What Doesn't Kill You Makes You Robust(er): How to Adversarially Train against Data Poisoning
Authors: Jonas Geiping
Liam H Fowl
Gowthami Somepalli
Micah Goldblum
Michael Moeller
Tom Goldstein
Venue: ICLR 2022 Submitted (2022)
Keywords: Data Poisoning
Poisoning Defenses
Adversarial Training
Empirical Defenses
Robustness
Security

Abstract: Data poisoning is a threat model in which a malicious actor tampers with training data to manipulate outcomes at inference time. A variety of defenses against this threat model have been proposed, but each suffers from at least one of the following flaws: they are easily overcome by adaptive attacks, they severely reduce testing performance, or they cannot generalize to diverse data poisoning threat models.  Adversarial training, and its variants, are currently considered the only empirically strong defense against (inference-time) adversarial attacks.  In this work, we extend the adversarial training framework to defend against (training-time) data poisoning.  Our method desensitizes networks to the effects of such attacks by creating poisons during training and injecting them into training batches. 
We show that this defense withstands adaptive attacks, generalizes to diverse threat models, and incurs a better performance trade-off than previous defenses.

Under review as a conference paper at ICLR 2022
WHAT DOESN'T T KILL YOU MAKES YoU ROBUST(ER): HOW TO ADVERSARIALLY TRAIN AGAINST DATA POI- SONING
Anonymous authors Paper under double-blind review
ABSTRACT
Data poisoning is a threat model in which a malicious actor tampers with training data manipulate outcomes inference time. A variety of defenses against this threat model have been proposed, but each suffers from at least one of the following flaws: they are easily overcome by adaptive attacks, they severely reduce testing performance, or they cannot generalize to diverse data poisoning threat models. Adversarial training, and its variants, are currently considered the only empirically strong defense against (inference-time) adversarial attacks. In this work, we extend the adversarial training framework to defend against (training-time data poisoning, including targeted and backdoor attacks. Our method desensitizes networks to the effects of such attacks by creating poisons during training and injecting them into training batches. We show that this defense withstands adaptive attacks, generalizes to diverse threat models, and incurs a better performance trade-off than previous defenses.
INTRODUCTION
As machine learning systems consume more and more data, the data curation process is increas- ingly automated and reliant on data from untrusted sources. Breakthroughs in image classification (Russakovsky et al., (015)) as well as language processing (Brown et al.) (020) are built on large corpora of data scruped from the internet. Automated scraping, which data is collected directly from online sources, leaves practitioners vulnerable data poisoning in which bad actors tamper with the data so that models trained on this data perform poorly or contain backdoors embedded in them (Gu al., 2019; Shafahi et al. 2018). These attacks present security vulnerabilities that persist even if the data is labeled and checked by crowd-sourced human supervision. In essence, be compromised if the input data is modified maliciously even if the modification appears minor and inconspicuous to human observer. This mounting threat has instilled fear especially in industry practitioners whose business models rely on powerful neural networks trained on massive volumes of scraped data (Kumar et al. 2020).
entire machine learning pipelines
In response this growing threat, recent works have proposed number of defenses against data poisoning attacks (Li et al., 22221b Goldblum et al. 2020). Existing defense strategies suffer from up to three primary shortcomings:
1. In exchange for robustness, they trade off test accuracy degree that is intolerable to real-world practitioners (Geiping et al. 2021).
2. They only robust specific threat models but not to adaptive attacks specially designed to circumvent the defense (Koh et al. 2018 Tan & Shokri 2020).
3. They apply only specific threat model and do not lend a generally applicable framework to practitioners (Wang et al. 2019), that could extend to novel attacks.
We instead propose a variant of adversarial training that harnesses adversarially poisoned data in the place of (test-time) adversarial examples. We show that this strategy exhibits both an improved bubustness-accuraal trade-off as well as greater flexibility for defending against a wide range of threats including adaptive attacks.
Adversarial training desensitizes neural networks to test-time adversarial perturbations by augmenting the training data with on-the-fly crafted adversarial examples (Madry et al. 2018). Similarly,

---

Under review as a conference paper at ICLR 2022
Split
Modify lower split misclassify upper split
Concatenate
Robust Training
Minibatch
poisoned data
Figure 1: Data poisoning attacks require approach to adversarial training robustify machine learning models against this threat model.
modify training data order to desensitize neural networks to the types of perturbations caused by data poisoning yet adapting this robust training framework data poisoning requires special consideration this new threat model. In contrast to contemporary work on training against data poisoning such Li al. a)02T) the defense dynamically trains robust model without the need to identify poisoned data points and provide benefits by adapting specifically to targeted data poisoning instead of using default adversarial training as in Tao et al. (2021) poisoned data generated offline as in Radiya-Dixx & Tramer (2021). We demonstrate the effectiveness of this framework at defending against range of data tampering threat models across several training regimes and visualize the impact of the defense in feature space. We further compare the proposed defenses to range of related defense strategies.
RELATED WORK
Data poisoning is class of threat scenarios focused on malicious modifications to the training data of machine learning model. See Goldblum al. (2020) for an overview of dataset security. Data poisoning attacks can either focus reducing the overall model performance or on backdoor attacks that introduce malicious behavior which is triggered by specific visual pattern or target image, thus breaking model integrity (Barreno et al. 2010).
In this work, we focus on attacks against model integrity which can further distinguished by nature of their trigger mechanism. In backdoor trigger allacks (Gu et al., 2019 Turner et al. (2018). the attack is triggered by a specific backdoor pattern or patch that can added to target images at test time, whereas targeted data poisoning (Shafahi et al. 2018 Zhu et al. 2019) is triggered by predefined target image. contrast to targeted poisoning, backdoor trigger attacks can be applied to multiple target images but require target modifications to be active during inference, while targeted attacks are activated by specific, but unmodified targets.
2. DATA POISONING ATTACKS:
Attacks can be further categorized by the precise training setup they anticipate their victims to employ, Some attacks assume that the victim will only fine-tune their model on the poisoned data will train a linear classifier on top of pre-trained feature extractor (Saha et al. 2020 Zhu et al. 2019 Shafahi al. 2018). These methods are effective against practitioners engaging in transfer learning with pre-trained models like those found in popular repositories such as Paszke et al. (2017). Other attacks work even when the victim trains their model from scratch on the poisoned data (Huang al. 2020) Geiping et al.) 2021). Generally, the simpler the victim's training procedure, the more readily the attacker can anticipate the effect of their perturbations and thus, the easier the attacker's 's job. Here, briefly detail prominent attacks against which test defense:
Feature Collision: Shafahi et al. 2018)) present a targeted poisoning attack that perturbs training data so that its deep features collide with those corresponding to target image. At the same time, the attack penalizes the size of perturbations applied to training data in order to maintain visual similarity between the original and perturbed images.
2

---

Under review as a conference paper at ICLR 2022
Aghakhani et al. 2020), as opposed to colliding images in feature space, surround target images in feature space to incraase the success of feature collision. This method improves on the work of Zhu et al. 2019), which constructs convex polytope around the target based on ensemble of models.
Bilevel Optimization: MetaPoison (Huang et al. 2020) generates poisoned data based on unrolling the bilevel objective encountered in targeted data poisoning for several steps and optimizing the unrolled objective over an ensemble of models at different stages in training, leading to an attack that robust new initializations in from-scratch training as well to changes in model architectures.
Witches' Brew (Geiping et al. 2021 instead approximates the bilevel objective by gradient matching, leading to computationally efficient attack (in comparison MetaPoison (Huang et al. 2020)), combining the efficiency of feature collision attacks with the success of bilevel approximations. This attack was already adapted to be effective against data augmentation and differential privacy, showing the need for strong defenses against adaptive attacks.
Gradient Matching:
Hidden Trigger Backdoor Attacks: Saha et al. (2020) present a backdoor trigger attack wherein the attacker modifies training data within an 8 bound to cause a vulnerability for a pre-selected to patch added to target validation images. This way, the attack is generalize from a chosen target to only a chosen patch with can be added to any data.
2.2 DEFENSES AGAINST DATA POISONING:
Defenses can be broadly classified into filter defenses which attempt to detect and remove or sanitize malicious training data, robust training algorithms which use a training routine that yields robust models even on malicious training data, and model repair methods that train models on poisoned data and attempt to repair the poisoned models after training. Filter defenses are easy to deploy as they simply add a pre-processing step, but they require extensive hyperparameter tuning and rely on the assumption that only small fraction of the dataset is poisoned. Furthermore, filter defenses can often be overcome by adaptive attacks (Koh et al., 2018 Tan & Shokri, 2020). Any defense reduces model performance (in terms of validation accuracy); filter defenses reduce performance as result of training the new model on fewer samples, robust training methods do so by deviating from standard training practices which are tuned for accuracy in order to increase robustness, and model repair methods harm accuracy by pruning away potentially important neurons.
Numerous options have been proposed for detecting poisoned data, Tran et al. (2018a) detect spectral signatures associated with backdoor triggers based on their correlation with the top right singular vector of the covariance matrix formed by feature representations, and additional detection scores can be found in Paudice et al.) (2018). |Prr et al.) (2020) detect poisoned data by clustering based on deep KNN, re-labeling data based on the nearest neighbors in feature space. [Chen et al.) (2019) cluster training data based on activation patterns in feature space. Another method detects images containing backdoor triggers by flagging images whose corresponding predictions do not change when they are combined with clean samples (Gao et al., 2019). Yet. any measure of "anomaly" that is used to filter images can also be used to generate poisoned data which minimizes the anomalous property, thus adaptively defeating the defense (Koh et al. 2018).
Robust training algorithms may incorporate strong data augmentations (Borgnia et al. 2020 random- ized smoothing (Weber et al.) 20200) may partition data into disjoint pieces and train individual models on the partitions, performing classification via majority voting at test-time (Levine & Feizi 2021). Another popular robust training strategy harnesses differentially private SGD (Abadi et al.
2016 Ma et al. 2019). Hong et al. (2020) note that the addition of noise in differentially private SGD is the primary factor controlling robustness to poisoning. However, differential privacy is an extreme and general definition of robustness to data manipulations, compared to robustness specific to data poisoning. This strategy consequently incurs a significant performance penalty (Jayaraaan & Evans) 2019), and these algorithms can even be adaptively attacked by modifying gradient signals during poison generation in the same manner as in the defense (Veldanda & Garg 2020). Other robust training schemes are proposed in Li et al. (202 Tao et al. (2021), b): Radiya-Dixi & Tramer (202I). Yet, adaptive attacks can bypass these defenses by creating poisoned data whose activation patterns mimic those of clean data. In order to counteract the loss of performance induced by pruning, some methods fine-tune the pruned model on clean data (Liu et al. 2018: Chen et al. 2020). But this process is only effective when the defender possesses large quantities of trustworthy clean data.
3

---

Under review as a conference paper at ICLR 2022
3 UNDERSTANDING ADVERSARIAL TRAINING FOR DATA POISONING
Adversarial training (Madry et al. 2018 Sinha et al. 2018) reduces the impact of test-time adversarial attacks and generally considered the only strong defense against adversarial examples. Adversarial training solves the saddle-poin point problem,
max Co + A, A.w)
min LE (nay)
(I)
AES
where Lg denotes the loss function of model with parameters 0. and the adversary perturbs inputs from a data distribution D, subject to the constraint that perturbation A is in S. (Per et al. (2020) notes that adversarial training against test-time evasion attacks already confers a degree of robustness against data poisoning a performance cost. Our proposed strategy an adaptation of adversarial training to poisoning, resulting in stronger defense that degrades performance less than differentially private SGD or adversarial training against evasion attacks. The capabilities of an attacker depend on its knowledge of the defender's 's training setup, so we now enumerate series of assumptions concerning the knowledge of the attacker and defender before presenting our framework in precise detail.
Preparing for strong threat model. In order to harden the model against a wide range of poisons, we train against strong surrogate attacker. The differences between the surrogate threat model and that of a real-world attacker concern the attacker's access to the defender's training routine. The surrogate attacker in our training algorithm is aware of the defender's 's training protocol (e.g. learning rate, optimization algorithm), architecture, and defense strategy but can neither influence training nor intercept random factors such as initialization and mini-batch sampling. In cases where the defender only re-trains a component of a model or fine-tunes the model, the exact baseline pre-trained model, including its parameters, is known to both parties. The attacker's 's trigger (a target image for targeted poisoning we do not assume that the defender possesses additional, vetted, clean data. In order to constrain the attacker, the defender chooses a ||. |l-norm perturbation budget against which they seek robustness.
specific patch for backdoor attacks) is unknown to the defender, and
Since the attacker possesses such strong knowledge concerning the defender's 's training routine, the threat it poses constitutes near worst-case analysis. If more factors, such as model definition or parts of the training protocol, are hidden from the attacker, then the quality of the defense can only improve. On the other hand, the defender needs to set a bound within which to be robust against attacks if there were no such limit, then the attacker could arbitrarily modify data.
3.1 ADVERSARIALLY TRAINING POISON IMMUNITY
In contrast to adversarial attacks at test-time, the objective for targeted data poisoning is itself already a bilevel objective. For given target w, with intended adversarial label yt. targeted data poisoning optimizes poisoned data points Tp. so that models trained with them exhibit low loss on the adversarial label ye instead of its original label yo:
N Σ
min L(x,, yc, (Ix)) Zy
s.t. 0(x) = arg min
C(a, Vi. O).
The optimal defense against this attack then develops into a two-player game between attacker and defender. While the attacker minimizes the loss on the target, the defender maximizes it with (0(x)) as defined above).
min max C(r,, Vt, 0(5,p + A)).
dP AES
However, this formulation reveals the central trick of adversarially training against data poisoning: Both attacker and defender modify the same variable x = Ip + A. This means that any known algorithm used approximate the bilevel poisoning objective for attack (e.g. Huang et al. (002 or Geiiiig et al. (2021), is valid as an approximation for a defense.
In the formulation above, the solution for the defender is simple, the defender optimize the poisoning objective Eq. (2) for p, and then set A = -Ip- Yet, in practice, the defender has neither knowledge of the specific data point na targeted by the attack, nor change their strategy in
4

---

Under review as a conference paper at ICLR 2022
response to the attacker. The optimal choice is hence to sample surrogate targets (It, ye) from the data distribution, optimize pp, and take steps to maximize [(x yt, 8):
min [( Vt. 0(x, + A)) max E (e AES
MDDD
(4)
zy
Instead of maximizing the adversarial loss, the defender can equivalently minimize the loss of the true label for the sampled target, i.e. [( Mo. 0). To implement the objective Eq. (4). we sample mini-batches of data and then first split these batches into two subsets of data, (Ip, yp) and (zz, Mt). with probability for data point to be placed in the first ("poison") split. For the sampled target 24 we then approximately minimize [( yt, ((r + A) through a known data poisoning attack, which yields Tp-- We can then update the model parameters to minimize [a Mo: f) and [(T Up- 0), Le. we train the model on the concatenated output of surrogate poisons and targets, as seen in Fig. This way we alternate between both steps in Eq. effectively.
Interestingly, contemporary work in Li et al. a)) instead maximizes adversarial loss in the outer objective and instead of optimizing in expectation detects and maximizes over detected poisoned data points 2p. leading to different approximation to Eq. (3). whereas work in Tao et al. (202T) focuses on adversarial training against attacks on model availability, which can be understood by replacing Eq. (2) by maximization of loss over all data points, leading to the min-max structure of Eq. (1)
as a defense. Work in Radiya-Dixit & Tramer (202T) instead proposes to optimize Eq. 44) by first generating additional poisoned data Tp for a fixed pretrained model and then training with all data, which corresponds to approximating the objective Eq. 4 in an offline fashion.
These considerations hold not only for targeted data poisoning, but likewise apply to poisoning with other modalities, such as backdoor triggers. those cases, both Tp and xe are optimized (or sampled) instead of only Zp.
Example: Defending against Gradient Matching. While our methodology can be applied to any data poisoning attack, there are several considerations to make when adapting the attack into format that is applicable and practical to run in each mini-batch of training. We detail these considerations for recent attack (Geiping et al. 202T).
The cosine similarity objective of the attack is originally evaluated on a clean surrogate model trained by the attacker and the attack optimized for a significant number of iterations (n 250 in the original work). To apply it during training, we first replace the clean model used in the attack with the current model in the current state of training - this is actually an advantage for the defender. While the attacker needs to create poisons on surrogate model, the defender can use the exact model, making it easier to create effective poisons. Secondly, similar to adversarial training, the number of attack iterations can be reduced. In practice, we choose creating a strong attack and spending limited time budget, as the attack is naturally stronger due to its basis in the current state of training. Third, we need to choose malicious labels yt- These labels could be chosen entirely at random, however then the average gradient over all targets would likely be small. However, poisoned data points in Geiping al. (2021) in practice chosen from the same class as the target adversarial label, and this choice can be replicated for the randomly chosen subset of poisoned data points Tp with labels yp by choosing y, the label that appears most often in Vp-
5 during the defense, as a compromise between
Other defenses can be viewed as special cases of poison immunity. This methodology general- izes and explains previous work on defenses against poisoning. In Borggia et al. (2020), strong data augmentations such as mixup (Zhang et al. 2018) and cutout (DeVries & Taylor 2017) are proposed as defenses against data poisoning. These defenses are special cases of the proposed poison immunity; when Algorithm is used to defend against a watermarking attack (which superimposes the target data onto poison data with low opacity), then the attack equivalent to mixup data augmentation with mixing factor =
Likewise, implementing Algorithm I aggainst patch attack attack reduces to patching randomly selected pairs of data points with random patch. this patch chosen to be uniformly gray, then this defense is exactly equivalent to the cutout data augmentation.
3.2 ADAPTIVE ATTACK SCENARIOS
Crucial for the design of new defense algorithms is their ability to withstand adaptive attacks, i.c attacks that can be modified to respond to a defense algorithm when the attacker is aware of the
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Modified iterative training routine for poison immunity.
Input: Split probability (0, 1).
repeat
Sample mini-batch of data {are, yi N==1
Split data randomly into two subsets Tpp 2y with probability
Draw malicious labels yt for It
Apply data poisoning attack to minimize (( Me. (x²)) via 2p Concatenate 2p. 24 into a new batch Zm with unchanged labels {yr ):=
Update model based on new data Fm
until training finished
defense. While this principle has been well-regarded in literature about adversarial attacks at test-time Carlini et al. (2019); Tramer et al. (2020), it has not been applied as rigorously for data poisoning.
The defense proposed in this work is exceedingly effective against non-adaptive models (evaluating the exemplary case of gradient matching), as the difference in training regimes leads to incorrect perturbations computed by the attacker that relies a pre-trained surrogate model. However, this would also be the case for most modifications to the training procedure, such as adding data augmentations or changing learning rates or optimizer settings. As such, we find that the optimal way to attack this defense is for the attacker to re-train their pre-trained model with exactly the same defense and the same hyperparameters. The attacker can then more accurately estimate the target gradient (for gradient matching) or target features (for feature collision). We also investigated the possibility of applying Algorithm [ during the optimization of poisoned data itself as an additional stochastic input modification. However, this modification weakens the attack by gradient masking, making too difficult for the attacker to optimize the poisoned data. This behavior mirrors (test-time) adversarial attacks, where it is non-optimal to add additional perturbations during the creation of an adversarial perturbation. As we will find in the next section, the defense has a major impact on the feature space of model, which may make it difficult to bypass it with better adaptive attacks.
4 ANALYSIS
To understand the effect of the proposed poison immunity scheme qualitatively, we conduct an analysis of feature space visualizations for several attacks. [Shafahi et al. (2018), who introduced feature collision attacks, illustrate their poisoning method by visualizing the feature space collisions between poisoned data and the target. These experiments are carried out in the transfer setting, where the feature representation of the model is fixed and known to both parties. We run Bullseye Polytope, recent and improved feature collision attack (Aghakhani et al., 2020), in strong attack setting of 500 poisoned examples for a ResNet-18 pre-trained on CIFAR- 10 and re-trained on the poisoned dataset. We visualize the feature space by plotting the projections of feature vectors of data to the vector connecting the centroids of the poison (base) class and the target class and it's orthogonal (generated using PCA) in the x-y plane and the softmax output for poison (base) class for each of the points on the 2 axis. This way we expect to see both classes to form separate clusters in feature space (x-y plane) and to be further separated by the poison class probability, which low for images from the target class and high for images from the poison class.
Fig. 22 shows the effects of an attack on baseline model. The poisoned images (red) move to collide with the target (black triangle) in feature space as seen by their overlap in the æ-y plane, while maintaining their original label (z-axis), subsequently leading to a misclassification of the target image as image from the poison class this is a feature collision attack. Fig. howeerr contrasts this collision with the effects of poisons on a defended model. Two effects stand out: First, the poisons, which were optimized to collide with the target, no longer cluster around the target (refer also to the 2D visualization Fig. 12). indicating that straightforward collisions are difficult to achieve against the robust model. Second, poisoned images close to the target are now predicted the target base class shown by their descend on the z-axis; While we would expect the first effect for any defense, the second effect really breaks the attack. the defended model is robust enough assign poisoned images a label that agrees with their feature representation, even though this assignment contradicts the given labels of these images. In essence, the poisoned images are treated like images from the

---

Under review as a conference paper at ICLR 2022
target class
poison class
poison
target
pakke claa ireeaaabiiy
Phlce tey
eirnnee erbento po GRAN imeeasty
Chrtacco ontrecy or
Perretcc controdd
dirteccs controlas
(a) Undefended model, clean (left) and fine-tuned on poisoned data containing feature collisions.
(b) Defended model, clean (left) and fine-tuned on poisoned data containing feature collisions. (right).
(right).
pohuee clate prebebiiigg
Jaree class arrraaatl
polure clas
diriace aisoice ent enthosmmnn
deiaaee ot
aisaace orthenenn
dimans shonp aissice
sertavce stong
contralds
(c) Undefended model, clean (left) and retrained on poisoned data containing gradient matching attacks (right).
(d) Defended model, clean (left) and retrained on poisoned data containing gradient matching attacks (right).
Figure 2: Visualization of the effects of poisoning attacks against undefended and defended model. Top: Feature collisions via Aighhhhaaa et al. (2020). Bottom: Gradient matching as in Geiping al. 202T). The target image is marked by black triangle and is originally part of the class colored blue. The poisoned images are colored red and are part of the class colored green. The z-y axis in each diagram corresponds to a projection of the principal direction separating both classes, while the confidence in the original target class is marked on the z-axis. The defended models generate a feature space in which poisons (red) behave consistently with the robust model, prevening collsions in Fig. 2b and alignment in Fig. 2d so that the target (black) remains correct.
target class but with a noisy label. This property reverses the attack. Instead of moving the target into the poison class, the poisoned images are drawn into the target class as matches their feature representation. The model stays consistent and is able to defend against the strong attack analyzed here.
In addition to feature collision attacks, in Fig. we analyze the defense against the gradient matching attack of Geiping et al. (202I) | in the From-scratch setting, where the model is fully re- traiedded. The attack can be seen to be effective in Fig. 2 changing the decision boundary of the model to fit the target without collisions by clustering poisons opposite to the target in feature space, significantly moving the target. However, this is prevented by the defense as seen in Fig. 24] The robust model is not modified by the clustering of poisoned images, and outliers seen in the undefended model are again reclassified as the target class leading to consistent decision. An interesting side effect of the defense for both attacked and clean models is that the model itself is generally less over-confident in its clean predictions. We compute similar outcomes also for other attacks, such as patch attacks, which we show in the appendix in Fig. to Fig. 15
EXPERIMENTS
This section details a quantitative analysis of the proposed defense for the application of image classification with deep neural networks. To fairly evaluate all attacks and defenses, especially in light of /Schwazzschild et al. (2020) discussing the difficulty in comparing attacks across different evaluation settings, we implement all attacks and defenses in single unified framework, which we will make publicly available. For all experiments, we measure avg. poison success over 20 trials, where each trial represents randomly-chosen attack trigger from random class and separately
7

---

Under review as a conference paper at ICLR 2022
Table 1: Quantitative result for several attacks and their defense by poison immunity with 0.75, showing avg. poison success with standard eтror (where all trials have equal outcomes, we report the worst-case error estimate 5.59%). Additional details about each attack threat model can be found in the Appendix [.2] The proposed defense significantly decreases success rates over a wide range of attacks and scenarios without any hyperparameter changes. This table evaluates gradient matching (GM) with both squared error (SE) and cosine similarity (CS). The evaluation column with the strongest attack uses gradient matching with cosine similarity.
Attack
Scenario
Undefended
Defended
Same Attack
Strongest Attack
MetaPoison
From-scratch
69.50% (1:9.44)
10.00% (19.49)
10.00% (19.49)
Gradient Matching (CS)
From-scratch
90.00% (++6.11
0.00% (+5.59)
0.00% (+5.59)
Bullseye Polytope
Fine-tuning
80.00% (1.8.44)
0.00% (15.59)
8.33% (17.98)
Bullseye Polytope
Transfer
100.00% (+:5.59
10.00% (+6.71)
0.00% (+55.9)
Poison Frogs
Transfer
100.00% (+5,
15.00%
(#7.98)
0.00% (+5.59)
Gradient Matching (SE)
Transfer
95.00% (#4.87)
0.00% (=5.59
5.00% (+:4.7)
Hidden Trigger Backdoor
Transfer
55.59% (155.65)
24.78% (1-6.82)
3.32% (1.0.79)
attacked and trained model. The sampling of randomized attack triggers is crucial estimate the average performance of poisoning attacks, which are generally more effective for related class labels. We discuss additional experimental details in Appendix E
We focus on defending against threat models, in which the attacker may modify some percentage of training data within an 8 bound, i.c. change every pixel slightly. This covers all mentioned targeted data poisoning attacks as well as hidden trigger backdoor attacks (Saha al. 2020).
Defending against f threat models
To evaluate the proposed defense mechanism thoroughly, we consider variety of attacks in different scenarios and distinguish three scenarios with increased difficulty for the attacker, transfer where the defender only re-trains the last linear layer of a model, fine-tuning, where the defender re-trains all layers, and from-scratch where the defender trains a completely new model.
We first apply the proposed defense against range of attacks and settings in Table I choosing s = 0.75 for all targeted data poisoning attacks and no additional modifications. All attacks shown adaptive, if possible. In the fine-tuning and transfer scenarios, the pre- pre-trained model is defended but known to the attacker exactly. In all cases, we observe that while the attacks are highly effective against an undefended model, our defense steeply reduces their effectiveness. These encouraging results suggest that the proposed methodology is a strong strategy that can be robustly applied across range of attacks and may also generalize to future attacks.
natural question to ask whether this defense, which trains against one specific surrogate attack, can be circumvented when the real attacker utilizes different attack. Surprisingly, we find in Table [ that using gradient matching, strong attack, as surrogate during training successfully defends against a range of other attacks. This corroborates findings in Madry al. (2017) where was shown that training with the strongest test-time ad adversarial attack, also defends against weaker test-time attacks an mechanism which fortunately appears to also hold when defending against data poisoning.
Comparison to Other Defenses In this subsection, we compare the proposed defense to other existing defense strategies against data poisoning including differentially private SGD,, adversarial training, various data augmentations, and filter defenses. For differentially private SGD and adver- sarial training, we test several noise levels and perturbation budgets, respectively. When comparing to filtering defenses, we allow optimal hyperparameter choice by supplying the exact number of poisons in the training set, although this information would be unknown in practice. We analyze poison immunity training with varying levels of to show the trade-off of performance and security. We test the gradient matching attack proposed in Geiping et al. (202I). for a ResNet- 18 trained on CIFAR-10 10 with budget 1 and 16 (the same setting proposed in that work). While previous
8

---

Under review as a conference paper at ICLR 2022
Undiltened
Undefended (Baseline)
Spectral Sigusuns
Puison Immunity (propoeed)
Differentilly private SGD
Advensarial Training
Ang. Polcon Success
CunMix
Unaversal Adversarial Training
Filler Delense
0.8
Augmestation
Marap eqoel)
0.4
p=0.25
Chastering
open2
-05-116
Inpiliinntr opsnd
<psmG
pmo.5
pmot0225
p-0475
Figure 3: Avg. Poison Success versus validation accuracy for various defenses against the gradient matching attack of (Geiping et al. (02I)) in the from-scrated setting. The baseline undefended model is shown in blue, the proposed defense in red. The differentially private SGD is shown for noise values from 0.0001 0.01. The proposed defense is strong trade-off of robustness and accuracy.
defenses were shown to be ineffective in Geiping et al. (2021), we now show in Fig. that the proposed poison immunity defense is extremely effective defense in the from-scratch setting, yielding a much stronger protection than filter defenses, but with only mild trade-off in validation accuracy compared to differential privacy and classical adversarial training.
Releasing Robust Models So far, we have considered scenarios in which the defense described in Algorithm [ always active, even during the fine-tuning procedure in the transfer setting. However, especially in the trans- fer setting, we are interested in the inherent robustness of models and its transferability. We thus analyze setting in which the base model is trained robustly via poison immunity, but the last layer is re-trained non-robustly poisoned data. For the bullseye polytope attack in the transfer setting, this approach leads to an avg. poison success of only 20.00% (0.00%((#..99) (..94) and validation accuracy of 88.66 when using a base model trained robustly with poison immunity via 0.75, compared to the best de- significant part of the overall robustness is already encoded into the base model. The inset figure visualizes that the target confidence remain consistent, even if the fine-tuning is nonrobust.
cines probablliyy
distancc arthonormat
distance elong controlds
target class
posson class
poison
target
fense 10.00%(=6.71))
Figure 4: Feature Defense. A robust base model can withstand feature colli- sion attacks even when fine-tuning non- robustly on poisoned data. Note how no collision forms around target and the model remains robust.
CONCLUSIONS
In this work, we adapt adversarial training to defend against data poisoning attacks. In addition to demonstrating the strong defensive capabilities our method, poison immunity, we analyze the feature space of defended models and observe mechanisms of defense. We further evaluate the proposed defense against a variety attacks on deep neural networks for image classification, successfully adapting to and defending against feature collisions, gradient matching, and hidden trigger backdoors. We stress that we believe this strategy to be a general paradigm for defending against data tampering attacks that can extend novel future attacks.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Data poisoning attacks have the potential to disrupt machine learning pipelines and hinder data collec- tion and curation. Especially when collected data is of unknown quality and possibly contaminated with poisoned samples, we hope that defense strategies like we propose can be useful in mitigating harmful effects.
REPRODUCIBILITY STATEMENT
We refer to the appendix for details for all attacks and defenses evaluated in this work and regarding the experimental setup. We further supply code that allows for the reproduction of all experiments in a consistent framework.
REFERENCES
Martin Ahadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Tlya Mironov, Kunal Talwar, and 7hang. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS 16. pp. 508-318, Vienna, Austria, October 2016. Association for Computing Machinery. ISBN 978- 450114440-4199----
doi: (0.11445/97767992978811
Hojjat Aghakhani, Dongyu Meng, Yu- -iang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye Polytope: Scalable Clean Label Poisoning Attack with Improved Transferability. arXiv:2005.0019) (cs, stat), April 2020. htt.ttt::/// http://xxxvvv..// abs/ 2005. 0C191
Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. The security of machine learning, Machine Language, 81(2):221-148 November 2010. ISSN 0885-6125, doi:
Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta, Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradcoff. arXiv:2011.090 [es), November 2020. URLptttp:/arxxte org/ 2abps/2211.000
2ttpo//////.////..222222 09527
Fitan Borgnia, Jonas Geiping, Valeriia Cherepanova, Liam Fowl, Arjun Gupta, Amin Ghiasi, Furong Huang. Micah Goldblum, and Tom Goldstein. DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations. arXiv:2103.02079 [es], March 2021. URL /
arxiv org/ abs/2103. 72079
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-Vosss Gretehen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In 34th Conference Neural Information Processing Sysiems (NeurIPS 2020), December 2020. URL http:ttp://aae org. abs 2005 14165
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Tan Good- fellow, Aleksander Madry, and Alexey Kurakin. On Evaluating Adversarial Robustness. arXiv: 1902.06705 [cs, star], February 2019. URL httphhtttt/xxiii org/ abs/ 00..0022..000
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy. and Biplav Srivastava. Detecting Backdoor Attacks Deep Neural Networks by Activation Clustering. In SafeAI@AAAL, January 2019, /ttp/ttt::///pecceeeee
Xinyun Chen, Wenxiao Wang, Chris Bender, Yiming Ding, Ruoxi Jia, Bo Li, and Dawn Song. REFIT: A Unified Watermark Removal Framework for Deep Learning Systems with Limited Data. 1911v7991..02200 [cs], January 2020. URL http::htpp://tt/t http://rgg/////setoooooooooss
httpp:://aaxii..rrg/ab////// 1911 07205
Terrance DeVries and Graham W. Taylor. Improved Regularization of Convolutional Neural Networks with Cutout,
[es], August 2017. htt.:/tpp://aavv.oo///a/a/////// wttp://.wgg/bb////ooo/ C4552
Yansong Gao, Change Xu, Derui Wang. Shiping Chen, Damith C. Ranasinghe, and Surya Nepal. STRIP: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC '19, pp. 113-125, New York, NY, USA, December 2019. Association for Computing Machinery. ISBN 778-1-4503-76284) doi: 10. 10.1145/3359789.33599
10

---

Under review as a conference paper at ICLR 2022
Jonas Geiping, Liam H. Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor. Michael Moeller, and Tom Goldstein. Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching. International Conference on Learning Representations, April 2021. URL httptt:://ooeeeevlliieeee http:///wwweevlee.eoooooo/oooooe wwtp:/uune/eoeuet///////
010InfLibe
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset Security for Machine Learning: Data Poisoming, Backdoor Attacks, and Defenses. arXiv:2012. 10544 [cs], December 2020. /Rllctte://axxi..orr///../ 10544
Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu. MaxUp: A Simple Way to Improve Generalization of Neural Network Training. arXiv:2002.09024 [cs, stat], February 2020.
2.09/2022.00 09024
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Evaluating Backdooring Atlacks on Deep Neural Networks. IEEE Access, 7:47230-47244, 2019. ISSN 2169-3536. doi: 10.1109/ACCESS 2019.2909068.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. arXiv:151200338. [cs]. December 2015. URL (Rttt:////axxaii wwwpp//ag/ag..o.oo.o...r/r.eeeea 03382
Sanghyun Hong, Varun Chandrasekaran, Yigitean Kaya, Tudor Dumitraş, and Nicolas Papernot. On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping. arXiv:2002. 1499 [cs], February 2020. URL /tattt::/arxiyve org/ abs 2002. 11497
W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstcin. MctaPoison: Practical General- purpose Clcan-label Data Poisoning. In Advances in Neural Information Processing Systems, volume 33, Vancouver, Canada, December 2020. hat.:///ee...0000e.77777777a75aa
neurips. /apperapp.r...e files/
.html
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial Transformer Networks.
In Advances in Neural Information Processing Systems 28, pp. 2017-2025. Curran Associates, Inc., 2015.
UtI/http:///paperr ni www..ippppoooo....mm
-aan-ttttaaaooooaaa netorkss.
Bargav Jayaraman and David Evans. Evaluating Diffcrentially Private Machine Learning in Practice. In
28th (USENIX) Security Symposium (/USENIX) Security 19), pp. 1895 1912, 2019. ISBN 178-1-999111
06-9. httthhtts:://// //w:w///wwwwww.o/.. usenix
jayaraman
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger Data Poisoning Attacks Break Data Sanitization Defenses,
(cs, stat), November 2018. URL/ccpp://ardi..oo////aa/
Alex Krizhevsky, Learning Multiple Layers of Features from Tiny Images. 2009. URL https://wwk. pd.
werooto..eu//kiie www.rriz/eearnnng... Sernoo---auttteeee
R. S. Siva Kumar, M. Nyström, Lambert, A. Marshall, M. Goertzel, A. Comissoneru, M. Swann, and S. Xia. Adversarial Machine Haaming-Industry Perspectives. In 2020 JEEE Security and Privacy Workshops (SPW). pp. 69 69-75, May 2020, doi: 200/://PW50008222000000
Alexander Levine and Soheil Feizi. Deep Partition Aggregation: Provable Defenses against General Poisoning Attacks. In International Conference on Learning Representations, 2021. URL http:tts://pppeeiiiiiiee
net /
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-Backdoor Learning: Training Clean Models on Poisoned Data. In Thirty-Fifth Conference Neural Information Processing Systems, May net worrm/orummmm/m 1 6UcCLRM
2021a. URL
Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor Leamingg A Survey, arXiv:2007.08745 [cs], February 2021b. httt:t///aaxii// org/ abs /tbp/2007..om 08745
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks. In Research in Attacks, Intrusions, and Defenses, Lecture Notes in Computer Science, pp. 273-294, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00477055 doi: 00000//978-3000-0077700
Yuzhe Ma, Xiaojin Zhu, and Justin Hsu. Data Poisoning against Differentially-Piin Private Learners: Attacks and Defenses. In Proceedings of the Twenty-Eighe International Joint Conference on Artificial Intelligence. pp. 4732-4738, Macao, China, August 2019. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-0-999241 1-4-1, doi: 10. 201.6////jcaa.221166617

---

Under review as a conference paper at ICLR 2022
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks, arXiv:/706.0660 06083 fes, stat), June 2017. URL
06083
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations. February 2018. URL http::/::////peeew/eeeee www.ccpenvvvicccco net/ forum? d 1B
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in Py Torch. In NIPS 2017 Autodiff Workshop, Long Beach, CA, 2017. URL
wwt.eeewwwwwwwweeeeeeeeeeeeeeeee net wttpte/e//ooooomnn//oaaaaaaaaaa
Andrea Paudice, Luis Muñoz--onzzlez, Andras Gyorgy. and Emil C. Lupu. Detection of Adversarial Training [cs, stat], February 2018.
Examples in Poisoming Attacks through Anomaly Detection. URL /tRL/tttt:///rxiv.orrr httpr///bs//88..... 03041
Nechar Peri, Neal Gupta, W. Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John P. Dickerson, Deep k-NN Defense Against Clean-L Data Poisoning Attacks. In Computer Vision ECCV 2020 Workshops, Lecture Notes in Computer Science, pp. 55-70, Cham, 2020. Springer International Publishing. ISBN 033-6-000-- doi: (L0079978-3-000-66-1155
Evani Radiya-Dixit and Florian Tramer. Data Poisoning Won't Save You From Facial Recognition. In ICML 2021 Workshop Adversarial Machine Learning, June 2021. URL /tttps:://ppennrviiew..to net. wtw.rrrrewiieeeeeeeumemo
1d=
ap5PEix2H
Sukrut Rao, David Stutz, and Bernt Schiele. Adversarial Training against Location Adversarial Patches. 2:X1::2005.02313 [cs, stat], December 2020. URL htt.:ttt///aexi...oo/aaaa///////
02313
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Lii Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-255 December 2015. ISSN 1573-1405. doi: 10. 12101115-1222001116662
Aniruddha Saha, Akshayvanu Subramanya, and Hamed Pirsiavash. Hidden Trigger Backdoor Attacks. Proceed- ings of the AAAI Conference on Artificial Intelligence, 14907):11957-11966 11965, April 2020. ISSN 2374-3468.
doi: 10. wtt.9//aaa....i0..... 16871.
Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P. Dickerson, and Tom Goldstein. Just How Toxic is Data Poisoning? a Unified Benchmark for Backdoor and Data Poisoning Attacks, arXiv:2006.122: [cs, stat], June 2020. WRRLbbtpp//arxxve
org/ ab0/ 12557
Ali Shafahi. W. Ronny Huang, Mahyar Najibi, Octavian Sueiu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18 pp. 6106-61 16, Red Hook, NY, USA, December 2018. Curran Associates Inc.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying Some Distributional Robustness with Principled Adversarial Training. In International Conference on Learning Representations, February 2018. URL
htttps://
net wttp:/ommmmmmaakkkeeaa2
T.J. Lan and R. Shokri. Bypassing Backdoor Detection Algorithms in Deep Learning. In 2020 IEEE European Symposium on Security and Privacy (EuroS P). pp. 175-183, September 2020. doi: 10.1 109/Eu49SP44549 2020.00019.
Lue Tao. Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training. Thirty Fifth Conference on Neural Information Processing Systems, May 2021. URL htttts:://pppneeeeeeeee
net. http://ormmmddddImmm.ee 39789067]
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On Adaptive Attacks to Adversarial Example Defenses. In 34th Conference on Neural Information Processing Systems (NeuriPS 2020), Vancouver, Canada, December 2020. URL //ap:://///xvvv/ arwiv org/ abs 2002. 08347
Brandon Tran, Jerry Li, and Aleksandor Madry. Spectral Signatures in Backdoor Attacks. In Advances in
Neural Information Processing Systems 31, pp. 8010- Curran Associates. Inc., 80. htt
papers nips. cc/ wttpccc/ppppee/ee0 -pect wiigrruurssinnnsasssseeeeee waw.eeeoooinaabocooooooo-oooo--e attaoorrrrtttt...... pdf
Brandon Tran, Jeryy Li, and Aleksander Madry. Spectral Signatures in Backdoor Attacks. 4rX1v:88/1000336 fcs. stat], November 2018b. Cw..nt.p://arsiv.orr///bbb (00600
12

---

Under review as a conference paper at ICLR 2022
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-Label Backdoor Attacks. openreview: September 2018. URL
Akshaj Veldanda and Siddharth Garg, On Evaluating Neural Network Backdoor Defenses. arXiv:2010.12180 [cs), October 2020.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks. In 2019 JEEE Symposium on Security and Privacy (SP), pp. 707-723, San Francisco, CA, USA, May 2019. IEEE, ISBN 978-1-5386- 6660-9. doi: 10.1109/5P2019.00033
Maurice Weber. Xiaojun Xu, Bojan Karlas, Ce Zhang, and Bo Li RAB: Provable Robustmess Against Backdoor Attacks. arXiv: [cs, stat], June 2020. URL http://arxiy http:///aaxxvv..//bb.bs
abs abs///s///2003 08904
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. Pro- ceedings of the IEEE/CVE International Conference on Computer Vision, pp. 6023-6032. 2019.
URL htttp:://openacees: thecvÊ com/ content ICCV 2019/htm1 Yur CutMix Reqularization Strategy to Train Strong Classifiers With Loca zable html
Peatures_Iccy_
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond Empirical Risk
Minimization. In International Conference on Learning Representations, February 2018. URL https
/openreview net/ www.:nee.ffrumm. Forummid-rrr Ddpl- Rb
Chen Zhu, W. Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable Clean-Labe Poisoning Attacks on Deep Neural Nets. International Conference on Machine Learning, pp- 7614-7623. PMLR, May 2019. URL
mnr.. wwwpprressveee V zhul 9a, html
A DEFENDING AGAINST e THREAT MODELS
In this section we further investigate patch attacks as Gu al. (2019) who trigger backdoors by adding an l bounded patch to training images in a given class, causing a network trained on these images associate the patch with the given class. Then, the attacker patches test-time images (of different class) with the same patch in the hopes that the network mis-classifies the images. Most backdoor attacks fall under e threat model, more specifically a limited e norm for connected rectangular region, patch. Defending against such attacks through adversarial training techniques is not as well-understood, even for test-time adversarial attacks (Rao et al., 2020), as for threat models. Although Eq. (4) remains valid, finding the arg min, i.e. the worst-case patch to apply, appears difficult from an optimization perspective.
We evaluate this threat model for the example of challenging backdoor trigger attack in Table 2 where we insert a 4x4 trigger patch into 5% of the CIFAR-10 training data. This patch either is a noisy checkerboard, and such out-of-distributiin for CIFAR- 10, or firefox logo, i.e. potentially in-distribution semantic feature. The evaluated filtering defenses do not defend well against both patches, whereas differentially private SGD provides some protection against the noise patch. When comparing different instances of the proposed poison immunity strategy, we can compare several strategies approximate Eq. { We can sample randomized patches of varying sizes ([large] noise patch). We can optimize for a worst-case patch content at a random position (optimized patch). We can sample randomized, but in-distribution, patch (image patch) by sampling patch from another image in the batch, which is equivalent to CutMix (Yun et al. 2019). The results show that optimizing the worst-case patch fails to significantly improve upon a well-chosen sampled patch. For the noisy checkerboard, we reach the best results when like-wise sampling random noise patches during training, although sampling image patches is also competitive. With the semantically meaningful firefox logo however, we find that training with noisy patches actually only incurs robustness such noisy patterns, and only limited robustness to other patterns. In contrast, the image- based patch can succeed in both settings.
13

---

Under review as a conference paper at ICLR 2022
Table Avg. poison success for various defenses against backdoor triggers attacks, for an attack via 4x4 patch on 5% of training data. Top table: Baseline and defenses via filtering and differential privacy. Bottom table: Variations of adversarial training against poisons. We evaluate each on a noisy checkerboard patch (Noise) and a patch with semantic meaning, firefox logo (Sem).
Filtering Defenses deep-KNN
Undefended
DP-SGD
Patch:
Spectr. Sign.
Act. Clust.
Noise
69.89
(+:4.47)
68.62 (#7.97)
73.34 (+ 5.25)
79.72 (+6.82)
50.24 (+9.41) 94.40 (+3.53)
Sem.
79.68
(++4550)
83.93 (+-4.55
76.96
(+5.07)
86.84 (+3386)
Poison Immunity
Patch:
Noise Patch
Large Noise Patch
Optimized Patch
Image Patch 25.80 (#33993) 25.62 (#3.85)
Noise
29.77
(1...5)
33.87 (#5.64)
62. (+: 12.28) 79.86 (=..2
Sem.
66.06 (#5.17)
54.45 (#6.11)
0.7
Nu Poison Success
0.6
0.5
0.4
0.2
0.1
10
Attack Budget Data
Figure 5: Effectiveness of the Defense against large attack budgets for the case of a defense against gradient matching in the cerm-scrathh setting on ResNet-18. In this scenario, the attack already succeeds when 1% of the data is poisoned, but the attacker can increase their budget further and poison more of the data. The defense still effectively reduces poison success even with large levels of poisoned data with roughly logarithmic scaling. Note that 10% is the limit for this attack on CIFAR- 10, for which all data points from the poisoned class are modified.
B ABLATION STUDIES
This section contains ablation studies and additional information. We visualize the effect increased poison budgets in Fig. 5 Table 3 contains information clean validation accuracy and runtimes for all experiments in Table In the same way Table exxendss Table 2 with additional information.
C VISUALIZATIONS
We repeat the three-dimensional visualizations shown Fig. 2 in the main work for additional attacks. For easy comparison, also repeat the figures appearing in the main work. We show feature collisions via Poison Frogs Fig. and repeat Bulleye Polytope in Fig. 7 We then repeat gradient matching in Fig. 88 in comparison to backdoor trigger in Fig. 9 and gradient matching (SE) in Fig. 10
All three-dimensional visualizations are also shown in two dimensions, showing Poison Frogs in
Fig. III Bullseye Polytope in Fig. 12 gradient matching in Fig. [3 backdoor triggers in Fig. 14 and
gradient matching (SE) Fig. 15
D COMPARISON TO OTHER DEFENSES FOR TRANSFER LEARNING
We additionally compare poison immunity to other existing defenses in the transfer setting, where only the last layer is re-trained on poisoned data. We test feature collisions via Bullseye Polytope (Aghakhani et al. 2020). also for a ResNet-18 18 trained on CIFAR- 10 with a budget 1% poisoned data within ' bound of æ 16. This a setting that is ideal for commonly used filtering defenses, as a large number poisoned data is collided with the target image, which can be detected and filtered, while the setting is difficult for robust training methods, both due the large perturbations
14

---

Under review as a conference paper at ICLR 2022
Table 3: This is the same table Table in the main body, with additional information of natural validation accuracy and timing for each run. Best viewed on screen. All shown timings are wall-clock time for an NVIDIA GTX-2080ti with 4 assigned CPUs. Some timings missing (due machine heterogeneity), but can be inferred from other rows in the same block (i.e. all transfer experiments take roughly the same amount of time for the "Strongest Attack" defense).
Attack
Secnario
Undefended
Defended Strongest Attack
Attack
Poison Acc.
Nat.Acc.
Time
Poison Acc.
Nat.Acc.
Time
Poison Acc.
Nat.Acc.
Time
MetaPoison
From scratch
69.50% (4.9.4))
86.11%
0:28:18
10.00% (1:9.49)
81.34%
2:06:48]
10.00% (1:9.49)
78.40%
1:09:47
Gradient Matching (CS)
Prom-scratch 90.00% (::6.71)
92.01%
0:15:48
0.00%
5.9)
88.32%
4:13:51
0.00%
(::5.59)
88.32%
4:13:51
Bullseye Polytope
Fine tuning
80.00% (.94)
0:16:31
0.00%
(.55)
88.49%
0:59:58
8.33%
(:17.98)
88.14%
Bullseye Polytope
Transfer
100.00% (::5.59)
91.97%
0:08:04
10.00%
(1:6.71)
88.64%
0:39:53
0.00%
(::5.59)
90.34%
Poison Frogs
Transfer
100.00%
(+5.59)
91.93%
0:08:46
15.00%
(#:7 .88
88.50%
0:39:53
0.00%
(#:5.59)
90.54%
Gradient Matching (SE)
Transfer
95.00% (#:4.87)
92.02%
0:08:08
0.00%
(:15.5
87.68%
0:41:08
5.00%
(::4.87)
90.62%
Hidden Trigger Backdoor
Transfer
55.59% (#5.65) 65) 86.07%
224.78% (#:6
86.44%
3.32% (+00799)
87.94% 0:40:26
Table 4: Avg. poison success for various defenses against backdoor triggers attacks, for an attack via a 4x4 patch on 5% of training data. Extended version of Table 2 columns. Best viewed on screen.
but note the flip in rows and
Patch:
Noise
Somantic
Poison Acc.
Nat.Acc.
Time
Poison Acc.
Nat.Acc.
Time
Undefended
69.89% (#4.47)
91.77%
0:16:47
79.68% (#4.50)
91.67%
0:16:49
Spectral. Sign.
68.62% (17.97)
77.31%
0:23:09
83.93% (14455)
77.42%
0:23:06
Deep-KNN
73.34% (+5.25)
91.27%
2:46:10
76.96% (=5.07)
90.97%
2:45:59
Act. Clustering
79.72% (#6.82)
81.06%
0:28:34
86.84%
(#3.86)
80.43%
0:28:36
DP-SGD
50.24% (1.9.1))
69.33%
0:17:47
94.40% (1:3.53
69.24%
0:17:43
Noise Patch
29.77% (#6.75)
92.00%
0:20:20
66.06% (#5.177
91.88%
0:20:21
Large Noise Patch
33.87% (1:5644)
88.91%
0:19:43
54.45% (1:6.11)
89.02%
0:19:28
Optimized Patch
62.70% (4:12.28)
91.77%
6:08:03
79.86%
(1-6.44)
92.01%
6:00:42
Image Patch
25.80% (4:3.93)
91.41%
0:16:45
25.62% (#:385)
91.15%
0:16:40
Table 5: Additional quantitative result for several attacks and their defense by poison immunity with 0.75, showing avg. poison success with standard error (where all trials have equal outcomes, we report the worst-case error estimate 5.59%). Additional details about each attack threat model can be found in the Appendix
Attack
Scenario
Dataset
Model
Undefended
Defended
Same Attack
Strongest Attack
Gradient Matching
From-Scratch Transfer
GTSRB ResNet18 40.00% (1.15.49)
10.00% (.L9.49)
10.00% (1.949))
Bullseye Polytope
GTSRB
ResNet18 10.00% (1.9.49) 0.00% (155.5) 0.00% (15.59)
15

---

Under review as a conference paper at ICLR 2022
target class
poison class
poison
target
poiken daso probaaaar
Of
distaton sang centroks
distston BlAng
stang
stang centrois.
(a) Undefended model, clean (left) and fine-tune fic-tuned on (b) Defended model, clean (left) and retrained on poi- poisoned
soned
data containing feature collisions. (right).
data containing gradient matching attacks (right).
Figure 6: 3D Visualization of feature collision attack (via Poison-Frooe Frogs) against an undefended and defended model. The defended model significantly hinders feature collisions. The target image is marked by black triangle and is originally part of the class marked in blue. The poisoned images are marked in red and are part of the class marked in green.
pakan dass probisati
Delum das probisatt
dilanee orthomore
aitan controu
ditante
distunee contrenn.
dobunee contrant
oistance controsa
(a) Undefended model, clean (left) and fine-tuned (b) Defended model, clean (left) and retrained on poi- poisoned
soned
data containing feature collisions. (right).
data containing gradient matching attacks (right).
Figure 7: 3D Visualization of the effects of a feature collision (Bullseye Polytope) attack against an undefended and defended model. The defended model significantly hinders feature collisions. The target image is marked by black triangle and is originally part of the class marked in blue. The poisoned images marked in red and part of the class marked in green. Notably the strong collision seen in the baseline is inhibited by the defense.
target class
poison class
poison
target
poiim daas pobatneeyy
destance artooamma
disance contrics
ditanca centrisat
ditance centesiai
chitance contriide
(a) Undefended model, clean (left) and retrained (b) Defended model, clean (left) and retrained on poi- poisoned
soned
data containing gradient matching attacks (right).
data containing gradient matching attacks (right).
Figure 8: 3D Visualization of the effects of a gradient matching attack (Witches' Brew) against an undefended and defended model. The defended model significantly hinders feature collisions. The target image is marked by black triangle and is originally part of the class marked in blue. The poisoned images are marked in red and are part of the class marked in green.
16

---

Under review as a conference paper at ICLR 2022
pivon deis proddaai prodadans
Dekon das probiasn
Weturee contront.
dobanse contrns
eibhane rostress
covtrous
(a) Undefended model, clean (left) and retrained (b) Defended model, clean (left) and retrained on poi- pnisoned
soned
data containing backdoor patches. (right).
data containing backdoor patches (right).
Figure 9: 3D Visualization of the effects of backdoor trigger patch attack against an undefended and defended model. The target trigger is applied to number of target images shown in black and originally part of the class marked in blue. The poisoned images are marked in red and are part of the class marked in green. Note how the black datapoints associated with the poison class in the undefended case, but correctly associate with the target class in the defended case.
pakven daso prebaniyy
2. peebiasry
das rabuniy
didance erthnnated
Rizaace contrss
delance or
distane covtrens
dotante controns
distance covtronn
Gelavre or
(a) Undefended model, clean (left) fine-tuned (b) Defended model, clean (left) and fine-tuned on poisoned
poisoned
data containing gradient matching attacks (right).
data containing gradient matching attacks (right).
Figure 10: 3D Visualization the effects a gradient matching attack (Witches' Brew with squared loss) against undefended and defended model. The defended model significantly hinders feature collisions. The target image marked by a black triangle and is originally part of the class marked in blue. The poisoned images are marked in red and part of the class marked in green. We see that the attack effectively moves the decision boundary opposite of the target in feature space. However this is completely prevented in the defended model.
target class
poison class
poison
target
Rlaaree OFTH SETTR
dicance cantrits
desacs slong contrids
dicance contrals
dicance cantroie
(a) Undefended model, clean (left) and fine-tuned on (b) Defended model, clean (left) and retrained on poi- poisoned
soned
data containing feature collisions. (right).
data containing gradient matching attacks (right).
Figure 11: 2D Visualization of feature collision attack (via Poison-Froo Frogs) against an undefended and defended model. The defended model significantly hinders feature collisions. The target image marked by black triangle and is originally part of the class marked in blue. The poisoned images are marked in red and are part of the class marked in green.
17

---

Under review as a conference paper at ICLR 2022
(a) Undefended model, clean (left) and fine-tuned Defended model, clean (left) and retrained on poi- poisoned
soned
data containing feature collisions. (right).
data containing gradient matching attacks (right).
Figure 12: 2D Visualization of the effects of a feature collision (Bullseye Polytope) attack against an undefended and defended model. The defended model significantly hinders feature collisions. The target image marked by black triangle and originally part of the class marked in blue. The poisoned images are marked in red and are part of the class marked in green. Notably the strong collision seen in the baseline is inhibited by the defense.
diaan ottornmaal
datuncz conbrits
dicunce contrott
dacance controlds
dalance contrets
(a) Undefended model, clean (left) and retrained on (b) Defended model, clean (left) and retrained on poi- poisoned
soned
data containing gradient matching attacks. (right).
data containing gradient matching attacks (right).
Figure 13: 2D Visualization of the effects of gradient matching attack (Witches' Brew) against an undefended and defended model. The defended model significantly hinders feature collisions. The target image is marked by black triangle and is originally part of the class marked in blue. The poisoned images are marked in red and are part the class marked in green.
target class
poison class
poison
target
(a) Undefended model, clean (left) and retrained (b) Defended model, clean (left) and retrained on poi- poisoned
soned
data containing backdoor patches. (right).
data containing backdoor patches (right).
Figure 14: 2D Visualization of the effeets of backdoor trigger patch attack against undefended and defended model. The target trigger is applied to number of target images shown in black and originally part of the class marked in blue. The poisoned images marked in red and part of the class marked in green. Note how the black datapoints are associated with the poison class in the undefended case, but correctly associate with the target class in the defended case.
18

---

Under review as a conference paper at ICLR 2022
ditunce along contrcto
disance sking controts
dxsance abong conbiiss
dasance slong contrib
(a) Undefended model, clean (left) and fine-tuned on (b) Defended model, clean (left) and fine-tuned on poisoned
poisoned
data containing gradient matching attacks. (right).
data containing gradient matching attacks (right).
Figure 15: 2D Visualization of the effects of gradient matching attack (Witches' Brew with squared error) against undefended and defended model. The defended model significantly hinders feature collisions. The target image is marked by a black triangle and is originally part of the class marked in blue. The poisoned images are marked in red and are part of the class marked in green. We see that the attack effectively moves the decision boundary by placing poisoned data opposite of the target feature space. However this is completely prevented in the defended model.
Table 6: Defenses against feature collision via (Aghakhani al. 2020) in the transfer setting for budget of 1% and bound of e = 16.
Defense
Poison Success
Val. Acc.
Time
None
100.00% (15..99
91.97%
0:08:04
Random Noise
90.00% (1...11)
90.45%
0:08:33
Deep K-NN
75.00% (+9.68)
91.94 11.94%
3:29:13
Activation Clustering
0.00% (#5.59)
91.34%
0:07:08
Spectral Signatures
0.00% (+5.599
92.09%
0:09:10
Diff. Priv. SGD (n 0.0001)
100.00% (#:5559)
92.72%
0:08:08
Diff. Priv. SGD
0.001)
100.00% (+:5.59)
91.26%
0:08:07
Diff. Priv. SGD (n 0.01)
85.00% (+7.98)
69.78%
0:08:12
Adv. Training (s 8)
5.00% (+:4.87)
77.80%
0:42:20
Adv. Training (s 16)
5.00% (1:4.87)
58.98%
0:42:37
Adv. Poisoning (s = 0.25)
90.00% (+-6.71)
91.24%
0:30:44
Adv. Poisoning (s 0.5)
70.00% (+10)25)
89.83%
0:35:06
Adv. Poisoning 0.75)
10.00% (++6.71)
88.64%
0:39:53
and due to the limitation that only the last layer is re-trained leaving less control over the model. We record results in Table finding that poison immunity can be effective even in a scenario that favors filter defenses, matching filter defenses, while beating adversarial training significantly trade-off against validation performance.
E EXPERIMENTAL SETUP
In general terms, the goal of our experimental setup is to standardize the experimental conditions encountered in various works in data poisoning to degree that allows for convenient comparisons across attacks. Furthermore previous works have focused on showcasing the smallest possible adversarial modifications that still achieve substantially malicious effect. However, such setups "on the edge" are broken too easily by any defenses, that we generally consider stronger attacks in this work than in their original implementations. On the other hand, there an upper limit this design because attacks have to be sufficiently realistic, modifying only parts of the dataset within limits unlimited adversarial modifications would allow for unlimited attack strength.
For all experiments shown in this work, we standardize the machine learning model that is attacked to deep neural network for image classification, namely always a ResNet- 18 model trained on the CIFAR- 10 dataset. The ResNet- 18 model follows (He et al.) 20155.. with the customary CIFAR-10 modification of replacing the initial 7x7 convolution and max- max-pooling with a 3x3 convolution. We train this model using SGD with Nesterov momentum (m 0.9) with batch size of 128 for 40
19

---

Under review as a conference paper at ICLR 2022
epochs with initial learning rate of 0.1, which is reduced by a factor of 10 after 3 and 1 of all epochs. The model is additionally regularized by weight decay with weight 5 x 10 The CIFAR-10 dataset is augmented with horizontal flips and continuous random crops from images with a zero padding of 4.
We directly train with these hyperparameters in the FROM-SCRATCH setting. For the TRANSFER experiments we first train a ResNet-18 18 with this setup, which call the base model, and then freeze its feature representation. We then retrain the linear layer from a random initialization with the same hyperparameters. For the PINE-TUNING experiments we drop the learning rate by 0.001 before fine-tuning from the base model. likewise reinitializing the linear layer, but not freezing the feature representation. This setup for fine-tuning and transfer is arguably easier to attack than a transfer to an unknown dataset (as investigated for example in (Shafahi et al. 201 as all features are already optimized to be relevant to the given task, and as such we do not recommend it as the only evaluation of an attack, but believe it is an appropriate worst-case setting for the defense experiments considered in this work. Note that we apply slightly different setting for the convex polytope attack of et al. (2019) which we will detail together with the attack in the next section and mark by TRANSFER* * in the main table.
E. MEASURING POISON EFFECTIVENESS
We run data poisoning attacks with the goal of maliciously classifying single target image for all targeted data poisoning attacks, and with the goal of maliciously classifying 1000 images patched with a single target trigger for backdoor attacks. In both cases these images are drawn from the validation set without replacement. We report success for an attack if the target image, or patched image is classified with the adversarial label, we do not count mis-classificatios into third label. In all experiments, we then report avg. poison success. This metric represents the average success over N random trials, where each trial consists of a randomly drawn target trigger or image, randomly chosen adversarial class, randomly chosen subset of images to be poisoned (from the adversarial class) and random model initializations. We control these trials with by specifying their random seed. All experiments in this work are based on the same 20 fixed trials, which we list by their seed within the supplemented code submission and as such comparable.
For all attacks we consider an e00 bound of = 16 for targeted data poisoning attacks.
E.2 ATTACK SETTINGS FOR 0o0 THREAT MODELS
For all attacks we optimize the adversarial perturbation through projected descent (PGD). We found signed Adam with step size of 0.1 be a robust first-order optimization tool to this end which we run for 240 steps, reducing the step size by a factor of 10 after 3 g.8 and I of total steps. Basic data poisoning attacks are often brittle when encountering simple data augmentations (Schwarzschild et al. 2020). As we include data augmentations in our experimental setup. we also include them during the attack algorithm for all iterative attacks, sampling a random augmentation in every update step and differentiating through the resulting transformation via grid sampling (Jaderberg et al.] 2015). Some works such as Geiping et al. (202T) consider multiple restarts of the attack algorithm, however we consistently run all attacks with single restart, mostly due to computational constraints, and given that restarts appear to confer only a minor benefit.
Poison Frogs. We implement the feature collision objective as proposed in Shafahi al. (2018).
However, while perturbation bounds were weakly enforced in the original version by an additional penalty, we instead optimize the objective directly by projected (signed) gradient descent in line with other attacks. We find this to be at least equally effective.
Convex Polytope. Poisoned data created by Convex Polytope can be brittle in terms of the amount of training data and optimizer settings (Schwarzschild et al. 2020). Therefore, to get an idea of how well our defense works against this attack, we implement a modified setting wherein the attack succeeds. This includes using feature extractor trained CIFAR-100, and training the last linear layer for only 10 epochs using the Adam optimizer with Ir= 0.1. We otherwise applied the attack as proposed in Zhu al. 2019). This experiment is found in Table
20

---

Under review as a conference paper at ICLR 2022
Table 7: Transfer* refers to the explicit setting of Zhu et al. (2019). The proposed defense significantly
decreases success rates, even in this setting.
Attack
Scenario
Undefended
Defended
Convex Polytope
Transfer
90.00% (+10000)
40.00 % (+16.32)
Bullseye Polytope. We directly re-implement the attack based on (2) in Aghakhani et al. (2020)
Witches' Brew. We implement gradient matching as in Geiping et al. (2021). However, we modify the original attack in the transfer setting. The original attack is posed for large models and the objective of cosine similarity of parameter gradients does not scale well to small models. For small models (such as the transfer case, where only the last linear layer is retrained), we instead measure similarity in the squared Fuclidean norm. We refer to this variant as gradient matching with squared error, e.g. Gradient Matching (SE) in table 1.
attacks on
MetaPoison We download premade poisoned datasets for MetaPoison from httpt:tt///// /ttp:s:///tttiie thub
com/ htmm//oonnyhuang metapoi son| validate their effectiiveness and and then deploy our proposed defense using our own surrogate attack at the batch level, which we unroll for 2 steps as also proposed for the attacks in Huang al. (2020). We defend using only the current estimate of the model as discussed in previous sections (instead of replicating the ensemble of 24 models used to create the poisoned dataset in some fashion). We train the usual ResNet- 18 model on the downloaded poisoned CIFAR-10 for 40 epochs without data augmentations, conforming to the training setup without augmentations based on which the poisons for a budget of 1% for the bird-dog setting with bird target ids 0 to with perturbations bounded by E 8 in 10) -oom and perturbed by 4% color space.
created. We download poisoned datasets
Hidden Trigger Backdoor For the hidden trigger backdoor attack of Saha et al. (2020) we use trig- gers identical to the ones used in the original work (see /ttts:///ithhtt: com/ UMBCvi sion/ The number of poisons we use is indicated in the main body experiments, and is same order the number used in the original work. Specifically, we evaluate the attack on 1000 patched target images and choose a budget of 5%. The adversarial perturbations the subset of poisoned data are optimized described in the general attack settings, minimizing the hidden trigger objective of matching poison features to features of patched images.
Hidden-Trigger Bawprr-r-ccccoorra-eeeeooo -
E.3 DEFENSE SETTINGS
Input Noise As a sanity check we include a comparison to input noise. We draw random noise from the boundary of the set of allowed perturbations by independently sampling from Bernoulli distribution for each value, and assigning either - or to each value.
CutMix and Maxup We apply CutMix Yun et al. 2019) as a defense against data poisoning as proposed in BBorgnia al. (2020). We attack this defense adaptively by creating poisoned data based on a clean model trained with CutMix as well. The same considerations apply for Maxup (Gong al. 2020). We Cutout (DeVries & Taylor 2017) as base augmentation for Maxup, and select the warst-case augmentation from four examples.
Spectral Signatures We implement the defense proposed in Tran et al. (0188), using the provided overestimation factor 1.5. We supply the attack budget as additional info for this defense.
Deep K-NN We implement the defense as proposed in Peri et al. 2020). using the provided overestimation factor of 2. We supply the attack budget as additional info for this defense.
Activation Clustering We run the defense of Chen et al. 2019), clustering the available training data into two clusters.
21

---

Under review as a conference paper at ICLR 2022
Differentially private SGD We implement variant of differentially private SGD with gradient clipping to a value of 1 on a bani-batch level (as suggested in Hong et al. (2020), and varying levels of Gaussian noise applied to the mini- gradient. Attacks adapt to this defense by adding gradient noise to their surrogate estimation of gradients (this is mostly relevant for gradient matching where surrogate gradients appear explicitly).
Adversarial Training We implement straightforward adversarial training, starting from a randomly initialized perturbation and maximizing cross entropy for 5 steps via signed descent. Interestingly, for small E values, this defense can be overcome by the poisoner by creating poisoned data while adversarial noise is sampled and added during the poison optimization. This however this only helpful when the adversarial training is smaller than the attack e.
Poison Immunity For all implementations of adversarial poisoning we replicate the original objective of the attack in the mini-batch setting, but optimize for only steps, based on features or gradients from the current model. The surrogate attacks are optimized via signed Adam descent with the same parameters as described in the attack section. For the hidden trigger backdoor attack we draw random patches surrogate targets, and then optimize the eo perturbations for 5 steps usual.
E.4 ATTACK SETTINGS FOR e7 THREAT MODELS
For backdoor triggers, we allow triggers with a size of 4 by 4 pixels, i.e. a rectangular arrangement of the e bound of 16. We allow a budget of 3% of the dataset to be modified. We then imprint this patch in the lower right corner of all images in the poison set. During evaluation same patch in the same location for 1000 target images. We first evaluate "noisy checkerboard" patch, which is computed by sampling Bernoulli variable for each patch pixel and RGB channel independently and assigning either 0 or 255. This patch is arguably not part of the distribution of CIFAR- 10 images. Secondly, select a resized firefox logo as the second patch, which leads to patch that appears semantically similar to CIFAR-10 10 content.
imprint the
We evaluate several possible defenses:
Poison Immunity Noise Patch We sample random noisy checkerboard pattern (Bernoulli samples in each pixel and channel as above) with a random rectangular shape with lengths within [3, 12) for an approximate 6 < 45 (overestimating the actual e bound for gray-box setting) as well as a random location. We sample such a patch from for every class in the dataset and then apply them to randomly chosen pairs of classes, replicating the attack without knowing the targeted class.
Poison Immunity Large Noise Patch We repeat the previous setup with random noisy checker- boards sampled with lengths within [8, 28). but otherwise the same setup above. Note that even these large patches improve upon the patches with lengths in [3, 12] only in minor ways for the semantic patch.
Poison Immunity Optimized Patch Previously we only sampled random patches during the defense. This mirrors the methodology of previous sections on targeted attacks, in the sense that a data poisoning attack is used to attack each mini-batch in the same way that it would be used by an attacker - and the backdoor trigger attack also samples these patches randomly as described above. However, this arguably non-optimal approximation of the objective described in Equation (2) in the main body and further we can conjecture more optimal backdoor trigger attack that would optimize patch contents. As such we optimize the contents of a patch with random rectangular shape as described for the noise patch, while still drawing the location at random. However, we need to choose a surrogate objective on which to optimize the patch. For this task we choose gradient matching and as such optimize the patch so that the gradients of patched target data and patched poisoned data are optimally aligned. We apply the same procedure as otherwise in this work and optimize with signed Adam for 5 steps with step size 0.01.
As we find in Table 2 though, this approach does not improve upon the noise patch sampling (and even performs worse). Possible reasons for this behavior are the choice of surrogate objective and optimization setting or the question of whether the patch location and shape should also be optimized, which would however further complicate the optimization.
22

---

Under review as a conference paper at ICLR 2022
Poison Immunity- Image Patch In contrast to sampling noise patch, patch from in-distribution data, i... CIFAR-10 content. To do so we sample these patches from other images in the mini-batch. We choose larger. but fixed size of 16 16 patches with a random location. To correct for the semantic impact of these patches on unrelated images we have to adjust the label information match, that finally, this option reduces exactly CutMix (Yun al.. 2019). also proposed as defense in Borgnia et al. (2021).
also sample this
Filtering Defenses for targeted attacks above.
apply activation clustering, spectral signatures and deep KNN as described
Differentially private SGD We apply differentially private SGD as described for targeted attacks above with noise level of 0.01 and gradient clipping to 0.1.
E.5 COMPUTATIONAL REQUIREMENTS
We run all reported experiments using Nvidia GEFORCE RTX 2080 Ti GPUs, using one GPU per experimental run which are scheduled from an internal SLURM system. We also ran select ablation pre-trials using Nvidia V100 setup. Training robust model in the (most expensive) from-scratch setting requires approx. 4 hours and minutes 10 for the RTX 2080 Ti. After training clean model (for the adaptive attack), the attack takes approx. 5 minutes. Finally the model is trained again on the poisoned data and its resilience against the poisoning is measured. In total this requires approx 8h of compute per experiment, of which we run 20 experiments for every data point with random poison-target class pairs and samples as described above. Training time reduces approx. 2 1h for the fine-tuning setting and 2 x 35 minutes for the transfer setting.
the evaluated ResNet-18 18 on CIFAR-
E.6 ASSET LICENCES
We use CIFAR-10 data (Krizeessyy) 2009) found at hhttps://www.. cs toronto..ed.i.. edu/ -rii/cc ftlz/cifaa far html The code submission is based forked repositories of the projects of
Geiping et al. (2021) and Schwarzschild et al. (2020) as described in detail in the attached code
submission folder.
23


Reviews:
[{"review_id": "sdMn-0mIM9J", "reviewer": "Reviewer_xUH6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The introduced method is sounds and a simple adaptation of adversarial training to make it more effective against data poisoning attacks.\n- The paper is well-written and flows smoothly\n- The authors put decent effort in detailing prior work and replicating their results within the same framework to foster reproducibility. The authors attach the code to the submission.\n- The experimental setup is detailed in the appendix.\n\n### Weaknesses\n- The analysis section S4 and the corresponding qualitative results (Fig 2) are hard to follow and understand. What is the takeaway in this section? Making this clear in the paper/captions is important in my opinion.\n\n### Small changes:\n- Typo: “Defineda” -> “defined”\n- Fig 2 captions are almost overlapping. Try to increase the space between the subplots for better readability.\n", "summary_of_the_paper": "The paper adapts adversarial training to build robust models against data poisoning. Specifically, the paper shows how one can effectively apply data-poisoning attacks in the training loop to successfully defend against potential data poisons. The introduced method is rigorously evaluated by (adaptive) known poisoning attacks. The paper demonstrates that the new defense outperforms existing data-poisoning defenses on CIFAR-10. Finally, the paper qualitatively analyzes the feature space to understand the effect of the proposed defenses.", "summary_of_the_review": "Overall, I believe the paper is good in that it introduces a simple yet effective adaptation of adversarial training to make it effective against data poisoning. The presented empirical results are convincing. Though, I don’t quite understand the qualitative assessment of the introduced defense, so it would be great if the authors can improve the presentation there.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fFqFdi4sXN0", "reviewer": "Reviewer_E8z9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is interesting to extend adversarial training to defending against data poisoning attacks. I reviewed this paper before and the following problems have not yet been addressed.\n\n1. The key of the proposed training is based on the expectation that the generated poisoned data can counter the effect from the real injected data. However, this relies how many data are polluted in the training set. If only a very small portion is injected, then it is possible that the proposed training may eliminate the effect from poisoned data. What if there is a much larger portion of the training set poisoned, say 50% or even 80%? Can the proposed training still reduce the attack success rate without sacrificing normal accuracy? For those poisoning attacks that change labels of poisoned data, the chosen label $y_t$ in Algorithm 1 may still be the true target label, which does not counter the original effect of poisoned data. The current version of the paper does not seem to study the impact of different poisoning rates on the proposed approach.\n\n2. The results show the proposed training is effective against $L_\\infty$ but not against $L_0$ (in Appendix D.4). This may be due to the approximation of generating trigger patterns during training. The paper only uses 5 steps to generate those patterns. This is useful for $L_\\infty$ attacks as it is more like classical adversarial training. As long as the perturbation is pointing towards the target direction, the added perturbation can help the model to be insensitive to those types of perturbations. However, for patch-like attacks, a small number of steps may just generate some random pattern and do not help the training. The results on the semantic patch is poor by the proposed approach. The best result is achieved using image patch, which is actually the baseline CutMix. This means the proposed method is no better than the baseline. This limitation should be discussed early in the paper, preferably in the introduction. Or the authors can adjust their threat model and just focus on $L_\\infty$ attacks. The introduction of the current version does not clarify the limitation of the proposed method regarding the threat model. The evaluation on $L_0$ attacks does not include more studies.\n\n3. Generating $L_\\infty$ poisoned data during the proposed training procedure is similar to crafting universal perturbations. How about using universal adversarial training [1] to defend against data poisoning attacks? This is not investigated in the current version.\n\n4. The evaluation is only conducted on one model (ResNet-18) and one dataset (CIFAR-10). The observations and experimental results may not be general and applicable to other cases. Including more model structures (e.g., VGG, Inception, etc.) and datasets (GTSRB, ImageNet or its subset, etc.) can provide a better understanding of the performance of the proposed training method. There is a public dataset of poisoned models called TrojAI dataset. The original training datasets and the code for generating those models are public available. An extensive evaluation on a larger benchmark can better assess the performance of the proposed approach.\n\n5. There are missing critical results. The accuracies on clean data are missing in Table 1 and Table 2.\n\n6. The writing of the paper needs improvement. This paper categorizes backdoor attacks into backdoor trigger attacks and targeted data poisoning, which is good. But then the paper lists 5 attacks that are evaluated in the experiment. The two parts are not well organized. I would suggest to put the relevant attacks in the corresponding categories and then elaborate their similarities and differences. Subsections can be used to highlight different parts of the related work. For instance, use subtitle \"poisoning attacks\" to indicate the discussion on attacks and \"poisoning defenses\" to indicate the discussion on defenses. The references to figures and tables are not consistent. The paper uses \"Figure 2a\" on page 6 but uses \"fig. 2c\" on page 7. The paper directs the readers to the supplementary material on page 7 but does not point out the specific section.\n\n[1] Shafahi, Ali, et al. \"Universal adversarial training.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.", "summary_of_the_paper": "This paper proposes to leverage classical adversarial training to defend against data poisoning attacks. Specifically, the paper splits each training batch randomly into two sets: one set for generating the trigger pattern and the other for injecting the pattern. The subject model is then trained on the modified batch. This paper evaluates the proposed training method on a few $L_\\infty$ based backdoor attacks and shows that it has a better balance between normal accuracy and defense performance compared to baselines.", "summary_of_the_review": "1. There is no study on the impact of different poisoning rates.\n2. The method is only effective against $L_\\infty$ but not $L_0$ and it is not clarified in the introduction.\n3. A baseline defense method is not evaluated in the paper.\n4. The number of evaluated models and datasets is small and limited.\n5. The writing needs improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CxYlCz-UUzJ", "reviewer": "Reviewer_dQnr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is clear and easy to understand. The idea is interesting and the arguments are convincing. However, since it is some kind of combination of two existing robust models, I give weak acceptance based on the novelty. It would be great if the authors can provide more evidence on the novelty of this algorithm, e.g. what difficulties you met when designing this algorithm.\n\nBelow are some other minor comments:\n\n(1) Could you provide some insights on why adversarial training performs worse than the proposed method in Figure 3? For example, what attack does adversarial training tend to defend, and what is the formula of the attack used in Figure 3?\n\n(2) After adding adversarial training idea, the algorithm involves three levels of optimization. The adversarial training is known to be time consuming, so I expect the proposed algorithm takes an even longer time to run. Please report the running time of your experiment.\n\n(3) It takes me several minutes to understand the layout of Figure 2: there are four groups (a) to (d) and each group contains two subfiguires. Please enhance the display of Figure 2. At the first glance I was thinking there are two groups (upper and lower), and the title of the upper group is \"(a) Undefended model, clean (left) and fine-tuned on (b) Defended model, clean (left) and fine-tuned on\".\n\n", "summary_of_the_paper": "This paper propose to use the idea of adversarial training to defend data poisoning. The idea is interesting and the numerical results \nalso demonstrate the promising performance of this algorithm.", "summary_of_the_review": "This paper is clear and easy to understand. The idea is interesting and the arguments are convincing. However, since it is some kind of combination of two existing robust models, I give weak acceptance based on the novelty. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7YVWXv4nEa_", "reviewer": "Reviewer_pSWq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengthes:\n\n1. The proposed training objective for poison immunity is reasonable and well-motivated.\n2. Experiments are thorough. Many baselines are compared. Results are promising.\n\nQuestions/Weaknesses:\n\n1. Is the proposed defense objective in Eq. (4) a natural generalization of the original adversarial training objective in Eq. (1). In other words, is Eq. (1) a special case of Eq. (4)? The authors are suggested to discuss this. If it is not the case, the wording of some parts of the paper, such as the title of Section 3 (i.e., \"Generalizing Adversarial Training to Data Poisoning\"), may need to be modified to avoid misunderstandings.\n2. There are some closely related papers [1, 2, 3], which the authors may want to discuss. In particular, [1] proposed \"an adaptive defense\", where they can teach a model to correctly classify both unperturbed and perturbed pictures generated by Fawkes and LowKey. Their procedure shares some similarities with this paper. [2] also extended the use of adversarial training for defending against one type of poisoning attacks (they called delusive attacks). Note that those papers are largely concurrent to this paper, so they would not affect the contribution of this paper. Yet, elaborating on the differences from those papers would increase the integrity of this paper.\n3. A minor typo. A punctuation mark is missing after \"those of clean data\" on the third-to-last line on the second page.\n\n[1] Radiya-Dixit et al., Data Poisoning Won’t Save You From Facial Recognition. ICML Workshop, 2021.  \n[2] Tao et al., Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training. NeurIPS 2021.  \n[3] Li et al., Anti-Backdoor Learning: Training Clean Models on Poisoned Data. NeurIPS 2021.", "summary_of_the_paper": "This paper proposes a defense against backdoor attacks and targeted attacks. Specifically, the proposed defense injects randomly chosen poisoned instances and targeted instances into the training data, and then forces the model to have right behaviors on these instances. Experimental results demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "Overall, I am leaning towards acceptance. The proposed defense outperforms many existing defense strategies including differentially private SGD, adversarial training, various data augmentations, and filter defenses, providing a strong trade-off of robustness and accuracy.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewer E8z9 remained with a number of serious concerns, including efficacy of the defense in higher poisoning setting, overclaiming contributions in terms of L0 defenses (which are mostly achieved by the baseline CutMix), as well as novelty and generalizability of the approach. The author responses were unconvincing, and all other reviewers participated in the discussion, conceding that they too were unable to provide compelling arguments against E8z9's comments. Other reviewers claimed that these drawbacks may be "acceptable" for a first step, but were not willing to defend it very strongly. 

We note that E8z9 claims they are a reviewer for a previous version of the paper and these issues were present before. The authors claim that E8z9's key points had been addressed in this version, but the reviewer maintains that the issues still persist in the latest version. The authors are advised to take their comments into account for further versions of this paper.
Decision: Reject