Incompatibility Clustering as a Defense Against Backdoor Poisoning Attacks
Authors: Charles Jin
Melinda Sun
Martin Rinard
Venue: ICLR 2023 poster (2023)
Keywords: data poisoning
defense

Abstract: We propose a novel clustering mechanism based on an incompatibility property between subsets of data that emerges during model training. This mechanism partitions the dataset into subsets that generalize only to themselves, i.e., training on one subset does not improve performance on the other subsets. Leveraging the interaction between the dataset and the training process, our clustering mechanism partitions datasets into clusters that are defined by—and therefore meaningful to—the objective of the training process.

We apply our clustering mechanism to defend against data poisoning attacks, in which the attacker injects malicious poisoned data into the training dataset to affect the trained model's output. Our evaluation focuses on backdoor attacks against deep neural networks trained to perform image classification using the GTSRB and CIFAR-10 datasets. Our results show that (1) these attacks produce poisoned datasets in which the poisoned and clean data are incompatible and (2) our technique successfully identifies (and removes) the poisoned data. In an end-to-end evaluation, our defense reduces the attack success rate to below 1% on 134 out of 165 scenarios, with only a 2% drop in clean accuracy on CIFAR-10 and a negligible drop in clean accuracy on GTSRB.

Published as a conference paper at ICLR 2023
INCOMPATIBILITY CLUSTERING AS A DEFENSE AGAINST BACKDOOR POISONING ATTACKS
Charles Jin CSAIL MIT Cambridge, MA 02139 cc ccjecsail.m. aataaiimmmii..... .mit.ed
Melinda Sun
Martin Rinard CSAIL
MIT
Cambridge, MA 02139
MIT
mmsun@mit.edu
Cambridge, MA 02139 rinard@csail wis.sailllmm.........mm. mit.edu
ABSTRACT
We propose novel clustering mechanism based incompatibility property between subsets of data that emerges during model training. This mechanism partitions the dataset into subsets that generalize only themselves, i.e., training on one subset does not improve performance on the other subsets. Leveraging the interaction between the dataset and the training process, our clustering mechanism partitions datasets into clusters that are defined by-and therefore meaningful to- the objective of the training process.
We apply our clustering mechanism to defend against data poisoning attacks, in which the attacker injects malicious poisoned data into the training dataset affect the trained model's output. Our evaluation focuses on backdoor attacks against deep neural networks trained perform image classification using the GTSRB and CIFAR-10 datascts. Our results show that (1) these attacks pro- duce poisoned datasets in which the poisoned and clean data are incompatible and (2) technique successfully identifies (and removes) the poisoned data. In an end-to-end evaluation, our defense reduces the attack success rate to below 1% 134 out of 165 scenarios, with only 2% drop in clean accuracy on CIFAR-10 and a negligible drop in clean accuracy on GTSRB.
OVERVIEW
Clustering, which aims to partition a dataset to surface some underlying structure, has long been fundamental technique in machine learning and data analysis, with applications such outlier detection and data mining across diverse fields including personalized medicine, document analysis, and networked systems (Xu & Tian 2015). However, traditional approaches scale poorly to large, high-dimensional datasets, which limits the utility of such techniques for the increasingly large, minimally curated datasets that have become the norm in deep learning (Zhou al. 2022).
We present a new clustering technique that partitions the dataset according to an incompatibility property that emerges during model training. In particular, it partitions the dataset into subsets that generalize only to themselves (relative to the final trained model) and not to the other subsets. Di- rectly leveraging the interaction between the dataset and the training process enables our technique to partition large, nigh-dimensiooal datasets commonly used to train neural networks (or, more gen- erally, any class of learned models) into clusters defined by- and therefore meaningful to the task the network is intended to perform.
Our technique operates as follows. Given dataset, it iteratively samples a subset of the dataset, trains model on the subset, then selects the elements within the larger dataset that the model scores most highly. These selected elements comprise a smaller dataset for the next refinement step. By decreasing the size of the selected subset in each iteration, this process produces final subset that (ideally) contains only data which is compatible with itself. The identified subset is then removed from the starting dataset, and the process is repeated to obtain collection of refined subsets and corresponding trained models. Finally, a majority vote using models trained on the resulting refined subsets merges compatible subsets to produce the final partitioning.

---

Published as a conference paper at ICLR 2023
Our evaluation focuses backdoor data poisoning attacks (Chen al. 2017: Adi al. 2018) against deep neural networks (DNNs), in which the attacker injects a small amount of poisoned data into the training dataset to install backdoor that can be used to control the network's ' behavior during deployment. For example, /uu et al. (019) install a backdoor in a traffic sign classifier, which causes the network misclassify stop signs speed limit signs when a (physical) sticker is applied. Prior work has found that directly applying classical outlier detection techniques to the dataset fails separate the poisoned and clean data (Tran et al. 2018). A key insight is that training on poisoned data should not improve accuracy on clean data (and vice versa). Hence, the poisoned data satisfies the incompatibility property and can be separated using our clustering technique.
This paper makes the following contributions:
Clustering with Incompatibility and Seff-Expansion. Section 2 defines incompatibilityy between subsets of data based on their interaction with the training algorithm, specifically that training on one subset does not improve performance on the other. We prove that for datasets containing in- compatible subpopulations, the subset which is most compatible with itself (as measured by the self-expansion error) must be drawn from a single subpopulation. Hence, iteratively identifying such subsets based on this property provably separates the dataset along the original subpopulations. We present a tractable clustering mechanism that approximates this optimization objective.
Formal Characterization and Defense for Data Poisoning Attacks. Section 3 presents formal characterization of data poisoning attacks based on incompatibility, namely, that the poisoned data is incompatible with the clean data. We provide theoretical evidence to support this characterization by showing that a backdoor attack against linear classifiers provably satisfies the incompatibility property. We present a defense that leverages the clustering mechanism to partition the dataset into incompatible clean and poisoned subsets.
Experimental Evaluation. Section 4 presents empirical evaluation of the ability of the incom- patibility property presented in Section 2 and the techniques developed in Section to identify and remove poisoned data defend against backdoor data poisoning attacks. We focus on three dif- ferent backdoor attacks (two dirty label attacks using pixel-based patches and image-based patches, respectively, and clean-label attack using adversarial perturbations) from the data poisoning litera- ture (Gu et al. [2019] Gao et al. 2019; Turner et al. (018). The results (1) indicate that the considered attacks produce poisoned datasets that satisfy the incompatibility property and (2) demonstrate the effectiveness of our clustering mechanism in identifying poisoned data within a larger dataset con- taining both clean and poisoned data. For attacks against the GTSRB and CIFAR-10 datasets, our defense mechanism reduces the attack success rate to below 1% 134 out of the 165 scenarios, with a less than 2% drop in clean accuracy. Compared to three previous defenses, our method suc- cessfully defends against the standard dirty-label backdoor attack in 22 out of 24 scenarios, versus only 5 for the next best defense.
We have open sourced implementation at https://github.ccm//haarr/
compat cemmpaaibilityyllle htt.lllytcyyllynnggggooooo
INCOMPATIBILITY CLUSTERING
This section presents our clustering technique for datasets with subpopulations of incompatible data. In our formulation, two subsets incompatible when training one subset does not improve performance on the other subset. Because we cannot measure performance on, e.g., holdout data with ground truth labels, we introduce a "self-expansion" property: given a set of data points, we first subsample the data to obtain smaller training set, then measure how well the learning algorithm generalizes to the full set of data. Our main theoretical result is that set which achieves optimal expansion is homogeneous, i.e., composed entirely of data from a single subpopulation. Finally, we propose the Inverse Self-Paced Learning algorithm, which uses an approximation of the self- expansion objective to partition the dataset.
2.1 SETTING
We present our theoretical results in the context a basic binary classification setting. Let X be the input space, Y = {-1, 1,,+1 +1) be the label space, and T.(..) be the 0-1 loss function over Y Y, i.e.,
2

---

Published as a conference paper at ICLR 2023
L1V1 y2) (1 y1 * 92) Given target distribution D supported on X x Y and a parametric family of functions fo. standard objective in this setting is to find 8 minimizing the population risk R(0) I ((f(())) y v)dPo(P.PP y). Given n data points D {Z), yi),.... (Iz, Vr)). the empirical risk defined as Remp(0; D) := n Ei (zx,,, y:). We fix learning algorithm A(-) which takes as input training set and returns parameters 0. For example, if so is a family of DNNs, then A might train a DNN via stochastic gradient descent.
2.2 SELF-EXPANSION AND COMPATIBILITY
To measure generalization without holdout data and independent of any ground truth labels, propose the following self-expansion property of sets:
Definition 2.1 (Self-expansion of sets.). Let S and T be sets (S nonempty), and let a E (0, 1] denote the subsampling rate. We define the a-expansion error of S given T as
c == Sur(srsee (A(");;: where the expectation is over both the randomness in A and S', a random variable that samples a uniformly random fraction per class (rounded up) from SUT. When T = 0 we also write cS; a)).
(1)
Here is hyperparameter that typically selected a priori on the basis of some domain knowledge (for example, if = and A trains a deep neural network, then the network can memorize the training set). A typical value of a is 1/4 or 1/2: in our ablation studies, we find that our particular application is robust to wide range of a.
The self-expansion property measures the ability of the learning algorithm A to generalize from a subset of S U T to the empirical distribution of S. Intuitively, when T 0, a smaller expansion error means that the set is both "easier" and "more homogeneous" with respect to the learning algorithm,
random subset is sufficient to lear the contents of S. When T is not empty, expect that the self-expansion error will be lower when T contains similar data as S (for instance, both S and T are drawn from the same distribution D).
This observation leads directly to our formal definition of compatibility between sets:
Definition 2.2 (Compatibility of sets.). A set T is a-compatible with set if <(SIT; a) e(S;a).
Conversely, T is a-incompatible with S if the opposite holds, i.e.,
c(; a) S <())Taa).
(3)
Furthermore, T is completely a-compatible with S if every subset Tf is a-compatible with every subset of S, and similarly for complete incompatibility.
In other words, T is (formally) compatible with S if T improves the ability of A to generalize S.
2.3 SEPARATION OF INCOMPATIBLE DATA
Our next result allows us to separate subpopulations of data satisfying the incompatibility property (proof in Appendix [A]. The main idea is that, given any set, we can always achieve better self- expansion by removing incompatible data (if it exists).
Theorem 2.3 (Sets minimizing expansion error are homogeneous.). Let = AUB be set with A and nonempty and mutually completely a-incompatible. Define
min (S);
(4)
S'CS
let S be the collection of subsets of S that achieve e*. Then for any smallest subset Smin E S*, either Smnnn C A or Sii. C B. Furthermore, if at least one of the incompatibilities strict, then for all 5* E S* we have either S C A or S* B.
Theorem suggests an iterative procedure for separating subsets of data that satisfy the incom- patibility property. Namely, at each step i, identify St in the current dataset D,, then repeat the procedure with D.+1 D; S The procedure terminates when Di+1 0. Assuming the sub- populations in the dataset satisfy the incompatibility property, this process partitions the training set into subsets that guaranteed to contain data from a single subpopulation. Furthermore, if subpop- ulations satisfy strict incompatibility, we can instead take the largest S e S* at each step. The next section develops this insight into practical clustering mechanism for incompatible subpopulations.
3

---

Published as a conference paper at ICLR 2023
Algorithm Inverse Self-Paced Learning
Input: training set S, total iterations N, annealing schedule 1 > 80 2 BN
Bmin > 0.
expansion a S I, momentum 7/ [0, 1]. learning algorithm A, initial parameters 0n
Output: SS S such that |Sx] SSin|S]
1: So 5
2: L 4
3: for = 1 to do
4:
5" SAMPLE(S,-1,0))
5:
0: + A(S", 0z-1)
6:
f nL+ (1 -1) Bemp (0,
7: St < TRIM L, 51)
8: end for
2.4 INVERSE SELF-PACED LEARNING
major question raised by Theorem 2.3 is how to identify the set S In general, even computing single self-expue celf-cxpansion error of a given set S is intractable it involves taking the expectation over all subsets of size a a.SS The Inverse Belf-Paced Learning (ISPL) algorithm presented below solves this intractibilty problem. Rather than optimizing all possible subsets of the training data, instead seek to minimize the expansion error over subsets of fixed size S)S). The new optimization objective is defined as:
5 arg
S -Ssss5'88811 a))
min
(5)
We alternate between optimizing parameters 0g and the training subset Si-given St-1, update 0, using a single subset from S; of size a|S,-1|. Then we use 0g to compute the loss for each element in S, and set S, be the 8 fraction of the samples with the lowest losses. To encourage stability of the learning algorithm, the losses are smoothed by momentum term 1). To encourage more global exploration in the initial stages, we anneal the parameter 3 from an initial value 9 down to the target value 8min- Algorithm Serrithmmppreeeee presents the full algorithm. SAMPLE takes set S and samples an a fraction each class uniformly at random; TRIM takes losses and returns B fraction with the lowest loss.
To conclude, we show for certain choices of parameters that Algorithm converges to local opti- mum of the following objective over the training set S:
F(e., S. == L(fo,, vi) + (0, [S,))
(6)
VES:
where S, is a subset and 3r is decreasing (proof Appendix A
Proposition 2.4. Set a = 1 and O in Algorithm and assume that A returns the empirical risk minimizer. Then we have that for each round of the algorithm, F(Or, Si5 Br) is decreasing in t and furthermore, [F Sc; F(6441, S1:1; Be41) 20000 0.
3 DEFENDING AGAINST BACKDOOR ATTACKS
We present novel characterization of poisoning attacks based on the formal definition of compat- ibility introduced in Section 2 In our threat model, the attacker aims to affect the trained model's behavior in ways that do not emerge from clean data alone, so we characterize the poisoned data by its incompatibility with clean data. We prove this observation holds for backdoor attack in the con- text of linear classifiers. As Theorem 2.3 yields method of partitioning the dataset into clusters of clean and poisoned data (when the clean and poisoned data are incompatible), we present a boosting algorithm to identify which clusters contain clean data (and hence remove the poisoned data).
3.1 THREAT MODEL.
The attacker's goal is to install a backdoor in the trained network that 1) changes the poisoned network's predictions poisoned data while preserving the accuracy of the poisoned network
4

---

Published as a conference paper at ICLR 2023
clean data. For the backdoor to be meaningful, the behavior of poisoned model on poisoned data should also be inconsistent with a model trained on clean data.
To begin, set of clean training samples D is drawn iid from D. After observing D, the attacker selects at most n - samples A A D (leaving behind at least n/2 clean samples = \ A) and fixes a pair of perturbation functions Ttrain X x X X x Y (the attacker the poisoned dataset for training, changing the image and potentially the label of each poisoned element) and Tteat X + X (the attacker uses Trest construct poisoned test images without changing the ground-truth label of the poisoned image). The attacker then poisons the samples in A obtain poisoned samples P [Ttrain (a) a E A}. The threat model places no restrictions on Ttrain, but requires that Treat should not affect the predictions of network trained on C. The final poisoned training set is C CUP. P.
Ttrain construct
Let & be the parameters of the network under attack. Given m fresh test samples T drawn iid from D, the attacker seeks to maximize the targeted misclassification rate (TMR):
TMR(0;
= m Σ L(fo(z:). -y.) [[fo((ica(((:)))
1
m
i.e, the attack succeeds if can flip the label of a correcty-classified instance by applying the trigger Tteat()) at test time. This metric captures both the "hidden" nature of the backdoor (the first term rewards the attacker only if clean inputs are correctly classified) as well as control upon application of the trigger (the second term rewards the attacker only poisoned inputs are misclassified).
3.1.1 DEFENSE CAPABILITIES AND OBJECTIVE
The defender is given only the poisoned dataset C' U P. In particular, Tirain, Trests A, and C are not available to the defender. The defender's 's objective is to return a sanitized dataset D S CUP such that the learned parameters 0 == A( (D) are as close possible to the ground-truth parameters 0" == AC).. Note that 0" is guaranteed to exhibit low targeted misclassification rate-the the threat model requires that for fresh (z, y) drawn from D, fo- (z)) with high probability.
3.2 CHARACTERIZING BACKDOOR POISONING ATTACKS USING INCOMPATIBILITY
The following property formalizes the intuition that increasing the amount of clean data in the train- ing set should not improve the performance on poisoned data (and vice versa):
Definition 3.1 (Incompatibility Property). Let C and P be the clean and poisoned data produced by an attacker according to threat model defined in Section 3.1 We say that the attacker satisfies the incompatibility property if C and P are mutually completely incompatible. Furthermore if at least one of the incompatibilities between C and P is strict, then the atlacker satisfies the strict incompatibility property.
Note that incompatibility defined with respect to the learning algorithm A as well as the sub- sampling rate a. To motivate this definition, we exhibit backdoor attack against linear classifiers which falls under the threat model in Section [31 and provably satisfies the incompatibility property:
Theorem 3.2. Let the input space be X = =R (N > 1). We use the parametric family of functions mm. consisting of linear separators given by fmu,(())
sigu((w,a0 + b) for RR b E R. A selects the maximum margin classifier. Then there exists distribution D and perturbation functions Terain and Trest such that if the adversary poisons 1/4 of each class at random, then with high probability over the initial training samples:
1. Given the clean datasel, A returns parameters with 0 empirical (and population) risk.
2. Given the poisoned dataset, A returns parameters with targeted misclassification rate of I.
3. The clean and poisoned data are mutually completely incompatible.
The proof in Appendix A The idea is to construct dataset such that there exist dimensions in the original, clean feature space that are uncorrelated with the true labels, i.e., consist of pure noise. The attacker makes use of these extra dimensions by replacing the noise with a backdoor signal. Although this result uses linear classifiers, the setting is analogous to many existing attacks against DNNs: for example, many backdoor attacks against image classifiers insert synthetic patches on the border of the image, which is location that does typically affect the original classification task.
5

---

Published as a conference paper at ICLR 2023
Algorithm 2 Boosting Homogeneous Sets
Input: Homogeneous sets S1, Sv, number of samples n. > | learning algorithm A
Output: Votes V1,... VN
1: Ci. CN f- 0
2: for i = 1 to do
3: 0: < AS:)
4: for k to do
5: if LoSSo, (0;; Sk) <
S then
6:
Ck < Ck + [S
7: end if
8: end for
9: end for
10: for 1 to N do
Vi C; n/2
12: end for
3.3 IDENTIFICATION USING BOOSTING
We conclude this section by showing how identify the clean data given the output of our clustering mechanism, which is a collection of homogeneous (i.e., entirely clean or entirely poisoned) subsets. One idea to measure the compatibility between each pair of subsets, then take the largest mutually compatible (sub)collection of subsets (under the assumption that the clean data is always compatible with itself). However in general this requires training on each subset multiple times sequence. We instead propose simplified boosting algorithm for identifying the clean data which involves training on each subset exactly once. The main idea is to fit a weak learner to each cluster, then use each weak learner to vote the other clusters.
The following sufficient conditions guarantee the success of our boosting algorithm:
Property 3.3 (Compatibility property of disjoint sets). Let C' and C" be any two disjoint sets of clean data, and let P' be a set of poisoned data. Then
Remp (AlC') C") 1/2 s (eC,'
The first inequality says that training and testing on disjoint sets of clean data is an improvement over random guessing. The second inequality says that training with clean data then testing poisoned data is at best equivalent to random guessing; this property is natural in that the threat model already guarantees Remp (A(C); P) is large (i.e., close to 1) for random poisoned data P.
Algorithm presents our approach for boosting from homogeneous sets. The subroutine Lossp,1 takes a set of parameters and set S, and returns the total risk over using the zero-one loss. Each subset's vote is weighted by its size. We have the following correctness guarantee (proof in Appendix A
Theorem 3.4 (Identification of clean samples). Let St,..,, SN be a collection of homogeneous sub- sets containing either entirely clean or entirely poisoned data. Then if the clean and poisoned data satisfy Property 3.3 Algorithm votes V. = True if St is clean (and V, False otherwise).
4 EXPERIMENTAL EVALUATION
We report empirical evaluations of the incompatibility property (Section 4.1 and the proposed de- fense (Section 4.2). We use the CIFAR-10 dataset (Krizhevsky Hinton 2009) for general image recognition, containing 10 classes of 5000 training and 1000 test images each, and the GTSRB dataset (Stallkamp et al., 2012)) for traffic sign recognition, containing 43 classes with 26640 train- ing and 12630 test images, respectively. The GTSRB training set is highly imbalanced, with classes ranging from 150 to 1500 instances.
Our experiments consider three types of backdoor attacks. Dirty label backdoor (DLBD) attacks use small patches ranging from a single pixel to a 3x3 checkerboard pattern, following the imple- mentation Tran al. (2018). WATERMARK uses 8x8 images such copyright sign peace

---

Published as a conference paper at ICLR 2023
Figure I: Pairs of poisoned (top) and clean (bottom) samples for all datasets, selected for visible triggers. From left right: CIFAR-10 with CLBD using GAN, f2, and lxx perturbations, DLBD, and WATERMARK attacks; and GTSRB with DLBD and WATERMARK attacks.
Performance CIFAR-10
Performance vs. CIFAR-10
Absolute Change in Mccuracy (h)
0.5
3.0
of
Foison Data
Figure 2: Results from incompatibility experiments on CIFAR-10, DLBD (left) and CLBD (right). Each subplot shows how accuracy changes different subsets of data as the amount of poison increases. The top plot displays change in accuracy on clean training data, i.c., <CCi a) (C;aa); the next plots separate the change in accuracy on C into source and non-source classes; the bottom plot shows change in accuracy on poisoned data P. All axes use linear scales.
sign, taken from Gao et al. (2119). Both of these attacks insert a patch into a training image from a source class, then change the training label the target class. The attacker's objective is induce the learner to misclassify images of the source class as the target class upon application of the patch. Clean label backdoor (CLBD) attacks use the poisoned datasets from Turner et al. (2018), which are only available for CIFAR-10. Each attack first strongly perturbs each image of the source class independently using either GAN, lz. or foe bounds, then inserts several patches (without changing the label); the attacker's objective is to make the learner misclassify an image from any non-source class as the source class by applying the patches. Figure [ Additional dataset construction details can be found in Appendix B All experiments presented in the main text use a PreActResNet- 18 architecture. Full experimental details and additional results, including performance against adaptive white-box attacker and experiments testing the sensitivity and
presents examples from each dataset.
of ISPL to its hyperparameters, are contained in Appendices C
4.1 INCOMPATIBILIT BETWEEN CLEAN AND POISONED DATA
We evaluate whether the attacks satisfy the incompatibility property by measuring the gap between e(C;a) and e(C² P: a) for clean data C and poisoned data /' generated by the 3 attacks (DLBD, WATERMARK, CLBD) on CIF Figure presents selected plots for the DLBD (e.g., rela- belling airplanes as birds) and GAN-based CLBD (e.g., perturbing airplanes toward other classes) scenarios. Along the x-axis, we increase the ratio of poisoned to clean data in the source class from 0 to 1. Our experimental procedure is follows: we first sample 1/8 of each class in the clean dataset to create C. For each poison ratio, we sample P' ~ P according to the poison ratio, train on P' U C' with entire process is repeated for samples of C for each poisoned dataset our evaluations (8 total for DLBD, 5 total for GAN-based CLBDD.. To better present the gap Pa c(C;a), we report
1/2, then take the mean over 5 samples of P' as the self-expansion error. The

---

Published as a conference paper at ICLR 2023
Table 1: Summary of performance for ISPL+B across several datasets and poisoning strategies. Columns display different datasets and poisoning strategies (e.g., poisoning CIFAR-10 10 using the CLBD attack). The rows display different amounts of poison (e-g., c c= refers 5% of the source class being poisoned). An entry of the form "7 8" indicates that ISPL+B successfully defended of the scenarios in the corresponding setting. Please refer to Section 4.2 for a detailed description.
CIFAR-10
GTSRB
DLBD
totals
WATERMARK
CLBD
DLBD
WATERMARK
1-to-1
all-to-1
all-to-all
7/8
4/4
4/4
6/8
15/
6/8
8/8
50/55
8/8
4/4
2/4
5/8
15
6/8
8/8
48 / 55
20
7/8
4/4
2/4
1/8
11/15
5/8
6/8
36/ 55
totals
22/24
12 12
8/ 1/12
12/24
41/45
17
22/224
134/ 165
each metric as an absolute difference in accuracy from the unpoisoned case (ratio = 0). A trendline connects the medians.
Our first observation is that the key quantity, e(C)P;a) - e(C: a), is negative for all poisoned datasets, which empirically supports our insight that clean and poisoned data are incompatible (or even strictly incompatible), despite the qualitative difference behind the attack mechanisms, The trend in the top plot also indicates that the gap between PC)) and c(C:a) increases with the amount of poisoned data. The next two plots show how the gap breaks down over the source and non-source classes. For the DLBD attack, the source class shows a strong negative trend in accuracy as the amount of poison increases, while accuracy the non-source classes stays largely constant. This fact is consistent with the hypothesis that the poisoned data is impairing the accuracy of the classifier on clean data from the source class (e.g., clean airplanes are getting misclassified as birds).
For the CLBD attack, even though the total clean accuracy decreases, the source class accuracy actually increases the amount of poison increases. We attribute this phenomenon to the classi- fier learning the perturbed poisoned training instances in the source class, which yields improved accuracy the clean instances well. The decrease in clean accuracy is therefore due to drop in accuracy among the non-source classes. Because the CLBD attack produces perturbations of the source class toward other classes (thus making the other classes more similar to, e.g., airplanes), this behavior is consistent with the classifier misclassifying clean instances of non-source classes as the source class. These observations are corroborated by the increase in accuracy on poisoned data in both cases, suggesting that the classifier's 's mistakes on clean data are correlated with increased accuracy on similar- limilar-lokking poisoned data. Appendix containss further discussion and results.
4.2 PERFORMANCE OF PROPOSED DEFENSE
We next evaluate the performance of our proposed defense on the full range of attacks. Most settings one-to-one attacks (e.g.. place trigger on dogs and mislabel cats; e percentage of the source class is poisoned), which is the most common setting in the literature (Gu et al. 2019). For the CIFAR-I0 DLBD scenario,
additionally use all-to-one (e.g., place a trigger on all non-cats and mislabel as cats; an c/9 percentage of each non-target class is poisoned) and all-to-all attacks (i.e.. place trigger on any image and mislabel according to a cyclic permutation on classes; an percentage of every class is poisoned).
Table presents a summary of our results. Each entry is of the format "success total", where total indicates the total number of different scenarios for the attack setting (i.e., different source and target class and trigger combinations), and run is successful when the defense achieves TMR below 1%. We note that the clean accuracy is around 91% % on CIFAR-10 and 94% on GTSRB after running ISPL+B, compared to around 93% and 94% when training on the full, unpoisoned datasets, respectively. These results indicate that approach succeeds in defending against various backdoor poisoning attacks, particularly for the standard settings of e = 5, 10 in the literature (Tran) et al. 2018), for small cost in clean accuracy. Appendix D contains the unabridged results.
8

---

Published as a conference paper at ICLR 2023
epsilon=5
epsilon=10
epsilon=20
100%
Targeted Misclassification Rate
40%%
10%
34%
1%
55
ITLM
AC
ISPL+8 (this work)
55
ITLM
AC
ISPL+B (this work)
55
ITLM
AC
IS (this work)
Comparison with baseline defenses on the CIFAR-I0 DLBD (1-to-1) scenario for 10, 20. Each plot displays TMRs (lower is better) 24 runs The orange lines indicate the median TMR, the boxes indicate the top and bottom quartile, and the whiskers cover the entire range of performance. The blue line displays the cutoff for success used in Table TMR <1%).
Figure 3:
Comparison with Existing Approaches. Figure compares the performance of our defense with
3 existing defenses -Spectral Signatures (Tran et al.) 2018), Iterative Trimmed Loss Minimization (Shen & Sanghavi| 2019), and Activation Clustering (Chen al. (2088) on the CIFAR-10 DLBD (1-to-1) scenario. ISPL+B outperforms all 3 baselines in terms of reducing the TMR. Table 2 in
Appendii(Dcoontaaie contains the detailed results of this experiment.
5 RELATED WORK
Many prior works propose methods for defending against backdoor attacks neural networks. One common strategy, which can be viewed as type of deep clustering (Zhou et al.) 2022), is to first fit deep network the poisoned distribution, then partition the learned representations. The Activation Clustering defense (Chen et al. 2018) runs k-means clustering (k=2) on the activation patterns, then discards the smallest cluster. Tran al. (2018) propose defense based on the as- sumption that clean and poisoned data are separated by their top eigenvalue in the spectrum of the activations. TRIM (Jagielski et al. 202T) and ITLM (Shen & Sanghavi] 2019) iteratively train on subset of the data created by removing samples with high loss. However, each iteration removes same small number samples; both direct comparisons with ITLM and our ablation studies suggest that the dynamic resizing in ISPL is crucial to identifying the poison. (Shan et al. (2022) is motivated by a similar intuition as our compatibility property: however their formalization differs significantly from ours, and they do not present defense but rather forensies technique, which is only able to identify the remaining poison given access to a known poisoned input. Finally, several works provide certified guarantees against data poisoning attacks (Steinhardt et al.] [017]] Jia al.) 2022]; in exchange for the stronger guarantee, these works defend a small fraction of the dataset, e.g., the state-of-the-ad certifies 16% of CIFAR-1O against 50 poisoned images (Wang et 2022a) We refer the reader to Wang el al. (2022b) for a comprehensive survey of data poisoning defenses. Appendix ErpendixxEcootaiinn contains a discussion of other related works.
6 CONCLUSION
Backdoor data poisoning attacks on deep neural networks are emerging class of threats in the growing landscape of deployed machine learning applications. We present new, state-of-the-art defense against backdoor attacks by leveraging novel clustering mechanism based patibility property of the clean and poisoned data. Because our technique works with any class of trained models, we anticipate that our tools may be applied to wide range of datasets and training objectives beyond data poisoning. More broadly, this work develops an underexplored perspective based the interaction between the data and training process.
incom-

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We gratefully acknowledge support from DARPA Grant HR001 120C0015. The views expressed those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
REFERENCES
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weak- ness into a strength: Watermarking deep neural networks by backdooring. In 27th {USENIX} Security Symposium ({USENIX) Security 18), pp. 1615-1631, 2018.
Maria-Florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: Towards bridging theory and practice. Advances in neural information processing systems, 17:89-96, 2005.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning, pp. 41-48, 2009.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee. Ian Molloy, and Biplav Srivastava, Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint 11iv:1811.03725 2018
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05522 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-248822 200,-255,220 2009. doi: 10. 109/C//R.2200000 2009.506848. 5206848.
Yoav Freund, Robert E Schapire, et al. Experiments with a new boosting algorithm. In ICML, volume 96, pp. 158-156. Citeseer, 1996.
Frossyniotis, Aristidis Likas, and Andreas Stafylopatis. A clustering method based on boosting. Puttern Recognition Letters, 25(6):641-6544 2004.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C. Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC '19, pp. 113-125, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781 9781450376280. 3359790. URL https:: //tp::://oi... www.ddi........oooooooeeeooooo.. 10 01145/33399899 ..1.793999
doi: 10.1145/3359789,
Jonas Geiping, Liam Fowl, w Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint arXiv:2009.02276 2020.
Tianyu Gu, Brendan Dolan-Gave and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint urXiv:1708.063336 2017.
Tianyu Gu, Brendan Gavit,,, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.067. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. arXiv preprint arXiv: 2016b.
Jeremy Howard. Imagenette: A smaller subset of casily classified classes from imagenet, March 2019. Utthttps:: / h/tp://g.hhht....// com wwsaai///////..........a.....a htttm///aaaaaammmmeatetttee magenette)
Matthew Jagielski. Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru and Bo Li. Ma- nipulating machine learning: Poisoning attacks and countermeasures for regression learning. arXiv preprint arXiv: a 1804.00304 2021.
10

---

Published as a conference paper at ICLR 2023
Jinyuan Jia, Yupei Liu, Xiaoyu Cao, and Neil Zhenqiang Gong. Certified robustness of nearest neighbors against data poisoning and backdoor attacks. AAAI, 2022.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pp. 2304-2313, PMLR, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report 0, University of Toronto, Toronto, Ontario, 2009.
M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in neural information processing systems, 23:1189-1197, 2010.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1607.v660.02220
2016.
Yi Liu, Rong Jin, and Anil K Jain. Boostcluster: boosting clustering by pairwise constraints. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 450-459, 2007.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0608. 2017.
Deyu Meng, Qian Zhao, and Lu Jiang. What objective does self-paced learning indeed optimize? arXiv preprint A.IV::5511.00455 2016.
Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. In Proceedings the AAAI conference on artificial intelligence, volume 33, pp. 4610-4611 2019.
Maria- Irina Nicolae, Mathicu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wis- tuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian M. Molloy, and Ben Edwards. Adversarial robustness toolbox v1.0.0. arXiv preprint arXiv: arXiv:/807.00 1807.01069, 2019.
Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? unified benchmark for backdoor and data poisoning attacks, 2021.
Shawn Shan, Arjun Nitin Bhagoji, Haitao Zheng, and Ben Y Zhao. Poison forensics: Traceback of data poisoning attacks in neural networks. In 31st USENIX Security Symposium (USENIX Security 22), pp. 3575-3592, 2022.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss mini- mization. arXiv preprint 1rXiv:1810.11874 2019.
Hossein Souri, Liam H Fowl, Rama Chellappa. Micah Goldblum, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Pro- cessing Systems, 2022. URL https: www.revieeeeeeeppeeeee.e.v net boorumidd=poLJaa _a6____6__MMMM__________
J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking ma- chine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332, 2012. ISSN 0893-6080. doi:
URL https::/ wwww.
sciencedirect com/ http://ecee//areaa///////3331000 pers from IJCNN 2011.
Selected Pa-
Jacob Steinhardt, Pang Wei W Koh, and Percy $ Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. arXiv 2018
preprint arXiv:
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. 2018.
11

---

Published as a conference paper at ICLR 2023
Wenxiao Wang, Alexander J Levine, and Soheil Feizi. Improved certified defenses against data poi- soning with (deterministic) finite aggregation. In International Conference on Machine Learning, pp. (27769227788 PMLR, 2022a.
Zhibo Wang, Jingjing Ma, Xue Wang, Jiahui Hu, Zhan Qin, and Kui Ren. Threats training: A survey of poisoning attacks and defenses on machine learning systems. ACM Comput. Surv,, 55 (7), dec 2022b. ISSN 0360-0300.
doi: 10. 10.1145/3538707. URL https:://oi.czq//
1145/3538707
Dongkuan Xu and Yingjie Tian. A comprehensive survey of clustering algorithms. Annals of Data Science, 2:1, 2015.
Linxiao Yang, Ngai-Man Cheung, Jiaying Li, and Jun Fang. Deep clustering by gaussian mixture variational autoencoders with graph embedding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6440-6449, 2019.
Xuchao Zhang, Xian Wu, Fanglan Chen, Liang Zhao, and Chang- Thang-Tien Lu. Self-paced robust learn- ing for leveraging clean labels in noisy data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6853-6860, 2020.
Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Zhao li, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, and Martin Ester. A comprehensive survey on deep clustering: Taxonomy, chal- lenges, and future directions, 2022. UR/httpss:/arrii..om wwtp://wwwrr///////e////////s/// /tt..///22....... C7579
Zhuowen Tu. Probabilistic boosting-tree: learning discriminative models for classification, recog- nition, and clustering. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 2, pp. 1589-1596 Vol. 2, 2005. doi: 10.1 10.1100//CC.
12

---

Published as a conference paper at ICLR 2023
Cean
Poison
10
-5
-10
Figure 4: The construction used in our proof of Theorem 3.2 for N 2 dimensions. The balls de- pict the support of the clean training distribution (blue) and the poisoned data (red). The perturbation function is shown by the grey arrows. The lines the ground truth maximum margin classifier (blue) and the maximum margin classifier when training with a mixture of clean and poisoned data (red). The red and blue arrows point in the direction of half-plane labelled by the corresponding classifier as the positive class.
A DEFERRED PROOFS
Proof of Theorem 3.2 First we give construction when the dimension N = 2. We define the training distribution as D (D+ + D22 where D4 and 'D_ are both distributed arbitrarily within a ball of radius 1 (c.g..
truncated Gaussian); D4 is centered at (10, 0) with labels +1 and The adversary chooses the perturbation function r which flips the label during training and sets the last entry be 10 if the original label was +1, and -10 the original label was 1. Note that 7 is bounded the sense that as - oc, the (expected) magnitude data point can grow as O(vN) while the magnitude the perturbation is 0(1)- Figure 4 depicts the construction.
D_ is centered at (-10, 0) with labels
Clearly, one sample from each clean class is sufficient to learn a maximum margin classifier (MMC) achieving zero population risk, i.e., the MMC will be close
01, O.
Next, we show that the attacker succeeds under the threat model in Section 3.1 Recall that the attacker selects 1/4 of the positive samples and 1/4 of the negative samples poison. Given poi- soned dataset containing at least one example of each class (both clean and poisoned), the classifier achieves perfect accuracy on both the clean data and the poisoned data, and hence the MMC also has poison misclassification rate of 1.
given by (1, 2)),
We begin by showing that the poisoned data is not compatible with the clean data. Let C' be any set of clean data, and P be any set of poisoned data. We have two cases. First, C contains samples of both classes, then by definition, any subset C" of C' in the measurement of the self-expansion error will also include at least one sample of each clean class. Hence, the MMC learned from C' will close to (1, b 0, which achieves perfect accuracy on all clean data. On the other hand, if C contains only samples of the positive class, the MMC w = (0, 0), b = +1, and again all samples in C' are correctly classified. The same argument works when C' contains only samples from the negative class. Thus, in neither case can poisoned data improve the self-expansion error C.
We now show that this setup satisfies the incompatibility property defined in Section 3.2
As the exact same line of reasoning holds for proving incompatibility of clean data with respect to poisoned data, we omit the proof. Therefore the clean and poisoned data are mutually completely incompatible.
13

---

Published as a conference paper at ICLR 2023
To generalize this to N The entire construction can also be scaled then translated and rotated without affecting any of the conclusions.
can simply embed D in the first two dimensions of the space.
we first prove a useful lemma: Lemma A.1. Ler A and B be arbitrary sets, and define S = -AUB B. Then for all a,
Before moving on to the proof of Theorem 2.3
()) () + [B((BB),;a)
with the convention that <W a:) for all T and a.
Proof. Notice that all the expansion errors sample from the same training set S = AU B. The lemma then follows from the linearity the unnormalized empirical risk the test set, i.e.,
| (|(((S E (Amp(A(S)):))
(10)
[1s|-
Σ
(11)
Egnn
Σ
((i)())((i)..))
Σ
(12)
(ELIB))A
(E..)))B
[el) Eg-5 | Remp (A(S'): A) + +Esins
(13)
|A)c(A|B:a) + (B)
(14)
as claimed.
Proof of Theorem 2.3 Assume first that the incompatibility is not strict. Define Smin 0 A and
V S O B. From Theorem A.1 we have that
min
S Sminkc = [
(15)
Applying now the definition of incompatibility gives
(16)
As |U| + IV| =S S it follows that at least one of ((U;a) e(V;a) is at most et, which contradicts the optimality S (as it was assumed to be a smallest set achieving e*).
Now consider the case when least one of the incompatibilities is strict. Let S* be any set in S* Using the definition of strict incompatibility, we get that
| > (U]e(U:a) + [V)e(V:a)
(17)
Hence, at least one of aU aV: is strictly less than c", which again contradicts the opti- mality of S".
Proof of Theorem 3.4 Let S. and Sy be a clean component and poison component, respectively. By the second inequality in Property 3.3 we have that Remp(A(S.):: 2 1/2. Thus S; votes False on Sy. Conversely, both S; and Sj are clean components, then Remp (A(S.): S,) < 1/2, so S; votes True on Sy. Putting these together and using the fact that at least half of the data is clean, we find that the poisoned components have weighted vote strictly less than [S|/2, while the clean components have weighted vote strictly greater than /8) As all the clean components vote correctly, using the weighted majority correctly identifies all the clean and poisoned components as claimed.
Proof of Proposition 2.4 Recall that = 1 and 7) = 0. We prove the statement in two steps.
First, we show that
S1,80) S F(0.,
(18)
14

---

Published as a conference paper at ICLR 2023
The inequality follows from the optimization Line 5 in Algorithm which sets 0u11 to the empirical risk minimizer of the set S,. Next, we claim that
F( St+1: Bo41) S F(
(19)
Since [(:::)] S 1, the optimal size of the set St is |Sa] = B.S)). Since 3r is decreasing, we have that |St+1 S [S&]. Thus the number of elements in the trimmed empirical loss is non-increasing (Line 7, Algorithm
Combining the two inequalities shows that the objective function is decreasing in t. Since F(Or, Sr; Be) is a decreasing sequence bounded from below by zero, the monotone convergence theorem gives the second result.
B DATASET CONSTRUCTION DETAILS
Our implementation of the 1-to-1 dirty label backdoor (DLBD) adversary follows the threat model described in Gu al. (2017). For evaluation, we use the same dataset (CIFAR-I0 (Krizhevsky & Hinton 2009)) and setup for our experiments as the Spectral Signatures work (Tran et al. 2018). Each scenario has single source and target class, and we use the same (source, target) pairs as in Tran et al. (2018); (airplane, bird), (automobile, cat), (bird, dog), (cat, dog), (cat, horse), (horse, deer), (ship, frog), (truck, bird).
The perturbation function overlays a small pattern on the image fixed location. All patterns fit within a 3x3 pixel box. To generate a perturbation, we choose shape (L-shape, X-shape, or pixel) uniformly at random. The (x.y) coordinates of the perturbation are randomly selected to guar- antee that the entire shape is visible before data augmentation (e.g., the pixel-based perturbation can be placed anywhere within the 32x32 image, but the X- X-shape is larger and must be centered in a 30x30 region, one pixel away from the border). The color of the perturbation is also selected uniformly at random, with each of the GR, coordinates ranging from 0 to 255. Finally, ran- domly select an = 5, 10, 20% percentage of the source class, apply the perturbation by replacing the pixels in the corresponding locations with the selected shape and color, then relabel the poisoned images as the target class. For the all-to-1 case, for given target class, we poison an equal propor- tion of every non-target class such that the total amount of poison e times the size of the source class. For the all-to-all case, we select cyclic permutation of the classes, then poison a percentage of each class.
The construction of the WATERMARK dataset is the same, except that we instead use 8x8 images depicting: peace sign, the letter A, bullet holes, or a colorful patch, taken from Nicolae et al. (2019). Each trigger has a transparent background, and is blended into the upper left corner of the original image with 80% opacity (where 0% opacity would return the original image without the trigger, and 100% opacity would superimpose the trigger on top of the original image).
For the CLBD dataset, we used the official datasets provided by Turner et al. (2018). There are three datasets in total, one for each perturbation type: Eg, foc. and GAN. For the &g and foo -based attacks, each image is perturbed independently to maximize the loss with respect to a reference model trained on clean data; the perturbation size is bounded by the respective norm. For the GAN-based attack, for each image to be poisoned, a perceptually similar target image in another class is identified using the latent space of the GAN, and the perturbed image an interpolation of the original and target images in the latent space. In all three cases, 3x3 checkerboard patch is then placed in each of the 4 corners of the image. The label of the image is not changed. At test time, only the checkerboard patches are placed on the image. We refer the reader to Turner et al. (2018) for more details.
For the GTSRB dataset, we use the same construction of the DLBD and WATERMARK attacks on CIFAR- 10. As the CLBD dataset was only created for the CIFAR-10 10 dataset, we did not evaluate the CLBD attack on GTSRB
15

---

Published as a conference paper at ICLR 2023
C DEFENSE SETUPS AND HYPERPARAMETERS
ISPL Boosting (this work). For our defense, we use the set of hyperparameters described here unless otherwise noted. For CIFAR-10 10, we run rounds of ISPL, each of which returns a component consisting of roughly 12% of the total samples. For GTSRB, we run 4 rounds of ISPL that each component contains 24% of the total samples. Let p be the target percentage of samples over the remaining samples (e.g., for CIFAR-10 p 1/(8
+ 1) in the 7th iteration). Then the number of iterations N' is set to 2 + min(3, 1/p)- 3 starts at 3 p in the first iteration, then drops linearly to its final value of p over the next 2 iterations. When trimming the training set, we also additionally include the top p/8 samples per class to prevent the network from collapsing to a trivial solution. For the learning procedure A. we use standard SGD, trained for 4 epochs per iteration, with warm-up in the first iteration of 8 epochs. The expansion factor a is set to 1/4, and the momentum factor is set to 0.9.
We run ISPL 3 times to generate 24 weak learners for CIFAR-10, and 12 weak learners for GTSRB. For the boosting framework, each weak learner trained for 40 epochs on its respective subset, then votes on a per-sample basis. The sample is preserved if the modal vote equals the given label, with ties broken randomly, or the sample is in the lower half of its class by loss.
For CIFAR-10, we also include final self-training step by training fresh model for 100 epochs on the recovered samples. The main idea is that a model fit to the full "clean" training data can be used to test the excluded training data, thereby recovering additional consistent data which may have been originally excluded because the weak learners were fit to a small subset of data for fewer epochs. However, it may take several repetitions of training a model from scratch before this self- training process no longer identifies new samples to recover. Therefore, we use a simple self-paced learning algorithm to dynamically adjust the samples during training to limit the self-training to a single iteration. More explicitly, we start with the "clean" samples as returned by the boosting framework. Every 5 epochs, we update the training set be the samples whose labels agree with the model's current predictions. Due to the relative frequency with which we resample the training set, we smooth the predictions by momentum factor of 0.8 so that the training process is less noisy. The samples used for training in the last epoch are returned as the defended dataset. In our experiments, this process decreases the false positive rate (and thus increases the clean accuracy) but does not materially affect the false negative rate (nor the targeted misclassification rate). We did not use self-training for GTSRB as we found the clean accuracy was sufficiently high.
Spectral Signatures. We use the official implementation of the Spectral Signatures (SS) defense (Tran al. 2018) by the authors, available on Github, except that replace the training procedure with PyTorch (instead of Tensorflow 1.x as in the authors' original implementation). The authors suggest removing 1.5 times the maximum expected amount of poison from each class for the de- fense. We remove 20% of each class for e = 5, 10% (to match the procedure in their paper) and 30% of each class for 20%. In selecting the layer for the activations, for the ResNet32 architecture we use the input to the third block of the third layer (taken from the SS defense authors' public implementation), and for the Ne architecture we use the input to the first block of the fourth layer (which was found empirically to remove the most poison on the first set of scenarios). We note that the authors indicate the defense should be fairly successful at any of the later layers of the network.
Iterative Trimmed Loss Minimization. The Iterative Trimmed Loss Minimization (ITLM) de- fense (Shen & Sanghavi, (219)) consists of an iterative procedure. Given a setting 0 < a S 1, one first trains a model for a number of epochs. Then the a fraction of samples with the lowest loss are retained for the next iteration. This process is repeated several times, with a fresh model beginning each iteration. The defended dataset is the a fraction of samples with the lowest loss after the last iteration. For the backdoor data poisoning experiments on CIFAR-10, the authors use 80 epochs for the first round of training, then 40 epochs thereafter; they also set a = 98% for e = 5%, and do not test at other values of e. We use the same settings, and scale a linearly with e, i.e., = 96% for e = 10% and a = 92% for e = 20%.
Activation Clustering. The Activation Clustering (AC) defense (Chen et al. 2018) has an actively maintained official implementation in the Adversarial Robustness Toolbox (ART) (Nicolae et al.;
16

---

Published as a conference paper at ICLR 2023
open-source collection of tools for security in machine learning. We use the official implementation with the default parameters values ART v1.6.2. In selecting the layer for the activations, used the same layers as for Spectral Signatures.
2019).
Models. For the ResNet32 (He et al. 2016a) model, we use vanilla SGD with learning rate 0.1, momentum 0.9, and weight decay le-4. For the final dataset, we train for 100 epochs (unless oth- erwise noted) and drop the learning rate by 10 at epochs 75 and 90. Using these parameters, we achieve around 94.5% accuracy on GTSRB when trained and tested with clean data.
For the PreActResNer18 8 (He et al. 201 model, we use vanilla SGD with learning rate 0.02, mo- mentum 0.9, and weight decay 5e-4. For the final dataset, we train for 100 epochs (unless otherwise noted) and drop the learning rate by 10 at epochs 50, 75, and 90. Using these parameters, we achieve around 93. 33.0% accuracy on C-FAR-10 when trained and tested with clean data.
For the ResNet56 (He et al. 2016b) model, we use vanilla SGD with learning rate 0.05, momentum 0.9, and weight decay 5e-4. For the final dataset, we train for 100 epochs (unless otherwise noted) and drop the learning rate by 10 at epochs 75, 90, 95, and 98. Using these parameters, we achieve around 91.0% accuracy on Imagenette when trained and tested with clean data.
17

---

Published as a conference paper at ICLR 2023
D ADDITIONAL EXPERIMENTAL RESULTS
This section contains additional experimental results to complement the main text. Appendix D.1 ex-
pands upon the empirical evaluations of the compatibility property from Section 4.1 Appendix D.2 reports comprehensive results from the evaluations of the ISPL+B defense reported in Section 4.2 Appendix DappendixD.2aahaee also includes a detailed breakdown of the performance for the three defenses (Spectral Signatures, Activation Clustering, and Iterative Trimmed Loss Minimization) we use as baselines in our comparisons. Appendix D.2.1 reports results for our defense against the Sleeper Agent attack (Souri et al. 2022), and Appendix D.2.2 reports results for the DLBD attack on a high-resolution dataset. Appendix D.2.3 evaluates our defense against an adaptive attack that is given white-box access to the ISPL subroutine, and selects which elements to poison based on the behavior of ISPL on clean data. Our results show that ISPL+B maintains excellent performance against the adaptive attacker, defending against 20 out of 24 scenarios. Appendix rep.4ts results for the CIFAR-10 DLBD attack after training for 200 epochs on both the PreAAtRes.Net18 and ResNet32 architec- tures, to test the robustness of our results to the training setup. Finally, evaluate the sensitivity of ISPL to the main hyperparameters a and B, Appendix D.2.5 compares the performance of ISPL. on selection of attack scenarios across a range of and B3. Our main finding is that performance remains high for range of choices of a and 3.
All results reported use the median over random seeds, unless otherwise noted.
D.1 INCOMPATIBILITY EVALUATION
Figures 9 displays aggregated results of the empirical study of incompatibility for the CLBD, DLBD, and WATERMARK attacks on CIFAR-10. 10. Plots on the left side measure the compatibility of poisoned with clean data (i.e.. e(C;a) a()) and plots on the right side measure the compatibility of clean with poisoned data (i.e., (P; - ((P(C:a)
In all cases, the trend in the first plot is decreasing, which suggests that the gap increases in mag- nitude as the size of the incompatible data grows. Additionally, the clean label scenarios all exhibit the behavior that the source class accuracy increases as the amount of poison (and poisoned accu- racy) increases, while the non-source accuracy drops. These attacks share the characteristic that the poisoned data are "harder" versions of the source class in the sense that they are perturbed toward instances of other classes. In the case of the GAN-based attack, the perturbations occur in the latent space of a GAN trained on clean data (e.g., the attacker identifies a bird and an airplane that are close in the latent space, linearly interpolates the latent representation of the airplane toward the latent representation of the bird, then uses the generative network to convert the perturbed represen- tation into an image of a "bird-like airplane"). In the case of the l and E.x bounds, the attacker uses the Projected Gradient Descent (PGD) algorithm commonly used for adversarial training (Kurakin) et al., 2016; Madry et al., (017) to identify an image within norm nomn-ddudddd neighborhood of the original image that maximizes the loss of a trained network, i.c., the perturbed image is close in pixel-space to the original image, but was misclassified by the trained network different class. We therefore attribute the simultaneous increase in source class accuracy and decrease in non-source class accuracies to similar mechanism for all the clean label cases.
Conversely, when measuring the compatibility of poisoned data with clean data (plots on the left) for the DLBD and WATERMARK attacks, we see that, as the amount of poisoned data increases, the source class accuracy drops. The implication is that the poisoned data causes the classifier to misclassify clean data by presenting similar (both quantitatively in terms of the distance between the poisoned and clean distributions, as well as perceptually in most settings) instances of clean source data that are mislabelled as the target class, thus complicating the learning of the clean source class.
Figures 14 plot the scenarios individually. We note that incompatibility holds generally across the range scenarios, with only small deviations to the contrary.
18

---

Published as a conference paper at ICLR 2023
Performance Paéson Ratio CIFAR-10 CLSD GAN
Performence Clean Ratio CIFAR-10 CLBD GAN
Alaulutte Chargee Accutayyy
Absullute Chainge Accuracy
Souaci
10-
0.3
0.6
0.1
0.6
0.7
Relio of Puisan Dala
Relio of Clean Dala
Figure 5: Aggregated compatibility results for the GAN-based CLBD attack CIFAR-10.
Performace os. Pakkn Ratio CIFAR110 CLBD
Performance ve Clean Ratie CIFAR-10 CLBD 12
Posson
Absolute Change in Accuracy
Sant
har-suite Sounce
10
Nowwwute
Aboulute
-1-
Poius
0.1
62
03
0.6
05
0.6
37
0.8
0.0
60
0.1
0.1
05
0.6
07
69
Ralio of Poison Dala
Ret of Clean Data
Figure 6: Aggregated compatibility results for the ba-ba CLBD attack on CIFAR-10.
Performence
Absohulie Chai got Accuraicy 196) Nonoouutte
11
Alautuute Che ge
Phoum
Figure Aggregated compatibility results for the lxx- -based CLBD attack on CIFAR-10.
19

---

Published as a conference paper at ICLR 2023
Perfommance
Rerformance OIFRR110 wazrmark
CIFAN-10
2000011
Absciute Changge Accuracy
Norwousee
Absulule
Chaam
Ratlo Polson
Rat Clean
Figure Aggregated compatibility results for the WATERMARK attack on CIFAR-10.
Performance Paison Rabi CIRV 10 DLBD
Performance vs. Clean Rabi CIFN 10 DL5D
Alaublee Chargge Accuratty
Abscolute Chargee in Accuraty
500000
Poous horeetuee
50
Chan
00
0.9
0.6
09
Rabio of Poison Data
Rat of Clean Data
Figure 9; Aggregated compatibility results for the DLBD attack on CIFAR-10.
Figure 10: Full compatibility results for the GAN-based CLBD attack on CIFAR-10.
20

---

Published as a conference paper at ICLR 2023
Figure 11: Full compatibility results for the ba-baded CLBD attack on CIFAR-10.
Coo
Figure 12: Full compatibility results for the loo based CLBD attack on CIFAR-10.
21

---

Published as a conference paper at ICLR 2023
Figure 13: Full compatibility results for the DLBD attack on CIFAR-10.
Figure 14: Full compatibility results for the WATERMARK attack on CIF
22

---

Published as a conference paper at ICLR 2023
D.2 DEFENSE EVALUATION
This section includes additional experimental results concerning the performance of our defense. Tables 8 display the unabridged results of our main evaluation results for our defense (condensed in the main text as Table D. We note that even though we have selected 1% as the threshold for a "successful" defense, in many of the failure cases, ISPL+B still achieves relatively low TMR. For instance, setting the threshold to 5% increases the number of successful scenarios to 142 (compared 134 1%, out of 165 total).
Table 2 also includes results of the three baseline defenses for the standard 1-to-1 DLBD attack on CIFAR- 10. In particular, we observe that our clean accuracy is comparable to (or even slightly better than) that of Activation Clustering (AC), the next best defense, despite AC achieving significantly lower TMR our evaluations.
Table 2: Performance the CIFAR-10 DLBD scenario (1-to-1) using the PreActResNer18 archi- tecture. The S / column lists the CIFAR-10 source and target classes. refers to the percentage of the source class which is poisoned. For the remainder of the columns, the top level column headers give the defense type: Oracle (training only on clean data), ND (no defense), SS (spectral signa- tures), AC (activation clustering), ITLM (iterative trimmed loss minimization, and ISPL+B (ISPL + boosting, this work); the second level column headers give the metric type: C (clean accuracy, higher is better), (targeted misclassification rate, lower better), FP (false positives, lower better), FN (false negatives, lower is better). Successful runs (TMR < 1%) are highlighted.
Oracle C
ND
SS
AC
ITLM
ISPL+B
S/T
C
A
C
A
C
A
C
A
C
A
FP
FN
93.3
93.1
17.2
92.5
2.1
90.3
92.5
4.7
90.9
0.0
3761
23
0/2
10
93.1
0.0
92.8
42.6
91.3
18.2
90.7
14.7
92.5
43.7
90.6
0.0
3811
22
20
93.1
0.0
92.4
79.8
89.5
4.6
90.3
23.9
92.1
59.4
90.7
0.2
3773
118
93.1
0.0
92.7
1.4
92.4
0.3
90.6
0.3
92.6
0.0
90.9
0.0
3787
2
1/3
10
93.0
0.0
92.4
11.1
91.3
1.1
90.3
0.4
92.3
7.7
90.8
0.0
3758
5
93.1
92.5
55.7
88.2
4.3
90.0
4.8
92.2
51.6
90.6
0.0
3819
5
93.1
0.8
92.9
81.7
92.8
81.7
89.4
73.5
92.8
81.0
90.5
1.2
3778
44
2/5
92.9
83.3
92.1
81.0
90.4
76.2
92.5
82.1
90.6
0.6
3928
47
20
93.0
0.8
92.9
81.2
88.2
72.9
90.9
76.2
92.4
77.9
90.8
74.4
3552
981
93.2
92.8
72.9
92.6
69.9
91.0
56.5
92.8
68.0
90.8
0.6
3843
26
3/5
10
93.1
0.2
92.9
86.6
91.8
76.8
90.8
76.7
92.8
87.6
91.0
0.4
3759
42
20
93.1
0.2
92.8
89.3
90.2
81.7
90.4
80.6
92.7
88.9
90.4
0.2
3925
59
93.1
0.2
92.9
8.0
92.3
2.0
90.0
0.8
92.8
0.3
90.9
0.1
3656
12
3/7
10
92.9
0.0
92.8
46.6
91.3
20.8
89.7
3.4
92.4
36.5
90.7
0.1
3725
19
20
93.0
0.0
92.5
60.8
88.6
36.0
90.7
59.1
92.4
73.3
90.3
0.2
3895
100
93.0
0.0
93.2
76.2
92.5
58.2
90.7
15.6
92.6
40.6
91.0
0.0
3729
2
7/4
10
93.1
0.0
93.3
87.8
92.0
80.1
90.3
72.5
92.6
84.2
90.7
0.0
3781
3
93.2
93.1
92.9
88.8
36.3
90.6
74.0
92.6
90.8
0.0
3637
8
93.1
0.0
93.0
3.0
92.5
1.3
89.6
0.3
92.9
0.6
90.8
0.0
3818
2
8/6
10
93.3
93.1
87.8
91.2
47.0
91.0
2.9
92.3
15.3
90.6
0.0
4018
2
20
93.2
0.0
93.1
93.2
89.3
7.4
90.4
62.1
92.5
89.4
91.0
0.0
3697
4
93.3
92.9
75.4
92.7
14.5
90.3
31.0
92.7
1.0
91.0
0.0
3625
4
9/2
10
93.1
0.0
93.0
83.0
91.8
77.6
90.9
79.8
92.6
81.2
90.8
0.2
3718
37
20
93.0
0.0
92.3
41.2
89.4
76.6
91.1
78.5
92.6
81.2
90.6
0.2
3699
74
23

---

Published as a conference paper at ICLR 2023
Table Performance on the CIFAR-10 DLBD scenario (all-to-1) using the PreActResNet11 archi- tecture. The T column lists the target class (i.e., all poisoned images have label T). Each of the source classes contain e/9 percent poison.
Oracle C
ND
ISPL.+B
T
A
FN
5
93.2
0.0
93.0
0.1
91.1
0.0
3624
9
93.1
0.1
93.0
90.9
0.0
3632
25
20
93.3
0.0
92.3
74.2
90.6
0.1
3770
49
93.1
0.1
3921
8
3
93.0
0.1
92.4
25.4
90.6
0.1
3855
24
20
93.3
0.1
92.8
87.6
90.3
0.1
3947
67
5
93.2
0.2
93.0
62.8
90.8
0.1
3682
6
5
10
93.0
0.2
93.1
92.2
90.7
0.1
3764
21
20
93.2
0.1
92.9
92.1
90.9
0.5
3698
66
5
93.3
0.1
93.0
76.2
90.9
0.2
3605
8
7
93.0
0.1
92.8
83.3
90.8
0.2
3542
21
93.0
0.1
92.8
90.9
0.5
3592
39
Table 4: Performance on the CIFAR-10 DLBD scenario (all-to-all) using the PreActResNet118 ar- chitecture. The $ T column lists offset of the eyclic permutation (i.e., 2 means that source class 0 is poisoned as target class 2, source class is poisoned as target class 3, etc.). refers to the percentage of the source class which is poisoned.
Oracle
ND
ISPL+B
S
C
A
C
A
FP
FN
92.7
0.0
92.4
78.0
90.3
0.1
3790
130
2
10
92.8
0.0
92.5
87.8
90.0
0.1
3648
256
92.2
0.1
92.6
88.2
88.2
0.2
3695
821
92.8
0.0
93.2
88.2
90.6
0.0
3664
58
10
92.5
0.0
92.9
89.7
89.6
0.0
3892
172
20
92.2
0.0
92.6
90.7
88.4
0.1
3902
384
93.1
93.0
90.1
90.7
3446
5
10
92.8
0.0
93.2
90.6
90.4
8.9
3416
410
20
92.4
0.1
92.9
91.0
90.3
88.5
3367
9027
93.1
0.1
93.1
85.6
90.6
0.1
3505
58
7
10
92.7
0.0
93.2
89.0
90.6
69.4
3532
3576
20
92.5
0.0
93.2
90.0
90.6
86.7
3076
8952
24

---

Published as a conference paper at ICLR 2023
Table 5: Performance on the CIFAR-10 WATERMARK scenario (1-to- 1) using the PreActResNet18 architecture. The S column lists the CIFAR-10 source and target classes, refers to the percent- age of the source class which is poisoned.
Oracle C
ND
ISPL+B
S/T
A
A
A
FP
FN
93.1
0.2
93.3
3826
0/2
10
93.2
0.3
93.2
84.5
90.6
4.4
3868
20
93.1
0.1
93.0
87.0
90.6
76.8
3741
787
5
93.1
0.0
93.0
47.7
90.8
0.0
3752
1/3
10
93.0
0.0
93.1
74.4
90.8
0.0
3746
20
93.2
0.0
92.9
85,8
90.9
0.0
3634
11
5
93.0
2.0
93.0
45.2
90.7
11.9
3685
109
2/5
10
93.1
2.4
92.9
64.4
90.7
7.1
3664
122
93.1
2.4
92.9
73.7
35.4
3499
93.0
0.1
45.1
90.6
0.2
3814
93.1
0.1
93.0
74.3
3715
20
92.9
0.5
92.9
85.1
90.4
21.7
3796
289
93.2
0.7
92.9
84.2
90.7
6.5
3920
3 3/7
10
93.1
0.9
92.9
85.9
90.7
73.5
3717
406
20
93.3
1.2
92.9
86.1
91.0
74.9
3772
928
5
92.9
0.0
93.1
77.6
90.8
0.0
3877
7/4
10
92.9
0.0
92.9
82.9
90.7
0.0
3756
20
93.0
0.0
92.9
89.4
90.6
72.4
3907
711
5
93.1
0.0
93.1
82.4
90.7
0.0
3639
8/6
93.2
0.0
93.2
87.7
90.7
0.1
3751
20
93.1
0.0
93.2
92.5
90.5
83.2
3795
859
93.1
0.3
93.1
69.6
3634
18
9/2
10
92.9
0.1
92.9
75.0
90.7
0.4
3707
17
20
92.9
0.2
92.8
78.8
90.6
1.3
3519
58
25

---

Published as a conference paper at ICLR 2023
Table 6: Performance on the CIFAR-10 CLBD scenario using the PreActResNet18 architecture. The T column lists the target class (i.e., all poisoned images have label T). refers the percentage of the target class which is poisoned.
Oracle C
ND
ISPL+B
M
T
A
FP
FN
93.2
3654
192
0
93.1
0.0
93.0
0.1
90.9
0.1
3763
406
20
93.1
0.0
93.3
0.3
91.0
0.2
3688
968
93.0
0.1
93.2
1.3
90.7
0.2
3635
129
2
10
92.9
0.1
92.9
12.1
90.7
0.4
3739
324
20
93.1
0.1
93.3
27.2
90.8
4.3
3621
907
GAN
5
92.9
0.1
93.3
0.2
90.9
0.1
3666
204
4
10
92.9
0.1
93.0
0.4
90.8
0.1
3735
406
20
92.9
0.0
93.1
3.4
91.0
0.3
3532
875
5
93.1
0.0
93.2
0.5
90.9
0.0
3763
124
7
93.3
0.0
93.3
4.8
90.6
0.1
3832
217
92.9
93.0
25.2
3595
574
93.2
0.0
93.0
0.3
91.0
0.1
3742
143
93.1
90.9
3663
20
93.1
0.0
93.2
10.6
90.7
0.7
3943
913
93.0
93.2
3633
162
0
93.2
0.0
93.1
6.1
90.6
0.1
3747
351
20
92.9
0.0
93.2
45.8
90.6
1.1
3848
757
92.9
0.0
93.2
30.1
91.0
0.2
3694
95
2
10
93.2
0.1
93.1
68.9
90.4
0.2
3877
219
20
92.9
0.1
93.1
76.4
90.6
20.3
3716
633
82
5
93.0
0.1
93.5
6.8
90.9
0.1
3639
140
4
10
93.0
0.1
93.0
57.4
90.6
0.5
3793
299
93.0
0.0
93.1
74.4
90.2
3731
587
5
92.9
0.0
93.0
0.3
91.0
0.1
3769
157
7
93.3
0.0
22.1
90.7
0.1
3796
366
20
92.9
0.0
93.1
68.0
90.5
0.4
3932
732
93.1
0.1
3871
174
9
93.4
0.1
93.2
12.5
90.6
3846
332
20
92.9
0.0
93.1
68.6
90.9
0.9
3622
731
5
93.2
0.0
93.1
0.5
91.1
0.0
3732
109
0
10
93.1
0.0
93.0
3.8
91.1
0.1
3591
301
20
93.2
0.0
93.1
51.3
90.7
0.2
3686
729
5
93.1
0.1
93.2
13.7
90.8
0.1
3723
44
2
10
93.3
0.1
93.3
52.9
90.9
0.2
3641
120
20
93.0
0.1
93.0
66.2
90.4
1.0
4157
430
too
5
93.1
0.1
93.2
0.6
90.7
0.1
3919
67
4
93.2
0.1
93.1
14.4
90.8
0.1
3918
183
93.1
93.1
4.1
90.6
3807
716
93.2
0.0
93.0
1.0
90.6
0.0
3592
97
93.2
93.0
32.6
90.6
3780
235
20
93.0
0.0
93.1
67.4
90.5
0.2
3625
492
93.1
0.1
3683
109
9
93.1
0.0
93.1
3.7
90.6
0.2
3929
308
20
93.2
0.1
93.4
1.0
90.7
0.2
3697
792
26

---

Published as a conference paper at ICLR 2023
Table 7: Performance on the GTSRB DLBD scenario (I-to-1) 1) using the ResNet32 architecture. The S/ /TT column lists the GTSRB source and target classes. refers the percentage of the source class which is poisoned.
Oracle C
ND
ISPL+B
S/T
A
C
A
C
A
FP
FN
5
94.2
0.0
94.4
95.1
628
0/2
10
94.9
0.0
94.2
0.0
94.8
0.0
591
0
20
94.4
0.0
94.4
0.0
94.1
0.0
558
1
5
93.9
0.0
94.5
0.1
94.4
0.0
327
1/2
10
94.3
0.0
94.5
0.1
94.3
0.0
430
8
20
94.2
0.0
93.7
53.2
93.7
0.1
418
25
5
94.5
0.0
94.1
87.5
94.3
0.0
582
0
1/0
10
94.0
0.0
94.6
97.6
94.5
0.0
460
6
20
94.2
0.0
94.5
97.9
93.0
82.6
569
79
94.2
0.0
94.5
91.9
94.2
89.7
269
75
2/
94.3
0.0
94.8
97.5
94.7
95.7
366
150
20
94.2
0.0
93.8
96.8
93.9
98.1
472
300
94.5
0.0
94.1
94.2
453
69
38 /
10
94.1
0.0
92.7
3.0
93.8
2.3
668
138
20
94.6
0.0
94.1
9.3
93.7
30.1
665
276
5
94.3
0.0
94.0
47.4
95.2
0.0
616
0
31 / 24
10
94.2
0.0
93.5
60.7
93.8
0.0
553
0
20
93.9
0.0
93.9
83.0
94.7
0.0
643
7
5
94.5
0.0
94.5
0.5
94,1
0.0
674
7
15 / 29
10
94.3
0.0
94.4
0.5
94.5
0.0
649
20
94.1
0.0
94.4
0.0
94.6
0.5
706
74
92.8
0.0
93.8
61.9
94.3
586
2
9/18
10
93.8
0.0
94.6
96,9
94.4
0.0
607
2
20
94.3
0.0
94.2
98.8
94.3
0.0
586
26
27

---

Published as a conference paper at ICLR 2023
Table 8: Performance on the GTSRB WATERMARK scenario (I-to-1) using the ResNet32 architec- ture. The S/ T column lists the GTSRB source and target classes. refers to the percentage of the source class which is poisoned.
Oracle C
ND
ISPL+B
S/T
A
C
A
C
A
FP
FN
5
94.6
0.0
94.5
93.0
369
0/2
10
94.4
0.0
93.8
0.0
94.0
0.0
416
0
20
94.1
0.0
94.1
0.0
93.7
0.0
783
4
5
94.4
0.0
94.5
10.3
94.2
0.0
452
1/2
10
94.1
0.0
94.2
54,0
94.7
0.0
359
8
20
94.5
0.0
94.1
77.5
94.2
0.0
747
17
5
94.2
0.0
94.1
0.6
93.7
0.0
379
0
1/0
10
94.9
0.0
94.2
10.7
94.3
0.0
305
5
20
94.9
0.0
94.5
73.3
93.1
360
94.5
0.0
94.1
1.6
94.7
270
2/
93.7
0.0
93.9
32.1
94.8
278
20
94.4
0.0
94.0
76.9
94.2
0.1
401
300
94.4
0.0
94.2
50.0
93.9
308
38 /
10
94.2
0.0
93.3
65.4
94.3
0.0
273
0
20
94.2
0.0
94.2
78.1
94.0
66.4
685
275
5
94.4
0.0
94.5
69.3
94.0
0.0
378
0
31 / 24
10
94.6
0.0
94.7
81.1
94.3
0.0
292
0
20
94.8
0.0
94.5
80.4
94.4
0.0
566
8
5
94.4
0.0
94.5
1.0
94.2
0.0
302
0
15 / 29
10
94.5
0.0
94.1
0.5
94.6
0.0
344
0
20
94.4
0.0
94.5
47.6
94.4
19.0
482
59
93.9
0.0
93.8
6.9
94.3
331
9/18
10
94.3
0.0
94.3
75.0
94.3
0.0
269
0
20
94.8
0.0
94.6
94.0
94.2
0.0
482
0
28

---

Published as a conference paper at ICLR 2023
D.2.1 SLEEPER AGENT ATTACK
Sleeper Agent (Souri al. 2022) clean label attack that uses gradient matching (Geiping al, 2020) to create a poisoned training set. The objective of the attacker is to induce a model trained on the poisoned training set to miselassify patched versions of source class as the target class. Figure [6] displays examples of clean and poisoned training and test data created by the Sleeper Agent attack, where the source class horse and the target class is dog (training with the poisoned dogs causes patched horses to be misclassified as dogs). Similar to the CLBD attack, Sleeper Agent perturbs each image in the training set independently subject to an foo bound, however, the Sleeper Agent attack is more subtle because does not insert any patches during training. The Sleeper Agent attacker also differs from the DLBD, CLBD, and WATERMARK attacks in using an informed mechanism to select which subset of the training set D to poison (rather than poisoning a random subset). Finally, unlike prior gradient matching approaches, the Sleeper Agent attack is also intended apply in the black-box setting, where the attacker does not have access the victim model's 's architecture or training hyperparameters.
We first used the official repository released by the authors generate 24 poisoned datasets with the default settings for CIFAR-I0 using different random seeds, targeting a ResNet32 as the victim architecture]] | We observed an attack success rate (ASR) of 49.1% (#38.3%) over the generated datasets, when training ResNet32 model with the attacker's ' original training hyperparameters. We then transferred the poisoned dataset to the setting of our defense (training ResNet32, using the same training and defense hyperparameters
in our main experiments for ResNet32 on CIFAR- 10). Our defense delivers an ASR of 4.3% (1..1%) 1%), compared to an oracle, which achieves an ASR of 3.0% (+.4.%) However, we also observed that the attack only achieves success rate of 4.5% (+4.3%) on an undefended model in our setting. We attribute this anomaly (between the ASR for undefended model in our setting and the ASR reported by the attacker) to the difference in training hyperparameters, despite using the same victim architecture. In all three cases defended, undefended, and oracle- the clean aceuracy is consistent between 89-90%.
We next evaluated the effectiveness of our defense when retraining with the attacker's training setup. Specifically, we took the poisoned training set generated by the attacker, then ran our defense using our training setup to output a new training set with the detected poisoned examples removed. We then used the attacker's 's training setup to train new model our training set, and evaluated this new model on the poisoned test set.
Note that in our threat model, the training algorithm A is fixed, and our defense is designed around the interaction between the data and A. The above scenario is thus explicitly not the intended use case for our algorithm. Nevertheless, our defense still achieves an ASR of 14. 1% (=15.)%) with clean accuracy of 87.5% (+0.6%), compared to the original ASR of 49.1%. An oracle achieves an ASR of 2.4% (+2.)) These results thus indicate that our defense successfully defends against the Sleeper Agent attack in our threat model (where the defense is integrated in the intended training pipeline), while also remaining robust to an evaluation under the attacker's 's original training setup.
Figure 15: Example clean (top) and poisoned (bottom) images from the train (left) and test (right) datasets, taken from one run of Sleeper Agent attack; the source class is horse, and the target class dog (training with the poisoned dogs causes patched horses to be misclassified as dogs).
'lll results in this section are the mean over 24 runs, with the standard error in parentheses, to match the presentation in Souri (202
29

---

Published as a conference paper at ICLR 2023
D.2.2 IMAGENETTE DATASET
Imagenette (Howard 2019 is subset the ImageNet dataset (Deng et al. 2009) consisting of 9,469 training and 3,925 test images over 10 classes. We downsampled the 320x resolution version of Imagenette to 224x224, and use a ResNet56 achieving around 91 % aceuracy on the clean dataset. We used a DLBD adversary that places solid 5x5 patch of a random color in random position on the image. We use this setting to probe the applicability datasets. Figure (igurl/y examples of clean and poisoned data from the Imagenette experiments. reportrr the results of our defense on Imagenette, using the same defense hyperparameters as the main experiments for 10.. The performance of our defense remains consistent, with 21 of the 24 scenarios defended below 1% TMR and less than 2% drop in clean accuracy.
our technique to higher-resolution
Figure 16: Examples of clean (top) and poisoned (bottom) data from the Imagenette experiments.
30

---

Published as a conference paper at ICLR 2023
Table 9: Performance on the Imagenette DLBD scenario (1-to-1) using the ResNet56 architecture.
The S T column lists the Imagenette source and target classes. refers the percentage of the source class which is poisoned.
Oracle A
ND
ISPL+B
S/T
C
A
C
A
FP
FN
90.9
0.0
91.2
85.5
89.3
0.0
544
0/2
10
91.3
0.0
91.3
93.8
89.5
0.0
535
2
20
91.2
0.0
91.0
92.2
88.7
0.3
608
17
5
91.1
0.0
91.3
79.7
89.1
0.0
536
10
1/3
10
90.7
0.0
91.0
87.1
89.2
0.5
513
19
20
91.4
0.0
91.2
91.9
88.9
89.1
586
185
5
91.1
0.0
91.0
0.5
88.7
0.0
619
7
2/5
10
91.5
0.0
90.8
72.8
89.1
0.0
642
23
20
90.9
0.3
90.7
77.2
88.3
0.8
639
65
5
0.0
91.3
69.7
89.5
0.0
554
6
3/5
10
90.8
0.0
91.0
79.0
88.7
599
13
20
91.2
0.0
90.9
79.7
88.5
72.8
572
182
90.6
0.0
90.3
13.2
89.9
0.0
536
3 3/7
10
91.5
0.0
91.0
88.2
89.4
0.0
520
3
20
91.7
0.0
91.1
90.8
88.9
0.0
617
24
5
91.7
0.0
90.7
72.2
89.7
0.0
562
0
7/4 4
10
91.3
0.0
91.3
81.7
89.2
0.0
521
2
20
91.4
0.0
90.6
86.2
89.8
0.0
542
10
5
90.7
0.0
91.3
83.3
89.8
0.0
540
1
8/6
10
91.5
0.0
91.2
91.3
89.1
0.0
569
5
20
90.9
0.0
91.5
92.1
89.1
0.0
495
5
0.0
90.9
17.9
89.6
0.0
496
3
9/2
10
91.0
0.0
90.9
79.8
89.4
0.0
544
4
20
91.4
0.0
90.7
76.2
88.5
2.3
581
58
31

---

Published as a conference paper at ICLR 2023
D.2.3 ADAPTIVE ATTACKER
The attacks used in the main experiments are oblivious the presence of defender when selecting the data to poison. In this section, we devise an strong adaptive attacker with white-box access to the ISPL algorithm when constructing the poisoned dataset.
The success of the defense rests crucial on the partition of training data returned by the ISPL al- gorithm. In order for the boosting phase to succeed, the partition should consist of homogeneous sets, i.e., the poison must be concentrated in small number of sets within the partition; hence, the adaptive attacker's objective to try and distribute the poisoned samples across multiple sets.
In particular, rather than randomly selecting fraction of the source class to poison, an adaptive attacker with white-box access to defense can first run the ISPL algorithm (using same parameters) on the clean dataset to create partition of the dataset, then random select fraction from each partition to poison. The idea is that the ISPL algorithm returns a similar partition, then the poison will be still be distributed across multiple sets, thereby breaking the defense.
Table I0 presents the results our defense against the strong adaptive attacker described above. Against the adaptive attack, our defense still succeeds in 20/24 of the scenarios (versus 22/24 for the standard oblivious attack). These results suggest the ISPL algorithm remains robust even to white- box adaptive attacks.
Table 10: Performance the CIFAR-10 DLBD scenario using the PreActResNer18 architecture with an adaptive attack. The S T column lists the CIFAR-10 source and target classes. e refers to the percentage of the source class which is poisoned.
ISPL+B
S/T
C
A
FN
90.6
0.0
3899
23
0/2
10
91.0
0.0
3594
17
90.6
0.2
3819
90.9
0.0
3733
91.0
0.0
3796
90.8
0.0
3797
90.7
0.9
3741
2/5
10
90.8
64.5
3656
336
20
90.6
74.0
3558
953
90.8
0.3
3562
3/5
10
90.7
8.2
3665
51
20
90.4
69.4
4162
731
90.6
0.1
3665
3/
10
90.6
0.1
3754
23
90.6
0.3
3526
90.8
0.0
3746
7/4
91.0
0.0
3730
20
90.8
0.0
3559
90.7
0.0
3760
8/6 6
10
90.9
0.0
3723
20
90.7
0.0
3818
4
90.8
0.0
3738
11
9/2
10
90.9
0.1
3660
21
90.8
0.4
3640
70
32

---

Published as a conference paper at ICLR 2023
D.2.4 VARYING THE TRAINING SETUP
As prior works (Schwarzschild ularly with DNNs, can vary significantly with the training setup, we additionally ran our defense (along with the three baseline defenses) on the standard CIFAR-10 DLBD attack using both the PreActResNet18 and ResNet32 models, respectively, and report results after training for 200 epochs (instead of 100, as in our main experiments).
2021) have observed that results in data poisoning, partic-
Tables 11 and 12 report the results of these experiments. We observe that the defense success rates are largely consistent with those in the main experiments across all defenses; the biggest change is that the Spectral Signatures defense successfully defends against 9 scenarios instead of just (out of 24 total), ISPL+B still achieves the best performance terms of reducing the targeted misclassification rate below I % (with 2-3% drop in clean accuracy).
Table 11: Performance on the CIFAR- 10 DLBD scenario (I-to-1) using the PreActResNet18 ar- chitecture, when training for the final model for 200 epochs. The S / T column lists the CIFAR-10 source and target classes. refers to the percentage of the source class which is poisoned.
Oracle A
ND
SS
AC
ITLM
ISPL+B
S/T
C
A
C
A
FP
FN
C
A
FP
FN
C
A
FP
PN
C
A
FP
FN
1.3
94.5
91.3
94.5
79.9
7381
130
91.9
58.3
18926
155
94.6
84.9
994
244
92.8
3640
23
0/2
10
1.3
94.1 90.6
94.5
66.0
7383
383
92.3
85.9
19790
320
94.2
92.8
1961
461
92.8
0.0
3479
25
20
1.1
94.6
80.2
94.2
64.2
14335
335
92.4
73.9
19127
601
93.8
93.8
3897
897
91.6
22.9
3711
237
5
0.0
94.4
92.9
94.7
5.5
7319
68
90.9
19.5
19791
155
94.8
96.5
986
236
92.8
0.0
3434
2
1/3
10
0.0
94.6
98.4
94.5
0.0
7035
35
92.5
68.7
19033
333
94.6
98.0
1981
481
93.0
0.0
3348
3
20
99.6 94.4
94.6
14007
92.2
91.8
18317
94.4
99.4
3914
914
3253
5
1.1
94.4
80.4
94.6
76.4
7494
243
92.4
53,9
19937
172
94.7
79.4
985
235
92.6
0.2
3704
5
10
0.9
94.4 97.2
94.4
0.1
7053
53
92.9
81.8
18657
94.7
92.4
990
490
3200
16
20
1.2
94.4
94.0
94.5
87.7
14263
263
92.6
89.4
20531
406
94.6
95.4
966
966
92.8
0.3
3540
38
7.6
94.7
91.0
94.7
88.5
7281
30
92.6
90.8
19172
172
91.4
81.1
3693
167
3/5
10
5.9
94.8
94.0
94.4
8.6
7014
14
92.3
91.6 20293
296
94.6
92.3
988
488
92.6
90.2
3355
496
20
7.6
94.7
90.8
94.3
90.6
14092
92
92.6
92.6 18236
652
94.7
91.7
974
974
92.6
87.4
3210
995
5
0.6
94.3
33.8
94.6
98.3
7500
249
91.9
97.2 21289
246
94.7
98.4
995
245
92.7
0.0
3692
5
3/7
10
0.7
94.4 98.5
94.6
96.6
7135
135
92.2
98.2
20630
484
94.5
98.6
1979
479
92.7
2.4
3427
22
20
0.7
98.5 94.4
93.7
78.5
14675
675
92.3
98.2
20470
987
94.4
98.9
980
980
92.9
10.5
3259
43
5
1.5
94.7
92.0
94.6
0.6
7280
29
92.3
39.2
19353
150
94.8
91.6
992
242
92.8
0.5
3258
30
7/4
10
1.5
94.6
94.7
94.4
0.0
7023
23
92.0
47.9
19631
285
94.4
94.5
1979
479
93.0
0.3
3627
293
20
1.5
94.6
96.5
94.3
0.1
14000
0
92.1
78.6
21292
94.4
3910
910
3203
867
5
0.2
94.7
97.0
94.8
80.5
7470
219
92.8
73.3
19293
152
94.5
98.3
993
243
92.8
0.0
3494
2
8/6
10
94.4
99.5
94.7
0.0
7008
8
92.6
97.6
19544
94.4
99.4
1981
92.9
3571
20
0.2
94.7
99.5
94.4
0.0
14000
0
92.5
96.4
18946
597
94.2
99.5
3908
908
92.6
0.2
3492
8
0.1
95.0
92.0
94.4
93.3
7501
250
91.7
85.0
23573
94.6
97.3
988
238
93.0
3291
9
10
0.1
94.5
93.9
94.3
93.1
7500
500
92.1
95.1
22896
251
94.4
98.6
1970
471
93.1
0.0
3133
1
20
0.1
94.4
96.1
94.7
0.0
14010
10
93.1
98.9
18651
575
94.1
99.0
3906
906
93.1
0.0
3223
2
33

---

Published as a conference paper at ICLR 2023
Table 12: Performance on the CIFAR-10 DLBD scenario (I-to-1) using the ResNet32 architecture, when training for the final model for 200 epochs. The $ T column lists the CIFAR-10 source and target classes. 6 refers to the percentage of the source class which poisoned.
Oracle A
AC
ITLM
ISPL+B
S/T
C
A
C
A
FP
FN
C
A
FP
FN
FP
FN
FP
FN
5
1.1
92.6
85.0
91.8
57.3
7499
248
88.8
0.6
24211
133
91.8
83.3
988
238
89.8
0.0
5752
31
0/2
10
1.1
92.4
92.6
91.7
91.5
7422
422
88.8
78.6
24031
250
91.7
92.3
1969
469
5445
20
1.5
94.9 92.5
90.4
64.2
14590
590
88.6
62.7
23686
455
91.9
94.2
3890
890
88.0
0.0
6992
86
0.1
92.7
98.5
91.7
91.7
5.7
7479
227
88.8
2.4
23903
125
91.9
12.7
988
238
89.8
5776
1/3
10
0.1
91.6
35.0
91.1
3.3
7436
436
88.3
24.9
23891
254
92.0
69.2
988
488
89.8
0.0
5736
3
20
0.0
92.1
98.1
91.5
0.0
14004
4
89.1
0.4
21145
70
91.9
97.3
3900
900
89.4
0.0
5833
5
1.7
92.2
62.7
91.2
43.2
7493
242
88.0
1.4
24001
128
92.2
75.7
994
244
88.8
0.1
6358
6
2/5
10
1.6
92.5
92.4
91.9
89.8
7476
476
89.3
81.5
23827
275
91.6
93.2
1975
475
89.2
0.0
5985
26
20
91.7
95.8
89.9
3.5
14349
349
88.5
87.2
21086
91.9
95.1
3905
905
89.8
5684
40
5
6.3
91.3
88.9
91.7
87.9
7466
215
88.9
80.0
23817
131
92.2
90.8
996
246
89.2
27.5
5974
59
3/5
10
6.2
91.8
90.8
91.6
86.4
7348
348
88.7
73.1
21299
89.5
990
82.1
5681
344
20
6.3
92.3
90.8
90.5
71.9
14090
90
89.3
78.5
20694
156
90.9
88.8
3918
918
86.7
5093
994
92.6
98.4
91.0
97.1
7452
201
86.2
72.8
23823
92.1
97.1
994
244
5498
13
3/7
10
1.1
92.4
98.6
91.8
96.7
7315
315
88.6
96.2
23648
482
92.3
98.0
1981
481
89.8
0.0
5126
16
20
1.1
92.9
98.6
90.9
97.0
14167
167
89.1 95.4
20600
551
92.4
98.1
3924
924
88.6
0.2
6642
26
5
1.7
92.1
88.5
91.6
91.6
87.5
7486
235
89.3
72.2
23928
149
92.4
92.2
992
242
88.8
0.4
6643
27
7/4
10
1.7
92.7
93.9
91.9 94.2
7371
371
88.3
59.2
23753
193
92.1
96.2
1973
473
88.9
0.8
6478
32
20
1.7
92.6
96.9
91.1
94.1
14397
397
88.2
47.6
23737
423
91.7
95.8
3904
904
88.6
46.9
6322
209
5
0.2
92.7
98.0
91.3
97.8
7441
190
89.1
88.7
23658
164
92.5
96.9
988
238
89.6
0.0
5991
0
8/6
10
0.2
92.1
97.7
91.8 97.2
7089
89
90.2
95.1
19585
280
92.3
98.7
973
473
89.3
0.0
6007
2
20
0.2
92.8
98.8
91.3
96.0
14297
297
87.2
64.2
23347
646
92.3
98.6
975
975
89.4
5691
5
0.1
92.9
93.2
91.2
93.2
7478
225
88.7
1.2
23518
136
92.1
94.0
991
241
90.3
0.0
5444
2
10
0.1
92.6
98.6
91.4
94.7
7497
497
88.5
91.5
24034
92.5
97.8
88.1
7220
20
0.1
92.6
97.4
90.5
0.0
14011
11
90.6
97.7
18658
578
91.9
99.2
3881
811
89.6
0.0
5742
1
34

---

Published as a conference paper at ICLR 2023
D.2.5 SENSITIVITY OF ISPL TO HYPERPARAMETERS
We next conduct some ablation studies better understand the effects of the two main hyperpa- rameters in Algorithm D the expansion factor a and the subset size 3. Computationally, larger / means fewer components and fewer outer iterations of ISPL (and is thus more efficient); in our main experiments, we use / 1/8 and run ISPL times in sequence to generate 8 components, Addi- tionally, when estimating the self-expansion error, a larger leads to lower variance estimates the expansion error, but less effective measures of generalization; hence, we should prefer taking as large as possible to produce hetter estimates (until memorization effects take over). Our experiments use both 1/2 (for GTSRB, due to the imbalanced class sizes) and 1/4 (for CIFAR-10).
Tables 13 and present our results from the ablation studies, conducted on subset of the CIFAR- 10 DLBD and CLBD subsets. We do not include the self-training step after ISPL as in the main experiments, in order to better highlight the trends. In particular, without self-training, we expect both lower clean accuracies and higher targeted misclassification rates due to not recovering all the compatible clean data.
Our main finding is that our method is quite robust both the expansion factor and subset size 9. In particular, 3 E [1/8, 1/16 and a E [1, 1 /4] all yield strong defenses with less than 10% targeted misclassification. As expected, increasing leads to a stronger defense but lower clean accuracies. In particular, 3 = 1 (which identifies only a single subset consisting of 96% of the original dataset) fails in all settings of a; this the setting for which ISPL+B conceptually the most similar the ITLM defense, which offers a possibly explanation for ITLM's 'ss poor performance in our experi- ments. Additionally,
avoids memorization, mainly because do not train until convergence within the ISPI. loop. Finally, we note that 1 1/2 appears to achieve a modest optimum, which is particularly noticeable when paired with smaller 3.
Table 13: Average clean accuracies after running ISPL on three scenarios of CIFAR-10 DLBD and CLBD using the PreActResNett18 architecture, with various settings of and B.
B
a
1/2
1/4
1/8
1/16
average
1
92.55
91.65
90.37
87.92
85,41
89.58
1/2
92.63
91.85
90.25
87.28
83.54
89.11
1/4
92.64
91.93
89.92
86.52
83.01
88.80
1 1/8
92.70
91.69
89.21
86.56
83.83
88.80
1 16
92.58
91.63
89.63
86.98
85.15
89.20
average
92.62
91.75
89,88
87.05
84.19
Table 14: Average targeted misclassification rates after running ISPL on three scenarios of CIFAR- 10 DLBD and CLBD using the PreAAtResNet118 architecture, with various settings of and B.
B
a
1/2
1
1/8
1/16
average
29.21
15.38
12.06
5.88
7.43
13.99
1/2
30.52
15.74
12.26
3.54
4.18
13.25
1/4
39.15
15.12
11.10
7.07
9.59
16.41
1/8
36.54
21.99
12.71
10.96
10.78
18.60
1/16
37.15
26.31
12.60
11.91
11.67
19.93
average
34.51
18.91
12.15
7.87
8.73
35

---

Published as a conference paper at ICLR 2023
E ADDITIONAL RELATED WORK
Deep Clustering. The literature on clustering is vast, and we refer the reader to Xu & Tian (2015) for survey of classical techniques. Classical techniques often do scale to large, high- dimensional datasets, so recent years have seen increasing development in techniques for deep clus- tering. Such techniques generally use deep learning to embed some high-dimensional input data into a now-dimensional space (such as the latent space of a generative adversarial network (Mukherjee al., 2019) or variational autoencoder (Yang et al. 2019), then perform a classical clustering on the embeddings (Zhou et al. 2022). The structure revealed by the clustering is thus tied to the ob- jective used to learn the dimensinsional embedding, e.g., the reconstruction loss. In contrast, our clustering mechanism is based on an interaction between the dataset and the training process, and therefore produces clusters based on the incompatibility of subsets with respect to specific training objective (and can also be applied to any class of learned model).
Compatibility. Our separation results in the context of data poisoning can be viewed as clustering by exploiting weak supervision in the form of (possibly poisoned) class labels. [Balcan et al. (2005) introduce a method called "co-training", and show that under a similar compatibility property, learn- ers fit independently two different "views" of the data can supervise each other improve the joint performance. However, they (and similar works) assume access to a small set of trusted labels not present our setting and do not consider the case of malicious training data.
Boosting. To identify which components belong together and recover the primary distribution, we use a method that can interpreted as a basic form of boosting, in which an ensemble weak learners is aggregated into stronger learner. For instance, the seminal boosting algorithm AdaBoost (Freund et al. 1996) adaptively reweights the training set so that subsequent weak learners focus on samples of poor performance. The use of boosting methods for clustering has also been explored previously (Frossyniotis et al. 22044] Zhuowen Tu [2005] Liu et al.) (007)) ; as far as we know, we are the first to use our incompatibility objective cluster data.
Self-Paced Learning. Self-Paced learning (SPL) (Kumar et al. 2010) is type of curriculum learning (CL) (Bengio et al. 2009) that dynamically creates a curriculum by including samples of inereasing difficulty based the losses of the partially trained model. Prior works generally apply SPL to improve convergence on noisy datasets (Meng et al., 2016 Jiang et al., (2088) Zhang al. 2020). In contrast, ISPL. discards progressively easier samples identify subset with good expansion; to the best of our knowledge we are also the first to apply CL. ideas to defend against backdoor attacks.
36


Reviews:
[{"review_id": "wpJ5xCCHZ0j", "reviewer": "Reviewer_HYv9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the need to detect backdoor poisoned data, in which a subset of training data is permuted with a watermark, resulting in some elements of a testing set being incorrectly classified.  The paper first identifies an incompatibility property, in which poisoned training data does not improve model accuracy on clean data, and vice-versa.  They show this property applies with high probability to maximum-margin classifiers whose data have been backdoor poisoned.  They then introduce an algorithm which leverages this property to identify clean data.  Finally, they introduce an algorithm which loosens many of the requirements for the above, and tractably trains a classifier on data which is likely to be clean.", "summary_of_the_review": "This is potentially an important result.  The bones of the work is interesting, useful, and potentially provides one of the first real defenses against data poisoning I've seen.  However, there are significant unexplained aspects of the experimental setup that make it difficult to judge and impossible to reproduce.  With editing, this would be an acceptable paper; without, it is not.", "strength_and_weaknesses": "The paper addresses the important data poisoning problem.  The approach is innovative and interesting, and effective defenses to data poisoning are rare enough that even incremental improvement on this problem is important and worth reading.  Experimental results are promising, and incompatibility provides a useful way of thinking about data poisoning.\n\nThe experiments, however, are very poorly introduced.  I was unclear if these were still being run on the maximum margin classifiers from the theoretical results, or if they were being run on something more \"realistic.\"  I believe from the appendix they were run on ResNet variants, but it is still somewhat unclear.  This is obviously a very important point to be unclear on: does the theory, understandably developed on linear classifiers, extend well to more modern, nonlinear classifiers, or not?\n\nI am also interested in whether the incompatibility properties hold, in theory or in practice, for other types of data poisoning that allow for distinct perturbations on individual training images.  I suspect this will be every reader's first question, and so it should be addressed in the paper.  Incompatibility is much more significant if it does hold on these other types of poisoning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i1DunnUYS5y", "reviewer": "Reviewer_Sqdp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper notices that poisoned and clean \"domains\" do not help classify one another.  Therefore, identifying the subset of the data that does not help classify the rest can defend against poisoning attacks.  The paper also proposes a computationally feasible approach to this defense strategy and tries it on several datasets and compares its effectiveness against three existing defenses.", "summary_of_the_review": "In general, the concept and execution are strong.  Several problems exist in the writing and some additional experiments and error bars should be added.  Nonetheless, I would tend to accept this paper.", "strength_and_weaknesses": "I really like the concept of this paper, the writing is clear, and the experimental evaluations are solid.  I especially appreciate that you consider an adaptive attacker.\n\nI would make a suggestion though.  You might consider trying gradient-based backdoor attacks such as Sleeper Agent since (1) it may have very different properties than the types of poisons you already consider which contain patches or 8x8 watermarks in the poisons, and (2) you may be able to use it for a stronger adaptive attack.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z5XOib0a1n", "reviewer": "Reviewer_ZDJY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper identifies an incompatibility property of the interaction of clean and poisoned data, i.e., involving poisoned data does not improve model performance on clean data and vice versa. The authors then leverage this property to develop a detection algorithm for finding the clean samples among the poisoned dataset. It is achieved by first separating the dataset into multiple subsets of either all clean data or all poisoned data, and then identifying the clean data among these subsets through a voting algorithm. The proposed approach ISPL+B demonstrates great performance against various types of backdoor attacks.", "summary_of_the_review": "The problem is important and the solution is novel. The paper quality is good. I only have a few comments and questions. I'm leaning to acceptance of the paper.", "strength_and_weaknesses": "**Strengths**\n\n1. The authors identified an interesting universal property intrinsic to the backdoor poisoning attack, which inspires the design of defense algorithms. \n2. The authors conducted extensive evaluations and demonstrated effectiveness of their approach.\n\n**Weaknesses**\n\nI do not have major complaints about the proposed method. Below are a few comments which I hope can help improve the paper or questions to assist my understanding.\n\n1. A highly-relevant related work [1] is missing. This submission shares a similar flavor with [1]. Both works are developed based on the intuition that clean samples will not help the performance on poisoned samples, and both works iteratively refine the subsets. I recommend the authors to add [1] in Sec 6 related work and discuss the similarities and differences.\n2. The approach proposed in [1] requires access to a trained poisoned model as well as a misclassification event during testing time (their setup is forensics); your work does not assume access to these. It is pretty normal to assume the access to a trained poisoned model ([2] uses this assumption as well; see their Sec. 6). Do you think this knowledge can be leveraged to improve your approach in any principled way?\n3. I understand the goal of learning a parameter $\\tilde \\theta$ such that it is as close as possible to $\\theta^\\star=\\mathcal{A}(C)$ which is a model obtained on clean data only (Sec 2.2). But I have a few following thoughts: 1. How close is it to the model obtained on the original dataset $D=C \\cup A$? This can be answered through experiments (and I'd appreciate it if the authors can show some results.) 2. If the gap is large in the first step (e.g., when the size of $A$ is almost the same as that of $C$, is it possible to rectify the poisoned set $P$ to its original form $A$ in some way? This could be an interesting direction for future work.\n4. The definition of compatibility Def 3.2 is not symmetric. Could the authors elaborate more on this point?\n5. For the Incompatibility property in Def 3.3, I'm curious how well is it satisfied in the practical attacks, and thus have the two following questions: a) Can we propose a proxy to evaluate this? b) If an attack doesn't satisfy it very well, what's the reason, and how significant it will degrade the defense?\n\nExperiments wise:\n\n6. Issues with Fig. 2 and the descriptions: a) left column, 2nd subfigure from the top: 0 is above 10 in the y-axis; this looks incorrect to me. b) right column, I do not understand the \"source\" and \"non-source\" for CLBD. There is no \"source\" but only \"target\" in CLBD. I find the interpretations of the subfigure difficult to understand. c) The last row shows absolute change in accuracy on poisoned data; I would love to see the absolute accuracy on poisoned data as well to understand how effective the attack is.\n7. In Sec. 5.2, the authors compared to \"training on the original clean samples\". Does it refer to training on $C$ or $D$? (This is related to point 3.)\n8. In Sec. 5.2 \"Comparison with Existing Approaches\", the authors said they compared with \"3 existing defenses\". I would recommend the authors to add the names or at least references for the three methods in this sentence.\n\n**Minors**\n\n* In Eq. (1), $A_{emp}$ should be $R_{emp}$.\n* Typos\n  * \"as an absolute different in accuracy\" --> \"difference\"\n \n\n\n**References**\n\n[1] Shan, Shawn, et al. \"Poison Forensics: Traceback of Data Poisoning Attacks in Neural Networks.\" 31st USENIX Security Symposium (USENIX Security 22). 2022.\n\n[2] Cui, Ganqu, et al. \"A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks.\" Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WDHyq2IrTS", "reviewer": "Reviewer_F4Dd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The backdoor attack has recently received increased attention from the community. The incompatibility property is proposed in this paper in terms of the interaction of clean and poisoned data with the training algorithm, specifically that including poisoned data in the training dataset does not improve model accuracy on clean data and vice versa. The results of the experiments show that prior dirty-label and clean-label backdoor attacks in the literature produce poisoned datasets with behavior consistent with the incompatibility property. Based on this motivation, the authors propose to partition the original dataset into disjoint clean and poisoned components. Experiment results on the CIFAR-10 and GTSRB datasets show that the proposed defense method can reduce attack success rates to less than 1% in the majority of scenarios with negligible accuracy drop.\n\n=========\nBecause the authors' response partially addressed my concern, I have raised my rating to 6.", "summary_of_the_review": "Although this work has some merit, I am concerned about its applicability to more difficult real-world datasets with high-resolution images.", "strength_and_weaknesses": "Strength:\n- Addressing the backdoor defense is an important and timely topic right now.\n- The paper is overall well written.\n- The proposed incompatibility property is novel, and experimental results validate its effectiveness in defending against backdoor attacks.\n\nWeaknesses:\n- This work considers the case where the defenders have complete access to the poisoned training set. The CIFAR-10 and GTSRB datasets used in the experiments are both low-resolution, allowing the defenders to easily examine the entire dataset to find the poisoned samples. In other words, the two toy datasets used are far too simple. The successful implementation of the incompatibility property on these two datasets does not imply that the proposed incompatibility property will work well for more difficult high-resolution real-world datasets (e.g., 224 by 224 resolution images with 5 by 5 resolution trigger patterns). It would be great if the authors could demonstrate the applicability of the proposed incompatibility property to these difficult cases.\n- When the defenders have no knowledge of the ratio of poisoning samples, will the proposed partition solution still be useful?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster