Backdoor or Feature? A New Perspective on Data Poisoning
Authors: Alaa Khaddaj
Guillaume Leclerc
Aleksandar Makelov
Kristian Georgiev
Andrew Ilyas
Hadi Salman
Aleksander Madry
Venue: Submitted to ICLR 2023 (2023)

Abstract: In a backdoor attack, an adversary adds maliciously constructed ("backdoor") examples into a training set to make the resulting model
vulnerable to manipulation. Defending against such attacks---that is, finding and removing the backdoor examples---typically involves viewing these examples as outliers and using techniques from robust statistics to detect and remove them.

In this work, we present a new perspective on backdoor attacks. We argue that without structural information on the training data distribution, backdoor attacks are indistinguishable from naturally-occuring features in the data (and thus impossible to ``detect'' in a general sense). To circumvent this impossibility, we assume that a backdoor attack corresponds to the strongest feature in the training data. Under this assumption---which we make formal---we develop a new framework for detecting backdoor attacks. Our framework naturally gives rise to a corresponding algorithm whose efficacy we show both theoretically and experimentally.

BACKDOOR OR FEATURE? A NEW PERSPECTIVE ON DATA POISONING
Anonymous authors Paper under double-blind review
ABSTRACT
a backdoor attack, an adversary adds maliciously constructed ("backdoor") examples into training set to make the resulting model vulnerable to manipula- tion. Defending against such attacks that is, finding and removing the backdoor examples- -typiallyyy involves viewing these examples outliers and using tech- niques from robust statistics to detect and remove them.
In this work, we present a new perspective on backdoor attacks. We argue that without structural information on the training data distribution, backdoor attacks are indistinguishable from ceturally-occuriigg features in the data (and thus impossible to "detect" in general sense). To circumvent this impossibility, we assume that a backdoor attack corresponds to the strongesl feature in the training data. Under this assumption which we make formal- we develop a new framework for detecting backdoor attacks. Our framework naturally gives rise to corresponding algorithm whose efficacy we show both theoretically and experimentally.
INTRODUCTION
A backdoor attack is technique that allows an adversary to manipulate the predictions of supervised machine learning model (Gu al., 2017: Chen al., 2017; Adi et al., 2018; Shafahi et al., 2018 8; Turner et al., 2019). To mount a backdoor attack, an adversary modifies small subset of the training inputs a systematic way, e.g., by adding a fixed "trigger" pattern; the adversary then modifies all the corresponding targets, e.g., by setting them all to some fixed value y/. This intervention allows the adversary to manipulate the resulting models' predictions at test time, e.g., by inserting the trigger into test inputs.
Given the threat posed by backdoor attacks, there is an increasing interest in defending ML models against them. One such line of work (Jin al., 2021; Tran al., 2018; Hayase et al., 2021a; Chen et al., 2018) aims to detect and remove the manipulated samples from the training set. Another line of work (Levine & Feizi, 2021; Jia al., 2021) seeks to directly train ML models that are robust against backdoor attacks (without necessarily removing any training samples).
prevailing perspective on defending against backdoor attacks treats the manipulated samples as outliers, and thus draws parallel between backdoor attacks and the classical data poisoning setting of robust statistics. In the latter setting, one receives data that is from a known distribution with probability 1 1-6 e, and adversarially chosen with probability 8-the goal is to detect (or learn in spite of) the adversarially chosen points. This perspective is natural one to take and has lead to a host of defenses against backdoor attacks, but is it the right way to approach the problem?
In this work, we take a step back from the above intuition and offer new perspective on data poisoning: rather than viewing the manipulated images as outliers, we view the trigger pattern itself as just another feature
feature in the data. Specifically, we demonstrate that backdoors inserted in a dataset can be indistinguishable from features already present in that dataset. On one hand, this immediately pinpoints the difficulty of detecting backdoor attacks, especially when they can correspond to arbitrary patterns. On the other hand, this new perspective suggests there might be equivalence between detecting backdoor attacks and surfacing features in the data.
Equipped with this perspective, introduce framework for featuing in input data and characterizing their strength. Within this framework, we can view backdoor attacks simply as particularly strong features. Furthermore, the framework naturally gives rise algorithm for

---

detecting using datamodels (Ilyas al., 2022) the strongest features in given dataset. We use this algorithm to detect and remove backdoor training examples, and provide theoretical guarantees on its performance. Finally, demonstrate through range of experiments the effectiveness of our framework in detecting backdoor examples for a variety of standard backdoor attacks.
Concretely, our contributions
follows:
We argue that the absence of any knowledge about the distribution of natural image data, backdoor attacks are in a natural sense indistinguishable from existing features in the data.
We make this intuition more precise by providing a formal definition of feature that naturally captures backdoor triggers as a subcase. We then re-frame the problem of detecting backdoor examples as one detecting particular feature in the data. To make the problem feasible (i.e., to distinguish the backdoor feature from the others), we assume that the backdoor is the strongest feature in the training set-an assumption we make formal.
Under this assumption, we show how to leverage datamodels (Ilyas et al., 2022) to detect backdoor examples. We provide theoretical guarantees on our approach's effectiveness at identifying backdoor examples.
We show experimentally that our algorithm (or rather, an efficient approximation to it) effectively identifies backdoor training examples in range of experiments.
2 A FEATURE-BASER PERSPECTIVE ON BACKDOOR ATTACKS
The prevailing perspective on backdoor attacks casts them an instance of data poisoning, a concept with rich history in robust statistics (Hampel et al., 201 1). In data poisoning. the goal is to learn from dataset where most of the points-say D, and the remaining points (an e-fraction) are chosen by adversary. The parallel between this "classical" data poisoning setting and that of backdoor attacks is natural. After all, a backdoor attack the adversary inserts the trigger in only a small fraction of the data, which is otherwise primarily drawn from a data distribution D.
- =) fraction are drawn from distribution
This threat model is tightly connected the classical poisoning setting in robust statistics. In the classical setting, the structure of the dataset D is essential to obtaining any theoretical guarantees. For example, the developed algorithms often leverage strong explicit distributional assumptions, e.g. (sub~ GGussianity (Lugosi & Mendelson, 2019). In settings such as computer vision, however, no such structure is known. In fact, lack almost any characterization of how benchmark image datasets are distributed. In this section, we argue that without such structure,
Backdoor attacks are fundamentally indistinguishable from features already present in the dataset.
Backdoor attacks can be "realistic" features. First, we show that one can mount backdoor attack using features that are already present in the dataset. In Figure 1, we mount a backdoor attack on ImageNet (Deng et al., 2009) by using hats place of a fixed trigger pattern. The resulting dataset is entirely plausible in that the images are (at least somewhat) realistic, and the corresponding labels are unchanged-wilt some more careful photo editing, one could imagine embedding the hats in a way that makes the dataset look unmodified even to human. At test time, however, the hats act as an effective backdoor trigger: model predictions are skewed towards cats whenever a hat added on the test sample. Should we expect backdoor detection algorithum flag these examples?
Backdoor attacks can occur naturally. In fact, the adversary need not modify the dataset at all-one can use features already present in the dataset manipulate models test time. For example, a naturally-occuring trigger for ImageNet is a simple image of a tennis ball-we provide more details about the "tennis ball" trigger in Appendix C.
These examples highlight that without making additional assumptions, trigger patterns for backdoor attacks are more than features in the data. Thus detecting them should be easier than detecting hats, backgrounds, or any other spurious correlation the data. In the next sections, we will use this insight to craft more specific conditions under which we can hope detect backdoor examples.
2

---

(a) Sample images of cats with generated hats
Output distribution
Top Predicted Classes
104
Poisoned
Cats
Validation Set
Frequency
Clean
Prequeney
Not Cats
Clean Accuracy
Poisoned
Accuricy
107
63.72%
42.04%
102
200
ImageNel Classes (sarted by frequency)
Class
(c) Model accuracy
(b) Predictions a poisoned R on the fully poisoned validation set
Figure 1: An adversary craft trigger that is indistinguishable from natural feature and use it as a backdoor. (a) We "backdoor" the ImageNet training set by generating (using 3DB (Leclere et al., 2021)) images of hats and pasting them on 20% of the cats images of ImageNet (Deng et al., 2009). We train a ResNet-18 (He et al., 2015) on the backdoored training set, and evaluate it on both the clean validation set, and validation set with the trigger added each image. (b) On the clean validation set, model predictions are distributed uniformly across classes (as one would expect); on the backdoored validation set, predictions are skewed towards cat classes, (c) The accuracy of the model drops from 63.72% on the clean validation set to 42.04% on the poisoned validation set.
3 WHEN CAN WE DETECT POISONED SAMPLES?
The results of the previous section suggest that without distributional assumptions, the deliniation between a backdoor and a naturally-occurring feature largely artificial. In this section, we work towards making this observation more precise. We first offer concrete but simplistic definition of feature, which (as expected) captures backdoor attacks as a subcase. We then try to characterize the conditions under which we can distinguish a backdoor from a naturally-occurinn feature.
For task with input space X' and label space y, and for fixed dataset S (X' x x))", a feature is simply a function Φ E X' - 1), For example, é might map from an image x E X' to whether the image contains "dog fur." Rather than focusing on features themselves, however, in this work we will mainly think of features in terms of their supports.
Definition 1 (Feature support). For a feature & X + {0, 1}. its support (SS) is the subset of the training set S where the feature f is present, ie.., ((5) = {2 S
Clearly, this definition enables us capture backdoor attacks as features-the the feature function simply detects the trigger and the corresponding support o(S) is the set of backdoor training examples. Being able to distinguish backdoors from other features in the dataset thus necessitates making additional assumptions. Intuitively, we assume that the backdoor is the strongest feature present in the poisoned dataset. This is captured in the following informal assumption, which we later (in Assumption 1) make precise:
Informal Assumption 1 (Backdoors are strong features). Let Φ be a feature corresponding to a backdoor attack, and let (S) be its support (i.e., the backdoor examples). We assume that adding backdoor example (x (p(x)) = 1) to the training set changes predictions on the backdoor examples (p(S) more than adding an example x ((z) changes predictions on ((S).
To make the above more precise, we formally define a learning algorithm's sensitivity to given feature Φ. To that end, we define the margin function (x: S) as follows:
Definition 2 (Margin function). For dataset St C S and a fixed input X, the margin function f(;; is defined as
f(x;S) == the correct-class margin on of model trained on S'
While this definition is technically correct, in practice would want to limit ø & for some restricted class of "valid" features 4 In this case, all of our results still hold with only minor adaptations.
3

---

Average Margin
Average Margin
-5
Poisoned Image
Poisoned Image
Image
Image
25
30
35
40
45
100
110
120
133
140
Number Poisoned Samples
Number Poisoned Samples
Figure 2: We plot how the average margin for poisoned and clean images changes the number of poisoned samples in the dataset increases. Each plot is a different backdoor attack on the CIFAR- 10 training set, for which we randomly select few backdoor (orange) and non-backdoor (blue) examples. For each attack, we sample (100,000) random 3-fraction subsets of the training set and train model on each one. We then stratify the models by the number of backdoor training examples they were trained on, and plot this number (on the axis) against the average margin of the models on each selected example (on the y axis). The (instantenous) slopes sensitivities so(-, k) of the corresponding examples to the poison feature Øp. We provide details on our experimental setup in Appendix D.
k of the lines are the
where the correct-class margin is the logit of the correct class minus the largest incorrect logit.
Intuitively, (x; maps from an example x and any subset of the training dataset S' C S to the correct-class margin after training (using any fixed learning algorithm) S'. Now, define the sensitivity of an example x to a feature ø as:
Definition 3 (Sensitivity to feature Φ). Given feature o X - 10, 1] with support @(S) C S. the sensitivity of an input æ to the feature & is
80(x, k) = Essras ((x::)))
- - Es ((or:S)) (ó(S') =b]
(I)
where for any E (0, we let S' aS represent sumpling random o-fruction subset of S.
Intuitively, sy (x, k) captures the effect on example of adding one more example with feature e to the training set. Note that while (z, k) is technically function of a, we drop this dependence in our notation since we do not vary throughout our analysis while all our results hold for any (0, 1), in our experiments we simply fix
To provide further intuition for Definition in Figure 2 we illustrate examples' sensitivity backdoor feature. In particular, for two datasets containing backdoor trigger, we randomly select a few inputs both containing and not containing the backdoor trigger. We plot models' margin on function of the number of backdoor examples included in the training set. For any k E N and x E S, the sensitivity SΦp (x. k) as defined in (1) is simply the slope at z-value k of the corresponding line in Figure 2.
these examples
We are now ready to give formal version of Informal Assumption 1:
Assumption 1. Let Up be the backdoor feature, and let Sp be the support (i.e., the training examples containing a backdoor). Then, for some (0, and all features & with == [0p(S)] (o(S)
Σ
SØp (x, a p)p.pppp 5
Σ
(
xEdr(()
REO(S)
This assumption directly inspires our algorithm for detecting backdoor attacks.
3.1 APPROXIMATING SENSITIVITY USING DATAMODELS
Our goal now is to translate Assumption into an algorithm for detecting backdoor training examples. To accomplish this goal, need a way of estimating model sensitivity s (x, k) for a given feature
Again, throughout experiments simply fix
4

---

Paisoned
Poisoned
True Sensitivity
Clean
True Sensitivity
Clean
0.02
0,02
0.01
0.00
0.00
-0.1
0002
0.22
-0.005
0.000
0,005
0,010
-0,002
0,000
0,022
0,004
Datamodels Istimate
Datamodels Estimate
Figure 3: We plot for two different experiments the sensitivities (y axis) and the corresponding data- model approximations (z axis), computed with (4). We observe- -atervve-rrttiularryy for poisoned samples (in orange) good linear correlation between sensitivities and the corresponding approximations.
é and example Now, if we had specific feature in mind, we could simply compute (1) (from Definition 3) directly, In our case, however, finding the feature & of interest precisely our goal. To this end, in this section we show how to estimate sensitivity for all possible features simultaneously.
Our key tool here will the recently proposed datamodeling framework (Ilyas et al., 2022). In particular, Ilyas et al. (2022) show experimentally that (when corresponds to training deep neural network) for every example there exists weight vector wa E RISI such that for subsets S aS,
BIf(;; 1g
where s E (0, 13181 the indicator vector S In other words, we can approximate the outcome of training deep neural network on given subset S' C C as linear function of the presence of each training data example.
It turns that this property alone-captured sensitivity of any example r to any feature øb
a formal assumption below-suffices estimate the
Assumption (Datamodel accuracy). For any example x, let wa be the corresponding datamodel weight. We assume that the datamodel approximates model outputs well on unseen sets, i.e., that
E.gaa
-3...) Se.
Assumption 2 essentially guarantees that datamodels provide an accurate estimate of the average margin for random subsets S ~ aS. Note that we can in fact verify this assumption by sampling sets S' and computing (3) directly (replacing inner expectation with an empirical average). turns out that this Assumption alone suffices to estimate the senstivity of example to a feature s. as shown the following Lemma.
Lemma 1. For a feature o, let 10(s) be the indicator vector of support, and h R -> Ra be
1
h(w) =
|ell v - ||ll|1 (1n - v).
Then, given that Assumption holds, we have that for all there exists some C such that
(s ( w. h(1 S C/2
Proof. See Appendix
Lemma 1 provides a closed-form expression-iivollving only the datamodel w. w the (approxi- mate) sensitivity of example feature á.
We validate our assumptions in Figure 3 by comparing approximate sensitivities (derived from Lemma 1) true example sensitivities (i.e., the slopes in Figure 2). Our results show that sensitivities as estimated by datamodels (as in (4)) are good approximations of ground truth sensitivities. (We refer the reader to Appendix D for details on how calculate ground-truth sensistivities.)
5

---

3.2 POISONED SAMPLES AS MAXIMUM- SUM SUBMATRIX PROBLEM
In the previous section, we showed how to leverage datamodels (and in particular, Assumption 2) to estimate model sensitivity. In particular, Lemma 1 tells us that (assuming they are accurate) we can use datamodels to approximate examples' sensitivity to all possible features @ simultaneously. In this section, we transform Lemma i-in combination with Assumption (i.e., that the backdoor constitutes the strongest feature in the dataset)-ing training examples.
algorithm that provably finds backdoor
To this end, recall that = |S| and p ( Assumption I says that Enco, 60, (z, p) is large-frroe from here, would like to obtain optimization procedure that allows to identify poisoned examples. Inspired by Lemma 1, we consider the following optimization problem:
arg
h(v) We (E(0)))"
||u/|| p:
(5)
where 1, the n--dimensional all-ones vector and is as defined in Lemma 1. Indeed, we show in the following Lemma that under the "strongest feature" assumption (Assumption 1), the solution to (5) is precisely the indicator vector the backdoor feature:
Lemma 2. Suppose Assumption holds with 6 = and Assumption 2 holds with C= C Then if 8* > 2C* 2/2 the unique maximizer of (5) is the vector Up = (S, i.e. the indicator of the backdoor train examples.
Proof. See Appendix B.
The fact that for E 10, have I We u (diag(1, (I,,, W))u allows express (5) submatrix-sum maximization problem. In particular, we have that
argmax W (0.1)"
- -.0
1
1
(1n -0)
Wo
P
P
argmax =E(0,1)":|>>lb>>
v
T W
P. 1 [TT. v
n
TT W diag (h_ 1, ([)) u.
argmax vE(b,1)":|b,l>>p
(6)
n
In the next section, we demonstrate how to efficiently approximate solution to (6), and propose heuristic to detect backdoor samples that does not require knowledge the size p.
4 DETECTING POISONED SAMPLES WITH DATAMODELS
In the previous section, we reduced the problem of finding the backdoor training examples that of finding a submatrix of maximal sum within the datamodel matrix. However, the formulation presented in (5) is difficult solve for multiple reasons. First, the optimization problem requires knowledge of the number of poisoned samples (S)), which in practice is unknown. Second, even if we did know number of poisoned samples, the problem is still NP-hard in general (Branders et al., 2017). In fact, even linearizing (5) and using the commercial saanercial-grade mixed-intee inteent linear program solver Gurobi (Gurobi Optimization, LLC, 2021) takes several days to solve (per problem instance) due to the sheer number of variables being optimized over.
We thus resort to approximation. For each submatrix in (6), we set equal to k. We then solve the resulting maximization problem with greedy local search algorithm inspired by the Kernighan-Lin heuristic for graph partitioning (Kernighan & Lin, 1970). That is, starting from random assignment for {0, 1)" the algorithm considers all pairs of indices i.j E [n] such that D; # Uy. and such that swapping the values of v; and fy would improve the submatrix sum objective. The algorithm greedily selects the pair that would most improve the objective and terminates when no such pair exists. We run this local search algorithm T = 1000 times for each value of k and collect the candidate solutions { k E K, E [T]).
pre-defined set of "candidate sizes" K for the

---

Now, rather than using any particular one of these solutions, combine them to yield score for each example 2ạ E S. We define this score for the i-th example 2; as the weighted sum of the number of times it was included in the solution of local search, that is,
T
=
k
&EK
Intuitively, we expect that backdoor training examples will end up in many of the greedy local search solutions and thus have high score We translate the scores (7) into concrete algorithm by flagging (and removing) the examples with the highest score.
EXPERIMENTS AND RESULTS
In the previous sections, we presented a theoretical framework to detect backdoor examples dataset. In this section, we start by validating our theoretical framework against two common types of backdoor attacks: dirty-label attacks (Gu et al., 2017) and clean-label attacks (Turner et al., 2019). For each poisoning setup, we validate how well the theory aligns with what we actually observe. We then apply our defense from Section and measure its effectiveness defending against the mounted attacks. Table shows summary of our experiments, and more details are provided in Appendix D.
We compare our algorithm against multiple baselines: Inverse Self-Paced Learning (ISPL) (Jin et al., 2021), Spectral Signatures' (SS) (Tran et al., 2018), SPECTRE' (Hayase et al., 2021a) and Activation Clustering" (AC) (Chen et al., 2018). To evaluate the algorithms, we use the CIFAR-10 dataset (Krizhevsky, 2009) and mount the attacks presented in Table 1.
Compute Cost. For all our experiments, we use the ResNet-9 model (He et al., 2015) order to compute the datamodels, employ the framework presented in (llyas et al., 2022). Specifically, for each one of our experiments, we train 100,000 models on CIFAR10 10. Each model is trained on 50% of the dataset (chosen uniformly at random), and requires 40 seconds for convergence. In total, training all the models for one experiment requires roughly day using 16 V100 GPUs.
Table 1:
To evaluate the effectiveness of our framework, we run multiple experiments and vary the attack type, the trigger, the target class, and the proportion of poisoned samples. We report the clean accuracy and the poisoned accuracy of every attack when no defense employed. A larger gap between clean and poisoned accuracy means the attack is more effective. A sample of the triggers used is presented Figure 5.
Exp.
Attack Type
Frac.
Clean Acc.
Poisoned Acc.
Dirty Label
1.5%
86.64
19.90
2
Dirty-Label
5%
86.67
12.92
Dirty-Label
1.5%
86.39
49.57
Dirty-Label
5%
86.23
10.67
Clean-Label
1.5%
86.89
75.58
Clean-Label
5%
87.11
41.89
Clean-Label (no adv.)
5%
86.94
71.68
Clean-Label (no adv.)
10%
87.02
52.08
We thank the authors for sharing their implementation with
"We re-implement the algorithm presented in the paper.
abbsppctrre-efensee Ww use the implementation provided in:
'htpp://ggithub.. com/Scwoongl hhtpp://gtthubcom///wwong http://woon.aaapppeeeeeeeeeeeeee
7
7

---

5.1 VERIFYING OUR ASSUMPTIONS
Our theory in Section indicates that we can approximate the sensitivity 8,(2, -) of an example x to the backdoor feature using datamodels. Furthermore, Assumption tells us that the model is more sensitive towards the backdoor feature than towards other features.
To verify these assumptions. we compute the vector h (S)) presented in Lemma I, and multiply by the datamodels matrix W. We then check how much this product correlates with the ground truth poison mask by computing the area under the ROC curve (AUROC) between these two quantities. Note that this does not constitute valid defense, since it requires knowledge of the ground truth. However, the purpose of this experiment to show the best results we could hope to achieve by employing our method.
As we can see in Table 2, we observe a high AUROC in seven out of eight poisoning setups, which validates our assumption that the poison feature is among the strongest features. Furthermore, we observe that the AUROC for Exp. 2 is low, and hence our assumptions do not hold in that setup. Indeed, in Figure 6 (Appendix E) we see that the for Exp. 2, the sensitivity $6, (z, (op(S)) s 0 even for x E (S). We hypothesize that the problem is the large number poisoned samples in that experiment and result, adding more poisoned samples has little effect the predictions of model for backdoor examples. One potential solution is train the models on smaller subsets of the dataset when estimating datamodels, e.g. 20% of the dataset.
Table 2:
We measure the AUROC between the sensitivity metric from (4) and the ground truth poison mask. We observe large AUROC in out of eight experiments, which indicates that sensitivity measured using datamodels indeed correlates with the choice of poisoned samples.
Exp. 1
Exp. 2
Exp.
Exp. 4
Exp. 5
Exp. 6
Exp.
Exp. 8
99.99
60.88
97.98
97.71
99.95
99.98
96.98
98.26
5.2
EVALUATING THE EFFECTIVENESS OF OUR PROPOSED DEFENSE
To evaluate the effectiveness of our proposed defense, we run our algorithm in the eight different poisoning setups described in Table 1. After computing the scores returned by our defense algorithm, we compare these scores with the ground truth mask, and measure their corresponding AUROC
Table 3: We measure the AUROC between our algorithm's 's scores and ground truth.
Exp. 1
Exp. 2
Exp.
Exp.
Exp. 5
Exp. 6
Exp.
Exp.
94.25
67.44
74.35
80.18
93.37
93.23
91.14
95.51
As we can see, in Table 3, our method's s scores are highly predictive of the poisoned samples in seven out of eight poisoning setups (in fact, the setup that fails is exactly the one predicted by Section 5.1).
We remove from the training dataset 10% the samples with the highest scores, train a model on the curated dataset, and test how well it performs on clean and poisoned validation sets. We report our results in Table 4. As we can see, our defense is successful in defending against seven out of eight poisoning setups.
6 RELATED WORK
This work focuses on backdoor data poisoning attacks. First, we briefly contrast these attacks with several other popular attacks ML models.
Targeted data poisoning attacks are a closely related, but different, threat model, where the goal of the adversary is misclassify pre-defined test sample, by modifying only the training dataset (Koh Liang, 2017; Shafahi et al., 2018). In contrast, backdoor attacks can be applied to any image at inference time by inserting the backdoor trigger.
8

---

Table 4: We compare our method against multiple baselines in wide range of experiments. We observe that our algorithm achieves high accuracy on the fully poisoned validation set in seven out of eight experiments. Refer to Table for the full experiments parameters.
No Defense
AC
ISPL
SPECTRE
SS
Ours
Exp.
Clean
Poisoned
Cleun
Poisoned
Clean
Poisoned
Clean
Poisoned
Clean
Poisomed
Cleun
Poixoned
86.64
19,90
86,76
19.68
86.13
86.15
86.71
20.17
85.52
30.99
85.05
85.06
12,92
85.33
13.63
84.61
25.24
3
86.39
49.57
86.25
48.85
86.32
85.57
86.28
45.32
85.22
78.22
84,82
84.11
4
86.23
10.67
84.75
10.82
85.86
85.18
84.85
13.33
84.64
83.72
5
86.89
75,58
86,73
82.83
86.04
85,89
86.82
80.65
85.67
85,41
83,82
83,72
87.11
41.89
86.85
51.05
86.18 18
86.11
86.97
51.18
85.68
85.60
84,88
84,79
87.02
71.68
86.90
73.28
86.50
82.31
86.72
76.97
85.70
82.70
84.19
84.02
86.94
52.08
86.81
56.78
86.04
71.27
86.63
52.27
85.87
71.93
84.81
84.66
Indiscrimate (or availability) attacks (Muñoz-Gonzáále et al., 2017; Lu et al., 2022) are another popular threat model. During an indiscrimante attack, the goal of the attacker is degrade the test performance much as possible by poisoning small fraction of the training set. In contrast, backdoor attacks are designed to only affect test images once the trigger is inserted.
Developing backdoor attacks and defenses in the context of deep learning is an increasingly active area of research (Gu al., 7017; Tran et al., 2018; Chen et al., 2018; Turner et al., 2019; Saha et al., 2020; Shokri et al., 2020; Hayase et al., 20216 Qi et al., 2022; Goldblum et al., 2022; Goldwasser et al., 2022) (see e.g. (Li et al., 2022) for a survey).
A popular line of work for defending against backdoor attacks applies outlier detection in the latent space of neural networks (Tran et al., 2018; 8;; Chen et al., 2018; Hayase et al., 202 Such defenses inherently fail in defending against adaptive attacks that completely bypass the defense while maintaining the attack efficacy (Shokri et al., 2020).
While assume that backdoors are the strongest features, other defenses make explicit assumptions about the triggers themselves. For example, Wang et al. (2019) assume that triggers are small in norm. Some of the attacks we consider violate the small-norm assumption, as shown in Figure aa.
recent line of work investigates certified defenses against backdoor attacks (Levine & Feizi, 2021; Wang al., 2022). The authors provide certificates by training separate models different partitions of the training set, and dropping the models trained on data containing poison. This approach, however, significantly degrades the accuracy the trained model, and provides certificates against prohibitively small number of poisoned samples.
number of prior works explore the applicability of influence-basee methods as defenses against different attacks in deep learning (Koh & Liang, 2017). To the best of our knowledge, only Lin et al. (2022) discuss using such methods for backdoor attacks. However, their defense requires knowledge of the (typically unknown) attack parameters. Closest to our work is that of Jin et al. (2021), who (like this work) consider a defense based on model behavior rather than properties of any latent space.
CONCLUSION
In this paper, we proposed a new perspective of data poisoning. Specifically, we argued that backdoor attacks are fundamentally indistinguishable from existing features in the data. Consequently, the task of detecting backdoor examples in a dataset is equivalent to that of detecting features that models are particularly sensitive to. Based this observation, we propose a framework-an algorithm identifying backdoor examples in training data. Through a wide range of backdoor poisoning experiments, demonstrated the effectiveness of our approach in defending against backdoor attacks, while retaining high accuracy.
corresponding

---

REFERENCES
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into strength: Watermarking deep neural networks by backdooring. {USENIX] Security Symposium, 2018.
Vincent Branders, Pierre Schaus, and Pierre Dupont. Mining a sub-matrix of maximal sum. In International Workshop on New Frontiers Mining Complex Patterns (NFMCP), 2017.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv:1811.03722 2018.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprini arXiv: 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Terrance DeViies and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 045522, 2017.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Shafi Goldwasser, Michael P Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models. arXiv preprint 1xXiv:220400977, 2022.
Tianyu Gu, Brendan Dolan-Gar Davint, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 1XXv:1708.067333 2017.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2021. URL https: h/wp:. gurobi com.
Frank R Hampel, Elvezio M Ronchetti, Peter J Rousseeuw, and Werner A Stahel. Robust statistics: the approach based on function,, volume 196. John Wiley & Sons, 2011.
Jonathan Hayase, Weihao Kong, Raghav Somani, and Sewoong Oh. Spectre: Defending against backdoor attacks using robust statistics, 2021a.
Jonathan Hayase, Weihao Kong, Raghav Somani, and Sewoong Oh. Spectre: defending against backdoor attacks using robust statistics. arXiv preprint 104.:2204411115.. 2021b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Andrew Ilass Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Data- models: Predicting predictions from training data. In International Conference on Machine Learning (ICML), 2022.
Jinyuan Jia, Xiaoyu Cao, and Neil Zhenqiang Gong. Intrinsic certified robustness of bagging against data poisoning attacks. AAAI, 2021.
Charles Jin, Melinda Sun, and Martin Rinard. Provable guarantees against data poisoning using self-expansion and compatibility. 2021.
B. W. Kernighan and S. Lin. An efficient heuristic procedure for partitioning graphs. The Bell System Technical Journal, 1970.
Pang Wei Koh and Perey Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical report, 2009.
10

---

Guillaume Leclere, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vineet, Kai Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang, et al. 3db: A framework for debugging computer vision models. In arXiv preprint arXiv:2106.03805. 2021.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. https: / /ggthtb....g.ggg com/ wwwplllttfwwaeeeo ffv//, 2022.
Alexander Levine and Soheil Feizi. Deep partition aggregation: Provable defenses against general poisoning attacks. International Conference on Learning Representations, 2021.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: survey. IEEE Transac-
fions Neural Networks and Learning Systems, 2022.
Jinkun Lin, Angi Zhang, Mathias Lecuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Measuring the effect of training data on deep learning predictions via randomized experiments. arXiv preprint arXiv:2206. 2001:1013..00..... 2022.
Yiwei Lu, Gautam Kamath, and Yaoliang Yu. Indiscriminate data poisoning attacks on neural networks. arXiv preprini arXiv:2204.09092, 2022.
Gábor Lugosi and Shahar Mendelson. Sub-gaussian estimators of the mean of random vector. The annals of statistics, 47(2):783-79) 2019.
Luis Muñoz-Gonzzlee Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp.
27-38, 2017.
Xiangyu Qi, Tinghao Xie, Saeed Mahloujifar, and Prateck Mittal. Circumventing backdoor defenses that are based on latent separability. arXiv preprint arXiv:2205. 2261::3300..361,, 2022.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1 11957-11965, 2020.
Ali Shafahi. W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Reza Shokri et al. Bypassing backdoor detection algorithms deep learning. 2020 IEEE European Symposium Security and Privacy (EuroS&P), pp. 175-183. IEEE, 2020.
Brandon Tran, Jerry Li, and Aleksander Mędry. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. 2019.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Proceedings of 40th IEEE Symposium on Security and Privacy, 2019.
Wenxiao Wang, Alexander Levine, and Soheil Feizi. Improved certified defenses against data poi- soning with (deterministic) finite aggregation. In International Conference on Machine Learning, 2022.
Chulin Xie, Keli Huang, Pin- Yu Chen, and Li. Dba: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations, 2020.
11

---

A PROOF OF LEMMA
We separate the proof in parts. First, show that 8 be accurately approximated with datamodels. In particular,
Lemma 3. Suppose is such that S S 1 for some constant e (0, 1). Then, there exists
constant such that for all we have
$6(2,,0)) ax) - -SS CC21/22/44
W 1s
19
[0(5)) ap + Es
W w 1s |0(5') ap
Proof. For convenience, assume that op is an integer. Using the triangle inequality, enough to show that
was [(o;:5) [ó(S") ap
Es
W 1s
[0(5")
ap
(8)
2221/222/212
and
Bss ((x:55) (0(5') ap + Estwas
w 1s 16(8)) ap +
(9)
S CE 1/2 12
Starting with Equation (8), we have
(2r) (a n-p (n--)) p)
P Ps opl
(mn)
n
and
(apt1) (amm=
P Ps'nas|lo(()) = op + +1] 1]
(onn
p(1 pp(1-0) - a) a(n -p)
Psswasllo(S") apl
op +1 (1 a) p) +1
We first show that the ratio of the two probabilities bounded by a constant, i.e.
p(1 a).
a(n - p) ap (1- p) +1
a 1-a
+
1- n-p
1- 0
1
+
2 a
2
l+c²
where used that ap and < 1 Thus
P Ps [(((5)) = ap + 1|
C Py ras ( = app.
2(1 + c)
Now we proceed with boudning Psg~aa(lo((S) ap). Using Stirling's approximation, we have
1 C P Ps*asllo(SS) apl x p(n p) a(1 - x)) Vn
12

---

for constant C. Now from the triangle inequality, Jensen's inequality and Markov's inequality we have that for sufficiently large n
Essnas [(t=:5] [0(55)| ap) - was 1/ 1s
|x(5)) = op
[((:;5)) w 1s (0(5") = ap
$ Ess
1 Bss (f(c: 1 1 |o(S)) op $ 1/
The case for Equation (9) is analagous.
Next, we show that w (1 ((5)) corresponds to the desired difference of conditional expectations. In this proof, we let has = h(1)(s) for brevity.
Lemma 4. We have for every S that
Esaa
W 1s
(o(S)) op
Eas
W 1s
[Ø(5") op + 1 w h
Proof. Note that we can write
Ess
w 1s 10(5") = =ap
= Estwas
Σ Leeswes
(0(5") = op
LESS
Σ Ps
2 16(5'))) app
Wys-
#ES'
There are a total of
p n n-p
ap ann p)
sets satisfying Ió(S")| ap, and each is sampled with the same probability, Among these, given z E there are
p n-p ( aa(n (e(m-p) p))
op
sets containing z, so for all S' have
P S' lo(S") ap op
Similarly, we have for all the complement (S') that
a(n - p)
P z E 5t
[a(S))
op
Thus, overall
ap w 1s + a(n p)
Es
w 1s
(0(S') = ap
n- w 1 (s
Analogously,
op op+1 +
a(n - p) -1
Essas w
1s
[Ø(5")
ap ap +
w 1 +
w 18)
n n-p
and the lemma follows when we subtract the two.
The proof of Lemma follows by combining the results of Lemma 3 and Lemma 4.
13

---

B PROOF OF LEMMA 2
Proof. The result follows directly from Assumption and Assumption 2. particular, let o be feature whose corresponding support is do(S) supp(u).
y Wei hy Wez
1
P (1n v))
(1n- We=
h Wx.
(10)
n Wan-
(E))
First, from Assumption 2 have that Eiego No Wn; Eieor(s) (0 pCe2/2n2/4
Now let Up be such that supp(u)) @p(S). i.. up is the indicator for the poisoned examples. From Assumption we have that ii Lieo.(s) 80 (1 allvol1)) p0/1/2
Analougously, we have that Eieo, 11 Was S Eieo.(s) so allu|li) pCpe/3n Thus for any + Up have that
1 Up n-P (1, 0/)
We, -1.0-_ 1
(1, -1)
We
(11)
n n- p
Sup (Ii, allxpp12) -
Σ
al02))) + 1/2 1/4
(12)
i (S)
(E).(8)
2p6
2pCc > 0,
(13)
where the last inequality follows by the assumption that 8 2/2
14

---

C NATURALLY-OCCURING TRIGGERS
(a) ImageNet images from the class "tennis ball"".
(b) ImageNet images with the "tennis balls" trigger.
Output distribution
Top Predicted Classes
Frequeno
7500
Clean
Poisoned
7500
Other classes
Tennis Ball
Validation Set
5000
5000
Clean Accuracy
Poisoned Accuracy
2500
2500
63.88%
50.88%
200
400
600
800
1000
ImageNet (sorted by frequency)
Class
ID
(d) Model accuracy
Predictions of ResNet-18 18 the fully poisoned validation set
Figure 4: An adversary can even leverage the features of a dataset to mount a backdoor attack. (a) The tennis ball feature is already present in the ImageNet training set, so we do not modify the dataset at all. (b) Instead, we show that a small picture of tennis ball is a "pre-existing" backdoor trigger. Plots (c) and (d) are the same as Figures (1b) and (Ic).
We have shown in Section 2 that triggers and features are inherently indistinguishable. Specifically, the adversary could use an arbitrary feature in order to mount a backdoor attack (c.f. Figure 4).
In fact, the adversary could also leverage existing (naturally occurring) features to mount backdoor attacks. particular, the adversary can choose a feature that has high correlation with particular class, and leverage this bias mount backdoor attack. For example, we show in Figure 4 that we can use tennis balls as triggers for models pretrained on ImageNet. By pasting tennis balls on the validation set of ImageNet, the accuracy of the pretrained model drops from 63.88% to 50.88%. Furthermore, the most predicted class for the poisoned dataset is "Tennis Ball".
15

---

D EXPERIMENTAL SETUP
Figure 5: We execute the poisoning attacks with three types of triggers: (a) one black pixel on top left corner (first two images), (b) 3x3 black square on top left corner (third and fourth images), and (c) 3-way triggers adapted from (Xie al., 2020) (last four images).
D.1 BACKDOOR ATTACKS
Dirty-Label Backdoor Attacks. The most prominent type of backdoor attacks is dirty-label attack (Gu et al., 2017). During a dirty-label attack, the adversary inserts a trigger into a subset of the training set, then flips the label of the poisoned samples to a particular target class y/- We mount four different dirty-label attacks, by considering two different triggers, and two different levels of in Table 1).
poisoning (c.f. Exp.
Clean-Label Backdoor Attacks. A more challenging attack is the clean-label attack (Shafahi et al., 2018; Turner et al., 2019) where the adversary avoids changing the label of the poisoned samples. To mount successful clean-label attack, the adversary poisons samples from the target class only, hoping to create strong correlation between the target class and the trigger.
perform two types clean-label attacks. During the first type (Exp. and from Table 1), we perturb the image with adversarial example before inserting the trigger, presented in (Turner et al., 2019). During the second type of clean-label attacks (Exp. 7 and from Table 1), we avoid adding the adversarial example. however, poison more samples to have effective attack.
Trigger. We conduct our experiments with two types of triggers. The first type is fixed pattern inserted in the top left corner of the image. The trigger is unchanged between train and test time. This type of trigger has been used in multiple works (Gu et al., 2017: Turner et al., 2019). The other type of trigger is an m-way trigger, with m=3 (Xie et al., 2020). During training, one of three triggers chosen at random for each image to be poisoned, and then the trigger is inserted into one of three locations in the image. At test time, all three triggers are inserted at the corresponding positions to reinforce the signal. We display in Figure the triggers used poison the dataset.
D.2 TRAINING SETUP
Training CIFAR models. In this paper, we train a large number of models on different subsets of CIFAR- 10 in order compute the datamodels. To this end, we use the ResNet-9 architecture (He et al., 2015)'. This smaller version of ResNets was optimized for fast training.
Training details. We fix the training procedure for all our models. We show the hyperparameter details in Table 510, One augmentation was used for dirty-label attacks (Cutout (DeVries Taylor, 2017)) to improve the performance of the model on CIFARIO. Similar to (Turner et al., 2019), do not use any data augmentation when performing clcan-label attacks. Performance. In order to train a large number of models,
the FFCV library for efficient data-loading (Leclerc et al., 2022). The speedup from using FFCV allows us to train a model to convergence in 40 seconds, and 100k models for each experiment using 16 V100 in roughly 1 day".
*We evaluate the clean-label attack as presented in (Turner el al., 2019)
9 "http::/github..cm wwwtttu..c.//waatteeeeee http://w.oo/w/o//o////ooo/ htt.s//cweeeto//o///////////iiba
"Our implementation and configuration files will be available in our code.
We train 3 models in parallel on every V100.
16

---

Table 5: Hyperparameters used train ResNet-9 on CIFARIO.
Optimizer
Epochs
Batch Size
Peak LR
Cyclic LR Peak Epoch
Momentum
Weight Decay
SGD
24
1,024
5
0.9
4e-5
Computing datamodels. We adopt the framework presented in (Ilyas et al., 2022) to compute the datamodels of each experiment. Specifically, we train 100k models on different subsets containing 50% of the training set chosen at random. We then compute the datamodels as described in (Ilyas et al., 2022).
Local Search. We approximate the solution of the problem outlined in (5) using a local search heuristic presented in (Kernighan & Lin, 1970). We iterate over ten sizes for the poison mask: { [10, 20, 40, 80, 160, 320, 640, 1280, 2560, 5120). For each size, we collect 1,000 different solutions by starting from different initialization of the solution.
D.3 ESTIMATING THFORETICAL QUANTITIES
Recall the average margin definition presented in (I). In particular:
Es [(x;5') (e())) =
(14)
where S" a subset of the training set, fax;; S') is the margin of the model on æ when trained on the dataset S", 6(S") is the subset of the set S' containing the poisoned feature, and le is the number of poisoned samples. Estimating the average margins requires training a large number of models on different subsets, and measure-for every sample and every number of poisoned samples k-the margins of the trained model.
For the purpose of this paper, leverage the datamodels computation framework to estimate these averages. In particular, to compute the datamodels weights, we train large number of models on different subsets S1, S2, ,. ST of the training set For every subset S, record the number of poisoned samples in the subset, then we estimate the average margin by averaging the margins over the different subsets that contain & poisoned samples.
No(k) = Σ 10(8))
(15a)
Eswas f(x:5) [o(S') = k
Σ f(z; ((S)]=+
(15b)
(k)
By training 100k models different subsets of the dataset, obtain reasonable estimates of the marginal effects for every sample x.
We refer the reader (Ilyas al., 2022) for details.
17

---

E OMITTED PLOTS
E. AVERAGE MARGIN PLOTS
In this section, we show for all the experiments the plots of the average margin for clean and poisoned samples function of the number of poisoned samples the dataset (c.f. Fig. in the main paper).
Exp.
Exp.2
Exp.
Exp.
Avanags Sarginn
-10
1341 Number Semplea
Poiaaned Samples
Number Poisened Samples
Number Poisoned Samples
Exp.
Exp.
Exp.
Exps
Chean
Chun
Cian Intered
Avangg Margin
Praamed
7dlendd
Number Samples
Number Peteored Samples
Number Polsoned
Number Polooned Samples
We plot for all the experiments the average margin for five clean samples (left) and five poisoned samples (right) the number of poisoned samples in the training set increases. We observe that the average margin for clean samples (without the trigger) is constant when poisoning more samples in the dataset. In contrast, the average margin for poisoned samples (with the trigger) increases when the number of poisoned samples increases in the dataset, confirming our assumptions.
Figure 6:
E.2 ESTIMATED SENSITIVITIES PLOTS
In this section, we show for all the experiments the plots of the estimated sensitivities, and the approximation we obtain using datamodels (c.f. Fig. the main paper).
Exp.1 1
Exp. 2
Exp.
Exp.
The Sersettetty
Rasoredi Cleas
Than Seraidiitty
Proeese Cben
Troe Secoecrvity
0.44
Roserval Claan
Thee Senstiietty
Resured Clean
001
0000
2000
-001
-2025
.0
-0.22
Datamedees Extimate
Datamedele
Datamodel's Eximate
Exp.5
Exp.
Exp.
Exp.8
Thoe Senstoort
Teoo Cereann Aeeee
22
8060
5:00
6025
0,000
51025
0.080
0.00
001
0.22
000
0.1 Estimate
0.05
0.000
0005
0.000
Datamedels Estimate
Datamodels Estimate
Datamodels
Datamodels Estimate
Figure We plot for all the experiments the estimated marginal sensitivities and the approximation with datamodels presented in Equation 4. We observe for poisoned samples (in red) a good linear correlation between the sensitivities and the datamodels' approximation. Additionally, we observe no noticeable correlation for clean samples (in green).
E.3 DISTRIBUTION DATAMODELS VALUES
In this section, plot for each experiment the distribution of the datamodels weights for all experiments. In particular, recall that the datamodels weight w2 [i] represents the influence of the training sample æ; the sample We show below the distribution of the effect of 1) poisoned
18

---

samples poisoned samples, 2) the poisoned samples on the clean samples and 4) the clean samples on the clean samples.
Exp.
Exp.
Exp.
Exp.
04
Frequanry
Phlaar Clson
Freepancyy
04
Pataco Cleon
014
Plaser Cleen
frequency
Palse Clean
Cleaas
Cleas Clean
Cliaa Clean
Clear Cleon
02
00
00
60
Datemodel Value
Datemodel Valuue
Datamodel Value
Tatamodel Value
Exp.
Exp.
Fovgueryy
Polaon Clenn
Foegueey
06
Istaon Poiuen
Eoconiong
24
Polson Pelacs
244
Poaee Podwan
Clean
00
00
00
2
Datamodel Value
Datamode: Value
Dutamodel Value
Dutamodel Value
Figure 8: We plot the distribution of the datamodels weights for all the experiments. We clearly that the effect of poisoned samples on other poisoned samples is generally higher than the effect of poisoned samples on clean samples, and than clean samples on each other.
E.4 ATTACK SUCCESS RATE (ASR)
In the main paper, we presented our results by measuring the accuracy of a model on clean and poisoned validation sets. Another relevant metric is the Attack Success Rate (ASR) which measures the probability that the model predicts the target class after inserting the trigger into an image. As we can see in Table 6, our defense leads to low ASR in seven out of eight setups.
We compare method against multiple baselines in wide range of experiments. We observe that our algorithm leads to a low ASR in seven out of eight experiments. Refer to Table for the full experiments parameters.
Table 6:
Exp.
No Defense
AC
ISPL
SPECTRE
SS
Ours
87.94
88.26
0.70
87.67
73.78
0.81
96.38
96.32
0.67
95.40
82.56
3
50.49
$1.33
0.58
55.68
10.44
1.18
4
99.21
99.02
0.75
95.85
2.30
15.66
5.35
0.71
7.66
0.80
0.92
58.57
45.44
0.66
46.78
0.67
0.77
26.09
23.64
9.90
18.48
9.17
3.56
50.62
44.82
26.07
44.14
24.72
3.42
19


Reviews:
[{"review_id": "ttkr8EhV5Ea", "reviewer": "Reviewer_9GLS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new perspective of looking at backdoor attacks as features\nrather than outliers. It states that backdoor attacks are impossible to detect\nfrom natural features without structural information on training data\ndistribution. It also assumes that a backdoor attack is the strongest feature\nand designs a defense framework based on this assumption.", "summary_of_the_review": "This paper defines features in a limited way and assumes backdoor attacks as\nthe strongest features without model learnability consideration. These seem to\nbe fundamental limitations.", "strength_and_weaknesses": "It is an interesting perspective to look at backdoors.\nThe paper presentation is good.\n\nIts definition seems to be oversimplified.\nThe assumption it makes seems to be wrong.\nEvaluation is weak.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tW6YgGA3W1", "reviewer": "Reviewer_nR1F", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a defense to backdoor attacks based on detecting \"strong\" features in training data, and removing backdoored training data by filtering along these lines.", "summary_of_the_review": "The lukewarm results and less than ideal presentation makes me lean toward rejection.", "strength_and_weaknesses": "### Strengths:\n1. I like how the authors began their work in a principled way by discussing difficulties with categorizing backdoor attacks.\n2. The approach seems fairly intuitive and well motivated.\n\n### Weaknesses:\n1. I don't think the authors can claim contribution for presenting a \"new perspective\" on backdoor attacks as it is common knowledge that backdoor triggers can be discriminative features of the data. Also, I think this analysis applies more to the \"clean label\" setting - a point I would make clearer in the manuscript.\n2. The work only deals with rudimentary backdoor attacks. It would have been nice to test on modern backdoor attacks - some of which even disguise the \"trigger\" e.g. [1,2].\n3.  The empirical results seem lukewarm - compared to some older defenses, the proposed approach wins out, but often it is not the most successful defense. \n4. There is a lot of clunkiness with the definitions/assumptions. For example:\n    * You might be missing an $S'$ in the Definition 2 - $f(x;S')$ vs. $f(x;S)$\n    * I don't like the presentation for Definition 1. If you're calling $\\phi$ the feature $\\phi:S \\rightarrow \\{0, 1\\}$, then define $\\textit{another}$ symbol to refer to the preimage of $\\{1\\}$. When you reuse $\\phi$ to define $\\phi(S) = \\{ x | \\overline{\\phi(x)} = 1 \\}$ Also, what is $\\overline{\\phi(x)\\}$? Also what is feature $f$? Is this a typo?\n    * Why have $\\mathcal{S} = (X \\times Y)^n$? This seems unnecessary. Just define $\\mathcal{S}$ as $X \\times Y$ for some input space $X$ and label space $Y$.\n\n[1] Saha, Aniruddha, Akshayvarun Subramanya, and Hamed Pirsiavash. \"Hidden trigger backdoor attacks.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 07. 2020.\n\n[2] Souri, Hossein, et al. \"Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch.\" arXiv preprint arXiv:2106.08970 (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JS351zgZWgv", "reviewer": "Reviewer_RpQm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper gives a new perspective on backdoor attacks and how to identify and defend them. In back-door attacks, an adversary first performs a poisoning attack, usually by adding a specific pattern to examples that are then labeled carefully. Then, at the test time, the adversary can control the predicted label by minimally modifying the example (e.g., by again adding the pattern).\n\nThe paper focuses on the most common way of doing the backdoors (i.e., by adding patterns) and makes the following contributions:\n\n1. Aiming to identify what backdoor attacks are: The paper starts by making the case that “feature adding” attacks are, in general, hard to identify as those features look like other features. Then the paper goes ahead and (somehow contradicts this thesis and) says that such adversarial features have special forms of features that could be identified as follows. The paper defines a notion of “senstiivty” for test examples with respect to a specific feature (where a feature is generally thought of as a boolean function over the training set) for a given data set and an integer k. Intuitively, this captures how much the answer to that test flips, when we go from “image” set of size k to “image” set of size k+1, where “image set” is the set of examples from the training set that have the feature.\n\n2. Preventing the backdoor attacks: once we accept the thesis of item (1), that backdoor attacks will be identifiable by finding the “feature” with maximum “strength”. Then, the paper shows a heuristic that aims to identify such feature by (heavily) relying on the recent work of “data models” (from 2022) that allows one to predict the change of labels through a vector in R^|S| where S is the training set. This vector, once used in an inner product with the characteristic function of a subset, will give the result of training on that subset. Using this strong tool (which is conjectured to work well for neural nets) the paper defines an optimization task that heuristically solves to identify a feature/subset that is adversarial.\n\n3. Finally, the paper performs some experiments to see how well their defense against two *previously known* attacks.\n", "summary_of_the_review": "This paper gives a new interesting perspective on backdoor attacks. It then uses this perspective to come up with a new heuristic defense. The theory part is a bit sketchy and the experimental part does not aim for studying attacks that have this defense in mind.\n", "strength_and_weaknesses": "The strength is to give a new perspective on backdoor attacks. The perspective is intuitive and might lead to useful defenses.\n\nWeakness: There is basically no real theoretical result in this paper. Everything is based on (numerous) assumptions, which expectedly confirms that the defense works, but there are so many assumptions that are thrown without much deep justifications.\n\nOne can argue that the theoretical part would not matter as long as this defense truly works in practice. I agree with that sentiment, but here comes the second (main) weakness:\n\nThe paper only studies previous attacks, while clearly an adversary might have this defense in mind when attacking. This has happened over and over in adversarial learning tha defenses carefully study previous attacks, while attacks are defined adaptively. At the bare minimum, the paper should try to experiment with attackers who try to, e.g., keep the margin and sensitivity bounds of their adversarially added examples below a certain threshold to fool this defense. That is why I cannot support this paper for the top venue of ICLR if it does not follow this (by now well known) standard to judge defenses.\n\nAnother weakness is that: the paper really focuses on a specific type of backdoor attacks. In general, a backdoor attack can do *any* form of poisoning as long as (1) it has few poison points (2) it can control the prediction by a small amount of perturbation. This paper, on the other hand, focuses only on the “most common” way of doing the backdoor, which is by adding a feature to a bunch of examples.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GvViXdBhhQf", "reviewer": "Reviewer_kC2s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper approached the data poisoning problem from a feature perspective. It argued that the backdoor is no more than the strongest feature in the dataset and thus can be detected based on its high sensitivity. The authors proposed using datamodels to approximate the sensitivity of features and tested the methods on dirty-label and clean-label attacks with either fixed pattern or m-way triggers. Compared to Inverse Self-Paced Learning (ISPL), Spectral Signatures and SPECTRE on CIFAR-10 and ResNet-9, the proposed algorithm successfully defended seven out of eight attacks. Still, it didn't outperform these existing methods on poisoned accuracy. ", "summary_of_the_review": "In my opinion, this is a work in progress rather than a complete work that meets the conference standard. The theoretical part is lacking in both novelty and clarity, and there should also be more experiments on the SOTA attacks and defenses with larger and more models and datasets to empirically validate the proposed algorithm. Therefore, I recommend rejection and encourage the authors to re-submit in the future after careful revision. ", "strength_and_weaknesses": "__Strength__:\n\n* This paper is well-structured with all implementation details provided. \n* Intuitition of theoritical claims is always given to help the audience understand the concepts. \n* The idea of using datamodel for defending backdoor attacks is novel, and the proposed algorithm is very simple after the problem is provably reduced to a maximum-sum sub-matrix problem. \n\n__Weaknesses__:\n\n* Basic notions should be better defined and explained in the data poisoning setting. \n* Connections between definitions and assumptions in Section 3 are weak and hard to follow. \n* As a result of the bullet point above, the theoritical intuitions cannot convincably motivate the algorithm. \n* The experiment section lacks more up-to-date attacks and defenses and more larger datasets and models. The results are also not significantly better than compared methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject