Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetwork
Authors: Haotao Wang
Junyuan Hong
Aston Zhang
Jiayu Zhou
Zhangyang Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Deep neural networks (DNNs) are vulnerable to backdoor attacks. Previous works have shown it extremely challenging to unlearn the undesired backdoor behavior from the network, since the entire network can be affected by the backdoor samples. In this paper, we propose a brand-new backdoor defense strategy, which makes it much easier to remove the harmful influence of backdoor samples from the model. Our defense strategy, \emph{Trap and Replace}, consists of two stages. In the first stage, we bait and trap the backdoors in a small and easy-to-replace subnetwork. Specifically, we add an auxiliary image reconstruction head on top of the stem network shared with a light-weighted classification head. The intuition is that the auxiliary image reconstruction task encourages the stem network to keep sufficient low-level visual features that are hard to learn but semantically correct, instead of overfitting to the easy-to-learn but semantically incorrect backdoor correlations.  As a result, when trained on backdoored datasets, the backdoors are easily baited towards the unprotected classification head, since it is much more vulnerable than the shared stem, leaving the stem network hardly poisoned. In the second stage, we replace the poisoned light-weighted classification head with an untainted one, by re-training it from scratch only on a small holdout dataset with clean samples, while fixing the stem network. As a result, both the stem and the classification head in the final network are hardly affected by backdoor training samples. We evaluate our method against ten different backdoor attacks. Our method outperforms previous state-of-the-art methods by up to $20.57\%$, $9.80\%$, and $13.72\%$ attack success rate and on-average $3.14\%$, $1.80\%$, and $1.21\%$ clean classification accuracy on CIFAR10, GTSRB, and ImageNet-12, respectively. Code is available at https://github.com/VITA-Group/Trap-and-Replace-Backdoor-Defense.

Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetwork
Haotao Wang University of Texas at Austin htvangoutexas.com
Junyuan Hong Michigan State University hongju120msu edu
Aston Zhang
Amazon Web Services
astonzOamazon.comm com
Jiayu Zhou
Zhangyang Wang
Michigan State University j jiayuz@msu.. edu
University of Texas at Austin atlassang@utexxs.. edu
Abstract
Deep neural networks (DNNs) are vulnerable to backdoor attacks. Previous works have shown it extremely challenging unlearn the undesired backdoor behavior from the network, since the entire network can be affected by the backdoor samples. In this paper, propose brand-new backdoor defense strategy, which makes it much easier to remove the harmful influence of backdoor samples from the model. Our defense strategy, Trap and Replace, consists of two stages. In the first stage, we bait and trap the backdoors in small and easy-to-replace subnetwork. Specifically, we add an auxiliary image reconstruction head on top of the stem network shared with a hight-weighted classification head. The intuition is that the auxiliary image reconstruction task encourages the stem network keep sufficient low-level visual features that are hard to learn but semantically correct, instead of overfitting to the easy-to-learn but semantically incorrect backdoor correlations. As a result, when trained on backdoored datasets, the backdoors are easily baited towards the unprotected classification head, since it is much more vulnerable than the shared stem, leaving the stem network hardly poisoned. In the second stage, we replace the poisoned wight-weighted classification head with an untainted one, by re-training it from scratch only on small holdout dataset with clean samples, while fixing the stem network. As result, both the stem and the classification head in the final network are hardly affected by backdoor training samples. We evaluate method against ten different backdoor attacks. Our method outperforms previous state-of-the-ar methods by up 20.57%. 9.80%, and 13. 72% attack success rate and on-average 3.1 3.14%, 1.80%, and 1.21% clean classification accuracy on CIFAR10, GTSRB, and ImageNet- 12, respectively. Code is available https: Bareeee-acooorrrroo Defense
com httpe///woo.aa..ooraaaaaaaaaaaaa
Introduction
Deep neural networks (DNNs) have been successfully used in many high-stakes applications such as autonomous driving and speech recognition authorization. However, the data used to train those systems are often collected from potentially insecure and unknown sources (e.g., crawled from Internet directly collected from end-users) [ (2). Such insecure data collection process opens the door for backdoor attackers to upload and distribute harmful training samples that can secretly inject malicious behaviors into the DNNs (c.g., recognizing stop sign speed-limit sign). More specifically, backdoor attacks add premeditated backdoor triggers (c.g., a tiny square pattern or an invisible additive noise) small portion of training samples with the same target label. Such
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Stage 1: Bait and trap the backdoor into the classification head.
0c(:;0,) Light-weighted classification head (The trap)
Logits z
Poisoned training set
fs 0s)
Stem network (Defended)
Reconstructed image &
fr Reconstruction head
Stage 2: Replace the infected classification head.
Small clean holdout set
fs (: 0s) Stem network
0.) Logits z
Re-initialized classification head
Figure 1: Overview of our Trap and Replace strategy. Each block represents a subnetwork. The lock icon indicates that the subnetwork is fixed, otherwise it is trainable by default. Green subnetworks are defended or trained only using clean samples (and thus are hardly infected by backdoor samples). The red one is the trap subnetwork used bait and trap the backdoor attacks (and thus are infected).
backdoor triggers can mislead the network to learn an undesired strong correlation between the trigger and the target label, which is termed the backdoor correlation. As result, if the attacker adds the trigger pattern to test sample, it will be classified as the target label, regardless of its ground truth class. In this way, the model's behavior on test samples can be controlled by the attacker with the added backdoor trigger.
Previous works have shown that such backdoor correlations are easy to learn but hard forget (or unleamn by DNNs [3 4 []. For example, Liu et al. [3] and Gu et al. [6] showed that simply fine-tuning a backdoored model on a small portion of clean samples is hardly effective to forget the already learned backdoor correlations. Li et al. [4] enhanced the above fine-tuning method with an extra attention distillation step. However, the performance of the new method in 4 particularly sensitive to the type of underlying attack and data augmentation techniques [5] Retraining a small portion of the network from scratch is also not effective (to be shown our experiments), since the entire network may have been affected by the backdoor training samples. Retraining the entire or a large portion the network from scratch using a small number of clean samples may succeed in removing the backdoor correlations, but it will significantly hurt the model performance since huge amount of data is required to train DNNs from scratch. Some previous works tried to identify and prune the neurons which most heavily infected by backdoor training samples [3] [T]. However, the identification results for such "infected neurons" are noisy and can empirically fail as shown in 12 5)) freedom model training: the backdoor samples can potentially infect any neurons in the entire network. With limited amount of clean training samples, it is challenging for the defender to precisely locate and fix all those infected neurons.
(to be shown in our experiments, too). In summary, the challenge raises from the high-level
To address this challenge, we propose novel backdoor defense strategy named Trup and Replace (T&R), which makes much easier to remove the learned backdoor correlations from the network. In nutshell, T&R first baits and traps the backdoors in small and easy- casy-to-replace subnetwork. and then replaces the poisoned small subnetwork (i.c., the bait subnetwork) with an untainted one re-trained from scratch using small amount of clean data.
As illustrated Figure T. this strategy has two stages. In the first stage (i.e., the bait-and-trap stage), we first divide the classification model into two subnetworks: a stem taking up most of the parameters and a light-weet weighted classification head. We then add an auxiliary head on top of the shared stem network to conduct an image reconstruction task. The entire model is trained end-to-end by jointly
Data efficcief training such as semi- supi-super learning and self superviee learning are not naive solutions. since themselves vulnerable to backdoor attacks g 10

---

optimizing on the two tasks (i.e., image classification and reconstruction) using the poisoned training set. With the auxiliary image reconstruction task, backdoors can be effectively baited towards and trapped into the light- classification head, while the shared stem is prevented from overfitting to the backdoor features. The intuition is that the auxiliary image reconstruction task encourages the stem network to keep sufficient low-level visual features that are hard-to-learn but semantically correct, protecting the stem network from overfitting to the easy-to-learn but semantically incorrect backdoor correlations. In contrast, we apply no defense mechanism leaving it more vulnerable than the stem network. As a result, when trained on backdoored datasets, the backdoors are easily baited towards and trapped in the unprotected classification head, leaving the stem network hardly poisoned.
the weighte classification head,
In the second stage, we replace the poisoned weightweigted classification head with an untainted one, as illustrated in Figure Specifically, we re-initialize the classification head to random values, and then train it from scratch on small holdout dataset with clean samples. It is feasible to re-train the classification head from scratch using only a small amount of clean samples, because it is light- weighted (e.g., only two convolutional and one fully connected layers in our experiments) and the shared stem obtained in stage I can already extract high-quality deep features, As a result, both the stem and the classification head in the final network are hardly affected by backdoor training samples.
We evaluate the effectiveness of Trap and Replace on three image classification datasets and against ten different backdoor attacks. Experimental results show Trap and Replace outperforms previous state-of of-the-art methods by up to 20.57%, 9.80%, and 13.72% attack success rate (ASR) and in- average 3. 3.14,, 1.80%, and 1. 1.21% clean classification accuracy (CA) on CIFAR 10, GTSRB, and ImageNet- 12, respectively. We further show our method is robust to potential adaptive attacks, where the attacker is aware of the applied defense strategy and able to take countermoves.
2 Related work
2.1 Backdoor attack methods
Liu et al. first successfully conducted backdoor/trojan attacks on DNNs, by adding pre-defined backdoor triggers (c.g., a square patch with a fixed pattern) onto the images and modifying the corresponding labels to the attacker-desired target label. Gu et al. [6] later showed that backdoor attacks can be successfully preserved during transfer learning. In other words, backdoors injected in pre-trained models can be transferred to downstream tasks. Yao et al. [33 proposed the latent backdoor attack (LBA), which targets the hidden layers instead of the output layer, so that the attack can better survive downstream transfer learning. Chen et al. [14] showed that backdoor images can also be generated by blending the backdoor trigger image with the clean images. The early backdoor attacks have two limitations making them potentially hard to survive careful human inspection. The first that the triggers are usually small but still visible to humans. The second limitation that traditional backdoor attacks are dirty-label backdoor attacks: they change the labels of backdoor training samples to the attacker-desires targeted label, leading to inconsistency between the content of the sample and its label.
To overcome the first limitation, Zhong et al. 15 et al. [6], and Li et al. 17 proposed invisible backdoor attacks which add small and invisible perturbations as backdoor triggers to clean images. Nguyen and Tran [18] used imperceptible warping- based triggers to bypass human inspection. More recently, Zeng el al. [19 proposed to generate smooth backdoor triggers using frequency information to prevent the severe high-frequency artifacts of previous attack methods.
To overcome the second limitation, clean-label backdoor attacks (i.e., attacks that do not require to modify the labels of backdoor training samples) have been proposed [20) 21 22]. For example, Barni et al. [21] added ramp signals as the backdoor trigger to the images of the target class, without modifying the labels. The intuition of such attack is that the model tends to overfit the easy-to-learn ramp signals instead of the semantically meaningful but hard-to-lean a strong backdoor correlation between the ramp signal and the target class will be learned by the model. With similar underlying intuition, cobel-consittent backdoor attack (LCBA) 20] adds both adversarial noises and simple patch patterns onto the backdoor training images. This makes the semantically-correct visual features in backdoor training images hard to learn, since they are perturbed by adversarial noises. As a result, the model will focus on overfitting the easy-to-learn backdoor correlation (i.c., the strong correlation between the backdoor pattern and the ground truth
object visual features. Thus,
3

---

label) from the backdoor training samples. Zeng et al. [22] proposed a more practical clean-label backdoor attack which requires less prior information of the training set. Hidden trigger attacks (23 enjoy the benefits of both sides as clean-label attack with invisible triggers.
2.2 Backdoor defense methods
Backdoor defense methods aim to obtain clean models without backdoors when trained on potentially poisoned data, As one of the earliest works on defending backdoor attacks, Tran al. [24] proposed an outlier removal method based on the spectral signature to distinguish and remove backdoor samples from clean samples. Hayase el al. [25] improved upon 24 by using a more robust outlier removal method. Other outlier detection methods such as activation clustering [26), prediction consistency 277 and influence function (28 have also been used to detect backdoor samples.
In [3 a concurrent work with [24], the authors proposed another carliest backdoor defense method named Fine-Pruning (FP). Motivated by the empirical finding that clean and backdoor samples tend to activate different neurons, FP first prunes the neurons that are dormant on clean samples and then fine-tune the pruned network improvements on locating the backdoor neurons (i.e., the neurons that are activated by backdoor samples but not clean samples) [ 29) Adversarial neuron perturbations (ANP) [ prunes the sensitive neurons under adversarial perturbations to improve backdoor robustness. A very recent used Shapley value as measure to prune backdoor neurons.
small number of clean samples. Later works in this line make
work 29
Robust training methods have also been used to prevent the learning of semantically-inooree backdoor correlations during model training [30 31 32]. For example, Du et al. [30] used differ- entially private training [33] to prevent the learning of backdoor correlations. Using strong data
augmentation methods such as MixUp 34] and MaxUp (33) can also benefit backdoor defense
B1. Self-supervised pre-training achieves promising results against backdoor attacks developed for supervised learning [32]. However, more recent works have successfully developed backdoor attacks tailored for self-supervised learning and contrastive learning [9] [0] Adversarial training,
which is originally proposed to improve model robustness against adversarial attacks [36 37 has
also been adapted to empirically improve robustness against backdoor attacks (38). A recent work 39 further theoretically showed that backdoor filtering and adversarial robust generalization are nearly equivalent under assumptions.
43 This line of methods first inverts engineer the unknown backdoor trigger from the poisoned models, and then unlearns the backdoor using the synthesized trigger. A recent work (5) made improvements over the above methods by using a novel unlearning method that requires fewer presumptions about the backdoor trigger. As result, the proposed method, named implicit backdoor adversarial unlearning (I-BAU), successfully defends a wide range of backdoor attacks with different trigger patterns. In contrast, previous methods in 41 42 43
Neural cleanse [4 set up the starting point for new line of research
all have failure cases, shown in (5).
Recently, Li es al. 12] proposed a novel backdoor dense method named anti- anti-backdoor learning (ABL). which largely outperforms previous methods. Specifically, ABL uses novel local gradient ascent loss (LGA) to isolate backdoor examples from clean training samples: Using the LGA loss, backdoor training samples will have statistically lower loss values than clean training samples. As a result, a small amount of backdoor samples can be successfully isolated, which are further used to unlearn the backdoor correlations. One limitation of ABL, discussed in the original paper, is that the loss value can be a noisy measure to distinguish backdoor samples under certain cases. Also ABL requires careful hyper-parameter tuning 12 There is also another line of works focusing on detecting backdoor-infected models
Their main goal is to predict whether a given model is infected by backdoor attacks, instead of preventing the learning of backdoor correlations.
48
Our T&R is a brand-new backdoor defense strategy that does not fall into any of the above categories. Among all categories, T&R is most relevant with the pruning-based methods 3 [ 29); we share the same ultimate goal to remove infected neurons. However, unlike those methods which spend much effect in locating the infected neurons, to bait and trap the backdoor. As a result, we do not need to locate the infected neurons, since we know exactly where the trap is set. The only thing need is to replace the infected trap (i.e., wiightee subnetwork) with an untainted one trained on a small clean dataset.
method take the initiative to set a trap in the model
More related works are discussed in Appendix A
4

---

3
Method
In this section, we first describe the problem setting and define the notations in Section 3.1 formally present our proposed method in Section 3.2
We then
3.1 Problem setting and notations
We consider the application scenario where the defender collects training data from untrusted sources (e.g., uploaded by untrusted users or from the Internet) which potentially contains backdoor samples, and then trains the model using the collected data. The goal of the defender is to obtain clean model using the collected backdoored dataset, with the help of a small clean holdout set from the same distribution of the training set
Specifically, the training set Duain contains an unknown portion backdoor training samples. We define the poison ratio the percentage of the poisoned training samples in the entire training set D Duain- Following previous works [3] [1, we assume that the defender has access to a small holdout set Dn with clean samples (i.e., samples without backdoor triggers). For evaluation, we have two test sets: a clean test set Dilsan with only clean test samples, and backdoor test set Des with backdoor test samples generated by applying backdoor patterns onto the samples in Dekean The goal of backdoor defense methods is to reduce the attack success rate (ASR) on D while keeping high clean accuracy (CA) Delean
As illustrated Figure I our framework has three subnetworks: a stem network fs with parameters O.. light weighted classification head fe with parameters O.. and an auxiliary image reconstruction head fr with parameters 0,. For an input image z, the stem network outputs a hidden feature h(x)= = (z; 0g). The classification head outputs the classification logits 2(x) fe(h(z); 0c). The image reconstruction head outputs the reconstructed image 2(x) = f.((());
3.2 The proposed method
Our Trap and Replace (T&R) backdoor defense strategy consists of two stages. Below we formally describe the process of each stage.
Stage 1: Bait and trap the backdoor into the classification head. As intuitively explained in Section we use the auxiliary image reconstruction task to regularize the stem network fs to keep enough semantically correct low-level visual features. The stem network is thus protected from overfitting to the casy-to-learn but semantically incorrect backdoor correlations 3 In contrast, we apply no defense mechanisms on the light- weighted classification head fer leaving more vulnerable than the stem. As a result, the backdoor attacks are easily baited towards and trapped in the classification head, leaving the stem network hardly infected.
Specifically, we jointly train the classification task and the auxiliary image classification task on the poisoned training set Drrain The entire model is updated end-to-end. More formally, we solve the following optimization problem:
min ExwDin Carlz) X Cre P..0.
(1)
where Lar(z) and Lree (x) the image classification and reconstruction losses, respectively, and Y is the trade-off weight between the two loss terms. The image classification loss Lar(2)) is simply the
cross-entropy loss between logits 2 z(x) and corresponding ground truth label yzz). The reconstruction loss is the &g loss between the original and reconstructed images with the total variation regularization:
Live (x) [(((x)) æl|2 + Tv (f(=)), where TV(-) is the total variation function, which is widely
used to improve the spatial smoothness and visual quality of the reconstructed image ((x) 44 50].
and X is the loss trade-off weight.
"The relation and difference in application scenario with previous works discussed in Appendix D 'In contrast, the stem network obtained from normal training (i.e., without the auxiliary image reconstruction task) will overfit to the semantically incorrect but casy-to-learn backdoor correlations. See our ablation study in
Section 4.4 Table 4 for details.
5

---

Stage 2: Replace the infected classification head. After stage 1, the stem network keeps enough semantically correct low-level visual features. Now the missing piece of a disinfected image classifier is an untainted light-weighted classification head, which can be obtained by re-training Se from scratch using the small clean holdout set Dn. Specifically, we first discard the auxiliary image reconstruction head f,, and re-initialize the classification head parameters 0g to random values. We then re-train 0g on Dn. while keeping the stem parameters Og fixed to the values learned in stage 1. More formally, we solve the below optimization problem:
min Er Car(x))
(2)
Note that solving Eq. (2) from scratch is feasible since fo is light-weighted. To further prevent overfitting the small Db. we apply Dropout (with 50% ratio) [5T] and label smoothing (with
smoothing factor 0.1) [52] as regularization when solving Eq. Finally, we summarize the
workflow of T&R in Algorithm[[
Algorithm 1: Trap and Replace (T&R)
Input: Poisoned training set Danin a small holdout set Dp.
Output: A disinfected image classifier [a(Sa(;; On); 0s).
Stage 1:
Randomly initialize On. O.. and 0,. 3 Optimize Op. tc. and 8, on Duain according to Eq.
// Stage 2:
5 Randomly initialize 0e- Optimize 0₂ Dr according to Eq. 2
Experiments
4.1 Experimental settings
Datasets and models Following 12 5 32), we conduct experiments on CIFARIO (33, GTSRB
544, and ImageNet-12 subset ImageNet (55) with 12 classes); we use WideResNet (WRN16-1)
56 on the first two datasets and ResNet34 [57 on ImageNet-12.
Backdoor attacks We evaluate the effectiveness of our method against 10 different backdoor attacks. Specifically, we use all 7 attacks in [5] BadNet with white square pattern (denoted as BadNet-White) (6, Blend 14] Invisiiee [16), Ln-Invisible
[16), Smooth 19, Trojan with square pattern (denoted as Trojan-SQ) [. and Tojan with watermark pattern (denoted as Trojan-WMi [. We further include 3 more attacks used in [2]] BadNet with grid square pattern (denoted as BadNet-Grid) 66, label-consistent backdoor attack (LCBA) (20). and SIG (21). Note that LCBA and SIG are clean-label attacks and all others are dirty-label attacks. Following 12], we set poison ratio a 10% for all attacks (i.e., 5000 training images are poisoned in CIFAR10). Detailed settings and visualization for each attack are shown in Appendix B.I. As pointed out by [12, some attacks fail to be reproduced following their original papers GTSRB and Imagenet- 12. Following the suggestions in [2], we omit CLBA attack on GTSRB and use four attacks on ImageNet-12. 12.
Baseline methods We compare our method with six baseline methods: normal training (denoted as "No defense"), differentially private training (DP) [30], NAD [4], ANP [II]. ABL [12), and I-BAU (5). The last three are recent state-of-the-art methods that have been shown largely outperform previous baselines. The results of DP are only shown in Appendix due to space limit (and also since it achieves the least competitive results). We omit ANP and I-BAU on ImageNet- 12, since the original papers only provided experimental settings on CIFAR1O and we fail to produce reasonable results on ImageNet after careful
tuning.
Evaluation measures Following [12
5]. we use attack success rate (ASR) and clean accuracy (CA) as the evaluation measures. ASR defined as the percentage backdoor test samples that can successfully fool the model to the target class desired by the attacker. CA is simply the classification accuracy on a clean test set. The smaller ASR (L) and the larger CA (t) indicate better backdoor defense performance.

---

Defense settings Previous works usually use 10% of the training set as the clean holdout set B 4 In this paper, we consider a more challenging setting where less clean holdout data are available for defenders. Specifically, for all methods requiring a clean holdout set (i.e., NAD, ANP, I-BAU, and our T&R), we use 5% of the training set as the clean holdout set on CIFAR1O and GTSRB. The ratio is still set to 10% on ImageNet-12 for all methods since it is more challenging dataset. Please see Appendix B for more details on experimental settings.
4.2 Main results
Table 1: Results on CIFARIO using WRN16-1. The best results are shown in bold. All numbers are reported in percentages.
No defense ASR
NAD
ANP
ABL
I-BAU
Ours
Attack method
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
BadNet-Grid
99.98
88.36
8.60
80.23
2.64
84.55
3.20
86.35
8.24
81.34
1.21
84.42
BadNet- White
96.91
88.87
12.80
79.68
3.68
85.74
9.32
80.32
8.80
86.17
3.14
83,96
Blend
99.11
88.95
10.56
81.89
5.62
82.35
19.91
80.63
12.27
75.18
10.59
83.82
In-iniisible
99.93
89.36
30.25
76.84
15.76
72.5
16.76
76.73
9.21
81.15
2.91
84.04
Invisiiibe
100.00
89.28
10.06
75.63
22.97
70.52
7.49
76.82
2.64
82.48
0.74
84.01
Smooth
98.14
89.24
15.68
79.10
33:42
77.49
20.21
70.79
24.80
67.93
4.23
83.63
Trojan-SQ
98.23
89.64
12.52
80.56
23.84
72.06
3.16
80.44
16.61
83.27
6.54
79.92
Trojan-WVP
99.61
89.48
33.64
79.60
18.04
75.66
7.31
84,89
4.55
85.30
12.66
79.97
SIG
99.93
89.27
3.20
75.24
0.05
85.67
8.79
60.07
1.89
77.05
0.02
82.97
LCBA
90.42
86.35
10.64
77.82
1.02
84.21
6.54
78.56
8.88
78.05
5.41
82.57
Average
97.95
88.54
14.79
78.66
12.70
79.08
10.27
77.56
9.79
79.79
4.75
82,93
Table 2: Results GTSRB. The best results are shown in bold. All numbers are reported in
percentages.
No defense
NAD
ANP
ABL
I-BAU
Ours ASR
Attack method
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
CA
BadNet-Grid
100
96.36
1.62
93.56
8.97
95.44
4.11
95.52
5.56
95.36
0.20
95.94
BadNet-White
96.32
96.11
0.96
90.61
12.68
92.06
0.00
95.14
6.63
95.55
0.01
95.74
Blend
99.95
96.50
5.64
91.26
20.36
90.53
4.94
92.61
0.00
82.03
1.98
95.62
fy Invisbleeee
100
95.50
25.60
91.35
34.96
88.54
0.60
96.39
4.73
94.31
1.32
95.87
f2 Invisiiie
99.93
96.81
12.43
72.69
24.83
90.41
100
94.74
9.83
94.60
0.03
96.10
Smooth
100
96.58
33.20
75.74
37.95
82.56
32.24
76.70
2.68
94.74
0.11
96.12
Trojan SQ
97.75
96.53
1.52
90.56
12.06
90.74
0.45
95.88
5.71
94.49
1.47
95.17
Trojan WM
100
96.68
1.58
92.26
25.68
88.64
0.55
95.41
7.47
95.30
7.06
91.95
SIG
87.51
96.32
50.68
88.56
20.60
82.34
66.80
96.50
10.04
94.53
0.54
94.56
Average
97.94
96.38
16.45
87.40
22.01
89.03
23.30
93.21
5.85
93.43
1.41
95.23
Table 3: Results on ImageNet-12. The best results are shown in bold. All numbers are reported in percentages.
No defense ASR
NAD
ABL
Ours ASR
Attack method
CA
ASR
CA
ASR
CA
CA
BadNet-Grid
99.67
74.17
13.67
71.67
3.59
74.06
0.67
72.64
Blend
99.17
74.83
33.50
70.17
25.06
70.12
11.33
67.33
Trojan-WM
100
77.33
29.33
73.33
3.26
72.08
7.52
70.83
SIG
86.00
71.67
19.67
70.83
7.67
67.01
5.33
77.33
Average
96.21
74.50
24.04
71.50
9.89
70.82
6.21
72.03
The main results on CIFAR10, GTSRB, and Image 12 are shown in Table As we can see, our method largely outperforms previous state-of-the-ar art methods in terms of both
2 and 3 respectively.
7

---

ASR and CA on all three datasets. Specifically, CIFARIO, our method outperforms I-BAU (the most competitive baseline method) by 20.57% ASR against Smooth attack, 10.07% ASR against Trojan-SQ attack, and 3.14% average CA. On GTSRB dataset, our method outperforms I-BAU by 9.80% ASR against In-invise attack, 9.50% ASR against SIG attack, and 1.80% average CA. On ImageNet-12 12 dataset, our method outperforms ABL by 13. 72% ASR against Blend attack, 3.68% average ASR, and 1.21% average CA.
Moreover, we can observe that our method has almost no failure cases on all three datasets. In contrast, all baseline methods have failure cases, with either a significant drop CA compared with normal training (i.e., "No defense") or an unacceptably high ASR. For example, ABL fails on In-Invisible attack and SIG attack on GTSRB, and I-BAU fails on Smooth attack on CIFARIO and Blend attack on GTSRB.
4.3 Is the stem network really backdoor-free?
clean airplane samples
clean airplane samiples
clean automobile samples
clean automobile samples
poisoned automobile samples
poisoned automobile samples
(a) No defense
(b) Our T&R
(without image reconstruction head)
(with image reconstruction head)
Figure 2: Output feature scatters of the stem network fa when trained without (left) and with (right) the auxiliary image reconstruction head. Principle component analyses (PCA) is applied to project the features to two-dimensional space. Three types of test samples are visualized: clean "airplane" samples, clean "automobile" samples, and backdoored "automobile" samples (i.e., automobile images with & Invisilee pattern). Both models are trained on CIFAR10 R10 dataset poisoned by Invisibee attack which aims to map the backdoor pattern to the target "airplane".
In this section, we verify our assumption that the image reconstruction task can protect the stem network from overfitting to the backdoor correlations, by visualizing the feature scatters. Specifically, we compare the output features of the stem network fs when trained with or without the auxiliary image reconstruction task. We visualize the feature distributions of clean "airplane" samples, clean "automobile" samples, and poisoned "automobile" samples via PCA in Figure 2
As shown in Figure a),, without the auxiliary image reconstruction task, the stem network fa maps poisoned "automobile" " samples to the feature space of clean "airplanc" samples instead of the clean "automobile" feature space. This demonstrates that fa has learned the backdoor correlation: It ignores the semantic features related to "automobile" in the poisoned "automobile" samples, but overfits the semantically-incoree backdoor correlations between the backdoor pattern and the backdoor target "airplane". In contrast, as shown in Figure (bb), when trained with the image reconstruction head, clean and poisoned "automobile" samples are projected to similar feature space, which is different from the "airplane" feature space, This indicates that fs learns semantically-cores features, instead of the backdoor correlations, on the poisoned samples. In conclusion, the visualization results show that the auxiliary image reconstruction task successfully hinders the stem network from learning the backdoor correlations.
8

---

4.4 Ablation study
The importance of the auxiliary image reconstruction task In this paragraph, we show the importance of two building blocks in T&R: the auxiliary image reconstruction task in stage I, and the re- re-training of the classification head in stage 2. The results are shown in Table 4] As shown in the first row, if we remove the auxiliary image reconstruction task in stage 1, the model will not effectively unlearn the backdoor correlations even we retrain the classification head from seratch. For example, the ASRs of fz-Invisible and Trojan-WM WMMM are still as high as 99.99% and 51.87%, respectively. This indicates that without the auxiliary image reconstruction task, the backdoor training samples have infected large portion of the network, including both the classification head and the stem network. In contrast, with the help of the auxiliary image reconstruction task, we successfully trapped the backdoor correlations into the light-weighted classification, while leaving the stem network largely uninfected. On the other hand, as shown in the second row in Table 4] using the auxiliary image reconstruction task alone will not directly reduce ASR, since the classification head is still infected. In conclusion, both building blocks are necessary: The defense will fail with either one missing.
Table 4: Ablation study on the building components of T&R using CIFAR1O. In the last row, we report the mean and standard deviation of the results over three random runs of our method.
Auxiliary image reconstruction task
Replace classification head
In-Invisible CA
Trojan-Wii WM
SIG
LCBA
ASR
ASR
CA
ASR
CA
ASR
CA
/
99.99
85.08
51.87
84.26
26.96
84.03
45.81
84.31
x
99.96
89.06
99.81
88,50
97.24
81.10
92.46
81.22
0.72
83.72
12.01
80.01
0.01
82.99
5,37
82.47
+0.09
+0.25
+1.16
+0.33
+0.01
+0.03
+0.45
+0.11
We also report the error bars of our method in the last row of Table 4
Specifically, we run our method three times with different random seeds and report the mean (/4) and standard deviation (o) of ASR and CA in the form of μ + a. As we can see, the performance of our method is statistically stable under all four different attacks.
From which layer should we separate the stem and classification head? The ablation study results on the stem-head separation layer are shown in Table 5] As we can see, if the stem contains too many layers (e.g.. as in the first row), the backdoor defense will fail, since it is hard to trap all the backdoors in too tiny classification head (e.g., only one fully connected layer as the first row). On the other hand, if the head contains too many layers (e.g, as in the third row), the backdoor defense will be successful, but CA will significantly drop. This is because, with a limited amount of holdout clean samples, it is infeasible to re-train too large classification head from scratch. In summary, properly sized classification head is important for the practical success of our method.
Table 5: Ablation study on the stem-head separation layer. Experiments are conducted on CIFARIO with WRN16-1 backbone, which has total of convolutional and fully connected (FC) layers. In the first row, fo has only one FC layer, and all convolutional layers belong to fs. The second row is our default setting, where fe has the last two convolutional layers and the FC layer. The best results are shown in bold and the second-best ones are underlined.
Number of layers
Blend
fn-Invisible ASR
Smooth ASR
Trojan- WM ASR
fz
fc
ASR
CA
CA
CA
CA
13
1
97.41
88.04
99.99
87.93
96.02
87.60
99.63
87.52
11
3.
10.59
83.82
0.74
84.01
4.23
83.63
12.66
79.97
9
5
3.01
74.99
0.37
73.50
4.17
73.96
13.83
77.36
Ablation study on poison ratio In this paragraph, we study the performance of T&R under different poison ratios. Specifically, we set the poison ratio a to 5%, 10% (the default value used in our main experiments), and 20%, which are the popular values used in previous works [2 (5 We also compare T&R with I-BAU, which is the most competitive baseline as shown in Table 1 under these different poison ratios. The results are summarized Table As we can see, the advantage of our method holds across different poison ratios.

---

Table 6: Ablation study on the poison ratio a using CIFARIO. The best results are shown in bold.
Blend ASR
fy-Invisible ASR
Smooth ASR
Trojan-WM ASR
Defense method
CA
CA
CA
Ours I-BAU
9.37
83,52
0.54
84.21
2.04
83.96
7.61
80.12
5%
10.16
81.96
1.68
81.04
8,58
80.92
5,34
82.69
Ours I-BAU
10.59
83,82
0.74
84.01
4.23
83.63
12.66
79.97
10%
12.27
75.18
2.64
82.48
24.80
67.93
4.55
85.30
Ours I-BAU
8.74
82.15
0.84
84.25
3.51
82,96
14.28
79.84
20%
15.57
81.06
4.38
83.06
16.69
80.44
6.94
83.33
More experimental results, including results against potential adaptive attack, ablation study on hyper-parameeers, ablation study on the size of clean holdout set Dn, and results on non-poisoned datasets (i.e., when poison ratio = 0), are presented in Appendix
5 Discussion
The major finding in this paper is that the auxiliary image reconstruction loss can successfully trap backdoors into a small subnetwork while preserving the rest of the network largely uncontaminated, as illustrated in Figure 2 The impact of this finding is that it provides promising decrease-and- conquer strategy for backdoor defense. Since the stem network is largely uncontaminated, defenders only need to tackle the small backdoored subnetwork, which is easier than sanitizing the entire network. This finding inspires multiple future work directions. First, our "trapping" strategy can potentially be combined with previous pruning-based backdoor neuron removal methods [1]] [29]
for better results. This is because it reduces the search space of potential backdoor neurons: We only need to search and remove backdoored neurons in the small backdoored subnetwork. instead of the entire network as done previously in 29).. Second, although sufficiently effective against existing backdoor attacks, the image reconstruction loss is not guaranteed be the best option for the "trapping" strategy, and one may research better alternatives as future work.
Limitation Our method has less flexible application scenario compared with previous backdoor defense methods, since it requires the backdoored training set. Most previous defense methods 4 take the backdoored model, instead of the original backdoored training set, as input. Unlike those methods, our method cannot sanitize the backdoored models when the backdoored training set not available. With that said, the application scenario of our method is still popular in real-world applications. Please see Appendix D for details.
6 Conclusion
In this paper, we show that an auxiliary image reconstruction loss can successfully hinder the learning of backdoor attacks in image classification tasks. Based on this observation, we propose brand- new backdoor defense strategy named Trap and Replace (T&R). Empirical results on three image classification datasets show advantage our method over previous state-of-the-arr methods.
Acknowledgement
This material is based in part upon work supported by the National Science Foundation under Grant 11S-2212174, IIS- 118-2212176, IIS- 174994,,, and Office of Naval Research N00014-20-1-2888
References
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang. and Xiangyu Zhang Trojaning attack on neural networks. In Network and Distributed System Security Symposium, 2017.
[2] Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? A unified benchmark for backdoor and data poisoning attacks. In International Conference on Machine Learning, pages 9389-9398, 2021.
10

---

[3] Kang Liu,, Brendan Davit,, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 273-294, 2018.
[4] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distilla- tion: Frasing backdoor triggers from deep neural networks. In International Conference on Learning Representations, 2020.
[5] Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations, 2022.
[6] Tianyu Gu. Kang Liu, Brendan DavintGavitt, and Siddharth Garg. BadNets: Evaluating backdooring attacks on deep neural networks. IEFE Access, 0177230-47244, 2019.
[7] Zhicong Yan, Gaolei Li, Yuan Tian, Jun Wu, Shenghong Li, Mingzhe Chen, and H Vincent Poor DeHliB: Deep hidden backdoor attack on semi-supervised learning via adversarial perturbation. In AAAI Conference on Artificial Intelligence, pages 10585-10593, 2021.
[8] Nicholas Carlini. Poisoning the unlabeled dataset semi-supervised learning. USENIX Security Symposium, pages 1577-1592, 2021.
[9] Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Backdoor attacks on self-supervised learning. In JEEE Conference on Computer Vision and Pattern Recognition, 2022.
[10] Nicholas Carlini and Andreas Terzis, Poisoning and backdooring contrastive learning. In International Conference on Learning Representations, 2022.
[11] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In Advances in Neural Information Processing Systems, 2021.
[12] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In Advances in Neural Information Processing Syxtems, 2021.
[13] Yuanshun Yao, Huiying Li. Haitao Zheng, and Ben Zhao. Latent backdoor attacks on deep neural networks. In ACM SIGSAC Conference on Computer and Communications Security, pages 2041-2055 2019.
[14] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, arXiv preprint arXi:: 2017.
[15] Haoti Zhong, Cong Liao, Anna Cinzia Squicciarini, Sencun Zhu, and David Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. In ACM Conference Data and Application Security and Privacy, pages 97-108, 2020.
[16] Shaofeng Li, Minhui Xue, Benjamin Zi Hao Zhao, Haojin Zhu, and Xinpeng Zhang. Invisible backdoor attacks on deep neural networks via steganography and regularization. IEEE Transactions on Dependable and Secure Computing, 1865):2088-2155, 2020.
[17] Yuezun Li, Yiming Li, Baoyuan Wu. Longkang Li. Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In IEEE International Conference Computer Vision, pages 16463-16472, 2021.
[18] Tuan Anh Nguyen and Anh Tuan Tran. Wanet-impe warping-based backdoor attack. In Interna- tional Conference on Learning Representations, 2021.
[19] Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In IEEE International Conference on Computer Vision, pages 16473-16481, 2021.
[20] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks, arXiv preprint 12 2019.
[21] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set coruptiinn without label poisoning. In IEEE International Conference on Image Processing, pages 101-105. 2019.
[22] Yi Zeng. Minzhou Pan, Hoang Anh Just, Lingjuan Lyu. Meikang Qiu, and Ruoxi Jia. NARCISSUS: A practical label backdoor attack with limited information. arXiv preprint wxXx:2204.002255. 2022.
11

---

[23] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In AAAI Conference on Artificial Intelligence, pages 11957-11965, 2020.
[24] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems, 2018.
[25] Jonathan Hayase and Weihao Kong. SPECTRE: Defending against backdoor attacks using robust covari- estimation. In International Conference on Machine Learning, 2020.
[26] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig. Benjamin Edwards, Taesung Lee. Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks deep neural networks by activation clustering. arXiv preprint arXiv: 181 1.03728, 2018.
[27] Yansong Gao, Change Xu, Derui Wang. Shiping Chen, Damith C. Ranasinghe, and Surya Nepal. STRIP: A defence against trojan attacks on deep neural networks. Annual Computer Security Applications Conference, pages 113-125, 2019.
[28] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, In International Conference on Machine Learning, pages 1885-1894, 2017.
[29] Jiyang Guan, Zhuozhuo Tu, Ran He, and Dacheng Tao. Few-shot backdoor defense using shapley estimation. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[30] Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. In International Conference on Learning Representations, 2020.
[31] Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages
3855-3859, 2021.
[32] Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In International Conference on Learning Representutions, 2022.
[33] Cynthia Dwork. Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of Computation, pages 1-19, 2008.
[34] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mix Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[35] Chengyue Gong. Tongzheng Ren, Mao Ye, and Qiang Liu. MaxUp: Lightweight adversarial training with data augmentation improves neural network training. In IEEE Conference on Computer Vision and Puttern Recognition, pages 2474-2483, 2021.
[36] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learming Representattinss 2013.
[37] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[38] Jonas Geiping, Liam Fowl, Gowthami Somepalli, Micah Goldblum, Michael Moeller, and Tom Goldstein. What doesn't 't kill you makes you robust (er): Adversarial training against poisons and backdoors. arXiv preprint 1xXxx:2002.13222 2021.
[39] Naren Manoj and Avrim Blum. Excess capacity and backdoor poisoning. Advances in Neural Information Processing Systems, 2021.
[40] Bolun Wang. Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE Symposium on Security and Privacy, pages
2019.
[41] Huili Chen, Cheng Fu, Jishen Zhao and Farinaz Koushanfar. DeepInspect: A black-box trojan detection and mitigation framework for deep neural networks. In International Joins Conference Artificial Intelligence, pages 1658-4664, 2019.
[42] Wenbo Guo, Lun Wang, Xinyu Xing. Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems. arXiv preprint 11Xiv:1908.01763, 2019.
12

---

[43] Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An. Qiuling Xu, Shiqing Ma. Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[44] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting Al trojans using meta neural analysis. In IEEE Symposium on Security and Privacy, pages 103-120, 2021.
[45] Yinpeng Dong. Xiao Yang. Zhijie Deng. Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection of backdoor attacks with limited information and data. In IEEE International Conference on Computer Vision, pages 16482-16491, 2021.
[46] Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-bor backdoor detection using adversarial extreme value analysis. In International Conference on Learning Representations, 2021.
[47] Xiaoling Hu, Xiao Lin, Michael Cogswell, Yao, Susmit Jha, and Chao Chen. Trigger hunting with a topologiaal prior for trojan detection. In International Conference on Learning Representations, 2022.
[48] Tianlong Chen. Zhenyu Zhang, Yihua Zhang, Shiyu Chang, Sijia Liu, and Zhangyang Wang- Quarantine: Sparsity can uncover the trojan attack trigger for free. In IEEE Conference on Computer Vision and Pattern Recognition, pages 598-608 609, 2022.
[49] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pages 694-711, 2016.
[50] Hussein A Aly and Eric Dubois. Image up- sampling using total-variation regularization with a new observation model. IEEE Transactions on Image Processing, 14410))1647-1655, 2005.
[51] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout:
A simple way to prevent neural networks from overtitting. Journal of Machine Learning Research, 15()):1929-1958 2014.
[52] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2818-2826, 2016.
[53] Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[54] Scbastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference on Neural Networks, 2013.
[55] Jia Deng, Wei Dong. Richard Sacher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hicrarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[56] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, pages 87. 12. 2016.
[57] Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[58] Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can adversarial weight perturbations inject neural backdoors. In ACM International Conference on Information & Knowledge Management, pages
2020.
[59] Zhiyuan Zhang, Lingjuan Lyu. Weiqiang Wang, Lichao Sun, and Xu Sun. How to inject backdoors with better consistency: Logit anchoring on clean data. In International Conference on Learning Representations, 2022.
[60] Jakub Breier, Xiaolu Hou, Dirmanto Jap, Lei Ma. Shivam Bhasin, and Yang Liu. Practical fault attack on deep neural networks. In ACM SIGSAC Conference on Computer and Communications Security, pages 2204 2206, 2018
[61] Jiawang Bai, Baoyuan Wu, Yong Zhang, Yiming Li, Zhifeng Li, and Shu-Tao Xia. Targeted attack against deep neural networks via flipping limited weight bits. In International Conference on Learning Representations, 2021.
[62] Xiangyu Qi. Tinghao Xie, Ruizhe Pan, Jifeng Zhu, Yong Yang, and Kai Bu. Towards practical deployment- stage backdoor attack on deep neural networks. In IEEE Conference on Computer Vision and Patern Recognition, 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Section 5
(c) Did you discuss any potential negative societal impacts of your work? [No] We do not see significant potential negative societal impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] All codes and pre-trained models will be released after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 4.1 and Appendix B
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [Yes] Please see Table 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] All datasets we used are public and cited properly.
(b) Did you mention the license of the assets? [Yes] All datasets used are public and cited properly.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 'r using/curating? [Yes] All datasets we used are public and cited properly.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We only use public image classification datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "dqCQTiY9cLy", "reviewer": "Reviewer_Nfpj", "summary": "This paper proposes a new defense strategy *Trap and Replace* to protect deep neural networks from the backdoor in the poisoned dataset. This paper presumes that the backdoor pattern is easy to learn, so it trains a standard image classification model consisting of a stem network and a classification head and an extra image reconstruction model, which consists of the stem network and reconstruction head at the same time to encourage stem model to learn the low-level visual features. Then freeze the stem work and initialize the classification head by setting those parameters to random values. And train the new model in a small but clean dataset. The experimental results show that Trap and Replace outperforms SOTA defense strategies in most cases.\n", "questions": "N/A", "limitations": "Comments:\n1. In previous works, the classification head is considered the fully connected layer. You can explain the definition of classification head and reconstruction head in the introduction session.\n2. In line 156, there is a repetitive word \"assumptions\".", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Introducing a reconstruction task to force the stem model to learn visual features is a novel idea. It is self-supervised training that does not need extra labeled samples.\n2. Training classification head-on clean dataset can mitigate the attack success rate and keep accuracy at a high level.\n3. The paper's summary of relative works is well organized and comprehensive.\n4. The experiments are well designed and clearly shown in the tables.\n5. Ablation study showing that both Trap and Replace are necessary is convincing.\n\n\t \nWeakness:\n1. Why using the fully connected layer as classification head fails to defend the backdoor should be discussed.\n2. How many layers should be chosen as the classification head in others models that are not included in this paper should be considered.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dZ1rkH6OVZB", "reviewer": "Reviewer_cjmb", "summary": "The proposed work introduces a defense which uses an auxiliary reconstruction task along with the classification to ‘trap’ the backdoor within the classification head. In the second stage, a new classification head is trained from scratch to completely remove the effect of the backdoor. The intuition is that the auxiliary task ensures that low-level features of the image is preserved within the backbone or the stem network, hence reducing the effectiveness of the backdoor. Results are shown on CIFAR-10, GTSRB and ImageNet-12 against various attacks.", "questions": "— It seems that replacing the classification head without the auxiliary task results in significant drop in ASR. Is it possible to consider an experiment where the learning rate for samples from clean holdout set is 10x the learning rate for the untrusted dataset? There is no auxiliary task in this scenario. Such an experiment would show whether the effectiveness of the defense is purely due to the correct decision boundaries learned by the classification head or if it is due to the trapping mechanism.\n\n— A more detailed ablation study on the size of clean holdout set is necessary. From, Table 9 of the appendix, it seems there is no difference between the 2.5% and 5% setting. It would be interesting to know what is the least amount of samples required to achieve similar results. \n\n\nAfter Rebuttal:\nThanks to the authors for providing the rebuttal. Having gone through the other reviews, I will keep my score unchanged and strongly encourage the authors to include some of the new experiments about the relationship between reconstruction and ASR in the main paper.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n— The defense method introduced is novel and effective. It is also easy to use without requiring too much hyper-parameter tuning compared to previous methods.\n\n— Authors conduct extensive ablation study to illustrate the effectiveness of the method. An Adaptive attack is also considered, against which the defense still remains effective.\n\n— Results are shown on a variety of backdoor attacks, indicating that it can be used to defend against multiple attacks.\n\n**Weaknesses:**\n\n— Although the authors do provide an intuition that low-level features are preserved better with the auxiliary task, it is not clear why the proposed method should be effective. A feature analysis on the difference with and without the reconstruction task can be helpful in identifying why the defense is effective to such a degree.\n\n— The reconstruction task proposed by the authors is not very well understood. An experiment where the quality of reconstructed images vs ASR can improve understanding. Although authors do provide a variant of this experiment in Table 8 of the appendix, a qualitative analysis would be better in this scenario. Another experiment would be to vary the size or capacity of the ‘decoder’ and observe the variation in ASR.\n\n— Similar to previous methods, a clean heldout dataset is required which is typically 10%. This can be difficult to obtain for larger datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WtXWsvAJhSi", "reviewer": "Reviewer_goHi", "summary": "The authors of this work proposed a defense method against backdoor attacks. The defense method consists of two stages. In the first stage, they trapped the backdoors in a subnetwork. In the second stage, they replace the poisoned subnetwork and retrain the network with clean samples. Consequently, this method outperforms previous state-of-the-art methods.", "questions": "1. I think the reconstruction feature is pretty different from the classification feature which may cause both tasks to be difficult to train in more complex datasets. Can you show some reconstruction results? Does the reconstruction network reconstruct the trigger as well?\n2. It is confusing that in the SIG attack on ImageNet, the clean accuracy becomes higher in Table 3. Why your finetuning is so powerful in this case?\n3. Is your framework only suitable for convolutional neural networks? How about vision transformers?\n4. As you mentioned in the footnote on page 2, semi-supervised learning and self-supervised learning are not naive solutions, since they are vulnerable to backdoor attacks. Is your work aim to provide a defense mechanism in this scenario?\n5. As a suggestion and a question, I think that the poisoned data is aim to fool the classifier hence finetuning will work in this case. In multi-task learning, you can try to create poisoned data that aims to fool any downstream tasks. As finetuning might work for each task, how about putting various purpose attacks at the same time?\n", "limitations": "Yes. The authors state that there are no potential negative societal impacts of this work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The writing of this paper is very clear and easy to follow.\n2. The experimental results show that the method outperforms the other six baseline methods in various attack methods except in some cases (Blend, Trojan SQ, Trojan WM).\n3. The ablation study clearly shows the significance of the two stages.\n\nWeaknesses:\n\n1. In my opinion, the threat model is pretty unrealistic (with some holdout clean samples) however you can follow this assumption from previous works. \n2. The auxiliary image reconstruction task encourages the stem network to keep sufficient low-level visual features that are hard-to-learn but semantically correct, protecting the stem network from overfitting to the easy-to-learn but semantically incorrect backdoor correlations. I cannot figure out the relation between this intuition and how the effect of poisoned data can be trapped.\n\nTypos:\n1. Line 156 has two “assumptions”.\n2. Line 238 “Tojan” --> “Trojan”.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9hWtHlz6jI", "reviewer": "Reviewer_CZwS", "summary": "The authors propose to trap the backdoor by a lightweight classification head on top of a low level feature extractor and replace it with a clean classifier to remove the backdoor. Extensive experimental results on different dataset against various attacks show the effectiveness of the proposed method.", "questions": "a. I am curious about the performance of the other methods. For example, in [11], ANP use 500 images (i.e., 1% of the training data) to purify the model against SIG on CIFAR-10, and the degradation of the clean accuracy is only 0.24% (93.64% to 93.40%). In this paper, the authors use 5% of the training data (2500 images), but the performance of ANP is far worse (89.27% to 85.67%). It is strange that the baseline methods perform such badly.", "limitations": "No. I think the largest limitation is the threat model. As far as I know, this is the first method that require both a poisoned dataset and a clean holdout set. If so, I suggest the author to discuss the practical scenario of such threat model.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\na. The idea is interesting, and the motivation is reasonable.\n\nb. The authors provide rich ablation experiments to evaluate the proposed method.\n\nWeaknesses:\n\na. Most of the previous methods only need access to either the poisoned dataset or a small clean dataset, but the proposed method requires access to both, which limits its practical value.\n\nb. According to Table 1, the degradation on clean accuracy is considerably large. This may because the final classifier is only trained on limited data (the holdout dataset).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept