Backdoor Defense via Decoupling the Training Process
Authors: Kunzhe Huang
Yiming Li
Baoyuan Wu
Zhan Qin
Kui Ren
Venue: ICLR 2022 Poster (2022)
Keywords: Backdoor Defense
Backdoor Learning

Abstract: Recent studies have revealed that deep neural networks (DNNs) are vulnerable to backdoor attacks, where attackers embed hidden backdoors in the DNN model by poisoning a few training samples. The attacked model behaves normally on benign samples, whereas its prediction will be maliciously changed when the backdoor is activated. We reveal that poisoned samples tend to cluster together in the feature space of the attacked DNN model, which is mostly due to the end-to-end supervised training paradigm. Inspired by this observation, we propose a novel backdoor defense via decoupling the original end-to-end training process into three stages. Specifically, we first learn the backbone of a DNN model via \emph{self-supervised learning} based on training samples without their labels. The learned backbone will map samples with the same ground-truth label to similar locations in the feature space. Then, we freeze the parameters of the learned backbone and train the remaining fully connected layers via standard training with all (labeled) training samples. Lastly, to further alleviate side-effects of poisoned samples in the second stage, we remove labels of some `low-credible' samples determined based on the learned model and conduct a \emph{semi-supervised fine-tuning} of the whole model. Extensive experiments on multiple benchmark datasets and DNN models verify that the proposed defense is effective in reducing backdoor threats while preserving high accuracy in predicting benign samples. Our code is available at \url{https://github.com/SCLBD/DBD}.

Published as a conference paper at ICLR 2022
BACKDOOR DEFENSE VIA DECOUPLING THE TRAIN- ING PROCESS
Kunzhe Huang"., Yiming Li Baoyuan Wu Zhan Qin Kui Ren"
1 "ccoool of Cyber Science and Technology. Zhejiang University
Schooo of Data Science, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen
TTiinguaa Shenzhen International Graduate School, Tsinghua University
[hkunzhe, zhanqin, kuiren) ) @ zww..edu.cn; wubaoyuan www.hc.cuu..... li-ym mailss.sinnguu..duuu
ABSTRACT
Recent studies have revealed that deep neural networks (DNNs) are vulnerable to backdoor attacks, where attackers embed hidden backdoors in the DNN model by poisoning a few training samples. The attacked model behaves normally on benign samples, whereas its prediction will be maliciously changed when the backdoor is activated. We reveal that poisoned samples tend to cluster together in the feature space of the attacked DNN model, which is mostly due to the end- to-end supervised training paradigm. Inspired by this observation, we propose novel backdoor defense via decoupling the original end-to-end training pro- cess into three stages. Specifically, we first learn the backbone of a DNN model via self-supervised learning based on training samples without their labels. The learned backbone will map samples with the same ground-truth label to similar locations the feature space. Then, we freeze the parameters of the learned back- bone and train the remaining fully connected layers via standard training with all (labeled) training samples. Lastly, further alleviate side-effects of poisoned samples in the second stage, we remove labels of some 'low 'low-credible" samples determined based on the learned model and conduct a semi-supervised fine-tuning of the whole model. Extensive experiments DNN models verify that the proposed defense is effective in reducing backdoor threats while preserving high accuracy in predicting benign samples. Our code is available at https:: gighuuuuubh com/ SCLBD/DDD
multiple benchmark datasets and
INTRODUCTION
Deep learning, especially deep neural networks (DNNs), has been widely adopted many realms (Wang et al.) 2020b Li et al. 2020a; Wen et al. 2020) for its high effectiveness. In general, the training DNNs requires large amount of training samples and computational resources. Accord- ingly, third-party -arty resources (e.g.. third-party data or servers) are usually involved. While the opacity of the training process brings certain convenience, it also introduces new security threats.
Backdoor attack poses new security threat to the training process of DNNs Li et al. 2020c). It maliciously manipulates the prediction of the attacked DNNs by poisoning a few training samples. Specifically, backdoor attackers inject the backdoor trigger (i.e., a particular pattern) to some benign training images and change their labels with the attacker-speciffed target label. The connection between the backdoor trigger and the target label will be learned by DNNs during the training process. In the inference process, the prediction of attacked DNNs will be changed to the target label when the trigger is present, whereas the attacked DNNs will behave normally on benign samples. As such, users are difficult to realize the existence of hidden backdoors and therefore this attack is serious threat to the practical applications of DNNs.
In this paper, we first investigate backdoor attacks from the hidden feature space. Our preliminary experiments reveal that the backdoor is embedded in the feature space, i.e.. samples with the back-
'The first two authors contributed equally to this work. This work was mostly done when Kunzhe Huang and Yiming Li were the research interns at The Chinese University of Hong Kong, Shenzhen. indicates corresponding authors: Baoyuan Wu (wubaoyuan @cuhk.edu.c)) and Zhan Qin (qinzhan (z..huunn.......ccn.

---

Published as a conference paper at ICLR 2022
door trigger (dubbed poisoned samples) tend to cluster together in the feature space. We reveal that this phenomenon is mostly due to the end-to-end supervised training paradigm. Specifically, the excessive learning capability allows DNNs to learn features about the backdoor trigger, while the DNNs can shrink the distance between poisoned samples in the feature space and connect the learned trigger-related features with the target label by the end-to-end supervised training. Based on this understanding, we propose to decouple the end-to-end training process for the backdoor defense. Specifically, we treat the DNNs as two disjoint parts, including a feature extractor (i.e., backbone) and a simple classifier (i.c., the remaining fully connected layers). We first learn the purified fea- ture extractor via sifp-uueerieed learning (Kolesnikov et al.) 2019 Chen et al., 2020a Jing & Tian, 2020) with unlabeled training samples (obtained by removing their labels), and then learn the sim- ple classifier via standard supervised training process based on the learned feature extractor and all training samples. The strong data augmentations involved in the self-supervised learning damage trigger patterns, making them unlearnable during representation learning; and the decoupling pro- cess further disconnects trigger patterns and the target label. Accordingly, hidden backdoors cannot be successfully created even the model is trained on the poisoned dataset based on our defense.
Moreover, we further reveal that the representation of poisoned samples generated by the purified extractor is significantly different from those generated by the extractor learned with standard train- ing process. Specifically, the poisoned sample lies closely to samples with its ground-truth label instead of the target label. This phenomenon makes the training of the simple classifier similar to label-noise learning (Wang et al., 2019b] Ma et al. 2020; Berthon al., 2021). As such, we first filter high-credible training samples (i.e., training samples that are most probably to be benign) and then use those samples labeled samples and the remaining part to form unlabeled samples to
fine-tune the whole model via supirvupee learning (Rasmus et al. 2015 Berthelot et al.. 2019 Sohn et al. 2020). This approach is to further reduce the adverse effects of poisoned samples.
The main contributions of this paper are three-fold. (1) We reveal that the backdoor is embedded in the feature space, which is mostly due to the end-to-end supervised training paradigm. (2) Based on our understanding, we propose a decoupling- based backdoor defense (DBD) to alleviate the threat of poisoning-bau based backdoor attacks. (3) Experiments on classical benchmark datasets are conducted, which verify the effectiveness of our defense.
2
RELATED WORK
2.1 BACKDOOR ATTACK
Backdoor attack is an emerging research area, which raises security concerns about training with third-party resources. In this paper, we focus on the poisoning-based backdoor attack towards image classification, where attackers can only modify the dataset instead of other training components
(e.g.. training loss). This threat could also happen in other tasks (Xiang et al. 2021;
Zhai et al.
2021 Li et al., 2022) and with different attacker's 's capacities (Nguyen & Tran 2020 Tang et al,,
2020 Zeng al. 2021a)) which are out-of-scope of this paper. In general, existing attacks can be divided into two main categories based on the property of target labels, as follows:
Poison-Label Backdoor Attack. is currently the most common attack paradigm, where the target label is different from the ground-truth label of poisoned samples. BadNets (Gu et al.] 2019) is the first and most representative poison-label attack. Specifically, it randomly selected few samples from the original benign dataset to generate poisoned samples by stamping the backdoor trigger onto the (benign) image and change their label with an attacker-specified target label. Those gener- ated poisoned samples associated with remaining benign ones were combined to form the poisoned training dataset, which will be delivered to users. After that, (Chen et al., 2017) suggested that the poisoned image should be similar to its benign version for the stealthiness, based on which they proposed the blended attack. Recently, (Xue et al.] 2020 Li et al., 20206] (021e) further explored how to conduct poison-label backdoor attacks more stealthily. Most recently, more stealthy and effective attack, the WaNet (Nguyen & Tran 2022), was proposed. WaNet adopted image warping as the backdoor trigger, which deforms but preserves the image content.
Clean-Label Backdoor Attack. Although the poisoned image generated by poison- attacks could be similar to its benign version, users may still notice the attack by examining the image-labe relationship. To address this problem, Turner et al. (2019) proposed the clean-label label attack paradigm, where the target label is consistent with the ground-truth label of poisoned samples. Specifically,
2

---

Published as a conference paper at ICLR 2022
they first leveraged adversarial perturbations or generative models to modify some benign images from the target class and then conducted the standard trigger injection process. This idea was gener- alized to attack video classification in (Zhao et al., 2020b), where they adopted the targeted universal adversarial perturbation (Moosavi-Dezfoolli et al. 2017) as the trigger pattern. Although clean-label backdoor attacks are more stealthy compared with poison-label ones, they usually suffer from rela- tively poor performance and may even fail in creating backdoors (Li et al. 2020c).
2.2 BACKDOOR DEFENSE
Currently, there are also some approaches to alleviate the backdoor threat. Existing defenses are mostly empirical, which can be divided into five main categories, including (1) detection- de-
fenses (Xu et al. 2 Zeng et al.
aa Xiang et al. 2022). (2) preprocessing based defenses
(Doan et al.
2020; Li et al. 2021b
Zeng et al. b02Tb),
(3) model reconstruction based defenses
Zhao et al.
2020a Li et al. 202 la Zeng et al. 2022 (4) trigger synthesis based defenses (Guo
ct al.
2020:
Dong et al., 2021; Shen et al. 2021), and (5) poison suppression based defenses (Du
al. 2020
Borgnia et al., 2021 Specifically, detection-based defenses examine whether sus-
picious DNN or sample is attacked and it will deny the use of malicious objects; Preprocessing based methods intend to damage trigger patterns contained in attack samples to prevent backdoor activation by introducing preprocessing module before feeding images into DNNs; Model recon- struction based ones aim at removing the hidden backdoor in DNNs by modifying models directly; The fourth type of defenses synthesize potential trigger patterns at first, following by the second stage that the hidden backdoor is eliminated by suppressing their effects; The last type of methods depress the effectiveness of poisoned samples during the training process to prevent the creation of hidden backdoors. general, our method is most relevant to this type of defenses.
In this paper, we only focus on the last four types of defenses since they directly improve the ro- bustness of DNNs. Besides, there were also few works focusing on certified backdoor defenses (Wang et al. 2020a Weber et al. (020)). Their robustness is theoretically guaranteed under certain assumptions, which cause these methods to be generally weaker than empirical ones in practice.
2.3 SEMI-SUPERVISED AND SELF-SUPERVISED LEARNING
Semi-supervised Learning. In many real-world applications, the acquisition of labeled data often relies on manual labeling, which is very expensive. In contrast, obtaining unlabeled samples is much easier. To utilize the power of unlabeled samples with labeled ones simultancously. a great amount of sumi-supervised learning methods were proposed (Gao al. 2017 Berthelot al. al.] 2019 Van Engelen& & Hoos 2020). Recently, -upervisd learning was also introduced in improving the security of DNNs (Stanforth et af.) 2019 Carmon et al.] 2019), where they utilized unlabelled samples in the adversarial training. Most recently, Yann et al. [222T) discussed how to backdoor semi-supervised learning. However, this approach needs to control other training components (e.g.. training loss) in addition to modifying training samples and therefore is out-of-scope of this paper. learning for backdoor defense remains blank.
How to adopt
Self-supervieed Learning. This learning paradigm is subset of unsupervised learning, where DNNs are trained with supervised signals generated from the data itself (Chen et al.) 2020a Grill
et al.
2020
2021). It has been adopted for increasing adversarial robustness (Hendrycks 2019 Wu et al. 2021 Shi et al., 2021). Most recently, there were also a few works (Saha
Liu et al.
et al.
2021 Carlini & Terzis [021] Jia et al. 2021) exploring how to backdoor self-supervised
learning. However, these attacks are out-of-scope of this paper since they need to control other training components (e.g., training loss) in addition to modifying training samples.
3 REVISITING BACKDOOR ATTACKS FROM THE HIDDEN FEATURE SPACE
In this section, we analyze the behavior of poisoned samples from the hidden feature space of at- tacked models and discuss its inherent mechanism.
Settings. We conduct the BadNets (Gu et al. 2019) and label-consistent attack (Turner et al. 2019) on CIFAR-10 10 dataset (Krizhevsky) 2009) for the discussion. They representative of poison-laber attacks and clben-label attacks, respectively. Specifically, we conduct supervised learning on the poisoned datasets with the standard training process and self-supervised learning the unlabelled
3

---

Published as a conference paper at ICLR 2022
30
20
10
-10
-10
-30
-20
-30
30
-40
-30
(8)
(b)
(c)
(d)
Figure 1: The t-SNE of poisoned samples in the hidden space generated by different models. (a)- (b): DNNs trained with supervised learning. (c)-(d): DNNs trained with self-supervised learning. (a))(c): DNNs under BadNets attack. (d) DNNs under label-consistent attack. No matter under the BadNets or label-consistent attack, poisoned samples tend to cluster together in the hidden space generated by DNNs trained with supervised learning, whereas lie closely to samples with their ground-truth label by those trained with self-supervised learning.
poisoned datasets with SimCLR (Chen al. 2020a). We visualize poisoned samples in the hidden feature space generated by attacked DNNs based on the t-SNE (Van der Maaten & Hinton 2008
More detailed settings are presented in Appendix A
Results. As shown in Figure ((a) [(b) poisoned samples (denoted by "black-cross') tend to cluster together to form separate cluster after the standard supervised training process, no matter under the poison-labed attack or elean-label attack. This phenomenon implies why existing poisoning -based based backdoor attacks can succeed. Specifically, the excessive learning capability allows DNNs to learn features about the backdoor trigger. Associated with the end-to-end supervised training paradigm, DNNs can shrink the distance between poisoned samples the feature space and connect the learned trigger-related features with the target label. In contrast, as shown in Figure [(c) [(d) poisoned samples lie closely to samples with their ground-truth label after the self-supervised training process on the unlabelled poisoned dataset. It indicates that can prevent the creation of backdoors by self-supervised learning, which will be further introduced the next section.
4 DECOUPLING-BASED BACKDOOR DEFENSE
4.1 PRELIMINARIES
General Pipeline of Backdoor Attacks. Let D {(x, v:)) denotes the benign training set, where E X' (0, 1..... 255] [x5]>xxx111 is the image, y/ E y== f0, 1, 1...., K} is its label, K is the number of classes, and l/ E y indicates the targer label. How to generate the poisoned dataset Dp is
the cornerstone of backdoor attacks. Specifically, D, consists of two subsets, including the modified
version of a subset of 'D and remaining benign samples, i.e., Dp = Dm U Dr, where DB C D,
y [D] is the poisoning rale, D,, ((w', Mt) )a) G(x), (æ, y) E D\Ds), and G X
(Dml
poisoned image generator. For example, G(x) = (1 - 2) & x & where an 10x t E X is the trigger pattern, and @ is the element-wiss product in the blended attack (Chen et al. 2017). Once D,, is generated, will be sent to users who will train DNNs on it.
A
Hidden backdoors will be created after the training process.
Threat Model. In this paper, we focus on defending against poisoning-basee based backdoor attacks. The attacker can arbitrarily modify the training set whereas cannot change other training components (e.g., model structure and training loss). For our proposed defense, we assume that defenders can fully control the training process. This is the scenario that users adopt third-party collected samples for training. Note that we do not assume that defenders have a local benign dataset, which is often required in many existing defenses (Wang al., 2019a] Zhao al. 2020a Li al., al.]a 2021a).
Defender's Goals. The defender's 's goals are to prevent the trained DNN model from predicting poisoned samples as the target label and to preserve the high accuracy on benign samples.
4.2 OVERVIEW OF THE DEFENSE PIPELINE
In this section, describe the general pipeline of our defense. As shown in Figure it consists of three main stages, including (1) learning a purified feature extractor via sulf-super learning, (2) filtering high-credible samples via label-noise learning, and (3) semi-supervised fine-tuning.
4

---

Published as a conference paper at ICLR 2022
Labeled High-crodiile Sumples
Training Images
Feature Estractor
Truining Sumples
Purifled Foaure Extractor
Simple Classifier
Bind
Purified DNN
Bird
Deg
Deg
train
filter
Eine tune
Dog
Dog
Dog
Uniabeled Low-credtile Sumples
Stuge
Stage 2
Stuge 3
Figure 2: The main pipeline of our defense. In the first stage, we train the whole DNN model via sepersueer learning based label-removed training samples. In the second stage, freeze the learned feature extractor and adopt all training samples to train the remaining fully connected layers via supervised learning. After that, filter high-credible samples based the training loss. In the third stage, we adopt high-credible samples as labeled samples and remove the labels of all low-credible samples to fine-tune the whole model via semi-supervised learning.
Specifically, in the first stage, we remove the label of all training samples to form the unlabelled dataset, based on which train the feature extractor via self-supervised learning. In the second stage, we freeze the learned feature extractor and adopt all training samples to train the remaining fully connected layers via supervised learning. We then filter a% high-credible samples based on the training loss. The smaller the loss, the more credible the sample. After the second stage, the training set will be separated into two disjoint parts, including high-credible samples and low- credible samples. We use high-credible samples low-credible samples to fine-tune the whole model via semi-supervised learning. More detailed information about each stage of our method will be further illustrated in following sections.
labeled samples and remove the label of all
4.3 LEARNING PURIFIED FEATURE EXTRACTOR VIA SELF-SUPERVISED LEARNING
Let D, denotes the training set and fw X + [o, 1 indicates the DNN with parameter w = [we, wy), where w, and wy indicates the parameters of the backbone and the fully connected layer, respectively. In this stage, we optimize w, based on the unlabeled version of Dr via self-supervised learning, as follows:
Σ
: = arg min
W.
(a.y)ED:
where (.).) indicates the self-supervised loss (e.g., NT-Xent in SimCLR (Chen et al., 2020a )
Through the self- supervupe learning, the learned feature extractor (i.e., backbone) will be purified even if the training set contains poisoned samples, as illustrated in Section 3
4.4 FILTERING HIGHICCEDIBLE SAMPLES VIA LABEL-NOISE LEARNING
Once w is obtained, the user can freeze and adopt D; to further optimize remaining wy, i.c.,
w arg min wy
Σ
L2 (x)
(T.
where C2(-) indicates the supervised loss (e.g., cross entropy).
After the decoupling-baat based training process [ even if the model is (partly) trained on the poi- soned dataset, the hidden backdoor cannot be created since the feature extractor is purified. However, this simple strategy suffers from two main problems. Firstly, compared with the one trained via su- pervised learning, the accuracy of predicting benign samples will have a certain decrease, since the learned feature extractor frozen in the second stage. Secondly, poisoned samples will serve as "outliers" to further hinder the learning of the second stage when poison-label attacks appear, since those samples lie close to samples with its ground-trute label instcad of the target label in the hidden feature space generated by the learned purified feature extractor. These two problems indicate that we should remove poisoned samples and retrain or fine-tune the whole model.
Specifically, select hieh-credible samples Dr based the loss L2(: w/)). The high- credible samples defined the a% training samples with the smallest loss, where a E [0, 100] is
5

---

Published as a conference paper at ICLR 2022
0.3
benign poisoned
Normalized Loss
0.6
Nommlized
0.2
benign poisoned
0.4
0.1
0.2
0.0
0.0
0
10
50
60
70
80
10
20
30
60
70
80
100
Epoch
Epoch
(a) Cross-Entropy
(b) Symmetric Cross-Entropy
Figure 3: Loss values of models under BadNets attack with 20% poisoning rate trained CIFAR- 10 dataset with the symmetric cross-entropy (SCE) and cross-entropy (CE) in the second stage. All loss values normalized [0, 1]. As shown in the figure, adopting SCE significantly increase the loss differences between poisoned samples and benign ones compared with the CE.
In particular, adopt the symmetric cross-entropy (SCE) (Wang al. 2019b) as (-)., inspired by the label-noise learning. As shown in Figure 3] compared with the CE loss, the SCE significantly increase the differences between poisoned samples and benign ones, which further reduces the possibility that high-credible dataset D, still contains poisoned samples.
Note that do not intend accurately separate poisoned samples and benign samples. We only want to ensure that the obtained Dr, contains as few poisoned samples as possible.
4.5 SEMI-SUPERVISER FINE-TUNING
After the second stage, the third-party training set D. will be separated into two disjoint parts, includ- ing the high-credible dataset Dr and the low-credible dataset Dr & DD Let D, & {x|(x, y) E Di} indicates the unlabeled version of low-credible dataset Dj. We fine-tune the whole trained model w with semi-supervised learning as follows:
min L. (Dn, Di;
where La(-) denotes the cemi-supervised loss (e.g., the loss in MixMatch (Berthelot et al. 2019
This process can prevent the side-effects of poisoned samples while utilizing their contained useful information, and encourage the compatibility between the feature extractor and the simple classifier via learning them jointly instead of separately. Please refer to Section 5.3 for more results.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets and DNNs. We evaluate all defenses on two classical benchmark datasets, including CIFAR- Krrihevskyy 2009) and (a subset of) ImageNet (Deng et al. 2009). We adopt the ResNet- 18 (He et al. (2016) for these tasks. More detailed settings are presented in Appendix B. Besides, we also provide the results on (a subset of) VGGFace2 (Cao et al. 2018) in Appendix
Attack Baselines. We examine all defense approaches in defending against four representative at- tacks. Specifically, we select the BadNets (Gu et al.) 2019), the backdoor attack with blended strat- egy (dubbed 'Blended') (Chen et al., 2017). WaNet (Nguyen & Tran (021). and label-consistent attack with adversarial perturbations (dubbed JabbllCooniittnt") (Turner et al. 2019) for the eval- uation. They the representative of patch-based visible and invisible poison- label attacks, patch-based paison-label attacks, and clean-label attacks, respectively.
Defense Baselines. We compared DBD with two defenses having the same defender's capaci- ties, including the DPSGD (Du et al., (020)) and ShrinkPad (Li et al. (2021b). We also compare with other approaches with additional requirement (i.e., having local benign dataset), including

---

Published as a conference paper at ICLR 2022
BadNets
Blended
WaNet
Consistent
Poisoned
Image
Trigger
Pattern
Figure 4: The illustration of poisoned samples generated by different attacks.
the neural cleanse with unlearning strategy (dubbed 'NC') (Wang et al. 2019a), and neural attention distillation (dubbed 'NAD') (Li et al. 2021a). They are the representative of poison suppression based defenses, preprocessing based defenses, trigger synthesis based defenses, and model recon- struction based defenses, respectively. We also provide results of DNNs trained without any defense (dubbed 'No Defense') as another important baseline for reference.
Attack Setups. We use 2 x 2 square as the trigger pattern on CIFAR- 10 dataset and the 32 x 32 Apple logo on ImageNet dataset for the BadNets, as suggested in (Gu et al.] 2019; Wang et al.) 2019a). For Blended, we adopt the 'Hello Kitty' pattern on CIFAR-10 and the random noise pattern on ImageNet, based on the suggestions in (Chen et al. (017), and set the blended ratio À 0.1 on all datasets. The trigger pattern adopted in label-consistent attack is the same the one used in BadNets. For WaNet, we adopt its default settings on CIFAR- 10 dataset. However, on ImageNet dataset,
different settings optimized by grid-search since the original ones fail. An example of poisoned samples generated by different attacks is shown in Figure 4 Besides, we set the poisoning rate ?1 2.5% for label-consistent attack (25% of training samples with the target label) and B.2
722 = 5% for three other attacks. More details are shown in Appendix
Defense Setups. For our DBI), we adopt SimCLR (Chen et al. 2020a) as the self-supervised method and MixMatch (Berthelot et al. 2019) as the semi-supervised method. More details about SimCLR and MixMatch are in Appendix The filtering rate is the only key hyper-parameter which is set to 50% in all cases. We set the shrinking rate to 10% for the ShrinkPad on all datasets, as suggested @Li al. 20216 Zeng et al., 202Tb).
DBD,
particular, DPSGD and NAD are sensitive to their hyper- parameters. We report their best results in each case based on the grid-searce (as shown Appendix D). Besides, we split 5% random subset of the benign training set as the local benign dataset for NC and NAD. More implementation details are provided in Appendix B.3
hyper-parameters.
Evaluation Metrics. We adopt the attack success rate (ASR) and benign accuracy (BA) to measure the effectiveness of all method Specifically, let Drest indicates the (benign) testing set and Cw X + 3' denotes the trained classifier, we have ASR Pr(a. D)) (G(x)) = yı ly * ye and BA & Pr(s ()) (Cw(w:) = y}. where y/ is the target label and G(-) the poisoned image generator. In particular, the lower the ASR and the higher the BA, the better the defense.
5.2 MAIN RESULTS
Comparing DBD with Defenses having the Same Requirements. As shown in Table D2 DBD is significantly better than defenses having the same requirements (i.c., DPSGD and ShrinkPad) in defending against all attacks. For example, the benign accuracy of DBD is 20% over while the attack success rate is 5% less than that of DPSGD in all cases. Specifically, the attack success rate of models with DBD is less than 2% in all cases (mostly < 0.5%), which verifies that our method can successfully prevent the creation of hidden backdoors. Moreover, the decreases of benign accuracy are less than 2% when defending against poison- label attacks, compared with models trained without any defense. Our method is even better relatively larger dataset where all baseline methods become less effective. These results verify the effectiveness of our method.
Among all defense methods, the one with the hest performance is indicated in holdface and the value with underline denotes the second-best result.
7

---

Published as a conference paper at ICLR 2022
Table 1: The effectiveness (%) of defending against three attacks. Note that NC and NAD need an additional local benign dataset, which is not required in DPSGD, ShrinkPad, and our method.
Dataset -
CIFAR-10 10
ImageNet Blended
Attack +
BadNets
Blended
WaNet
BadNets
WaNet
Defense +
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
No Defense
94.92
100
94.14
98.25
94.29
98.64
80.23
90.49
82.07
95.37
80.98
26.22
NC
94.25
2.33
93.01
5.70
92.44
98.52
80.67
32.22
82.07
27.33
80.32
99.32
NAD
90.47
1.20
89.72
1,51
91.83
9.49
73.49
5.81
70.88
8.15
75.27
31.43
DPSGD
44.24
14,56
53.05
20.68
52.13
39.78
51.43
25.40
55.12
84.17
20.72
54.12
ShrinkPad
90.32
10.34
79.67
55.71
74.76
85.59
66.08
45.61
64.47
4.20
63.74
3.58
DBD (ours)
92.41
0.96
92.18
1.73
91.20
0.39
80.99
0.26
81.63
0.16
82.05
0.33
Table 2: The effectiveness (%) of defending against label-consistent attack on CIFAR-I0 dataset.
Attac1, Defense-+
Metrie|
Benign
NC
NAD
DPSGD
ShrinkPad
DBD (ours)
Label-Consistee
BA
94.90
94.94
93.04
44.28
90.67
89.67
(c 16)
ASR
99.33
1.31
3.95
7.97
9.60
0.01
ConsisConsitt
BA
94.82
94.57
90.98
69.79
90.83
91.45
(c (==30 32)
ASR
99.19
1.36
6.54
9.06
13.04
0.34
Table 3:
The ablation study of our proposed method.
Attack +
BadNets
Blended
Label-Consistee
WaNet
Defense J. Metric +
BA
ASR
BA
ASR
BA
ASR
BA
ASR
No Defense
94.92
100
94.14
98.25
94.82
99.19
94.29
98.64
DBD without SS
93.66
100
93.47
99.93
90.70
98.50
81.91
98.40
SS with CE
82.25
5.20
81.85
12.19
82.08
5.87
80.29
9.48
SS with SCE
82.34
5.12
82.30
6.24
81.81
5.43
81.15
7.08
SS with SCE Tuning
78.94
4.02
78.78
3.70
77.44
1.44
78.51
5.87
DBD (ours)
92.41
0.96
92.18
1.73
91.45
0.34
91.20
0.39
'DBD without SS: use standard supervised training to obtain the feature extractor adopted in DBD.
2$S with CE: freeze the learned backbone and train remaining FC layers all samples with CE loss.
SSS with SCE: freeze the Iearned backbone and train remaining FC layers on all samples with SCE loss.
"SS with SCE+ + Tuning: fine-tune FC layers of the model in 'SS with SCE' on high-credible samples.
Comparing DBD with Defenses having Extra Requirements. We also compare our defense with two other methods (i.e., NC and NAD), which have an additional requirement that defenders have a benign local dataset. As shown in Table [2] NC and NAD are better than DPSGD and ShrinkPad, as we expected, since they adopt additional information from the benign local dataset. In particular, although NAD and NC use additional information, our method is still better than them, even when their performances are tuned to the best while our method only uses the default settings. Specifically, the BA of NC is par with that of our method. However, it is with the sacrifice of ASR. Especially on ImageNet dataset, NC has limited effects in reducing ASR. In contrast, our method reaches the smallest ASR while its BA is either the highest the second-highest in almost all cases. These results verify the effectiveness of our method again.
Results. As shown in Figure 7 our method can still prevent the creation of hidden backdoors even when the poisoning rate reaches 20%. Besides, DBD also maintains high benign accuracy. In other words, our method effective defending attacks with different strengths.
5.3 ABLATION STUDY
There are four key strategies in DBD, including (1) obtaining purified feature extractor, (2) using SCE instead of CE in the second stage, (3) reducing side-effects of low-credible samples, and (4) fine-tuning the whole model via cupe-supe learning. Here we verify their effectiveness.
Settings. We compare the proposed DBD with its four variants, including (1) DBD without SS, (2) SS with CE, (3) SS with SCE, and (4) SS with SCE + Tuning, on the CIFAR-10 dataset. Specifically, in the first variant, we replace the backbone generated by self-supervised learning with the one trained a supervised fashion and keep other parts unchanged. In the second variant, we freeze the backbone learned via self-supervised learning and train the remaining fully-connected layers with cnoss-yntropy loss on all training samples. The third variant is similar to the second one. The only difference is that it uses symmetric cross-entropy instead of cross-entropy to train conneccd layers. The last variant is an advanced version of the third one, which further fine-tunes fully- connected layers high-credible samples filtered by the third variant.
8

---

Published as a conference paper at ICLR 2022
Results. As shown in Table E] can conclude that decoupling the original end-to-end supervised training process is effective in preventing the creation hidden backdoors, by comparing our DBD with its first variant and the model trained without any defense. Besides, can also verify the effectiveness of SCE loss on defending against poison-label backdoor attacks by comparing the second and third DBD variants. Moreover, the fourth DBD variant has relatively lower ASR and BA, compared with the third one. This phenomenon is due to the removal of low-credible samples, It indicates that reducing side-effects of low-credible samples while adopting their useful information is important for the defense. We can also verify that fine-tuning the whole model via semi-supervised learning is also useful by comparing the fourth variant and the proposed DBD.
5.4 RESISTANCE TO POTENTIAL ADAPTIVE ATTACKS
In our paper, we adopted the classical defense setting that attackers have information about the defense. Attackers may design adaptive attacks they know the existence of our DBD. The most straightforward idea is to manipulate the self-supervised tranning process that poisoned samples are still in a new cluster after the self-supervised learning. However, attackers are not allowed to do it based on threat model about adopting third-party datasets. Despite this, attackers may design adaptive attacks by optimizing the trigger pattern to make poisoned samples still in a new cluster after the sulf-super learning if they can know the model structure used by defenders, as follows:
Problem Formulation. For a K -cllassification problem, let X" = { indicates the benign images selected for poisoning, X, (ar,), denotes the benign images with ground-truth label j. poisoned image generator G, the adaptive
and is a trained backbone. Given an attacker-predefing attack aims to optimize trigger pattern t by minimizing the distance between poisoned images while maximizing the distance between the center of poisoned images and centers of clusters of benign images with different label, i.c.,
K
Σ )) 1
min &
Σ d d
(4)
M
K
#EX"
where o' A 1 EvEx g(G((x)))
N. + EmEx g(x), and is distance metric.
Settings. We adopt the CIFAR-10 10 dataset and use the 12 norm as the distance metric to conduct the experiment. Specifically, we assume that attackers have the entire benign dataset, based on which they can train a backbone adopted in the first stage of our DBD. We use the Adam optimizer to solve the above optimization problem for 100 epochs with learning rate of 0.1. The trigger size set to 32 x2, which means the attacker can completely modify the content of poisoned samples, regardless of its original semantic information and the stealthiness of the attack. This setting to ensure the attack ability, since clustering poisoned samples together is very difficult in self sulfrsuuee learning.
Results. The adaptive attack works well when there is no defense (BA=94,9653 ASR=99.70%%
However, this attack still fails to attack our DBD (BA=93.219% % ASR=1 In other words, our defense is resistant to this adaptive attack. on the backbone is far less effective when the model is retrained since model parameters are changed due to the random initialization and the update of model weights during the training process.
most probably because the trigger optimized based
CONCLUSION
The mechanism of bissoning-based backdoor attacks is to establish a latent connection between trigger patterns and the target label during the training process. In this paper, we revealed that this connection is learned mostly due to the end-to-end supervised training paradigm. Motivated by this understanding, we proposed decoupling based backdoor defense, which first learns the backbone via self-supervised learning and then the remaining tully-connected layers by the classical super- vised learning. We also introduced the label-noise learning method to determine high-credible and low-credible samples, based on which
fine-tuned the whole model via semi-supervised learn- ing. Extensive experiments verify that our defense is effective on reducing backdoor threats while preserving high accuracy on predicting benign samples.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
Baoyuan Wu is supported in part by the National Natural Science Foundation of China under Grant 62076213, the University Development Fund of the Chinese University Hong Kong. Shenzhen under Grant 01001810, and the Special Project Fund of Shenzhen Research Institute of Big Data under Grant T00120210000 Zhan Qin is supported in part by the National Natural Science Foun- dation of China under Grant U20A20178, the National Key Research and Development Program of China under Grant 2020AAA0107705, and the Research Laboratory for Data Security and Pri- vacy, Zhejiang University-Anr Financial Fintech Center. Kui Ren is supported by the National Key Research and Development Program of China under Grant 2020AA
ETHICS STATEMENT
DNNs are widely adopted in many mission-critical areas (e.g., face recognition) and therefore their security of great significance. The vulnerability of DNNs to backdoor attacks raises serious con- cerns about using third-pan harrd-patty training resources. In this paper, propose general training pipeline to obtain backdoor-fi free DNNs, even if the training dataset contains poisoned samples. This work has ethical issues in general since method is purely defensive and does not reveal any vulner- abilities of DNNs. However, we need to mention that our defense can be adopted only when training with untrusted samples, and backdoor attacks could happen in other scenarios. People should not be too optimistic about eliminating backdoor threats.
REPRODUCIBILITY STATEMENT
The detailed descriptions of datasets, models, and training settings in Appendix A We also describe the computational facilities and cost in Appendix [K] Codes of our DBD are also open- sourced.
REFERENCES
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 2019.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores
make instance-dependent label-noise learning possible. In ICML, 2021.
Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor atlacks without accuracy tradeoff. In ICASSP. 2021.
Qiong Cao, Shen, Weidi Xie, Omkar Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In IEEE FG, 2018.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. arXiv preprint (tt::2/00609666. 2021.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. In NewrlPS, 2019.
David M Chan. Roshan Rao, Forrest Huang, and John F Canny. Gpu accelerated t-distributed stochastic neighbor embedding. Journal of Parallel and Distributed Computing, 131:1-13, 2019.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In AAAI Workshop. 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI. 2020a.
10

---

Published as a conference paper at ICLR 2022
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprini arXiv: 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Bao Gia Doan, Ehsan Abbasnejad, and Damith Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. In ACSAC, 2020.
Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang. Zihao Xiao, Hang Su, and Jun Zhu, Black- box detection of backdoor attacks with limited information and data. In ICCV, 2021.
Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. In ICLR, 2020.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ô Arik, Larry S Davis, and Tomas Pfister. Consistency- based semi-supervised active learning: Towards minimizing labeling cost. In ECCV, 2020.
Yuan Gao, Jiayi Ma, and Alan L Yuille. Semi-supervised sparse representation based classification for face recognition with insufficient labeled samples. IEEE Transactions on Image Processing, 26(5):2545-255 2560, 2017.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In NeurIPS, 2020.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access,
2019.
Wenbo Guo, Lun Wang, Yan Xu, Xinyu Xing, Min Du, and Dawn Song. Towards inspecting and eliminating trojan backdoors in deep neural networks. In ICDM, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. NeurIPS, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In IEEE S&P, 2021.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In CVPR, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Ang Li, Shanshan Zhao, Xingjun Ma, Mingming Gong, Jianzhong Qi, Rui Zhang, Dacheng Tao, and Ramamohanarao Kotagiri. Short-term and long-term context aggregation network for video inpainting. In ECCV, 2020a.
11

---

Published as a conference paper at ICLR 2022
Shaofeng Li, Minhui Xue, Benjamin Zhao, Haojin Zhu, and Xinpeng Zhang. Invisible backdoor attacks on deep neural networks via steganography and regularization. IEEE Transactions on Dependable and Secure Computing, 2020b.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR, 2021a.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning; A survey. arXiv preprint N04V:2007.008755 2020c.
Yiming Li, Tongqing Zhai, Yong Jiang. Zhifeng Li, and Shu-Tao Xia. Backdoor attack in the physical world. In ICLR Workshop, 2021b.
Yiming Li, Haoxiang Zhong, Xingjun Ma, Yong Jiang, and Shu-Tao Xia. Few-shot backdoor attacks on visual object tracking. ICLR, 2022.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, 2021c
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data En- gineering, 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2016.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Nor- malized loss functions for deep learning with noisy labels. In ICML, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In ICLR, 2018
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In CVPR, 2017.
Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurlPS, 2020.
Anh Nguyen and Anh Tran. Wanet-imperceptible warping-based backdoor attack. In ICLR, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. In NeurIPS, 2019.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In NewrIPS, 2015.
Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Back- door attacks on self-supervised learning. arXiv preprint arXiv:2105. 2021.
Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. In ICML, 2021.
Changhao Shi, Chester Holtz, and Gal Mishne. Online adversarial purification based on self- supervision. In ICLR, 2021.
12

---

Published as a conference paper at ICLR 2022
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying supervupe learning with consistency and confidence. In NeurIPS, 2020.
Robert Stanforth, Alhussein Fawzi, Pushmeet Kohli, et al. Are labels required for improving adver- sarial robustness? In NeurlPS, 2019.
Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. In KDD, 2020.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. NeurIPS, 2018.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02777, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learn- ing, 109(():373 2020.
Binghui Wang, Xiaoyu Cao, Neil Zhenqiang Gong, et al. On certifying robustness against backdoor attacks via randomized smoothing. In CVPR Workshop, 2020a.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks neural networks. In JEEE S&P, 2019a.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV, 2019b.
Yue Wang, Alireza Fathi, Abhijit Kundu, David A Ross, Caroline Pantofaru, Tom Funkhouser, and Justin Solomon. Pillar-based object detection for autonomous driving. In ECCV, 2020b.
Maurice Weber, Xiaojun Xu, Bojan Karlas, Ce Zhang, and Bo Li. Rab: Provable robustness against backdoor attacks. arXiv preprint urXiv:2003.08904 2020.
Xin Wen, Biying Li, Haiyun Guo, Zhiwei Liu, and Guosheng Hu. Adaptive variance based label distribution learning for facial age estimation. In ECCV, 2020.
Haibin Wu, Xu Li. Andy T Liu, Zhiyong Wu, Helen Meng, and Hung-yi Lee. Adversarial defense for automatic speaker verification by cascaded self-supervised learning models. In ICASSP, 2021.
Zhen Xiang, David Miller, Siheng Chen, Xi Li, and George Kesidis. A backdoor attack against 3d point cloud classifiers. In ICCV, 2021.
Zhen Xiang, David J. Miller, and George Kesidis. Post-training detection of backdoor attacks for two-class and multi-attack scenarios. ICI.R, 2022.
Xiaojun Xu, Qi Wang, Huichen Li. Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In JEEE S&P, 2021.
Mingfu Xue, Can He, Jian Wang, and Weiqiang Liu. One-to-n & n-to-one: Two advanced backdoor attacks against deep learning models. IEEE Transactions Dependable and Secure Computing, 2020.
Zhicong Yan, Gaolei Li, Yuan Tlan, Jun Wu, Shenghong Li, Mingzhe Chen, and H Vincent Poor: Dehib: Deep hidden backdoor attack on semi-supervised learning via adversarial perturbation. In AAAI, 2021.
Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV, 2021a.
13

---

Published as a conference paper at ICLR 2022
Yi Zeng, Han Qiu, Shangwei Guo, Tianwei Zhang, Meikang Qiu, and Bhavani Thuraisingham. Deepsweep: An evaluation framework for mitigating dnn backdoor attacks using data augmenta- tion. In AsiaCCS, 2021b.
Yi Zeng, Si Chen, Won Park Z. Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In ICLR, 2022.
Tongqing Zhai, Yiming Li, Ziqi Zhang, Baoyuan Wu, Yong Jiang, and Shu-Tao Xia. Backdoor attack against speaker verification. ICASSP, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, 2018.
Pu Zhao, Pin- Yu Chen, Payel Das. Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity loss landscapes and adversarial robustness. In ICLR, 2020a.
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean- label backdoor attacks on video recognition models. In CVPR, 2020b.
14

---

Published as a conference paper at ICLR 2022
Table 4: Statisties of datasets and DNNs adopted in main experiments.
Dataset
Input Size
Classes
# Training Images
Test Images
DNN model
CIFAR- 10
3 x 32 x 32
10
50,000
10,000
ResNet-18 18
ImageNet
3 224 224
30
38,939
1,500
ResNet-18
A DETAILED SETTINGS FOR REVISITING BACKDOOR ATTACKS
Attack Setups. We conduct the BadNets (Gu et al. 2019) and label-consistent attack (Turner et al., 2019) with the target label y/ on the CIFAR- 10 dataset (Krizhevsky) 2009). The trigger patterns are the same as those presented in Section 5.2 In particular, we implement the label-consistent attack with adversarial perturbations, as suggested in its original paper (Turner et al. 2019). Specif- ically, we used the projected gradient descent (PGD) (Madry et al.) 2018) to generate adversarial perturbations within the 0o -bal where the maximum perturbation size e 16.
Training Setups. We conduct supervised learning on the poisoned datasets with the standard train- ing process and the self-supervised learning on unlabelled poisoned datasets with the SimCLR (Chen et al. 2020a). The supervised training is conducted based on the open-source code] Specif- ically, we use the SGD optimizer with momentum 0.9, weight decay of 5 x 10-4, and an initial learning rate of 0.1. The batch size is set to 128 and we train the ResNet-18 18 model 200 epochs. The learning rate is decreased by a factor of 10 at epoch 100 and 150, respectively. Besides, we add triggers before performing the data augmentation (e.g., random crop and horizontal flipping). For the self-supervised training, we use the stochastic gradient descent (SGD) optimizer with a momen- tum of 0.9, an initial learning rate of 0.4, and weight decay factor of 10-1 We use batch size of 512. and train the backbone for 1,000 epochs. We decay the learning rate with the cosine decay schedule (Loshchllov & Hutter 2016) without a restart. Besides, we also adopt strong data augmentation techniques, including random crop and resize (with random flip), color distortions, and Gaussian blur, as suggested (Chen al. 2020a). All models are trained until converge.
t-SNE Visualization Settings. We treat the output of the last residual unit as the feature representa- tion and use the tsne-cuda library (Chan al. [219) to get the feature embedding of all samples. To have a better visualization, we adopt all poisoned samples and randomly select 10% benign samples for visualizing models under the supervised learning, and adopt 30% poisoned samples and 10% benign samples for those under the supersspe learning.
B DETAILED SETTINGS FOR MAIN EXPERIMENTS
B. MORE DETAILS ABOUT DATASETS AND DNNS
Due to the limitations of computational resources and time, we adopt a subset randomly selected from the original ImageNet. More detailed information about the datasets and DNNs adopted in the main experiments of our paper is presented in Table
B.2 MORE DETAILS ABOUT ATTACK SETTINGS
Attack Setups. We conduct the BadNets (Gu et al. 2019), blended attack (dubbed 'Blended') (Chen et al. 2017), label-consistent attack (dubbed -oness-Conssstett) (Turner al. 2019), and WaNet (Nguyen & Tram 202I) with the target label y/ = 3 on all datasets. The trigger patterns are the same those presented in Section 5.2 In particular, set the blended ratio À 0.1 for the blended attack on all datasets and examine label-consistent attack with the maximum perturbation size (16. Besides, WaNet assumed that attackers can fully control the whole training process in its original paper. However, we found that WaNet only modified training data while other training components (c.g., training loss, training schedule, and model structure) are the same as those used in the standard training process. As such, we re-implement its code in the poisoning- based attack scenario based on its official code] Specifically, following the settings in its original paper, we set the noise rate Pn 0.2, control grid size k 4, and warping strength s 0.5 on
based_ _a____accooo______ Batlooor_Attcck-Pee
15

---

Published as a conference paper at ICLR 2022
Table The Effectiveness of WaNet with different kernel size &t when the noise rate to 0 and strength = 1 on the ImageNet dataset.
k:
128
224
BA
78.31
77.64
81.29
80.98
ASR
6,87
14.98
91.69
96.22
Table 6: The Effectiveness of WaNet with different strength when the noise rate is set to 0 and strength kernel size k 224 on the ImageNet dataset.
+
0.4
0.6
0.8
BA
81.01
81.45
81.32
80.98
ASR
80.06
92.22
94.23
96.22
Table The Effectiveness of WaNet with and without the noise mode when the kernel size 224 and strength = on the ImageNet dataset.
w/ noise mode
w/o noise mode
BA
79.37
80.98 96.22
ASR
31.72
the CIFAR-10 10 dataset. However, we found that the default it and are too small to make the attack works on the ImageNet dataset (as shown Table Besides, the 'noise mode" also significantly reduces the attack effectiveness (as shown in Table 7). As such, we set k = 224 and and train models without the noise mode on the ImageNet dataset.
Training Setups. On the CIFAR-10 dataset (Krizhevsky 2009). the settings are the same those described in Section[A] On the ImageNet dataset (Deng et al., 2009), we conduct experiments based on the open-source code" Specifically, we use the SGD optimizer with momentum 0.9, weight decay 10-4, and an initial learning rate of 0.1. The batch size is set to 256 and we train the ResNet- 18 model 90 epochs. The learning rate is decreased by factor of 10 at epoch 30 and 60, respectively. Besides, since the raw images in the ImageNet dataset are different sizes, we resize them to 3 224 224 before adding triggers.
B.3 MORE DETAILS ABOUT DEFENSE SETTINGS
Settings for NC. We conduct reverse engineering and anomaly detection based on its open-source code [5] We implement the 'unlearning' method patch attacked models, suggested in its paper (Wang et al. 2019a). We randomly select 5% benign training samples as the local benign dataset, which is used in the "unlearning" process. Unless otherwise specified, other settings the those used in (Wang et al.) 2019a).
Settings for NAD. We implement this method based its open-source code The origin NAD only conducted experiments on the WideResNet model. In our paper, we calculate the NAD loss over the last residual group for the ResNet-18. The local benign dataset is the same the one adopted in NC, which used in the fine-tuning and distillation process of NAD. Unless otherwise specified, other settings are the same as those used (Li et al. 202
Settings for DPSGD. The original DPSGD was conducted on the MNIST dataset implemented by the TensorFlow Framework. this paper, we re-implement it based on the differentially private SGD method provided by the Opacus]] Specifically. we replace the original SGD optimizer with the differentially private one, as suggested in (Du et al., 2220).. There are two important hyper- parameters in DPSGD, including noise scales and the clipping bound C. In the experiments, we set C and select the best a by the grid-search.
16

---

Published as a conference paper at ICLR 2022
BadNets
Blended
WaNet
Polsoned
Image
Trigger
Pattern
Figure 5: An example of poisoned samples generated by different attacks on VGGFace22 dataset.
Table 8: Statistics of the VGGFace2 dataset and model structure adopted in our experiments.
Dataset
Input Size
# Classes
# Training Images
# Test Images
DNN model
VGGFace2 3 x 224 224
30
9,051
2,263
DenseNet-12 121
Table Defending against attacks on VGGFace2 dataset. Note that NC and NAD need an additional local benign dataset, which is not required in DPSGD, ShrinkPad, and our method.
Attack +
BadNets
Blended
WaNet
Defense Metric
BA
ASR
BA
ASR
BA
ASR
No Defense
91.52
88.14
93.35
98.94
91.41
98.75
NC
92.16
18.82
93.43
86.76
91.32
95.09
NAD
87.29
9.49
85.27
10.25
80.69
4.41
DPSGD
60.83
6.61
81.90
93.63
54.04
30.85
ShrinkPad
77.64
61.50
79.18
75.61
78.62
30.13
DBD (ours)
89.74
0.17
93.29
0.09
89.32
Settings for ShrinkPad. We set the shrinking rate to 10% on all datasets, as suggested (Li et al. 2021b Zeng al.) 2021b). Following their settings, we pad O-pixels at the bottom right of the shrunk image to expand it to its original size.
Settings for our Defense. In this first stage, We adopt SimCLR (Chen et al. 2020a) to perform self-supervised learning. We train backhones 100 instead of 1,000 epochs to reduce computational costs while preserving effectiveness. Other settings the same those described in Section We use the same settings across all datasets, models, and attacks; In the second stage, we use the Adam optimizer with a learning rate 0.002 and the batch size to 128. We train the fully connected layers 10 epochs with the SCE loss (Wang et al. 2019b). Two hyper-pane parameters involved in the SCE (i.e., a and 3 in the original paper) are set to 0.1 and 1, respectively. After that, we filter 50% high-credible samples. We use the same settings across all datasets, models, and attacks; In the third stage, we adopt the MixMatch (Berthelot al. (019) for semi-supervised fine- fine-tuning with settings suggested in its original paper. Specifically, we use the Adam optimizer with a learning rate of 0.002, the batch size of 64, and finetune the model 190 epochs on the CIFAR- and 80 epochs on the ImageNet dataset, respectively. We set the temperature T = 0.5 and the weight of unsupervised loss X 15 on the CIFAR-10 and A = on the ImageNet dataset, respectively. Moreover, we re- re-filter high-credible samples after every epoch of the third stage based on the SCE loss.
C DEFENDING AGAINST ATTACKS ON VGGFACE2 DATASET
Dataset and DNN. Due to the limitations of computational resources and time, we adopt a subset randomly selected from the original VGGFace2 (Cao et al., 2018). More details are in Table
Settings for Attacks. For the training of models on the VGGFace2 dataset, the batch size set to 32 and we conduct experiments the DenseNet-122 121 model (Huang et al.) 2017). An example of poisoned samples generated by different attacks are in Figure Other settings are the same as those used on the ImageNet dataset.
Settings for Defenses. For NAD, we calculate the NAD loss over the second to last layer for the DenseNet- 121. Other settings the same as those used on the ImageNet dataset.
17

---

Published as a conference paper at ICLR 2022
Table 10: Results of DPSGD against the BadNets and blended attack with different noise scale d.
Dataset +
CIFAR-10
ImageNet
VGGFace2
Attack +
BadNets
Blended
BadNets
Blended
BadNets
Blended
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
89.78
100
90.03
92.64
56.04
97.01
55.12
84.17
85.96
99.91
85.34
97.68
0,005
89,38
100
89.48
91.15
55.67
95.44
55.92
85,99
84.64
99.91
84.97
97.29
0.01
88.85
100
88.67
91.50
57.19
93.10
55.46
84.79
83.12
99.87
81.90
93.63
0.05
81.04
100
80.85
81.03
55.04
87.26
54.22
86.06
60.83
6.61
58.02
98.48
0.1
69.77
100
68.10
72.47
51.43
25.40
50.12
89.35
43.55
17.80
43.26
100
0.3
52.48
99.99
53.05
20.68
39.33
7.49
36.97
87.13
17.59
22.08
13.32
53.20
0.5
44.24
14.56
45.26
14.19
32.26
14.46
31.60
84.23
10.17
48.90
9.27
53.50
0.7
40.37
21.23
39.71
15.49
29.06
21.46
25.38
88.69
8.74
41.55
6.55
61.50
35.16
32.27
36.51
29.24
23.70
31,30
22.27
82.23
6.64
31.59
6.46
21.43
Table 11: Results of DPSGD against the label-consistent attack with different noise scale
Poisoning Rate 1
0
0.005
0.01
0.05
0.1
0.3
0.5
0.7
Metric 4
BA
89.69
89.62
89.15
82.25
70.24
53.41
43.82
39.98
36.65
0.6%
16
ASR
88.79
89.00
88.16
4.30
6.68
8.80
3.51
7.97
16.04
BA
89.40
89.56
88.59
81.25
69.87
53.16
44.28
35.08
36.28
2.5%
ASR
97.95
98.51
98.57
99.42
80.32
69.04
7.97
13.98
16.27
BA
89.75
89.43
88.77
82.27
71.12
53.42
47.52
39.18
37.06
0.6%
32
ASR
98.29
99.02
98.38
9.23
5.76
13.60
6.29
15.64
14.70
BA
89.87
89.38
89.09
82.15
69.79
53.02
44.08
38.31
36.13
2.5%
ASR
99.69
98.89
99.57
99.86
9.06
18.88
6.62
20.97
7.32
Table 12: Results of DPSGD against the WaNet with different noise scale a.
Dataset
CIFAR-10
ImageNet
VGGFace2
4, Metrie
BA
ASR
BA
ASR
BA
ASR
0
87.76
87.06
53.26
96.06
83.62
95.22
0.005
88.21
84.71
53.38
98.14
80.72
96.47
88.13
82.05
54.03
97.72
79.88
83.57
0.05
81.03
80.59
51.48
97.
54.04
30.85
0.1
66.85
72.75
47.65
98.63
33.89
99.82
0.3
52.13
39.78
29.65
96.45
9.44
73.99
0.5
41.56
39.11
22.13
93.45
6.41
80.82
36.51
33.60
20.72
54.12
6.10
86.98
32.66
39.75
15.15
10
6.36
52.16
Results. As shown in Table 9 our defense still reaches the best performance even compared with NC and NAD. Specifically, the BA of NC is on par with that of our method whereas it is with the sacrifice of ASR. These results verify the effectiveness of our defense again.
D SEARCHING BEST RESULTS FOR DPSGD AND NAD
The effectiveness of DPSGD and NAD is sensitive to their their best results based on the criteria that 'BA- ASR' reaches the highest value after the defense.
Here we search for
D.1 SEARCHING BEST RESULTS FOR DPSGD
In general, the larger the a, the smaller the ASR while also the smaller the BA. The results of DPSGD are shown in Table To where the best results marked in boldface.
18

---

Published as a conference paper at ICLR 2022
Table 13: Results of the fine-tuning process in NAD against the BadNets and blended attack with different learning rates 7).
Datasct +
CIFAR-10
ImageNet
VGGFace2
Attack
BadNets
Blended
BadNets
Blended
BadNets
Blended
1) J. Metric +
ASR
BA
ASR
BA
ASR
BA
ASR
32.64
5.12
38.24
1.43
62.73
1.09
56.72
0.95
90.35
24.36
91.39
99.96
0.01 90.70
15.66
89.77
11.32
80.37
43.90
80.28
95.82
92.04
83.80
92.95
99.96
0.001
94.91
100
93.25
96.79
81.47
86.79
80.85
99.12
91.49
90.91
92.91
99.96
Table 14: Results of the fine-tuning process in NAD against the label-consistent attack with different learning rates on the CIFAR-10 dataset.
c
Poisoning Rate |
7- +
0.1
0.01
0.001
Metric
BA
36.77
89.91
95.38
0.6%
16
ASR
0.63
44.90
94.69
BA
31.12
93.07
95.03
2.5%
ASR
99.67
100
BA
30.73
91.91
94.89
0.6%
ASR
1.15
38.27
99.07
BA
35.93
91.23
94.79
2.5%
ASR
96.19
99.98
Table 15: Results of the fine-tuning process in NAD against WaNet with different learning rates 1).
Dataset +
CIFAR-10
ImageNet
VGGFace2
L, Metric
BA
ASR
BA
ASR
BA
ASR
0.1
48.74
2.75
59.11
0.34
89.05
91.40
0.01
90.60
10.37
79.11
67.09
89.01
99.69
0.001
94.16
97.65
80.63
99.39
88.19
99.73
Table 16: Results of the NAD against attacks on the CIFAR 10 dataset with different B.
Attack 1
500
1,000
1,500
2,000
2,500
5,000
Metric +
BA
92.96
91.87
90.47
86.26
65.26
19.91
BadNets
ASR
27.74
7.92
1.20
1.74
5.81
0.40
BA
91.14
89.72
85.30
75.43
61.18
26.87
Blended Attack
ASR
6.25
1.51
3.19
3.55
3.67
9.61
BA
92.43
91.83
88.08
78.22
58.32
26.63
WaNet
ASR
18.48
9.49
3.58
3.54
3.12
0.32
D.2 SEARCHING BEST RESULTS FOR NAD
We found that the fine-tuning stage of NAD is sensitive to the learning rate. We search the best initial learning rate from (0.1, 0.01, 0.001). As shown in Table 13 15
very large learning rate significantly reduces the BA, while a very small learning rate can not reduce the ASR effectively. To keep relatively large BA while maintaining small ASR, set n 0.01 in the fine-tuning stage.
The distillation stage NAD is also sensitive to its grid-search. The results shown in Table 16 19
B. We select the best 3 via the
19

---

Published as a conference paper at ICLR 2022
Table 17: Results of the NAD against attacks on the ImageNet dataset with different
-
Allack 4
5,000
6,000
7,000
8,000
9,000
10,000
Metric 4
BA
79.98
79.20
76.88
73.49
69.46
67.55
BadNets
ASR
24.65
15.50
16.99
5.81
5.95
2.51
BA
79.98
77.57
74.88
70.88
64.44
53.75
Blended
ASR
88.34
75.79
15.12
8.15
3.47
2.99
BA
79.65
79.06
77.63
76.77
75.27
74.88
WaNet
ASR
76.06
69.23
61.54
45.97
31.43
28.58
Table 18: Results of the NAD against attacks the VGGFace2 dataset with different B.
-
Attack 4
10
30
40
50
60
70
Metric +
BA
89.79
78.10
76.62
83.82
87.29
47.53
BadNets
ASR
36.02
17.87
0.18
14.52
9.49
BA
90.18
83.48
85.27
73.43
75.01
51.81
Blended
ASR
97.11
68.97
10.25
0.36
0.27
BA
82.98
80.69
84.27
26.87
20.71
75.10
WaNet
ASR
66.10
4.41
15.98
0.06
1.88
0.54
Table 19: Results of the NAD against the label consistent attack on the CIFAR-10 dataset with different <
--
Poisoning Rate
500
1,000
1,500
2,000
2,500
5,000
Metric
BA
92.94
91.66
90.29
85.14
73.05
18.49
0.6%
16
ASR
53.11
33.78
12.43
4.39
4.36
0.1
BA
93.36
93.04
90.31
75.11
54.55
24.37
2.5%
ASR
36.13
3.95
1.88
3.13
0.22
BA
92.81
92.13
91.21
85.72
60.06
18.81
0.6%
32
ASR
52.50
57.63
38.53
9.99
1.33
0.06
BA
93.40
92.02
90.98
86.32
68.45
20.53
2.5%
ASR
48.73
17.99
6.54
2.93
1.79
0
E DEFENDING AGAINST LABEL-CONSISTENT ATTACK WITH A SMALLER POISONING RATE
For the label-consistent attack, except for the 2.5% poisoning rate examined the main manuscript, 0.6% is also important setting provided in its original paper (Turner et al. 2019). In this section, we compare different defenses against the label-consistent attack with poisoning rate 7 0.6%.
As shown in Table 20 when defending against label-consistent attack with 0.6% poisoning rate, our method still significantly better than defenses having the same requirements (i.e., DPSGD and ShrinkPad). Even compared with those having the additional requirement (i.e., NC and NAD) under their best settings, our defense is still better or on par with them under the default settings. These results verify the effectiveness of method again.
20

---

Published as a conference paper at ICLR 2022
Table 20: The effectivenesss (%) of defending against the label-consistent attack with 0.6% and 2.5% poisoning rate on CIFAR-10 dataset. Among all defense methods, the one with the best performance indicated in boldface and the value with underline denotes the second-best result. Note that NC and NAD require to have an additional local benign dataset, which is not required in DPSGD, ShrinkPad, and our method.
Defense Metric
poison rate
No Defense
NC
NAD
DPSGD
ShrinkPad
DBD (ours)
BA
95.34
94.51
85.14
82.25
90.59
91.88
0.6%
ASR
92.20
0.73
4.39
4.30
10.22
0.46
16
BA
94.90
94.94
93.04
44.28
90.67
89.67
2.5%
ASR
99.33
1.31
3.95
7.97
9.60
0.01
BA
94.94
94.53
85.72
88.27
90.48
90.04
0.6%
ASR
97.69
0.72
9.99
9.23
11.47
0.15
32
BA
94.82
94.57
90.98
69.79
90.83
91.45
2.5%
ASR
99.19
1.36
6.54
9.06
13.03
0.34
Table 21: DBD against BadNets with different triggers CIFAR-IO dataset.
Location
Trigger Size
Upper Left
Upper Right
Lower Left
Lower Right
Center
1 x
2
x
X
No
BA
95.01
95.01
94.97
94.72
94.77
94.70
94.99
95.01
94.89
Defense
ASR
99.78
99.70
99.86
99.68
99.70
92.13
99.35
99.78
99.86
DBD
BA
92.46
92.47
93.32
92.92
92.58
93.10
92.91
92.46
92.46
(ours)
ASR
1.27
0.53
0.51
0.64
1.41
0.80
0.65
1.27
1.02
F DEFENDING AGAINST ATTACKS WITH DIFFERENT TRIGGER PATTERNS
In this section, we verify whether DBD is still effective when different trigger patterns are adopted.
Settings. For simplicity, we adopt the BadNets on the CIFAR- 10 dataset as an example for the discussion. Specifically, we change the location and size of the backdoor trigger while keeping other settings unchanged to evaluate the BA and ASR before and after our defense.
Results. As shown in Table although there are some fluctuations, the ASR is smaller than 2% while the BA is greater than 92% in every cases, In other words, our method is effective in defending against attacks with different trigger patterns.
G DEFENDING AGAINST ATTACKS WITH DYNAMIC TRIGGERS
In this section, we verify whether DBD is still effective when attackers adopt dynamic triggers.
Settings. We compare DBD with MESA (Qiao et al. 2019 in defending the dynamic attack dis- cussed in (Qiao et al. 2019) on the CIFAR-10 dataset as an example for the discussion. This dynamic attack a distribution of triggers instead of fixed trigger.
Results. The BA and ASR of DBD are 92.4% and 0.4%, while those of MESA are 94.8% and 2.4%. However, find MESA failed in defending against blended attack (for it can correctly detect the trigger) whereas DBD is still effective. These results verified the effectiveness of our defense.
H DISCUSSIONS
H.1 EFFECTS OF HYPER-PARE PARERRTAAS
Settings. Here we analyze the effect of filtering rate which is the only key method-related hyper- parameter in our DBD. We adopt the results on the CIFAR-10 10 dataset for discussion. Except for the studied parameter a, other settings are the same those used in Section 5.2
21

---

Published as a conference paper at ICLR 2022
BedNeb
Blodkd
WaNet
Labe Combitea
BA (%)
1%
(%)
90
AST
92
30
40
50
30
40
50
60
91 01
10
15
20
15 Polsoning Rate (%)
20
Filtering Rate (%)
Filtering Rate (%)
Paisoning Rate (7)
Figure 6: The effects of filtering rate.
Figure 7: The effects of poisoning rate.
Results. The number of labeled samples used in the third stage increase with the increase of filtering rate a, while the probability that the filtered high-credible dataset contains poisoned samples also increases. As shown in Figure 6 DBD can still maintain relatively high benign accuracy even when the filtering rate is relatively small (e.g., 30%). It is mostly due to the high-quality of learned purified feature extractor and the semi-superviser fine-tuning process. DBD can also reach a nearly 0% attack success rate in all cases. However, we also have notice that the high-credible dataset may contain poisoned samples when is very large, which in turn creates hidden backdoors again during the fine-tuning process. Defenders should specify a based on their specific needs.
H.2 DEFENDING ATTACKS WITH VARIOUS POISONING RATES
Settings. We evaluate our method in defending against attacks with different poisoning rate 7 CIFAR-10 10 dataset. Except for y, other settings are the same as those used in Section 5.2
I MORE DETAILS ABOUT SIMCLR, SCE, AND MIXMATCH
NT-Xent Loss in SimCLR. Given sample mini-batch containing N different samples, SimCLR first applies two separate data augmentations toward each sample to obtain 2.V augmented samples. The loss for positive pair of sample (i,j) can be defined as:
exp (sim (zi, 2,) /T)
D.j = - log E T(k exp (sim (z,, /T)
(5)
where sim(.,) is the cosine similarity, 2; is the feature representation of sample i, r is the temper- ature parameter, and I(k * i) {0, indicating whether k & i. The NT-Xent Loss is computed across all 2N positive pairs in this mini-batch.
SCE. The symmetric cross entropy (SCE) can be defined
LsCF H 4) + H(g. p):
where (p, q) is the cross entropy, H(g is the reversed cross entropy, p is the prediction, and g is the one-hot label (of the evaluated sample).
MixMatch Loss. For batch X' labeled samples and batch U of unlabeled samples (Xx) U ). MixMatch produces a guessed label q for each unlabled sample u E U and applies Mix Zhang et al. 2018 to obtain the augmented X' and u' The loss Lx and Lu can be defined as:
Lx
Σ
H (pa,
X
(x,4)EX"
where is the prediction of æ, is its one-hot label, and H ((..) is the cross entropy.
Lu
Σ llp. - 7112
(8)
K. U
where Pa is the prediction of u, is its guessed one-hot label, and K is the number of classes.
By combining Lx with Lue the MixMatch loss be defined as:
L = Lx + dee Lus
(9)
where X is hyper-parameter.
22

---

Published as a conference paper at ICLR 2022
J COMPUTATIONAL FACILITIES
We conduct all experiments on two Ubuntu 18.04 servers having different GPUs. One has four NVIDIA GeForce RTX 2080 Ti GPUs with 11GB memory (dubbed 'RTX 2080Ti') and the another has three NVIDIA Tesla V100 GPUs with 32GB memory (dubbed "V100").
Computational Facilities for Attacks. All experiments are conducted with a single RTX 2080 Ti.
Computational Facilities for Defenses. Since we do not use memory-efficient implementation of DenseNet-121, we conduct DPSGD experiments on the VGGFace2 dataset with a single V V100. Other experiments of baseline defenses are conducted with single RTX 2080 Ti. For our defense, we adopt PoTch (Paszke et al. 2019) distributed data-parallel and automatic mixed precision train- ing (Micikevicius et al. 20188) with two RTX 2080 Ti for self- superviee learning on the VGGFace2 dataset. Other experiments are conducted with a single RTX 2080 Ti.
K COMPUTATIONAL CoST
In this section, analyze the computational cost of our method stage by stage, compared to stan- drad supervised learning.
Stage 1. Self- Self-supervised learning is known to have a higher computational cost than standard super- vised learning (Chen et al. 2020aa; He et al.] 2020). In our experiments, SimCLR requires roughly four times the computational cost of standard supervised learning. Since intend to get a purified instead of well-trained feature extractor, we train the feature extractor (i.e., backbone) lesser epochs than the original SimCLR to reduce the training time. As described in Section B.3] find 100 epochs is enough to preserve effectiveness.
Stage 2. Since we freeze the backbone and only train the remaining fully connected layers, the computational cost is roughly 60% of standard supervised learning.
Stage 3. Semi-supervised learning is known to have extra labeling cost compared with standard supervised learning (Gao et al., 2020). In our experiments, MixMatch requires roughly two times the computation cost of standard supervised learning.
We will explore more computational efficient training method in our future work.
L COMPARING OUR DBD WITH DETECTION-BASEL BACKDOOR DEFENSES
In this paper, we do intend filter malicious and benign samples accurately, as we mentioned in Section 4.4 However, we notice that the second stage of our DBD can serve as a detection- based backdoor defense for can filter poisoned samples. In this section, we compare the filtering ability of our DBD (stage 2) with existing iirection-basede backdoor defenses.
Settings. We compare our DBD with two representative detection based methods, including, Spec- tral Signatures (SS) (Tran et al., 2018) and Activation Clustering (AC) (Chen et al., 2219), on the CIFAR- 10 dataset. These detection- the training set and train the model on non-malicious samples. Specifically, we re-implement SS in PyTorch based on its official code and adopt the open-source code for AC, following the settings in their original paper. In particular, since SS filters 1.5g malicious samples for each class, where is the key nyper-parameter means the upper bound of the number of poisoned training samples, we adopt different e for fair comparison.
methods (e.g.. SS and AC) filter malicious samples from
Results. As shown in Table 22 23 the filtering performance of DBD is on par with that of SS and AC DBD is even better than those methods when filtering poisoned samples generated by more complicated attacks (i.e., WaNet and Label-Consistent). Besides, we also conduct the standard training on non-malicious samples filtered by SS and AC. As shown in Table 24] the hidden backdoor will still be created in many cases, even though the detection-based defenses are sometimes accurate.
http:///ww...ooooooooooooaa/a//a jnvvis /n/////nn//eeeeeeeee/eeeeeee fenaes/
23

---

Published as a conference paper at ICLR 2022
Table 22: The successful filtering rate (# filtered poisoned samples / # all filtered samples, %) under different e in the target class on CIFAR-10 10 dataset.
Attack
Defense J, E
250
500
1000
1500
SS
95.73
93.20
87.60
80.71
BadNets
DBD
100
97.60
90.87
70.09
SS
0.80
10.53
28.87
35.29
Blended
DBD
97.87
94.67
87.27
75.16
SS
0
1.00
7.42
WaNet
DBD
100
100
99.47
97.46
SS
2.40
4.40
9.00
13.78
Label-Consistent
DBD
43.47
37.07
34.47
32.53
Table 23: The number of remaining poisoned samples over filtered non-malicious samples on
CIFAR-10 dataset.
Defense J, Attack +
BadNets
Blended
WaNet
SS (s = 500)
1801/42500
2421/42500
2500/42500
1217/42500
SS 1000)
1186/35000
2067/35000
2400/35000
1115/35000
AC
0/42500
0/37786
5000/45546
1250/39998
DBD
8/25000
6/25000
38/25000
13/25000
Table 24: The BA (%) and ASR (%) of models trained on non-malicious samples filtered by SS and AC on CIFAR-10 dataset.
Defense +
SS (E
500)
SS (E
1000)
AC
Attack J. Metric
BA
ASR
BA
ASR
BA
ASR
BadNets
92.99
100
93.27
99.99
85.90
Blended
92.84
99.07
92.56
99.18
77.17
WaNet
92.69
98.13
91.92
99.00
84.60
99.02
Label-Consisten
92.93
99.79
92.88
99.86
75.95
99.75
Table 25: The BA (%) and ASR (%) of our DBD defending against four attacks with different self-supervised methods on CIFAR- dataset.
Attack +
BadNets
Blended
WaNet
Method J. Metric +
BA
ASR
BA
ASR
BA
ASR
BA
ASR
SimCLR
92.41
0.96
92.18
1.73
91.20
0.39
91.45
0.34
MoCo-V2 V2
93.01
1.21
92.42
0.24
91.69
1.30
91.46
0.19
BYOL
91.98
0.82
91.38
0.51
91.37
1.28
90.09
0.17
This is mainly because these methods may not able to remove enough poisoned samples while preserving enough benign samples simultaneously, i.e., there is trade-off between BA and ASR.
M DBD WITH DIFFERENT SELF-SUPERVISED METHODS
In this paper, we believe that the desired feature extractor mapping visually similar inputs to similar positions in the feature space, such that poisoned samples will be separated into their source classes. This goal is compatible with that of self-supervised learning. We believe that any self- supervised learning adopted SimCLR with other self-supervised methods in our DBD and examine their performance.
be adopted in method. To further verify this point, replace the
Settings. We replace the SimCLR with two other self- self-supervised methods, including MoCo- V2 (Chen et al.] 2020bb and BYOL (Grill et al. 2020). our DBD. Except for the adopted self- supervised method, other settings the same those used in Section 5.2
Results. As shown in Table 25 all DBD variants have similar performances. other words, our DBD is not sensitive to the selection of self-supervised methods.
24

---

Published as a conference paper at ICLR 2022
Table 26: The BA (%) and ASR (%) of our DBD defending against four attacks with different
label-noise learning methods on CIFAR-10 dataset.
Attack -
BadNets
Blended
WaNet
Label- -onns-Cnnnsttett
Method J, Metric +
BA
ASR
BA
ASR
BA
ASR
BA
ASR
SCE
92,41
0.96
92.18
1.73
91.20
0.39
91.45
0.34
GCE
92.93
0.88
93.06
1.27
92.25
1.51
91.05
0.15
APL
92.95
1.00
92.65
0.78
92.24
1.40
91.08
0.14
N DBD WITH DIFFERENT LABEL-NOISS LEARNING METHODS
In the main manuscript, adopt SCE the label-noise learning method in second stage. In this section, we explore whether our DBD still effective if other label-noise methods are adopted.
Settings. We replace SCE in DBD with other label-noise learning methods, including generalized cross entropy (GCE) (Zhang & Sabuncu, 2018) and active passive loss (APL) (Ma et al., 2020 Specifically, we adopt the combination of NCE+RCE in APL and use the default hyper- parameters suggested in their original paper. Except for the adopted label-noise learning method, other settings the same as those used Section
Results. As shown in Table 26 all DBD variants are effective in reducing backdoor threats (i.e., low ASR) while maintaining high benign accuracy. In other words, our DBD is not sensitive the selection of label-noise learning methods.
O ANALYZING WHY OUR DBD IS EFFECTIVE IN DEFENDING AGAINST LABEL-CONSISTENT ATTACK
In general, the good defense performance of our DBD method against the label-consistent attack (which is one of the clean-label attacks) can be explained from the following aspects:
Firstly, shown in Figure [ there a common observation across different attacks (including both poisoned- and clean-label attacks) that poisoned samples tend to gather together in the feature space learned by the standard supervised learning. The most intuitive idea of our DBD is to prevent such a gathering in the Iearned feature space, which is implemented by sule-supe learning. As shown in Figure [(d) the poisoned samples of label-consistent attack are also separated into different areas in the feature space learned by self-supervised learning. This example gives an intuitive explanation about why our DBD can successfully defend against the tabel-consistent attack.
Furthermore, it is interesting explore why the poisoned samples in the label-consistent attack separated under self-supervised learning since all poisoned samples are from the same target class, rather than from different source classes in poisoned-label attacks. For each poisoned sample in this attack, there are two types of features: the trigger and the benign feature with (untargeted) adver- sarial perturbations. From the perspective perturbations are similar to samples from different source classes, though these samples look similar from the human's 's perspective. Thus, it is not surprising that poisoned samples clean-label attacks can also be separated under self- self-supervised learning, just like those in poisoned-lan labeel attacks.
DNNs. benign samples with (untargeted) adversarial
25


Reviews:
[{"review_id": "wYq2mkUoW8", "reviewer": "Reviewer_h7wA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: I enjoyed reading the paper as it has a good structure, strong sets of experiments and baselines. Instead of searching for the triggers as NC and other papers are trying to do, this paper proposes to address the label discrepancy introduced by backdoor samples.\n\nWeaknesses: \n1. The proposed method modifies an underlying training procedure multiplying training time, which I think is significant for practitioners. Addressing this issue is essential to support the practicality of the method.\n2. The primary assumption of the paper is that learning in semi-supervised learning is safe. However, Carlini [1] 's recent work demonstrates the attacker's effectiveness under the same threat model, i.e., when the attacker is only allowed to poison data. If the poisoning is efficient, then the proposed defense exposes the model to a different attack.\n\n[1] Carlini, N. (2021). Poisoning the Unlabeled Dataset of Semi-Supervised Learning. USENIX Security'21", "summary_of_the_paper": "Summary: authors propose a modification to the training procedure to prevent backdoor attacks. Instead of performing supervised training, they suggest first training the model in a self-supervised way, then in a supervised way on fully connected layers. Later they propose to remove low-credible samples and fine-tune the whole model on the remaining samples with labels.  They claim that this procedure eliminates the backdoored inputs that have incorrect labels.", "summary_of_the_review": "Overall, I believe the method is useful, however the above questions are important to be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "n7lnfnE5TyB", "reviewer": "Reviewer_G82o", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes decoupling-based backdoor defense (DBD) on poisoning-based backdoor attacks. Specifically, the authors first give a broad view of why backdoor works by visualizing the poisoned and benign data in the embedding space. They found that the supervised learning paradigm tends to cause poisoned data close to its target label rather than the ground truth label. To mitigate this, the DBD first uses a self-supervised learning to learn a purified feature extractor. It splits the training dataset into two disjoint parts by comparing the fully connected layers' training loss. Later, it uses the high-credible and low-credible samples to fine-tune the whole model in a semi-supervised manner. Experiments are performed on three datasets, three backdoor attack models, and four defense baselines. The authors also consider multiple potential scenarios in practice, such as trigger patterns, adaptive backdoor attacks.\n\nThe idea of this paper is simple but effective, and the authors' efforts in the experiments are highly appreciated. Details comments are as follows.\n\nThe primary concern of this paper is the extra computation cost of the proposed DBD pipeline. Considering training a self-supervised and semi-supervised model costs way more computational resources than a supervised learning model. Would it be possible to use a public pre-trained feature extractor to replace the self-supervised feature extractor? The authors are welcomed to discuss this.\n\nIn Section 5.2, the authors list the results on \"No defense\" to show the backdoor defenses' impact on the original model's accuracy and the effectiveness of the backdoor mitigation. It is not clear how the model works without a defense. Suppose the authors train the original model in a supervised learning manner. In that case, directly using a self-supervised learning paradigm should also be included to illustrate each step's contribution.\n\nIn the fine-tuning step, the sensitivity of lambda should also be discussed, since it controls the ratios of unlablelled data.\n\nIn Figure 1(a), the poisoned samples' embedding seems far from the target label (label 3) and closer to other un-targeted labels (label 1, 7, and 9). It would be great if the authors could give explanations.\n", "summary_of_the_paper": "This paper proposes a decoupling-based backdoor defense (DBD) on poisoning-based backdoor attacks where an adversary can modify the dataset only. Specifically, DBD combines a self-supervised feature extractor and a supervised noise-free classifier, with an additional semi-supervised learning fine-tuning step. The core idea is to decouple the feature extractor and the final prediction. The authors evaluate the effectiveness of DBD on three datasets, three backdoor attack models, and four defense baselines.\n", "summary_of_the_review": "I like this paper and would support an accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WNi_vribrJ", "reviewer": "Reviewer_RsZj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a novel defense method named DBD based on contrastive learning (SimCLR) and conduct extensive experiments to show that DBD is effective against  different types of attacks, including BadNets, Blended, WaNet, and Label-Consistent attacks.\n\nHowever, the author did not compare DBD with some recently works such as \"Spectral signatures in backdoor attacks\" in NIPS 2018 and \"Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering\" in AAAI 2019, which both are based on data removal. It is unclear how differently DBD removes the poisoned data as compared with the existing works.\n\nWhile the authors claim that their major contribution was to decouple the end-to-end training process, I suspect the proposed approach works only with SimCLR. What is the necessary conditions that makes a feature extractor scatter poisoned data points in the feature space? Can you point out other feature extractors having the same properties as SimCLR? Does DBD works with these extractors too?\n\nThe effect of the final fine-tuning step is unclear. How does DBD perform without this phase?\n\nI also suggest the authors to move the section about the resistance to adaptive attacks in Appendix into the main paper as the adaptive attacks are becoming a more serious threats today. Please explain your adaptive attack settings more clearly (for example, what trigger size you used and how you tuned the hyper-parameters).\n\nIt would also be good if the author discuss how an attack may work around the proposed defense, and how to further defend such workarounds.\n\nEdit after rebuttal:\n\nThe reviewer thank the authors for their response. Most of my concerns have been addressed. In particular, the DBD seems to be effective against the adaptive attacks with a large trigger size. The experiments also show that DBD works with other self-supervised learning methods. Also, comparison with the baseline methods such as Spectral Signatures have been made. Due to the above, I raise my score.\n", "summary_of_the_paper": "This paper shows that self-supervised, contrastive learning can give a feature extractor that scatters training data points with backdoor triggers in the feature space. With this observation, the authors propose a novel defense method based on contrastive learning and decouple end-to-end training to defend against backdoor attacks. They first train a feature extractor using self-supervised contrastive learning that turns the poisoned data points into outliers in the feature space. Then they train a cascade classifier that ignores the poisoned data points by leveraging the fact that a neural network tends to capture frequent patterns. Experiments are conducted and the results verify the effectiveness of the defense.", "summary_of_the_review": "Overall, the paper proposes a novel defense to the backdoor attacks that could benefit the field. However, the current experiments are not convincing enough because some critical baselines are missing. There is no discussion on how general the proposed defense work when different training techniques are used in different phases. And there is no discussion on how a backdoor attack may work around the proposed defense, and how to further defend such workarounds like many other security papers did. Therefore, I give a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DDFrKsIHy6", "reviewer": "Reviewer_MY71", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The proposed method is quite intuitive and easy to be extended to the more advanced methods.\n* The experiment part shows the effectiveness of the proposed method.\n\nWeakness:\n* The paper lacks a theoretical analysis of the proposed method. I understand that this paper mainly focused on empirical performance, but * it is quite surprising that the proposed methods perform well on label-consistent attacks. This is because that proposed method decouples the label corruption and feature corruption. When label corruption no longer exists, what is the advantage of the proposed method?\n* For the second step label noise learning, there are also many choices instead of just using the symmetric cross-entropy method. Investigating more noisy-label algorithms might be interesting.\n* The two-step method makes the algorithm not in an end-to-end fashion. It would be interesting to investigate the possibility to make an end-to-end algorithm.\n* I do not think excluding the detection-based method is fair since those methods are strong baselines especially for the badnet and blending attack. Also, since the main contribution of the paper is the empirical performance, it is necessary to compare with different kinds of baselines.", "summary_of_the_paper": "The paper uses self-supervised learning to get benign representation and then uses a noisy label algorithm (SCE) to optimize the prediction model. The empirical performance shows the effectiveness of the proposed method. ", "summary_of_the_review": "I think the proposed method is natural and simple, which makes the paper easy to be extended to different settings. I recommend weak acceptance since the experiment part did not fully convince me.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Inspired by the observtion that the poisoned samples tend to cluster together in the feature space of the attacked DNN model, which is mostly due to the end-to-end supervised training paradigm, the authors propose a novel defense method based on contrastive learning and decouple end-to-end training to defend against backdoor attacks.
The issues, including training time, difference from certain previous studies, ablation study, and so on, raised by the reviewers have been properly addressed and the reviewers are satisfied with the responses from the authors.
According to the consistent positive opinions from the reviewrs, this manuscript is recommended to accept.
Decision: Accept (Poster)