Anti-Backdoor Learning: Training Clean Models on Poisoned Data
Authors: Yige Li
Xixiang Lyu
Nodens Koren
Lingjuan Lyu
Bo Li
Xingjun Ma
Venue: NeurIPS 2021 Poster (2021)
Keywords: Backdoor Defense
Deep Neural Networks
Anti-backdoor learning

Abstract: Backdoor attack has emerged as a major security threat to deep neural networks (DNNs). While existing defense methods have demonstrated promising results on detecting or erasing backdoors, it is still not clear whether robust training methods can be devised to prevent the backdoor triggers being injected into the trained model in the first place. In this paper, we introduce the concept of \emph{anti-backdoor learning}, aiming to train \emph{clean} models given backdoor-poisoned data. We frame the overall learning process as a dual-task of learning the \emph{clean} and the \emph{backdoor} portions of data. From this view, we identify two inherent characteristics of backdoor attacks as their weaknesses: 1) the models learn backdoored data much faster than learning with clean data, and the stronger the attack the faster the model converges on backdoored data; 2) the backdoor task is tied to a specific class (the backdoor target class). Based on these two weaknesses, we propose a general learning scheme, Anti-Backdoor Learning (ABL), to automatically prevent backdoor attacks during training. ABL introduces a two-stage \emph{gradient ascent} mechanism for standard training to 1) help isolate backdoor examples at an early training stage, and 2) break the correlation between backdoor examples and the target class at a later training stage. Through extensive experiments on multiple benchmark datasets against 10 state-of-the-art attacks, we empirically show that ABL-trained models on backdoor-poisoned data achieve the same performance as they were trained on purely clean data. Code is available at \url{https://github.com/bboylyg/ABL}.

Anti-Backdoor Learning: Training Clean Models on Poisoned Data
Yige Xidian University yglee0stu. xidian edu, en
Xixiang Lyu f Xidian University xxlv0mail, xidian edu. en
Nodens Koren University of Copenhagen nodens f. koren@di. ku.
Lingjuan Lyu
Sony AI Lw@.ony. .com com
Bo Li
Xingjun Ma i Fudan University danx janxjma@gmail
University of Illinois at Urbana- Chammaagnn lbo@illing wwwliiiiii.....ii....i....ii edu
Abstract
Backdoor attack has emerged as a major security threat to deep neural networks (DNNs). While existing defense methods have demonstrated promising results detecting or erasing backdoors, is still not clear whether robust training methods be devised prevent the backdoor triggers being injected into the trained model in the first place. In this paper, we introduce the concept of anti-backdoor learning, aiming to train clean models given backdoor-poisonee learning process as a dual-task of learning the clean and the backdoor portions of data. From this view, we identify two inherent characteristics of backdoor attacks as their weaknesses: 1) the models learn backdoored data much faster than learning with clean data, and the stronger the attack the faster the model converges on backdoored data; 2) the backdoor task is tied to specific class (the backdoor target class). Based on these two weaknesses, we propose general learning scheme, Anti-Backdoor Learning (ABL), to automatically prevent backdoor attacks during training. ABL introduces two-stage gradient ascent mechanism for standard training I) help isolate backdoor examples at early training stage, and 2) break the correlation between backdoor examples and the target class at a later training stage. Through extensive experiments against 10 state-of-th thet--t--artttt attacks, we empirically show that ABL-traine models on backdoor- backdoor-poisoned data achieve the same performance they were trained on purely clean data. Code available at https: // //ggttttbb.//. cww.coo/////////o commbboylly. ABL.
data. We frame the overall
multiple benchmark datasets
Introduction
A backdoor attack is type of training-time data poisoning attack that implant backdoor triggers into machine learning models by injecting the trigger pattern(s) into small proportion of the training data [1]. It aims trick the model to learn strong but task-irrelevant correlation between the trigger pattern and a target class, and optimizes three objectives: stealthiness of the trigger pattern, injection (poisoning) rate and attack success rate. A backdoored model performs normally clean test data yet consistently predicts the target class whenever the trigger pattern is attached to a test example.
Correspondence to: Xixiang Lyu, Xingjun Ma.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Studies have shown that deep neural networks (DNNs) are particularly vulnerable to backdoor attacks [2]. Backdoor triggers are generally easy implant but hard detect or erase, posing significant security threats to deep learning.
Existing defense methods against backdoor attacks can be categorized into two types: detection methods and erasing methods [3]. Detection methods exploit activation statistics or model properties to determine whether model is backdoored [4, 5J, or whether a training/test example is a backdoor example [6. 7J. While detection can help identify potential risks, the backdoored model still needs to be purified. Erasing methods [8-10] take one step further and remove triggers from the backdoored model. Despite their promising results, it is still unclear in the current literature whether the underlying model learns clean and backdoor examples in the same way. The exploration of this aspect leads to a fundamental yet so far overlooked question, "Is it possible to train a clean model on poisoned data?"
Intuitively, if backdoored data be identified during train- ing, measures can be taken to prevent the model from learn- ing them. However, we find that this is not trivial task. One reason is that we do not know the proportion nor the distribution of the backdoored data in advance. As shown
Attack success rate (%)
Mini
in Figure 1, on CIFAR-10, even if the poisoning rate is less than 1%, various attacks can still achieve high attack success rates. This significantly increases the difficulty of backdoor data detection as the model's learning behavior may remain the same with or without a few training examples. Even worse, we may accidentally remove lot of valuable data when the dataset is completely clean. Another important reason is that the backdoor may have already been learned by the model even if the backdoor examples are identified at a later training stage.
18
18
10
Poldoninng
Figure 1: Attack success rate (ASR) of 6 backdoor attacks under different poisoning rates on CIFAR-1O. 4 out of the 6 attacks can achieve nearly 100% ASR at poisoning rate 0.5%.
In this paper, we frame the overall learning process on backdoor -ooisoooo-ooooddddooooooo dataset as a dual-task learning problem, with the learning of the clean portion as the original (clean) task and the learning of the backdoored portion as the backdoor task. By investigating the distinctive learning behaviors of the model on the two tasks, we identify two inherent characteristics of backdoor attacks as their weaknesses. First, the backdoor task is a much easier task compared to the original task. Consequently, the training loss of the backdoored portion drops abruptly early epochs of training, whereas the loss of clean examples decreases at a steady pace. We also find that the stronger the attack, the faster the loss on backdoored data drops. This finding indicates that the backdoor correlations imposed by stronger attacks are easier and faster to learn, and marks one distinctive learning behavior on backdoored data. Second, the backdoor task is tied to a specific class (i.e., the backdoor target class). This indicates that the correlation between the trigger pattern and the target class could be
casily broken by simply randomizing the class target, for instance, shuffling the labels of a small proportion of examples with low loss.
Inspired by the above observations, we propose a principled anti-Backdoor Learning (ABL) scheme that enables the training of clean models without any prior knowledge of the distribution of backdoored data in datasets. ABL introduces gradient ascent based anti-backdoor mechanism into the standard training to help isolate low-loss backdoor examples in early training and unlearn the backdoor correlation once backdoor examples are isolated. In summary, our main contributions are:
We present novel view of the problem of robust learning with poisoned data and reveal two inherent weaknesses of backdoor attacks: faster learning on backdoored data and target-class dependency. The stronger the attack is, the more easily it can be detected disrupted.
We propose novel Anti-Backdoor Learning (ABL) method that is capable of training clean models on poisoned data. To the best of our knowledge, ABL is the first method of its kind in the backdoor defense literature, complementing existing defense methods.
We empirically show that our ABL is robust to 10 stateoof-th--ae backdoor attacks. The models trained using ABL are of almost the same clean accuracy as they were directly trained on clean data and the backdoor attack success rates on these models are close to random guess.

---

2 Related Work
Backdoor Attack, Existing backdoor attacks aim to optimize three objectives: 1) making the trigger pattern stealthier; 2) reducing the poisoning (injection) rate; 3) increasing the attack success rate [3]. Creative design of trigger patterns can help with the stealthiness of the attack. These can be simple patterns such as a single pixel [6] and black-white checkerboard [1], or more complex patterns such as blending backgrounds [11]. natural reflections [12], invisible noise [13-16], adversarial patterns [17] and sample-wise patterns [18, 19). Backdoor attacks can be further divided into two categories: dirty-label attacks [1, 11, 12] and clean-label attacks [20-22, 17, 16]. Clean-label attacks are arguably stealthier as they do not change the labels, Backdoor attackers can also inject backdoors via retraining the victim model on eeverse-engineeree dataset without accessing the original training data [23]. Most of these attacks can achieve a high success rate (e.g.. > 95%) by poisoning only 10% or even less of the training data. A recent study by Zhao et al. [24] showed that even models trained on clean data can have backdoors, highlighting the importance of anti- baackoor learning.
Backdoor Defense. Existing backdoor defenses fall under the categories of either detection or erasing methods. Detection methods aim to detect anomalies in input data [7, 6, 25-28] or whether a model is backdoored [4, 5, 29, 30J. These methods typically show promising accuracies; however, the potential impact of backdoor triggers remains uncleared in the backdoored models. On the other hand, erasing methods take step further and aim to purify the adverse impacts models caused by the backdoor triggers. The current state-of-the, art erasing methods are Mode Connectivity Repair (MCR) [9], Neural Attention Distillation (NAD) [10] and Adversarial Neuron Pruming (ANP) [31]. MCR mitigates the backdoors by selecting a robust model in the path of loss landscape, NAD leverages attention distillation to erase triggers, while ANP prunes ad adversarially sensitive neurons to purify the model. Other previous methods, including finetuning, denoising, and fine-pruning [8], have been shown to be insufficient against the latest attacks [32, 33, 12]. An early work [34] found that DNNs are more accurate on clean samples in an early training stage, while we find that the backdoor attacks studied in [34] are only limited to simple BadNets attacks. Thus, it would be interesting to further study the different properties of diverse types of backdoor attacks.
In this paper, we introduce the concept of anti-backloor learning. Unlike existing methods, our goal is to train clean models directly on poisoned datasets without further altering the models or the input data. This requires more in-depth understanding of the distinctive learning behaviors backdoored data. However, such information is not available in the current literature. Anti-backdoor learning methods may replace the standard training prevent potential backdoor attacks in real- world scenarios where data sources are not 100% reliable, and the distribution or even the presence of backdoor examples are unknown.
3 Anti-Backdoor Learning
In this section, we first formulate the Anti-Backdoor Learning (ABL) problem, then reveal the distinctive learning behaviors on clean versus backdoor examples and introduce proposed ABL method. Here, we focus on classification tasks with deep neural networks.
Defense Setting. We assume the backdoor adversary has pre-generated a set of backdoor examples and has successfully injected these examples into the training dataset. We also assume the defender has full control over the training process but has no prior knowledge of the proportion nor distribution of the backdoor examples a given dataset. The defender's ' goal is to train a model on the given dataset (potentially poisoned) that is good models trained purely clean data. Moreover, an isolation method is used, the defender may identify only a subset of the backdoor examples. For instance, in the case of 10% poisoning, the isolation rate might only be 1%. Robust learning methods developed under our defense setting could benefit companies, research institutes, government agencies or MLaaS (Machine Learning a Service) providers to train backdoor-fri models on potentially poisoned data. More explanations about our threat model and how our proposed ABL method can help other defense settings can be found in Appendix B.
Problem Formulation. Consider a standard classification task with a dataset D D. UD, where D. denotes the subset of clean data and Dj denotes the subset of backdoor data. The standard training
3

---

trains a DNN model fe by minimizing the following empirical enror:
C C-E(as)~D(((((ee)) v) = ((a.))>,,l((Vvaa),) +
v)
(1)
clean task
backddoor task
where E(.) denotes the loss function such as the commonly used cross-entropy loss. The overall learning task is decomposed into two tasks where the first clean task is defined on the clean data D while the second backdoor lask is defined on the backdoor data Dr. Since backdoor examples are often associated with particular target class, all data from Dd may share the same class label. The above decomposition indicates that the standard learning approach tends to learn both tasks, resulting in backdoored model.
To prevent backdoor examples from being Iearned, we propose anti-backdoor learning to minimize the following empirical error instead:
C E (a.))DD ([o(())) v)] ((a.))DD, -F y)]
Note the maximization of the backdoor task is defined on Dj. Unfortunately, the above objective is undefined during training since we do not know the Dd subset. Intuitively, Dj can be detected and isolated during training if the model exhibits an atypical learning behavior on the backdoor examples. In the following subsection, we will introduce one such behavior, which we recognize as the first weakness of backdoor attacks.
3.1 Distinctive Learning Behaviors on Backdoor Examples
We apply 6 classic backdoor attacks including BadNets [1], Trojan [23], Blend [11], Dynamic [18), SIG [35], and CL [21], and 3 feature-space attacks including FC [20], DFST [36], and LBA [32] to poison 10% of CIFAR- 10 training data. We train WideRes model [37] on the corresponding poisoned dataset using the standard training method by solving equation (1) for each attack. Each model is trained following the standard settings (see Section 4 and Appendix A.2). We plot the average training loss (i.c.. cross- cross-entropy) on clean versus backdoored training examples in Figure 2. Clearly, for all attacks, the training loss on backdoor examples drops much faster than that on clean examples in the first few epochs. Both pixel- and feature-space attacks exhibit this faster-learning pattern consistently, although some feature-space attacks (FC and LBA) can slow down the process to some extent. For all attacks except SIG, FC and LBA, the training loss reaches almost zero after only two epochs of training. Moreover, according to the attack success rate, the stronger the attack is, the faster the training loss on backdoor examples drops. More results on GTSRB and an ImageNet subset can be found in Appendix B.2.
The above observation indicates that the backdoor task is much easier than the clean task. This is not too surprising. In a typical clean dataset, not all examples are easy examples. Thus, it requires a certain number of training epochs to minimize the loss on those examples, even for small datasets like CIFAR-10. On the contrary, backdoor attack adds an explicit correlation between the trigger pattern and the target class to simplify and accelerate the injection of the backdoor trigger. We argue that this a fundamental requirement and also a major weakness of backdoor attacks. For backdoor attack to work successfully, the trigger(s) should be easily learnable by the model, or else the attack would lose its effectiveness or require much higher injection rate, which goes against its key objectives. Therefore, the stronger the attack is, the faster the training loss on backdoor examples drops to zero; e.g., compare FC with other attacks in Figure 2. We also show in Figure 6 in Appendix B. that the training loss of the backdoor task drops more rapidly as we increase the poisoning rate.
Based on the above observation, one may wonder if backdoor examples can be casily removed by filtering out the low-loss examples at an early stage (e.g., the 5th epoch). However, we find that this strategy is ineffective for two reasons. First, the training loss in Figure is the average training loss which means some backdoor examples can still have high training loss. Additionally, several powerful attacks such as Trojan and Dynamic can still succeed even with very few (50 or 100) backdoor examples. Second, if the training progresses long enough (e.g., beyond epoch 20), many clean examples will also have a low training loss, which makes the filtering significantly inaceurate. Therefore, we need a strategy to amplify the difference in training loss between clean and back examples. Moreover, we need to unlearn the backdoor since the backdoor examples can only be identified when they are learned into the model (i.c., low training loss).

---

25
26
20-
20
20
Trakino tou
0.6
0.
0.0
10
00
Epochs
Epecto
(a) BadNets (ASR-100%)
(b) Trujan (ASR-100%)
(e) Blend (ASR-100%)
2.5
25
25
sumppss
2.0
2.0
1.5
66
0.5
0.5
60-
30
08
00
30
Epoche
Epocho
(d) Dynamic (ASR-100%)
(e) SIG (ASI--9946%)
(f) CL (ASR-199.83%
25
exapples
2.
hoss
4.5
Trannng
1.0
20
19
15
00
15
20
90
00
0
Epochs
Epaciti
(g) FC (ASR-88.52%)
(b) DFST (ASR-99.76%)
(i) LBA (ASR-99.13%)
Figure 2: The training loss on clean versus backdoor examples crafted by 9 backdoor attacks including
BadNets [1]. Trojan [23], Blend [11]. Dynamic [18], SIG [35], and CL [21], FC [20], DFST [36], and
LBA [32]. This experiment is conducted with WideResNet- 16-1 [37] on CIFAR-10 under poisoning
rate 10% ASR: attack success rate (on WideResNet-16-1).
3.2 Proposed Anti-Backdoor Learning Method
Suppose the total number of training epochs is T. we decompose the entire training process into two stages, i.e., early training and later training. We denote the turning epoch from early training to later training by Tie. Our anti-backdoor learning method consists of two key techniques: 1) backdoor isolation during early training, and 2) backdoor unlearning during later training. The turning epoch is chosen to be the epoch where the average training loss stabilizes at a certain level.
Backdoor Isolation. During early training, we propose local grudient ascent (LGA) technique to trap the loss value of each example around certain threshold y. We use the loss function LLGA in equation (3) achieve this. The gradient ascent is said to be "local" because the maximization is performed around a fixed loss value ς. In other words, if the loss of a training example goes below y, gradient ascent will be activated to boost its loss to y; otherwise, the loss stays the same. Doing so will force backdoor examples to escape the 7 constraint since their loss values drop significantly faster. The choice of an appropriate y lies in the core of this strategy, as an overly large y will hurt the learning of the clean task, while an overly small may not be strong enough to segregate the clean task from the backdoor task. Note that 7 can be determined by the strength of potential attacks: stronger attacks only need a smaller 7 to isolate. Since most backdoor attacks are strong, the poisoned data can easily reach a small loss value below 0.5. So, we set 7 = 0.5 our experiments and show its consistent performance across different datasets and models in Section 4.2 and Appendix B.5. At the end of carly training, we segregate examples into disjoint subsets: p percent of data with the lowest loss values will be isolated into the backdoor set D. (p = DD.I/(D), and the rest into the clean set D. (D= = Dr U D.). An important note here is that the isolation rate (e.g., p = 1%) is assumed to be much smaller than the poisoning rate (e.g., 10%).
Backdoor Unlearning, With the clean and backdoor sets, we can then proceed with the later training. Note that at this stage, the backdoor has already been learned by the model. Given the above low isolation rate, an effective backdoor unlearning method is required to make the model unlearn the backdoor with small subset D, of backdoor examples while simultancously learning the remaining
5

---

(unisolated) backdoor examples in the clean set De. We make this possible by exploiting the second weakness of backdoor attacks: the backdoor trigger usually associated with a particular backdoor target class. We propose to use the loss LOGA defined in equation (2) for this purpose. In LoGs, global gradient ascent (GGA) is defined on the isolated subset Dz. Unlike the local gradient ascent, it is not constrained to be around a fixed loss value. We will show in Section 4.2 that a low isolation rate of 1% is able to effectively unlearn the backdoor against 50% poisoning.
The loss functions used by our ABL for its two training stages are summarized as follows,
CLOAN = B(a (f -)- [(fo(z),
i S t Tie
CARL. =
(3)
Lara = ((.a)))) [z) v)] 16 (e) [((f(())))) w) ih. St<T,
where t E [0,T- -1 1] is the current training epoch, sign(-) is the sign function, is the loss threshold for LGA and D3 is the isolated backdoor set with the isolation rate p |D),//D). During early training (0 I t < T,e), the loss will be automatically switched to -((fe(z)))) v) if E(.) is smaller than y by the sign function; otherwise the loss stays the same, i.e., {(fo(z). 1y. Note that CLGA loss may also be achieved by the flooding loss proposed in [38] to prevent overfitting: [e(fo(z), v) - bl + b where b is flooding parameter. We would like to point out that LGA serves only one part of our ABL and can potentially be replaced by other backdoor detection methods. Additionally, we will show that a set of other techniques may also achieve backdoor isolation and unlearning, but they are far less effective than our ABL (see Section 4.3 and Appendix B.3).
4 Experiments
Attack Configurations. We consider 10 backdoor attacks in our experiments, including four dirty-
label attacks: BadNets [1], Trojan attack [23), Blend attack [11], Dynamic attack [18), two clean-label
attacks: Sinusoidal signal attack(SIG) [35] and Clean-label attack(CL) [21]. and four feature-space
attacks: Feature collision (FC) [20], Deep Feature Space Trojan Attack (DFST) [24], Latent Backdoor Attack (LBA) [32], and Composite Backdoor Attack (CBA) [39]. We follow the settings suggested by [10] and the open-sourced code corresponding to their original papers to configure these attack algorithms. All attacks are evaluated on three benchmark datasets, CIFAR- 10 [40], GTSRB [41] and an ImageNet subset [42], with two classical model structures including WideResNet (WRN-16-1) [37] and ResNet-34 [43]. No data augmentations are used for these attacks since they hinder the backdoor effect [12]. To keep their original configurations of dataset and parameter settings, here we only run the four reature-space attacks on CIFAR-10 dataset. We also omit some attacks on GTSRB and ImageNet datasets due to a failure of reproduction following their original papers. The detailed settings of the 10 backdoor attacks are summarized in Table 5 (see Appendix A.2).
Defense and Training Details. We compare our ABL with three state-of-the-art defense methods: uneepruming (FP) [8], Mode Connectivity Repair (MCR) [9], and Neural Attention Distillation (NAD) [10]. For FP, MCR and NAD, we follow the configurations specified in their original papers, including the available clean data for finetuningheeparrdisiilltttt and training settings. The comparison with other data isolation methods are shown Section 4.3. For our ABL, we set T= 100, Tie = 20, 7 = 0.5 and an isolation rate p = 0.01 (1% in all experiments. The exploration of different Tie. 7, and isolation rates p are also provided in Section 4.1. Three data augmentation techniques suggested in [10] including random crop (padding 4), horizontal flipping, and cutout, are applied for all defense methods. More details on defense settings can be found in Appendix A.3.
Evaluation Metrics. We adopt two commonly used performance metrics: Attack Success Rate (ASR), which is the classification accuracy on the backdoor test set, and Clean Accuracy (CA), the classification accuracy on clean test set.
4.1 Effectiveness of Our ABL Defense
Comparison Existing Defenses. Table 1 demonstrates our proposed ABL method on CIFAR- 10. GTSRB, and an ImageNet Subset. We consider 10 state-of-thh-ar backdoor attacks and compare the performance of ABL with three other backdoor defense techniques. It is clear that our ABL achieves the best results on reducing ASR against most of backdoor attacks, while maintaining an extremely high CA across all three datasets. comparison to the best baseline method NAD, our ABL achieves 12.7 1% (7.69% vs. 20.40%), 11.90% (7.27% vs. 19.17%) and 7.35% (6.00% vs. 13.35%) lower

---

Table 1: The attack success rate (ASR %) and the clean accuracy (CA %) of 4 backdoor defense methods against 10 backdoor attacks including 6 classic backdoor attacks and 4 sature--pace attacks. None means the training data is completely clean.
No Defense
FP
MCR
NAD
ABL
(Ours)
Dataset
Types
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
None
0%
89.12%
0%
85.14%
0%
87.49%
0%
88.18%
0%
88.41 %
BadNets
100%
85.43%
99.98%
82.14%
3.32%
78.49%
3.56%
82.18%
3.04%
86. %
Trojan
100%
82.14%
66.93%
80.17%
23.88%
76.47%
18.16%
80.23%
3.81 %
87.46 %
Blend
100%
84.51 %
85.62%
81.33%
31 1.5%
76.53%
82.04% 4.56%
16.23%
84.06%
CIFAR-10
Dynamic
100%
83.88%
87.18%
80.37%
26. 26.86% 70.36%
22.50%
74.95%
18.46
85.34% %
SIG
99,46%
16%
76.32%
81.12%
0.14%
78.65%
82.01 % 1.92%
0.09%
88,27 %
CL
99.83%
83.43%
54.95%
81.53%
19. 19.86% 77.36%
16.11%
80.73%
0 %
89.03%
FC
88.52% 83.32%
69.89%
80.51%
44.43%
77.57%
58.68%
81.23%
0.08%
82.36%
DFST
99.76%
82.50%
78.11% 1 80.23%
39.22%
75.34%
35.21%
78.40%
5.33%
79.78%
LBA
99.13%
81.37%
54,43%
79.67%
15.52%
78.51%
10.16%
79.52%
0.06%
80.52%
CBA
90.63%
84.72%
77.33%
79.15%
38.76%
76.36%
33. 1 11% 82.40%
29.81 %
84.66 %
Average
97.73%
83.55%
75.07%
80.62%
24.38%
76.56%
20.40%
80.37%
7.69%
84.76 %
None
0%
97.87%
0%
90.14%
0%
95.49%
0%
95.18%
0%
96.41 %
BadNets
100%
97.38%
99.57%
88.61 %
1.00%
93.45%
0.19%
89.52%
0.03%
96.01 %
Trojan
99. 80%
96.27%
93.54%
84.22%
2.76%
92.98%
0.37%
90.02%
0.36 %
94.95%
GTSRB
Blend
100%
99. 99.50% 95.97%
86.67%
6.83%
92.91%
8.10%
89.37%
24.59%
93.14%
Dynamic
100%
97.27%
99. 99.84% 88.38%
64.82%
43.91%
68.71%
76.93%
6.24%
95.80% %
SIG
97.13%% 97. 97.13%
79.28%
90.50%
33.98%
91.83%
4.64% %
89.36%
5.13%
96.33 %
Average
99.38%
96.80%
94.35%
87.68%
21.88%
83.01%
19.77%
87.04%
7.27% %
95.25% %
None
0%
89.93%
0%
83. 14%
0%
85.49%
0%
88. 18%
0%
88.31 %
ImageNet Subset
BadNets
100%
84.41%
97.70%
82.81% %
28.59%
78.52%
6.32%
81.26%
0.94%
87.76 %
Trojan
100%
85.56%
96.39%
80.34%
6.67%
76.87%
15.48%
80.52%
1.47%
88.19 %
Blend
99 99.9%% 86. 16%%
99. 99%3%% 81.33%
19.23% % 75.83%
26.47%
82.39%
21. 21.42%
85.12%
SIG
98.60% 86. 86.02%
78.82% 85.72%
25, 144
78.87%
83.01 % 5.15%
0.18%
86.42% %
Average
99.63% 85.53%
93.06% 82.55%
19.91%
77.52%
13.35%
81.80%
6.00%
86.87% %
average ASR against the 10 attacks on CIFAR-10, GTSRB, and the ImageNet subset, respectively. This superiority becomes more significant when compared to other baseline methods.
We notice that our ABL is not always the best when looking at the 10 attacks individually. For instance, NAD is the best defense against the Blend attack on CIFAR-10 10 and against the SIG attack on GTSRB, while MCR the best against Blend on GTSRB and the ImageNet subset. We suspect this is because both Blend and SIG mingle the trigger pattern (i.c., another image or superimposed sinusoidal signal) with the background of the poisoned images, producing an effect of natural artifacts. This makes them harder to be isolated and unlearned, since even clean data can have such patterns [24]. This is one limitation of our ABL that needs to be improved in future works. Our ABL achieves much better performance than the baselines on defending against the 4 feature-space attacks. For example, NAD only manages to decrease the attack success rates of the FC and the DFST attacks to 58.68% and 35.21%, respectively. In contrast, our ABL can bring their ASRs down to below 10%. The Dynamic and the CBA attacks are found to be the toughest attacks to defend against in general. For example, baseline methods NAD, MCR and FP can only decrease CBA's ASR to 33.11% %, 38. 38.76%. and 77.33% on CIFAR- 10, and Dynamic's ' ASR to 68.71%, 64.82%, and 99.84% on GTSRB, respectively, a result that is much worse than the 29.81 % and 6.24% ASRs of our ABL.
Maintaining the clean accuracy as important as reducing the ASR, as the model would lose utility if its clean accuracy is much compromised by the defense. By inspecting the average CA results in Table 1, one can find that our ABL achieves nearly the same clean accuracy as models trained on 100% clean (shown in row None and column 'No Defense") datasets. Specifically, our ABL surpasses the average clean accuracy of NAD by 4.39% (84.76% vs. 80.37%), 8.21 % (95.25% vs. 87.04%) and 5.07% (86.87% vs. 81.80% on CIFAR- GTSRB, and the ImageNet subset, respectively. FP defense decreases the model's performance even when training data are clean (the None row). This makes our ABL defense more practical for industrial applications where performance is equally
7

---

100
100
90
BadNets Trejan
Bland
SIG
90
BodNats
Blond
Dynamic
CL
Troyan
Dynamic
G1
80
Clean accuracy (%)
80
70
70
60
80
50
50
40
40
30
30
20
20
10
10
096
0%
1%
20%
Isolation rate
Isolation
Figure 3: Performance of our ABL with different isolation rate p E [0.01. 0.2] on CIFAR-10 10 dataset. Left: attack success rate (ASR); Right: clean accuracy of ABL against 6 classic backdoor attacks.
2.5
2.0
+
Losu 1.0
Loau Ri: y-u 0.5
BadNets
Loca on Da. 0.3
1-0
100
80
BadNets
Lous Rocy-1..
80
1.5
70
1.0
60
ASRICA
0.5
50
0.0
40
30
-0.5
20
DA y
AS2 y
CA y- 1.0
ABR y-
10
DA 1-1
ASE
-1.5
-10
10 Epochs
20
Epochs
Figure 4: Separation effect of local gradient ascent with different y on CIFAR-10 10 against BadNets. Left: Training loss on the ground truth backdoor (Da) and clean (D,) subsets; Right: Attack success rate (ASR) and clean accuracy (CA). The gap between the two lines of the same color becomes wider for larger i.e., better separation effect.
important as security. Results of our ABL with 1% isolation against 1% poisoning can be found in Table 10 (Appendix B.9).
Effectiveness with Different Isolation Rates. Here, we study the correlation between the isolation rate p = [D, [ and the performance of our ABL, on the CIFAR-10 10 dataset. We run ABL with different p E (0.01, 0.2] and show attack success rates and clean accuracies in Figure 3. There is trade-off between ASR reduction and clean aceuracy. Specifically, high isolation rate can isolate more backdoor examples for the later stage of unlearning, producing a much lower ASR. However, it also puts more examples into the unlearning mode, which harms the clean accuracy. In general, ABL with isolation rate < 5% works reasonably well against all 6 attacks, even though the backdoor poisoning rate much higher, i.e., 70% (see Figure in Section 4.2). Along with the results Table 1. this confirms that it is indeed possible to break and unlearn the backdoor correlation with only tiny subset of correctly-ides identified backdoor examples, highlighting one unique advantage of backdoor isolation and unlearning approaches.
Effectiveness with Different Turning Epochs. Here, we study the impact of the timing to switch from the learning stage (LLGA) to the unlearning stage (LGGs) on CIFAR-10. We compare four different turning epochs: the 10th, 20th, 30th, and 40th epoch, and record the results of our ABL in Table 6 (see Appendix B. B.4). We find that delayed turning epochs tend to slightly hinder the defense performance. Despite the slight variations, all choices of the turning epoch help mitigate backdoor attacks, but epoch 20 (i.e., at 20% 30% of the entire training progress) achieves the best overall results. This trend consistent on other datasets as well. We attribute these results to the success of LGA in preserving the difference between clean and backdoor samples over time, which enables us to select the turning epoch flcxibly. A more comprehensive discussion on LGA is given in Section 4.2.
4.2 Comprehensive Understanding of ABL
Importance of Local Gradient Ascent. data, we visualize and compare in Figure the training loss and the model's performances (ASR and CA) under three different settings where is set to 0.5, 1.0, and 1.5, respectively. It is evident that L.A can segregate backdoor examples from clean examples to a certain extent under three settings of y by preventing the loss of clean examples from converging. Moreover, larger y leads to
To help understand how LGA works in isolating backdoor
8

---

Table 2: Stress testing with poisoning rate up to 50% and 70% for 4 attacks including BadNets, Trojan, Blend, and Dynamic on CIFARIO dataset.
BadNets
Trojan
Blend
Dynamic
Poisoning Rate Defense
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
None
100%
75.31%
100%
70.44%
100%
69.49%
100%
65.15%
50%
ABI.
4.98%
70.52%
16.11%
68.56%
27.28%
64.19%
25.74%
61.32%
None
100%
74.8%
100%
69.46%
100%
67.32%
100%
65.15%
70%
ABL
5.02%
70.11%
29.29%
68.79%
62.28%
64.43%
69.36%
62.09%
a wider difference in training loss as well as ASR and CA. However, we note that this may cause training instability, as evidenced by the relatively larger fluctuations with 7 1.5.
We also examine the precision of the 1 1% isolated backdoor set under different y of 0, 0.5, 1.0, and 1.5 on CIFAR-10 10, GTSRB, and the ImageNet subset. We use BadNets attack with poisoning rate 10% and set the turning (isolation) epoch of ABL to 20. We report the isolation precision results in Table 7 (see Appendix B.5). As can be seen, when y O, the detection precision is poor; this indicates that is tough for the model to tell apart backdoor examples from the clean ones without the LGA, which foreseeable because the clean training loss is uncontrolled and overlaps with the backdoor training loss. Note that as soon as we set 7 > 0, the precision immediately improves on both CIFAR- 10 and the ImageNet subset. Additionally, the precision of the isolation task not sensitive the change in 7. which again allows the hyperparameter value to be flexibly chosen. In our experiments, 7 0.5 works reasonably well across different datasets and models.
In summary, LGD creates and sustains a gap between the training loss of clean and backdoor examples, which plays vital role in extracting isolated backdoor set.
Fixing 1 % Isolation Rate While Increasing Poisoning Rate. Now that we know we can confidently extract a tiny subset of backdoor examples with high purity, the challenge remains whether the extracted set is sufficient for the model to unlearn the backdoor. We demonstrate that our ABL is stronger method, even under this strenuous setting. Here, we experiment against 4 attacks including BadNets, Trojan, Blend, and Dynamic with poisoning rates up to 50%/70% and show the results in Table 2. We can find that even with high poisoning rate of 50%, our ABL method can still reduce the ASR from 100% to 4.98%. 16. %, 27.28%, and 25. 25.74% for BadNets, Trojan, Blend, and Dynamic, respectively. Note that ABL will break when the poisoning rate reaches 70%. In this case however, the dataset should not be used to train any models in the first place. Overall ABL remains effective against up to 1) 70% BadNets; and 2) 50% Trojan, Blend, and Dynamic. This finding is very compelling, considering ABL needs to isolate only 1 1% of training data. As we mentioned before, this is because the correlation between the backdoor pattern and the target label exposes a weakness of backdoor attacks. Our ABL utilizes the GGA to break this link and achieve defense goals effortlessly.
Stress Testing:
CIFAR-10
4.3 Exploring Alternative Isolation and Unlearning Methods
Alternative Isolation Methods. In this section, we compare the isolation precision of our ABL with two backdoor detection methods, namely Activation Clustering (AC) [7] and Spectral Signature Analysis (SSA) [6]. The goal is to isolate 1 %% of training examples into the backdoor set (Db), and we provide in Figure 7 (see Appendix B.3) the precision of these methods alongside our ABL in detecting the 6 backdoor attacks on CIFAR-10 dataset. We find that both AC and SS achieve high detection rates on BadNets and Trojan attacks while perform poorly on 4 other attacks. A reasonable explanation is that attacks covering the whole image with complex triggers (c.g.. Blend, Dynamic, SIG, and CL) give confusing and unidentifiable output representations of either feature activation or spectral signature, making these detection methods ineffective. It is worth mentioning that our ABL is effective against all backdoor attacks with the highest average detection rate. In addition, we find that the flooding loss [38] proposed for mitigating overfitting is also very effective for backdoor isolation. We also explore a confidence- based isolation with label smoothing (LS), which unfortunately fails on most attacks. More details of these explorations can be found in Figure and in Appendix B.6.
Alternative Unlearning Methods. Here we explore several other empirical strategies, including mmage-based. label-based, model-based approaches, to rebuild a clean model on the poisoned data.

---

Table 3: Performance of various unlcarning methods against BadNets attack on CIFAR-10.
Discard
Backdoored
After Unlearning
Backdoor Unlearning Methods
Method Type
D.
ASR
CA
ASR
CA
Pixel Noise
Image-based
No
100%
85.43%
57.54%
82.33%
Grad Noise
Image-based
No
100%
85,43%
47.65%
82.62%
Label Shufflig
Label-based
No
100%
85.43%
30.23%
83.76%
Label Uniform
Label-based
No
100%
85.43%
75.12%
83,47%
Label Smoothing
Label-based
No
100%
85,43%
99.80%
83.17%
Self-Learning
Label-based
No
100%
85.43%
21.26%
84.38%
Finetuning All Layers
Model-based
Yes
100%
85,43%
99.12%
83.64%
Finetuning Last Layers
Model-based
Yes
100%
85.43%
22.33%
77.65%
Finetuning ImageNet Model
Model-hased
Yes
100%
85.43%
12.18%
75.10%
Re-training from Scratch
Model-hased
Yes
100%
85.43%
11.21%
86.02%
ABL
Model-hased
No
100% 85.43%
3.04%
86.11%
These approaches are motivated by the second weakness of backdoor attacks, and are all designed to break the connection between the trigger pattern and the target class. We experiment on CIFAR-10 with BadNets (10% poisoning rate), and fix the backdoor isolation method to our ABL with a high isolation rate 20% (as most of them will fail with 1% isolation). Table 3 summarizes our explorations. Our core findings can be summarized as: a) adding perturbations to pixels or gradients is not effective; b) changing the labels of isolated examples is mildly effective: e) finetuming some (not all) layers of the model cannot effectively mitigate backdoor attacks; d) "self-learning" and "retraining the model from scratch" on the isolated clean set are good choices against backdoor attacks; and e) our ABL presents the best unlearning performance. Details of these methods are given in Appendix A.4. The performance of these methods under the 1 % isolation rate also reported Table in Appendix B.7. We also considered another widely used attack settings that the user only allowed to access a backdoored model and hold limited benign data. In this case, we proposed to combine our ABL unlearning with Neural Cleanse [4] to erase the backdoroed model (sce Appendix B.10).
Conclusion
In this work, we identified two inherent characteristics of backdoor attacks as their weaknesses: 1) backdoor examples are easier and faster to learn than clean examples, and 2) backdoor learning establishes strong correlation between backdoor examples and the target label. Based on these two findings, we proposed a novel framework of two stages of learning utilizing local gradient ascent (LGA) and global gradient ascent (GGA), respectively. At the early learning stage, we use LGA to intentionally maximize the training loss gap between clean and backdoor examples to isolate out the backdoored data via the low loss value. We use GGA to unlearn the backdoored model with the isolated backdoor data at the last learning stage. Empirical results demonstrate that our ABL is resilient to various experimental settings and can effectively defend against 10 state-of-the-art backdoor attacks. Our work introduces simple but very effective ABL method for industries to train backdoor-frr models on real-world datasets, and opens up an interesting research direction for robust and secure machine learning.
Learning (ABL) which consists
Broader Impact
Large-scale data have been key the success of deep learning. However, is hard to guarantee the quality and purity of the training data in many cases, and even high-quality datasets may contain backdoors, especially those collected from the internet. By introducing the concept of anti-backdoor learning (ABL), our work opens up a new direction for secure and robust learning with not-fully- trusted data. Even in the clean setting, ABL can prevent deep learning models from overfitting to those overly easy samples. Beyond backdoor defense, ABL could be explored as a generic duta-quality-waare learning mechanism in place of the traditional data-quality- Such mechanism may help reduce many potential data-quality-relle rrelttdd risks such as memorization, overfitting, backdoors and biases. Although not our initial intention, our work may adversely be exploited to develop advanced backdoor attacks. This essentially requires new defenses to combat.
learning.
10

---

Acknowledgement
This work is supported by China National Science Foundation under grant number 62072356 and in part by the Key Research and Development Program of Shaanxi under Grant 20192DLGY122-0
References
[1] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint /rXiv:/70..0673 2017.
[2] Cheng-Hsin Weng, Yan-Ting Lee, and Shan-Hung Brandon Wu. On the trade-off between adversarial and backdoor robustness. In NeurlPS, 2020.
[3] Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: survey, arXiv preprint arXiv:2007.08745, 2020.
[4] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In S&P. IEEE, 2019.
[5] Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: black-box trojan detection and mitigation framework for deep neural networks. In IJCAI, 2019.
[6] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In NeurlPS, 2018.
[7] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In AAAI Workshop, 2019.
[8] Kang Liu, Brendan DolintGavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In RAID, 2018.
[9] Pu Zhao, Pin- Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In ICLR, 2020.
[10] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR, 2021.
[11] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 171iv: 2017.
[12] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV, 2020.
[13] Cong Liao, Haoti Zhong, Anna Squicciarini, Sencun Zhu, and David Miller. Backdoor embed- ding in convolutional neural network models via invisible perturbation. CODASPY, 2020.
[14] Shaofeng Li, Minhui Xue, Benjamin Zi Hao Zhao, Haojin Zhu, and Xinpeng Zhang. Invisible backdoor attacks on deep neural networks via steganography and regularization. arXiv preprint arXiv:1909.02743 2019.
[15] Jinyin Chen, Haibin Zheng, Mengmeng Su, Tianyu Du, Changting Lin, and Shouling Ji. Invisible poisoning: Highly stealthy targeted poisoning attack. In ICISC, 2019.
[16] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In AAAI, volume 34, pages 11957-11965, 2020.
[17] Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean-label backdoor attacks on video recognition models. In CVPR, pages 14443-14452. 2020.
[18] Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurIPS, 2020.

---

[19] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, pages 16463-16472, 2021.
[20] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks neural networks. NeurIPS, 2018.
[21] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. https///people.sail..ie haw.:iaiiii..uu/mm/////abaaa 2019.
[22] Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In ICML., pages 7614-7623. PMLR, 2019.
[23] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS, 2018.
[24] Shihao Zhao, Xingjun Ma, Yisen Wang, James Bailey, Bo Li, and Yu-Gang Jiang. What do deep nets learn? blass-wise patterns revealed in the input space. arXiv preprint wxXiv:210100680) 2021.
[25] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In ACSAC, 2019.
[26] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In S&P, 2021.
[27] Jonathan Hayase, Weihao Kong. Raghav Somani. and Sewoong Oh. Spectre: Defending against backdoor attacks using robust statistics. In ICML, 2021.
[28] Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of dnns for robust backdoor contamination detection. In USENIX Security, 2021.
[29] Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In CVPR, 2020.
[30] Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimiza- tion. In ICML, 2021.
[31] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. NeurlPS, 2021.
[32] Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks on deep neural networks. In CCS, 2019.
[33] Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004.04683 2020.
[34] Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In ICML, 2019.
[35] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In ICIP, 2019.
[36] Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. In AAAI, 2021.
[37] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[38] Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Do we need zero training loss after achieving zero training error? In ICML, 2020.
[39] Junyu Lin, Lei Xu, Yingqi Liu, and Xiangyu Zhang. Composite backdoor attack for deep neural network by mixing existing benign features. In CCS, 2020.
12

---

[40] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[41] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-333 332, 2012.
[42] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[44] Chulin Xie, Keli Huang. Pin- Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In ICLR, 2019.
[45] Lingjuan Lyu, Han Yu, Xingjun Ma. Lichao Sun, Jun Zhao, Qiang Yang. and Philip Yu. Privacy and robustness in federated learning: Attacks and defenses. arXiv preprint 2012:77012.06337, 2020.
13


Reviews:
[{"review_id": "j39-mUFIgb8", "reviewer": "Reviewer_KE43", "summary": "Authors have introduced the concept of anti-backdoor learning to learn clean models using backdoor-poisoned data. They have proposed Anti-Backdoor Learning (ABL) to automatically break backdoor attack during training with poisoned data. They have identified two weaknesses from backdoor attack. First weakness is that models learn from poisoned data at a much faster rate than these learn from clean data. The stronger the attack, the faster the models converge to a backdoor. The second weakness is that the backdoor task is tied to a specific class (the backdoor target class). Based on the two weaknesses, the authors designed a two-step method to learn a clean model with poisoned data.  The first step is to isolate poisoned samples from the training dataset, and the second step is to use the isolated samples to unlearn backdoor. The first step happened in the early training stage and the second stage takes over the training after the first stage. As compared to other state of the arts, the isolation step is very precise, and in many case, the precision is easily to reach 100%. The second step is very effective to unlearn backdoors. Even if only 1% of the poisoned samples are isolated, it can counter up to 70% poisoned samples to learn a clean model. Overall, this is a good paper and the proposed method outperformed the competing methods under the authors’ experimental setting. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?\n\nThis is a new method as long as I am aware of, and I think references are adequate. \n\nQuality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\nThe paper is technically sound. Claims are well supported by their experimental results. However, the models being used in the paper are way below the performances of the state of the arts. For example, state of the arts on CIFAR10 are way above 90% (close to 100%), and the models being used in this paper obtained less than 90% of accuracies. How would this affect your conclusions? \n\nClarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\n\nThe paper is well organized, pleasant to read and easy to follow. \n\nSignificance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n\nI believe the solution provided by the paper is significant. However, the authors did not discuss the computational complexities. For example, if the user gives a pass of poisoned sample detection before using the data to train a model, how much extra time s/he needs to spend, assuming the poisoned data screening process is effective? This information will give important information of the importance of the paper. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In addition to the limitations mentioned by the authors, here are some other concerns of this reviewer: all the showcases are based on the relatively simple attack: BadNets. For example, The Stress Testing showed that with a fixing 1% Isolation rate, the proposed method is still very effective with up to poising rate of 70%, which is fantastic meaning that the 1% identified poisoned samples can overpower the other unidentified 69% poisoned samples with the proposed method. Is it true for other relatively complicated attacks? ", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "hMQpFQvf0JZ", "reviewer": "Reviewer_tvph", "summary": "The proposed work identifies that backdoor attack examples converge fairly quickly compared to clean examples during training, due to the nature of their objective. Using this, the authors identify backdoor examples and use a gradient ascent method to unlearn the backdoor, while gradient descent is used for clean examples. The proposed method is shown to be effective in reducing Attack Success Rate (ASR) while increasing Clean Accuracy (CA) on multiple datasets and attacks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n— The proposed work is shown to be effective against multiple attacks for CIFAR10,GTSRB and ImageNet subset dataset.\n\n— The paper is well written and easy to understand.\n\n— The proposed method is compared against different unlearning and detection methods, showing best performance.\n\nWeaknesses:\n\n— While the authors consider different class of attacks, it is not known if the training loss observation holds for other forms of optimization or feature-collision based attacks [15,17,19]. \n\nAdditional comments:\n\n— In L141, it is mentioned that 6 backdoor attacks are used to poison 10% of CIFAR10 training data. Does that mean the poisoned dataset consists of all 6 attacks? If so, it would be interesting to see if transferability across attacks hold in this scenario- training data consists of 5 backdoor attacks and test set backdoor data comes from the 6th. \n\nDue to the gradient ascent procedure, the network might unlearn only those 5 backdoors. As a result, the Re-training from scratch method (presented in Table 2) might perform better compared to ABL on ASR. An experiment in this setting would help understand the effect of L_GGA.\n\n— How does Tuning epoch affect the weaker SIG attack?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss limitations and potential negative impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "cvkJQNBE4Q", "reviewer": "Reviewer_9SWb", "summary": "This paper observes that deep neural networks learn backdoored data faster than\nbenign samples. Based on this finding, the paper proposes Anti-Backdoor Learning\n(ABL), which can learn a benign model on poisoned datasets. Specifically, at the\nbeginning of model training, ABL maximizes the training loss gap between clean\nsamples and backdoored examples and detects backdoored data based on the\ndifferences in their loss values. Then, ABL unlearns backdoored model with the\ndetected data.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper finds that deep neural models learn backdoored samples faster than\nbenign samples, which is interesting. However, it lacks sufficient analysis to\nsupport this conclusion. For example, the paper only shows the results on\nResNet-18 and CIFAR-10 in figure 2 to support the conclusion. One model and one\ndata set are not representative enough.\n\nAlso, the observation is drawn from simple backdoor attacks. I suggest training\nthem on complicated backdoor attacks, such as feature space backdoor and latent\nspace backdoor, e.g., Cheng et al. AAAI 2021, Lin et al. CCS 2020, Yao et al.\nCCS 2019.\n\nMoreover, the paper assumes that data poisoning happens at the beginning of the\nmodel training. In the data poisoning attack scenario, backdoor attacks can\nhappen at any time. If the attacker poisons the data dynamically, how would this\ndefense effectively defend such attacks?\n\nIt is unclear to me how this approach can generalize to other threat models,\nsuch as federated learning.\n\nFinally, it seems the selection of the fixed loss value is quite important and\nhard to decide. If the fixed loss value is large, the training loss of the model\ncannot decrease after it reaches the fixed loss value and the accuracy of the\nmodel cannot increase. If the fixed loss value is small, it is harder to detect\nbackdoored samples. Based on my understanding, the fixed loss value needs to be\nadjusted based on the validation dataset but it is hard to create a validation\ndataset for poisoned datasets because we do not know whether the attack exists\nand the attack type. The paper can add a discussion about this problem.\n\nA minor comment:\n\nIn Table 1, it would be helpful if the paper can provide results of the SIG\nattack on ImageNet subset dataset because it seems ABL may perform worse when\ndefense against SIG attack (from the results on GTSRB).\n\nIn \"Effectiveness with Different Turning Epochs\", the paper claims that epoch 20\nachieves the best results for all datasets but only shows results on CIFAR-10.\nIt would be better if the paper can show results on other datasets.\n\nUpdates after rebuttal: \n\nThank the authors for providing new experiments and justify their design/arguments. They addressed most of my concerns. I would like to raise my score but I still have a concern that when the trigger is complex enough (i.e., close to benign features learned by neural networks), the model will take longer time to learn, in which case, the foundation of this defense is broken.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work uses existing assets but did not discuss license issues.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "JsaiHWWlSa8", "reviewer": "Reviewer_oQMj", "summary": "This paper observes that models converge much faster on backdoored data than clean data. Within a few epochs, the training loss for backdoored data will decrease to zero, but not for clean data. Based on such an observation, this paper aims to train clean models on poisoned data without knowing the backdoored portion in the training set. Specifically, it proposes a new training procedure called anti-backdoor learning (ABL), which splits the traditional training procedure into two stages. In the first stage, ABL uses a threshold to prevent loss values for individual samples being too small by flipping the sign of the loss. It then selects a small portion (1%) of the training set with small losses as the backdoored set and the remaining as the clean set. In the second stage, ABL applies gradient ascent on the backdoored set and gradient descent on the clean set. The evaluation is conducted on three standard benchmarks and ABL is compared with three existing backdoor removal baselines. The experimental results show ABL has the best performance on both reducing attack success rate and retaining normal accuracy in most cases. A list of studies on different components of ABL and comparisons with other alternative methods are interesting and informative.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The observation on the training loss of backdoored data is interesting. The proposed training procedure leverages this observation to counter backdoor effects, and the results show the effectiveness of the proposed technique. This work may have potentials in defending against backdoor attacks in general. I have a few concerns regarding the observation, the proposed method and experiments. It would be appreciated if the authors could elaborate in the following points.\n\n1. The observation of the distinctive train losses between backdoored data and clean data is quite interesting. But the six attacks studied in Figure 2 all seem to be universal attacks, where the backdoor flips all other classes to one single target class. How about label-specific attacks where only one or two victim classes are compromised? Can we still have similar observations for the training loss as in Figure 2? If so, the proposed method would probably be much more general and useful. Also, 5 out of 6 studied attacks are static attacks. Although Dynamic attack stamps input-specific triggers, the type of triggers is still at the pixel-level, where such patterns might easily be learned by models. This could be the reason of the distinctive train losses. To further understand different types of backdoors, filter attacks would be a good candidate for the study as such attacks universally transform the input images and generate more natural backdoored data. It would be interesting to see whether such natural-looking backdoored data are still distinguishable from clean data based on their training losses.\n\n2. From Figure 2, by just looking at the training loss for the first a few epochs, backdoored data are already clearly separated from clean data. Why does ABL only select a small portion (1%) as the backdoored data? Why cannot we just directly use a threshold, for instance 0.4, to remove all the backdoored data? In Figure 2, the 0.4 threshold at epoch 5 can clearly separate the two sets (backdoored and clean). Are there some poisoned images still having large training loss after a few epochs or clean images having very small loss? If this is the case, Figure 2 does not well reflect the true training loss. Could the authors clarify this?\n\n3. The proposed local gradient ascent loss is very similar to the flooding loss. The results in Figure 9 in Appendix shows these two also have very similar performance. Since the evaluation is only conducted on 4 attacks on CIFAR-10, it is hard to tell whether the proposed loss is indeed better than the flooding loss. Also, the flooding level is set to 0.5 as stated in the appendix. Is this the best parameter for the flooding loss? Could there be a better parameter? I would suggest the authors to extend the evaluation on the comparison with the flooding loss as the proposed loss has a very similar characteristic and performance.\n\n4. In the introduction, the authors motivate their work by claiming \"a high attack success rate can still be achieved on CIFAR-10 by different attacks even though the poisoning rate is less than 1%\". This is very intriguing as such a low poisoning rate might make defense harder. However, in the evaluation, this paper uses 10% poisoning rate as the default setting. Even when studying the performance of ABL against different poisoning rates, this paper only increases the poisoning rate. I am interested in whether ABL is still effective if a very low poisoning rate is chosen. I would like to see ABL's performance on the poisoning rates studied in Figure 1.\n\n5. Have the authors considered adaptive attacks? For instance, the adversary can intentionally generate poisoned data having training loss similar to normal data. Will the proposed training procedure still be effective defending against backdoor attacks?\n\n6. Could the authors explain why ABL has a higher clean accuracy on the clean-label poisoned training set (89.03%) than on the clean training set (88.41%)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations of ABL against Blend and SIG attacks. But there are missing discussions on the applicability of ABL on label-specific attacks, filter-like attacks. Please see details in the main review.\n\nThe authors have addressed the potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I read the paper and considered the discussions carefully, especially that of the authors with 9SWb who brought up many thoughtful comments and criticisms. I believe that the concerns and criticisms have been largely addressed by the authors during the rebuttal, which I will discuss below.

I will now discuss the merits of the paper:
1. Well-established threat model. The threat scenario of classical (non-federated) training on a dataset poisoned such that inserting the backdoor during inference will result in the target class is a well-established scenario. It is also one of the simplest scenarios, which means the attack is more likely to be deployed in the real world.
2. Novelty. To mine and all the reviewers' agreement, the observations and methods presented in this paper are quite novel. Compared to other backdoor defense methods which often aim to learn the backdoor pattern from among a large search space, I believe the easy-learning detection and backdoor unlearning methods are quite elegant since they exploit what seems to be a fundamental weakness of backdoors (that to be strong they must be easy to learn) and can be incorporated directly into the training procedure (as opposed to a separate stage).
3. Results. The authors have done a comprehensive study on 3 datasets and 6 backdoor attacks, and included some well-established datasets/architectures for comparison with prior art (i.e. resnet18 on cifar, along with resnet34 upon the request of 9SWb). The authors covered 4 additional attacks upon the request of reviewer 9SWb who subsequently raised their score from 3 to 5.
4. The authors provided ample ablation studies to provide better insight into why their method works, which helps the readers obtain some learnings from this paper.

There were some concerns about different threat models (e.g. adversary having control over more things than just a fraction of the dataset) and different training settings (federated learning). While interesting, these are outside the scope of the paper and that the current scope is sufficiently interesting.

I therefore recommend acceptance.
Decision: Accept (Poster)