Anti-Backdoor Learning: Training Clean Models on Poisoned Data

1 Introduction . A backdoor attack is a type of training-time data poisoning attack that implant backdoor triggers into machine learning models by injecting the trigger pattern ( s ) into a small proportion of the training data [ 1 ] . It aims to trick the model to learn a strong but task-irrelevant correlation between the trigger pattern and a target class , and optimizes three objectives : stealthiness of the trigger pattern , injection ( poisoning ) rate and attack success rate . A backdoored model performs normally on clean test data yet consistently predicts the target class whenever the trigger pattern is attached to a test example . †Correspondence to : Xixiang Lyu , Xingjun Ma . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Studies have shown that deep neural networks ( DNNs ) are particularly vulnerable to backdoor attacks [ 2 ] . Backdoor triggers are generally easy to implant but hard to detect or erase , posing significant security threats to deep learning . Existing defense methods against backdoor attacks can be categorized into two types : detection methods and erasing methods [ 3 ] . Detection methods exploit activation statistics or model properties to determine whether a model is backdoored [ 4 , 5 ] , or whether a training/test example is a backdoor example [ 6 , 7 ] . While detection can help identify potential risks , the backdoored model still needs to be purified . Erasing methods [ 8–10 ] take one step further and remove triggers from the backdoored model . Despite their promising results , it is still unclear in the current literature whether the underlying model learns clean and backdoor examples in the same way . The exploration of this aspect leads to a fundamental yet so far overlooked question , “ Is it possible to train a clean model on poisoned data ? '' Intuitively , if backdoored data can be identified during training , measures can be taken to prevent the model from learning them . However , we find that this is not a trivial task . One reason is that we do not know the proportion nor the distribution of the backdoored data in advance . As shown in Figure 1 , on CIFAR-10 , even if the poisoning rate is less than 1 % , various attacks can still achieve high attack success rates . This significantly increases the difficulty of backdoor data detection as the model ’ s learning behavior may remain the same with or without a few training examples . Even worse , we may accidentally remove a lot of valuable data when the dataset is completely clean . Another important reason is that the backdoor may have already been learned by the model even if the backdoor examples are identified at a later training stage . In this paper , we frame the overall learning process on a backdoor-poisoned dataset as a dual-task learning problem , with the learning of the clean portion as the original ( clean ) task and the learning of the backdoored portion as the backdoor task . By investigating the distinctive learning behaviors of the model on the two tasks , we identify two inherent characteristics of backdoor attacks as their weaknesses . First , the backdoor task is a much easier task compared to the original task . Consequently , the training loss of the backdoored portion drops abruptly in early epochs of training , whereas the loss of clean examples decreases at a steady pace . We also find that the stronger the attack , the faster the loss on backdoored data drops . This finding indicates that the backdoor correlations imposed by stronger attacks are easier and faster to learn , and marks one distinctive learning behavior on backdoored data . Second , the backdoor task is tied to a specific class ( i.e. , the backdoor target class ) . This indicates that the correlation between the trigger pattern and the target class could be easily broken by simply randomizing the class target , for instance , shuffling the labels of a small proportion of examples with low loss . Inspired by the above observations , we propose a principled Anti-Backdoor Learning ( ABL ) scheme that enables the training of clean models without any prior knowledge of the distribution of backdoored data in datasets . ABL introduces a gradient ascent based anti-backdoor mechanism into the standard training to help isolate low-loss backdoor examples in early training and unlearn the backdoor correlation once backdoor examples are isolated . In summary , our main contributions are : • We present a novel view of the problem of robust learning with poisoned data and reveal two inherent weaknesses of backdoor attacks : faster learning on backdoored data and target-class dependency . The stronger the attack is , the more easily it can be detected or disrupted . • We propose a novel Anti-Backdoor Learning ( ABL ) method that is capable of training clean models on poisoned data . To the best of our knowledge , ABL is the first method of its kind in the backdoor defense literature , complementing existing defense methods . • We empirically show that our ABL is robust to 10 state-of-the-art backdoor attacks . The models trained using ABL are of almost the same clean accuracy as they were directly trained on clean data and the backdoor attack success rates on these models are close to random guess . 2 Related Work . Backdoor Attack . Existing backdoor attacks aim to optimize three objectives : 1 ) making the trigger pattern stealthier ; 2 ) reducing the poisoning ( injection ) rate ; 3 ) increasing the attack success rate [ 3 ] . Creative design of trigger patterns can help with the stealthiness of the attack . These can be simple patterns such as a single pixel [ 6 ] and a black-white checkerboard [ 1 ] , or more complex patterns such as blending backgrounds [ 11 ] , natural reflections [ 12 ] , invisible noise [ 13–16 ] , adversarial patterns [ 17 ] and sample-wise patterns [ 18 , 19 ] . Backdoor attacks can be further divided into two categories : dirty-label attacks [ 1 , 11 , 12 ] and clean-label attacks [ 20–22 , 17 , 16 ] . Clean-label attacks are arguably stealthier as they do not change the labels . Backdoor attackers can also inject backdoors via retraining the victim model on a reverse-engineered dataset without accessing the original training data [ 23 ] . Most of these attacks can achieve a high success rate ( e.g. , > 95 % ) by poisoning only 10 % or even less of the training data . A recent study by Zhao et al . [ 24 ] showed that even models trained on clean data can have backdoors , highlighting the importance of anti-backdoor learning . Backdoor Defense . Existing backdoor defenses fall under the categories of either detection or erasing methods . Detection methods aim to detect anomalies in input data [ 7 , 6 , 25–28 ] or whether a model is backdoored [ 4 , 5 , 29 , 30 ] . These methods typically show promising accuracies ; however , the potential impact of backdoor triggers remains uncleared in the backdoored models . On the other hand , erasing methods take a step further and aim to purify the adverse impacts on models caused by the backdoor triggers . The current state-of-the-art erasing methods are Mode Connectivity Repair ( MCR ) [ 9 ] , Neural Attention Distillation ( NAD ) [ 10 ] and Adversarial Neuron Pruning ( ANP ) [ 31 ] . MCR mitigates the backdoors by selecting a robust model in the path of loss landscape , NAD leverages attention distillation to erase triggers , while ANP prunes adversarially sensitive neurons to purify the model . Other previous methods , including finetuning , denoising , and fine-pruning [ 8 ] , have been shown to be insufficient against the latest attacks [ 32 , 33 , 12 ] . An early work [ 34 ] found that DNNs are more accurate on clean samples in an early training stage , while we find that the backdoor attacks studied in [ 34 ] are only limited to simple BadNets attacks . Thus , it would be interesting to further study the different properties of diverse types of backdoor attacks . In this paper , we introduce the concept of anti-backdoor learning . Unlike existing methods , our goal is to train clean models directly on poisoned datasets without further altering the models or the input data . This requires a more in-depth understanding of the distinctive learning behaviors on backdoored data . However , such information is not available in the current literature . Anti-backdoor learning methods may replace the standard training to prevent potential backdoor attacks in real-world scenarios where data sources are not 100 % reliable , and the distribution or even the presence of backdoor examples are unknown . 3 Anti-Backdoor Learning . In this section , we first formulate the Anti-Backdoor Learning ( ABL ) problem , then reveal the distinctive learning behaviors on clean versus backdoor examples and introduce our proposed ABL method . Here , we focus on classification tasks with deep neural networks . Defense Setting . We assume the backdoor adversary has pre-generated a set of backdoor examples and has successfully injected these examples into the training dataset . We also assume the defender has full control over the training process but has no prior knowledge of the proportion nor distribution of the backdoor examples in a given dataset . The defender ’ s goal is to train a model on the given dataset ( potentially poisoned ) that is as good as models trained on purely clean data . Moreover , if an isolation method is used , the defender may identify only a subset of the backdoor examples . For instance , in the case of 10 % poisoning , the isolation rate might only be 1 % . Robust learning methods developed under our defense setting could benefit companies , research institutes , government agencies or MLaaS ( Machine Learning as a Service ) providers to train backdoor-free models on potentially poisoned data . More explanations about our threat model and how our proposed ABL method can help other defense settings can be found in Appendix B.10 . Problem Formulation . Consider a standard classification task with a dataset D = Dc [ Db where Dc denotes the subset of clean data and Db denotes the subset of backdoor data . The standard training trains a DNN model f✓ by minimizing the following empirical error : L = E ( x , y ) ⇠D [ ` ( f✓ ( x ) , y ) ] = E ( x , y ) ⇠Dc [ ` ( f✓ ( x ) , y ) ] | { z } clean task +E ( x , y ) ⇠Db [ ` ( f✓ ( x ) , y ) ] | { z } backdoor task , ( 1 ) where ` ( · ) denotes the loss function such as the commonly used cross-entropy loss . The overall learning task is decomposed into two tasks where the first clean task is defined on the clean data Dc while the second backdoor task is defined on the backdoor data Db . Since backdoor examples are often associated with a particular target class , all data from Db may share the same class label . The above decomposition indicates that the standard learning approach tends to learn both tasks , resulting in a backdoored model . To prevent backdoor examples from being learned , we propose anti-backdoor learning to minimize the following empirical error instead : L = E ( x , y ) ⇠Dc [ ` ( f✓ ( x ) , y ) ] E ( x , y ) ⇠Db [ ` ( f✓ ( x ) , y ) ] . ( 2 ) Note the maximization of the backdoor task is defined on Db . Unfortunately , the above objective is undefined during training since we do not know the Db subset . Intuitively , Db can be detected and isolated during training if the model exhibits an atypical learning behavior on the backdoor examples . In the following subsection , we will introduce one such behavior , which we recognize as the first weakness of backdoor attacks .

Summary: This paper observes that models converge much faster on backdoored data than clean data. Within a few epochs, the training loss for backdoored data will decrease to zero, but not for clean data. Based on such an observation, this paper aims to train clean models on poisoned data without knowing the backdoored portion in the training set. Specifically, it proposes a new training procedure called anti-backdoor learning (ABL), which splits the traditional training procedure into two stages. In the first stage, ABL uses a threshold to prevent loss values for individual samples being too small by flipping the sign of the loss. It then selects a small portion (1%) of the training set with small losses as the backdoored set and the remaining as the clean set. In the second stage, ABL applies gradient ascent on the backdoored set and gradient descent on the clean set. The evaluation is conducted on three standard benchmarks and ABL is compared with three existing backdoor removal baselines. The experimental results show ABL has the best performance on both reducing attack success rate and retaining normal accuracy in most cases. A list of studies on different components of ABL and comparisons with other alternative methods are interesting and informative.