Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks

Deep neural networks ( DNNs ) are known vulnerable to backdoor attacks , a training time attack that injects a trigger pattern into a small proportion of training data so as to control the model ’ s prediction at the test time . Backdoor attacks are notably dangerous since they do not affect the model ’ s performance on clean examples , yet can fool the model to make incorrect prediction whenever the trigger pattern appears during testing . In this paper , we propose a novel defense framework Neural Attention Distillation ( NAD ) to erase backdoor triggers from backdoored DNNs . NAD utilizes a teacher network to guide the finetuning of the backdoored student network on a small clean subset of data such that the intermediate-layer attention of the student network aligns with that of the teacher network . The teacher network can be obtained by an independent finetuning process on the same clean subset . We empirically show , against 6 state-of-the-art backdoor attacks , NAD can effectively erase the backdoor triggers using only 5 % clean training data without causing obvious performance degradation on clean examples . Our code is available at https : //github.com/bboylyg/NAD . 1 INTRODUCTION . In recent years , deep neural networks ( DNNs ) have been widely adopted into many important realworld and safety-related applications . Nonetheless , it has been demonstrated that DNNs are prone to potential threats in multiple phases of their life cycles . A type of well-studied adversary is called the adversarial attack ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Ma et al. , 2018 ; Jiang et al. , 2019 ; Wang et al. , 2019b ; 2020 ; Duan et al. , 2020 ; Ma et al. , 2020 ) . At test time , state-of-the-art DNN models can be fooled into making incorrect predictions with small adversarial perturbations ( Madry et al. , 2018 ; Carlini & Wagner , 2017 ; Wu et al. , 2020 ; Jiang et al. , 2020 ) . DNNs are also known to be vulnerable to another type of adversary known as the backdoor attack . Recently , backdoor attacks have gained more attention due to the fact it could be easily executed in real scenarios ( Gu et al. , 2019 ; Chen et al. , 2017 ) . Intuitively , backdoor attack aims to trick a model into learning a strong correlation between a trigger pattern and a target label by poisoning a small proportion of the training data . Even trigger patterns as simple as a single pixel ( Tran et al. , 2018 ) or a black-white checkerboard ( Gu et al. , 2019 ) can grant attackers full authority to control the model ’ s behavior . Backdoor attacks can be notoriously perilous for several reasons . First , backdoor data could infiltrate the model on numerous occasions including training models on data collected from unreliable sources or downloading pre-trained models from untrusted parties . Additionally , with the invention of more complex triggers such as natural reflections ( Liu et al. , 2020b ) or invisible noises ( Liao et al. , 2020 ; Li et al. , 2019 ; Chen et al. , 2019c ) , it is much harder to catch backdoor examples at test time . On top of that , once the backdoor triggers have been embedded into the target model , it is hard to completely eradicate their malicious effects by standard finetuning or neural pruning ( Yao et al. , 2019 ; Li et al. , 2020b ; Liu et al. , 2020b ) . A recent work also proposed the mode connectivity repair ( MCR ) to remove backdoor related neural paths from the network ( Zhao et al. , 2020a ) . On the other hand , even though detection-based approaches have been performing fairly well on identifying backdoored models ( Chen et al. , 2019a ; Tran et al. , 2018 ; Chen et al. , 2019b ; Kolouri et al. , 2020 ) , the identified backdoored models still need to be purified by backdoor erasing techniques . †Correspondence to : Xixiang Lyu ( xxlv @ mail.xidian.edu.cn ) , Xingjun Ma ( daniel.ma @ deakin.edu.au ) In this work , we propose a novel backdoor erasing approach , Neural Attention Distillation ( NAD ) , for the backdoor defense of DNNs . NAD is a distillation-guided finetuning process motivated by the ideas of knowledge distillation ( Bucilua et al. , 2006 ; Hinton et al. , 2014 ) and neural attention transfer ( Zagoruyko & Komodakis , 2017 ; Huang & Wang , 2017 ; Heo et al. , 2019 ) . Specifically , NAD utilizes a teacher network to guide the finetuning of a backdoored student network on a small subset of clean training data so that the intermediate-layer attention of the student network is wellaligned with that of the teacher network . The teacher network can be obtained from the backdoored student network via standard finetuning using the same clean subset of data . We empirically show that such an attention distillation step is far more effective in removing the network ’ s attention on the trigger pattern in comparison to the standard finetuning or the neural pruning methods . Our main contributions can be summarized as follows : • We propose a simple yet powerful backdoor defense approach called Neural Attention Distillation ( NAD ) . NAD is by far the most comprehensive and effective defense against a wide range of backdoor attacks . • We suggest that attention maps can be used as an intuitive way to evaluate the performance of backdoor defense mechanisms due to their ability to highlight backdoored regions in a network ’ s topology . 2 RELATED WORK . Backdoor Attack . Backdoor attack is a type of attack emerging in the training pipeline of DNNs . Oftentimes , a backdoor attack is accomplished by designing a trigger pattern with ( poisoned-label attack ) ( Gu et al. , 2019 ; Chen et al. , 2017 ; Liu et al. , 2018b ) or without ( clean-label attack ) ( Shafahi et al. , 2018 ; Turner et al. , 2019 ; Liu et al. , 2020b ) a target label injected into a subset of training data . These trigger patterns can appear in forms as simple as a single pixel ( Tran et al. , 2018 ) or a tiny patch ( Chen et al. , 2017 ) , or in more complex forms such as sinusoidal strips ( Barni et al. , 2019 ) and dynamic patterns ( Li et al. , 2020c ; Nguyen & Tran , 2020 ) . Trigger patterns may also appear in the form of natural reflection ( Liu et al. , 2020b ) or human imperceptible noise ( Liao et al. , 2020 ; Li et al. , 2019 ; Chen et al. , 2019c ) , making them more stealthy and hard to be detected even by human inspection . Recent studies have shown that a backdoor attack can be conducted even without access to the training data ( Liu et al. , 2018b ) or in federated learning ( Xie et al. , 2019 ; Bagdasaryan et al. , 2020 ; Lyu et al. , 2020 ) . Surveys on backdoor attacks can be found in ( Li et al. , 2020a ; Lyu et al. , 2020 ) . Backdoor Defense . Existing works primarily focused on two types of strategies to defend against backdoor attacks . Depending on the methodologies , a backdoor defense can be either backdoor detection or trigger erasing . Detection-based methods aim at identifying the existence of backdoor adversaries in the underlying model ( Wang et al. , 2019a ; Kolouri et al. , 2020 ) or filtering the suspicious samples from input data for re-training ( Tran et al. , 2018 ; Gao et al. , 2019 ; Chen et al. , 2019b ) . Although these methods have been performing fairly well on distinguishing whether a model has been poisoned , the backdoor effects still remain in the backdoored model . On the other hand , Erasing-based methods aim to directly purify the backdoored model by removing the malicious impacts caused by the backdoor triggers , while simultaneously maintain the model ’ s overall performance on clean data . A straightforward approach is to directly finetune the backdoored model on a small subset of clean data , which is typically available to the defender ( Liu et al. , 2018b ) . Nonetheless , training on only a small clean subset can lead to catastrophic forgetting ( Kirkpatrick et al. , 2017 ) , where the model overfits to the subset and consequently causes substantial performance degradation . Fine-pruning ( Liu et al. , 2018a ) alleviates this issue by pruning less informative neurons prior to finetuning the model . In such a way , the standard finetuning process can effectively erase the impact of backdoor triggers without significantly deteriorating the model ’ s overall performance . WILD ( Liu et al. , 2020a ) proposed to utilize data augmentation techniques alongside distribution alignment between clean samples and their occluded versions to remove backdoor triggers from DNNs . Other techniques such as regularization ( Truong et al. , 2020 ) and mode connectivity repair ( Zhao et al. , 2020a ) have also been explored to mitigate backdoor attacks . While promising , existing backdoor erasing methods still suffer from a number of drawbacks . Efficient methods can be evaded by the latest attacks ( Liu et al. , 2018a ; 2020b ) , whereas effective methods are typically computationally expensive ( Zhao et al. , 2020a ) . In this work , we propose a novel finetuning-based backdoor erasing approach that is not only effective but efficient against a wide range of backdoor attacks . Knowledge Distillation ( KD ) . KD was first proposed to compress a bigger or an ensemble of welltrained network ( s ) into a compact smaller network ( Bucilua et al. , 2006 ; Hinton et al. , 2014 ) . In this process , the more knowledgeable network is referred to as the teacher network , and the smaller network is the student network . Feature maps and attention mechanisms have been demonstrated to be very useful in KD to supervise the training of student networks ( Romero et al. , 2015 ; Zagoruyko & Komodakis , 2017 ; Huang & Wang , 2017 ; Song et al. , 2018 ; Ahn et al. , 2019 ; Heo et al. , 2019 ) . They can help the student network to learn more high-quality intermediate representations , leading to an improved distillation effect and better student network performance ( Romero et al. , 2015 ; Zagoruyko & Komodakis , 2017 ) . KD has also shown its potential in other fields such as adversarial robustness ( Papernot et al. , 2016 ) , multi-granularity lip reading ( Zhao et al. , 2020b ) , and data augmentation ( Bagherinezhad et al. , 2018 ) . In this work , we propose a new backdoor defense technique based on the combination of knowledge distillation and neural attention transfer . 3 PROPOSED APPROACH . In this section , we first describe the defense setting , then introduce the proposed NAD approach . Defense Setting . We adopt a typical defense setting where the defender outsourced a backdoored model from an untrusted party and is assumed to have a small subset of clean training data to finetune the model . The goals of backdoor erasing are to erase the backdoor trigger from the model while retaining the performance of the model on clean samples . 3.1 NEURAL ATTENTION DISTILLATION . Overview . We illustrate the differences between NAD and the traditional finetuning approach in Figure 1 . Instead of using the finetuned network directly as our final model , we employ it as a teacher network and use it in conjunction with the original backdoored network ( i.e . student network ) through an attention distillation process . The job of NAD is to align neurons that are more responsive to the trigger pattern with benign neurons that only responsible for meaningful representations . The main challenge for NAD is thus to find the proper attention representations to distill . Over the next few subsections , we will define the attention representation used throughout our work formally and introduce the loss functions used in the process of attention distillation . Attention Representation . Given a DNN model F , we denote the activation output at the l-th layer as F l ∈ RC×H×W with C , H andW being the dimensions of the channel , the height , and the width of the activation map respectively . We define A : RC×H×W → RH×W to be an attention operator that maps an activation map to an attention representation . Specifically , A takes a 3D activation map F as input and outputs a flattened 2D tensor along the channel dimension . We explore three possible formulations of the attention operator A as suggested in ( Zagoruyko & Komodakis , 2017 ) : Asum ( F l ) = C∑ i=1 ∣∣F li ∣∣ ; Apsum ( F l ) = C∑ i=1 ∣∣F li ∣∣p ; Apmean ( F l ) = 1C C∑ i=1 ∣∣F li ∣∣p , ( 1 ) where F li is the activation map of the i-th channel , | · | is the absolute value function and p > 1 . Intuitively , Asum reflects all activation regions including both the benign and the backdoored neurons . Apsum is a generalized version of Asum that amplifies the disparities between the backdoored neurons and the benign neurons by an order of p. In other words , the larger the p is , the more weight is placed on the parts with highest neuron activations . Amean aligns the activation center of the backdoored neurons with that of the benign neurons by taking the mean over all activation regions . An empirical understanding of the three attention representations is provided in the experiments . Attention Distillation Loss . A detailed structure of our NAD framework is illustrated in Figure 1 ( c ) . For ResNets ( He et al. , 2016 ) , we compute attention representations using one of the proposed attention functions after each group of residual blocks . The teacher network is kept fixed throughout the distillation process . The distillation loss at the l-th layer of the network is defined in terms of the teacher ’ s and the student ’ s attention maps : LNAD ( F lT , F l S ) = ∥∥∥∥∥ A ( F lT ) ∥∥A ( F lT ) ∥∥2 − A ( F l S ) ∥∥A ( F lS ) ∥∥2 ∥∥∥∥∥ 2 , ( 2 ) where ‖·‖2 is the L2 norm andA ( F lT ) /A ( F lS ) is the computed attention maps of the teacher/student network . It is worth mentioning that the normalization of the attention map is crucial to a successful distillation ( Zagoruyko & Komodakis , 2017 ) . Overall Training Loss . The overall training loss is a combination of the cross entropy ( CE ) loss and the sum of the Neural Attention Distillation ( NAD ) loss over all K residual groups : Ltotal = E ( x , y ) ∼D [ LCE ( FS ( x ) , y ) + β · K∑ l=1 LNAD ( F lT ( x ) , F lS ( x ) ) ] , ( 3 ) where LCE ( · ) measures the classification error of the student network , D is a subset of clean data used in finetuning , l is the index of the residual group , and β is a hyperparameter controlling the strength of the attention distillation . Before the attention distillation , a teacher network should be in place . We finetune the backdoored student network on the same clean subset D to obtain the teacher network . We will investigate how the choice of the teacher network affects the performance of NAD in Section 4.4 . We only distill the student network once as our NAD approach is effective enough to remove the backdoor by only a few epochs of distillation . A comprehensive analysis of iteratively applied NAD is also provided in the Appendix G .

Summary: This paper proposes a novel approach to erase backdoor triggers from neural networks through distillation. The defense method, called neural attention distillation (NAD), first finetunes the backdoored model on a set of clean data to get a teacher model. The second part of NAD then finetunes another copy of the original backdoored model (student) on the same clean data while minimizing the difference between the activation maps of the student model and teacher model. The teacher model’s weights are frozen in this distillation phase. The authors show that NAD outperforms or matches previous backdoor defenses over a range of attacks on the CIFAR-10 and GTSRB datasets. Further experiments are also conducted to show NAD’s performance under different conditions such as the percentage of clean data and configuration of teacher-student models.