Adversarial Neuron Pruning Purifies Backdoored Deep Models

1 Introduction . In recent years , deep neural networks ( DNNs ) achieve satisfactory performance in many tasks , including computer vision [ 15 ] , speech recognition [ 45 ] , and gaming agents [ 36 ] . However , their success heavily relies on a large amount of computation and data , forcing researchers to outsource the training in “ Machine Learning as a Service ” ( MLaaS ) platforms or download pretrained models from the Internet , which brings potential risks of training-time attacks [ 17 , 35 , 2 ] . Among them , backdoor attack [ 14 , 6 , 42 ] is remarkably dangerous because it stealthily builds a strong relationship between a trigger pattern and a target label inside DNNs by poisoning a small proportion of the training data . As a result , the returned model always behaves normally on the clean data but is controlled to make target misclassification by presenting the trigger pattern such as a specific single-pixel [ 41 ] or a black-white checkerboard [ 14 ] at the test time . Since DNNs are deployed in many real-world and safety-critical applications , it is urgent to defend against backdoor attacks . While there are many defense methods during training [ 38 , 41 , 8 , 29 , 20 ] , this work focuses on a more realistic scenario in outsourcing training that repairs models after training . In particular , the defenders try to remove the injected backdoor without access to the model training process . Without knowledge of the trigger pattern , previous methods only achieve limited robustness [ 7 , 24 , 22 ] . Some works try to reconstruct the trigger [ 43 , 5 , 33 , 50 , 44 ] , however , the ∗Work was done as an internship at Peking University when he was a student at Tsinghua University . Now , he is a Post-doc at the University of Tokyo . †Corresponding author : Yisen Wang ( yisen.wang @ pku.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . trigger pattern in brand-new attacks becomes natural [ 26 ] , invisible [ 49 ] , and dynamic [ 30 ] , leading to the reconstruction infeasible . Different from them , in this paper , we turn to explore whether we can successfully remove the injected backdoor even without knowing the trigger pattern . Usually , the adversary perturbs inputs to cause misclassification , e.g. , attaching triggers for backdoored models or adding adversarial perturbations . Parallel to this , we are also able to cause misclassification by perturbing neurons of DNNs [ 47 ] . For any neuron inside DNN , we could perturb its weight and bias by multiplying a relatively small number , to change its output . Similar to adversarial perturbations , we can optimize the neuron perturbations to increase its classification loss . Surprisingly , we find that , within the same perturbation budget , backdoored DNNs are much easier to collapse and prone to output the target label than normal DNNs even without the presence of the trigger . That is , the neurons that are sensitive to adversarial neuron perturbation are strongly related to the injected backdoor . Motivated by this , we propose a novel model repairing method , named Adversarial Neuron Pruning ( ANP ) , which prunes most sensitive neurons under the adversarial neuron perturbation . Since the number of neurons is much smaller than weight parameters , e.g. , only 4810 neurons for ResNet-18 while 11M parameters for the same model , our method can work well only based on 1 % of clean data . Our main contributions are summarized as follows : • We find that through adversarially perturbing neurons , backdoored DNNs can present backdoor behaviors even without the presence of the trigger patterns and are much easier to output misclassification than normal DNNs . • To defend backdoor attacks , we propose a simple yet effective model repairing method , Adversarial Neuron Pruning ( ANP ) , which prunes the most sensitive neurons under adversarial neuron perturbations without fine-tuning . • Extensive experiments demonstrate that ANP consistently provides state-of-the-art defense performance against various backdoor attacks , even using an extremely small amount of clean data . 2 Related Work . 2.1 Backdoor Attacks . The backdoor attack is a type of attacks occurring during DNN training . The adversary usually poisons a fraction of training data via attaching a predefined trigger and relabeling them as target labels ( dirty-label setting ) [ 14 , 6 ] . All poisoned samples can be relabeled as a single target class ( all-to-one ) , or poisoned samples from different source classes are relabeled as different target classes ( all-to-all ) [ 30 ] . After training , the model can be controlled to predict the target label in the presence of the trigger at the test time . Different from evasion attacks ( e.g. , adversarial attacks ) [ 3 , 39 , 13 ] , backdoor attacks aim to embed a input- and model-agnostic trigger into the model , which is a severe threat to the applications of deep learning [ 12 ] . Since incorrectly-labeled samples are easy to be detected , some attacks attach the trigger to samples from the target class ( clean-label setting ) [ 35 , 42 , 1 ] . Apart from simple forms like a single-pixel [ 41 ] or a black-and-white checkerboard [ 14 ] , the trigger patterns can be more complex such as a sinusoidal strip [ 1 ] and a dynamic pattern [ 30 ] . These triggers in recent attacks become more natural [ 26 ] and human-imperceptible [ 49 , 31 ] , making them stealthy and hard to be detected by human inspection . Besides , powerful adversaries who have access to the model can optimize the trigger pattern [ 25 ] , and even co-optimize the trigger pattern and the model together to enhance the power of backdoor attacks [ 32 ] . 2.2 Backdoor Defense . Defense during Training . With access to training , the defender is able to detect the poisoned data or make poisoning invalid during the training process . Regarding poisoned data as outliers , we can detect and filter them out using robust statistics in the input space [ 38 , 18 , 8 , 9 ] or the feature space [ 17 , 41 , 28 , 10 ] . Meanwhile , other studies focus on training strategies to make the poisoned data have little or no effect on the trained model [ 23 , 40 ] through randomized smoothing [ 34 , 46 ] , majority vote [ 21 ] , differential privacy [ 29 ] , and input preprocessing [ 27 , 4 ] . Defense after Training . For a downloaded model , we have lost control of its training . To repair the risk model , one direct way is to first reconstruct an approximation of the backdoor trigger via adversarial perturbation [ 43 ] or generative adversarial network ( GAN ) [ 5 , 33 , 50 ] . Once the trigger is reconstructed , it is feasible to prune neurons that are activated in the presence of the trigger , or fine-tune the model to unlearn the trigger [ 43 ] . As the trigger patterns in recently proposed attacks become more complicated such as the dynamics trigger [ 30 ] or natural phenomenon-based trigger [ 26 ] , reconstruction becomes increasingly difficult . There are other studies on trigger-agnostic repairing via model pruning [ 24 ] or fine-tuning [ 7 , 22 ] on the clean data . While they may suffer from severe accuracy degradation when only small clean data are available [ 7 ] . Unlike previous defensive pruning methods which are based on the rule-of-thumb , our proposed method is data-driven and does not require additional fine-tuning . 3 The Proposed Method . 3.1 Preliminary . Deep Neural Network and Its Neurons . In this paper , we take a fully-connected network as an example ( other convolutional networks are also applicable ) . Its layers are numbered from 0 ( input ) to L ( output ) and each layer contains n0 , · · · , nL neurons . The network has n = n1 + n2 + · · ·+ nL parameterized neurons in total3 . We denote the weight parameters of the k-th neuron in l-th layer as w ( l ) k , the bias as b ( l ) k , and its output is h ( l ) k = σ ( w ( l ) > k h ( l−1 ) + b ( l ) k ) , ( 1 ) where σ ( · ) is the nonlinear function and h ( l−1 ) is the outputs of all neurons in the previous layer , i.e. , h ( l−1 ) = [ h ( l−1 ) 1 , · · · , h ( l−1 ) nl ] . For simplicity , we denote all weights of the network as w = [ w ( 1 ) 1 , · · · , w ( 1 ) n1 , · · · , w ( L ) 1 , · · · , w ( L ) nL ] , and all biases as b = [ b ( 1 ) 1 , · · · , b ( 1 ) nl , · · · , b ( L ) 1 , · · · , b ( L ) nL ] . For a given input x , the network makes the prediction f ( x ; w , b ) . DNN Training . We consider a c-class classification problem . The parameters of weights and biases in DNN are learned on a training dataset DT = { ( x1 , y1 ) , · · · , ( xs , ys ) } containing s inputs , where each input is xi ∈ Rd , i = 1 , · · · , s , and its ground-truth label is yi ∈ { 1 , · · · , c } . The training procedure tries to find the optimal ( w ? , b ? ) which minimize the training loss on the training data DT , LDT ( w , b ) = E x , y∼DT ` ( f ( x ; w , b ) , y ) , ( 2 ) where ` ( · , · ) is usually cross entropy loss . Defense Setting . We adopt a typical defense setting that an untrustworthy model is downloaded from a third party ( e.g. , outsourcing training ) without knowledge of the training data DT . For defense , we are assumed to have a small amount of clean data DV . The goals of model repairing are to remove the backdoor behavior while keeping the accuracy on clean samples . 3.2 Adversarial Neuron Perturbations . In previous studies , the adversary causes misclassification by perturbing inputs before feeding them to DNNs . For example , attaching triggers to inputs to make backdoored DNN output the target label or adversarially perturbing inputs to make normal/backdoored DNN output incorrectly . Here , we try another direction to perturb DNN neurons to cause misclassification . Given the k-th neuron in the l-th layers , we can perturb its weight w ( l ) k and bias b ( l ) k by multiplying a small number respectively . As a result , the perturbed weight is ( 1 + δ ( l ) k ) w ( l ) k , the perturbed bias is ( 1 + ξ ( l ) k ) b ( l ) k , and the output of the perturbed neuron becomes h ( l ) k = σ ( ( 1 + δ ( l ) k ) w ( l ) > k h ( l−1 ) + ( 1 + ξ ( l ) k ) b ( l ) k ) , ( 3 ) where δ ( l ) k and ξ ( l ) k indicate the relative sizes of the perturbations to the weight and bias respectively . Similarly , all neurons can be perturbed like Eq . ( 3 ) . We denote the relative weight perturbation size of all neurons as δ = [ δ ( 1 ) 1 , · · · , δ ( 1 ) n1 , · · · , δ ( L ) 1 , · · · , δ ( L ) nL ] , and the relative bias perturbation 3The input neurons are not considered since they have no parameters . 0 0.05 0.10 0.15 0.20 0.25 0.30 Perturbation Budget 0 10 20 30 40 50 Er ro r R at e ( % ) Benign BadNets IAB ( a ) The error rate ( ± std ) of different models under neuron perturbations . 0 0.05 0.10 0.15 0.20 0.25 0.30 Perturbation Budget 0 10 20 30 40 50 Pr ed ict io n Pr op . ( % ) Class 0 Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9 0 0.05 0.10 0.15 0.20 0.25 0.30 Perturbation Budget 0 10 20 30 40 50 Pr ed ict io n Pr op . ( % ) 0 0.05 0.10 0.15 0.20 0.25 0.30 Perturbation Budget 0 10 20 30 40 50 Pr ed ict io n Pr op . ( % ) ( b ) The proportion ( ± std over 5 random runs ) of different classes in predictions by a benign model ( Left ) and two models backdoored by BadNets ( Middle ) and IAB attack ( Right ) under neuron perturbations . Figure 1 : The performance of neuron-perturbed models with different perturbation budgets . size as ξ = [ ξ ( 1 ) 1 , · · · , ξ ( 1 ) n1 , · · · , ξ ( L ) 1 , · · · , ξ ( L ) nL ] for simplicity . For a clean input x , the output of the perturbed model is f ( x ; ( 1 + δ ) w , ( 1 + ξ ) b ) , ( 4 ) where neuron-wise multiplication multiplies the parameters by the perturbation sizes belonging to the same neuron as follows : ( 1 + δ ) w = [ ( 1 + δ ( 1 ) 1 ) w ( 1 ) 1 , · · · , ( 1 + δ ( 1 ) n1 ) w ( 1 ) n1 , · · · , ( 1 + δ ( L ) 1 ) w ( L ) 1 , · · · , ( 1 + δ ( L ) nL ) w ( L ) nL ] , ( 5 ) ( 1 + ξ ) b = [ ( 1 + ξ ( 1 ) 1 ) b ( 1 ) 1 , · · · , ( 1 + ξ ( 1 ) n1 ) b ( 1 ) n1 , · · · , ( 1 + ξ ( L ) 1 ) b ( L ) 1 , · · · , ( 1 + ξ ( L ) nL ) b ( L ) nL ] . ( 6 ) Given a trained model , similar to adversarial perturbations [ 39 , 13 ] , we optimize the neuron perturbations δ and ξ to increase the classification loss on the clean data : max δ , ξ∈ [ − , ] n LDV ( ( 1 + δ ) w , ( 1 + ξ ) b ) , ( 7 ) where is the perturbation budget , which limits the maximum perturbation size . Specifically , we generate adversarial neuron perturbations for three ResNet-18 models [ 15 ] on CIFAR10 [ 19 ] : a benign model , a backdoored one with a predefined trigger ( Badnets [ 14 ] ) , and a backdoored one with a dynamic trigger ( Input-aware Backdoor Attack [ 30 ] , IAB ) . We all set the target label in backdoor as class 0 . We solve the maximization in Eq . ( 7 ) using project gradient descent ( PGD ) with random initialization similar to generating adversarial examples . The number of iterations is 300 and the step size is 0.001 . Figure 1 ( a ) shows adversarial neuron perturbations leads to misclassification , and the larger the perturbation budget is , the larger the error rate is . In addition , backdoored models always have larger error rates with the same perturbation budget compared to the benign one . To explore the misclassification in detail , we illustrate the proportion of different classes in predictions by three models in Figure 1 ( b ) . For benign model , adversarial neuron perturbations mislead the model to output the untargeted class ( e.g. , class 3 ) . While for backdoored models , the majority of misclassified samples are predicted as the target label ( e.g. , class 0 ) whose proportion rate is much higher than the benign model . More results to support these findings can be found in Appendix B . Therefore , even without the trigger , we can still induce the backdoor behaviours via adversarial neuron perturbations . The reasons can be explained as follows : assuming the k-th neuron in l-th layers is backdoor-related , it is dormant on clean sample [ 24 ] ( i.e. , output is close to 0 : h ( l ) k ≈ 0 ) . If w ( l ) > k h l−1 > 0 and b ( l ) k > 0 , we can increase its output by enlarging the norm of the weights ( i.e. , δ ( l ) k > 0 ) , and further continue to increase it by adding a larger bias ( i.e. , ξ ( l ) k > 0 ) 4 . With suitable perturbations , the backdoor-related neuron is activated even on clean samples . Thus , the neurons that are sensitive to adversarial neuron perturbation are strongly related to the injected backdoor .

Summary: This paper proposed Adversarial Neuron Pruning (ANP) to detect the backdoored DNNs. APN prunes the most sensitive neurons under adversarial neuron perturbations. Extensive experiments show that ANP provides state-of-the-art defense performance.