Adversarial Neuron Pruning Purifies Backdoored Deep Models

1 Introduction . In recent years , deep neural networks ( DNNs ) achieve satisfactory performance in many tasks , including computer vision [ 15 ] , speech recognition [ 45 ] , and gaming agents [ 36 ] . However , their success heavily relies on a large amount of computation and data , forcing researchers to outsource the training in “ Machine Learning as a Service ” ( MLaaS ) platforms or download pretrained models from the Internet , which brings potential risks of training-time attacks [ 17 , 35 , 2 ] . Among them , backdoor attack [ 14 , 6 , 42 ] is remarkably dangerous because it stealthily builds a strong relationship between a trigger pattern and a target label inside DNNs by poisoning a small proportion of the training data . As a result , the returned model always behaves normally on the clean data but is controlled to make target misclassification by presenting the trigger pattern such as a specific single-pixel [ 41 ] or a black-white checkerboard [ 14 ] at the test time . Since DNNs are deployed in many real-world and safety-critical applications , it is urgent to defend against backdoor attacks . While there are many defense methods during training [ 38 , 41 , 8 , 29 , 20 ] , this work focuses on a more realistic scenario in outsourcing training that repairs models after training . In particular , the defenders try to remove the injected backdoor without access to the model training process . Without knowledge of the trigger pattern , previous methods only achieve limited robustness [ 7 , 24 , 22 ] . Some works try to reconstruct the trigger [ 43 , 5 , 33 , 50 , 44 ] , however , the ∗Work was done as an internship at Peking University when he was a student at Tsinghua University . Now , he is a Post-doc at the University of Tokyo . †Corresponding author : Yisen Wang ( yisen.wang @ pku.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . trigger pattern in brand-new attacks becomes natural [ 26 ] , invisible [ 49 ] , and dynamic [ 30 ] , leading to the reconstruction infeasible . Different from them , in this paper , we turn to explore whether we can successfully remove the injected backdoor even without knowing the trigger pattern . Usually , the adversary perturbs inputs to cause misclassification , e.g. , attaching triggers for backdoored models or adding adversarial perturbations . Parallel to this , we are also able to cause misclassification by perturbing neurons of DNNs [ 47 ] . For any neuron inside DNN , we could perturb its weight and bias by multiplying a relatively small number , to change its output . Similar to adversarial perturbations , we can optimize the neuron perturbations to increase its classification loss . Surprisingly , we find that , within the same perturbation budget , backdoored DNNs are much easier to collapse and prone to output the target label than normal DNNs even without the presence of the trigger . That is , the neurons that are sensitive to adversarial neuron perturbation are strongly related to the injected backdoor . Motivated by this , we propose a novel model repairing method , named Adversarial Neuron Pruning ( ANP ) , which prunes most sensitive neurons under the adversarial neuron perturbation . Since the number of neurons is much smaller than weight parameters , e.g. , only 4810 neurons for ResNet-18 while 11M parameters for the same model , our method can work well only based on 1 % of clean data . Our main contributions are summarized as follows : • We find that through adversarially perturbing neurons , backdoored DNNs can present backdoor behaviors even without the presence of the trigger patterns and are much easier to output misclassification than normal DNNs . • To defend backdoor attacks , we propose a simple yet effective model repairing method , Adversarial Neuron Pruning ( ANP ) , which prunes the most sensitive neurons under adversarial neuron perturbations without fine-tuning . • Extensive experiments demonstrate that ANP consistently provides state-of-the-art defense performance against various backdoor attacks , even using an extremely small amount of clean data . 2 Related Work . 2.1 Backdoor Attacks . The backdoor attack is a type of attacks occurring during DNN training . The adversary usually poisons a fraction of training data via attaching a predefined trigger and relabeling them as target labels ( dirty-label setting ) [ 14 , 6 ] . All poisoned samples can be relabeled as a single target class ( all-to-one ) , or poisoned samples from different source classes are relabeled as different target classes ( all-to-all ) [ 30 ] . After training , the model can be controlled to predict the target label in the presence of the trigger at the test time . Different from evasion attacks ( e.g. , adversarial attacks ) [ 3 , 39 , 13 ] , backdoor attacks aim to embed a input- and model-agnostic trigger into the model , which is a severe threat to the applications of deep learning [ 12 ] . Since incorrectly-labeled samples are easy to be detected , some attacks attach the trigger to samples from the target class ( clean-label setting ) [ 35 , 42 , 1 ] . Apart from simple forms like a single-pixel [ 41 ] or a black-and-white checkerboard [ 14 ] , the trigger patterns can be more complex such as a sinusoidal strip [ 1 ] and a dynamic pattern [ 30 ] . These triggers in recent attacks become more natural [ 26 ] and human-imperceptible [ 49 , 31 ] , making them stealthy and hard to be detected by human inspection . Besides , powerful adversaries who have access to the model can optimize the trigger pattern [ 25 ] , and even co-optimize the trigger pattern and the model together to enhance the power of backdoor attacks [ 32 ] . 2.2 Backdoor Defense . Defense during Training . With access to training , the defender is able to detect the poisoned data or make poisoning invalid during the training process . Regarding poisoned data as outliers , we can detect and filter them out using robust statistics in the input space [ 38 , 18 , 8 , 9 ] or the feature space [ 17 , 41 , 28 , 10 ] . Meanwhile , other studies focus on training strategies to make the poisoned data have little or no effect on the trained model [ 23 , 40 ] through randomized smoothing [ 34 , 46 ] , majority vote [ 21 ] , differential privacy [ 29 ] , and input preprocessing [ 27 , 4 ] . Defense after Training . For a downloaded model , we have lost control of its training . To repair the risk model , one direct way is to first reconstruct an approximation of the backdoor trigger via adversarial perturbation [ 43 ] or generative adversarial network ( GAN ) [ 5 , 33 , 50 ] . Once the trigger is reconstructed , it is feasible to prune neurons that are activated in the presence of the trigger , or fine-tune the model to unlearn the trigger [ 43 ] . As the trigger patterns in recently proposed attacks become more complicated such as the dynamics trigger [ 30 ] or natural phenomenon-based trigger [ 26 ] , reconstruction becomes increasingly difficult . There are other studies on trigger-agnostic repairing via model pruning [ 24 ] or fine-tuning [ 7 , 22 ] on the clean data . While they may suffer from severe accuracy degradation when only small clean data are available [ 7 ] . Unlike previous defensive pruning methods which are based on the rule-of-thumb , our proposed method is data-driven and does not require additional fine-tuning . 3 The Proposed Method . 3.1 Preliminary . Deep Neural Network and Its Neurons . In this paper , we take a fully-connected network as an example ( other convolutional networks are also applicable ) . Its layers are numbered from 0 ( input ) to L ( output ) and each layer contains n0 , · · · , nL neurons . The network has n = n1 + n2 + · · ·+ nL parameterized neurons in total3 . We denote the weight parameters of the k-th neuron in l-th layer as w ( l ) k , the bias as b ( l ) k , and its output is h ( l ) k = σ ( w ( l ) > k h ( l−1 ) + b ( l ) k ) , ( 1 ) where σ ( · ) is the nonlinear function and h ( l−1 ) is the outputs of all neurons in the previous layer , i.e. , h ( l−1 ) = [ h ( l−1 ) 1 , · · · , h ( l−1 ) nl ] . For simplicity , we denote all weights of the network as w = [ w ( 1 ) 1 , · · · , w ( 1 ) n1 , · · · , w ( L ) 1 , · · · , w ( L ) nL ] , and all biases as b = [ b ( 1 ) 1 , · · · , b ( 1 ) nl , · · · , b ( L ) 1 , · · · , b ( L ) nL ] . For a given input x , the network makes the prediction f ( x ; w , b ) . DNN Training . We consider a c-class classification problem . The parameters of weights and biases in DNN are learned on a training dataset DT = { ( x1 , y1 ) , · · · , ( xs , ys ) } containing s inputs , where each input is xi ∈ Rd , i = 1 , · · · , s , and its ground-truth label is yi ∈ { 1 , · · · , c } . The training procedure tries to find the optimal ( w ? , b ? ) which minimize the training loss on the training data DT , LDT ( w , b ) = E x , y∼DT ` ( f ( x ; w , b ) , y ) , ( 2 ) where ` ( · , · ) is usually cross entropy loss . Defense Setting . We adopt a typical defense setting that an untrustworthy model is downloaded from a third party ( e.g. , outsourcing training ) without knowledge of the training data DT . For defense , we are assumed to have a small amount of clean data DV . The goals of model repairing are to remove the backdoor behavior while keeping the accuracy on clean samples . 3.2 Adversarial Neuron Perturbations . In previous studies , the adversary causes misclassification by perturbing inputs before feeding them to DNNs . For example , attaching triggers to inputs to make backdoored DNN output the target label or adversarially perturbing inputs to make normal/backdoored DNN output incorrectly . Here , we try another direction to perturb DNN neurons to cause misclassification . Given the k-th neuron in the l-th layers , we can perturb its weight w ( l ) k and bias b ( l ) k by multiplying a small number respectively . As a result , the perturbed weight is ( 1 + δ ( l ) k ) w ( l ) k , the perturbed bias is ( 1 + ξ ( l ) k ) b ( l ) k , and the output of the perturbed neuron becomes h ( l ) k = σ ( ( 1 + δ ( l ) k ) w ( l ) > k h ( l−1 ) + ( 1 + ξ ( l ) k ) b ( l ) k ) , ( 3 ) where δ ( l ) k and ξ ( l ) k indicate the relative sizes of the perturbations to the weight and bias respectively . Similarly , all neurons can be perturbed like Eq . ( 3 ) . We denote the relative weight perturbation size of all neurons as δ = [ δ ( 1 ) 1 , · · · , δ ( 1 ) n1 , · · · , δ ( L ) 1 , · · · , δ ( L ) nL ] , and the relative bias perturbation 3The input neurons are not considered since they have no parameters . 0 0.05 0.10 0.15 0.20 0.25 0.30 Perturbation Budget 0 10 20 30 40 50 Er ro r R at e ( % ) Benign BadNets IAB ( a ) The error rate ( ± std ) of different models under neuron perturbations . 0 0.05 0.10 0.15 0.20 0.25 0.30 Perturbation Budget 0 10 20 30 40 50 Pr ed ict io n Pr op . ( % ) Class 0 Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9 0 0.05 0.10 0.15 0.20 0.25 0.30 Perturbation Budget 0 10 20 30 40 50 Pr ed ict io n Pr op . ( % ) 0 0.05 0.10 0.15 0.20 0.25 0.30 Perturbation Budget 0 10 20 30 40 50 Pr ed ict io n Pr op . ( % ) ( b ) The proportion ( ± std over 5 random runs ) of different classes in predictions by a benign model ( Left ) and two models backdoored by BadNets ( Middle ) and IAB attack ( Right ) under neuron perturbations . Figure 1 : The performance of neuron-perturbed models with different perturbation budgets . size as ξ = [ ξ ( 1 ) 1 , · · · , ξ ( 1 ) n1 , · · · , ξ ( L ) 1 , · · · , ξ ( L ) nL ] for simplicity . For a clean input x , the output of the perturbed model is f ( x ; ( 1 + δ ) w , ( 1 + ξ ) b ) , ( 4 ) where neuron-wise multiplication multiplies the parameters by the perturbation sizes belonging to the same neuron as follows : ( 1 + δ ) w = [ ( 1 + δ ( 1 ) 1 ) w ( 1 ) 1 , · · · , ( 1 + δ ( 1 ) n1 ) w ( 1 ) n1 , · · · , ( 1 + δ ( L ) 1 ) w ( L ) 1 , · · · , ( 1 + δ ( L ) nL ) w ( L ) nL ] , ( 5 ) ( 1 + ξ ) b = [ ( 1 + ξ ( 1 ) 1 ) b ( 1 ) 1 , · · · , ( 1 + ξ ( 1 ) n1 ) b ( 1 ) n1 , · · · , ( 1 + ξ ( L ) 1 ) b ( L ) 1 , · · · , ( 1 + ξ ( L ) nL ) b ( L ) nL ] . ( 6 ) Given a trained model , similar to adversarial perturbations [ 39 , 13 ] , we optimize the neuron perturbations δ and ξ to increase the classification loss on the clean data : max δ , ξ∈ [ − , ] n LDV ( ( 1 + δ ) w , ( 1 + ξ ) b ) , ( 7 ) where is the perturbation budget , which limits the maximum perturbation size . Specifically , we generate adversarial neuron perturbations for three ResNet-18 models [ 15 ] on CIFAR10 [ 19 ] : a benign model , a backdoored one with a predefined trigger ( Badnets [ 14 ] ) , and a backdoored one with a dynamic trigger ( Input-aware Backdoor Attack [ 30 ] , IAB ) . We all set the target label in backdoor as class 0 . We solve the maximization in Eq . ( 7 ) using project gradient descent ( PGD ) with random initialization similar to generating adversarial examples . The number of iterations is 300 and the step size is 0.001 . Figure 1 ( a ) shows adversarial neuron perturbations leads to misclassification , and the larger the perturbation budget is , the larger the error rate is . In addition , backdoored models always have larger error rates with the same perturbation budget compared to the benign one . To explore the misclassification in detail , we illustrate the proportion of different classes in predictions by three models in Figure 1 ( b ) . For benign model , adversarial neuron perturbations mislead the model to output the untargeted class ( e.g. , class 3 ) . While for backdoored models , the majority of misclassified samples are predicted as the target label ( e.g. , class 0 ) whose proportion rate is much higher than the benign model . More results to support these findings can be found in Appendix B . Therefore , even without the trigger , we can still induce the backdoor behaviours via adversarial neuron perturbations . The reasons can be explained as follows : assuming the k-th neuron in l-th layers is backdoor-related , it is dormant on clean sample [ 24 ] ( i.e. , output is close to 0 : h ( l ) k ≈ 0 ) . If w ( l ) > k h l−1 > 0 and b ( l ) k > 0 , we can increase its output by enlarging the norm of the weights ( i.e. , δ ( l ) k > 0 ) , and further continue to increase it by adding a larger bias ( i.e. , ξ ( l ) k > 0 ) 4 . With suitable perturbations , the backdoor-related neuron is activated even on clean samples . Thus , the neurons that are sensitive to adversarial neuron perturbation are strongly related to the injected backdoor .

Summary: This paper proposes a simple yet effective method to purify the injected backdoor based on a small portion of clean data without access to training phase. The proposed method dubbed Adversarial Neuron Pruning (ANP) prunes some sensitive neurons to defend backdoor attacks. Empirical results validate ANP can significantly improve robustness against various backdoor attacks.