Adversarial Neuron Pruning Purifies Backdoored Deep Models
Authors: Dongxian Wu
Yisen Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: backdoor attacks
backdoor defense
AI security
trustworthy machine learning

Abstract: As deep neural networks (DNNs) are growing larger, their requirements for computational resources become huge, which makes outsourcing training more popular. Training in a third-party platform, however, may introduce potential risks that a malicious trainer will return backdoored DNNs, which behave normally on clean samples but output targeted misclassifications whenever a trigger appears at the test time. Without any knowledge of the trigger, it is difficult to distinguish or recover benign DNNs from backdoored ones. In this paper, we first identify an unexpected sensitivity of backdoored DNNs, that is, they are much easier to collapse and tend to predict the target label on clean samples when their neurons are adversarially perturbed. Based on these observations, we propose a novel model repairing method, termed Adversarial Neuron Pruning (ANP), which prunes some sensitive neurons to purify the injected backdoor. Experiments show, even with only an extremely small amount of clean data (e.g., 1%), ANP effectively removes the injected backdoor without causing obvious performance degradation.

Adversarial Neuron Pruning Purifies Backdoored Deep Models
Dongxian Wul* Yisen Wang
Dept. of Computer Science and Technology, Tsinghua University, China Key Lab. of Machine Perception, School of Artificial Intelligence, Peking Univesity, China Instittte for Artificial Intelligence, Peking Univesity, China
Abstract
As deep neural networks (DNNs) are growing larger, their requirements for compu- tational resources become huge, which makes outsoureing training more popular. Training in a third-party platform. however, may introduce potential risks that a malicious trainer will return backdoored DNNs, which behave normally on clean samples but output targeted misclassifications whenever trigger appears at the test time. Without any knowledge of the trigger, is difficult distinguish or recover benign DNNs from backdoored ones. In this paper, we first identify an unexpected sensitivity of backdoored DNNs, that is, they are much easier to col- lapse and tend to predict the target label on clean samples when their neurons are adversarially perturbed. Based on these observations, we propose a novel model repairing method, termed Adversarial Neuron Pruning (ANP), which prunes some sensitive neurons to purify the injected backdoor. Experiments show, even with only an extremely small amount of clean data (e.g.. 1%), ANP effectively removes the injected backdoor without causing obvious performance degradation. Our code is available athttss:/ /gi thub. com/ cttp//ssoonancac//eeeeee AMP _bckdooaccorrrrr
Introduction
In recent years, deep neural networks (DNNs) achieve satisfactory performance in many tasks, including computer vision [15], speech recognition [45]. and gaming agents [36]. However, their success heavily relies large amount of computation and data, forcing researchers outsource the training in "Machine Learning as a Service" (MLaaaS) platforms or download pretrained models from the Internet, which brings potential risks of training-time attacks [17 35 2 Among them, backdoor attack [4
is remarkably dangerous because it stealthily builds a strong relationship between a trigger pattern and a target label inside DNNs by poisoning small proportion of the training data. As result, the returned model always behaves normally on the clean data but is controlled to make target misclassification by presenting the trigger pattern such as specific single-pixel 4 black-white checkerboard 14 at the test time.
Since DNNs deployed in many real-world and safety- cafety-critical applications, it is urgent defend against backdoor attacks. While there are many defense methods during training [8 this work focuses
20), more realistic scenario in outsourcing training that repairs models after training. In particular, the defenders try to remove the injected backdoor without access to the model training process. Without knowledge of the trigger pattern, previous methods only achieve limited
29
robustness 7 24 22 Some works try to reconstruct the trigger 4 5 33 50, 44), however, the
"Wrk was done internship Peking University when he student at Tsinghua University. Now, he is Post-doc the University of Tokyo. Corresponniig author: Yisen Wang (yisen. wanen.wangGe pku.edu.cn).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

trigger pattern in brand-new attacks becomes natural [6], invisible 49] and dynamic [30, leading to the reconstruction infeasible. Different from them, in this paper, we turn explore whether we can successfully remove the injected backdoor even without knowing the trigger pattern.
Usually, the adversary perturbs inputs cause misclassification, e.g., attaching triggers for back- doored models or adding adversarial perturbations. Parallel to this, we are also able to cause misclassification by perturbing neurons of DNNs [47). For any neuron inside DNN, we could per- turb its weight and bias by multiplying a relatively small number, to change its output. Similar to adversarial perturbations, we can optimize the neuron perturbations increase its classification loss. Surprisingly, we find that, within the same perturbation budget, backdoored DNNs are much easier to collapse and prone to output the target label than normal DNNs even without the presence of the trigger. That is, the neurons that are sensitive to adversarial neuron perturbation are strongly related to the injected backdoor. Motivated by this, we propose a novel model repairing method, named Adversarial Neuron Pruning (ANP), which prunes most sensitive neurons under the adversarial neuron perturbation. Since the number of neurons is much smaller than weight parameters, e.g., only 48 10 neurons for ResNet-18 while 1 1M parameters for the same model, our method can work well only based on I % of clean data. Our main contributions are summarized as follows:
We find that through adversarially perturbing neurons, backdoored DNNs present backdoor behaviors even without the presence of the trigger patterns and are much easier to output misclassification than normal DNNs.
To defend backdoor attacks, propose simple yet effective model repairing method, Ad- versarial Neuron Praning (ANP), which prunes the most sensitive neurons under adversarial neuron perturbations without fine-tuning.
Extensive experiments demonstrate that ANP consistently provides state-of-the-art defense performance against various backdoor attacks, even using an extremely small amount of clean data.
2 Related Work
2.1 Backdoor Attacks
The backdoor attack is a type of attacks occurring during DNN training. The adversary usually poisons a fraction of training data via attaching predefined trigger and relabeling them target labels (dirty-label setting) 14] 6). All poisoned samples can be relabeled as a single target class (all-to-one), poisoned samples from different source classes are relabeled as different target classes (all-to-all) [30]. After training, the model can be controlled to predict the target label in the presence of the trigger at the test time. Different from evasion attacks (e.g., adversarial attacks) 3 39 [3]
backdoor attacks aim to embed a input- and model-agnostic trigger into the model, which is a severe threat to the applications of deep learning
[12] Since pooorrectly-labllee samples are easy to be detected, some attacks attach the trigger to samples from the target class (clean-label setting)
[35 42 D. Apart from simple forms like single-pixel 4 or a black-and white checkerboard [4] the trigger patterns can be more complex such as a sinusoidal strip [1] and dynamic pattern 30
These triggers in recent attacks become more natural 26 and numan-imperceetion (49 31, making them stealthy and hard to be detected by human inspection. Besides, powerful adversaries who have access to the model can optimize the trigger pattern 25), and even co-optimize the trigger pattern
and the model together to enhance the power of backdoor attacks [32]
2.2 Backdoor Defense
Defense during Training, With access to training, the defender is able to detect the poisoned data or make poisoning invalid during the training process. Regarding poisoned data as outliers, we can detect and filter them out using robust statistics in the input space [8 [8 g
or the feature space [1] 28 10). Meanwhile, other studies focus training strategies to make the poisoned data have
little or no effect on the trained model [23] 40) through randomized smoothing [34] 46), majority
vote 22 differential privacy 22). and input preprocessing 27
Defense after Training. For a downloaded model, we have lost control of training. To repair the risk model, one direct way is to first reconstruct approximation of the backdoor trigger via

---

adversarial perturbation 43 reconstructed, it is feasible to prune neurons that are activated the presence of the trigger. or fine-tune the model to unlearn the trigger 43 As the trigger patterns in recently proposed attacks become more complicated such as the dynamics trigger (30] or natural phenomenon-ban baased trigger (26). reconstruction becomes increasingly difficult. There are other studies on trigger-agnostie repairing via model pruning [24] or fine-tuning 7 [2] on the clean data. While they may suffer from severe accuracy degradation when only small clean data are available 77. Unlike previous defensive pruning methods which are based on the rule-of-thumb, our proposed method is data-driven and does not require additional fine-tuning.
generative adversarial network (GAN) S 33 50].
Once the trigger
3
The Proposed Method
3.1 Preliminary
Deep Neural Network and Its Neurons. In this paper, we take fully-connected network as an example (other convolutional networks are also applicable). Its layers are numbered from 0 (input) to (output) and each layer contains no. ny neurons. The network has n nn + n2 + ........ n parameterized neurons in total] We denote the weight parameters of the k-th neuron in I-th layer as w the bias 6 and its output is
oW (1) h1-1)) +b6)) b)
where (-) is the nonlinear function and (1-1) is the outputs of all neurons in the previous layer, (e-.h(11)) [h1 (1-1) 1 (1-1)) For simplicity,
denote all weights of the network
w (1)) and all biases as b = [1(2) For given input x, the network makes the prediction f(x; w, b).
[w]]] , W
w|
lng (()) Daz
DNN Training. We consider a c-class classification problem. The parameters of weights and biases in DNN are learned on training dataset Dy = {(x1 (()1 V/),.... (x,, ys)} containing inputs, where each input is x, E Rd, = -,.... ;8, and its ground-truth label is yi E {I..... c}. The training procedure tries to find the optimal (w*, bb) which minimize the training loss on the training data Dr
(D, b) = E x.y-Dr
E(f(x: w, b).
where E(:,)) is usually cross entropy loss.
Defense Setting. We adopt typical defense setting that an untrustworthy model is downloaded from a third party (e.g., outsourcing training) without knowledge of the training data Dy. For defense, we are assumed to have a small amount of clean data Dy. The goals of model repairing are to remove the backdoor behavior while keeping the accuracy on clean samples.
3.2 Adversarial Neuron Perturbations
In previous studies, the adversary causes misclassification by perturbing inputs before feeding them to DNNs. For example, attaching triggers inputs to make backdoored DNN output the target label or adversarially perturbing inputs make normal/bacckooore DNN output incorrectly.
Here, try another direction to perturb DNN neurons cause misclassification. Given the k-th neuron in the 1-th layers, we can perturb its weight w and bias (()) by multiplying small number
respectively. As a result, the perturbed weight is + (() yw and the output of the perturbed neuron becomes
(!) the perturbed bias is (1 (1+ (6))6())
h = ((1 + 8 W
(- + (1+ (1++5)) (0)),
(3)
where 5(0) and (() indicate the relative sizes the perturbations to the weight and bias respectively. Similarly, all neurons can be perturbed like Eq. (3). We denote the relative weight perturbation
size of all neurons as 18(1) o/
g(()) sl1. and the relative bias perturbation
'The input neurons considered since they have parameters.
3

---

Benign Beditiis IAB
dass
Class
dass
(lass 5
dass 2
Class 7
dass
Class
Class
Emnr
20
0.30 015
Perturbatio Budgule
Pertrrbation Sutget
Perturbation budgut
Perturbation Budget
(a) The error rate std) of different models under neuron perturbations.
(b) The proportion (#t std over random runs) of different classes in predictions by a benign model (Left) two models backdoored by BadNets (Middle) IAB attack (Right) under neuron perturbations.
Figure I: The performance neuron-perturbed models with different perturbation budgets.
size as & = 14 perturbed model
(L)
<(L) for simplicity. For a clean input x, the output of the
f(x: (1 + 6) O w, (1 +) b),
(4)
where neuron-We wise multiplication @ multiplies the parameters by the perturbation sizes belonging to the same neuron follows:
o w = [1 +
(1 6 ((2) ( (1 + 0( | (1 )
[w!! (L)
(5)
(1 5 ④ [1 + c 10
1 (1 104 ( ... (1 + 1 (6)
Given trained model, similar to adversarial perturbations [39 13. optimize the neuron perturba- tions and g to increase the classification loss on the clean data:
max 6.EE|-c,0]
LD, ((1 + @ w, + ()+
where is the perturbation budget, which limits the maximum perturbation size.
Specifically, we generate adversarial neuron perturbations for three ResNet- 18 models 15] on CIFAR- 10 [19] a benign model, backdoored one with predefined trigger (Badnets [14). and backdoored one with a dynamic trigger (Input-aware Backdoor Attack [30] IAB). We all set the target label in backdoor class O. We solve the maximization in Eq. 7 using project gradient descent (PGD) with random initialization similar to generating adversarial examples. The number of iterations is 300 and the step size is 0.001. Figure [(a] shows adversarial neuron perturbations leads misclassification, and the larger the perturbation budget is, the larger the error rate is. In addition, backdoored models always have larger error rates with the same perturbation budget compared the benign one. To explore the misclassification in detail, we illustrate the proportion of different classes in predictions by three models in Figure I(b) For benign model, adversarial neuron perturbations mislead the model to output the untargeted class (e.g., class 3). While for backdoored models, the majority of misclassified samples are predicted the target label (e.g., class whose proportion rate is much higher than the benign model. More results to support these findings can be found in Appendix Therefore, even without the trigger, we can still induce the backdoor behaviours via adversarial neuron perturbations. The reasons can be explained as follows: assuming the k-th neuron in 1-th
layers is backdoor-related. it is dormant on clean sample 24 (i.e., output close to O: n
(1)
0). If
w (1yT h- 0 and b/4) > O, we can increase its output by enlarging the norm of the weights (i.e.,
> 0). and further continue to increase it by adding a larger bias (i.e., e >> 0f With suitable perturbations, the backdoor-relane neuron is activated even clean samples. Thus, the neurons that are sensitive to adversarial neuron perturbation are strongly related to the injected backdoor.
3.3 The Proposed Adversarial Neuron Pruning (ANP)
As mentioned above, the sensitivity under adversarial neuron perturbations is strongly related to the injected backdoor. Inspired by this, we try to prune some sensitive neurons to defend backdoor
'If W (1)T h or 6 < O, still can increase its output by reducing the norm of the weight bias.

---

attacks, named Adversarial Neuron Pruning method (ANP). We denote the pruning masks as m (1) mi (L) m [0, For the k-th neuron in the (-th layer, we its
[m] May
weight W k = 0 if m = 0, and keep it unchanged if m = 1. Similar to Ghorbani and Zou
we always keep the bias g to avoid extra harm to accuracy on clean data
Continuous Relaxation and Optimization. Due to the binary masks, pruning is a discrete opti- mization problem that difficult solve within feasible time. To address this, we apply continuous relaxation to let m E [0, 1J and optimizes them using projected gradient descent (PGD). To restore the continuous masks discreteness after optimization, we all mask smaller than threshold (prune neurons with small masks), and others as (keep neurons with large masks). We expect the pruned model not only behaves well on clean data but also keep stable under adversarial neuron perturbations. Therefore, we solve the following minimization problem,
min me 10
aL (m W, ( aa)
LDV 8.EE(-c.r)"
max
(8)
where a E [0, 1] is a trade-off coefficient. If a is close to 1, the minimization object focuses more on accuracy of the pruned model on clean data, while if a is close it focuses more on robustness against backdoor attacks.
Implementation of ANP. As shown in Algorithm 1 we start from unpruned network, that is, initializing all masks as (Line 2). We randomly initialize perturbations (Line 5), and update them using step (Line 6-7), then project them onto the feasible regime (Line 8-9). Following that, update mask values based on the adversarial neuron perturbation (Line 10) and project mask value onto [0, 1] (Line 11). As convergence after iterations, we pruned neurons with small mask value and keep others unchanged (Line 13). Note that the proposed ANP is data-driven, different from previous pruning- based defenses which are based on the thumb-of-rule 24 43). Besides, ANP can work well on an extremely small amount of clean data due to the small number of masks.
Algorithm 1 Adversarial Neuron Pruning (ANP)
Input: Network f(; w, b). hyper-parameter a, learning rate 1), batch size b, maximum perturba-
tion size
2: Initialize all elements m as
3: repeat
4: Read mini-batch B [(x), 91), (X6. }6)} from training set
5: do. Eo U(-e,e), E). where U (-E,e) is the uniform distribution
6:
+ 8, + esign( V V
8,) w,
+ b))
7:
x-= Eu esign( C 6,) (1+ (
8:
+ (enx(-e,min (d,))
9:
g-
10: m + m - V/ O w,b) + (1 o) 0w (1+ bb]
11:
max(0, min(m,
12:
until training converged
(1) [[mm threshold), for all k
13:
my
14: Output: A robust network f(; m 3 w, b) against backdoor attacks
Adaptation to Batch Normalization. Batch Normalization (BatchNorm) [16] always normalizes its input and controls the mean and variance of the output by a pair of trainable parameters (scale ? and shift 3). If BatchNorm is inserted between matrix multiplication and the nonlinear activation, the perturbations to weight and bias may cancel each other out. For example, if we perturb the weight and bias of neuron to the maximum by multiplying (1 + e), the normalization offsets them and nothing changes after BatchNorm. To address this problem, we perturb the scale and shift parameters instead. We also make similar changes to the pruning masks.
We can still compress the model since these biases can be absorbed by their following layers.
5

---

Table 1: Performance (average over random runs) of 4 defense methods against backdoor attacks on 1% (500 images) of clean data on CIFAR-IB training set using ResNet-18. The ArgDrop indicates the average changes in ACC or ASR over 6 backdoor attacks compared to no defense results (Before).
Metric
Defense
Badnets
Blend
IAB-one
IAB-all
CLB
SIG
AvgDrop
Before
93.73
94.82
93.89
94.10
93.78
93.64
FT(Ir 0.01)
90.48
92.12
88.68
89.06
91.26
91.19
4 3.53
F 0.02) FP
87.23
88.98
84.85
83.77
88.25
88.63
[ 7.04
ACC
92.18
92.40
91.57
92.28
91.91
91.64
2.00
MCR(t 0.3)
85.95
88.26
86.30
84.53
86.87
85.88
7.70
ANP
90.20
93.44
92.62
92.79
92.67
93.40
+ 1.47
Before
99.97
100.0
98.49
92.88
99.94
94.26
FT((r 0.01)
11.70
47.17
0.99
1.36
12.51
0.40
85.24
FT(rr 0.02)
2.95
10.20
1.70
1.83
1.17
0.39
94.55
ASR
FP
5.34
65.39
20.73
32.36
3.40
0.32
76.33
MCR( 0.3)
5.70
13.57
30.23
35.17
12.77
0.52
81.26
ANP
0.45
0.46
0.88
0.86
3.98
0.28
96.44
4
Experiments
In this section, we conduct comprehensive experiments to evaluate the effectiveness of ANP, including its benchmarking robustmess, ablation studies and performance under limited computation resources.
4.1 Experimental Settings
Backdoor Attacks and Settings. We consider state-of-the-art backdoor attacks: 1) BadNets
[4]. 2) Blend backdoor attack (Blend) [6], 3) Input-aware backdoor attack with all-to-one target label (LAB-one) all-to-all target label (IAB-all) [00, 4) Clean-label backdoor (CLB) 42), and 5) Sinusoidal signal backdoor attack (SIG) (1] For fair comparisons, we follow the default configuration in their original papers such the trigger patterns, the trigger sizes and the target labels. We evaluate the performance of all attacks and defenses on CIFAR-10 [19] using ResNet- [5] as the base model. We 90% training data to train the backdoored DNNs and use the all or part of the remaining 10% training data for defense. More details about implementation can be found in Appendix A
Backdoor Defenses and Settings. We compare proposed ANP with 3 existing model repairing methods: 1) standard fine-tuning (FT), 2) fine-prumn ten-ppunnnng (FP) [24]. and 3) mode connectivity repair (MCR) 48 All defense methods assumed to have access to the same 1% of clean training data (500 images). The results based on 10% (5000 images) and 0.1% (50 images) of clean training data be found in Appendix. For ANP, optimize all masks using Stochastic Gradient Descent (SGD) with the perturbation budget = 0.4 and the trade-off coefficient = 0.2. We set the batch size 128, the constant learning rate 0.2, and the momentum 0.9 for 2000 iterations in total. Typical data augmentation like random crop and horizontal flipping are applied. After optimization, neurons with mask value smaller than 0.2 are pruned.
Evaluation Metrics. We evaluate the performance of different defenses using two metrics: 1) the accuracy clean data (ACC), and 2) and the attack success rate (ASR) that is the ratio of triggered samples that are misclassified as the target label. For better comparison between different strategies for the target label (e.g., all-to-one and all-to-all), already belong to the target class in the all-to-one setting before calculating ASR. Therefore, an ideal defense always has close-to-zero ASR and high ACC.
remove the samples whose ground-truth labels
4.2 Benchmarking the State-of-the-art Robustness
To verify the effectiveness of the proposed ANP, we compare its performance with other existing model repairing methods using ACC and ASR in T All experiments are repeated over runs with different random seeds, and we only report the average performance without the standard deviation due to the space constraint. More detailed results including the standard deviation can be found in Appendix

---

Table 2: The average training time of defense method against 6 backdoor attacks on 1% (500 images) of clean data on CIFAR-10 training set using ResNet-18.
Defense
FT
FP
MCR
ANP
Time (s) 93.80
1427.1
286.1
241.5
The experimental results show that the proposed ANP remarkably provides almost the highest robustness against several state-of-the-art backdoor attacks. In particular, in 5 of total 6 attacks, our proposed ANP successfully reduces the ASR to lower than 1% while only has a slight drop (~ 1.47%
on average) in ACC. Note that we only use 1% of clean data from CIFAR-10 10 training set, which is extremely small. We find the standard fnn-ttunig (FT) with a larger learning rate (Ir = 0.02) also provides strong robustness similar to ANP. However, it has obvious drop in ACC since large learning rate may destroy the features learned before and the new ones learned on limited data are poor in generalization. The poor accuracy hinders its usage in practical scenarios. In contrast, ANP has similar robustness against backdoor attacks while maintains ACC at a relatively high level at the same time. Similar to our method, Fine-pruning (FP) [24] also prunes some neurons and fine-tunes the pruned model on clean data. Specifically, It only prunes neurons in the last convolutional layer using rule-of-thumt and provides limited robustness against backdoor attacks. Differently, the proposed ANP is data-drive duta-driven (i.e., optimizing pruning masks based on data) and prunes neurons globally (i.e., pruning neurons in all layers), leading higher ACC and lower ASR than FP. MCR suggests mitigating backdoor by using a curve model lying on a path connection between two backdoored models in the loss landscapes. However, with limited data, it fails in providing high robustness against complex triggers (e.g. random trigger sampled from Gaussian in blend backdoor attack or the dynamic trigger input-aware attack). By contrast, ANP always reduces ASR significantly no matter what the trigger looks like. Note that ANP only prunes neurons and never changes any parameters of backdoored DNNs, while other methods all fine-tune parameters. Under this situation, ANP still provides the almost best robustness against these sfate-of-the-art attacks, which indicates that pure pruning (without fine-tuning) is still promising defense against backdoor attacks.
We also compare the training time of ANP and other baselines in Table 2 which indicates the efficiency of the proposed ANP, In particular, we apply 2000 iterations for FT, FP, and ANP, and 200 epochs (i.e.. 800 iterations) for MCR following the open-sourcen code] Note that ANP is not as slow as the vanilla adversarial training (AT) since ANP only requires one extra backpropagation while PGD inside AT usually needs 10 extra backpropagations (PGD-10) in each update of model parameters. Besides, although ANP takes 2.5x time compared to FT, it improves accuracy (e.g.. 88.98% -> 93.44% for Blend) and reduce vulnerability (c.g..0..001 + 0.46% for Blend) significantly, making the overhead caused by ANP acceptable.
4.3 Ablation Studies
we apply adversarial neuron perturbations to induce the injected backdoor and then defend against backdoor attacks. Here, we evaluate the effect of the adversarial neuron perturbation in our defense strategy. In the following experiments, we defend against Badnets attack by default, unless otherwise specified. First, we optimize masks with perturbations (a = 0.2 and 0.4, i.e., ANP) and without perturbations (a = 1.0, i.e., the vanilla pruning method). Figure 2(a) shows the ACC and ASR on the test set during optimization. We find the optimization under adversarial neuron perturbations helps the backdoored model suppress ASR significantly while the vanilla one always has high ASR throughout the training process. Next, we illustrate the mask distribution after optimization. Figure ( illustrates a certain fraction of masks becomes zero, which means that adversarial neuron perturbations indeed find some sensitive neurons and helps the model to remove them. In contrast, there is no mask close to 0 for the vanilla pruning method in Figure 2(c) Finally, we prune different fractions of neurons according to their mask in Figure 2(d) For both methods, ASR becomes low after pruning more than 5% of neurons
Effect of Adversarial Neuron Perturbation. Recalling Section 3.2
& However, ACC also starts to degrade significantly for the vanilla pruning method with more
FFine Prine-pruning prunes neurons that are less active on clean samples until there is an obvious drop in accuracy on a clean validation set.
Sam/IBM/mooded 'This is conceivable. As long as the backdoor related neurons are assigned slightly smaller masks than other neurons, we still can prune them first and remove the backdoor.
/estps://githuus
sani tization
7

---

Rola 100
2000
2530
250
ACC w/o
Numbet
100
190
ACC wio
ABR
191
ASR wio
ACC
ASR
64-
50-
Rata
42-
ASR
1990
90
0.1
00
(x10%)
Masic Valu
Mask Value
Prusing Fraction (96)
(a) Optimization
(b) With perturbations
(c) Without perturbations
(d) Pruning
Figure 2: Comparison between optimization with and without adversarial neuron perturbations.
Table 3: Results with small budget 0.1 against Blend attack on 500 clean images. "Neurons{" indicates the number of neurons pruned by ANP.
Table Results with large budget 0.4 against Blend attack on 500 clean images. "Neurons," indicates the number of neurons pruned by ANP.
Steps
2
10
Steps
1
2
5
10
Time (s)
239.9
359.2
551.8
943.9
Time (s)
241.5
357.2
557.1
950.1
Neurons
159
188
235
259
Neurons
233
239
281
296
ASR (%)
65.19
13.40
1.06
0.90
ASR (%)
0.46
1.30
5.30
31.34
ACC (%)
93.62
93.07
92.95
92.72
ACC (%)
93.44
94.07
93.57
94.28
than 5% of neurons pruned. Since backdoored-relater neurons are still mixed with other neurons (e.g. discriminant neurons), the vanilla pruning method prunes some discriminant neurons incorrectly. Meanwhile, ANP always keeps ACC at relatively high level. In conclusion, the adversarial neuron perturbation the key to distinguish the backdoor-related neurons from the other neurons, which thereby successfully obtains high ACC as well as low ASR.
In addition, we explore the effects number of iterations in crafting adversarial neuron perturbations. We conduct experiments with varying numbers of steps (1/2/5/10) in ANP with small perturbation budget 0.1) and large perturbation budget (e 0.4) respectively. The other settings are the same as Section 4.1 The experimental results are shown in Tables B Under small perturbation budget, with more steps for ANP, the ASR decreases with a slight drop in ACC. This because ANP with a single step and small size is too weak to distinguish benign neurons and backdoor-related neurons, while more steps can help ANP find more backdoor-related neurons. However, under large perturbation budget, ANP with more steps has worse robustness. This is because, with large perturbation budget, more neurons become sensitive. many "suspicious" neurons, and it is unable to identify backdoor-related neurons from them. In conclusion, we can strengthen the power of ANP to find backdoor- backdoor-related rellttd neurons using a larger perturbation budget more steps. Among them, single-step ANP with slightly larger budget is more practical due to its time efficiency, and we adopt this by default.
a result, ANP with more steps finds too
Results with Varying Hyperparameters. As mentioned in Section ANP controls the trade-off between the accuracy on clean data and the robustness against backdoor attacks. To test the performance with different we optimize masks for Badnets ResNet-18 18 based on 1% of clean data using different a E [0, 1] with fixed budget e = 0.4. pruning, we always prune neurons by the threshold 0.2. As shown in the left plot of Figure 4 ANP is able to achieve high robustness (ASR < 4%) when a & 0.6. Meanwhile, ANP maintains high natural accuracy (ACC > 90%) as long range of a E (0.1, 0.6].
3.3 the hyper- parameter a in
0.1. As result, ANP behaves well with high ACC and low ASR in
Similarly, we also test the performance with different perturbation budgets c. The experiment settings are almost the same except a fixed trade-off coefficient a 0.2 and varying budgets e E [o, 1]. For example, we can provide obvious robustness (ASR becomes 7.45%) with small perturbation budget = 0.2) as shown in the right plot Figure However, we find the accuracy on clean data degrades with large perturbation budget. This is because too large perturbation budget brings much difficulty to converge well and ANP only finds a poor solution, which fails identifying these discriminant and robust neurons and prunes part of them. As result, ACC decreases significantly. In conclusion, the

---

ACC (96)
ASK EN1
ASS
RoMMel-L
Pechlet-11
Ra
Rachee--5 50
VCG-12
Pruning Fraction (%)
Pruning Fraction (56)
Threshald
Threshold
(a) ACC by fraction
(b) ASR by fraction
(c) ACC by threshold
(d) ASR by threshold
Figure 3: Performance (dtstd over random runs) of pruned models using different architectures by
different pruning fractions or different thresholds.
proposed ANP is stable in large range of the trade-off coefficient E [.61, and the perturbation budget E E (0.2.0.7, demonstrating that ANP is not sensitive to hyper-parameters.
From the discussion above, we find that the hyperparameters are insensitive as shown in Figure 4 and ANP performs well across a wide range of hyperparameters, e.g., trade-off coefficient a E [0.1. 0.6) against Badnets attack. In addition, with varying hyperparameters, the performance trends are very consistent across different backdoor attacks as shown in Figure though the attack (e.g., Blend) is unknown, the defender could tune against known attack (e.g., Badnets) and find the 0.2 is good choice. ANP with a = 0.2 also achieves satisfactory performance against Blend attack. In conclusion, the selection of hyperparameters in ANP is not difficult.
Appendix [.I As a result, even
Pruning by Fraction or by Threshold. Different from previous pruning methods that use fixed pruning fraction, the proposed ANP prunes neurons by threshold when generalized across different model architectures. To verify this, we train different Badnets models using ResNet-18 [5 ResNet- 50 15 and VGG-19 [37] with the same settings Section 4.1 We also train model with Badnets MobileNet- V2 with 300 epochs We optimize the pruning masks for these models with 0.4, 0.2 for ResNet-18, ResNet-50, and VGG-19. For MobileNet-V2, we found the we apply a smaller 0.2. We show the ACC and ASR after pruning by varying pruming fractions in Figure ((a) and ((b) which indicates that the suitable fraction varies across model architectures. For example, ResNet- 18 only has low ASR after pruning 5% of neurons, while MobileNet- V2 has large accuracy drop on clean samples with more than 5% neurons pruned. Figure FCc) and Figure ((d) show ACC and ASR after pruning by different thresholds respectively. When pruning by the threshold, we find there is a large overlap in [0.2, 0.5) (the gray zone) in which all models have high ACC and low ASR simultaneously. That is why by the threshold in ANP. Note that, even for low-capacity models as MobileNet- V2, it is still possible remove the injected backdoor by neuron pruning (without the fine-tuning).
perturbation budget = 0.4 too large, leading to a failure convergence.
budget e 0.2 and the same trade-off coefficient
adopt the strategy of pruning
Pruning on Different Components. We also compare the performance when applying ANP to different components inside DNNs. We prune a backdoored 4-layer CNN (2 conv layers 1 fully- connected layer + output layer) on MNIST, and find it more efficient to prune neurons in some layers. In particular, we achieve 0.43% of ASR (99.04% of ACC) when two neurons in the 2nd layer pruned, while we only achieve 4. 4.42% of ASR (92.48% of ACC) even after pruning 150 neurons in the fully-connected layer. The experimental settings and more results can be found in Appendix C.2 This indicates that the structure of components matter in pruning-barn based defense. We leave this in our future work.
4.4 Performance under Limited Computation Resource
In practical scenarios, the defender usually has limited clean data and computation resources, bringing more difficulty to repair backdoored models. While Section 4.2 has discussed the effectiveness of the proposed ANP the extremely small amount of data, this part focuses the performance with limited computation resources defense.
In particular, we optimize all masks for Badnets ResNet-18 18 with varying number of iterations (20, 100, 400, and 2000) on 1% of clean data (500 images) and prune the model from neurons with the small mask to neurons with the large mask. Figure shows the ACC and ASR with varying pruning

---

109
20
Rolm (84)
AGD (N4
A.RR 1%4
2000
Rati
20
ACC
ACC
2000
1.
Trade-off Coufiiient O7
Perturbation on Budget
Prining Fraction (%)
Pruning Fractio (54)
Figure 4: Performance (#: std over random runs) of the pruned model by threshold 0.2 with different hyper-parameters (Left: trade-off coefficient a; Right: perturbation budget e).
Figure 5: Performance (=st over 5 random runs) of the proposed ANP with varying prun- ing fractions based on different numbers of iter- ations.
fractions based on different numbers of iterations. We find that some backdoor-related neurons have already been distinguished just after 20 iterations. For example, after removing 4% of neurons with the smallest masks, the pruned model has 6.28% of ASR and 86.65% of ACC. As the pruning fraction increases, ASR falls significantly first and raises again after 10% of neurons pruned. This because ANP has not yet separated them completely due to extremely limited computation (only 20 iterations). With more than 100 iterations, this phenomenon disappears. And ANP based on 100 iterations already has comparable performance to ANP on 2000 iterations, especially when we have not pruned too many neurons (< 8%).
We also find that ANP with 2000 iterations should prune more (+1%) neurons to degrade ASR than ones with fewer iterations. We conjecture this is because there also exist some sensitive neurons that are unrelated to the injected backdoor which are assigned with small masks at the late optimization stage. Besides, for ACC, we can see that more iterations maintain a higher ACC than fewer iterations, especially when more neurons are pruned, as the left neurons are almost the discriminant and robust ones. In conclusion, even with extremely small number of iterations (e.g., 20 iterations), ANP is able to distinguish the backdoor-related neurons from other neurons and obtains satisfactory robustness against backdoor attacks.
Conclusion
In this paper, we identified a sensitivity of backdoored DNNs to adversarial neuron perturbations, which induces them to present backdoor behaviours even without the presence of the trigger patterns. Based on these findings, we proposed Adversarial Neuron Pruming (ANP) to prune the sensitive neu- rons under adversarial neuron perturbations. Comprehensive experiments show that ANP consistently removes the injected backdoor and provides the highest robustness against several backdoor attacks even with limited amount of clean data and computation resources. Finally, work also reminds researchers that pruning (without fine-tuning) is still promising defense against backdoor attacks.
Broader Impact
The backdoor attack has become a threat to outsourcing training and open-sourcing models. We propose ANP improve the robustness against these backdoor attacks, which may help build more secure model even trained in an untrustworthy third-party platform. Further, we do not want this paper to bring overoptimism about Al safety the society. Since the backdoor attack only part of potential risks e.g., adversarial attacks, privacy leakage, and model extraction), there is still long way towards secure AI and trustworthy AI.
we can see that the benign model also suffers from the adversarial neuron perturba- tions, although its sensitivity much weaker than that of backdoored models.
'Recalling Figure l(b)
10

---

Acknowledgments
Yisen Wang partially supported by the National Natural Science Foundation of China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund.
References
[1] Mauro Barni, Kassem Kallas, and Benedetta Tondi. new backdoor attack in cnns by training corruption without label poisoning. In ICIP, 2019.
[2]
Battista Biggio, Blaine Nelson, and Pavel Laskov. Support vector machines under adversarial label noise. In ACML, 2011.
[3] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmdiić Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In ECML/PKDD, 2013.
[4] Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff. In ICASSP, 2021.
[5] Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. IJCAI, 2019.
[6] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning arXiv preprint 1xxiv:171.005520 2017.
[7] Xinyun Chen, Wenxiao Wang, Chris Bender, Yiming Ding, Ruoxi Jia, Bo Li, and Dawn Song. Refit: a unified watermark removal framework for deep learning systems with limited data. arXiv preprint arXiv: pet::///111000000 2019.
[8] Ilaas Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In ICML, 2019.
[9] Chao Gao, Yuan Yao, and Weizhi Zhu. Generative adversarial nets for robust scatter estimation: proper scoring rule perspective. Journal of Machine Learning Research, 21(160):1 1-48, 2020.
[10] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In ACSAC, 2019.
[11] Amirata Ghorbani and James Zou. Neuron shapley: Discovering the responsible neurons. In NeurlPS, 2021.
[12] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Data security for machine learning: Data poisoning, backdoor attacks, and defenses. arXiv preprint arXiv:2012. 2020.
[13] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In ICLR, 2015.
[14] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 2019.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[16] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[17] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.
[18] Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. arXiv preprint arXiv:1811.00741, 2018.
11

---

[19] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
[20] Alexander Levine and Soheil Feizi. Deep partition aggregation: Provable defense against general poisoning attacks. arXiv preprint arXiv:2006. 14768, 2020.
[21] Alexander Levine and Soheil Feizi. Deep partition aggregation: Provable defense against general poisoning attacks. In ICLR, 2021.
[22] Yige Li, Nodens Koren, Lingjuan Lyu, Xixiang Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks, In ICLR, 2021.
[23] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. NeurIPS, 2021.
[24] Kang Liu, Brendan Dolan-Gavity and Siddharth Garg. backdooring attacks on deep neural networks. In RAID, 2018.
Defending against
[25] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. In NDSS, 2018.
[26] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV, 2020.
[27] Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In ICCD, 2017.
[28] Shiqing Ma and Yingqi Liu. Nic: Detecting adversarial samples with neural network invariant checking. In NDSS, 2019.
[29] Yuzhe Ma, Xiaojin Zhu Zhu, and Justin Hsu. Data poisoning against differentially--rivaal learners: Attacks and defenses. In ISCAI, 2019.
[30] Anh Nguyen and Anh Tran. Inpur-aware dynamic backdoor attack. In NeurlPS, 2020.
[31] Anh Nguyen and Anh Tran. Wanet- imperciimeeee warping- based backdoor attack. In ICLR, 2021.
[32] Ren Pang, Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Alex Liu, and Ting Wang. A tale of evil twins: Adversarial inputs versus poisoned models. In CCS, 2020.
[33] Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. In NeurIPS, 2019.
[34] Elan Rosenfeld, Ezra Winston, Pradeep Ravikumar, and Zico Kolter. Certified robustness to label-flipping attacks via randomized smoothing. In ICML, 2020.
[35] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NeurlPS, 2018.
[36] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
[37] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[38] Jacob Steinhardt, Pang Wei Koh, and Percy Liang. Certified defenses for data poisoning attacks. In NeurlPS, 2017.
[39] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
12

---

[40] Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Preventing delusive adversaries with adversarial training. In NeurIPS, 2021.
[41] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In NeurlPS, 2018.
[42] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint 19 2019.
[43] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In S&P, 2019.
[44] Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin- Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. In ECCV, 2020.
[45] Yisen Wang, Xuejiao Deng, Songbai Pu, and Zhiheng Huang. Residual convolutional ctc networks for automatic speech recognition. arXiv preprint 17Xiv:1702.07793, 2017.
[46] Maurice Weber, Xiaojun Xu, Bojan Karlas, Ce Zhang, and Bo Li. Rab: Provable robustness against backdoor attacks. arXiv preprint arXiv:2003.08904 2020.
[47] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurIPS, 2020.
[48] Pu Zhao, Pin- Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In ICLR, 2019.
[49] Haoti Zhong, Cong Liao, Anna Cinzia Squicciarini, Sencun Zhu, and David Miller. Backdoor embedding in convolutional neural network models invisible perturbation. CODASPY, 2020.
[50] Liuwan Zhu, Rui Ning, Cong Wang, Chunsheng Xin, and Hongyi Wu. Gangsweep: Sweep neural backdoors by gan. In MM, 2020.
13


Reviews:
[{"review_id": "ZMOn3ZWYT4q", "reviewer": "Reviewer_BQVU", "summary": "The paper focuses on the problem of backdoor defense without any knowledge of triggers, which is an emerging topic recently (models are trained at the third party platform). The authors proposed to use adversarially neuron perturbation to find the most sensitive neuron to backdoor trigger, then used Adversarial Neuron Pruning (ANP) to purify the model. The experiments are complete and comprehensive to validate the effectiveness of ANP. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- It is interesting to know how to trigger the backdoor effect without knowing any knowledge of the trigger. The authors found that through adversarially perturbing neurons, backdoored DNNs can present backdoor behaviors and is much easier to output misclassification than normal DNNs. The methods are novel and the findings are inspiring. \n- The proposed Adversarial Neuron Pruning (ANP) is simple but effective. ANP prunes the most sensitive neurons under adversarial neuron perturbations without fine-tuning. This work also convinces the community that pruning (without fine-tuning) is still a promising defense against backdoor attacks.\n- The experiments are comprehensive. Even given a limited amount of clean data and computation resources, ANP consistently removes the injected backdoor and provides the highest robustness against several state-of-the-art backdoor attacks. \n\nWeakness:\n- How to choose the hyperparameters like alpha? \n- What is the overhead of ANP?\n- Can ANP be extended to the defense during training? E.g., the sensitive neurons will be pruned along with the training process.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No flaw found", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "30 minutes"}, "extra_text": {}}, {"review_id": "UHXi1DwmC9G", "reviewer": "Reviewer_jkbL", "summary": "This paper proposed Adversarial Neuron Pruning (ANP) to detect the backdoored DNNs. APN prunes the most sensitive neurons under adversarial neuron perturbations. Extensive experiments show that ANP provides state-of-the-art defense performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is inspired by adversarial training (AT), which tries to find the most adversarial data around the original data.\nDifferent from adversarial training, this work finds the most adversarial training parameters. \nI like the idea of utilizing AT techniques in backdoor attacks, which can better enhance the defense in backdoor attacks. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My main concerns:\n1. On page 3, you take a fully-connected network as an example to show the training procedure. However, in my opinion, applying ANP on fully-connected layer and convolutional layer are different. FC layer has more neurons while conv layer has much fewer neurons. Thus, compared to conv layer, I think the FC layer can have a better performance but hard to train. Can you conduct a small experiment to show the results of ANP on FC layers?\n2. AT is very show, can you report the training time of your methods and other baselines?\n3. In each round, you apply PGD only once (PGD-1), how is the performance if you use PGD-10[1] (i.e., apply PGD 10 times each round) instead?\n[1] Madry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks[J]. arXiv preprint arXiv:1706.06083, 2017.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "IofeAGrXbIW", "reviewer": "Reviewer_5ThB", "summary": "This paper propose to distinguish and prune neurons which are sensitive to adversarial perturbations so that backdoor attacks can be better defended. Different from traditional methods which focuses on the perturbations on the input images, this paper explores the correlations between the sensitive neurons to injected backdoor and the multiplicative perturbation on neurons, and proposes Adversarial Neuron Pruning (ANP) algorithm to prune the these neurons to improve the robustness against backdoor attacks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this paper is clear and well-organized. Proposed algorithm seems natural. Extensive experiments are conducted to demonstrate the effectiveness of proposed algorithm. The comparison with other baselines show superiority. My only concern lies in the discussion in Section 3.2.\n\nThe authors argue that there exist a strong connection between the neurons which are sensitive to adversarial neuron perturbations and those to injected backdoor, which is kind of counter intuitive since adversarial attacks, such as PGD, always generate perturbations with different patterns for different samples, and could activate different neurons. There is no sufficient analysis or empirical observations to support this assumption.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work could benefit AI safety in terms of outsourcing training and open-sourcing models.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "-Aj1xM4PFJ", "reviewer": "Reviewer_H7Ya", "summary": "This paper proposes a simple yet effective method to purify the injected backdoor based on a small portion of clean data without access to training phase.\nThe proposed method dubbed Adversarial Neuron Pruning (ANP) prunes some sensitive neurons to defend backdoor attacks. Empirical results validate ANP can significantly improve robustness against various backdoor attacks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The proposed method seems to be well motivated. Studying the relation between the backdoor trigger patten and the sensitivity of neurons under adversarial neuron perturbations is novel. \n2. The experimental results comprehensively show that ANP seems to achieve the state-of-the-art robustness against several strong backdoor attacks. In addition, ANP can be conveniently implemented to repair the model without fine-tuning. \n\nCons:\n1. It seems hard to intuitively understand why the sensitivity of neurons under adversarial neuron perturbation becomes closely related to the injected backdoor although this phenomenon is empirically validated.\n2. Does the neuron pruning incur the degradation of generalization? It is better to compare and report the utility of purified model as well.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: Please carefully refer to \"Main Review\".\nThis paper targets to defend backdoor attacks. Therefore, it has no negative societal impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on the problem of backdoor defense without any knowledge of triggers. The proposal is an adversarially neuron perturbation to find the most sensitive neuron to backdoor trigger, and then use Adversarial Neuron Pruning (ANP) to purify the model. The philosophy behind sounds quite interesting to me, namely, exploring the correlations between the sensitive neurons to injected backdoor and the multiplicative perturbation on neurons. This philosophy leads to a novel algorithm design I have never seen.

The clarity and novelty are above the bar of NeurIPS. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)