Pre-activation Distributions Expose Backdoor Neurons
Authors: Runkai Zheng
Rongjun Tang
Jianze Li
Li Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Backdoor Defense
Backdoor Attack
Adversarial Learning

Abstract: Convolutional neural networks (CNN) can be manipulated to perform specific behaviors when encountering a particular trigger pattern without affecting the performance on normal samples, which is referred to as backdoor attack. The backdoor attack is usually achieved by injecting a small proportion of poisoned samples into the training set, through which the victim trains a model embedded with the designated backdoor. In this work, we demonstrate that backdoor neurons are exposed by their pre-activation distributions, where populations from benign data and poisoned data show significantly different moments. This property is shown to be attack-invariant and allows us to efficiently locate backdoor neurons. On this basis, we make several proper assumptions on the neuron activation distributions, and propose two backdoor neuron detection strategies based on (1) the differential entropy of the neurons, and (2) the Kullback-Leibler divergence between the benign sample distribution and a poisoned statistics based hypothetical distribution. Experimental results show that our proposed defense strategies are both efficient and effective against various backdoor attacks. 

Pre-activation Distributions Expose Backdoor Neurons
Runkai Zheng', Rongjun Tang"*, Jianze Li", Li Liu
Schooo of Data Science,
The Chinese University of Hong Kong. Shenzhen
School of Science and Engineering,
The Chinese University of Hong Kong. Shenzhen
"Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen rkteddy@outlook com, rongjuntang@ling cuhk. edu. cn 1ij anze@gmail com, Liuli@cuhk. edu. cn
Abstract
Convolutional neural networks (CNN) can be manipulated to perform specific behaviors when encountering a particular trigger pattern without affecting the performance normal samples, which is referred door attack is usually achieved by injecting a small proportion poisoned samples into the training set, through which the victim trains model embedded with the designated backdoor. In this work, we demonstrate that backdoor neurons are exposed by their rretaitivttion distributions, where populations from benign data and poisoned data show significantly different moments. This property is shown to be attack-invariant and allows efficiently locate backdoor neurons. On this basis, we make several proper assumptions on the neuron activation distributions, and propose two backdoor neuron detection strategies based on (1) the differential entropy of the neurons, and (2) the Kullback-Leide nign sample distribution and poisoned statistics based hypothetical distribution. Experimental results show that our proposed defense strategies are both efficient and effective against various backdoor attacks. Source code available here
backdoor attack. The back-
divergence between the be-
Introduction
Convolutional neural networks (CNNs) have achieved tremendous success during the past few years wide range of areas. However, training a CNN from scratch involves a large amount of data and expensive computational costs, which is sometimes infeasible. A more practical strategy is to obtain pretrained models or utilize public datasets from a third party, which brings convenience but also raises severe security problems into the deployment of models. For example, malicious third party may provide pretrained models embedded with a designated backdoor, such that the model will have predefined response to some specific pattern, which is also called the trigger. More realistically, the attacker can inject only small proportion of malicious data into the public dataset to mislead the trained model, which is referred to buckdoor poisoning attacks [24]. For instance, the malicious data can be created by patching a particular pattern into the benign data and changing the label the desired target. The correlation between the trigger and the specified target label will be learned by the models during the training time. In this way, the infected model will misclassify the input to the attack target when the pattern is patched, while behaving normally otherwise, shown in Figure
According to previous studies, it was empirically found that an infected model always possesses one or more neurons that have high correlation with the trigger activation, and pruning these neurons can
Fual Contribution
Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0
Figure 1: An overview of an infected learning system. The images with a white square are classified class 0. It is empirical observation that the backdoor behaviors are always triggered by or more backdoor neurons.
significantly alleviate the backdoor behaviors, while retaining the model performance 41 27 7. Nevertheless, how precisely find out these backdoor neurons in an infected model still challenging problem, and has attracted a lot of attentions from the community.
In this work, we take inspection on the pre-activation distributions of infected models on each layer. In general, the pre-activations in each neuron follow an unimodal distribution that can be approximated by a Gaussian distribution. We demonstrate that backdoor neurons do not hold such property. Instead, in typical backdoor neuron, the pre-activation distributions of benign data and poisoned data present significant different moments, and can be approximated by mixture two Gaussian distributions. This property allows us to locate potential backdoor neurons through simple statistical analysis on pre-activations. Specifically, in case the defender has access the poisoned dataset, the abnormal pre-activation distributions can be directly observed by forward propagating the data. The mixture of benign and poisoned data, where the small proportion of poisoned points being away from the benign mean, leads to a skewed, and even bimodal distribution. Based on the maximum entropy property of the Gaussian distribution, the skewness will cause a reduction on the differential entropy, compared with a single Gaussian distribution with the same variance. Hence, after standardizing the pre-activation distribution to unit variance, the abnormal distributions should have lower differential entropy. Those neurons could potentially separate benign and poisoned data. As for another defense setting, in which only an infected model and set of benign data are provided, we are not able to observe the bimodal distributions since the poisoned data is not available. In this case, we propose to rely on the recorded statistics in Batch Normalization (BN) layers. Specifically, the infected model is trained on poisoned data, the population statistics in backdoor neurons recorded in the BN layer will be different from those of only benign data. More importantly, benign neurons will not exhibit such mismatch in statistics, allowing for differentiation between benign and backdoor neurons. Based on the differential entropy and the statistics discrepancy, we are able to locate and prune potential backdoor neurons recover the model flexibly under two defense settings.
In summary, our contributions include:
1. We take a deep inspection on the infected model, and summarize the law pre- are-activation distributions on poisoned dataset. We find that (1) the standardized entropy of backdoor neurons can be significantly lower than benign neurons, and (2) the BN statistics in infected model are mismatched with the benign sample statistics.
2. We propose to prune potential backdoor neurons based on either the differential entropy of pre-activation distribution or the statistics discrepancy, depending on the defense settings. Under certain assumptions, we claim that both the proposed indices can successfully separate the benign neurons and backdoor neurons by an appropriate threshold.
3. We conduct extensive experiments to verify our assumptions and evaluate our proposed methods, and achieve the otate-of-the-ar results under two different defense settings.
2 Related work
In this section, we briefly discuss recent works in backdoor attack and defense, and a specific branch of related studies on distributional properties of poisoned features.

---

2.1 Backdoor attacks
The most famous backdoor attack is introduced in [14], where the adversary injects small set of targeted hhbpp-f data with specific trigger into the training set, leading to a misclassification when predicting the samples with such trigger. To make the trigger pattern even more invisible to
human beings, the blending strategy is used in [6] to generate poison images, while the form of
natural reflection is utilized in trigger design in [0) The input image is perturbed in [9] to keep its content consistent with the target label such that the model better memorizes the trigger pattern, and keep it imperceptible to human beings. Moreover, the multi-targed and multi-trigger attacks are proposed in [42] [2], and make the attack more flexible and covert. Recently, some sample-specifti trigger design strategies [26] are proposed, making the defense against such backdoor attack much harder. Generally, the above attacks can be referred to as the poisoning based backdoor attacks.
Under some settings, the attackers can control the training process to inject the backdoor without modifying the training data, referred as the zon-poisoning based backdoor attacks. This is achieved in [29 35 5 through targeted modification of the neurons' weight in a network. Such attacks will not be evaluated in our work due to its strong attack setting.
2.2 Backdoor defenses
Training stage defense.
Under such setting, the defender has access to the training process, so that they can detect and filter the poisoned data or add some restrictions to suppress the backdoor effect in training. Since the poisoned data can be regarded as outliers, different strategies are applied in [10 12 I 38 [5], such as the robust statistics in feature space and input perturbation techniques to filter them out of training data. Other methods aim suppressing the backdoor effect during
training phase with strong data augmentation methods [2] 22 34 31 including CutMix [4] Flip,
ShrinkPad (25, CutOut and MaxUp 131, or differential privacy constraints [1] 18
Model post-processing defense. Under some specific scenarios, the defenders are only given a suspicious DNN model without access to the training process or the full training set. Therefore, they must eliminate the backdoor threat with limited resources, such as a small set of clean data. A straightforward way is to reconstruct the trigger, and then mitigate the model with the knowledge of the reversed trigger [40). Some try to find the relationship between backdoor behaviors and the neurons in a DNN model. Different levels of stimulation to a neuron are introduced in (28) to see how to determine the output activation change, if the model is attacked. Simple neuron pruning strategies are applied in 77] repair the model, while redundant neuron pruning and fine-tuning are combined in [27] to erase the backdoor effect. Adversarial perturbations are added to the neurons neurons with more limited clean data requirement and better performance. There are other fine-tuning based methods with the implementation of knowledge distillation 23 17 Mode connectivity repair technique [14] is also explored to mitigate the backdoored model. Recently, the K-Arm optimization [37 is applied in backdoor detection, helping curtail the threat of backdoor attack.
in [41 and precisely prunes the
2.3 Distributional properties in poisoned dataset
One branch of research on backdoor learning focuses specifically on using statistical differences in the distribution of benign and poisoned samples to filter out malicious data. Activation clustering 4 uses K-means to separate benign and poisoned samples in feature space. Spectral signatures [38] reveal that feature vectors tend to leave strong signals in the top eigenvectors of their covariance matrix. SPECTRE (Spectral Poison Excision Through Robust Estimation) [15] utilizes tools from robust statistics to amplify the spectral signature by outlier-robust data whitening. Our work differentiated from the above works from the following three aspects: 1) the above works focus on the penultimate layer feature representation, while our work inspects deep into each layer 2) the above works take the representation space from all neurons as a whole, while our finding indicates that the distributional difference only exists in some neurons 3) the above works aim at filtering out poisoned samples for retraining, while our methods directly repair the trained neural network by pruning potential backdoor neurons.
3

---

Preliminaries
3.1 Notations
Consider a multi-class classification problem with C classes. Let the original training set D {(r,,, yz) ] contains N i.i.d. sample images a; Rd. (xxxdxxxd and the corresponding labels Vi E (1.2,..., C} drawn from X' x y. Here, we denote by de, d₁ and d, the number of channels, the height and the width of images, respectively. In particular, we have de for RGB images.
As Section 2.1 the backdoor poisoning attack involves changes to the input images and corresponding labels on a subset D, S D. In this work, we define the ratio [2ul rale. We denote the poisoning function to the input images 6(x)). A dataset D is said to be p-poisoned if the poisoning rate of the dataset is
as the poisoning
[o]
Consider neural network F 9) with L layers. Denote
F()) ( a Φ f(l 06.
for < < L,, where (() is linear function (e.g., convolution) in the 1-th layer, and á is nonlinear activation function applied element wise. In this work, we may denote F(x;0) 0) as F(x) or for simplicity.
We denote by WCO E Rd.xxd xdy xd the weight tensor of a convolutional layer. To do pruning, we apply a mask M0 E {0, 1]d. xde xd starting with M( = dld xd. xd, xd., in each layer. Pruning neurons on the network refers to getting collection of indices I = {1. k): and setting 0 0d,xdx xd., if (1, I. The pruned network F-z has the same architecture as F but with all the weight matrices of convolutional layers set W @ M(), where @ denotes the Hadamard product.
3.2 Differential entropy
To measure the uncertainty of discrete random variable Z, the entropy 36 3
defined as ((2) E. p(z) p(z). At the same time, as an extension of entropy, the differential entropy also introduced for continuous random variable. More concretely, if Z is continuous random variable, then it was defined as
M(Z)
p(2) log
An important fact about the differential entropy is that, among all the real-valued distributions supported on (-o0, oo) with a specified finite variance, the Gaussian distribution maximizes the differential entropy [8]. In this work, the differential entropy (1) will be utilized to identify the distributions that are different from Gaussian distribution.
3.3 Backdoor neurons
was found that there exist one or more neurons that contribute the most to the backdoor behaviors
in infected model 41 27 If some of all of these neurons pruned, the attack success rate
will be reduced greatly 41 In this work, to better quantify the importance of neurons to backdoor behaviors, would like to introduce the sensitivity of neurons to the backdoor. We first introduce the definition of backdoor loss on a specific poisoning function:
Definition 1. Given model F and poisoning function 5, the backdoor loss dataset D is defined as:
(sa(f) DDees
where DoE denotes the cross entropy loss.
Then:
Given model F, the index of neuron (1, k)) and the backdoor loss Lbd, the sensitivity of that neuron to the backdoor is defined as:
Definition 2.
a(F,1,k) Lba(F) Lba(F (F-(()())))))
where F-((A)) is the network after pruning the k-th neuron of the (-th layer.

---

(a) Benign neuron
(b) Backdoor neuron
(c) Benign neuron
(d) Backdoor neuron
Figure In (a) and (b), we compare the pre-activation distributions in backdoor neurons and benign neurons. In benign neurons, the pre-activation distributions on benign data and poisoned data are nearly the same, while in backdoor neurons, they show a great difference. In (c) and (d), we plot the empirical distributions with benign samples (in blue) and the BN statistics induced Gaussians (in green). In backdoor neurons, the discrepancy between the empirical and BN-induced distribution large (all the neurons selected from infected ResNet- 18 trained CIFAR-10, 10, and use 1,000 images with (poisoned data) or without trigger (benign data) as the inputs).
The backdoor loss high when the model infected, and will be reduced when the backdoor effect is alleviated.
Using the quantity defined in (2). are able find the neurons that are mostly correlated with the backdoor behaviors:
Definition 3. Given a model F and a threshold 7 > the set of backdoor neurons are defined as:
Br,
k1) (FF,
3.4 Pre-activation distribution
During the forward propagation of input æ, denote (1) (2) E Rd D xdd xa!?
as the
output of the (-th layer. For the k-th neuron the /-th layer, the pre-activation of = = is defined as the maximum value the k-th slice matrix of dimension d²² de in ( The reason we choose pre-activations instead of activations is that the distribution of activations after non-linear function might be distorted. For example, ReLU (rectified linear unit) will cut out all negative values.
is a common assumption that, for every neuron, the pre-activations before the non- non-linear function follow Gaussian distribution, if the network is randomly initialized and the number of neurons is large enough [20). This assumption is based on the central limit theorem under weak dependence 3 In trained network, although this assumption may not strictly hold, the pre-activation of every neuron can be still regarded as approximately following Gaussian distribution. However, this work, for the first time, observe bimodal pre-activation distribution in backdoor neurons formed by the benign data and poisoned data. This phenomenon is shown in Figure 2 where a typical backdoor neuron compared with the benign neurons. It be seen that, after the model is infected, the pre-activation distributions of benign neurons hardly change when the data is poisoned, while the pre-activation distributions of backdoor neurons become significantly different.
4 Methodology
4.1
Basic assumptions
We now introduce two preliminarily assumptions the pre-activation distribution of infected neural network.
Assumption 1. Given infected model F, we have |B for some threshold > 0.
This is a primary assumption that guarantees proper pruming of neurons can correct the network's predictions poisoned samples some extent. Hence, it is prerequisite of good performance of all the pruning-ben based defense methods.
The next assumption provides precondition for our methods:
Assumption Given a model F infected by a poisoning function 8 with p-poisoned dataset D, the pre-activation of sample from D each single neuron of F follow Gaussian mixture distribution,
5

---

that is:
6 (1-11p11 pN .0 2 + PN o
with
1
(1)
f4
<,
if (1, () $ BF (1, k) E By
3 e,
and
1 0
(1)
(1)
<
Ak & BF
where c > is a small enough value, Pk and 0 (1)2 (b(F((())) x ~ X3 { (6())()())))) a ~ )) respectively.
pk and ox (1)2
are the mean and variance of
This assumes that the mean value of pre-activation distribution of benign and poisoned samples only significantly differentiated in backdoor neurons. This assumption is made based on empirical observation, and our methods work only when this assumption holds.
Entropy-based pruning (EP)
Based on the given assumptions, standardizing the pre-activation distributions (by subtracting the mean and dividing by the standard deviation) will maximize the differential entropy in benign neurons, which approximately follow standard Gaussian distribution (W 1)). However, in backdoor neurons, the mixture distributions resulting from differences in the moments of Gaussian components cannot be Gaussian distributions, leading smaller differential entropy than that of standardized Gaussian distribution.
Corollary 1. Let 1
42
be the standardized pre-activations, then the following inequality o
(1)
satisfied:
m M( S M(Z), vk E BF,,, k & Bp
where Z N(0, 1) is the standardized Gaussian distribution.
This inequality gives guarantee that with an appropriately chosen threshold, the backdoor neurons can be well separated with the benign neurons.
4.3 BN statistics-based pruning (BNP)
BN layer involves using the statistics of mini-batch to normalize the data in each layer for each neuron. known be able to smooth the optimization landscape, and has gradually become a common setting of neural networks [19]. During inference, BN uses the fixed statistics obtained by averaging the sample statistics of mini-batches during training time, including the mean and the variance. If the model is trained on poisoned dataset, BN will record the mean and the variance of the poison-bente bensgn mixed data. Note that the mean and variance here are not defined on the
pre-activations o but z (!) Based on the above discussions, we know that the poisoned pre- activations in backdoor neurons follow a different distribution from the benign samples. The recorded statistics during training are actually that of the mixture distribution. Hence, we can expect that the BN statistics of a trained backdoor neural network are biased. we are able to access a small set of benign data, can calculate approximation of the true statistics benign data. Then we calculate the Kullback-Leibler (KL) divergence [9] between the sample distribution and the BN induced distribution as the measurement of the bias. By assuming both of the distributions follow Gaussian distributions, we have a closed-forn solution:
(1) log
(012
(1
(1) 1
DKL (Av sample BN
(l))
0k
20 (1)
2'
o
where Arl1)
are the statistics obtained are the BN statistics. Note that BN statisties is the mixture statistics of benign and poisoned distributions. Thus, we have the following corollary:
NN ( (1) o
.v(0) BN
NV o (1) ), P and of (1)2
(
sample from benign samples, j/ and a

---

Corollary 2. According to Assumption 2
when - 0, the following inequality is satisfied:
DKL (N !
sample N BR & > DKL
(Ar .N
BN = 0, vk E BF k' & BF
sumple!
The corollary indicates that backdoor neurons should have larger KI. divergences than benign neurons, as illustrated in Figure 2(c)
4.4 Overview of the two pruning strategies
In Section[3.4] we reveal the discrepancy between the pre-activation distributions in backdoor neurons and that in the benign neurons. This enables fast detecting the neurons that are more related the backdoor behaviours. The index we choose to detect the abnormal neurons depends on what kind of data are able access.
Mixture training data In this case, the victim is given poisoned training dataset with specified poisoning rate p. Our goal is to obtain a benign model based on the poisoned dataset. To achieve this, first train infected model F on the poisoned dataset. The resulting model should have certain number of backdoor neurons based on empirical observation and assumption. Since p > 0 for the dataset, all the neurons follow Gaussian mixture distributions, and we have h((*)) < h((a()) for all E BF k' & B This implies that with an appropriate threshold Ti we can perfectly separate the benign neurons and backdoor neurons, which can be formulated as:
T h(t The h(#)))
Vk E BF
> The VK' & BF
Setting the threshold 7 is crucial to the solution, and it is a trade-off between the accuracy on benign samples and that on the backdoored samples. Note that |B << d!",, We can treat the low entropy neurons as outliers in each layer, and set different thresholds for different layers. Specifically, let h!!) = [h(z]]) ), h(xz22 (1) )..... h(x de )IT Rd.h be a vector of differential entropy
of the 1-th layer calculated from the poisoned dataset. Then we set Th
(1) = f - un
(l) where
7,000 M
(1) h
and s V 2 I (hg (1) h00y are the mean and standard deviation of
(,01) is a hyperparameter controlling how low the threshold is. Then we have a set of indices of potential backdoor neurons In {(1,k) 1 <Th (1), Finally, we prune the infected model F using I, which results in final model F-I.-
Benign training data This is the case that the victim is given a trained poisoned model with a small set of benign data. Our goal is to utilize the benign data clean up the poisoned model and eliminate the backdoor threat. Similar to the pruning process based on differential entropy, we first
construct vector of KL divergences of all neurons for each layer (())
[x(). K(()) K
IT
according to equation L
We set TK = k(0 + uk
SK. where (()) d!
Σ K (I)
and K
kWyz are the mean and standard deviation of K uk is
hyperparameter. The set of selected neurons Ix = {11, k) K(D) > ((0) and the pruned model can be represented F-Ix.
Experiments
5.1 Implementation details
Datasets
In this section, the experiments are conducted on two influential benchmarks, CIFAR-10 21 and ImageNe [22]. We use 90% of the data set for training, the rest of the data is used for validating recovering the poisoned model.
7

---

Models We use 18 [16] as the baseline model to evaluate our proposed method, and compare it with other methods. We train the network for 150 epochs on CIFAR-10 and 100 epochs on Tiny-ImageNet with SGD optimizer. The initial learning rate is set to 0.1 and the momentum is set to 0.9. We adopt the cosine learning rate scheduler adjust the learning rate. The batch size is set to 128 by default.
Attacks Our experiments are based on both the classical and the most advanced attack strategies. including the BadNet [14), Clean Label Attack (CLA) R Reflection Backdoor (Refool) [30), Warping-basee baannd poisoned Nelworks [3], Blended buckdoor attack (Blended) 66, Inpul-aware back- door allack (IAB) [32 and Sample Specific Backdoor Attack (SSBA) [26]. For BadNets, we test both the All-to-All (A2A) attack and All-to-One (A20) attack, i.e., the attack target labels are set to particular label y/+ C₁, respectively. The target for A20 attacks of all the attack strategies is set to class O. The triggers for BadNets and CLA are set to randomly generated patterns with size 3 3 for CIFAR-10 10 and 5 5x5 for Tiny-ImageNet. The poisoning rate is set to 10% by default. Note that, due to the image size restraint, SSBA is only performed on Tiny- Tiny-Imagenet.
3/- (y + 1) mod C, or
Defenses We conduct experiments under two defense settings, one of which allows the defender to access the poisoned training set, while the other only has a small clean data set. Both the defense goals are to obtain clean model without backdoor behaviors. We compare our approaches with the loe pruning (7]. fine-tuiinn ,FT), finee--runing (FP) [27], neural attention distillation (NAD) 23 and adversarial neuron pruning (ANP) [41]. The number of benign samples allowed to access is set to 500 (1 9) for CIFAR-10 10 and 5000 (5%) for Tiny-Imaged by default. We set the threshold hyperparameter un/uus Ug. to 3 on CIFAR-10 and 4 ImaeemteeNet of all tested attacks by default.
Evaluation metrics In this work, we use the clean accuracy (ACC) and attack success rate (ASR) to evaluate the effectiveness of different methods. The ACC for given model F is defined as:
ACC(F, Drest) =
Σ [(argma((F(x)) = y).
(2.,))
where I is the indicator function. The ASR is defined as:
ASR(F, Deest)
Σ
Hargmax(F((((2)) ye
(t.
where is the attack target label. The ACC measures the model performance on benign samples, while the ASR reflects the degree of backdoor behavior retainment in the model. Given an infected model, our goal is to reduce the ASR, while keeping the ACC from dropping too much.
5.2 Experimental results
CIFAR-10 We show the results CIFAR-10 in Table The recently proposed NAD and ANP perform significantly better than other defense methods, reducing the ASR to a very low level with a slight drop on ACC. However, they also have significant drop ~ 4%) on ACC when defending CLA, which is the most robust backdoor attack in our experiments, and ANP even failed when defending BadAe Nevertheless, both of our methods successfully eliminate the backdoor (ASR < 1%) with negligible loss on ACC. We even observe little rise on ACC when defending BadNets by EP. This phenomenon demonstrates that backdoor neurons may hurt the ACC in some way, and thus the ACC will rise when the backdoor neurons are precisely pruned. Overall, our methods achieve the most advanced defense results.
ImageNe Tiny-ImageNet is larger scale dataset with higher resolution images, and it is harder to defend against the attacks performed on it. Note that the A2A attack is absent, since we cannot successfully perform the attack due to the large number of its classes (up to 200). Our experimental results show that all the defense methods suffer from the performance degradation compared with the results in CIFAR- 10, and they fail to defend against WaNet with large ACC drop but even unchanged ASR, especially the ANP and lxx defense. This phenomenon shows that the principles for finding backdoor neurons of both ANP and Lxx don't work in such case. Nevertheless, our methods totally remove the backdoor and the ACC are not even affected, which indicates that our methods can precisely locate the backdoor neurons even on such large scale dataset.
8

---

Table I: Experimental results of the proposed approaches against different attacks compared with
other defense methods CIFAR-10
BadNets (A20)
BadNets (A2A)
CLAA
WaNet
Blended
Refool
IAB
Attacks
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
Origin
93.86
100.00
94.60
93.89
94.99
98.83
94
99.67
94.17
99.62
94,24
98.40
93.87
97.91
FT
92.03
92.88
95.73
92.93
9.37
93.9
FP
92.18
2.97
91.75
66.82
92.60
99.36
92.07
1.03
70.92
90.92
92.36
75.98
87.04
16.13
92.12
100.00
93.67
6.67
92.75
98.76
93.48
99.74
86.99
99.77
91.19
98.47
88.37
88.48
NAD
2.43
92.18
91.36
15.31
93.08
3.05
92.72
6.74
ANP
93.47
3.53
90.29
86.22
91.13
11.76
94_12
0.51
93.66
5.03
91.71
26.96
93.52
10.61
(Ours)
93.88
94.49
94.42
0.91
2.80
93.67
2.24
93.35
93.17
BNP (Ours)
93.60
1.60
94.25
0.72
94.14
7.03
94.05
3.39
94.17
2.71
93.69
6.48
93.15
0.64
Table 2: Experimental results of the proposed approaches against different attacks compared with
other defense methods in Tiny-ImageNett [22]
BadNets (A20)
CLA
WaNet
Refeol
Blended
IAB
SSBA
Attacks Origin
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
65.61
56.58
99.98
53.26
FT
46.93
99.84
61.19
63.20
54,28
99.96
47.09
91.77
56.83
29.12
52.39
99.10
52.39
33.19
FP
35.41
99.48
62.30
39.05
53,65
100.00
42.10
86.62
59.59
99.76
52.67
98.47
53.36
31.96
53.13
90.39
59.15
23.12
42.01
99.84
46.84
81.19
56.33
99.85
54.81
86.97
49.35
99.98
NAD
44,20
90.13
62.80
17.35
53.40
99.98
51.06
70.63
57.35
55.60
53.32
98.85
52.52
25.08
ANP
53.85
59.69
3.64
54,82
86.98
50.67
0.21
62.49
61.39
EP (Ours)
60.68
0.86
64,47
0,10
60.53
0.02
51.29
17.07
60.67
0.69
61.26
0.60
64.2
0.11
BNP (Ours)
61.60
1.60
64.86
0.05
61.58
0.01
52.41
23.79
60.77
0.85
61.30
0.60
64.64
0.01
5.3
Ablation study
To be fair, we compare BNP with other re-training based methods using 500 benign samples in Section 5.2 However, BNP doesn't require re-training the model, and the samples are just used for detecting the distribution discrepancy. As the statistical differences may be detected with much fewer samples, now study how the number of samples affects the effectiveness of BNP. We train BadNets, CLA, Refool and Blended on CIFAR-10 10 with p 10%, and use 10 to 500 benign samples recover the model using BNP. We record the changes of ACC and ASR with respect to the number of benign samples. The results are shown in Section 5.3 The influence of the number of samples to our methods comes from the randomness on estimating moments. As the number of samples grows, the randomness is reduced and BNP has more stable performance, but the average performances are not improved, except for Refool. Compared with other attacks, Refool clearly needs more samples to reduce the ASR. A possible reason is that the mixture distribution in Refool has closer moments and is harder to distinguish. Besides, we surprisingly find that BNP can recover BadNets, CLA and Blended using only 10 benign samples. We also conduct experiments to show the high correlation between the backdoor neurons and our proposed evaluation metrics, and the results are shown in Appendix D.
6 Discussion
The proposed methods are superior to other existing defense methods in the following three aspects:
BNP
Badnets
Badnecs
400
CLA
Paccanage
Blunded
Blanded
ARR
IAB
WaNeet
WaNet
82
65
Number of
Number
Figure 3: This figure shows the influence of sam- Figure 4: This figure shows the performance of the ples used for BNP.
proposed EP and BNP with different hyperparame- ter against badnet on CIFAR- 10 with ResNet-18.

---

Better performance As demonstrated in Section 5 both of the proposed methods achieve state- of-the-art results. Moreover, according to the ablation study, the proposed BNP can successfully defend most of the attacks within 10 benign samples, which shows the amazing effectiveness of our proposed methods.
Higher efficiency The proposed methods are highly efficient. We record the running time of several defense methods on 500 CIFAR-10 images with ResNet-18. 18, and show the results in Table It can be seen that both of the proposed methods require less time than the baseline defense methods. Since both methods require scanning each neuron once, the computational complexity scale linearly with the number of the neurons in the neural network. Therefore, the efficiency of our methods is promised.
Table 3: The overall running time of different defense methods on 500 CIFAR-10 images with ResNet-18.
Defense Method
FT
FP
NAD
ANP
EP (ours)
BNP (ours)
Runing Time (scc.)
12.35s
14.59a
22.088
25.68s
10.69a
0.39s
More robust to hyperparameter choosing One of the most general problems in backdoor defense is the choice of hyperparameters. Under realistic settings, the defenders can only perform defenses without any prior knowledge about poison data, including poisoning rate and examples of poisoned data. So the defenders should carefully tune the hyperparameters, or the ACC and ASR can change suddenly even under small fluctuations of those hyperparameters. In comparison, both of the proposed pruning strategies only require one universal hyperparameter u. Moreover, they show reliable consistency against different attacks in the same dataset, only vary from different datasets, which is inevitable. Besides, we leave wide range of parameters to choose, so that the ACC remains high while the ASR is controlled to very small number, as shown Section 5.3
7 Conclusion
In this work, we take an inspection on the characteristics of an infected model, and find backdoor sen- sitive neurons distinguishable by their pre-activations on poisoned dataset. Specifically, in backdoor neurons, pre-activations from benign data and poisoned data form distribution with extraordinarily different moments. This property makes it possible for defenders to efficiently locate the potential backdoor neurons based on the distributional property of aeeaaiiivtionn. When direct access the poisoned dataset available, we propose to measure the mixture property of pre-activations via differential entropy to detect potential backdoor neurons. In another case, where defenders only have access to benign dataset, we propose to check abnormality of pre-activation distribution based on the inconsistency of the recorded BN statistics and the sample statistics on the given benign dataset. We then do pruning on potential backdoor neurons to recover the model. Experiments show that the proposed defending strategies can efficiently locate the backdoor neurons, and greatly reduce the backdoor threat with negligible loss of clean accuracy. Our approaches achieve superior results compared with all other defense methods under various attacks on the tested datasets. The results shed lights on the field of backdoor defense, and can be guidance for designing more robust backdoor attacks.
8 Acknowledgement
This work is supported by the National Natural Science Foundation of China (No. 62101351), the GuangDong Basic and Applied Basic Research Foundation (No.2000A15151 10376), Shen- zhen Outstanding Scientific and Technological Innovation Talents PhD Startup Project (No. 2BSS2022100010444771000 the Key-Area Research and Development Program of Guangdong Province (2020B0101350000) and the Chinese University of Hong Kong (Shenzhen).
10

---

References
[1] Hassan Ali, Surya Nepal, Salil Kanhere, and Sanjay Jha. Hes-nets: A heal and select mechanism to defend dnns against backdoor attacks for data collection scenarios. CoRR, abs/2012.07474, 2020.
[2] Eitan Borgnia, Valeriaa Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradcoff. In ICASSP 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
IEEE, 2021.
[3] Richard C Bradley Jr. Central limit theorems under weak dependence. Journal of Multivariate Analysis, 11(1):1-16, 1981.
[4] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks deep neural networks by activation clustering. In SafeAI@ AAAI, 2019.
[5] Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Proflip: Targeted trojan attack with progressive bit flips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 77 1-77277 2021.
[6] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. CoRR, abs/1712.05526, 2017.
[7] Hao Cheng, Kaidi Xu, Sijia Liu, Pin-Yu Chen, Pu Zhao, and Xue Lin. Defending against backdoor attack on deep neural networks. CoRR, abs/2002. 12162, 2020.
[8] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
[9] Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pages 146-158, 1975.
[10] Bao Gia Doan, Ehsan Abbasnejad, and Damith C. Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. Annual Computer Security Applications Conference, ACSAC '20, page 897-912. Association for Computing Machinery. 2020.
[11] Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. In International Conference on Learning Representations, 2019.
[12] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 1 113-125, 2019.
[13] Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu. Maxup: A simple way to improve generalization of neural network training. arXiv preprint arXiv:2002.09024, 2020.
[14] Tianyu Gu, Kang Liu, Brendan Davint and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 1:47230-47244 2019.
[15] Jonathan Hayase and Weihao Kong. Spectre: Defending against backdoor attacks using robust covariance estimation. In International Conference on Machine Learning, 2020.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CORR, Abs/1503.00231, 2015.
[18] Sanghyun Hong, Varun Chandrasekaran, Yigitcan Kaya, Tudor Dumitras, and Nicolas Paper- not. On the effectiveness of mitigating data poisoning attacks with gradient shaping. CoRR, abs/2002. 1497, 2020.

---

[19] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMIR, 2015.
[20] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self- neural networks. Advances in neural information processing systems, 30, 2017.
[21] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[22] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23IN, 7(7):3, 2015.
[23] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations, 2020.
[24] Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[25] Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. ICLR Workshop on Robust and Reliable Machine Learning in the Real World, 2021.
[26] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with samplc-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16463-16472, 2021.
[27] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 273-294. Springer, 2018.
[28] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma. Yousra Aafer, and Xiangyu Zhang Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, pages 1265-12 1282,128,2000 2019.
[29] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network and Distributed System Security Symposium, NDSS. The Internet Society, 2018.
[30] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pages 182-199. Springer, 2020.
[31] Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In 2017 IEEE International Conference Computer Design (ICCD), pages 45-48. IEEE, 2017.
[32] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems, 33:3454-3464, 2020.
[33] Tuan Anh Nguyen and Anh Tuan Tran. iaper-impercoptibbl warping-base based backdoor attack. In International Conference on Learning Representations, 2020.
[34] Han Qiu, Yi Zeng. Shangwei Guo, Tianwei Zhang, Meikang Qiu, and Bhavani Thuraisingham. Deepsweep: An evaluation framework for mitigating dnn backdoor attacks using data augmen- tation. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security, ASIA CCS '21, page 363-377, Association for Computing Machinery, 2021.
[35] Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Tbt: Targeted neural network attack with bit trojan. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 13198 13198-13207, 2020.
[36] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-44 1948.
12

---

[37] Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm op- timization. In International Conference on Machine Learning, pages 9525-9536 PMLR, 2021.
[38] Brandon Tran, Jerry Li, and Aleksander Mądry. Spectral signatures in backdoor attacks. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 8011-8021, 2018.
[39] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprini arXiv: (9t..02711.2111 2019.
[40] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE, 2019.
[41] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34, 2021.
[42] M. Xue, C. He, Wang, and W. Liu. One-to-n n-to-one: Two advanced backdoor attacks against deep learning models. IEEE Transactions on Dependable and Secure Computing, (01):1-1, oct 5555.
[43] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023-6032. 2019.
[44] Pu Zhao, Pin-Yu Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In International Conference on Learning Representations (ICLR 2020), 2020.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See abstract.
(b) Did you describe the limitations of your work? [Yes] See 5.3
(c) Did you discuss any potential negative societal impacts of your work? [N/A] There is no negative social impacts our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results....
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
13

---

(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See experimental settings.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See experimental results.
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See experimental settings.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "cvSCmBYeJ0n", "reviewer": "Reviewer_Bhnv", "summary": "Based on the observation that clean and backdoor training samples have different feature statistics, the paper proposes to detect and remove the neurons that are mostly affected by backdoor training samples. Empirical results show the effectiveness of the proposed method under certain scenarios. ", "questions": "The title in the submitted PDF is different from that in Openreview. ", "limitations": "Please see above. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The proposed method is intuitively correct: Clean samples and backdoor samples should have somehow different feature statistics. This is because the model overfits superficial correlation on backdoor samples while it learns semantic information on clean samples. I don't doubt the proposed method is effective in improving backdoor robustness. \n2. The paper is well-written and easy to follow. All notations and terms are defined clearly without ambiguous. \n3. The proposed method can defend some backdoor attacks when few clean samples are available. \n\nWeakness:\n1. My main concern is how such good performance reported in this paper can be generalized to different settings. \n\n1.1 In practice, we don't know what type of backdoor attacks are used. So, it is very important for the defense method to be generally robust against multiple different attacks using the **same** hyper-parameters in the defense method. However, in this paper, the authors only mentioned the only hyperparameter $\\mu_k$ \"is **usually** set to 3\". The use of the word \"usually\" makes me feel skeptical about the results. The readers need to know what value $\\mu_k$ is used under all cases (on all datasets and against all attacks). The ideal case is we have a value that works well for all attacks, even under the mixture attack setting (i.e. when multiple different attack simultaneous exists in the training set). But that should be challenging in my point of view since the distributions of backdoor samples from different attacks can be very different. So what are the \"unusual\" cases in this paper? If different hyper-parameter values are used against different attacks in the proposed method, then it is unfair comparison with the baselines. As far as I know, NAD can achieve better performance if the hyper-parameters are tuned for different attacks. \nAgain, I don't doubt for a second that the proposed method can help defense backdoor attacks. I just concern whether it can achieve such good performance across all reported different attacks using the same hyper-parameter setting. \nIt won't harm the contribution of this paper if the performance drops a little bit when the same $\\mu_k$ value is used for different attacks. \n\n1.2 Please consider more different attack settings. For example, when a different poisoning ratio (eg 5%, 20%, 30%...) is used? Does the proposed method need to use a different uk value under different poisoning ratios to achieve good performance?\n\n1.3 Please provide ablation study results on different uk values. \n\n\n2. In Figure 5 (c) in appendix, the visualization of Blend attack looks a bit weird to me. I can't see any effect of the blend image (which is usually a hello kitty image or strong random noise). Please see Figure 5 in [1] or the original Blend attack paper for example. May I ask what blending ratio and blend image/pattern you are using? \n\nI will improve my score if the concerns are addressed. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aN4urZwuxJw", "reviewer": "Reviewer_WDQn", "summary": "In this paper, the authors propose a defense approach to mitigate the backdoor attacks on convolution neural networks (CNNs). Specifically, the authors first observe that the neuron activation distributions within the backdoor-infected CNN are different for benign and backdoored samples. Based on such an observation, the authors further propose to calculate the differential entropy to distinguish the compromised neuron. In the experiment section, the authors evaluate the proposed approach to CIFAR-10 and TinyImageNet benchmarks under different settings(e.g., training phase, post-training phase, etc). Moreover, the authors also test the robustness of the proposed approach on two adaptive attacks. The results demonstrate that the proposed method can significantly outperform comparison work in the dimensions of robustness, efficacy, and efficiency.  \n", "questions": "1. Is this work can still perform efficacy under multiple-label and other practical scenarios?\n\n\n2. What device for you to test the running time in the experiment section?", "limitations": "Requiring clean validation data makes the approach somehow impractical.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+++++++Strengths+++++++\n\n1. Well-structured presentation and easy to follow.\n2. Comprehensive evaluation.  \n3. Considering two adaptive attacks.\n\n-------------Weakness------------\n\n1. The novelty is somehow limited. Since inspecting compromised neurons for defending against backdoor attacks has been widely adopted by previous work, this work may not seem that novel. \n\n2.  I think more previous unlearning-based work or training-phase stage defense should also be considered for comparison. For example, DBD[1] and the most recent approach[2]. \n\n3. Missing evaluation on several practical scenarios. I think the proposed approach should also be evaluated under some rather practical scenarios, for example,  multiple-label scenarios, and multiple backdoors within the same infected label which previous work (neural cleanse) has discussed that.\n\n[1]https://openreview.net/forum?id=TySnJ-0RdKI\n[2]ADVERSARIAL UNLEARNING OF BACKDOORS VIA IMPLICIT HYPERGRADIENT\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HaNLWTMOw6", "reviewer": "Reviewer_Ton2", "summary": "This paper reveals that poisoned neurons and benign neurons have different behaviors, measured by the entropy of neuron activations and the BN statistics. Motivated by these understandings, the authors propose DDE and MBNS targeting the defense of poison-only backdoor attacks and attacked pre-trained models, respectively. The proposed methods are evaluated on both CIFAR-10 and Tiny-ImageNet datasets under seven attacks. ", "questions": "\nCons\n1. Some statements need further support or modifications to avoid over-claim.\n- ‘we claim that both the proposed indices can perfectly separate...’ (Line 60-62, Page 2): This statement should be modified. This claim can be made only if your methods have solid theoretical foundations. \n- ‘there was no such quantity defined in the literature to measure...’ (Line 139-140, Page 4): This is an over-claim. Specifically, all existing pruning-based defenses provided a quantity (e.g., activation values on benign samples) to measure it.  \n2. Missing some important related works. \n- No discussion about pre-processing-based backdoor defenses (e.g., [1-3])\n- Missing two poison-suppression-based backdoor defenses [4, 5].\n3. My biggest concern is about the experiment parts.\n- How to select the hyper-parameter u? Do you use the same u under different attacks on all datasets?\n- To ensure a fair comparison among all pruning-based defenses, I would like to see the results (i.e., ACC and ASR) with respect to the pruning ratio. It is necessary to verify whether the proposed methods are truly better in finding malicious neurons.\n- I am interested in whether the proposed defenses are truly resistant to adaptive methods since only poison-only backdoor attacks are used in the evaluations. The author should also test their methods on some training-controlled attacks (e.g., [6, 7]), which aim to reduce the difference between poisoned samples and benign samples in the hidden feature space.\n\n\nMinor Comments\n1. The ‘we will’ should be ‘we’ in Line 36 (Page 1).\n2. There are two typos in Table 2: ‘0.1’-->‘0.10’, ‘64.2’-->‘64.20’.\n3. Please recheck and cite the official version of all references (e.g., [8-9]).\n", "limitations": "The authors fail to adequately mention the limitations and potential negative societal impact of their work. Please discuss them explictly.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros \n1. The topic is of sufficient significance and interest to NeurIPS audiences. \n2. The authors discuss the resistance to potential adaptive attacks, which should be encouraged.\n3. In general, the idea is easy to follow.\n4. The authors discuss the efficiency of different defenses, which should be encouraged.\n\n\nIn general, I enjoy reading this paper. However, I still have some concerns about this paper. I will increase my scores if the authors can (partly) address my concerns. The detailed comments are as follows:\n\n\nCons\n1. Some statements need further support or modifications to avoid over-claim.\n- ‘we claim that both the proposed indices can perfectly separate...’ (Line 60-62, Page 2): This statement should be modified. This claim can be made only if your methods have solid theoretical foundations. \n- ‘there was no such quantity defined in the literature to measure...’ (Line 139-140, Page 4): This is an over-claim. Specifically, all existing pruning-based defenses provided a quantity (e.g., activation values on benign samples) to measure it.  \n2. Missing some important related works. \n- No discussion about pre-processing-based backdoor defenses (e.g., [1-3])\n- Missing two poison-suppression-based backdoor defenses [4, 5].\n3. My biggest concern is about the experiment parts.\n- How to select the hyper-parameter u? Do you use the same u under different attacks on all datasets?\n- To ensure a fair comparison among all pruning-based defenses, I would like to see the results (i.e., ACC and ASR) with respect to the pruning ratio. It is necessary to verify whether the proposed methods are truly better in finding malicious neurons.\n- I am interested in whether the proposed defenses are truly resistant to adaptive methods since only poison-only backdoor attacks are used in the evaluations. The author should also test their methods on some training-controlled attacks (e.g., [6, 7]), which aim to reduce the difference between poisoned samples and benign samples in the hidden feature space.\n\n\nMinor Comments\n1. The ‘we will’ should be ‘we’ in Line 36 (Page 1).\n2. There are two typos in Table 2: ‘0.1’-->‘0.10’, ‘64.2’-->‘64.20’.\n3. Please recheck and cite the official version of all references (e.g., [8-9]).\n\n\nReferences\n1. DeepSweep: An Evaluation Framework for Mitigating DNN Backdoor Attacks using Data Augmentation. AsiaCCS, 2021.\n2. Backdoor Attack in the Physical World. ICLR Workshop, 2021.\n3. Neural Trojans. ICCD, 2017.\n4. Backdoor Defense via Decoupling the Training Process. ICLR, 2022.\n5. Anti-Backdoor Learning: Training Clean Models on Poisoned Data. NeurIPS, 2021.\n6. Backdoor Attack with Imperceptible Input and Latent Modification. NeurIPS, 2021.\n7. Bypassing Backdoor Detection Algorithms in Deep Learning. EuroS&P, 2020.\n8. Backdoor Learning: A Survey. IEEE TNNLS, 2022.\n9. Backdoor Scanning for Deep Neural Networks through K-Arm Optimization. ICML, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0fQ5UZZkHC2", "reviewer": "Reviewer_Yu18", "summary": "In this paper, the authors studies how to better defend models from backdoor attacks. Specifically, they identified that there are only a subset of neurons, which are responsible for the poisoned behaviors. The found that if one prunes these poisoned neurons, then the backdoor behavior can be effectively stopped. The authors found a simple way of identifying such neurons through the two measurements: discrepancy of differential entropy and mismatched batchnorm statistics. By using these two measurements, they are able to more effectively stop the backdoor behavior while requiring very few additional compute. The also can identified such backdoor neurons with as few as 10 benign examples.\n\n\n\n", "questions": "I don't have any questions about the paper as of now.", "limitations": "The authors have adequately address the limitation of their approach and societal impact of their work.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The biggest strength of the paper is the method's effective as well as simplicity\n- The presentation of the paper is very clear and easy to follow \n- The experiments are thorough and convincingly shows the benefit of the proposed approach.\n\n**Weakness**\n- I do not find any obvious weakness in the paper. The wording may be improved slightly, but it doesn't impact the message of the paper very much.\n- Given its superior effectiveness in defending against the standard backdoor attacks in standard settings, I am curious whether it can also help stop the backdoor attacks in self-supervised or unsupervised settings. I think the contribution of the paper is already solid, but am simply interested in learning more about its implication for other backdoor settings.\n\nOverall, I really enjoy reading the paper, and recommend the paper being accepted.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept