Defense against Backdoor Attacks via Identifying and Purifying Bad Neurons
Authors: Mingyuan Fan
Yang Liu
Cen Chen
Ximeng Liu
Wenzhong Guo
Venue: Submitted to ICLR 2023 (2023)
Keywords: backdoor defense
security
neuron importance evaluation

Abstract: Recent studies reveal the vulnerability of neural networks to backdoor attacks. By embedding backdoors into the hidden neurons with poisoned training data, the backdoor attacker can override normal predictions of the victim model to the attacker-chosen ones whenever the backdoor pattern is present in a testing input. In this paper, to mitigate public concerns about the attack, we propose a novel backdoor defense via identifying and purifying the backdoored neurons of the victim neural network. Specifically, we first define a new metric, called benign salience. By combining the first-order gradient to retain the connections between neurons, benign salience can identify the backdoored neurons with high accuracy. Then, a new Adaptive Regularization (AR) mechanism is proposed to assist in purifying these identified bad neurons via fine-tuning. Due to the ability to adapt to different magnitudes of parameters, AR can provide faster and more stable convergence than the common regularization mechanisms in neuron purifying. Finally, we test the defense effect of our method on ten different backdoor attacks with three benchmark datasets. Experimental results show that our method can decrease the attack success rate by more than 95% on average, which is the best among six state-of-the-art defense methods.

Under review as a conference paper at ICLR 2023
DEFENSE AGAINST BACKDOOR ATTACKS VIA IDENTI- FYING AND PURIFYING BAD NEURONS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent studies reveal the vulnerability of neural networks to backdoor attacks. By embedding backdoors into the hidden neurons with poisoned training data, the backdoor attacker can override normal predictions of the victim model to the attacker-chosen ones whenever the backdoor pattern is present in a testing input. In this paper, to mitigate public concerns about the attack, we propose novel backdoor defense via identifying and purifying the backdoored neurons of the victim neural network. Specifically, we first define a new metric, called benign salience. By combining the first-order gradient to retain the connections between neurons, benign salience can identify the backdoored neurons with high accuracy. Then, new Adaptive Regularization (AR) mechanism is proposed to assist in purifying these identified bad neurons via fine-tuning. Due the ability adapt to different magnitudes of parameters, AR can provide faster and more stable convergence than the common regularization mechanisms in neuron purifying. Finally, test the defense effect of our method ten different backdoor attacks with three benchmark datasets. Experimental results show that our method can decrease the attack success rate by more than 95% average, which is the best among six state-of-the-aa defense methods.
INTRODUCTION
The brilliant feat of neural network (NN) make it a focal point of many attacks, one of the most threatening among which is the backdoor attack (Liu et al. 2017 Barni et al. (2019). By mixing poi- soned data into the training set, backdoor attack can control the victim NN to output attacker-chosen predictions for triggered inputs, while hardly disturb the predictions of normal inputs. Moreover, the triggers crafted by the attacker can be only several blocks of pixels (Li et al.) 2020b) or even the
invisible noises (Zhong al. 2020), which makes the attack notoriously perilous in applications.
Based on the findings in (Gu et al. 2017). backdoor attacks can succeed because the neurons that memorize trigger patterns, often called bad neurons, only keep strong connection with the triggers but rarely react to normal features. Naturally, to defend the attack, direct idea is to remove or mitigate the effect of bad neurons, e.g., model-pruning fine-tuning (Gu et al. 2017). However, the existing defense methods are mainly flawed from two perspectives.
Neuron Evaluation. To evaluate which neurons backdoored, current approaches mostly based on the activation magnitude (AM) of neurons (Gu et al. 2017). Neurons with low AMs about normal inputs are deemed to "bad", and pruned during the defense stage. Despite being intuitive and easy to apply, such metric sometimes leads to the over-pruning of good neurons it may ignore the connections between neurons. For instance, consider a case where clean neuron A in the network has low AMs but has very high weights its downstream neurons. N has strong positive effect on the final predictions but is still judged to be bad based on the above rule.
Neuron Purifying. As bad neurons marked in the evaluation stage, the next step is to purify or prune them. Currently, most pruning based defense methods choose to roughly remove bad neuron from the backdoored network. However, as pointed out by prior works (Liu et al. 2018). some bad neurons can also be related to the predictions of normal data. Roughly removing these neurons can easily cause the performance degradation of the target model.

---

Under review as a conference paper at ICLR 2023
In this paper, we propose novel backdoor defense method (called WIPER) to overcome the two flaws. Specifically. instead of utilizzig AM, we define a more effective metric called benign salience (BS. Section 4.1 to evaluate the importance of neurons. Compared to AM, BS retains the connec- tions between neurons through the first-order gradient, hence bad neurons can be identified more accurately. Then, after identifying bad neurons, our method chooses fine-tune but not directly prune them in the neuron purifying stage (Section (.2) to avoid unexpected performance degra- dation. Through a newly designed adaptive piece-wise regularization mechanism, our fine-tuning method can be far more effective in mitigating the network's 's attention on trigger patterns than the existing fine-tuning method (Truong et al., 2020p. Our contributions are summarized in four folds:
We propose WIPER, a novel backdoor defense method that combine the advantages of both model pruning and fine-tuning to identify and purify backdoored neurons.
We design a new metric BS to mark the bad neurons whose attention misled to the backdoor trigger patterns. Since the connections between neurons are reserved via the first-order gradient, defenders can use BS to distinguish bad neurons with higher accuracy than the commonly used metrics, e.g., AM.
We develop a new type of regularization adaptive regularization (AR). Compared to the common regularization, AR can better accelerate and stabilize the purifying process of bad neurons by adaptively adjusting the penalty degree to different magnitudes of parameters.
We conduct extensive experiments on benchmark datasets to validate the effectiveness of WIPER. The result shows that our method outperforms state-of-the-art defenses signifi- cantly on both attack success rate decreasing and model performance maintenance.
2 RELATED WORKS
Backdoor attack. A typical backdoor attack is done by injecting small volume of poison data crafted with the attacker-chosen triggers into the training set. To maintain the stealthiness of back- door attacks, the triggers used by attackers are various. For instance, the trigger used in (Li et al., 20206) was a simple rect with the resolution of 3 x 3. Blend attack (Chen et al.] 2017) adopted common life devices, e.g. glasses, as trigger so to evade the inspection of humans. Moreover, the authors in (Zhong et al. 2020) attempted to design a human-inpercepptible yet effective noise. The clben-label attack (Shafahi et al., 2018) made the natural features in images difficultly learnable so that the model was forced to only rely on the trigger to correctly classify without modifying the label. More recently, similar to clean-label attack, sinusoidal signal attack (Barni et al., 2019) de- signed an easily-learned trigger to conduct the backdoor attack. WIPER is designed to provide an effective defend the
attacks and ensure model security in applications.
Backdoor defense. Backdoor defenses can be roughly divided into two categories: detection-based methods (Wang et al. 2019 [2000) Xu et al. (021) that aim to detect whether a neural network is backdoored, and pwrifying-baase
methods (Zhao et al.) 2019 Truong et al., 2020) Yoshida & Fujino, 2020 Li et al.) 2021a) that try to remove the backdoor while maintaining the performance of the target model. Up to now, the detection-
method has been well developed and many remarkable works (Gao et al., 2019, Wang et al. 2020 Xu et al.) 202T achieve quite a high detection rate. Thus, this paper mainly focuses on the bad neuron purifying remove the backdoor. Inspired by the fact that the bad neurons were dormant with the presence of clean data, fine-pruning (Liu let al. (018)) removed the backdoor by erasing the neurons with activation values below a certain threshold. However, the effectiveness of fine-pruning heavily depends on the quality of holding data, and can be casily evaded by some state-of state-of-the-ard attacks, such as TrojanNN (Liu et al., 2017) In (Truong et al. 2020), the authors suggested that referring to catastrophic forgetting (Delange et al.) 2021), fine-tuning the model with some clean data was a simple yet effective method to remove the backdoor. Similar to fine-pruning fine-tuning also need high-quality clean data to mitigate the effect of bad neurons. Recently, knowledge distillation (Yoshida & Fujino 2020; Li et al. 202 was proposed to achieve backdoor defense by distilling clean knowledge from the infected model to a fresh model but failed to defend TrojanNN (Liu et al. 2017) due to the lack of consideration for its special trigger attention mechanism. Recently, novel work, Al-Lancet (Zhao et al. 2021) was proposed to locate bad neurons by doing comparative experiments with triggered inputs and clean inputs whose trigger-pasted regions were cropped. Such idea achieves competitive defense
2

---

Under review as a conference paper at ICLR 2023
performance against pixel-level backdoor attacks, however, fails to respond to the state-of- state-of-the-art attacks that uses crafted noises as triggers to cover the whole input image.
3 PROBLEM DESCRIPTION
Problem Setting. We consider the typical backdoor attack scenario as discussed in prior works (Gu ict al., 2017; Liu et al. 2017; Barni et al. 2019 Li et al., 2020b). the scenario, model trainer is with the knowledge of the training set D. collected from multiple data providers and self-owned clean set D.. The trainer seeks to leverage D, to train clean neural network F. While, among the data providers, there exists at least malicious one A that attempts to backdoor the trained model F by stealthily adding poisoned data into D.-
Attack Goal. In the above scenario, the attacker A aims complete the following goals.
By crafting the testing inputs with specific triggers, A can mislead the backdoored net- work F to output desired labels that are different from the ground-truth predictions (or predictions of clean network F).
To maintain the stealthiness of the attack, A should make the backdoored network FF to perform similar to as being given the clean inputs.
Defense Goal. As opposed to the attack goals, the defender, i.e., the model trainer who has full access to the internal architecture of the target model, aims to achieve the following two goals.
The first is to erase the backdoors from F' and make the purified model perform correctly as being given with triggered inputs.
To maintain utility, the purified model should perform similar to the original model trained with clean data (i.e., Dr without poison data).
Note, to ensure the feasibility of the defense method in varying applications, e.g., federated learn- ing (Li al. 2020a), the defender is limited to only utilizing its self-owned validation set to achieve the defense goals.
APPROACH
In this section, we present the design details of WIPER. As illustrated in Algorithm [ and Figure [1] WIPER mainly proceeds in two steps. The first is find the bad neurons required to be cleansed by evaluating the impor- tance of neurons. Then, a neuron-pune puuron-purifying strategy is leveraged to disable the effect of these bad neurons from the backdoored model.
Torgrt Jobel
Priam data
Triggen activsti the bacldoorrd neneons muleed model predict labe)
4.1 NEURON IMPORTANCE EVALUATION
Clicss data
Loss
WIPER uses a novel metric called BS to evaluate the importance of neurons and distinguish bad neurons. As mentioned before, BS outperforms the conventional metric AM used in the existing pruning- defense method (Liu et al. (0188) because instead of evaluating each neuron independently, BS additionally considers the connections between neurons in importance evaluation.
Observe valuze and gralicats of d'scightt associstal certin nrrecn on clean data ditect bacisdorred nencons
2. puiity the comeesooudngg wzights detected backscoored zeureas
Figure 1: Backdoor attacks (top image) and WIPER (bottom image).
4.1.1 THE DEFNITIIO OF BS
In more details, the definition of BS basically follows principles: 1) the importance of neuron positively correlates to its contribution to the loss decreasing of the target network; 2) connections between neurons established during the gradient backward propagation process in the training
3

---

Under review as a conference paper at ICLR 2023
Protocol WIPER
Input: F': the backdoored model; De: a set of clean data owned by the defender; A: the set of adjust the purifying magnitude; fo: reserved ratio of neurons; I: the tag of the layer required to be purified; n: the total number of epoch of purifying: 77. momentum accumulation factor.
neurons needs to be purified; a: the hyperparameter
Output: F: the purified model.
1: Neuron Importance Evaluation.
2: Compute the gradient of all neurons the I-th layer over De.
3: Evaluate the importance for the neurons on the I-th layer of F based on Eq. Eq. Eq.
4: Select 30 neurons with least importance (i.e., the lowest MBS) in the 1-th layer and add them into A.
5: # Neuron Purifying.
6: for each iteration i = to n2 do
7: Update 8, 30 - -1-11
8: for 2, y De do
9: Optimize FF' based on Epurifying
10:
Update M MB; for each neuron in 1-th layer and reset A that select B; neurons with the lowest . BS to add into A.
11:
end for
12: end for
stage. Following the first principle, we first define BS to be the differential loss format.
BS ALD. (wo) (w.,
where w denotes the parameters of neurons in a trained (backdoored) network F and wo the initial parameters that usually approximates to 0 (Sun 2019). Then, combined with the second principle, we can rewrite the computation of Eq. through Taylor expansion.
BS =0 CD. (0) - (D. = (w) + (w<..()) w) R1) (D.(w)
(2)
= =wV
R1 x - vwLCD,ww)'
where R1 the Taylor remainder consisting of 2-th derivative of FD. (w) and quadratic term of w, which can be omitted in the final deduction. Note that, if LD. (.) is non-linear function, the errors induced by the ignorance of R₁ can be non-negligible. However, for most neural network architectures, the last a few layers (output layer) At this time, the linear approximation in Eq. - becomes reasonable and effective. Therefore, in practice, WIPER pays more attention to purify the top fully-connected layers with BS achieve backdoor defense. We argue that such a setting is effective due to the following reason. Consider the scenario where the attacker only backdoors the feature extraction layers. The extracted feature about triggers can be simply blocked by the output layer if the neurons related to the trigger features are allocated with low weights. Also, the blocking can easily happen because the trigger features are useless for normal samples. Conversely, if the output layer is backdoored, the attack can be always successful as soon as the trigger feature is extracted. Therefore, to ensure attack effectiveness, the objective of most backdoor attacks is focused more the output layers. The same goes for the defender. The experimental results in Section 5.3 also prove the above statement.
always fully connected, i.e., being linear layers.
Moreover, Eq. indicates that BS can be used to quantify the importance of w. Then, the importance of a certain neuron can be quantitatively evaluated by the summation of BSs of its all downstream weights. Intuitively, higher BS indicates higher contribution of the neuron to the loss decreasing of Ff, i.e. with more importance, Therefore, as illustrated in Algorithm | (Step 4), WIPER marks neurons with the lowest BS as bad neurons, and the downstream weights of the marked neurons punished to diminish the impact of the neurons. Moreover, reconsider the backward propagation based gradient computation process. can be observed that BS not only considers the activation weight of each neuron but also involves its connection with downstream neurons into importance evaluation via the first-order gradient V.LD. (w)T. Thus, unlike the conventional metric AMD BS can better avoid the problem of over-pruning good neurons.
In prior backdoor defense methods (Liu et al. 2018 AM is computed by AM |w where ur is the neuron parameters and in the input.
4

---

Under review as a conference paper at ICLR 2023
Except for BS, we define, that can be adopted to evaluate the importance of neuron, there many inspiring metrics that own the similar function, a typical one of which is neuron shapley (Ghorbani & Zou 2020). Theoretically, neuron shapley can perfectly assess the contribution of a neuron to the overall performance of the model but seriously suffers from the unaffordable exponential computa- tion costs (even if with some mitigation measurements such pruning algorithms Monte Carlo estimation (Ghorbani & Zou| ). For backdoor defense, it requires not only the excellent ability of backdoor erasing but also acceptable time complexity in applications. Therefore, we argue that neuron shapley is not a suitable metric to evaluate the importance of a neuron in backdoor defense scenarios. Thus, the beginning of our design, we decided to abandon these complex methods and propose new metric BS to resolve the problem.
4.1.2 FURTHER DISCUSSION
While implementing WIPER, we notice that although WIPER converges in the expected direction, its convergence curve suffers from great oscillation. Review the definition of BS in Eq. 2 It can be discovered that the unstability is mainly caused by the fact that limited by the real-world computation resource, WIPER sometimes has to adopt Stochastic Gradient Descent (SGD) to complete model purifying. With SGD, only one batch of data is taken into computation thereby easily leading the gradients used to compute BSs to be trapped into local optimum thereby degrading the performance of WIPER. At this time, we can use the momentum accumulation mechanism to mitigate the problem as follows.
(B)((w)= = n (I)) + (1 - 7)) BS(w), 0 S S 1, where (w) denotes the accumulated BS of w in the i-th epoch and v) is the momentum factor given by defenders.
(3)
Furthermore, in WIPER, the initial proportion 3 of purified neurons (with least BS) is an empirical hyperparameter. Considering that the neurons required be purified will be on the decrease along with the model purifying process, 3 usually starts with large value and then is slowly decayed until the purifying process is completed. Referring the decaying mechanism of learning rate, we define the following decaying function in WIPER.
B Bo epoch - cur-epoch
(4)
epoch
where An is initial value of B, epoch is the number of training epochs and cur eur-epoch denotes the current epoch.
4.2
NEURON PURIFYING
We now elaborate how WIPER leverages improved neuron fine-tuning strategy to achieve bad neuron purifying.
Without consideration of model performance, basic idea for neuron purifying is to wipe out bad neurons by forcing all the parameters of them to be O. Following the idea and adding the model performance into consideration, we can derive the following way to achieve bad neuron purifying like (Liu et al. 2018).
n
ypirifying
CFFFF(a))) + a Σ -|lk-
(5)
WEA
where denotes the set of bad neurons, CE(-) is the cross-entropy loss function, a is the penalty coefficient and ||w 0|l is the La-norm regularization term (k E {1,2)) that promotes the pa- rameters of bad neurons to lose impact on the model outputs. Compared to l-grregularization, L1 Lyrrgulaarzztion is known to be able to completely cleanse the impact of but requires more optimization rounds as w is large (Hoetler et al. 2021). In contrast, Ly-regularization can accelcrate the purifying process but never makes to exactly 0 thereby easily causing the overshoot of the global optimum (also discussed in Section [.6) To address the issue, we design a piece-wise regularization item called adaptive regularization (AR).
w-1
w < - 1,
AR(w)= =
|
|w| < 1,
(6)
w-1
1.
5

---

Under review as a conference paper at ICLR 2023
20
As shown in Fig. AR provide higher penalty degree than L2 to accelerate conver- genee when the imput parameter is large. Mcan- while, the penalty degree of AR is similar to the I. T--norm regularization close to O. In our evaluation, we will demon- strate that benefited from the ability process different magnitudes of parameters, AR ensures faster convergence rate and high stability than L1 and L2 regularization in neuron fine-tuning (see Fig. 8 in Section 5.6). Finally, with AR, the purifying loss can be rewritten as follows:
Gradient Size
10
the parameter is
-10
L2
AR
220
2
Figure 2: The penalty degree of three types of reg- ularization items over different magnitudes of in- puts
Epurifying (E(F Σ AR(w,).
JEA
EXPERIMENTS
In this section, we mainly experiment with the effectiveness of WIPER in backdoor defense and answer the following research questions (RQs).
RQ1 Effectiveness: Can WIPER defend the state-of-the-an backdoor attacks?
RQ2 A Closer Look BS: Is BS good metric in identifying bad neurons?
RQ3 Improvement: Is AR more suitable for bad neuron purifying than the common reg- ularization?
RQ4: Approximation Availability: Is purifying output layers enough to defend backdoor attacks?
Table 1: The performance of WIPER compared with six state-of-the-arr defense methods over backdoor attacks in CIFAR-IO.
Defense
Before
Pine-tuning
KD)
NAD
Fine- Fine-Pruning
IBAU
ANP
WIPER
Attack
ACC:
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASRR
ACC
ASR
BadNet
78.42
70.54
83.20
BA
87.67
99.77
78.69
18.10
50.66
6.94
70.58
13.51
80.77
25.06
80.97
82.26
5.46
82.82
5.22
ETA
87.2
97.24
75.57
28.62
50.93
9.71
71.68
10.42
81.85
15.79
82.85
9.03
81.20
6.45
82.4
5.80
IA
86.83
98.67
78.38
28.10
49.10
7.29
70.71
5.87
78.73
42.40
82.46
10.51
80.91
5.78
$2.04
6.02
TrojuNNN
79.46
73.37
51.27
71.71
67.92
34.67
4.68
IMC
87.10
99.78
78.40
77.56
50.62
8.66
70.86
33.73
81.62
75.63
44.54
15.87
80.04
7.4%
80.92
6.77
WaveNer
85.87
97.68
77.24
18.10
$1.87
7.55
70.41
13.35
81.45
17.94
80.66
9.96
82.48
7.21
$2.59
5.81
SSA
LogitAttakk
86.92
99.99
79.08
23.92
49.51
5.1 EXPERIMENT SETTINGS
Baselines. Four state-of-the-art defense methods are considered as baselines, which are Fine- Pruning (Liu al., 2018)) Fine-tuning (Truong et al. (2220), KD (Yoshida & Fujino 2020). NAD (Li et al., 2021a). IBAU (Zeng et al.) 2022), and ANP (Wu & Wang 2021). We compare their de- fense effect with WIPER against backdoor attacks, namely badNet (Gu et al. 2017). blend attack (BA) (Chen et al. (017). enhanced trigger attack (ETA) (Li et al.) 2020b). invisible attack (IA) (Zhong al. 2020), sinusoidal signal attack (SIG) (Barni al. (209)) TrojanNN (Liu al. 2017) IMC (Pang al. (020), WaveNet (Nguyen & Tran 2021). SampleSpecifiedAAtae
(SSA) (Li
ct al. 2021b). and LogitAttack (Zhang et al. (202)) on three benchmark datasets SVHN, CIFAR- 10, and CIFAR-100. For fair comparisons, all the configurations of these attacks and defense follow their corresponding original papers (Appendix A.3 and A.4). Moreover, all defense methods are implemented with access parts of the clean data randomly selected from the training set (5% to the size of training sets but never used in the training process like (Yoshida & Fujino 2020 Li al.
202 Liu al. 2018 Zeng et al. 2022

---

Under review as a conference paper at ICLR 2023
Attack
BadNet
BA
ETA
IA
SIG
TrojanNN
Method
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
Before
87.42
99.50
87.67
99.77
87.20
97.24
86.83
68.67
85.14
91.10
86.83
98.67
Fine. Pruning + AM
80.68
14.70
80.77
25.06
81.85
15.79
78.73
42,40
80.69
16.03
80.40
67.92
Fine-Pruning BS
81.11
10.47
81.70
14.22
82.34
10.72
80.20
18.11
80,79
11.71
81.85
16.50
Ours + AM
81.72
12.35
82.03
17.28
81.97
12.78
81.51
30.55
81.88
15.17
83.09
55.32
Ours BS
83.20
5.03
82.82
5.22
82.41
5.80
82.04
6.02
82.41
3.73
84.67
4.68
Table 2: Comparison of backdoor defense with different purifying strategies and neuron importance evaluation metries on CIFAR10.
Evaluation Metrics. The performance of backdoor defense is evaluated with two widely-used metrics: the accuracy of the model on clean samples (ACC), and the attack success rate (ASR). ASR denotes the ratio of triggered samples that succeed in misleading the target model to output attacker- chosen predictions. An effective defense method should significantly lower ASR and remain high ACC, All metrics are average values computed with the whole testing set that are never used in the training purifying stage.
Other Setups. While implementing WIPER, we fix a, fg. 17 to be 0.01, 0.5, and 0.9. The number of epochs for neuron purifying is set to 10 similar to (Li et al. 2021a). The purified layer is the output layer (usually the last full connection layer in the network) (Liu et al.) 2018). Moreover, we adopt standard SGD with constant learning rate of 0.01 and batch size of 128. All experiments are conducted using ResNet18 as the target model with a workstation equipped with Nvidia V100. Parts of the results with SVHN and CIFAR-I00 are left in the supplementary material.
5.2 RQ1: EFFECTIVENESS
To validate the effectiveness of WIPER, we first test and analyse the defense effect of WIPER against different backdoor attacks. Then, we compare the performance of WIPER with other defense methods. Table Table (Appendix A.2) and Table (Appendix A.2) summarize the experimental result of five different backdoor defense approaches against six state-of-the- backdoor attacks with SVHN, CIFAR- 10, and CIFAR-100. In the tables, Before means the ACC and ASR of backdoored models before being purified.
Overall, it can be observed that the performance (simultaneously consider both ACC and ASR) of WIPER considerably outperforms the other backdoor defense methods under different settings. Specifically, considering the decrease of ASR, two state-of-the methods, KD and NAD, can achieve competitive performance compared with WIPER but notably suffers from the problem of model performance degradation. For example, to maintain the ASR to be less than 10%, KD sacri- fices at most 35% loss of ACC while WIPER only 5%. For NAD, it attains similar performance to WIPER most backdoor attacks but fails to defend the TrojanNN attack. This is because besides adding poisoned data into the training set, TrojanNN allows the attacker to select and manipulate specific neurons enhance the memory of backdoored model about the trigger patterns. The model distillation method used in NAD make the purified model inherit the "enhanced memory" from the backdoored model thereby leading the low defense effect of NAD on TrojanNN. In contrast, the bad neuron filtering mechanism of WIPER can accurately mark these backdoored neurons and lower the ASR of TrojanNN significantly. Furthermore, considering performance maintenance, WIPER can ensure lower ACC degradation of the model compared with other defense methods in almost all conditions. For instance, in SVHN (Table 77. with only less than 1% degradation of the model performance, WIPER can drop the ASR of six attacks to less than 5%. However, the other method still struggles against the attacks, especially for Preuninung.
5.3 RQ2: A CLOSER LOOK AT BS
As mentioned before, one of the key factors make WIPER outperform other methods is the in- troduction of BS. Here, comprehensive experiments are conducted to explain why BS can achieve such an improvement
backdoor defense from three perspectives: 1) statistical analysis; 2) atten- tion analysis; 3) effect on neuron purifying. In the experiments, a ResNet18 network is trained with CIFAR10 10. During the training process, we backdoor the model according the widely-used bench-

---

Under review as a conference paper at ICLR 2023
Benign Salience (BS)
Activation Magnitude (AM)
6e-5
Clean Data
Clean Data
Attention of benign neurous
Poison Data
Attention of benign nearons
Poison Data
4e-5
0.8
0.6
3e-5
Attention of
Attention of bad neurons
bud neirróns
0.2
0c-5
The real place of trigger
The real place
of trigger
0.4
0.6
-4-55
For WIPER, the neurons with lowest BS on clean data are considered bad.
For Fine- Mre-Pruning, the neurons with lowest AM on clean data are considered bad.
Figure 3: The left graph is about experimental result of BS, and the right one about Fine-Pruning with AM. For both of the two graphs, the horizontal axis indicates the neurons of the victim model. Each bar corresponds to one neuron. All recorded BS and AM values are averaged over 1000 testing samples (500 for clean and 500 for poison data). Especially, the BS and AM for clean data are sorted in descending order for the convenience of observation.
mark attack BadNet (Gu et al. 2017) and ensure 100% ASR. The poison label and poison ratio set to O and 5%. The trigger is a 3 square in the upper left corner (Gu et al., 2017).
Statistical Analysis. We record the averaged BSs of all neurons by feeding batch of clean and poison data, 500 for each, as shown in the left of Figure 3 From the results, an interesting finding is that the BSs neurons are almost opposite for clean and poison data. In fact, such a phenomenon is precisely what we desire in bad neuron filtering. This is because the phenomenon proves that BS backdoor attacks: making some neurons to be strongly activated by
well describes the principle triggers to change the original label output and rarely activated by clean data to avoid lowering the performance of victim model on normal inputs. Also, we can use BS to accurately mark the neurons with low importance normal predictions but high significance to trigger activation. Referring to AM, shown in the right of Figure 3 can be observed that there is no significant difference in AM between neurons. Such flaw makes AM unable to provide a clear division between benign and bad neurons, and leads the bad neurons marked by AM to be mixed in some benign neurons.
visualize the attention of neurons on the original input image in Figure The results show that for both BS and AM, the attention of benign neurons is mostly focused on the main body of the input image, i.e.,around the airplane. Then, the upper left corner, the place where triggers are added, draws the attention of bad neurons marked by BS. Clearly, tribute the most to backdoor attacks. Conversely, the attention of the bad neurons marked by AM is not only focused on the upper left corner but also somewhere else.
Attention Analysis. Further, validate the above statement,
proved that with BS, WIPER accurately identifies which neurons con-
Effect on Neuron Purifying. Finally, we conduct experiments to show that the choice of neuron importance evaluation metric can lead to significant effect on neuron purifying. From the results of Table given the same purifying method (Fine-Pruning outperforms the ones with AM on both ACC and ASR. Especially, with AM, Fine-Pruning fails to defend the TrojanNN attack. However, with BS, Fine- Pruning succeeds in defending TrojanNN due to the better ability of BS to identify bad neurons. All the facts prove that BS a good metric.
WIPER), the defense effect with BS
5.4 RQ3: IMPROVEMENT OF AR ON NEURON PURIFYING
To show the advantages of AR on fine-tuning based neuron purifying, experimentally compare it with the other two commonly used regularization methods L, and Lg. Figure S (Appendix A.2)
illustrates the performance of WIPER employed with three kinds of regularization mechanisms.
is observed that the difference of these three kinds of regularization methods mainly stems from the changing trend of ASR. In more details, the intrinsic property of L2 makes it only "strong" on penalizing large parameters (more than 1, shown in Figure 2]. and unable to actually decrease parameters to O. However, for backdoored network, the parameters of bad neurons are usually
8

---

Under review as a conference paper at ICLR 2023
very small (close to 0). Thus, L2 fails to provide proper penalty on bad neurons and leads to the lowest ASR decreasing speed. Then, for L1, can always reduce the ASR to a similar level to AR However, since the penalty of L1 is too weak, its convergence speed is still lower than AR. Finally, the required epochs of AR is just around 2, which is significantly lower than I, and L2 regularization, which requires more than 15 epochs erase the backdoor. Thus, the adoption of AR in backdoor defense can greatly improve the performance of neuron purifying.
5.5 RQ4: IS PURIFYING OUTPUT LAYERS ENOUGH TO DEFEND BACKDOOR ATTACKS
In this subsection, we mainly answer the question of whether only purifying the last few layers is enough for erasing the backdoor from the victim model. Table 5 (Appendix A.2) reports the per- formance of WIPER as the target purifying layers are changed. From the results, adding feature extraction layer (i.e., convolution layers) into backdoor purifying shows little improvement fense effect, only less 1% more decrease on ASR. In contrast, the performance of the purified model degrades more significantly when more convolution layers are introduced. As aforementioned, such a phenomenon occurs because backdoor attacks usually pay less attention on the feature extraction layers than the output layer to achieve better attack effectiveness. Therefore, forcibly purifying the feature extraction layers contributes little to the backdoor defense but can easily lead to the over- pruning of the victim model and lower the model performance.
de-
5.6 OTHER CASES
Impact of Clean Data. As discussed in the prior works (Liu et al., 2018 Truong al. 2020
Yoshida & Fujino 2020 Li et al., (021a)). the amount of clean data owned by the defender is a crucial factor that affects the effect of backdoor defense significantly. Here, for the convenience of statistic and comparison, we use the ratio of clean data set to training set as the indicator to denote the amount of clean data owned by the defender. F (Appendix A.2) demonstrates the performance of five defense methods against six attacks over different data ratios. Strikingly, with the strict condition of 1% data ratio, WIPER can still maintain almost 90% ASR drops and less than 3.5% accuracy drops. In contrast, most of other methods fail to achieve effective backdoor defense (less than 40% ASR drops on some attacks) or suffer from significant ACC drops (more than 20%). ).
is vital factor to determine the degree of WIPER to punish the bad neurons. Table [6 (Appendix A.2) summarizes the experimental results of WIPER with coarse-tuning of a. The results show that larger makes WIPER focus more on backdoor pu- rifying and achieve lower ASR but is with the expense of slightly decreased ACC. Correspondingly, lower a allows the target model to maintain higher ACC but results in higher ASR. Therefore, a trade-off exists between ASR and ACC. However, the trade between ASR and ACC are not ex- changed at equal values. It can be found that in most cases, only less than 1% ACC drop can leads to the drop of ASR for about 50% to 60%. As a result, in practice, the defender is suggested to choose litter higher magnitude of ensure the effectiveness of WIPER.
Impact of a. As discussed in Section 4.2
Impact of Purifying Strategy. Furthermore, in Table 2 we also show the influence of purifying strategy choice on backdoor defense. As expected, despite of maintaining the performance of the purified model in an acceptable range, Fine-Pruning (fine-tuning with L1 + pruning) still suffers from poor capacity of backdoor erasing. However, if Fine-Pruning is replaced with our purifying strategy, the results can be improved significantly. For instance, referring to the TrojanNN attack, substituting Fine- Fine-Pruning with our purifying strategy can make the ACC rise from 81. 185% to 84.67% while the ASR drops from 16.50% to 4.68%, even with AM as the importance evaluation metric.
6 CONCLUSION
We proposed backdoor defense approach WIPER that could combine the advantages of both model pruning and fine-tuning. WIPER improved current backdoor defense method from two aspects The first was propose new metric called benign salience to evaluate the importance of neurons in network and filter the bad ones. The second was to design a new regularization item for backdoor purifying, which could accelerate the neuron purifying process significantly. Finally, extensive experiments to show that the WIPER achieved ideal performance on both crasing the backdoor and maintaining the model performance after being equipped with such two techniques.
conducted

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cmns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 101-105. IEEE, 2019.
Xinyun Chen, Chang I. Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. CoRR, abs/1712.05526, 2017. URL http::tttt/aaat
erg/aba/170 1712. 05526
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 113-125, 2019.
Amirata Ghorbani and James Zou. Neuron shapley: Discovering responsible neurons. arXiv
preprint arXhv:2002.09815. 2020.
Tianyu Gu, Brendan Dolan-Gand Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.0673:. 2017.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 1::11::1211:22 2021.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020a.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations, 2021a.
Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang. Zhifeng Li, and Shutao Xia. Rethinking the trigger backdoor attack. arXiv preprint arXiv:2004. 04692, 2020b.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference 2021b.
on Computer Vision, pp.
Kang Liu, Brendan Davin-Gavitt, and Siddharth Garg. Fine-pruning: Defending against back- dooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pp. 273-294. Springer, 2018.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen Ven-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. 2017.
Tuan Anh Nguyen and Anh Tuan Tran. Wanet-impercoptible warping-based backdoor attack. In International Conference Learning Representations, 2021.
Ren Pang, Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Alex Liu, and Ting Wang. tale of evil twins: Adversarial inputs versus poisoned models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp. 85-99, 2020.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp-
6106-6116 16, 2018
Ruoyu Sun. Optimization for deep learning: theory and algorithms. arXiv:1912.08955 2019.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2023
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Loc Truong, Chace Jones, Brian Hutchinson, Andrew August, Brenda Praggastis, Robert Jasper, Nicole Nichols, and Aaron Tuor. Systematic evaluation of backdoor data poisoning attacks on image classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 788-789 2020.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks, In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin- Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pp. 222-238. Springer, 2020.
Dongxian Wu and Yisen Wang. ArXiv, abs/21 10.110..4.. 14430, 2021.
Adversarial neuron pruning purifies backdoored deep models.
Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 103-120. IEEE, 2021.
Kota Yoshida and Takeshi Fujino. Disabling backdoor and identifying poison data by using knowl- edge distillation in backdoor attacks on deep neural networks. In Proceedings of the 13th ACM Workshop on Artificial Intelligence and Security, pp. 117-127, 2020.
Yi Zeng, Chen, Won Park, Z. Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. International Conference on Learning Representations, 2022.
Zhiyuan Zhang. Lingjuan Lyu, Weiqiang Wang. Lichao Sun, and Xu Sun. How to inject backdoors with better consistency: Logit anchoring clean data. International Conference on Learning Representations, 2022.
Pu Zhao, Pin-Ye Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In International Conference on Learning Representations, 2019.
Yue Zhao, Hong Zhu, Kai Chen, and Shengzhi Zhang. Ai-lancet: Locating error-inducing neurons to optimize neural networks. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2021.
Haoti Zhong, Cong Liao, Anna Cinzia Squicciarini, Sencun Zhu, and David Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. In Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy, pp. 97-108, 2020.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1
ALTERNATIVE METRICS
Here we first examine whether the gradient magnitude of neuron can be served as a good metric distinguish between good and bad neurons and Figure shows results. Similar AM, solely employing gradient magnitude is hard to detect bad neurons.
1.0
Clean Data
Poison Data
0.5
0.0
0.5
1.0
Figure 4: The gradient magnitude of each neuron with respect to clean data and poison data respec- tively. The evaluation settings follow Figure
BS vs. Neuron Shapley & Integrated Gradient. Some literature (Sundararajan et al. 2017 Ghor- bani & Zou 2020) developed various metrics quantify the importance of neurons. To show better practicality of BS, we compare our method with other importance evaluation metrics. Specifically, select common-used metrics: Neuron Shapley (Ghorbani & Zoul 2020) and Integrated Gra- dient Sundararajan et al. (2077). Table 3
reports the performance and overhead of three metrics. As can be seen, BS obtains competitive defense performance and only needs considerably smaller overheads compared with Integrated Gradient and Shapley value.
Table 3: The performance and overhead of different metrics against BadNet.
Metric
ACC
ASR
Time
BS
83.2
5.03
1,00
Integrated Gradient
83.25
5.10
10.76
Shapley Value
82.89
4.92
968.17
Table 4: The performance of Al-Lancet and WIPER against four backdoor attacks.
Defense
Al-Lancet
WIPER
Attack
ACC
ASR
ACC
ASR
BadNet
80.15
5.21
83.20
5.03
BA
73.81
27.17
82.82
5.22
ETA
68.67
29.93
82.41
5.80
SSA
68.47
22.14
81.7
4.76
A.2 ADDITIONAL EXPERIMENT ON DIFFERENT DATASETS
Firstly, demonstrate that the poor defense performance of Al-Lancet Zhao al. (2021) against state-of-the-an attacks. Table shows that Al-Lancet is vulnerable against BA, ETA, and SSA, which adopt more threatening triggers covering the entire images being dynamic.
12

---

Under review as a conference paper at ICLR 2023
Altack
BadNet
BA
ETA
SIG
TrojanNN
Purified Layer
ACC.
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
FC only
83,20
5.03
82.82
5.22
82.41
5.80
82.04
6.02
82.41
3.73
84.67
4.68
PC Cony
82.19
5.15
81.29
5,93
81.37
5.77
80.46
5,44
81.26
3.98
82.89
4.24
FC +C++2 Cony
80.29
4.79
79.31
5.24
79.42
5.38
77.66
5.69
79.82
4.05
81.00
4.01
PC 3 Conv
77.03
4.83
77.84
4.34
77
4.60
76.71
5.62
77.96
3.45
79.72
5.36
FC + 4 Conv
74.49
4.51
74.86
4.35
71.56
4.04
73.33
4.96
74.56
3.22
72.73
4.67
Table 5: The performance WIPER when purifying different layers. Here n Conv indicates puri- fying the last n convolution layers.
Attack
BadNet
BA
ETA
IA
SIG
TrojanNN
a
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
83,57
77.03
84.09
86.74
83.69
33.18
0.005
83.55
21.01
84.14
26.20
84,41
16.54
83.44
25.97
82.64
6.69
86.22
5.57
0.01
83.20
5.03
82.82
5.22
82.41
5.80
82.04
6.02
82.41
3.73
84.67
4.68
0.05
82.76
5.48
82.89
5.42
82. 10
5.48
81.97
5.69
81.78
4.02
84.25
4.86
0.1
82.20
5.30
82.18
5.07
81.55
3.96
81.33
4.72
81.20
3.44
84.39
4.39
Table 6: The performance of WIPER with different a against six backdoor attacks in CIFAR-10.
Here report some additional experimental results in SVHN and CIFAR-100. From experiments, we can further consolidate our conclusion that WIPER is an effective defense against backdoor attacks (also discussed in Section 5).
Vigure66ann Figure 7 iillurrate the defense performance of different defenses on SVHN and CIFAR- 100. Besides, Figure 9 and Figure show the defense performance of WIPER with varying regu- larization items on SVHN and CIFAR-100 100. Table 9 and Table report the defense performance of WIPER with different purifying strategies and neuron importance evaluation metrics on SVHN and CIFAR-100.
Defease
Before
Fine-tuning
KD
NAD
Fine-Pruning
IBAU
WIPER
Attack
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
BadNet
94.68
100
85.74
3.17
60.16
4.42
86.22
2.71
93.17
69.57
89.70
6.83
94,14
0.49
BA
94,62
100
86.54
1.44
52.4
2.61
86.47
3.11
92.29
79.65
87.95
9.91
94.09
1.43
ETA
94,59
99.98
87.61
6.34
53.08
4.98
87.56
6.28
93.41
68.52
91.22
10.98
94.11
4.94
3.26
52.45
86.25
1.87
94.23
87.53
89.71
94,53
SIG
94,38
96.97
84.85
1,31
53.22
6.87
85.3
3.22
93.16
38.53
88. 59
2.28
93.69
1.31
TrojanNN
49.64
31.97
92.31
88.22
11.34
94.52
IMC
94.15
100
87.52
10.56
52.62
5.15
86.41
32.53
93.50
98.23
87.52
18.16
93,90
1.92
WaveNet
6.26
4.57
92.54
88.06
3.25
93.90
SSA
94.60
97.41
85.22
2.22
52.83
7.59
85,30
4.07
94.95
40.76
87.52
5.27
93.33
3.17
LogitAttack
86.
51.68
86.30
93.60
68.75
90.01
6.62
Table The performance of WIPER compared with four state-of-the-art defense methods over six backdoor attacks in SVHN.
Defense
Before
Fine tuning
KD
NAD
Fine Pruning
IBAUI
WIPER
Attack
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
BadNet
69.66
99.83
60.74
11.01
18.84
1.51
32.80
2.32
63.48
25.53
59.29
3.44
63.71
1.12
BA
68.68
99.81
61.68
23.32
15.32
1.59
33.24
2.46
63.08
26.68
61.62
9.08
63.21
0.80
ETA
68.98
98.43
62.71
17.99
17.52
1.44
31.03
3.11
62.89
37.63
60.82
1.67
62.35
0.83
IA
69.12
99.88
59.44
27.79
16.57
1.91
31.37
3,02
60.97
23.30
60.27
5.94
62.11
0.55
SIG
68.92
90.21
57.48
21.18
15.48
1.58
30.50
1.62
61.43
28.07
59.11
8.12
63,89
0.26
TrojanNN
70.84
94.66
61.97
56.76
16.61
1.57
30.38
32.03
61.45
78.99
60.58
11.56
63.92
0.20
IMC
69.08
99.55
62.41
62.15
17.52
1.98
31.19
35.40
61.62
87.98
60.47
15.16
60.91
0.97
WaveNet
67.68
98.84
59.49
22.07
16.00
1.29
31.08
2.46
62.04
27.87
59.95
8.37
62.04
0.99
SSA
68.03
97.76
58.80
22.00
15.50
1.38
30.45
0.21
62.23
28.
61.00
6.46
60.82
1.38
LogitAllack
68.35
99.87
61.49
11.26
19.24
2.55
32.90
3.15
62.63
26.09
58.80
4.82
62.00
0.53
Table The performance of WIPER compared with four state-of-the-art defense methods over six backdoor attacks in CIFAR-100.
A.3 ATTACK IMPLEMENTATION DETAILS
We summarize the implementation details of ten backdoor attacks used in this paper as follows:
13

---

Under review as a conference paper at ICLR 2023
Attack
BadNet
BA
ETA
IA
SIG
TrojanNN
Method
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
before
94.68
100.00
94.62
100.00
94.59
99.98
94.68
100.00
94,38
96.97
94.71
100.00
Pruning AM
93.17
69.57
92.29
79.65
93.41
68.52
94.53
87.53
93.16
38.53
92.31
97.82
Pruning BS
93.58
17.58
93.01
19.25
93.58
24.92
93.07
13.78
93.52
12.8 12.84
93.41
20.88
Purifying AM
93.70
93.91
29.32
93.56
94.57
35.19
93.64
27.10
93.84
63.08
Purifying BS
94.14
0.49
94.09
2.87
94.11
4.94
94.23
9.48
93.69
1.48
94.52
2.41
Table 9: Comparison of backdoor defense with different purifying strategies and neuron importance evaluation metrics on SVHN.
Atlack
BadNet
BA
ETA
IA
SIG
TrojanNN
Method
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
before
69.66
99.83
68.68
99.81
68.98
98.43
69.12
99.88
68 92
90.21
70.84
94.66
Pruning AM
63.48
25.53
63.08
26.68
62.89
37.63
60.97
23.30
61.43
28.07
61.45
78.99
Pruning +
63.55
16.69
63.02
19.87
61.73
18.07
61.02
11.77
62.21
15.28
61.96
29.63
Purifying AM
63.62
20.31
63.15
23.51
63.01
25.04
61.75
19.27
62.52
22.67
62.83
70.57
Purifying BS
63.71
1.12
63.21
0.80
62.35
0.83
62.11
0.55
63.89
0.26
63.92
0.20
Table 10: Comparison of backdoor defense with different purifying strategies and neuron importance evaluation metries on CIFAR100.
BadNet: A 3x3 trigger with random pixel values is pasted in the top left corner of 5% training images, and labels tamper with O.
Blend Attack: We used the same trigger, hello kitty image, the original paper, blend ratio of 0.2, and inject rate of 0.05.
Enhanced Trigger Attack: Following the original paper, adopt the same random spa- tial transformation layer consisting rotation and scale to pre-process the poison data. The associated parameters of the trigger used in this attack are identical to BadNet.
Invisible Attack: We generated the same trigger with 32 32 resolution in the original paper and injection rate of 0.05.
Sinusoidal signal attack: We used the backdoor trigger generation function in the original paper with = 20 and f = 6 and injection rate of 0.1.
TrojanNN: Based on the implementation of the original paper, we utilized the same reverse engineer technology craft 3 square trigger from the fully connected layer. Other parameters are set the same in BadNet.
IMC: Similar to the original paper, used random noise with 3 x 3 size as the trigger and optimize the trigger during the training process.
WaveNet: We used the corruption technique with a transformation probability of 0.2 de- fined in the original paper process images.
We added noises produced by the generator used in the original paper for 20% of images.
SSA:
LogitAttack: We poisoned 5% training data with the trigger of 3 x 3 similar to BadNet.
A.4 DEFENSE IMPLEMENTATION DETAILS
We list the detailed defense settings of FinninPu Fine-tuning, KD, NAD, and I-BAU for refer- ence:
Pine-Pruning: As suggested in the original paper, we pruned the last layer of the model with a pruning rate of 0.1, where AM is measured over random subset of the training set as same to BS.
Fine-tuning: Following the original paper, adopted standard fine-tuning procedure with a fixed learning rate of 0.1, momentum factor 0.9, L2 weight decay factor of x 10-1 and cross-entropy loss function recover the backdoored model.
14

---

Under review as a conference paper at ICLR 2023
AX
1141
1141
1141
1144
1184
J0-
25
2'4
Figure 5: The defense performance of five defense methods over different data ratio (0.01, 0.05, 0.1, 0.15, 0.2) in CIFAR-10. The images from left to right indicate against BadNet, BA, ETA, IA, SIG, and TrojanNN.
E1
E1
Dea
16-
Figure 6: The defense performance of five defense methods over different data ratio (0.01, 0.05, 0.1, 0.15, 0.2) in SVHN.
-Traing
-Traing
-Traing
-Traing
-Traiig
3-4
Figure 7: The defense performance of five defense methods over different data ratio (0.01, 0.05, 0.1,
0.15, 0.2) in CIFAR-100.
KD: The standard knowledge approach is employed recover the backdoored model, where cross-entropy loss and KL divergence are fused together as the overall loss function. The weights of the two loss terms are set to 0.5 along with the temperature of and epoch of 50.
NAD: We replicated the defense of NAD, where the backdoored model is fine-tuned for 10 epochs with an initial learning rate of 0.1 and momentum of 0.9. The learning rate is divided by 10 after every 2 epochs.
I-BAU: For I-BAU, we reused their original hyperparameter Cs = 10.
15

---

Under review as a conference paper at ICLR 2023
Figure 8: The performance of WIPER with different regularization items (L1, L2, and AR) on
CIFAR-10. 10. The images from left to right indicate against BadNet, BA, ETA, IA, SIG, and TrojanNN.
1p
Figure 9: The performance of WIPER with different regularization items (I1, Lz, and AR) SVHN.
Figure 10: The performance of WIPER with different regularization items (L1, L2, and AR) on CIFAR-100 100.
16


Reviews:
[{"review_id": "yiG95VAI4xB", "reviewer": "Reviewer_krJA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a measure to distinguish benign neurons and malicious neurons. Based on the measure, the authors identify the malicious neurons and update their parameters. Experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "This paper misses many details and is hard to understand. Even if the proposed method might be promising, the author should clarify it well. In my opinion, it requires a major revision. Thus, I recommend rejecting this paper.", "strength_and_weaknesses": "**Strength**\n1. The empirical results demonstrate the effectiveness of the proposed method.\n2. If I understand correctly, the author defines the importance of a weight parameter using the element-wise product of the weights and their gradients. There is a similar formulation (element-wise product of the input and the gradient) in Saliency Maps [1, 2]. If we could find the similarities between them, it must be very interesting.\n\n**Weakness**\n1. This paper is written badly and requires major revision. I have a terrible time understanding this paper. For example, when citing related works, please use brackets “(” & “)” in most cases to include them, i.e., use \\citep in Latex.\n\n2. It is difficult to understand the formulation of Eqn (2). \n- Since DNN has an extremely large number of parameters, close-to-zero initialization still result in different loss compared to zero-initialization. Why could we replace  $L_{D_c}(w_0)$ with  $L_{D_c}(0)$?\n- In this formulation, BS is defined for the whole model and every neuron share this value inside the model. So how can we distinguish neurons using this shared BS? Is the neuron-wise BS defined as the sum of the products of the parameters and their derivatives inside a specific neuron?\n\n3. Where is Figure 4 in Sec 5.4?\n\n4. If this paper regards it as a pruning method, it neglects an important work, ANP [3]. Since this is another pruning-based method, this paper should compare it.\n\n[1] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713, 2016.\n\n[2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim. Sanity Checks for Saliency Maps. In NeurIPS, 2018.\n\n[3] Dongxian Wu and Yisen Wang. Adversarial Neuron Pruning Purifies Backdoored Deep Models. In NeurIPS, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TlEaHkfqC_7", "reviewer": "Reviewer_1xEU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a neuron-based method to defense against backdoor attacks. They first identify less important neurons in the network and then purify them using a fine-tuning based method. The paper contributes to both the selection of less important neurons and in purifying the network. Compared to the common method of selecting less important neurons based on the activation magnitude, authors argued that activation ignores the strong connections that a neuron may have and may result in selecting important neurons as the least important neurons. The authors proposed Benign salience (BS) that ranks the neurons based on how they contribute to reducing the loss. The neurons that contributed the least are identified as bad neurons. The bad neurons are then purified using an improved fine-tuning strategy. The experiments show that the proposed method outperforms all the baseline methods.", "summary_of_the_review": "- I found the writing of the paper a bit problematic. It mentions WIPER as their method but then mainly presents BS as their method. I did not get whether WIPER refers to only improved fine-tuning strategy or it refers to BS plus improved fine-tuning strategy.\n\n- The paper mainly ignores various methods to identify the role of a neuron with respect to the prediction such as integrated gradient, deep lift, layer-wise relevance back propagation. Looking at just activation is indeed not the best way to identify important neurons but I would expect authors to compare their proposed method with the methods from the literature.\n\n- Figure 2 (left): The fine neurons that authors are getting could be the result of the BadNet method that it learns to cleanly separate the poison neurons. Testing gradient based methods on this model would clarify the benefit of BS in comparison to other methods.\n\n- (Section 4.2) I did not fully get the description of L1 and L2. L1 brings sparsity so it should turn the neurons will low weight/activations to zero quickly and L2 normalizes the spikes in the scores and it won't turn neurons to zero. Section 5.4 seems to be correctly discussing the effect of L1 and L2.", "strength_and_weaknesses": "The results are promising.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RPkMTCOSSM", "reviewer": "Reviewer_pBHg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a novel backdoor defense, which used first-order gradient to identify bad neurons using clean dataset. Also, a new Adaptive Regularization (AR) mechanism is used to fine-tune the model using detected bad neurons. They performed experiments on ten different backdoor attacks with three benchmark datasets. Their proposed method has performed better than other six state-of-the-art defense methods. ", "summary_of_the_review": "The authors used clean data to detect bad neurons and designed a new Adaptive Regularization (AR) mechanism for finetuning. However, they failed to mention and compare a very similar approach published in CCS’21, which also achieved good clean accuracy, ASR. Since these two algorithms are belongs to the same category. Lacking this comparison makes its contribution not clear.  ", "strength_and_weaknesses": "Strong Points:\n\n1. A novel a gradient-based detection method of bad neurons.\n2. The method focused on the last fully connected layers to remove backdoor. \n3. A new Adaptive Regularization (AR) mechanism is proposed to assist in purifying these identified bad neurons via fine-tuning.\n4. Extensive experiment results illustrated the good performances. \n\nWeak point:\n\nThere is a paper published at CCS 2021: “AI-Lancet: Locating Error-inducing Neurons to Optimize Neural Networks”. This paper also considered gradient during finding bad neurons. They also achieved good results with respect to accuracy and ASR. Authors have not mentioned this paper nor compared with. They should include this paper for comparison to prove the effectiveness of their method. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ARefdKZqPnW", "reviewer": "Reviewer_etm6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel backdoor defense method (called WIPER) that includes: i) a new metric, called benign salience, to identify backdoored neurons; and ii) a new adaptive regularization mechanism to assist in purifying the identified bad neurons. ", "summary_of_the_review": "See \"Clarity, Quality, Novelty And Reproducibility\" Section of this review. ", "strength_and_weaknesses": "Strengths: \n- empirical evaluation and effectiveness, i.e. the proposed method yields better quantitative results than the included competitor methods.\n- relevant application: the fact of developing/designing safer and more resilient neural models is always welcome. \n\nWeaknesses: \n- not sure if technical/methodological novelty of this work with respect to the state-of-the-art is enough.\n- not sure if authors are comparing (quantitatively and qualitatively) with all relevant prior work. \n- the organization, presentation and writing of the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject