Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness
Authors: Weilin Lin
Li Liu
Shaokui Wei
Jianze Li
Hui Xiong
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Neural Network; AI security; Backdoor Defense
Primary area: safety_in_machine_learning

Abstract: The security threat of backdoor attacks is a central concern for deep neural networks (DNNs). Recently, without poisoned data, unlearning models with clean data and then learning a pruning mask have contributed to backdoor defense. Additionally, vanilla fine-tuning with those clean data can help recover the lost clean accuracy. However, the behavior of clean unlearning is still under-explored, and vanilla fine-tuning unintentionally induces back the backdoor effect. In this work, we first investigate model unlearning from the perspective of weight changes and gradient norms, and find two interesting observations in the backdoored model: 1) the weight changes between poison and clean unlearning are positively correlated, making it possible for us to identify the backdoored-related neurons without using poisoned data; 2) the neurons of the backdoored model are more active (*i.e.*, larger gradient norm) than those in the clean model, suggesting the need to suppress the gradient norm during fine-tuning. Then, we propose an effective two-stage defense method. In the first stage, an efficient *Neuron Weight Change (NWC)-based Backdoor Reinitialization* is proposed based on observation 1). In the second stage, based on observation 2), we design an *Activeness-Aware Fine-Tuning* to replace the vanilla fine-tuning. Extensive experiments, involving eight backdoor attacks on three benchmark datasets, demonstrate the superior performance of our proposed method compared to recent state-of-the-art backdoor defense approaches. The code is available at https://github.com/linweiii/TSBD.git.

Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness
Weilin Lin"
Li Liu Shaokui Wei Jianze Li Hui Xiong'
The Hong Kong University of Science and Technology (Guangzhou)
The Chinese University of Hong Kong, Shenzhen Shenzhen International Center for Industrial and Applied Mathematics SShnzzhnn Research Institute of Big Data
Abstract
The security threat backdoor attacks is central concern for deep neural networks (DNNs). Recently, without poisoned data, unlearning models with clean data and then learning pruning mask have contributed to backdoor defense, Additionally, vanilla fine-tuning with those clean data can help recover the lost clean accuracy. However, the behavior of clean unlearning is still under-explored, and vanilla fine-tuning unintentionally induces back the backdoor effect. In this work, we first investigate model unlearning from the perspective of weight changes and gradient norms, and find two interesting observations in the backdoored model: 1) the weight changes between poison and clean unlearning are positively correlated, making it possible for us to identify the buckdoored-relater neurons without using poisoned data; 2) the neurons of the backdoored model are more active (i.e., larger gradient norm) than those in the clean model, suggesting the need to suppress the gradient norm during fine-tuning. Then, we propose an effective two-stage defense method. In the first stage, efficient Neuron Weight Change (NWC)-based Backdoor Reinitialization is proposed based on observation 1). In the second stage, based on observation 2), we design Activeness-Aware the vanilla fine-tuning. Extensive experiments, involving eight backdoor attacks on three benchmark datasets, demonstrate the superior performance of our proposed method compared to recent state-of-the-ae backdoor defense approaches. The code is available httpss//gitaub.oom/inwwi///////t
to replace
1 Introduction
Over the past few years, deep neural networks (DNNs) have achieved surprising success in several real- world applications, such as face recognition D.3. medical image processing 4 () and autonomous driving 77, etc. However, DNNs are susceptible to malicious attacks that can compromise their security and reliability. One typical example is the backdoor attack [8-II), where the adversary maliciously manipulates the training dataset or training process to produce a backdoored model, which performs normally clean data while predicting any sample with particular trigger pattern to a pre-defined target label. In this work, we focus on the post-training defense scenario where, given backdoored model and small set of clean training samples, one aims mitigate the backdoor effect while maintaining the performance on clean data, thereby obtaining a benign model.
Up to now, several important methods have been developed for backdoor defense [14 One promising approach is poison unlearning, which involves updating backdoored model by unlearning
"Corresponds to Li Liu (avrillliu ...uut-z...-tggg..cccm
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Shaft Chrroa ofhrema (186)
Bu3 Chonect Sarrs Woded
Ouann Cowe Nour
Gurng dhen orborinn
buig dees samiing
buing daas untaaring
Datig Gasa daraiiing
Observation Weight changes between poison and clean unlearning are positively correlated.
Contoocei Hears
Ole Mertorr Hears
Osin-moc ta't
Pestege Ohereoor
Reason Index
Aeuran Indes
Meuroo tride
Observation 2. Neurons backdoored model are active than those in clean model.
Figure I: Illustration of two observations. Figures for Observation 1 show distributions of neuron weight changes during clean unlearning and poison unlearning. Figures for Observation compare the average gradient norm for each neuron on the backdoored model and clean model, which are calculated with one-epoch clean unlearning. Being "more active" means a larger gradient norm. Experiments are conducted on PreAct- Resseel1 [12] using CIFAR-10 [33 for the clean model, along with additional attacks using 10% poisoning ratio for the backdoored model. The last convolutional layers are chosen for illustration.
from poisoned data. This technique has been utilized in various backdoor defenses such ABL [14.
D-BR [9]. Neural Cleanse (NC) (20]. and i-BAU [2I] etc. To avoid approximating poisoned data, another approach called clean unlearning was conducted by RNP (22]. This technique only uses clean data for unlearning and then prunes the backdoored model, which has been proven to be effective. Through relevant experiments, we find an interesting connection between poison unlearning and clean unlearning, as illustrated in Observation of Figure [ Specifically, by calculating the weight changes of each neuron during the unlearning processes we find that they exhibit a strong positive correlation, ie., the neurons exhibiting significant weight changes during clean unlearning also tend to play crucial roles in poison unlearning, indicating stronger association with backdoor-related activities. Moreover, we further investigate the backdoor activeness during learning processe i.e., comparing the average gradient norm for each neuron in both the backdoored and clean models. The results are shown in Observation 2 of Figure revealing that neurons in the backdoored model are always more active compared to those in the clean model.
the backdoored model
Inspired by the above two observations regarding the backdoored model, we propose Stage Backdoor Defense (TSBD), consisting of stage 1) Neuron Weight Change-based Backdoor Reinitial- ization and stage 2) Activeness-Awars Fine-tuning. on the backdoored model, followed by the neuron weight change calculation, where both the changes of each subweigh and neuron are recorded. Then, we conduct zero reinitialization to mitigate the backdoor effect by reinitializing the most-changed subweights among the top-n% most-changed neurons as O in the original backdoored model. In the second stage, we adopt activeness- fine-tuning with gradient-norm regulation recover clean accuracy and suppress the reactivation of the backdoor effect. Extensive experiments demonstrate the superior defense performance of the proposed method compared of-t-oof-thearr (SOTA) backdoor defense methods.
the first stage, we first conduct clean unlearning
To summarize, our main contributions are three-fold. (1) Novel Insight: We are the first to uncover the strong positive correlation between neuron weight changes in clean unlearning and poison unlearning. We also reveal the high backdoor activeness in the backdoored model during the learning process. (2) Effective Defense Method: We further develop an effective two-stage defense method based
SSBA 9 and WaNet 24 are used for illustration. 3 Unleaminng as the opposite process of model learning, can also be considered as a kind of learning process.
Four attacked models on BadNets g Input-awarc
'A subweight represents learnable weight in neuron weight matrix.

---

on unlearning weight changes and backdoor activeness, considering both backdoor mitigation and clean-ace -ucarracy recovery, respectively. (3) SOTA Performance: Experimental results and analysis show that our proposed method achieves SOTA performance in backdoor defense.
Neuron Weight Change-based Backdoor Reinitialization
2) Activeness-aware Fine-tuning
Clean Unlearning
Neuron Weight Change Calculation
Fine-funing Gradient-Nom Regulation
Ex(0) C. 1V
Clean Datanet
Unieamedd O.:
Neuron Weight Changes
Subweight Changes
Zero Reinttillization
Backdoorad Modal
Weight Mask
Reinttialize Model
Clean Model
Figure 2: Overview of the proposed Two-Stage Backdoor Defense framework.
2
Related Work
2.1
Backdoor Attack
In the literature, various backdoor attacks on DNNs have been proposed, which can be categorized into data poisoning attacks and training-controllabe attacks. BadNets [8] is one of the earliest data poisoning attacks in this field. In this attack, small proportion of the original data is selected and patched with a pre-defined pattern, known as a trigger. The labels of these patched data points are then modified to target label. The mixed dataset, containing both clean and poisoned data, is used to train the DNNs, resulting in the implantation of the backdoor. Under a similar procedure, Blended [25] was proposed as stronger attuck by blending entire pre-defined image into the original clean data with controllable transparency. Recently, more advanced and stealthy attacks have been proposed enhance the trigger, such SIG [26). label consistent attacks (27 28], SSBA (9), etc. Another category is maining-controllann
29 31, where the attackers design triggers with permission control the training process. Two significant examples
attacks 23 24
WaNet [24]
and Input-aware [23), which generate unique triggers for different input data by incorporating an injection function into the model training process. This approach makes these attacks more difficult to detect compared to previous attacks with fixed triggers.
2.2 Backdoor Defense
According to the different stages of model training, backdoor defense methods can be classified into two types: training-stie sttgg defenses and post-training defenses.
Training-stage Defenses. In training-stage defenses, defenders have access to a mixed training dataset containing both clean data and poisoned data with triggers. ABL [14] discovers that the loss-dropping speed of poisoned data during the early stages of model training is faster, and thus isolates them for poison unlearning. DBD [32] splits the training process into three steps to separate the training of feature extraction from that of the subsequent classifier to evade the learning of trigger- label correlation. Similarly, D-ST/D-BR [19] observes that the transformations of poisoned-dan feature representations are more sensitive than clean ones, and thus proposes to modularize the training process.
Post-training Defenses. post-training defenses R3-34 36 defenders aim to erase the backdoor effect in the learned DNNs using a small portion of clean data. FP 37 is one of the earliest defense methods, which observes that poisoned data and clean data activate different neurons a backdoored DNN, and thus keeps pruning the less-activated neurons in response to clean data until significant

---

drop in accuracy occurs. After that, vanilla fine-tuning is employed to recover the lost clean accuracy. Using the pruning strategy [3) ANP [4]] observes that the backdoor-related neurons exhibit higher sensitivity to adversarial perturbations compared to others, and thus trains a pruning mask using minimax optimization. Continuing along this line, AWM 42 and RNP [2] use a similar mask training process with main modifications in neuron perturbations to data perturbations and clean unlearning, respectively. Different strategies are also proposed for defense. For example, NC [20] proposes to recover the trigger before the subsequent backdoor removal. NAD 43 for the first time, adopts model distillation to guide the learning of a benign student model. Additionally, employing unlearning techniques, SAU [44 treats backdoor triggers form of adversarial perturbation, and generates poisoned data through optimization on clean data, which are then used in poison unlearning.
Unlearning for Backdoor Defense. Model unlearning can be considered as an opposite process against learning, aiming to remove the impact of a training subset from a trained model [45]. In
the field of backdoor defense, unlearning the possible poisoned data (i.e., poison unlearning) is an
effective way to remove the learned backdoor. NC [20 and BTI-DBF 46 try to generate the possible poisoned data with either trigger inversion poison-data generator; ABL 14 and D-BR [19 focus on filtering out the poisoned data from the training dataset according to their attributes during training; i-BAU 21 and SAU 477 assume the adversarial perturbation as type of trigger and generate poisoned data with adversarial example. To avoid inducing bias, recent work tries to directly unlearn the available clean data (i.e., clean unlearning) for defense. RNP [22] finds that a leean-unleareed model can help expose the backdoor neurons for the subsequent pruning- gruning-mask learning.
However, there exist some limitations among those techniques, e.g., clean unlearning is still under- explored, and the vanilla fine-tuning unintentionally increases the attack success rate. In this paper, we propose comprehensive two-stage defense method breaking through the two limitations.
3 Methods
3.1 Problem Formulation
Threat Model. We assume that the attacker has full access to the training data. Their goal is to poison a portion of the dataset by injecting triggers into the data so that the trained model misclassifies the poisoned data to the target class while still performing normally clean data. The poisoning ratio (e.g., 10%) is used to depict the proportion of poisoned data within the entire dataset. We denote the parameters of the backdoored model as Ond = (0!)) hisgg. satisfying ( R x where K (Kihs numbers, respectively. Specifically, for the 1th {1...., I.) layer, there are K()) neurons in total and (() subweights for each neuron.
hiiis and I (1") heie represent the neuron numbers and learnable subweight
Defense Setting. The defender's goal is to remove the backdoor effect, which causes poisoned data to be misclassified to the target class, from the backdoored model while minimizing the impact on
the prediction accuracy for clean data. Following the previous defense setting [37 41, we assume
that the defender knows nothing about the poisoned data and possesses only 5% of the total dataset as clean data, denoted as De..
3.2 Neuron Weight Change & Suggestions Given by the Two Observations
In this subsection, we provide more details on the unlearning formulation and offer suggestions based on the two observed observations.
Model Unlearning. Model Unlearning can be defined as the reverse process of model training [22] which involves maximizing the loss value a given dataset. Given DNN model parameterized as 0 and a dataset DD for unlearning, the maximization problem can be formulated as:
max E ED [C(f(>; (x;()x))))))
where (x. y) E D represents the images and their corresponding labels, and L denotes the loss function used in this task, e.g. cross-entropy loss.
Intuitively, by maximizing the loss expectation, the unlearned model, parameterized as Out is prone to fail at the task specified in D. In this paper, we term the process clean unlearning when all

---

Comparison of Activtionn Clean Model
Comparison of Activations Racideored Model
Comparison of vatioon Changes
var tis zuris Polsen eariing Clea Add tis juing unierring
Actovations Change
Adtivatinn
Adtivaatoms
0.89
6,75
0.3-
Clear Ant
-6-
to son Autival ora
-10
WWC of Clean Unlearning
NVC of Clean Unteerning
NWC of Clean Unleamming
(a) Original clean model
(b) Original backdoored model
(c) Backdnored-moode unlearning
Figure 3: Illustration of clean and poison activations of each neuron. (a) and (b) represent the activations the original clean and backdoored model, respectively. (c) shows the activation changes during the clean and poison unlearning on backdoored model. Activations are captured from the last convolutional layer with an additional Relu activation function on PreAct-ResNeil8 [12]
the data in D are clean, denoted as De- On the other hand, poison unlearning refers to the scenario where all the data in D are poisoned with trigger. By default, both clean and poison unlearning are terminated when the model performs poorly on the corresponding tasks, such as achieving only 10% clean accuracy or attack success rate.
To comprehensively quantify the weight changes of neuron during the entire unlearning process, we define the Neuron Weight Change (NWC), where the L1 norm is calculated on every neuron's 's weight differences. The NWC for the hthh E (1...., K(D) neuron in layer / (I....,L) L can be formulated as:
Neuron Weight Change.
NWC
Σ 1=0
(hki
(())
1,
where E/ I1-111 is calculate the L1 norm for the differences on a neuron with totally (() subweights, (t)ss and @(iiké denote the zth E {1,..., ) subweights of w neuron after and before the entire unlearning process, respectively. changes occurring in neuron At during the unlearning process. Similarly, in Equation
larger NWC* indicates more significant
( the term
| ((hki - 1l2 represents the changes in the 5t subweight of neuron i in layer 1, i.e, defined as Subweight Change.
Suggestions Given by the Two Observations. As demonstrated in Section 1 we have two interesting observations regarding the backdoored model. Observation 1 shows that the neurons exhibiting significant weight changes during clean unlearning also tend to play crucial roles in poison unlearning. suggests that we can employ clean unlcarning to identify and climinate backdoor-related neurons using NWC, at the expense of reducing clean accuracy. On the other hand, Observation 2 reveals that neurons in the backdoored model are always more active compared to those in the clean model. suggests that should suppress the gradient norm during the learning process if we want to recover it to a clean model. These two suggestions act as the main supports to our proposed TSBD.
3.3 Further Investigations & Insights
Here, offer insights from the perspective of neuron activations, trying to answer two important questions: [Q1] What causes the clean unlcarning NWCs to exhibit a positive correlation with those in poison unlearning, and [Q2] What motivates the neurons more active in the backdoored model.
Neuron Activations & Activation Rise. The neuron activation is determined by computing the average value of all inputs to the specific neuron, e.g., h (f)) se o (1)k h (1-1) for simplicity, where a(.) is the activation function. In line with the terminology used in FP (37]. clean activation denotes the scenario where all the input samples are clean while poison activation refers to the presence of poisoned inputs. To better observe the changes in activation during unlearning, we calculate the activation rise from the original model the unlearned model, i.e., (1) (0) value indicates an increase in activation, while negative value signifies decrease.
ad 1i positive
5

---

Relationship between NWC and Activation Change. Considering that backdoored model has learned two tasks from the clean and poisoned data [1. the main influence of NWC on a neuron can be roughly attributed to its activation change on both clean and poisoned inputs. For neuron k in (1) layer we can formulate it as NWC x lake (1) + | where A (1) and Ah, represent the activation rise on clean and poisoned inputs, respectively.
Figure 3 iillustrates the clean and poison activations in (a) the original clean model, (b) the original backdoored model, and (c) the backdoored-modee unlearning. We now try to answer the above two questions from these observations. [A1] We can observe that poison activations are the main factors affected during both clean unlearning (increase) and poison unlearning (decrease), while clean
activations only slightly influenced (see Figure 3 (c)), i.e., NWC, Ahe (1) > + | (1)& It-
Also, the growing NWC during clean unlearning can indicate larger poison and clean activations
(where hy > ( to some extent (see Figure (b)). Thus, we deduce that the co-function of clean and poison activations dominates the performance on both tasks, while the higher values of poison activation in the backdoored model make it an easier target for modification. In this case, the neurons with higher poison activations tend to decrease their values during poison unlearning, thereby reducing the attack success rate. Conversely, during clean unlearning, these neurons increase poison activations, which suppresses the function of clean activations and reduces clean accuracy. [A2] Similarly, the significantly lower values of mixed clean and poison activations (maximum: 0,5676) on the clean model (see Figure (a)) indicate that it is less active compared to the backdoored model (maximum: 1.7053), where a similar pattern can also be seen on the bottom left of Figure (b).
3.4 Two-Stage Backdoor Defense Framework
Based on the above observations, we now propose defense framework incorporating Neuron Weight Change-hased Backdoor Reinitialization (including Clean Unlearning, Neuron Weight Change Calculation and Zero Reinitialization), and Activeness-aware Fine-tuming. The detailed defense process is illustrated in Figure and Algorithm (found in Appendix(A).
Stage 1) Neuron Weight Change-based Backdoor Reinitialization. We aim to mitigate the backdoor effect with acceptable clean-accuracy sacrificed in this stage. /a. Clean Unlearning.1 To identify the backdoor-related neurons, we first conduct a full clean unlearning using the available clean data De on the backdoored model. [b. Neuron Weight Change Calculation.] Then, we record the subweight changes and calculate the NWC for each neuron as described in Section 3.2 The resulting sorted order of neurons reflects the backdoor strength. (c. Zero Reinitializatiin, J After that, we can now eliminate the backdoor effect through zero reinitialization. Based on the NWC neuron order, we identify the top-n% neurons as strongly backdoor-related. As suggested in Section 33. NiC neurons may also contribute to clean accuracy to some extent. Therefore, we further choose to reinitialize the subweights of the most-changing m% among the selected neurons to zero in the backdoored model, while leaving the others unchanged. The reinitialized model parameter is denoted as 0.
Stage 2) Activeness-Aware Fine-tuning. To further repair the reinitialized subweights and avoid recovering the backdoor effect again, we conduct activeness aware fine-tuning on the reinitialized model (0) using the clean dataset, D This involves incorporating gradient-norm regulation into the original loss function, such as the cross-entropy loss Lee, to penalize high gradient values. This regulation serves to suppress neuron activity during fine-tuning, The final loss function is:
()) (0) + X | (0)(22
(3)
where || Sce (0) 12 represents the L2 norm of gradients, and À is the penalty coefficient controlling its impact. Hence, the objective of fine-tuning is minimize the loss function (+(()) using the available clean data De:
(e. (R (f ye)]-
During practical optimization for computational efficiency, we adopt the approximation scheme in 48), which can be formulated as:
VoCoe V VoCoo (0)122
v voL,(00) ~ (1 a) + vo
(5)
6

---

Table I: Comparison with the SOTA defenses CIFAR-10 dataset with PreAct-ResNett (%).
NAD
Attacks
ASR
DEK
ACC1
DER
A
13RK
ASR
D3R
ACC
ASR
DER
BadNeess g
91:32
15:03
89.96
1.45
96.30
91.31
57:13
08.35
89.57
2.14
95.72
55:05
1.27
95.75
Blended
2X
03.47
99.02
92.78
96.
51.55
93.17
90.30
50.18
92:
97.69
50.47
93:47
99.92
50.00
Input aware
22
90.67
98.26
93.12
1.72
98.27
91.74
0.04
99.11
93.18
1.68
98.
92.61
0.76
98.25
1F
99.11
99.28
92.57
78.44
60.01
92.90
98.97
50.01
92.37
47.83
75.31
91.62
1.41
98.15
SKG
PD
84.48
98.27
90.50
2.37
07.95
89.
26.20
86.03
90.00
10.66
93.81
84.48
98.
50,00
SSRA
9
92.88
97.86
92. 14
75.70
61
92.54
83.50
57.01
91.0 91
77.40
59.74
90.99
0.58
97.69
Trojan
50
93.42
100.00
92.42
96.51
92.46
71.37
03.94
91
3.73
97.36
91.76
8.22
95.05
WaNNr
17:30
86.32
91.44
Average
91.34
97.29
V2.13
34:75
80.38
91.84
54.67
71.19
51.76
90.72
Backshor
CLP [55)
1-RAIT 211
RNP 123
TSRD (Curr)
Attacks
ACCT
ASR
DER
ACCC
253R
DEK
ACCCT
ASSC
DER
ACC1
ASK
DEK
XCC
ASK
DEK
BaulNets 8
90.51
5.91
64.337
90.06
77.30
58.14
89.15
1.21
05.83
69.31
24.97
$4.25
90..22
1.31
96.55
Blended 25
93.00
54.90
57.28
91..32
99.74
49.01
57.00
50.53
71.46
58.76
79.74
57.25
91.61
2.61
97.73
Iepui-wwar
H3
91:04
1.30
95.47
90.30 30
2.17
97.55
89.17
27.08
84.84
90.52
1.84
98.13
93.06
1.94
98.16
1F kg
92.81
54.99
71.96
92.84
99.18
49
84.36
44.96
72:75
58.43
7.02
93.25
91.22
2.64
92.33
SIG 26
K3.36
36.43
80.36
83.30
98.91
49.05 19.66
85.67
3,68
97.29
B1.48
98.27
50.00
90:41
1.27
98,55
SSBA
60.16
91
N7.67
19.64)
17.89
91.57
92.98
96.
91.32
93.76
76.42
$9.49
91.35
90.43
91.22
Averge
91002
36
80.00
89.32
75.01
38.68
87.85
17.44
11
88 83.99
29.28
$2.85
91.70
2.18
97.09
Here, r used for appropriating the Hessian multiplication operation, and a = is the balance coef- ficient. Due to the space limit, the detailed derivation and algorithm are provided in Appendix B After the activeness-awan fine-tuning stage, we can obtain a repaired clean model, which demonstrates outstanding performance in the experiments.
4 Experiments
4.1 Experimental Setup
Attack Setup. We consider 8 SOTA backdoor attacks in the main experiment. There are BadNets
X
Blended (25), Input-aware (3). LF 49,, SIG (26), SSBA (9. Trojan (50) and WaNet 24 For
fair comparison, we follow the default attack configuration as in BackdoorBench [51] including
the trigger pattern, trigger size, the target label (i.e., the 0th label), etc. We choose 10% poisoning
ratio as the default setting. To fully evaluate the effectiveness of our proposed framework, all the attacks are implemented on three benchmark datasets, i.e., CIFAR-10 [3), Tiny ImageNet (52), and GTSRB (53) over two popular DNNs, i.e., PreAct-ResNetll 12 and VGG19-BN [54]. In particular, SIG is only applied to CIFAR- 10 since it cannot reach a 10% poisoning ratio on Tiny ImageNet and GTSRB. Besides, we also evaluate the defense methods under 5% and 1 1 poisoning ratios to verify the robustness of our proposed framework. Due to the space limit, we only exhibit parts of the main results in this section. More implementation details can be found in Appendix C
Defense Setup. We compare the proposed framework with SOTA backdoor defense methods: Fine-tuning (FT), Fine-pruning (FP) (37, NAD 43). NC [20). ANP [1], CLP [38], i-BAU 22I) and RNP [22]. We use the recommended configurations in BackdoorBench (5]] Since the defense settings for training-stage defenses are different [4 322,
we only compare the post-training defenses, where 5% clean data can be accessed following the previous settings 41. For TSBD, we set the learning rates for clean unlearning to 10 - and fine-tuning to 10 The default neuron ratio n and weight ratio m are set to 0.15 and 0.7, respectively. We follow the suggested settings of hyper-parameters 0.05 and 0.7 for fine-tuning [48].
Evaluation Metrics. We use three metrics to evaluate the performance of each defense method: Accuracy on clean data (ACC), Attack Success Rate (ASR), and Defense Effectiveness Rating (DER) [55] Specifically, ACC measures the proportion of clean data correctly predicted; ASR mea- sures the proportion of poisoned data misclassified to the target label; DER E [0, 1] evaluates the cost of ACC for reducing ASR, which is defined as: DER- = [mnax(,, AASR) max(0, (0, AACC) + 1]/2, where AASR and ACCC are the drop in ASR and ACC after applying defense on the backdoored model, respectively. Larger ACC, DER, and smaller ASR are desired for a successful defense. Note that in the following result tables, 5_" indicates that the value is inapplicable. The boldface values indicate the best performance and the underline values denote the second- -bett result.
4.2 Main Results
We validate the effectiveness of our proposed framework on SOTA backdoor attacks and compare with 8 defenses. In this section, we present the main results on CIFAR- 10 and Tiny ImageNet with
7

---

Table 2: Comparison with the SOTA defenses Tiny ImageNet dataset with PreAct-Res (%)
Backdloe
No
FP 17
NAD
Attacks
A
DEK
DE3
A
13ER
ASR
DER
ACC
ASR.
DER
BadNeis g
56.25
100000
55.18
0
99.43
51.73
99.99
47.76
46.37
0.27
94.15
15.26
0.10
55.96
Blended 2X
$6.03
99:71
55.04
97.73
50.49
51.89
95.94
49.81
46.80
95.00
47.79
52.55
93.21
$1.51
Input aware
23
57.45
98.85
57.45
1.65
98.00
55.28
02.91
56.88
47.91
1.36
93.75
56.20
0.09
98.76
1F
[29
55.97
95.57
54.90
94.87
51.36
51.44
95. 25.25
29.40
45.45
50.40
68.
52.99
85.56
55,02
SSBA 0
55.22
97.71
56.80
91.57
52.86
50.47
88.87
52.04
45.30
57.32
65.25
52.47
53.47
70.75
Trojan
50
55.89
99
3547
0.50
99.50
50.33
8.50
92:74
48.48
0.83
95.87
52.69
0.15
98.31
WaNet
56.78
56.74
53.84
3.94
96.30
0.43
94.63
52.33
0.23
97.80
Average
65.11
50.16
Backdrox
CTP
21
RNP 23
(Ohrrs)
Attecks
A
ASR
D3R
ACC
ASR
DER
ACC1
ASR
DAR
ACC1
ASR
DER 1
ACC
ASR
DFR
KadlNess g]
50:55
7.74
93.29
55.34
100100
49:36
51.43
97.36
25.05
21.71
[000
82.84
53.72
0.34
08.35
Dlended
[3]
$4.99
84.61
57.00
55.70
99.68
49.85
53.03
91,90
$2.40
34.60
0.11
59.08
53.62
2.30
97.50
Inpur lwpu
23
53.17
0.37
97:20
57.75
99.5%
50.00
52.45
72.98
50.45
15.57
0.00
78.49
55.38
0.10
99.34
1F k0
54.66
95.
50.94
55.61
08.49
49 -9.86
51.13
85.32
54.21
49
0.00
05.89
52.47
1.90
96.59
SSRA
5288
91.44
51.94
55,17
97.65
$0.01
49,86
81.90
55.32
57.64
0.00
50110
52.93
1.38
97.02
Trojan
50
50.37
1.40
96.53
55.86
6.39
95.78
52.65
98.49
49.12
66.27
0.00
95.18
53.66
0.31
98.72
WaNer
0.75
56.21
50.21
50.00
98.50
Average
52.92
77.83
96.04
36.51
52.05
86.17
54,42
a 10% poisoning ratio on PreAct-ResNer18 for illustration, which is shown in Table I and Table 2
More results on GTSRB and VGG19-BN can be found in Appendix D and respectively.
Results on CIFAR-10. Table 1 shows the results on CIFAR-10. Results show that our TSBD outperforms all the other SOTA defenses on the average of ASR (2. 18%) and DER (97.09% ) as well as a promising ACC (91.70%) higher than the original "No Defense" models (91.34%), which indicates its effectiveness in removing the backdoor effect with the least cost. Though most defenses fail in strong attacks Blended, LF, or SSBA, e.g., FT, FP, NAD, ANP, CLP, i-BAU, and RNP, our proposed TSBD success with the best ASR and DER on Blended and second best ASR and DER on LF and SSBA. FT and NAD perform similarly on each attack with a promising ACC, but they also fail on WaNet with a high ASR except for the mentioned strong attacks. FP and ANP perform well in ACC among all the defenses, while the defense performances on ASR and DER are unstable, which may be due to the unsuccessful backdoor-related neuron locating. CLP fails on most of the attacks with high ASR and low DER, which may be due to the structure constraint of computing channel Lipschitz only on the convolutional-bater normalization layer combination. i-BAU performs the second best on average ASR and DER, with failures three attacks. TSBD outperforms RNP on almost all performances, which indicates that using clean unlcarning with NWC is more effective than the unlearn-recove recovery process in RNP.
Results on Tiny ImageNet. Table presents the results on Tiny Image with PreAct-ResNet11. We observe that most defenses also fail on Blended, SSBA, and LF with high ASR. Similar performances of other attacks are shown on most of the defenses compared to CIFAR-IO, where FT and FP also perform well in ACC and CLP fails most attacks. Although i-BAU can successfully defend against most of the attacks in CIF it fails on all attacks here, indicating that its adversarial training fails with large classification categories. For RNP, though it performs well in ASR on almost all attacks, the ACC is sacrificed too much to be unacceptable, indicating an unbalanced defense performance. In comparison, our TSBD achieve SOTA on the average of DER, and perform second best on the average of ASR, which validates its superior defense performance.
4.3 Ablation Studies
Effectiveness of NWC order for Backdoor Strength. To verify the effectiveness of employing clean-unlearning NWC order in gaug- ing the backdoor strength, we borrow the Trigger-activated Change (TAC) 38 order as the ground truth and compare the neuron cov-
Coveog Ranoo on WCC 1h1
erage ratio on TAC under different proportions, i.e., measuring the overlap of the selected neurons on both metrics. Specifically, TAC measures the change in neuron activation before and after the input image attached with a trigger, where the larger value indicates a stronger backdoor effect. We select the following SOTA metrics for comparison: 1) the average neuron activations in FP [37]: 2) the channel Lipschitz in CLP [38] 2) the perturb-
CLP
AMP
RMP
Neuron Ratio (%)
learned mask in ANP [11: 3) the unlearn-recovery learned mask in RNP (22). The result is illustrated in Figure E where the x-axis represents the (reinitializinggpruuml
Figure 4: Comparison of neu- ron coverage ratio on TAC un- der different neuron ratios. neuron ratio and the y-axis represents the neuron coverage ratio on TAC. The higher values on the y-axis indicate better matching of the current metric and the TAC metric,
8

---

Bashlets
Plencer
Dacteet
Blended
ACC
ACC
ACC
MLC
Ask
ARR
ASR
Figure 5: Performance with different neuron ratios (two subfigures on the left) and weight ratios (two subfigures on the right) under the attacks of BadNets and Blended.
i.e. more backdoor-related neurons are chosen. We can observe that NWC is always the best under different neuron ratios compared to others. It validates the effectiveness of using NWC order as the metric for neuron reinitialization/rrmmm FP to show its superiority in assisting other defenses. The results can be found in Appendix F
Moreover, we also test the performance of using NWC in
Effectiveness of Zero Reinitialization on Sub- weight. We compare different combinations of reinitializing neurons and weights to validate the effectiveness zero reinitialization on the selected subweights. Specifically, three versions are designed for comparison: (1) Vı; all weights on the selected top-NWC neurons
Table 3: Comparison of different reinitialization
schemes on CIFAR-10 with
(%).
Backdour
V. (Ours)
Attacks
ACC
ASR
ASR
XCC
ASR
BadNett B
69.53
0.00
68,20
0.00
80.34
0.00
Blended
76.13
0.00
64.20
82.72
0.00
reinitial- ized to zero. (2) 22: m% (70% is used as de-
LF
49
76.41
0.00
78.64
0.02
82.42
0.00
SSBA
9
74.55
0.00
75.43
0.02
83.10
0.00
fault) of the top weights on each selected top-NWC neuron reinitialized to zero. (3) V3 (Ours):
our final version, where m% of the top weights among all selected top-NWC neurons are reinitialized to zero. As the ACC and ASR drop monotonously with more neurons selected to be reinitialized, we record the results when ASR first drops to almost zero (i.e., lower equal than 0.05%). Table 3 shows the performances selected neurons may help alleviate the hurt of clean accuracy. The failure of V2 may be blamed on the different backdoor strength of each neuron, i.e., some strong backdoor-related neurons fail to be removed thoroughly and thus more neurons are reinitialized to reach zero ASR.
four attacks. It validates that reinitializing the top weights among the
Effectiveness of Gradient-nore norm Regulation in Fine-tuning. To validate the essential role of gradient-norm regulation in the fine-tuning stage, we compare the performance on "no fine- tuning" (No-FT), "vanilla fine-tune one-tuning" (Vanilla- FT), and our "activeness- aware fine-tuning" (Aa- FT). For fair comparison, we follow the default settings and choose to fine-tune the reinitialized model after the second stage. The results are shown in Table 4 It shows that fine-tuning effective
Table 4: Comparison of different fine-tuning
schemes on CIFAR- 10 with PreAct-Ress ResNet18 (%).
Backdour
No.FT
Pa
(Ours)
Attacks
ACC1
ASR
ACC
ASR L
ACC1
ASR
BaiNets
8034
0.00
50.66
1.69
90.72
1.37
Blended
0.00
2.1
I.F
4
82.42
0.00
91.51
5.57
91.20
2.64
SSBA 9
83.10
0.00
91.12
7.6
91.57
1.66
in improving clean accuracy, which is sacrificed to erase the backdoored effect in reinitialization, though it is prone to bring back some extent of ASR. Furthermore, compared to vanilla fine-tuning. our implemented to the same level.
sitveenssswwwarr fine-tuning can suppress the rise of ASR as well as improve ACC
4.4 Further Analysis
Performance on Different Neuron Ratio and Weight Ratio. To test the sensitivity of our method towards hyper-parameter tuning, we record the performance of TSBD under a wide range of neuron ratio (n%) and weight ratio (m% %n) tuning. The experiments are conducted on CIFAR-10 10 with a 10% poisoning ratio on PreAct-ResNetl1.. The neuron ratios are tuned from 1% to 90%, and weight ratios are tuned from 40% to 80%. The results of BadNets and Blended attacks are depicted in Figure It shows that TSBD is insensitive to both neuron ratio and weight ratio, where the ASR kept at very low level, while ACC is maintained at a top level under a wide range of tuning. The high consistency in the performances may come from the promising recovery ability of
Table 5: Performance of hyper-
parameter tuning for stage 2.
ACC1
ASRL
Our Settings
0.03
0.70
50.72
1.31
0.01
0.70
90.68
1,30
Tuning
0.02
0.70
91.03
1.50
0,10
0.70
90.90
1.30
0.20
0.70
89.64
1.04
0.05
0.10
91.32
1.40
Taning
0.05
0.30
91.24
1.59
0.05
0.50
91.00
1.63
0.05
0.90
90.68
1.26
the fine-tuning stage. More results on other attacks are shown in the Appendix G

---

Performance on Different r and for Activeness-Aware Fine-tuning. To test the apper-prammters sensitivity for stage 2, i.e., r" and a in Activeness-Awar Fine-tuning, we follow the tuning range in [48] under our experimental settings. The results are shown in Teble which are obtained from BadNets-attacked PreAct-ResNetll We observe that the performance is insensitive (Changes <2% in ACC and <1% % in ASR) across different hyper-paramete parameter settings, maintaining a high level of performance.
Performance on Different Poisoning Ratio. We further investigate the performance of TSBD on different poison- ing ratios, e.g., 10%, 5%, and 1%. Note that larger poisoning ratio represents a stronger attack mode. We test the performance with six attacks on these three ratios. Figure ( shows the performances of ACC and ASR. We can observe that TSBD successfully defends all the attacks on 10% and 5% with a low ASR and a high ACC while performing less effectively that the unlearning weight changes of backdoor neurons
30%
Polkaiing
ACC: 5th Polioniig Rots
AGR: Pa Pon Pate
A0C Falsuniinng Rats
Palso sing Rate
100
80
Accuracy
60
40
I 1% A possible reason is
20
are less obvious in the weak attack mode compared to the strong attack mode with 10% or 5% poisoning ratios.
Baotaas
Blended
Injuf-wware
566
55B4
More Experiments and Analysis. Due to the space limit, we postpone the detailed discussion of the clean data ra- tio and the fine-tuning learning rate to Appendix H and Appendix ! respectively. We also evaluate the defense performance on the clean model in Appendix[aand on the VIT in Appendix K Further, we provide the computational overhead in terms runtime in Appendix L
Figure 6: ACC and ASR on different poisoning ratios.
Conclusion
In this work, we propose an effective two-stage backdoor defense method, TSBD, eliminate the backdoor effect in DNNs. Our research reveals two important observations regarding the backdoored models to support poison and clean unlearning in backdoored models. This finding enables us to identify and eliminate backdoor-related neurons through clean unlearning and zero reinitialization. Second, neurons in backdoored models are more active compared to those in clean models, which suggests regulating the gradient norm during fine-tuning. Furthermore, we also provide insights into these two observations from the perspective of neuron activations, which may be a valuable contribution to the field of backdoor defense. Extensive experiments demonstrate the superiority of our method over recent defenses. One current challenge as well as promising future work involves defending against backdoor attacks without any accessible clean data. The data generation techniques and data-free techniques may be the potential solutions.
method. First, there is a positive correlation between weight changes during
6 Acknowledgements
This work was supported in part by the Guangzhou Municipal Science and Technology Project: Basic and Applied Basic research projects (No. 2024A04J4232), National Natural Science Foundation of Joint Funding Program (Grant No.2023A03J00033,
China (No. 62101351). Srangzhou-HKUSTI Education Bureau of Guangzhou Municipality, and Hetao Shenzhen- Hong Kong Science and Tech- nology Innovation Cooperation Zone Project (No.17/SSS--KCCYB-22222)
References
[1] Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In CVPR, 2014.
[2] Divyarajsinh N Parmar and Brijesh B Mehta. Face recognition methods applications. arXiv preprint arXiv: 1403.0485, 2014.
[3] Ratnawati Ibrahim and Zalhan Mohd Zin. Study of automated face recognition system for office door access control application. In ICCSN, 2011.
10

---

[4] Yixiong Chen, Chunhui Zhang, Li Liu, Cheng Feng, Changfeng Dong, Yongfang Luo, and Xiang Wan. Uscl: pretraining deep ultrasound image diagnosis model through video contrastive representation learning. In MICCAI, 2021.
[5] Yixiong Chen, Li Liu, Jingxian Li, Hua Jiang, Chris Ding, and Zongwei Zhou. Metalr: Meta-tuning of learning rates for transfer learning in medical imaging. MICCAI, 2023.
[6] Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. survey autonomous driving: Common practices and emerging tochnologies. IEEE access, 2020.
[7] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[8] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 2019.
[9] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran Hc, and Siwei Lyu. Invisible backdoor attack with sapple-specific triggers. In ICCV, 2021.
[10] Baoyuan Wu, Li Liu, Zihao Zhu, Qingshan Liu, Zhaofeng He, and Siwei Lyu. Adversarial machine learning: A systematic survey of backdoor attack, weight attack and adversarial example. arXiv preprint 27X1v:2302.09455, 2023.
[11] Runkai Zheng. Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. In NeurIPS, 2022.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
[13] Alex Krizhevsky, Geoffrey Hinton, Learning multiple layers features from tiny images. 2009.
[14] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurIPS, 2021.
[15] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv:1811.0377 2018.
[16] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In NewrlPS, 2018.
[17] Xiangyu Qi. Tinghao Xie, Jiachen Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal, Towards proactive (ML) approach for detecting backdoor poison samples. In USENIX Security, 2023.
[18] Min Liu, Alberto Sangiovanni- Vincentelli, and Xiangyu Yue. Beating backdoor attack its own game, In ICCV, 2023.
[19] Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In NewrlPS, 2022.
[20] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao.
Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In SP, 2019.
[21] Yi Zeng. Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations, 2021.
[22] Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang. Reconstruc- tive neuron pruning for backdoor defense. arXiv preprint arXiv:2305. 14876, 2023.
[23] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurIPS, 2020.
[24] Anh Nguyen and Anh Tran, Wanet-imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10336 10369, 2021.
[25] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 1721::111..05526, 2017.
[26] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set comuptiio without label poisoning. In ICIP, 2019.

---

[27] Ali Shafahi. W Ronny Huang, Mahyar Najibi, Octavian Suciu. Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NewrIPS, 2018.
[28] Shihao Zhao, Xingjun Ma. Xiang Zheng. James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clban-label backdoor attacks on video recognition models. In CVPR, 2020.
[29] Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnalle, imperceptible and robust backdoor attacks. In ICCV, 2021.
[30] Eugene Bagdasaryan and Vitaly Shmatikov, Blind backdoors in deep learning models. In USENIX Security, 2021.
[31] Khoa Doan, Yingjic Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. In NeurlPS, 2021.
[32] Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR, 2022.
[33] Junfeng Guo, Ang Li, and Cong Liu. Aeva: Black-box backdoor detection using adversarial extreme value analysis. arXiv preprint arXiv:21 14880, 2021.
[34] Wei Jiang. Xiangyu Wen, Jinyu Zhan, Xupeng Wang. Ziwei Song, and Chen Bian. Critical path-based backdoor detection for deep neural networks. TNNLS, 2022.
[35] Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In CVPR, 2020.
[36] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li, Detecting ai trojans using meta neural analysis. In SP, 2021.
[37] Kang Liu, Brendan Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In RAID, 2018.
[38] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. ECCV, 2022.
[39] Tianlong Chen. Zhenyu Zhang, Yihua Zhang, Shiyu Chang, Sijia Liu, and Zhangyang Wang. Quarantine: Spansity can uncover the trojan attack trigger for free. In CVPR, 2022.
[40] Jiyang Guan, Zhuozhuo Tu, Ran He, and Dacheng Tao. Few-shot backdoor defense using shapley estimation. In CVPR, 2022.
[41] Dongxian Wu and Yisen Wang. Adversarial ncuron pruning purifies backdoored deep models. In NeurlPS, 2021.
[42] Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. In NeurlPS, 2022.
[43] Yige Li. Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930. 2021.
[44] Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. Advances in Neural Information Processing Systems, 36, 2024.
[45] Lucas Bourtoule, Varun Chandrasckaran, Christopher A Choquette-Chon Choo,, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lic, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP). pages 141-159. IEEE, 2021.
[46] Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren. Towards reliable and efficient backdoor trigger inversion via decoupling benign features. The Twelfth International Conference on Learning Representations, 2024.
[47] Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearinng shared adversarial examples. In NewrIPS, 2023.
[48] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning, pages 26982-26992. PMLR, 2022.
12

---

[49] Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV, 2021.
[50] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS Symposium, 2018.
[51] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor leaming. In NeurlPS Datasets and Benchmarks Track, 2022.
[52] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2013.
[53] Johannes Stallkamp, Marc Schlipsing. Jan Salmen, and Christian lgel. The german traffic sign recognition benchmark: multi-class classification competition. IJCNN, 2011.
[54] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint wrXiv:1499.15: 2014.
[55] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. ICCV, 2023.
13

---

Appendix Outline
This appendix is organized follows:
In Section
A detail the algorithm of TSBD.
In Section we detail the approximation process of the fine-tuning optimizations.
In Section we introduce the implementation details, including the details of datasets,
models, attacks, and defenses with our proposed method.
In Section
we compare the defense results on the GTSRB dataset.
In Section
compare the defense results VGG19-BN structure.
In Section
we show the effectiveness of using NWC in other defense.
In Section we show the comprehensive results with different neuron ratios and weight
ratios.
In Section
we discuss different clean data ratios on the defense performance.
In Section
we discuss different fine-tuning learning rates on the defense performance.
In Section
we evaluate the influence using our method on the clean model.
In Section test the performance of scaled-up experiments the VIT model.
In Section
we show the computational overhead in terms of runtime.
A Detailed Algorithm of TSBD
To clearly illustrate our proposed method, we provide the detailed algorithm of the entire process of TSBD, which is shown in Algorithm
B Details of the Approximated Fine-Tuning Optimizations
As illustrated in Section 3.4 our proposed Activaness-aware fine-tuning involves calculating an additional gradient-norm regulation in the loss function, making it computationally inefficient. Therefore, we adopt the approximation scheme from [48] during practical optimization. Here, we provide the details of the approximated fine-tuning optimization. The approximation deviation shown in the following. Specifically, for each step of the gradient calculation, we formulate it as:
VoCco (0)
[oC Cac 0
V Loe
(0)
V (0)llz
A R VoCce ( + V V 0 +
V VoCoe
V VoCce(
(6)
|V Lee (0)l22
V Co
= (1 a Vo 0) a Coe (
Le (0)
To avoid the Hessian computation, the second term is further approximated through an additional parameter update:
V Ce ~ VoCco (0)
CoCo +
Vo O0)l
0=0+r
Based on the approximation, the practical fine-tuning process illustrated in Algorithm
C More Implementation Details
We further illustrate the implementations here, covering the details datasets, models, attacks, and defenses.
Dataset Details.
The experiments are conducted on CIFAR-10
13 Tiny ImageNet [2 and
GTSRB [53]
14

---

BadNets
Blended
Input-aware
LF
SIG
SSBA
Trojan
WaNet
Figure 7: Examples backdoor-atte triggers on the same image of CIFAR-10.
CIFAR-1O. It contains 60,000 x2x colored images with 10 classes. Each class owns 6,000 images, consisting of 5,000 for training and 1,000 for testing.
Tiny ImageNet. It is a subset of the full ImageNet, consisting of 100,000 training data and 10,000 testing data. There are 200 classes in total and 500 images per class for training. All images are 64 x4 with color.
GTSRB, GTSRB (German Traffic Sign Recognition Benchmark) contains 39,209 images for training and 12,630 images for testing with 43 classes. All images are 32x32 colored images.
Models. We choose PreAct-ResNeet 12 and VGGI9-BN [44 as the target models to conduct attacks and defenses following the default configurations in BackdoorBench [5T]. Both of them contain convolutional layers and batch normalization layers, which can be implemented with all kinds of defense methods, e.g., FP (37 for the last convolutional layer and ANP [4] for the batch
normalization layers. The extensive experiments on ablation study and further analysis are conducted on PreAct-ResNet18 by default.
Attack Details. We conduct 8 SOTA attacks for comprehensive testing, consisting of BadNets (8) Blended (25). Input-aware (23] LF 49 SIG (26). SSBA 99, Trojan [50] and WaNet 24 All the attacks follow BackdoorBench'ss s default configurations. Figure 7 shows all 8 attack triggers with the same example of CIFAR-10. Specificallyy, for BadNets, 3 3x3 white square is patched at the bottom-right corner of the images for CIFAR-10 and GTSRB, and a 6x6 white square is for Tiny ImageNet. For Blended, a Ketty image blended in the images with a 0.2 transparent ratio. We choose the 10% poisoning ratio and 0 label as the default setting to conduct attacks and test all defenses following the previous works 55 47 5% and % poisoning ratios are conducted only for testing our proposed method.
Defense Details. We conduct 8 SOTA defenses for comprehensive comparison, containing Fine- tuning (FT), Fine-pruning (FP) (37. NAD 43). NC [20). ANP [4). CLP (38, i-BAU 21]. and RNP 22 The defenses also follow the BackdoorBench 's default configurations. Note that we compare only the post-training defenses with 5% benign data provided. The learning rate for all methods is set 10 2, the batch size is set 256. For RNP. the clean data ratio is set 0.5% since we found that it failed to defend well under the 5% setting. For our proposed method, we set the default learning rates for unlearning as 10-4 and fine-tuning as 10-2, The unlearning is stopped when clean accuracy drops to or below 10%. The default fine-tuning epoch is set to 20. The default neuron ratio n and weight ratio m are set to 0.15 and 0.7, respectively. We follow the suggested settings of hyper-parameters r 0.05 and = 0.7 for fine-tuning [48] Other settings are set following the default FackdoorBench configuration.
All experiments are conducted on server with GPU RTX 3090 and CPU AMD EPYC 7543 32-Core Processor. These experiments were successfully executed using less than 24G of memory on single GPU card.
D Evaluations on GTSRB dataset
We validate the effectiveness of our proposed method in the dataset GTSRB other than CIFAR-10 and Tiny ImageNet. Table/]sshww the corresponding performance on PreAct-ResNet18 with 10% poisoning ratio and 5% clean data ratio. We can observe that TSBD performs consistently with the lowest average ASR and the largest average DER as in CIFAR-10. Most of the defenses also fail in the strong attacks Blended, LF,, and Trojan, while NC performs the second best with comparable average ASR and DER. Compared to the other two datasets, TSBD performs much superior with
15

---

Algorithm Two-Stage Backdoor Defense
Input:
Small clean set De, backdoored model with parameter Ond, max iteration number T, neuron ratio n% and weight ratio m% for reinitialization
Output: Clean model with parameter 0
1 Neuron Weight Change-based Backdoor Reinitialization */
// a. Clean Unlearning
2: while Clean accuracy on al > 10% do
3: Sample mini-batch B. from De
4: Out < maxØ... Lf (B,: maa)
5: end while
Neuron Weight Change Calculation.
6: Record subweight changes and calculate NWC by Equation w.r.f. Out and Ond
c. Zero Reinitialization.
7: Obtain and sort the weight changes from the top-n% neurons w.e. NWC
8: 0 reinitialize m% of the most-changing weights into zero And
9: 1* Activeness-aware Fine-tuning */
10: for 1. 1 to 7 do
11: Sample a mini-batch B, from D,
12: Update 0 by Equation (4) with the approximated Equation 6
end for
14: 0" < fine-tuned 0
Algorithm 2 The Approximated Optimization of Activeness-aware Fine-tuning
Input: Small clean set De. the reinitialized model with parameter Ó, max iteration number T, approximation scalar balance coefficient
Output: Clean model with parameter 0"
1: / Activeness-aware Fine-tuning */
2: for = 1 to T do
3: Sample a mini-batch B. from DD Cee ()) with B. Temporally update the current parameter 0' VgC.((
4: Calculate the gradient
5:
6: Calculate the gradient 92 V Lae with B.
7: Calculate the final gradient (1 a)gı ag2
8: Update the parameter based on g
9:
end for
0" fine-tuned 0
most ASRs lower than 0.5% except for Blended attack. This suggests that TSBD might perform better when the model is learned with the input images with similar characteristics.
E Performance on VGG19-BN model structure
Except for PreAct-ResNet18, we also test another model, VGG19-BN, The performance on CIFAR- 10 with 10% poisoning ratio and 5% clean data ratio illustrated in Table 7 We follow similar settings in PreActtResNNet188 on CIFAR- 10, conducting 8 attacks and comparing our method with 8 defenses. The performance is also evaluated by ACC, ASR, and DER. As shown in the table, TSBD owns SOTA performance on VGG19-BI BN with the best average ACC and second-best average ASR and DER. Most of the other performances are in similar pattern as on PreAct-ResNet11. Although ANP performs almost the best in ASR and DER for all attacks, it damages the corresponding ACC as well, especially for the WaNet attacked model, where ACC decreases from 84.58% 78.04%. On the contrary, methods succeed in most attacks with high ACC and comparable ASR.
16

---

Table 6: Comparison with the SOTA defenses GTSRB dataset with PreAct-ResNetle (%).
Rackdoor
No
NAD
ASRC
BadNNnt
97:24
50:25
98.73
515
77.05
1119
75.38
95.05
0.6.1
79.31
97.48
0101
79.02
Blended [5
58.58
99.99
95.57
100.00
50.00
98.38
100.00
49.90
95.01
100000
50.00
97.76
5003
95.57
Input aware
97.26
92:74
98.40
29.81
61.46
98.08
2.32
05.21
98.27
50.65
76.04
95.55
0.01
96.25
LF
199
97.93
99.57
96.01
79.98
59.80
97.59
99.70
49.83
98.14
51.88
73.87
97.97
1.34
99.11
SSBA 0
97.98
99.56
97.92
99.10
50.20
97.75
99.46
94
97.95
99.39
50.07
07.72
0.29
99.50
Trojan 50
98.57
100,00
98.54
0.02
99.97
98.33
71,30
64,22
98.20
0.11
99.76
87.53
0.83
94.,7
WalNer 24
97.74
94.25
98.54
0.29
96.98
97. 62
58.07
53.63
98.61
0.56
96.84
95.25
0.00
97.12
97.90
92.19
98.39
$4.50
73.64
97.99
65.85
K3.10
98.35
41.85
75.13
95.47
1.50
CCP
1-5AN11211
KNP
Attacks
ASRC
DER
ASK
ACC
ASK
ASK
KidNett
97.60
0.001
Blended H5
95.75
99.82
50.09
98.48
100.00
49.95
92.35
86.35
53.71
98.84
99.64
50.18
95.57
5.94
95.52
Input aware
23
9911
0.00
96.37
99.664
90.01
50.00
97.09
0.52
90.03
97.17
0.00
96.32
99.23
0.29
56.22
1F 120
97.80
61. $1.38
99:03
97:70
99:50
49.92
95.78
16.15
90.64
97.91
99.06
50.25
96.83
0.83
99.22
SS0A g
97.86
98.93
50.26
98.08
99.24
50.16
96.14
1.88
97.92
97.78
99.43
49.96
96.06
0.10
98.77
Rojaan
50
98.10 188
0.90
99:75
98.11
95.
52.20
95 99
0.02
98. 70
98.56
100.00
50.00
97.16
0.15
99.22
WaNeet
24
97.08
0.00
96.80
7.16
100.00
4.71
96.72
0.00
96.62
96.77
0
96.64
98.80
0.03
97.11
40.05
75.96
83.13
93.85
43.85
95.79
14.99
57.55
56.97
67.52
0.93
95.10
Table Comparison with the SOTA defenses on CIFAR- 10 dataset with VGGI9-BN(%).
Backdoox
No Défcase
FP 57
NAD 43
NC 20
Attacks
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC 1
ASR
DER
Bulleet
90.12
11.33
15.15
27.39
K2.31
$5.96
10:23
91.33
55.31
3:47
92.51
39:21
11.31
90.35
Blended
23
91.91
99.
90.08
86.82
55.42
$9.95
57.46
55.00
58.64
53.86
56.17
50:00
83.33
57.16
Impui-wrars
M1
88.66
94.58
91.56
13.08
90.75
91:35
5.08
94.25
91.00
14.11
90.25
89:70
97.02
30,00
60
87.67
56.01
56.34
1.36
88.01
4.78
95.74
SSDA
95.11
Trojan
7.37
2.30
96.71
7.76
WiNerr
84.50 58
96-49
91.35
5.72
95.
91.12
4.74
88
90.73
10.33
93.08
91.20
6.88
94.81
Averuge
85.09
$6.00
89:47
27.09
79:28
89003
30.21
77.77
$7.78
22.05
31.10
$5.00
30,20
08.02
Backdloce
ANF [I]
CLPTS)
1-6AU 21
RNP
TSDD (Ours)
Attacks
ALC
ASR
13E.
XCCC
ASR
DER
ALCC
ASR
DEB
ACC
ASK
DEK
ALC
ASR
DER
RadNen B]
85.39
0.48
95.99
89.38
6.61
93.39
86.01
2:38
93.87
70.
0.12
91.90
58.76
3.58
94.60
Blended
25
89.19
4.77
96.01
90.66
98.59
69.83
87.58
69. 10.90
62.64
26.92
43.20
45.66
90.51
6.68
95.71
Isput-wess
24
86.56
0.71
87:54
2.66
49.41
35.26
75.37
58.14
2.58
98.14
SSRA
96.36
98.56
87.56
94.78
69.02
86.62
58.97
3.41
04.91
Trujan
50
89.30
0.00
98.91
90.56
99.70
69.65
88.63
8:23
94.11
53.40
57.30
52.27
90.22
6.63
96.01
Walee
24
78.04
0.63
94.96
88.46
1,75
97.37
89.76
1.61
97.44
87.90
$1.03
57.75
91.28
2.50
97.00
Avenge
86.05
0.93
91.75
87.07
$2.64
66.40 66.49
87.37
22.58
81.05
61.
33.63
63.05
89.76
1.06
90.38
F Effectiveness of using NWC in Other Defense
As illustrated in Section 4.3 it is effective to utilize NWC order in gauging the backdoor strength. To further test its ability to improve other defenses, we substitute the average neuron activation in FP to NWC (denoted as NWC-FP) and conduct the experiments on the default settings in PreAct-ResNet18 on CIFAR-10. Different from zero reinitialization, the pruned neurons will not be updated on the following fine-tuning. The performances of 8 attacks are shown in Figure g We can investigate that, by using NWC in FP, performances on ASR and DER are improved most of the attacks, with few effects on ACC, It exhibits the potential of using our NWC in more defense methods.
G Performance with Different Neuron Ratio and Weight Ratio
Section 4.4 shows some the performances with different neuron ratios and weight ratios and verifies the robustness of the byper-parameter tuning in our method. Here, exhibit the full results under all
Figure 8: Performance comparisons between the original FP and the FP with NWC (NWC-FP) under
8 attacks. Left: ACC; Middle: ASR; Right: DER.
17

---

BadNeh
154
100
104
MCC
ACC
ACC
ACC
500
3084
Toun
Wottet
ACc A2R
ACC A2e
Accuaoo
A2E
Figure 9: Performance with different neuron ratios under 8 attacks.
(50
Viest Ruta Infi
Westt Ruts teel
Westt Aristrii
Wesgr Aate
Toin
FRC
ACC
ACC
ACC
ACC
05
Weittr
Write
Weight a
Figure 10: Performance with different weight ratios under 8 attacks.
8 attacks in 8 on CIFAR-10 10 with 10% poisoning ratio. Figure 9 and Figure exhibii the tuning of neuron ratio and weight ratio, respectively. All performances are as good as expected.
H Evaluations with Different Clean Data Ratios
We are here to test the performance of TSBD under different clean data ratios. The performance on CIFAR-10 10 and PreAct-ResNet118 with 10% poisoning ratio is illustrated in Table Except for the default 5% clean data ratio, we also test the performance under 10%, 1%. and 0.5%. We can observe that larger ratio of clean data can always bring better performance in ACC, egg., 92.18% >> 91.70% > 89.63% > 86.42% on average values of the four ratios in decreasing order, respectively. For the defense performance in ASR, all of them can successfully defend the attacks to under 10%, and large clean data ratios can achieve promising DERs. Overall, TSBD is robust to the clean data ratio, with only 0.5% clean data can also defend most tested defenses successfully.
I Evaluations with Different Learning Rates on Fine-tuning
We test the performance of our proposed method under different learning rates on fine-tuning, Table illustrates the results. Specifically, with the same settings on CIFAR-10 and Reseet- with 10% poisoming ratio and 5% clean data ratio, we test the performance when the learning rate is set to 0. 0.01, 0.001, and 0.0001, where 0.01 is our default setting in the previous experiments. We can observe that the default learning rate of 0.01 is the most suitable one chosen for defense, which can achieve SOTA ASR and DER on average with comparable average ACC. Besides, setting the learning rate 0.001 can perform the best in ACC, while making it fail some strong attacks, e.g.
18

---

Table 8: Performance with Different Clean Data Ratios CIFAR-10 dataset with 8
(%)
Clean Data Katio
UNo Delense)
10% (TSOD)
5%5 (TSDD)
16 (ETSSDD:
0.5% (TSO))
Attaclas
ACCCT
ASR.
DEK
ACCT
ASK :
DEK
ACC
ASX
DEK
ACCC"
ASR.
DEK
ACCT
ASK
DEK'
BodNees
g
91.32
0.32
90.72
96.53
51.13
1.18
Blanded
25
03.47
5.79
1.32
97.25
lojut-wware
92167
1.56
52.05
1:32
08.16
91.54
97.82
[10
1.62
91.20
97.32
89.55
1.01
2.14
91.57
97.44
Thajan 50
93:42
100.00
92.37
7.00
53.98
91.15
3.06
96.64
90.18
4.66
96.05
87.98
7.21
93.68
WaNet
91
89.73
93.66
0.58
94.38
93.26
0.88
94.43
91.
1.24
54.22
89.57
0.69
93.68
Average
91:34
97.20
92.18
2.99
06.86
91.70
2.18
97.09
89.63
2.67
96.05
86.42
2.93
94,72
Table 9: Performance with Different Learning Rates of Fine-tuning on CIFAR-10 dataset with PreAct-ResNet18 (%).
Leamnng
(TSED)
(ISIIT)
BadNett
01.32
03.03
4.15
96172
2.10
01.44
4.37
Blended
25
03.47
99.92
74.79
0.57
90.34
91.61
2.61
97.73
92.35
12.19
92.10
92.88
29.33
55.00
Input uwaro
90067
98.26
88.70
3.00
95.64
03.00
1.54
99.10
93.22
2.05
98.10
91.46
0.64
98.81
139
93.19
09 20
79.38
3.50
90.98
9T:20
97.32
92.90
11.98
93.50
92.79
37.03
80.92
SKG 16
84.48
98.27
73.15
3.68
91.63
90:41
1.27
98.50
98.87
4.81
96.75
55.23
14.47
91.90
SSBA y
92.88
97.55
79.90
0.21
90.33
91.37
1.66
97.44
92.0
9:26
54.18
92.72
15.27
91.21
Trojen
50
93.42
100.00
73.83
32.87
73. 77
91.76
5.06
96.64
92.75
10.76
94.29
92.73
21.07
89.12
WaNNet
91.25
89.73
87.61
1448
92.31
93.26
0.58
94.43
93.
0.75
$4.87
91.36
0,31
94.71
Averge
92.50
Blended and LF. On the contrary, setting the learning rate 0.1 will hurt the ACC greatly, which may imply that clean knowledge cannot be re-learned properly.
J Evaluations on Clean Model
To test whether our defense method will hurt the performance of the clean model if no backdoor attack occurs, compare it with several defense methods on CIFAR-10 and Tiny ImageNet datasets on PreAct-ResNettla Figure II illustrate the results, where the ACC and ASR on the original clean model, ANP, and RNP, are used for comparison. We observe that most of the defense methods (including ours) will barely damage the clean model though there is no backdoor occurs, except for RNP on Tiny ImageNet. It exhibits the potential to largely employ our defense methods in real-world Al systems.
K Performance on Scaled-Up Experiments
To further verify the scalability of our method, evaluate its performance ViT-b-16 model with the CIFARIO dataset following the basic settings in Section 4.1, e.g., the poisoning ratio is set to 10% and the target label set to O. The results are shown in Table The results demonstrate that TSBD performs effectively on the scaled-up model, achieving low ASR and acceptable ACC. In contrast,
CLP and ANP fail completely with ASR still a high level, particularly for the WaNet attack.
L Computational Overhead
To show the computational overhead of TSBD, we record the average computational time of each defense step and its practical runtime compared to other defense methods. Table TT exhibits the computational time of TSBD, including Clean Unlearning. NWC Calculation, Zero Reinitialization. and Activeness-Aware Fine-Tuning. We observe that the main computational overhead lies the fine- tuning process. In contrast, the time required for clean unlearning does not increase proportionally with dataset complexity. This means that TSBD is as efficient as other fine-tuning based methods. Moreover, we present a practical runtime comparison with other SOTA defenses in Table [2] including the loading and testing time needed in practice. As we can see, TSBD is faster than most of the existing methods.
19

---

(a) CIFAR-10
(b) Tiny ImageNet
50
Accuracy (%)
Accuracy (%)
40
ACC
30
ACC
20
10
Clean Model
ANP
RNP
TS8D
Clean Model
ANP
RNP
TSBD
Figure 11: Performance of Defenses Clean Model.
Table 10: Performance on CIFAR-IO 10 with ViT-b-16 (%).
No Defense
CLP
[38
ANP 41
TSBD
Attacks
ACCT
ASR.
ACCT
ASR
ACCT
ASR
ACC1
ASR
Input-aware 133
91.65
92.30
90.55
79.34
90.40
50.69
86.11
4.43
WaNet 24
89.12
80.95
89.12
80.95
89.12
80.95
88.43
1.59
Table 11: Computational Time of Each Defense Step of TSBD
Delense Step
CIFAR-TO
Tiny ImageNel
Clean Unlearning
20.84s
17.90s
NWC Calculation
0.03s
0.03s
Zero Reinitialization
1.34s
1.29s
Activeness- Aware Fine-Tuning
21.08s
174.36s
Table 12: Practical Runtime Comparison on BackdoorBench
Datasets CIFAR-10
FT
FP [37]
ANP [4]
NC [20]
RNP [22]
TSBD
358s
855s
505s
733s
123s
159s
Tiny- ImageNet
1649s
20429s
2578s
37101s
285s
269s
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The contributions and scope are accurately written in the Introduction Section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The last sentence of the Conclusion section clearly states the limitation and potential solution.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
21

---

Justification:
All the results are empirically illustrated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All experimental settings are clearly illustrated the Experiment section and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: All the data are open-sourced. The code is provided Supplementary Material, and will be publicty available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//uuui guwde//ddee/ee/ wwt.:Suwmisssispoooooooooooooooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All settings and details are provided in the Experiment section and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We strictly follow the basic settings in the open-sourced BackdoorBench [1] which ensures the faimess of all the comparisons in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
23

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The computer resources are included in Appendix C, and the official informa- tion from BackdoorBench, since we follow their default settings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics h httpa/rpp...ppppupccccccccccceee
Answer: [Yes]
Justification:
We strictly conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The experiment in Appendix K exhibits the positive societal impacts of potentially employing our defense method without hurting the model performance in the real-world AI system.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our defense method has only a positive impact on use.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All licenses are explicitly mentioned.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the documentation for the summit code in Supplementary Mate- rial, and it will be publicly available.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "cZIkeo757f", "reviewer": "Reviewer_Vmzu", "summary": "The paper introduces two key observations in backdoored models, presenting a two-stage backdoor attack defense method, where the two observations are the following: a strong positive correlation between weight changes in poison and clean unlearning, and the stronger neuron activation in backdoored models compared to clean models. The proposed two-stage defense method leverages these observations by reinitializing a certain proportion of backdoor-related neurons and subweights, and by suppressing gradient normalization during the fine-tuning process. The proposed method demonstrates the state-of-the-art performance on the selected datasets and models with a wide range of ablation studies.", "questions": "1. There is no statement or analysis about scalability and generalizability. Can you provide more detailed information about these, to different datasets and model architectures?\n\n2. How can the clean unlearning process identifies and affects backdoor related neurons, without causing degradation of the models’ performance on clean data?\n\n3.\tThe section introducing the effectiveness of zero reinitialization on subweights in 4.3 Ablation Studies is a bit confusing. The difference between V2 and V3 is still not clear to me.\n\n4.\tI have questions about the insensitivity of TSBD to both neuron ratio and weight ratio as explained in Section 4.4. While I understand that the method effectively selects backdoor-related neurons indicated as active neurons and that reinitializing them is effective, it is confusing that the performance remains similar across both low and high ratios. Is reinitializing itself the important part?", "limitations": "The authors mention in the introduction that clean unlearning is still underexplored, but it is unclear if this limitation has been fully addressed in the paper. Additionally, the paper lacks a discussion of its own limitations. Providing a clear discussion on the limitations of the proposed method and potential areas for future research would enhance the paper.\n\nThe authors acknowledge that there remain challenges in fully mitigating backdoor effects without any access to clean data. They suggest that data generation techniques and data free techniques might offer potential solutions.\n\nThere is no discussion about the scalability of their proposed method, especially in large scale or real time applications. Addressing the computational overhead and resource requirements would be necessary for practical applications.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.  The paper introduces a novel approach to backdoor defense. It provides two insights in the weights of backdoor models. Based on this, the authors introduce a two-stage defense mechanism. Also, the use of clean unlearning that does not require poisoned data to prevent the backdoor attack.\n\n2. While the scope of the experiment is limited in model architectures and datasets, the experiment shows superior performance. The experimental setup is thorough, covering a certain range of backdoor attacks. \n\n3. The paper is well-organized, with figures and tables are used to illustrate key observations and experimental results. The descriptions of the observations and the proposed defense method are detailed and precise.", "weaknesses": "1. The scale of experiments are too limited. While the authors compare the results between 8 other backdoor attack methods, the main experiments are only conducted to specific datasets and models (CIFAR-10, PreAct-ResNet18). This makes the generalizability of the results rather unclear.\n\n2. The authors mention computational efficiency in Activeness-Aware Fine-Tuning by using an approximation scheme, but there are no experiments demonstrating the trade-off between efficiency and accuracy. Including such experiments would provide a more comprehensive understanding of the method's practical implications.\n\n3.\tThe concept of unlearning in backdoor attacks could be better explained in the Related Works section. Given its importance in this paper, a more detailed discussion would help contextualize the proposed method and highlight its significance in addressing existing challenges.\n\n4. There is no mathematical proof. While it employs various loss functions and approximation techniques, and the empirical results are quite promising, there is no analysis, explaining why the proposed method works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Vmzu, thank you very much for your careful review of our paper and thoughtful comments. We are encouraged by your positive comments on our **novel and insightful method**, **superior performance**, **thorough experimental setup**, and **good paper presentation**. We hope the following responses could help clarify the potential misunderstanding and alleviate your concerns.\n___\n\n**W(Weakness)1&Q(Question)1: Concern about the scale of main experiments.** \n\n**R(Response)1:** We appreciate your concern about the scale of the main experiments. We would like to clarify the following points:\n\n - Our main experiments are not only conducted on CIFAR-10, PreAct-ResNet18, but also cover the datasets of **Tiny ImageNet, GTSRB,** and the model of **VGG19-BN**. Please refer to **Section 4.2, Appendix D and E** for the details. \n - In this paper, we follow a similar testing range on models and datasets as in the previous SOTA works [1,2], which is considered sufficient to prove the generalizability of the results.\n \n To further prove the **generalizability** and **scalability**, we also evaluate our method on CIFAR-100 (as follows) and ViT-b-16 (due to the space limit, please refer to **RW1 of Reviewer obuU**). Note that we keep other settings the same as in section 4.1. The results show the superiority of TSBD.\n\n**Table 1: Experiment Results for CIFAR-100 on PreAct-ResNet18**\n|CIFAR-100| No Defense |No Defense|i-BAU|i-BAU|TSBD|TSBD|\n|--|--|--|--|--|--|--|\n| Attack | ACC |ASR|ACC|ASR|ACC|ASR|\n| BadNets | 67.22 |87.43|60.37|0.04|66.28|0.33|\n| Input-aware | 65.24 |98.61|65.21|85.14|69.67|0.18|\n___\n\n**W2: Suggestion for the experiment on computational efficiency of Activeness-Aware Fine-Tuning.** \n\n**RW(Response to Weakness)2:** Thanks for your interest in the computational efficiency. In fact, as pointed out in [3], it is infeasible to conduct such an experiment for DNNs since it involves calculating a Hessian matrix (refer to **Appendix B**) without approximation, where the *time* and *space complexity* are $O(n^2)$ theoretically. We now provide a brief example on *PreAct-ResNet18*, which contains $n\\approx11$ million parameter units. Its calculated Hessian matrix will contain $n^2 \\approx124,794$ billion units, which is much larger than a LLaMa 65B and fails to be calculated on one GPU, e.g., A6000 GPU with 49G memory.\n___\n\n**W3: Suggestion for adding unlearning to related work.**\n\n**RW3:** Thanks for your valuable suggestion. We will update some related works on unlearning-based backdoor defense to highlight our contribution in the revised version. Due to the space limit, a brief version is provided in the **RW3 of Reviewer kmL5**.\n___\n\n**W4: Suggestion for mathematical proof on explaining why TSBD works.**\n\n**RW4:** Thanks for your in-depth suggestion. We have explained the functionality of those important techniques in **Section 3**. For example, \"Suggestions\" in Section 3.2 clarify why we need both two stages. Additionally, in **Section 4**, we empirically validate the effectiveness of each component. \n\nIn fact, except for the empirical findings, we also attempted to mathematically derive the observations 1 and 2. However, this seems to be a very difficult task. For example, to prove observation 1, we need to estimate the NWC values, which occur in weight changes during the whole unlearning process. It involves estimating the total changes on a variant (e.g., $x$) for $K$ steps of gradient descents, i.e., $\\||x_K-x_1\\||=\\||\\sum_{t=1}^{K-1}(x_{t+1}-x_{t})\\||$. As far as we know, **in *optimization theory*, there exists no mathematical tool to estimate this quantity directly;** instead, more focus is placed on estimating the distance between $x_{t}$ and the limit point. In future work, we will continue to explore this issue.\n___\n\n**Q2: Explanation for the clean-unlearning capability.** \n\n**RQ(Response to Question)2:** Thanks for your question. We would like to point out that the *clean unlearning* in stage 1 will degrade the clean performance. While **the reinitialization is conducted on the original backdoored model**, clean degradation on the unlearned model will not affect the final performance. Based on **observation 1 in Figure 1**, we find that the backdoor-related neurons that change the most in weight during poison unlearning also change significantly during clean unlearning. Therefore, we define the neuron weight change to identify them, and then remove them from the backdoored model. For how unlearning affects the neurons, we offer insights from the perspective of neuron activations in **Section 3.3**. We would like to refer you to the **RW2 of Reviewer P1hB** for a brief summary.\n___\n\n**Q3: Explanation for the different model versions in ablation study.**\n\n**RQ3:** Thanks. Due to the space limit, we would like to refer you to the **RW2 of Reviewer kmL5** for the explanation of the differences among these three versions.\n___\n\n**Q4: Concern about the importance of reinitialization.**\n\n**RQ4:** We appreciate your concern about the importance of stage 1. While it exhibits a stablely good performance across different ratios in BadNets, **we cannot overlook its contribution to backdoor removal in some strong attacks**, e.g., Blended, LF, and SSBA, where ASR larger than 20% after defense (see **Figure 8 of Appendix G**). For more details, we would like to refer you to the **R2 of Author Rebuttal**, where we **validate the importance of stage 1** by comparing TSBD with the version containing only stage 2.\n___\n\n[1] Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples. NeurIPS 2023.\n\n[2] Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features. NeurIPS 2023.\n\n[3] Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning. ICML 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7VrAUE7Vx", "reviewer": "Reviewer_P1hB", "summary": "The paper addresses the security threat posed by backdoor attacks in deep neural networks (DNNs). The authors explore model unlearning from the perspective of weight changes and gradient norms, making two key observations: weight changes between poison and clean unlearning are positively correlated, and neurons in backdoored models are more active than those in clean models. Based on these observations, they propose a Two-Stage Backdoor Defense (TSBD) method, involving Neuron Weight Change-based Backdoor Reinitialization and Activeness-Aware Fine-Tuning. Extensive experiments demonstrate the superior performance of their method compared to state-of-the-art approaches.", "questions": "1. When selecting the clean dataset, how is the class distribution handled? Is it randomly selected or fully covered? Intuitively, would data related of the target label cause particularly high activation values?\n2. I think the NWC strategy is not significantly different from the pruning method in RNP[1], and the Activeness-Aware Fine-tuning is also similar to the fine-tuning in RNP. Therefore, I hope the authors can emphasize more clearly the differences between their approach and RNP.\n3. From the observations, it is apparent that there is a difference in neuron activation between clean and backdoored models. However, the use of gradient norm restriction in the unlearning process is applied to an already modified model (not the original backdoored model). The direct correlation between this and enhanced defense effectiveness does not seem sufficiently clear and is not within the scope of the initial observations.\n4. Regarding the selection of hyperparameters in Activeness-Aware Fine-tuning, I believe this can significantly impact the experimental results, but there is no discussion on this aspect in the paper.\n\n[1] Li Y, Lyu X, Ma X, et al. Reconstructive neuron pruning for backdoor defense[C]//International Conference on Machine Learning. PMLR, 2023: 19837-19854.", "limitations": "I think the paper still has some limitations that the authors didn't mention. Please refer to my questions and weaknesses.\nPlease add some discussions about data-free methods", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel perspective on backdoor defense by exploring the correlation between weight changes in poison and clean unlearning and the activeness of neurons. This approach provides new insights into identifying and mitigating backdoor vulnerabilities.\n2. The paper is well-organized and clearly presents its methodology, findings, and contributions.\n3. The proposed TSBD method is rigorously evaluated through extensive experiments involving eight backdoor attacks on three benchmark datasets.", "weaknesses": "1. The TSBD method involves additional steps such as clean unlearning, neuron weight change calculation, and activeness-aware fine-tuning, which may introduce computational overhead. An analysis of the computational cost and efficiency of the proposed method compared to existing defenses would be beneficial.\n2. What causes the clean unlearning NWCs to exhibit a positive correlation with those in poison unlearning? I think this question has not been solved clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer P1hB, thank you very much for your careful review of our paper and thoughtful comments. We are encouraged by your positive comments on our **novel and insightful method**, **good paper presentation**, and **rigorous evaluation**. We hope the following responses can help alleviate your concerns.\n___\n\n**W(Weakness)1: Suggestion for the analysis of computational cost.**\n\n**RW(Response to Weakness)1:** Thanks for your constructive suggestion. We would like to refer you to the **RW2 of Reviewer obuU** for a comprehensive analysis, which emphasizes that **TSBD is an effective and efficient method**.\n___\n\n**W2: Why are clean-unlearning NWCs positively correlated with those in poison unlearning?**\n\n**RW2:** Thanks for your in-depth question on the observation 1. This question has been discussed and analyzed from the perspective of neuron activation in **Section 3.3**. Briefly, **the neurons with higher *poison activations* are the main targets to be changed in both *clean* and *poison unlearning*.** They tend to decrease the poison activation during poison unlearning, while increasing it during clean unlearning, keeping the clean activation nearly unchanged (see **Figure 3**).  We also deduce that the weight change on a neuron positively influences its changes in clean and poison activation. Therefore, no matter clean or poisoned data are used for unlearning, those neurons with higher poison activations in the backdoored model are changed more in weights than the others, as reflected in the positive correlation of neuron weight changes (see **observation 1 of Figure 1**).\n___\n\n**Q(Question)1: Concern about clean-data selection and class distribution.**\n\n**RQ(Response to Question)1:** Thanks for your interest in the clean dataset used for defense. We strictly follow the same data setting in BackdoorBench [1] for a fair comparison with the baselines. Specifically, 5% of clean data are **randomly selected** from the unpoisoned dataset with no manipulation on the class distribution. Moreover, to answer the question of the correlation between target-label data and the activation values, we now average the activation values of the last convolutional layer on each class of CIFAR-10 for comparison. For each class, we randomly select 10 samples on each class as the input, and capture the output of the target layer with an additional *ReLU* activation function. The results are illustrated in Table 1 (as below), which shows that there is **no obvious relationship between the target-label data and the activation**.\n\n**Table 1: Average activation for the last convolutional layer on each class of CIFAR-10**\n| Class $\\rightarrow$ |  0 (target class) |1|2|3|4|5|6|7|8|9|\n|--|--|--|--|--|--|--|--|--|--|--|\n| Avg. Activation | 0.2212 |0.2146|0.2126|0.2175|0.2229|0.2231|0.2140|0.2177|0.2161|0.2186|\n___\n\n**Q2: Concern about the differences compared with RNP.**\n\n**RQ2:** Thanks for your concern about the differences compared with RNP. We would like to refer you to **R1 of Author Rebuttal** for a comprehensive comparison, which clarifies that **TSBD is a newly designed method with several fundamental differences compared with RNP**.\n___\n\n**Q3: Concern about the gap between observation 2 and loss regularization on gradient.**\n\n**RQ3:** Thanks for your in-depth concern. The main idea we want to convey in observation 2 of Figure 1 is that **for an arbitrary clean model, after it has been attacked by a backdoor, the corresponding neurons will become more active than before**, i.e., they will exhibit a larger gradient norm during learning processes. **This is a general phenomenon, not limited to the initial backdoored model**. Therefore, for the Activeness-Aware Fine-Tuning in stage 2, we add an additional gradient norm regularization in the loss function to encourage a low gradient norm status for the optimized model after fine-tuning, which is considered closer to a clean model. **This purpose has no relationship with the intermediate model status after the zero reinitialization.**\n___\n\n**Q4: Suggestion for the hyperparameters experiment on Activeness-Aware Fine-tuning .**\n\n**RQ4:** Thanks for your constructive suggestion. As the hyperparameters $r$ and $\\alpha$ have been well-discussed for their influence on approximation performance in the original paper [2], and since they are only indirectly related to the backdoor, we follow the suggested settings, i.e., $r=0.05$ and $\\alpha=0.7$. Here, we provide the tuning results following the tuning range in [2] under our experimental settings. These results were obtained on a BadNets-attacked PreAct-ResNet18. We observe that **the performance is insensitive (Changes <2% in ACC and <1% in ASR) across different hyperparameter settings**, maintaining a high level of performance.\n\n**Table 2: Hyperparameters Tuning for Activeness-Aware Fine-Tuning**\n|$r$| $\\alpha$ |ACC | ASR|\n|--|--|--|--|\n| 0.05 | 0.7 |90.72|1.31|\n| 0.01 | 0.7 |90.68|1.30|\n| 0.02 | 0.7 |91.03|1.50|\n| 0.1 | 0.7 |90.90|1.30|\n| 0.2 | 0.7 |89.64|1.04|\n| 0.05 | 0.1 |91.32|1.40|\n| 0.05 | 0.3 |91.24|1.59|\n| 0.05 | 0.5 |91.00|1.63|\n| 0.05 | 0.9 |90.68|1.26|\n___\n\n[1] BackdoorBench: A Comprehensive Benchmark of Backdoor Learning. NeurIPS 2022 Datasets and Benchmarks Track.\n\n[2] Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning. ICML 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A7zHzMp27s", "reviewer": "Reviewer_obuU", "summary": "The authors propose a novel two-stage backdoor defense method TSBD. The proposed method is based on two key observations 1) the weight changes of neurons during clean and poison unlearning are correlated, 2) the backdoored neurons exhibit a larger gradient norm during unlearning. Respectively, the proposed defense method consists of 1) reinitializing neurons with high weight changes, 2) fine-tuning under gradient-norm regularization. The proposed method is compared with several state-of-the-art defense methods on different attacks, and a comprehensive ablation study is conducted.", "questions": "Elaborated above.", "limitations": "The authors discuss the limitations.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and easy to follow. The figures are very clear.\n2. The observations are interesting, providing insights into this field.\n3. The proposed method is effective across different attacks, showing low attack successful rate and good defense effective rate. The authors provide sufficient and convincing results.", "weaknesses": "1. How is the performance on scaled-up experiments, e.g. ViT, or ImageNet?\n2. What is the computational overhead of the proposed method compared to others?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer obuU, thank you very much for your positive appraisal and great interest in our paper. We are encouraged by your positive comments on our **good paper presentation**, **insightful observations**, and **convincing evaluations**. We hope the following responses can help answer your questions.\n___\n\n**W(Weakness)1: Performance on scaled-up experiments.**\n\n**RW(Response to Weakness)1:** We appreciate your interest in the scaled-up cases in TSBD. To further verify the **scalability** of our method, we evaluate its performance on a ViT-b-16 model with the CIFAR10 dataset as follows:\n\n**Table 1: Experimental Results for ViT-b-16**\n| Method | No Defense |No Defense|CLP|CLP|ANP|ANP|TSBD|TSBD|\n|--|--|--|--|--|--|--|--|--|\n| Attack | ACC | ASR|ACC|ASR|ACC|ASR|ACC|ASR\n|Input-aware|91.65|92.30|90.55|79.34|90.40|50.69|86.11|**4.43**|\n|WaNet|89.12|80.95|89.12|80.95|89.12|80.95|88.43|**1.59**|\n\nNote that the other settings follow the basic ones in section 4.1, e.g., the poisoning ratio is set to 10% and the target label is set to 0. The results demonstrate that **TSBD performs effectively on the scaled-up model**, achieving a low ASR and acceptable ACC. In contrast, CLP and ANP fail completely with ASR still at a high level, particularly for the WaNet attack. Due to time constraints, we postpone the testing on large-scale datasets to future work.\n___\n\n**W2: What is the computational overhead?**\n\n**RW2:** Thanks for your interest in the computational overhead of TSBD. We would like to emphasize that **TSBD is an effective and efficient method** that can defend against backdoor attacks with acceptable overhead. To support our statement, we now show the average computational time of each defense step in Table 2, including *Clean Unlearning*, *NWC Calculation*, *Zero Reinitialization*, and *Activeness-Aware Fine-Tuning*. We follow the same experimental setting as in section 4.1, using PreAct-ResNet18 with a 10% poisoning ratio. The experiments here are conducted on a server with A6000 GPU and AMD EPYC 7543 32-Core Processor CPU. We observe that the main computational overhead lies in the fine-tuning process. In contrast, the time required for clean unlearning does not increase proportionally with dataset complexity. This means that **TSBD is as efficient as other fine-tuning-based methods**. Moreover, we present a practical runtime comparison with other SOTA defenses in Table 3, including the loading and testing time needed in practice. As we can see, **TSBD is faster than most of the existing methods**.\n\n**Table 2: Computational Time of Each Defense Step of TSBD**\n| Defense Step | CIFAR-10 |Tiny ImageNet|\n|--|--|--|\n| *Clean Unlearning* | 20.84s |17.90s|\n| *NWC Calculation* | 0.03s |0.03s|\n| *Zero Reinitialization* | 1.34s |1.29s|\n| *Activeness-Aware Fine-Tuning* | 21.08s |174.36s|\n\n**Table 3: Practical Runtime Comparison on BackdoorBench**\n| Dataset | FT |FP|ANP|NC|RNP|TSBD|\n|--|--|--|--|--|--|--|\n| CIFAR-10 | 358s | 855s| 505s| 733s| **123s**|159s |\n| Tiny ImageNet | 1649s | 20429s| 2578s| 37101s| 285s|**269s** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nRIpiWYR9X", "reviewer": "Reviewer_kmL5", "summary": "This paper first makes two observations that neurons exhibiting significant weight changes during clean unlearning also tend to play crucial roles in poison unlearning, and neurons in the backdoored model are always more active compared to those in the clean model. The authors showcase on commonly used backdoor attacks and provide further explanations for these two observations. Based on these two observations, a model is first unlearned on clean data where weights with the highest changes are re-initialized, and then the model is optimized by activeness-aware fine-tuning. Extensive experiments are provided to support the proposed defense.", "questions": "My main concern is the difference between TSBD and RNP, as well as the practicality of TSBD as a defense.", "limitations": "The authors did not provide analysis on limitations of the proposed method.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The analysis is clear and well-formulated, based on which the proposed defense is effective and easy to interpret. Extensive experiments on different types of backdoor attacks are provided.", "weaknesses": "The design principle of the proposed method (TSBD) is very similar to RNP [22]. Both TSBD and RNP follow the two-stage defense setup, including clean unlearning and recovering. The performance of TSBD is also similar to RNP except for the SIG backdoor. The authors provide some explanations about the difference between RNP and TSBD, but they are not convincing. The core technical difference can be further clarified. \n\nTable 3 provides the ablation study on the zero reinitialization ratios, indicating that reinitialization is a sensitive parameter. In a practical threat scenario, how would the defender select a proper reinitialization ratio? It is also suspicious whether the neuron ratio selection plays a key role in increasing the effectiveness. Closely related work [a] is missing, where clean unlearning by mask optimization is also discussed. \n\n\n[a] Towards reliable and efficient backdoor trigger inversion via decoupling benign features. In ICLR, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer kmL5, thank you very much for your careful review of our paper and thoughtful comments. We are encouraged by your positive comments on our **well-formulated analysis** and **effective method**. We hope the following responses could help clarify the potential misunderstanding and alleviate your concerns.\n___\n\n**W(Weakness)1: Concern about the differences compared with RNP.**\n\n**RW(Response to Weakness)1:** We appreciate your concern about the differences compared with RNP. We would like to clarify that **TSBD is a newly designed method with several fundamental differences compared with RNP**. Please refer to **R1 of Author Rebuttal** for more detailed comparisons.\n___\n\n**W2: Concern about the practicality of the proposed method.**\n\n**RW2:** Thank you for concerning about the practicality of TSBD. We would like to point out that the purpose of Table 3 in Section 4.3 is to compare the effectiveness of different reinitialization schemes, where a fixed neuron ratio $n\\%$ (e.g., 10%) is used for the three versions. Specifically, let's denote the  top-$n\\%$ selected neurons as $\\theta^n$. $V_1$ means that $\\theta^n$ are reinitialized thoroughly; $V_2$ means that only top-70% subweights of each neuron in $\\theta^n$ are reinitialized; $V_3$ means that the top-70% subweights of the whole $\\theta^n$ are reinitialized. Therefore, Table 3 in Section 4.3 indicates that some subweights in the selected neurons are important to the clean functionality, and thus we should keep them properly. The sensitivity analysis on neuron ratio is illustrated in **Section 4.4** and **Appendix G**, which prove that the hyperparameter $n$ is **insensitive but important** to the final performance. In a practical scenario, we can set $n$ freely from 10% to 70% (see **Figure 8 of Appendix G**). For a more detailed analysis of the importance of stage 1, please refer to the **R2 of Author Rebuttal**.\n___\n\n**W3: Suggestion for additional related work [1].**\n\n**RW3:** Thanks for providing us with this valuable information. We will add this paper (**BTI-DBF**) to the related work in the revised version. Here, we provide a brief version of *unlearning for backdoor defense* containing this paper as follows:\n\n> *Model unlearning* can be considered as an opposite process against learning, aiming to remove the impact of a training subset from a trained model [2]. In the field of backdoor defense, unlearning the possible poisoned data (i.e., *poison unlearning*) is an effective way to remove the learned backdoor. NC [3] and BTI-DBF [1] try to generate the possible poisoned data with either trigger inversion or poison-data generator; ABL [4] and D-BR [5] focus on filtering out the poisoned data from the training dataset according to their attributes during training; i-BAU [6] and SAU [7] assume the adversarial perturbation as a type of trigger and generate poisoned data with adversarial example. To avoid inducing bias, recent work tries to directly unlearn the available clean data (i.e., *clean unlearning*) for defense. RNP [8] finds that a clean-unlearned model can help expose the backdoor neurons for the subsequent pruning-mask learning. Different from the existing works that focus on utilizing the unlearning techniques, we fill the gap of exploring the *clean* and *poison unlearning* process on the backdoored model and provide insights from their correlation for defense.\n___\n\n[1] Towards reliable and efficient backdoor trigger inversion via decoupling benign features. ICLR 2024.\n\n[2] Machine unlearning. Symposium on Security and Privacy (SP) 2021.\n\n[3] Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. Symposium on Security and Privacy (SP) 2019.\n\n[4] Anti-backdoor learning: Training clean models on poisoned data. NeurIPS 2021.\n\n[5] Effective backdoor defense by exploiting sensitivity of poisoned samples. NeurIPS 2022.\n\n[6] Adversarial unlearning of backdoors via implicit hypergradient. ICLR 2022.\n\n[7] Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. NeurIPS 2023.\n\n[8] Reconstructive neuron pruning for backdoor defense. ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response
We sincerely thank all reviewers for their valuable time and constructive comments. 
___

**Q1: Systematic comparison with RNP [1].**

**R1:** We aim to address the concerns regarding the differences between our work and RNP. More precisely, we emphasize their differences from the perspectives of *technical details*, *motivation&insight*, and *experimental performance*, as follows:

 - **Different technical details.**
	- **The backdoor erasing techniques are different.** RNP utilizes *pruning* to erase the neurons permanently, while TSBD proposes *zero reinitialization* in the subweight level to modify the weights of neurons. By using the *zero reinitialization*, the modified neurons can be further repaired by the subsequent fine-tuning and better recover the clean functionality, which is infeasible in pruning.
	- **The goals are different for the second stage.** For RNP, as stated in [1], the purpose of *Filter Recovering* is to "*recover the clean features (features of the clean samples) erased by the previous unlearning step*", where a pruning mask is learned in this stage and backdoor erasing has not yet been conducted. However, for TSBD, our goal is to recover the clean functionality lost during erasing the backdoor effect by reinitializing the backdoor-related neurons, where fine-tuning is a commonly-used technique [2]. The lost of clean features comes from the *zero reinitialization* and has no direct relationship with unlearning.
	- **The target models are different in the second stage.** For RNP, the *Filter Recovering* is conducted based on the **unlearned model** from the previous stage, i.e., *Neuron Unlearning*. Differently, for TSBD, the unlearning stage is only to help find out the backdoor-related neurons, while the fine-tuning stage is based on the **reinitialized model** from the **original backdoored one**, not the unlearned model from the first stage.
	- **The subsequent operations are different after erasing backdoor.** The two stages in RNP are to finally find out and prune the backdoor neurons, and no further operation is conducted. For TSBD, after removing backdoor with *zero reinitialization* (as shown in Section 4.3, where the ASRs reduce to 0), a further fine-tuning is conducted to recover the sacrificed ACC. Moreover, we adopt a novel gradient norm regularization to enhance this process.
 - **Different motivations and insights.**
	-  **The motivations are different for conducting clean unlearning.** RNP aims to utilize the characteristics of a clean-unlearned model for mask learning and improving other techniques [1], i.e., exploring the utility of the unlearned model. In contrast, TSBD aims to explore the characteristics of the unlearning process, including the ones in different input data types and model types. The only reason for unlearning clean data is that it is an accessible data type for defense.
	- **The insights are from different perspectives.** For RNP, the authors state that "*the unlearned model tends to predict the backdoor label for all defense samples*" and it can be used to improve other defenses [1]. For TSBD, we emphasize the positive weight-change correlation of *clean* and *poison unlearning*, and we also uncover the neuron activeness of backdoored model. These insights are compensatory for a better understanding of backdoor learning. We believe that RNP and TSBD can both contribute to the community.
 - **Difference performances.**
	- **TSBD outperforms RNP in most cases.** For a fair comparison, we adopt the RNP method in the BackdoorBench framework and follow the basic experimental setup as other baselines; refer to **Appendix C** for more details. For all the main experiments illustrated in **Section 4.2**, **Appendix D and E**, TSBD outperforms RNP with the best average performance, e.g., for CIFAR-10, 97.09 (TSBD) > 82.83 (RNP) on DER; for Tiny ImageNet, 97.89 (TSBD) > 87.59 (RNP) on DER, etc.
	- **TSBD is more robust in clean data ratio than RNP.** Although RNP claims that only 1% clean data is needed for defense, it is not robust to the clean data ratio, which is validated in the **Appendix D.9 of [3]**. In contrast, TSBD is validated to be robust in the clean data ratio (see **Appendix I**). 

In conclusion, **our TSBD is a newly designed method with several fundamental differences compared with RNP**. Our paper provides several important insights to the research field of backdoor learning and the empirical results validate that TSBD is an effective defense method.
___

**Q2: Concern about the importance of stage 1.** 

**R2:** We aim to emphasize the importance of stage 1 for the final performance. As shown in **Figure 8 of Appendix G**, by reducing the neuron ratio to 1%, TSBD fails on several strong attacks, with ASR larger than 20%, which may indicate a bad performance without reinitialization. Here, we design a stage 2 only version, i.e., *Activeness-Aware Fine-Tuning* (**AaFT** for short), and compare it with the full version (TSBD) in **Table 1 (as follows)**. The results show that TSBD is more effective towards most attacks and **validates the importance of reinitialization for a successful defense**.

**Table 1: Comparison between **AaFT** (only stage 2) and **TSBD** (the full process)**
|Method| AaFT |AaFT|TSBD|TSBD|
|--|--|--|--|--|
| Attack | ACC |ASR|ACC|ASR|
| BadNets | 90.58 |**1.26**|90.72|1.31|
| Blended | 91.70 |20.52|91.61|**2.61**|
| Input-aware | 93.04 |**1.14**|93.06|1.94|
| LF | 92.07 |6.07|91.20|**2.64**|
| SIG | 90.01 |2.70|90.41|**1.27**|
| SSBA | 91.61 |31.97|91.57|**1.66**|
| Trojan | 91.71 |9.34|91.76|**5.06**|
| WaNet | 93.04 |1.21|93.26|**0.88**|
___

[1] Reconstructive Neuron Pruning for Backdoor Defense. ICML 2023.

[2] Fine-pruning: Defending against backdooring attacks on deep neural networks. RAID 2018.

[3] Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examples. NeurIPS 2023.

Meta-review: This paper aims to address the security threat posed by backdoor attacks in DNNs. The authors explore model unlearning from the perspective of weight changes and gradient norms, making two key observations: weight changes between poison and clean unlearning are positively correlated, and neurons in backdoored models are more active than those in clean models. Based on these observations, they propose a Two-Stage Backdoor Defense (TSBD) method, involving Neuron Weight Change-based Backdoor Reinitialization and Activeness-Aware Fine-Tuning. Extensive experiments demonstrate the superior performance of their method compared to state-of-the-art approaches. The reviewers agree that the paper provides interesting observations, and the paper will be benefitted by providing more clear unlearning process and analysis.
Decision: Accept (poster)