Title: Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Neural Network; AI security; Backdoor Defense
Abstract: The security threat of backdoor attacks is a central concern for deep neural networks (DNNs). Recently, without poisoned data, unlearning models with clean data and then learning a pruning mask have contributed to backdoor defense. Additionally, vanilla fine-tuning with those clean data can help recover the lost clean accuracy. However, the behavior of clean unlearning is still under-explored, and vanilla fine-tuning unintentionally induces back the backdoor effect. In this work, we first investigate model unlearning from the perspective of weight changes and gradient norms, and find two interesting observations in the backdoored model: 1) the weight changes between poison and clean unlearning are positively correlated, making it possible for us to identify the backdoored-related neurons without using poisoned data; 2) the neurons of the backdoored model are more active (*i.e.*, larger gradient norm) than those in the clean model, suggesting the need to suppress the gradient norm during fine-tuning. Then, we propose an effective two-stage defense method. In the first stage, an efficient *Neuron Weight Change (NWC)-based Backdoor Reinitialization* is proposed based on observation 1). In the second stage, based on observation 2), we design an *Activeness-Aware Fine-Tuning* to replace the vanilla fine-tuning. Extensive experiments, involving eight backdoor attacks on three benchmark datasets, demonstrate the superior performance of our proposed method compared to recent state-of-the-art backdoor defense approaches. The code is available at https://github.com/linweiii/TSBD.git.

Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness

Abstract

The security threat of backdoor attacks is a central concern for deep neural networks (DNNs). Recently, without poisoned data, unlearning models with clean data and then learning a pruning mask have contributed to backdoor defense. Additionally, vanilla fine-tuning with those clean data can help recover the lost clean accuracy. However, the behavior of clean unlearning is still under-explored, and vanilla fine-tuning unintentionally induces back the backdoor effect. In this work, we first investigate model unlearning from the perspective of weight changes and gradient norms, and find two interesting observations in the backdoored model: 1) the weight changes between poison and clean unlearning are positively correlated, making it possible for us to identify the backdoored-related neurons without using poisoned data; 2) the neurons of the backdoored model are more active ( i.e. , larger gradient norm) than those in the clean model, suggesting the need to suppress the gradient norm during fine-tuning. Then, we propose an effective two-stage defense method. In the first stage, an efficient Neuron Weight Change (NWC)-based Backdoor Reinitialization is proposed based on observation 1). In the second stage, based on observation 2), we design an Activeness-Aware Fine-Tuning to replace the vanilla fine-tuning. Extensive experiments, involving eight backdoor attacks on three benchmark datasets, demonstrate the superior performance of our proposed method compared to recent state-of-the-art backdoor defense approaches. The code is available at https://github.com/linweiii/TSBD.git.

1 Introduction

Over the past few years, deep neural networks (DNNs) have achieved surprising success in several realworld applications, such as face recognition [1-3], medical image processing [4, 5], and autonomous driving [6, 7], etc . However, DNNs are susceptible to malicious attacks that can compromise their security and reliability. One typical example is the backdoor attack [8-11], where the adversary maliciously manipulates the training dataset or training process to produce a backdoored model, which performs normally on clean data while predicting any sample with a particular trigger pattern to a pre-defined target label. In this work, we focus on the post-training defense scenario where, given a backdoored model and a small set of clean training samples, one aims to mitigate the backdoor effect while maintaining the performance on clean data, thereby obtaining a benign model.

Up to now, several important methods have been developed for backdoor defense [14-18]. One promising approach is poison unlearning , which involves updating a backdoored model by unlearning

Observation 1

. Weight changes between poison and clean unlearning are positively correlated.

Observation 2

. Neurons of backdoored model are more active than those in clean model.

from poisoned data. This technique has been utilized in various backdoor defenses such as ABL [14], D-BR [19], Neural Cleanse (NC) [20], and i-BAU [21], etc . To avoid approximating poisoned data, another approach called clean unlearning was conducted by RNP [22]. This technique only uses clean data for unlearning and then prunes the backdoored model, which has been proven to be effective. Through relevant experiments, we find an interesting connection between poison unlearning and clean unlearning, as illustrated in Observation 1 of Figure 1. Specifically, by calculating the weight changes of each neuron during the two unlearning processes on the backdoored models 2 , we find that they exhibit a strong positive correlation, i.e. , the neurons exhibiting significant weight changes during clean unlearning also tend to play crucial roles in poison unlearning, indicating a stronger association with backdoor-related activities. Moreover, we further investigate the backdoor activeness during learning processes 3 , i.e. , comparing the average gradient norm for each neuron in both the backdoored and clean models. The results are shown in Observation 2 of Figure 1, revealing that neurons in the backdoored model are always more active compared to those in the clean model.

Inspired by the above two observations regarding the backdoored model , we propose T woS tage B ackdoor D efense ( TSBD ), consisting of stage 1) Neuron Weight Change-based Backdoor Reinitialization and stage 2) Activeness-Aware Fine-tuning . In the first stage, we first conduct clean unlearning on the backdoored model, followed by the neuron weight change calculation, where both the changes of each subweight 4 and neuron are recorded. Then, we conduct zero reinitialization to mitigate the backdoor effect by reinitializing the most-changed subweights among the topn % most-changed neurons as 0 in the original backdoored model. In the second stage, we adopt activeness-aware fine-tuning with gradient-norm regulation to recover clean accuracy and suppress the reactivation of the backdoor effect. Extensive experiments demonstrate the superior defense performance of the proposed method compared to state-of-the-art (SOTA) backdoor defense methods.

To summarize, our main contributions are three-fold. (1) Novel Insight: Weare the first to uncover the strong positive correlation between neuron weight changes in clean unlearning and poison unlearning. We also reveal the high backdoor activeness in the backdoored model during the learning process. (2) Effective Defense Method: We further develop an effective two-stage defense method based

on unlearning weight changes and backdoor activeness, considering both backdoor mitigation and clean-accuracy recovery, respectively. (3) SOTA Performance: Experimental results and analysis show that our proposed method achieves SOTA performance in backdoor defense.

Backdoored Model

Clean Dataset

Reinitialized Model

2) Activeness-aware

Fine-tuning

Reinitialization

Weight Mask

a. Clean Unlearning

Neuron Weight Changes

Subweight Changes

b. Neuron Weight Change Calculation

Clean Model

Fine-tuning with Gradient-Norm

Regulation

2 Related Work

2.1 Backdoor Attack

In the literature, various backdoor attacks on DNNs have been proposed, which can be categorized into data poisoning attacks and training-controllable attacks . BadNets [8] is one of the earliest data poisoning attacks in this field. In this attack, a small proportion of the original data is selected and patched with a pre-defined pattern, known as a trigger . The labels of these patched data points are then modified to a target label. The mixed dataset, containing both clean and poisoned data, is used to train the DNNs, resulting in the implantation of the backdoor. Under a similar procedure, Blended [25] was proposed as a stronger attack by blending an entire pre-defined image into the original clean data with controllable transparency. Recently, more advanced and stealthy attacks have been proposed to enhance the trigger, such as SIG [26], label consistent attacks [27, 28], SSBA [9], etc . Another category is training-controllable attacks [23, 24, 29-31], where the attackers design triggers with permission to control the training process. Two significant examples are WaNet [24] and Input-aware [23], which generate unique triggers for different input data by incorporating an injection function into the model training process. This approach makes these attacks more difficult to detect compared to previous attacks with fixed triggers.

2.2 Backdoor Defense

According to the different stages of model training, backdoor defense methods can be classified into two types: training-stage defenses and post-training defenses .

Training-stage Defenses. In training-stage defenses, defenders have access to a mixed training dataset containing both clean data and poisoned data with triggers. ABL [14] discovers that the loss-dropping speed of poisoned data during the early stages of model training is faster, and thus isolates them for poison unlearning. DBD [32] splits the training process into three steps to separate the training of feature extraction from that of the subsequent classifier to evade the learning of triggerlabel correlation. Similarly, D-ST/D-BR [19] observes that the transformations of poisoned-data feature representations are more sensitive than clean ones, and thus proposes to modularize the training process.

Post-training Defenses. In post-training defenses [33-36], defenders aim to erase the backdoor effect in the learned DNNs using a small portion of clean data. FP [37] is one of the earliest defense methods, which observes that poisoned data and clean data activate different neurons in a backdoored DNN, and thus keeps pruning the less-activated neurons in response to clean data until a significant

drop in accuracy occurs. After that, vanilla fine-tuning is employed to recover the lost clean accuracy. Using the pruning strategy [38-40], ANP [41] observes that the backdoor-related neurons exhibit higher sensitivity to adversarial perturbations compared to others, and thus trains a pruning mask using minimax optimization. Continuing along this line, AWM [42] and RNP [22] use a similar mask training process with main modifications in neuron perturbations to data perturbations and clean unlearning , respectively. Different strategies are also proposed for defense. For example, NC [20] proposes to recover the trigger before the subsequent backdoor removal. NAD [43], for the first time, adopts model distillation to guide the learning of a benign student model. Additionally, employing unlearning techniques, SAU [44] treats backdoor triggers as a form of adversarial perturbation, and generates poisoned data through optimization on clean data, which are then used in poison unlearning .

Unlearning for Backdoor Defense. Model unlearning can be considered as an opposite process against learning, aiming to remove the impact of a training subset from a trained model [45]. In the field of backdoor defense, unlearning the possible poisoned data ( i.e. , poison unlearning ) is an effective way to remove the learned backdoor. NC [20] and BTI-DBF [46] try to generate the possible poisoned data with either trigger inversion or poison-data generator; ABL [14] and D-BR [19] focus on filtering out the poisoned data from the training dataset according to their attributes during training; i-BAU [21] and SAU [47] assume the adversarial perturbation as a type of trigger and generate poisoned data with adversarial example. To avoid inducing bias, recent work tries to directly unlearn the available clean data ( i.e. , clean unlearning ) for defense. RNP [22] finds that a clean-unlearned model can help expose the backdoor neurons for the subsequent pruning-mask learning.

However, there exist some limitations among those techniques, e.g. , clean unlearning is still underexplored, and the vanilla fine-tuning unintentionally increases the attack success rate. In this paper, we propose a comprehensive two-stage defense method breaking through the two limitations.

3 Methods

3.1 Problem Formulation

Threat Model. We assume that the attacker has full access to the training data. Their goal is to poison a portion of the dataset by injecting triggers into the data so that the trained model misclassifies the poisoned data to the target class while still performing normally on clean data. The poisoning ratio ( e.g. , 10%) is used to depict the proportion of poisoned data within the entire dataset. We denote the parameters of the backdoored model as θ bd = { θ ( l ) bd } 1 ≤ l ≤ L satisfying θ ( l ) bd ∈ R K ( l ) × I ( l ) , where K = { K ( l ) } 1 ≤ l ≤ L and I = { I ( l ) } 1 ≤ l ≤ L represent the neuron numbers and learnable subweight numbers, respectively. Specifically, for the l th ∈ { 1 , . . . , L } layer, there are K ( l ) neurons in total and I ( l ) subweights for each neuron.

Defense Setting. The defender's goal is to remove the backdoor effect, which causes poisoned data to be misclassified to the target class, from the backdoored model while minimizing the impact on the prediction accuracy for clean data. Following the previous defense setting [37, 41], we assume that the defender knows nothing about the poisoned data and possesses only 5% of the total dataset as clean data, denoted as D c .

3.2 Neuron Weight Change & Suggestions Given by the Two Observations

In this subsection, we provide more details on the unlearning formulation and offer suggestions based on the two observed observations.

Model Unlearning. Model Unlearning can be defined as the reverse process of model training [22], which involves maximizing the loss value on a given dataset. Given a DNN model f parameterized as θ and a dataset D for unlearning, the maximization problem can be formulated as:

where ( x , y ) ∈ D represents the images and their corresponding labels, and L denotes the loss function used in this task, e.g. , cross-entropy loss.

Intuitively, by maximizing the loss expectation, the unlearned model, parameterized as θ ul , is prone to fail at the task specified in D . In this paper, we term the process as clean unlearning when all

0.0

0.5

1.0

1.5

2.0

NWC of Clean Unlearning

0.1

0.2

0.3

0.4

0.5

Activations

Comparison of Activations on Clean Model

Clean Activations

Poison Activations

0.0

0.5

1.0

1.5

2.0

2.5

3.0

NWC of Clean Unlearning

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Activations

Comparison of Activations on Backdoored Model

Clean Activations

Poison Activations

0.0

0.5

1.0

1.5

2.0

2.5

3.0

NWC of Clean Unlearning

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

Activations Change

Comparison of Activation Changes

Poison-Activation Rise during Poison Unlearning

Poison-Activation Rise during Clean Unlearning

Clean-Activation Rise during Poison Unlearning

Clean-Activation Rise during Clean Unlearning

(a) Original clean model

(b) Original backdoored model

(c) Backdoored-model unlearning

the data in D are clean, denoted as D c . On the other hand, poison unlearning refers to the scenario where all the data in D are poisoned with a trigger. By default, both clean and poison unlearning are terminated when the model performs poorly on the corresponding tasks, such as achieving only 10% clean accuracy or attack success rate.

Neuron Weight Change. To comprehensively quantify the weight changes of a neuron during the entire unlearning process, we define the Neuron Weight Change (NWC), where the L 1 norm is calculated on every neuron's weight differences. The NWC for the k th ∈ { 1 , . . . , K ( l ) } neuron in layer l ∈ { 1 , . . . , L } can be formulated as:

where ∑ I ( l ) i =0 ∥·∥ 1 is to calculate the L 1 norm for the differences on a neuron with totally I ( l ) subweights, θ ( l ) ki ul and θ ( l ) ki bd denote the i th ∈ { 1 , . . . , I ( l ) } subweights of k th neuron after and before the entire unlearning process, respectively. A larger NWC k indicates more significant changes occurring in neuron k during the unlearning process. Similarly, in Equation (2), the term ∥ θ ( l ) ki ul -θ ( l ) ki bd ∥ 1 represents the changes in the i th subweight of neuron k in layer l , i.e. , defined as Subweight Change .

Suggestions Given by the Two Observations. As demonstrated in Section 1, we have two interesting observations regarding the backdoored model. Observation 1 shows that the neurons exhibiting significant weight changes during clean unlearning also tend to play crucial roles in poison unlearning. It suggests that we can employ clean unlearning to identify and eliminate backdoor-related neurons using NWC, at the expense of reducing clean accuracy. On the other hand, Observation 2 reveals that neurons in the backdoored model are always more active compared to those in the clean model. It suggests that we should suppress the gradient norm during the learning process if we want to recover it to a clean model. These two suggestions act as the main supports to our proposed TSBD.

3.3 Further Investigations & Insights

Here, we offer insights from the perspective of neuron activations, trying to answer two important questions: [Q1] What causes the clean unlearning NWCs to exhibit a positive correlation with those in poison unlearning, and [Q2] What motivates the neurons more active in the backdoored model.

Neuron Activations & Activation Rise. The neuron activation is determined by computing the average value of all inputs to the specific neuron, e.g. , h ( l ) k ≈ σ ( θ ( l ) k h ( l -1) ) for simplicity, where σ ( · ) is the activation function. In line with the terminology used in FP [37], clean activation denotes the scenario where all the input samples are clean while poison activation refers to the presence of poisoned inputs. To better observe the changes in activation during unlearning, we calculate the activation rise from the original model to the unlearned model, i.e. , ∆ h ( l ) k = h ( l ) k ul -h ( l ) k bd . Apositive value indicates an increase in activation, while a negative value signifies a decrease.

Relationship between NWC and Activation Change. Considering that a backdoored model has learned two tasks from the clean and poisoned data [14], the main influence of NWC on a neuron can be roughly attributed to its activation change on both clean and poisoned inputs. For neuron k in layer l , we can formulate it as NWC ( l ) k ∝ | ∆ h ( l ) k c | + | ∆ h ( l ) k p | , where ∆ h ( l ) k c and ∆ h ( l ) k p represent the activation rise on clean and poisoned inputs, respectively.

Figure 3 illustrates the clean and poison activations in (a) the original clean model, (b) the original backdoored model, and (c) the backdoored-model unlearning. We now try to answer the above two questions from these observations. [A1] We can observe that poison activations are the main factors affected during both clean unlearning (increase) and poison unlearning (decrease), while clean activations are only slightly influenced (see Figure 3 (c)), i.e. , NWC ( l ) k ↑ →| ∆ h ( l ) k c | ≈ + | ∆ h ( l ) k p | ↑ . Also, the growing NWC during clean unlearning can indicate larger poison and clean activations (where h ( l ) k p > h ( l ) k c ) to some extent (see Figure 3 (b)). Thus, we deduce that the co-function of clean and poison activations dominates the performance on both tasks, while the higher values of poison activation in the backdoored model make it an easier target for modification. In this case, the neurons with higher poison activations tend to decrease their values during poison unlearning, thereby reducing the attack success rate. Conversely, during clean unlearning, these neurons increase poison activations, which suppresses the function of clean activations and reduces clean accuracy. [A2] Similarly, the significantly lower values of mixed clean and poison activations (maximum: 0.5676) on the clean model (see Figure 3 (a)) indicate that it is less active compared to the backdoored model (maximum: 1.7053), where a similar pattern can also be seen on the bottom left of Figure 3 (b).

3.4 Two-Stage Backdoor Defense Framework

Based on the above observations, we now propose a defense framework incorporating Neuron Weight Change-based Backdoor Reinitialization (including Clean Unlearning , Neuron Weight Change Calculation and Zero Reinitialization ), and Activeness-aware Fine-tuning . The detailed defense process is illustrated in Figure 2 and Algorithm 1 (found in Appendix A).

Stage 1) Neuron Weight Change-based Backdoor Reinitialization. We aim to mitigate the backdoor effect with acceptable clean-accuracy sacrificed in this stage. [a. Clean Unlearning.] To identify the backdoor-related neurons, we first conduct a full clean unlearning using the available clean data D c on the backdoored model. [b. Neuron Weight Change Calculation.] Then, we record the subweight changes and calculate the NWC for each neuron as described in Section 3.2. The resulting sorted order of neurons reflects the backdoor strength. [c. Zero Reinitialization.] After that, we can now eliminate the backdoor effect through zero reinitialization. Based on the NWC neuron order, we identify the topn % neurons as strongly backdoor-related. As suggested in Section 3.3, high-NWC neurons may also contribute to clean accuracy to some extent. Therefore, we further choose to reinitialize the subweights of the most-changing m % among the selected neurons to zero in the backdoored model, while leaving the others unchanged. The reinitialized model parameter is denoted as ˆ θ .

Stage 2) Activeness-Aware Fine-tuning. To further repair the reinitialized subweights and avoid recovering the backdoor effect again, we conduct activeness-aware fine-tuning on the reinitialized model ( ˆ θ ) using the clean dataset, D c . This involves incorporating gradient-norm regulation into the original loss function, such as the cross-entropy loss L ce , to penalize high gradient values. This regulation serves to suppress neuron activity during fine-tuning. The final loss function is:

where ∥∇ ˆ θ L ce ( ˆ θ ) ∥ 2 represents the L 2 norm of gradients, and λ is the penalty coefficient controlling its impact. Hence, the objective of fine-tuning is to minimize the loss function L ft ( ˆ θ ) using the available clean data D c :

During practical optimization for computational efficiency, we adopt the approximation scheme in [48], which can be formulated as:

Here, r is used for appropriating the Hessian multiplication operation, and α = λ r is the balance coefficient . Due to the space limit, the detailed derivation and algorithm are provided in Appendix B. After the activeness-aware fine-tuning stage, we can obtain a repaired clean model, which demonstrates outstanding performance in the experiments.

4 Experiments

4.1 Experimental Setup

Attack Setup. We consider 8 SOTA backdoor attacks in the main experiment. There are BadNets [8], Blended [25], Input-aware [23], LF [49], SIG [26], SSBA [9], Trojan [50] and WaNet [24]. For a fair comparison, we follow the default attack configuration as in BackdoorBench [51], including the trigger pattern, trigger size, the target label ( i.e. , the 0 th label), etc . We choose 10% poisoning ratio as the default setting. To fully evaluate the effectiveness of our proposed framework, all the attacks are implemented on three benchmark datasets, i.e. , CIFAR-10 [13], Tiny ImageNet [52], and GTSRB [53], over two popular DNNs, i.e. , PreAct-ResNet18 [12] and VGG19-BN [54]. In particular, SIG is only applied to CIFAR-10 since it cannot reach a 10% poisoning ratio on Tiny ImageNet and GTSRB. Besides, we also evaluate the defense methods under 5% and 1% poisoning ratios to verify the robustness of our proposed framework. Due to the space limit, we only exhibit parts of the main results in this section. More implementation details can be found in Appendix C.

Defense Setup. We compare the proposed framework with 8 SOTA backdoor defense methods: Fine-tuning (FT), Fine-pruning (FP) [37], NAD [43], NC [20], ANP [41], CLP [38], i-BAU [21], and RNP [22]. We use the recommended configurations in BackdoorBench [51]. Since the defense settings for training-stage defenses are different [14, 32], we only compare the post-training defenses, where 5% clean data can be accessed following the previous settings [41]. For TSBD, we set the learning rates for clean unlearning to 10 -4 and fine-tuning to 10 -2 . The default neuron ratio n and weight ratio m are set to 0.15 and 0.7, respectively. We follow the suggested settings of hyper-parameters r = 0 . 05 and α = 0 . 7 for fine-tuning [48].

Evaluation Metrics. We use three metrics to evaluate the performance of each defense method: Accuracy on clean data ( ACC ), Attack Success Rate ( ASR ), and Defense Effectiveness Rating ( DER ) [55]. Specifically, ACC measures the proportion of clean data correctly predicted; ASR measures the proportion of poisoned data misclassified to the target label; DER ∈ [0 , 1] evaluates the cost of ACC for reducing ASR, which is defined as: DER = [max(0 , ∆ASR) -max(0 , ∆ACC)+1] / 2 , where ∆ASR and ∆ACC are the drop in ASR and ACC after applying defense on the backdoored model, respectively. Larger ACC, DER, and smaller ASR are desired for a successful defense. Note that in the following result tables, '-' indicates that the value is inapplicable. The boldface values indicate the best performance and the underline values denote the second-best result.

4.2 Main Results

We validate the effectiveness of our proposed framework on 8 SOTA backdoor attacks and compare it with 8 defenses. In this section, we present the main results on CIFAR-10 and Tiny ImageNet with

a 10% poisoning ratio on PreAct-ResNet18 for illustration, which is shown in Table 1 and Table 2. More results on GTSRB and VGG19-BN can be found in Appendix D and E, respectively.

Results on CIFAR-10. Table 1 shows the results on CIFAR-10. Results show that our TSBD outperforms all the other SOTA defenses on the average of ASR (2.18%) and DER (97.09%), as well as a promising ACC (91.70%) higher than the original 'No Defense' models (91.34%), which indicates its effectiveness in removing the backdoor effect with the least cost. Though most defenses fail in strong attacks Blended, LF, or SSBA, e.g. , FT, FP, NAD, ANP, CLP, i-BAU, and RNP, our proposed TSBD success with the best ASR and DER on Blended and second best ASR and DER on LF and SSBA. FT and NAD perform similarly on each attack with a promising ACC, but they also fail on WaNet with a high ASR except for the mentioned strong attacks. FP and ANP perform well in ACC among all the defenses, while the defense performances on ASR and DER are unstable, which may be due to the unsuccessful backdoor-related neuron locating. CLP fails on most of the attacks with high ASR and low DER, which may be due to the structure constraint of computing channel Lipschitz only on the convolutional-batch normalization layer combination. i-BAU performs the second best on average ASR and DER, with failures on three attacks. TSBD outperforms RNP on almost all performances, which indicates that using clean unlearning with NWC is more effective than the unlearn-recovery process in RNP.

Results on Tiny ImageNet. Table 2 presents the results on Tiny ImageNet with PreAct-ResNet18. We observe that most defenses also fail on Blended, SSBA, and LF with high ASR. Similar performances of other attacks are shown on most of the defenses compared to CIFAR-10, where FT and FP also perform well in ACC and CLP fails on most attacks. Although i-BAU can successfully defend against most of the attacks in CIFAR-10, it fails on all attacks here, indicating that its adversarial training fails with large classification categories. For RNP, though it performs well in ASR on almost all attacks, the ACC is sacrificed too much to be unacceptable, indicating an unbalanced defense performance. In comparison, our TSBD can achieve SOTA on the average of DER, and perform second best on the average of ASR, which validates its superior defense performance.

4.3 Ablation Studies

Effectiveness of NWC order for Backdoor Strength. To verify the effectiveness of employing clean-unlearning NWC order in gauging the backdoor strength, we borrow the Trigger-activated Change (TAC) [38] order as the ground truth and compare the neuron coverage ratio on TAC under different proportions, i.e. , measuring the overlap of the selected neurons on both metrics. Specifically, TAC measures the change in neuron activation before and after the input image is attached with a trigger, where the larger value indicates a stronger backdoor effect. We select the following SOTA metrics for comparison: 1) the average neuron activations in FP [37]; 2) the channel Lipschitz in CLP [38]; 2) the perturb-recovery learned mask in ANP [41]; 3) the unlearn-recovery learned mask in RNP [22]. The result is illustrated in Figure 4, where the x-axis represents the

0

10

20

30

40

50

60

0

10

20

30

40

50

60

70

Coverage Ratio on TAC (%)

FP

ANP

RNP

NWC

(reinitializing/pruning) neuron ratio and the y-axis represents the neuron coverage ratio on TAC. The higher values on the y-axis indicate a better matching of the current metric and the TAC metric,

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

BadNets

ACC

ASR

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

Blended

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

BadNets

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

Blended

ACC

ASR

i.e. , more backdoor-related neurons are chosen. We can observe that NWC is always the best under different neuron ratios compared to others. It validates the effectiveness of using NWC order as the metric for neuron reinitialization/pruning. Moreover, we also test the performance of using NWC in FP to show its superiority in assisting other defenses. The results can be found in Appendix F.

Effectiveness of Zero Reinitialization on Sub-

weight. We compare different combinations of reinitializing neurons and weights to validate the effectiveness of zero reinitialization on the selected subweights. Specifically, three versions are designed for comparison: (1) V 1 : all weights on the selected top-NWC neurons are reinitialized to zero. (2) V 2 : m % (70% is used as de-

fault) of the top weights on each selected top-NWC neuron are reinitialized to zero. (3) V 3 (Ours): our final version, where m % of the top weights among all selected top-NWC neurons are reinitialized to zero. As the ACC and ASR drop monotonously with more neurons selected to be reinitialized, we record the results when ASR first drops to almost zero ( i.e. , lower equal than 0.05%). Table 3 shows the performances on four attacks. It validates that reinitializing the top weights among the selected neurons may help alleviate the hurt of clean accuracy. The failure of V 2 may be blamed on the different backdoor strength of each neuron, i.e. , some strong backdoor-related neurons fail to be removed thoroughly and thus more neurons are reinitialized to reach zero ASR.

Effectiveness of Gradient-norm Regulation

in Fine-tuning. To validate the essential role of gradient-norm regulation in the fine-tuning stage, we compare the performance on 'no finetuning' (No-FT), 'vanilla fine-tuning' (VanillaFT), and our 'activeness-aware fine-tuning' (AaFT). For a fair comparison, we follow the default settings and choose to fine-tune the reinitialized

model after the second stage. The results are shown in Table 4. It shows that fine-tuning is effective in improving clean accuracy, which is sacrificed to erase the backdoored effect in reinitialization, though it is prone to bring back some extent of ASR. Furthermore, compared to vanilla fine-tuning, our implemented activeness-aware fine-tuning can suppress the rise of ASR as well as improve ACC to the same level.

4.4 Further Analysis

Performance on Different Neuron Ratio and Weight Ratio.

To test the sensitivity of our method towards hyper-parameter tuning, we record the performance of TSBD under a wide range of neuron ratio ( n % ) and weight ratio ( m % ) tuning. The experiments are conducted on CIFAR-10 with a 10% poisoning ratio on PreAct-ResNet18. The neuron ratios are tuned from 1% to 90%, and weight ratios are tuned from 40% to 80%. The results of BadNets and Blended attacks are depicted in Figure 5. It shows that TSBD is insensitive to both neuron ratio and weight ratio, where the ASR is kept at a very low level, while ACC is maintained at a top level under a wide range of

tuning. The high consistency in the performances may come from the promising recovery ability of the fine-tuning stage. More results on other attacks are shown in the Appendix G.

Performance on Different r and α for Activeness-Aware Fine-tuning. To test the hyper-parameters sensitivity for stage 2, i.e. , r and α in Activeness-Aware Fine-tuning, we follow the tuning range in [48] under our experimental settings. The results are shown in Teble 5, which are obtained from a BadNets-attacked PreAct-ResNet18. We observe that the performance is insensitive (Changes <2% in ACC and <1% in ASR) across different hyper-parameter settings, maintaining a high level of performance.

Performance on Different Poisoning Ratio. We further investigate the performance of TSBD on different poisoning ratios, e.g. , 10%, 5%, and 1%. Note that a larger poisoning ratio represents a stronger attack mode. We test the performance with six attacks on these three ratios. Figure 6 shows the performances of ACC and ASR. We can observe that TSBD successfully defends all the attacks on 10% and 5% with a low ASR and a high ACC while performing less effectively on 1%. A possible reason is that the unlearning weight changes of backdoor neurons are less obvious in the weak attack mode compared to the strong attack mode with 10% or 5% poisoning ratios.

More Experiments and Analysis. Due to the space limit, we postpone the detailed discussion of the clean data ratio and the fine-tuning learning rate to Appendix H and Appendix I, respectively. We also evaluate the defense

BadNets

Blended

Input-aware

LF

SIG

SSBA

0

20

40

60

80

100

Accuracy (%)

ACC: 10% Poisoning Rate

ACC: 5% Poisoning Rate

ACC: 1% Poisoning Rate

ASR: 10% Poisoning Rate

ASR: 5% Poisoning Rate

ASR: 1% Poisoning Rate

performance on the clean model in Appendix J and on the ViT in Appendix K. Further, we provide the computational overhead in terms of runtime in Appendix L.

5 Conclusion

In this work, we propose an effective two-stage backdoor defense method, TSBD, to eliminate the backdoor effect in DNNs. Our research reveals two important observations regarding the backdoored models to support our method. First, there is a positive correlation between weight changes during poison and clean unlearning in backdoored models. This finding enables us to identify and eliminate backdoor-related neurons through clean unlearning and zero reinitialization. Second, neurons in backdoored models are more active compared to those in clean models , which suggests regulating the gradient norm during fine-tuning. Furthermore, we also provide insights into these two observations from the perspective of neuron activations, which may be a valuable contribution to the field of backdoor defense. Extensive experiments demonstrate the superiority of our method over recent defenses. One current challenge as well as promising future work involves defending against backdoor attacks without any accessible clean data. The data generation techniques and data-free techniques may be the potential solutions.

References

Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In CVPR , 2014.

Divyarajsinh N Parmar and Brijesh B Mehta. Face recognition methods applications. arXiv preprint arXiv:1403.0485 , 2014.

Ratnawati Ibrahim and Zalhan Mohd Zin. Study of automated face recognition system for office door access control application. In ICCSN , 2011.

Yixiong Chen, Chunhui Zhang, Li Liu, Cheng Feng, Changfeng Dong, Yongfang Luo, and Xiang Wan. Uscl: pretraining deep ultrasound image diagnosis model through video contrastive representation learning. In MICCAI , 2021.

Yixiong Chen, Li Liu, Jingxian Li, Hua Jiang, Chris Ding, and Zongwei Zhou. Metalr: Meta-tuning of learning rates for transfer learning in medical imaging. In MICCAI , 2023.

Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access , 2020.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020.

Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access , 2019.

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV , 2021.

Baoyuan Wu, Li Liu, Zihao Zhu, Qingshan Liu, Zhaofeng He, and Siwei Lyu. Adversarial machine learning: A systematic survey of backdoor attack, weight attack and adversarial example. arXiv preprint arXiv:2302.09457 , 2023.

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. In NeurIPS , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV , 2016.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurIPS , 2021.

Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv:1811.03728 , 2018.

Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In NeurIPS , 2018.

Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards a proactive { ML } approach for detecting backdoor poison samples. In USENIX Security , 2023.

Min Liu, Alberto Sangiovanni-Vincentelli, and Xiangyu Yue. Beating backdoor attack at its own game. In ICCV , 2023.

Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In NeurIPS , 2022.

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In SP , 2019.

Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations , 2021.

Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang. Reconstructive neuron pruning for backdoor defense. arXiv preprint arXiv:2305.14876 , 2023.

Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In NeurIPS , 2020.

Anh Nguyen and Anh Tran. Wanet-imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10369 , 2021.

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 , 2017.

Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In ICIP , 2019.

Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NeurIPS , 2018.

Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean-label backdoor attacks on video recognition models. In CVPR , 2020.

Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In ICCV , 2021.

Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In USENIX Security , 2021.

Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. In NeurIPS , 2021.

Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR , 2022.

Junfeng Guo, Ang Li, and Cong Liu. Aeva: Black-box backdoor detection using adversarial extreme value analysis. arXiv preprint arXiv:2110.14880 , 2021.

Wei Jiang, Xiangyu Wen, Jinyu Zhan, Xupeng Wang, Ziwei Song, and Chen Bian. Critical path-based backdoor detection for deep neural networks. TNNLS , 2022.

Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In CVPR , 2020.

Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In SP , 2021.

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In RAID , 2018.

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In ECCV , 2022.

Tianlong Chen, Zhenyu Zhang, Yihua Zhang, Shiyu Chang, Sijia Liu, and Zhangyang Wang. Quarantine: Sparsity can uncover the trojan attack trigger for free. In CVPR , 2022.

Jiyang Guan, Zhuozhuo Tu, Ran He, and Dacheng Tao. Few-shot backdoor defense using shapley estimation. In CVPR , 2022.

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurIPS , 2021.

Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. In NeurIPS , 2022.

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930 , 2021.

Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. Advances in Neural Information Processing Systems , 36, 2024.

Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 141-159. IEEE, 2021.

Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren. Towards reliable and efficient backdoor trigger inversion via decoupling benign features. In The Twelfth International Conference on Learning Representations , 2024.

Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. In NeurIPS , 2023.

Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning , pages 26982-26992. PMLR, 2022.

Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV , 2021.

Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS Symposium , 2018.

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. In NeurIPS Datasets and Benchmarks Track , 2022.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 2015.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN , 2011.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In ICCV , 2023.

Appendix Outline

This appendix is organized as follows:

In Section A, we detail the algorithm of TSBD.

In Section B, we detail the approximation process of the fine-tuning optimizations.

In Section C, we introduce the implementation details, including the details of datasets, models, attacks, and defenses with our proposed method.

In Section D, we compare the defense results on the GTSRB dataset.

In Section E, we compare the defense results on VGG19-BN structure.

In Section F, we show the effectiveness of using NWC in other defense.

In Section G, we show the comprehensive results with different neuron ratios and weight ratios.

In Section H, we discuss different clean data ratios on the defense performance.

In Section I, we discuss different fine-tuning learning rates on the defense performance.

In Section J, we evaluate the influence of using our method on the clean model.

In Section K, we test the performance of scaled-up experiments on the ViT model.

In Section L, we show the computational overhead in terms of runtime.

A Detailed Algorithm of TSBD

To clearly illustrate our proposed method, we provide the detailed algorithm of the entire process of TSBD, which is shown in Algorithm 1.

B Details of the Approximated Fine-Tuning Optimizations

As illustrated in Section 3.4, our proposed Activaness-aware fine-tuning involves calculating an additional gradient-norm regulation in the loss function, making it computationally inefficient. Therefore, we adopt the approximation scheme from [48] during practical optimization. Here, we provide the details of the approximated fine-tuning optimization. The approximation deviation is shown in the following. Specifically, for each step of the gradient calculation, we formulate it as:

To avoid the Hessian computation, the second term is further approximated through an additional parameter update:

Based on the approximation, the practical fine-tuning process is illustrated in Algorithm 2.

C More Implementation Details

We further illustrate the implementations here, covering the details of datasets, models, attacks, and defenses.

Dataset Details. The experiments are conducted on CIFAR-10 [13], Tiny ImageNet [52], and GTSRB [53].

BadNets

Blended

Input-aware

LF

SIG

SSBA

Trojan

WaNet

CIFAR-10. It contains 60,000 32 × 32 colored images with 10 classes. Each class owns 6,000 images, consisting of 5,000 for training and 1,000 for testing.

Tiny ImageNet. It is a subset of the full ImageNet, consisting of 100,000 training data and 10,000 testing data. There are 200 classes in total and 500 images per class for training. All images are 64 × 64 with color.

GTSRB. GTSRB (German Traffic Sign Recognition Benchmark) contains 39,209 images for training and 12,630 images for testing with 43 classes. All images are 32 × 32 colored images.

Models. We choose PreAct-ResNet18 [12] and VGG19-BN [54] as the target models to conduct attacks and defenses following the default configurations in BackdoorBench [51]. Both of them contain convolutional layers and batch normalization layers, which can be implemented with all kinds of defense methods, e.g. , FP [37] for the last convolutional layer and ANP [41] for the batch normalization layers. The extensive experiments on ablation study and further analysis are conducted on PreAct-ResNet18 by default.

Attack Details. We conduct 8 SOTA attacks for comprehensive testing, consisting of BadNets [8], Blended [25], Input-aware [23], LF [49], SIG [26], SSBA [9], Trojan [50] and WaNet [24]. All the attacks follow BackdoorBench's default configurations. Figure 7 shows all 8 attack triggers with the same example of CIFAR-10. Specifically, for BadNets, a 3 × 3 white square is patched at the bottom-right corner of the images for CIFAR-10 and GTSRB, and a 6 × 6 white square is for Tiny ImageNet. For Blended, a Hello-Ketty image is blended in the images with a 0.2 transparent ratio. We choose the 10% poisoning ratio and 0 th label as the default setting to conduct attacks and test all defenses following the previous works [55, 47]. 5% and 1% poisoning ratios are conducted only for testing our proposed method.

Defense Details. We conduct 8 SOTA defenses for a comprehensive comparison, containing Finetuning (FT), Fine-pruning (FP) [37], NAD [43], NC [20], ANP [41], CLP [38], i-BAU [21], and RNP [22]. The defenses also follow the BackdoorBench's default configurations. Note that we compare only the post-training defenses with 5% benign data provided. The learning rate for all methods is set to 10 -2 , the batch size is set to 256. For RNP, the clean data ratio is set to 0.5% since we found that it failed to defend well under the 5% setting. For our proposed method, we set the default learning rates for unlearning as 10 -4 and fine-tuning as 10 -2 . The unlearning is stopped when clean accuracy drops to or below 10%. The default fine-tuning epoch is set to 20. The default neuron ratio n and weight ratio m are set to 0.15 and 0.7, respectively. We follow the suggested settings of hyper-parameters r = 0 . 05 and α = 0 . 7 for fine-tuning [48]. Other settings are set following the default BackdoorBench configuration.

All experiments are conducted on a server with GPU RTX 3090 and CPU AMD EPYC 7543 32-Core Processor. These experiments were successfully executed using less than 24G of memory on a single GPU card.

D Evaluations on GTSRB dataset

We validate the effectiveness of our proposed method in the dataset GTSRB other than CIFAR-10 and Tiny ImageNet. Table 6 shows the corresponding performance on PreAct-ResNet18 with 10% poisoning ratio and 5% clean data ratio. We can observe that TSBD performs consistently with the lowest average ASR and the largest average DER as in CIFAR-10. Most of the defenses also fail in the strong attacks Blended, LF, and Trojan, while NC performs the second best with comparable average ASR and DER. Compared to the other two datasets, TSBD performs much superior with

Algorithm 1 Two-Stage Backdoor Defense

Input : Small clean set D c , backdoored model with parameter θ bd , max iteration number T , neuron ratio n % and weight ratio m % for reinitialization

Output

: Clean model with parameter θ ∗

1: /* Neuron Weight Change-based Backdoor Reinitialization */

// a. Clean Unlearning

2: while Clean accuracy on θ ul > 10% do

3: Sample a mini-batch B c from D c

4: θ ul ← max θ bd L ( f ( B c ; θ bd ))

5: end while

6: Record subweight changes and calculate NWC by Equation (2) w.r.t. θ ul and θ bd // c. Zero Reinitialization.

7: Obtain and sort the weight changes from the topn % neurons w.r.t. NWC

8: ˆ θ ← reinitialize m % of the most-changing weights into zero on θ bd

9: /* Activeness-aware Fine-tuning */

10: for t = 1 to T do

11: Sample a mini-batch B c from D c

12: Update ˆ θ by Equation (4) with the approximated Equation (6)

13: end for

14: θ ∗ ← fine-tuned ˆ θ

Algorithm 2 The Approximated Optimization of Activeness-aware Fine-tuning

Input : Small clean set D c , the reinitialized model with parameter ˆ θ , max iteration number T , approximation scalar r , balance coefficient α

Output : Clean model with parameter θ ∗

1: /* Activeness-aware Fine-tuning */

2: for t = 1 to T do

3: Sample a mini-batch B c from D c

4: Calculate the gradient g 1 = ∇ ˆ θ L ce ( ˆ θ ) with B c

6: Calculate the gradient g 2 = ∇ ˆ θ ′ L ce ( ˆ θ ′ ) with B c

5: Temporally update the current parameter ˆ θ ′ = ˆ θ + r ∇ ˆ θ L ce ( ˆ θ ) ∥∇ ˆ θ L ce ( ˆ θ ) ∥ 2

7: Calculate the final gradient g = (1 -α ) g 1 + αg 2

8: Update the parameter ˆ θ based on g

9: end for

10: θ ∗ ← fine-tuned ˆ θ

most ASRs lower than 0.5% except for Blended attack. This suggests that TSBD might perform better when the model is learned with the input images with similar characteristics.

E Performance on VGG19-BN model structure

Except for PreAct-ResNet18, we also test another model, VGG19-BN. The performance on CIFAR10 with 10% poisoning ratio and 5% clean data ratio is illustrated in Table 7. We follow similar settings in PreAct-ResNet18 on CIFAR-10, conducting 8 attacks and comparing our method with 8 defenses. The performance is also evaluated by ACC, ASR, and DER. As shown in the table, TSBD owns SOTA performance on VGG19-BN with the best average ACC and second-best average ASR and DER. Most of the other performances are in a similar pattern as on PreAct-ResNet18. Although ANP performs almost the best in ASR and DER for all attacks, it damages the corresponding ACC as well, especially for the WaNet attacked model, where ACC decreases from 84.58% to 78.04%. On the contrary, our methods succeed in most attacks with high ACC and comparable ASR.

F Effectiveness of using NWC in Other Defense

As illustrated in Section 4.3, it is effective to utilize NWC order in gauging the backdoor strength. To further test its ability to improve other defenses, we substitute the average neuron activation in FP to NWC(denoted as NWC-FP) and conduct the experiments on the default settings in PreAct-ResNet18 on CIFAR-10. Different from zero reinitialization, the pruned neurons will not be updated on the following fine-tuning. The performances of 8 attacks are shown in Figure 8. We can investigate that, by using NWC in FP, performances on ASR and DER are improved on most of the attacks, with few effects on ACC. It exhibits the potential of using our NWC in more defense methods.

G Performance with Different Neuron Ratio and Weight Ratio

Section 4.4 shows some of the performances with different neuron ratios and weight ratios and verifies the robustness of the hyper-parameter tuning in our method. Here, we exhibit the full results under all

BadNets

Blended

Input-aware

LF

SIG

SSBA

Trojan

WaNet

0

20

40

60

80

ACC (%)

FP

NWC-FP

BadNets

Blended

Input-aware

LF

SIG

SSBA

Trojan

WaNet

0

20

40

60

80

100

ASR (%)

FP

NWC-FP

BadNets

Blended

Input-aware

LF

SIG

SSBA

Trojan

WaNet

0

20

40

60

80

100

DER (%)

FP

NWC-FP

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

BadNets

ACC

ASR

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

Blended

ACC

ASR

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

Input-aware

ACC

ASR

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

LF

ACC

ASR

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

SIG

ACC

ASR

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

SSBA

ACC

ASR

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

Trojan

ACC

ASR

1 5 1015202530

40

50

70

90

0

20

40

60

80

100

Accuracy (%)

WaNet

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

BadNets

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

Blended

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

Input-aware

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

LF

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

SIG

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

SSBA

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

Trojan

ACC

ASR

40

50

60

70

80

Weight Ratio (m%)

0

20

40

60

80

100

Accuracy (%)

WaNet

ACC

ASR

8 attacks in PreAct-ResNet18 on CIFAR-10 with 10% poisoning ratio. Figure 9 and Figure 10 exhibit the tuning of neuron ratio and weight ratio, respectively. All performances are as good as expected.

H Evaluations with Different Clean Data Ratios

We are here to test the performance of TSBD under different clean data ratios. The performance on CIFAR-10 and PreAct-ResNet18 with 10% poisoning ratio is illustrated in Table 8. Except for the default 5% clean data ratio, we also test the performance under 10%, 1%, and 0.5%. We can observe that a larger ratio of clean data can always bring better performance in ACC, e.g. , 92 . 18% > 91 . 70% > 89 . 63% > 86 . 42% on average values of the four ratios in decreasing order, respectively. For the defense performance in ASR, all of them can successfully defend the attacks to under 10%, and large clean data ratios can achieve promising DERs. Overall, TSBD is robust to the clean data ratio, with only 0.5% clean data can also defend most tested defenses successfully.

I Evaluations with Different Learning Rates on Fine-tuning

We test the performance of our proposed method under different learning rates on fine-tuning. Table 9 illustrates the results. Specifically, with the same settings on CIFAR-10 and PreAct-ResNet18 with 10% poisoning ratio and 5% clean data ratio, we test the performance when the learning rate is set to 0.1, 0.01, 0.001, and 0.0001, where 0.01 is our default setting in the previous experiments. We can observe that the default learning rate of 0.01 is the most suitable one chosen for defense, which can achieve SOTA ASR and DER on average with comparable average ACC. Besides, setting the learning rate to 0.001 can perform the best in ACC, while making it fail on some strong attacks, e.g. ,

Blended and LF. On the contrary, setting the learning rate to 0.1 will hurt the ACC greatly, which may imply that clean knowledge cannot be re-learned properly.

J Evaluations on Clean Model

To test whether our defense method will hurt the performance of the clean model if no backdoor attack occurs, we compare it with several defense methods on CIFAR-10 and Tiny ImageNet datasets on PreAct-ResNet18. Figure 11 illustrates the results, where the ACC and ASR on the original clean model, ANP, and RNP, are used for comparison. We can observe that most of the defense methods (including ours) will barely damage the clean model though there is no backdoor occurs, except for RNP on Tiny ImageNet. It exhibits the potential to largely employ our defense methods in real-world AI systems.

K Performance on Scaled-Up Experiments

To further verify the scalability of our method, we evaluate its performance on a ViT-b-16 model with the CIFAR10 dataset following the basic settings in Section 4.1, e.g. , the poisoning ratio is set to 10% and the target label is set to 0. The results are shown in Table 10. The results demonstrate that TSBD performs effectively on the scaled-up model, achieving a low ASR and acceptable ACC. In contrast, CLP and ANP fail completely with ASR still at a high level, particularly for the WaNet attack.

L Computational Overhead

To show the computational overhead of TSBD, we record the average computational time of each defense step and its practical runtime compared to other defense methods. Table 11 exhibits the computational time of TSBD, including Clean Unlearning , NWC Calculation , Zero Reinitialization , and Activeness-Aware Fine-Tuning . We observe that the main computational overhead lies in the finetuning process. In contrast, the time required for clean unlearning does not increase proportionally with dataset complexity. This means that TSBD is as efficient as other fine-tuning-based methods. Moreover, we present a practical runtime comparison with other SOTA defenses in Table 12, including the loading and testing time needed in practice. As we can see, TSBD is faster than most of the existing methods.

Clean Model

ANP

RNP

TSBD

0

20

40

60

80

Accuracy (%)

(a) CIFAR-10

ACC

ASR

Clean Model

ANP

RNP

TSBD

0

10

20

30

40

50

Accuracy (%)

(b) Tiny ImageNet

ACC

ASR

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and scope are accurately written in the Introduction Section. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The last sentence of the Conclusion section clearly states the limitation and potential solution.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: All the results are empirically illustrated.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experimental settings are clearly illustrated in the Experiment section and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the data are open-sourced. The code is provided in Supplementary Material , and will be publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All settings and details are provided in the Experiment section and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We strictly follow the basic settings in the open-sourced BackdoorBench [51], which ensures the fairness of all the comparisons in the Experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources are included in Appendix C, and the official information from BackdoorBench, since we follow their default settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We strictly conform the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The experiment in Appendix K exhibits the positive societal impacts of potentially employing our defense method without hurting the model performance in the real-world AI system.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our defense method has only a positive impact on use.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All licenses are explicitly mentioned.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the documentation for the summit code in Supplementary Material , and it will be publicly available.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first makes two observations that neurons exhibiting significant weight changes during clean unlearning also tend to play crucial roles in poison unlearning, and neurons in the backdoored model are always more active compared to those in the clean model. The authors showcase on commonly used backdoor attacks and provide further explanations for these two observations. Based on these two observations, a model is first unlearned on clean data where weights with the highest changes are re-initialized, and then the model is optimized by activeness-aware fine-tuning. Extensive experiments are provided to support the proposed defense.\n\nSTRENGTHS:\nThe analysis is clear and well-formulated, based on which the proposed defense is effective and easy to interpret. Extensive experiments on different types of backdoor attacks are provided.\n\nWEAKNESSES:\nThe design principle of the proposed method (TSBD) is very similar to RNP [22]. Both TSBD and RNP follow the two-stage defense setup, including clean unlearning and recovering. The performance of TSBD is also similar to RNP except for the SIG backdoor. The authors provide some explanations about the difference between RNP and TSBD, but they are not convincing. The core technical difference can be further clarified. \n\nTable 3 provides the ablation study on the zero reinitialization ratios, indicating that reinitialization is a sensitive parameter. In a practical threat scenario, how would the defender select a proper reinitialization ratio? It is also suspicious whether the neuron ratio selection plays a key role in increasing the effectiveness. Closely related work [a] is missing, where clean unlearning by mask optimization is also discussed. \n\n\n[a] Towards reliable and efficient backdoor trigger inversion via decoupling benign features. In ICLR, 2024\n\nQUESTIONS:\nMy main concern is the difference between TSBD and RNP, as well as the practicality of TSBD as a defense.\n\nLIMITATIONS:\nThe authors did not provide analysis on limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
The authors propose a novel two-stage backdoor defense method TSBD. The proposed method is based on two key observations 1) the weight changes of neurons during clean and poison unlearning are correlated, 2) the backdoored neurons exhibit a larger gradient norm during unlearning. Respectively, the proposed defense method consists of 1) reinitializing neurons with high weight changes, 2) fine-tuning under gradient-norm regularization. The proposed method is compared with several state-of-the-art defense methods on different attacks, and a comprehensive ablation study is conducted.

STRENGTHS:
1. The paper is well-written and easy to follow. The figures are very clear.
2. The observations are interesting, providing insights into this field.
3. The proposed method is effective across different attacks, showing low attack successful rate and good defense effective rate. The authors provide sufficient and convincing results.

WEAKNESSES:
1. How is the performance on scaled-up experiments, e.g. ViT, or ImageNet?
2. What is the computational overhead of the proposed method compared to others?

QUESTIONS:
Elaborated above.

LIMITATIONS:
The authors discuss the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the security threat posed by backdoor attacks in deep neural networks (DNNs). The authors explore model unlearning from the perspective of weight changes and gradient norms, making two key observations: weight changes between poison and clean unlearning are positively correlated, and neurons in backdoored models are more active than those in clean models. Based on these observations, they propose a Two-Stage Backdoor Defense (TSBD) method, involving Neuron Weight Change-based Backdoor Reinitialization and Activeness-Aware Fine-Tuning. Extensive experiments demonstrate the superior performance of their method compared to state-of-the-art approaches.\n\nSTRENGTHS:\n1. The paper introduces a novel perspective on backdoor defense by exploring the correlation between weight changes in poison and clean unlearning and the activeness of neurons. This approach provides new insights into identifying and mitigating backdoor vulnerabilities.\n2. The paper is well-organized and clearly presents its methodology, findings, and contributions.\n3. The proposed TSBD method is rigorously evaluated through extensive experiments involving eight backdoor attacks on three benchmark datasets.\n\nWEAKNESSES:\n1. The TSBD method involves additional steps such as clean unlearning, neuron weight change calculation, and activeness-aware fine-tuning, which may introduce computational overhead. An analysis of the computational cost and efficiency of the proposed method compared to existing defenses would be beneficial.\n2. What causes the clean unlearning NWCs to exhibit a positive correlation with those in poison unlearning? I think this question has not been solved clearly.\n\nQUESTIONS:\n1. When selecting the clean dataset, how is the class distribution handled? Is it randomly selected or fully covered? Intuitively, would data related of the target label cause particularly high activation values?\n2. I think the NWC strategy is not significantly different from the pruning method in RNP[1], and the Activeness-Aware Fine-tuning is also similar to the fine-tuning in RNP. Therefore, I hope the authors can emphasize more clearly the differences between their approach and RNP.\n3. From the observations, it is apparent that there is a difference in neuron activation between clean and backdoored models. However, the use of gradient norm restriction in the unlearning process is applied to an already modified model (not the original backdoored model). The direct correlation between this and enhanced defense effectiveness does not seem sufficiently clear and is not within the scope of the initial observations.\n4. Regarding the selection of hyperparameters in Activeness-Aware Fine-tuning, I believe this can significantly impact the experimental results, but there is no discussion on this aspect in the paper.\n\n[1] Li Y, Lyu X, Ma X, et al. Reconstructive neuron pruning for backdoor defense[C]//International Conference on Machine Learning. PMLR, 2023: 19837-19854.\n\nLIMITATIONS:\nI think the paper still has some limitations that the authors didn't mention. Please refer to my questions and weaknesses.\nPlease add some discussions about data-free methods", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces two key observations in backdoored models, presenting a two-stage backdoor attack defense method, where the two observations are the following: a strong positive correlation between weight changes in poison and clean unlearning, and the stronger neuron activation in backdoored models compared to clean models. The proposed two-stage defense method leverages these observations by reinitializing a certain proportion of backdoor-related neurons and subweights, and by suppressing gradient normalization during the fine-tuning process. The proposed method demonstrates the state-of-the-art performance on the selected datasets and models with a wide range of ablation studies.\n\nSTRENGTHS:\n1.  The paper introduces a novel approach to backdoor defense. It provides two insights in the weights of backdoor models. Based on this, the authors introduce a two-stage defense mechanism. Also, the use of clean unlearning that does not require poisoned data to prevent the backdoor attack.\n\n2. While the scope of the experiment is limited in model architectures and datasets, the experiment shows superior performance. The experimental setup is thorough, covering a certain range of backdoor attacks. \n\n3. The paper is well-organized, with figures and tables are used to illustrate key observations and experimental results. The descriptions of the observations and the proposed defense method are detailed and precise.\n\nWEAKNESSES:\n1. The scale of experiments are too limited. While the authors compare the results between 8 other backdoor attack methods, the main experiments are only conducted to specific datasets and models (CIFAR-10, PreAct-ResNet18). This makes the generalizability of the results rather unclear.\n\n2. The authors mention computational efficiency in Activeness-Aware Fine-Tuning by using an approximation scheme, but there are no experiments demonstrating the trade-off between efficiency and accuracy. Including such experiments would provide a more comprehensive understanding of the method's practical implications.\n\n3.\tThe concept of unlearning in backdoor attacks could be better explained in the Related Works section. Given its importance in this paper, a more detailed discussion would help contextualize the proposed method and highlight its significance in addressing existing challenges.\n\n4. There is no mathematical proof. While it employs various loss functions and approximation techniques, and the empirical results are quite promising, there is no analysis, explaining why the proposed method works.\n\nQUESTIONS:\n1. There is no statement or analysis about scalability and generalizability. Can you provide more detailed information about these, to different datasets and model architectures?\n\n2. How can the clean unlearning process identifies and affects backdoor related neurons, without causing degradation of the models’ performance on clean data?\n\n3.\tThe section introducing the effectiveness of zero reinitialization on subweights in 4.3 Ablation Studies is a bit confusing. The difference between V2 and V3 is still not clear to me.\n\n4.\tI have questions about the insensitivity of TSBD to both neuron ratio and weight ratio as explained in Section 4.4. While I understand that the method effectively selects backdoor-related neurons indicated as active neurons and that reinitializing them is effective, it is confusing that the performance remains similar across both low and high ratios. Is reinitializing itself the important part?\n\nLIMITATIONS:\nThe authors mention in the introduction that clean unlearning is still underexplored, but it is unclear if this limitation has been fully addressed in the paper. Additionally, the paper lacks a discussion of its own limitations. Providing a clear discussion on the limitations of the proposed method and potential areas for future research would enhance the paper.\n\nThe authors acknowledge that there remain challenges in fully mitigating backdoor effects without any access to clean data. They suggest that data generation techniques and data free techniques might offer potential solutions.\n\nThere is no discussion about the scalability of their proposed method, especially in large scale or real time applications. Addressing the computational overhead and resource requirements would be necessary for practical applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}