Title: Uncovering, Explaining, and Mitigating the Superficial Safety of Backdoor Defense
Venue: NeurIPS 2024
Accepted: yes
Keywords: Backdoor Safety

Safety Tuning

Superficial Safety
Abstract: Backdoor attacks pose a significant threat to Deep Neural Networks (DNNs) as they allow attackers to manipulate model predictions with backdoor triggers. To address these security vulnerabilities, various backdoor purification methods have been proposed to purify compromised models. Typically, these purified models exhibit low Attack Success Rates (ASR), rendering them resistant to backdoored inputs. However, \textit{Does achieving a low ASR through current safety purification methods truly eliminate learned backdoor features from the pretraining phase?} In this paper, we provide an affirmative answer to this question by thoroughly investigating the \textit{Post-Purification Robustness} of current backdoor purification methods. We find that current safety purification methods are vulnerable to the rapid re-learning of backdoor behavior, even when further fine-tuning of purified models is performed using a very small number of poisoned samples. Based on this, we further propose the practical Query-based Reactivation Attack (QRA) which could effectively reactivate the backdoor by merely querying purified models. We find the failure to achieve satisfactory post-purification robustness stems from the insufficient deviation of purified models from the backdoored model along the backdoor-connected path. To improve the post-purification robustness, we propose a straightforward tuning defense, Path-Aware Minimization (PAM), which promotes deviation along backdoor-connected paths with extra model updates. Extensive experiments demonstrate that PAM significantly improves post-purification robustness while maintaining a good clean accuracy and low ASR. Our work provides a new perspective on understanding the effectiveness of backdoor safety tuning and highlights the importance of faithfully assessing the model's safety.

Uncovering, Explaining, and Mitigating the Superficial Safety of Backdoor Defense

Abstract

Backdoor attacks pose a significant threat to Deep Neural Networks (DNNs) as they allow attackers to manipulate model predictions with backdoor triggers. To address these security vulnerabilities, various backdoor purification methods have been proposed to purify compromised models. Typically, these purified models exhibit low Attack Success Rates (ASR), rendering them resistant to backdoored inputs. However, Does achieving a low ASR through current safety purification methods truly eliminate learned backdoor features from the pretraining phase? In this paper, we provide an affirmative answer to this question by thoroughly investigating the Post-Purification Robustness of current backdoor purification methods. We find that current safety purification methods are vulnerable to the rapid re-learning of backdoor behavior, even when further fine-tuning of purified models is performed using a very small number of poisoned samples. Based on this, we further propose the practical Query-based Reactivation Attack (QRA) which could effectively reactivate the backdoor by merely querying purified models. We find the failure to achieve satisfactory post-purification robustness stems from the insufficient deviation of purified models from the backdoored model along the backdoor-connected path. To improve the post-purification robustness, we propose a straightforward tuning defense, Path-Aware Minimization (PAM), which promotes deviation along backdoor-connected paths with extra model updates. Extensive experiments demonstrate that PAM significantly improves post-purification robustness while maintaining a good clean accuracy and low ASR. Our work provides a new perspective on understanding the effectiveness of backdoor safety tuning and highlights the importance of faithfully assessing the model's safety.

1 Introduction

Backdoor attacks [2, 7, 16] have emerged as one of the most significant concerns [4, 5, 20, 25] in deep learning. These attacks involve the insertion of malicious backdoor triggers into the training set, which can be further exploited to manipulate the behavior of the model during the inference stage. To defend against these threats, researchers have proposed various safety tuning methods [20, 27, 32, 44, 49, 50, 55] to purify well-trained backdoored models. These methods can be easily incorporated into the existing model deployment pipeline and have demonstrated state-of-the-art effectiveness in reducing the Attack Success Rate (ASR) of backdoored models [32, 48].

However, a critical question arises: does achieving a low Attack Success Rate (ASR) through current safety tuning methods genuinely indicate the complete removal of learned backdoor features from the pretraining phase? If the answer is no, this means that the adversary may still easily reactivate the

implanted backdoor from the residual backdoor features lurking within the purified model, thereby exerting insidious control over the model's behavior. This represents a significant and previously unacknowledged safety concern, suggesting that current defense methods may only offer superficial safety [1]. Moreover, if an adversary can successfully re-trigger the backdoor, it raises another troubling question: how can we assess the model's robustness against such threats? This situation underscores the urgent need for a more comprehensive and faithful evaluation of the model's safety.

In this work, we provide an affirmative answer to these questions by thoroughly investigating the PostPurification Robustness of state-of-the-art backdoor safety tuning methods. Specifically, we employ the Retuning Attack (RA) [36, 42] where we first retune the purified models using an extremely small number of backdoored samples and tuning epochs. Our observations reveal that current safety purification defense methods quickly reacquire backdoor behavior after just a few epochs, resulting in significantly high ASR levels. In contrast, the clean model (which does not have backdoor triggers inserted during the pretraining phase) and Exact Purification (EP)-which fine-tunes models using real backdoored samples with correct labels during safety purification, maintain a low ASR even after the RA. This discrepancy suggests that existing safety tuning methods do not thoroughly eliminate the learned backdoor, creating a superficial impression of backdoor safety . Since the vulnerability revealed by the Retuning Attack (RA) relies on the use of retuned models, we further propose the more practical Query-based Reactivation Attack (QRA). This attack is capable of generating sample-specific perturbations that can trigger the backdoor in purified models, which were previously believed to have eliminated such threats, simply by querying these purified models.

To understand the inherent vulnerability of current safety purification methods concerning postpurification robustness, we further investigate the factors contributing to the disparity in postpurification robustness between EP and other methods. To this end, we utilize Linear Mode Connectivity (LMC) [13, 33] as a framework for analysis. We find that EP not only produces a solution with low ASR like other purification methods but also pushes the purified model further away from the backdoored model along the backdoor-connected path, resulting in a more distantly robust solution. As a result, it becomes challenging for the retuning attack to revert the EP model back to the basin with high ASR where the compromised model is located. Inspired by our findings, we propose a simple tuning defense method called Path-Aware Minimization (PAM) to enhance post-purification robustness. By using reversed backdoored samples as a proxy to measure the backdoored-connected path, PAM updates the purified model by applying gradients from a model interpolated between the purified and backdoored models. This approach helps identify a robust solution that further deviates our purified model from the backdoored model along the backdoor-connected path. Extensive experiments have demonstrated that PAM achieves improved post-purification robustness, retaining a low ASR after RA across various settings. To summarize, our contributions are:

Our work first offers a new perspective on understanding the effectiveness of current backdoor safety tuning methods. Instead of merely focusing on the commonly used Attack Success Rate, we investigate the Post-Purification Robustness of the purified model to enhance our comprehensive understanding of backdoor safety in deep learning models.

We employ the Retuning Attack by retuning purified models on backdoored samples to assess the post-purification robustness. Our primary observations reveal that current safety purification methods are vulnerable to RA, as evidenced by a rapid increase in the ASR. Furthermore, we propose the more practical Query-based Reactivation Attack, which can reactivate the implanted backdoor of purified models solely through model querying.

We analyze the inherent vulnerability of current safety purification methods to the RA through Linear Mode Connectivity and attribute the reason to the insufficient deviation of purified models from the backdoored model along the backdoor-connected path. Based on our analysis, we propose Path-Aware Minimization, a straightforward tuning-based defense mechanism that promotes deviation by performing extra model updates using interpolated models along the path. Extensive experiments verify the effectiveness of the PAM method.

2 Related Work

Backdoor Attacks. Backdoor attacks aim to manipulate the backdoored model to predict the target label on samples containing a specific backdoor trigger while behaving normally on benign samples. They can be roughly divided into two categories [48]: (1) Data-poisoning attacks: the attacker inserts

a backdoor trigger into the model by manipulating the training sample ( x , y ) ∈ ( X , Y ) , such as adding a small patch to a clean image x and change the sample's label to an attacker-designated target label y t [4, 7, 15, 16, 28, 43]; (2) Training-control attacks: the attacker has control over both the training process and the training data simultaneously [34]. Note that data-poisoning attacks are more practical in real-world scenarios as they make fewer assumptions about the attacker's capabilities [5, 15, 41] and have resulted in increasingly serious security risks [4, 39]. In this work, we focus on data-poisoning backdoor attacks .

Backdoor Defense. Existing backdoor defense strategies could be broadly categorized into robust pretraining [19, 26] and robust fine-tuning methods [32, 49, 53, 55]. Robust pretraining aims to prevent the learning of backdoor triggers during the pretraining phase. However, these methods often suffer from accuracy degradation and can significantly increase model training costs, making them impractical for large-scale applications. In contrast, robust purification methods focus on removing potential backdoor features from a well-trained model. Generally, purification techniques involve reversing potential backdoor triggers [44, 45, 46, 50] and applying fine-tuning or pruning to address backdoors using a limited amount of clean data [29, 32, 49, 55]. While these purification methods reduce training costs, they also achieve state-of-the-art defense performance [32, 55]. Therefore, in this work, we mainly focus on evaluations of robust purification methods against backdoor attacks.

Loss Landscape and Linear Mode Connectivity. Early works [12, 14, 22] conjectured and empirically verified that different DNN loss minima can be connected by low-loss curves. In the context of the pretrain-fine-tune paradigm, Neyshabur et al. [33] observe that the pretrained weights guide purified models to the same flat basin of the loss landscape, which is close to the pretrained checkpoint. Frankle et al. [13] also observe and define the linear case of mode connectivity, Linear Mode Connectivity (LMC). LMC refers to the absence of the loss barrier when interpolating linearly between solutions that are trained from the same initialization. The shared initialization can either be a checkpoint in early training [13] or a pretrained model [33]. Our work is inspired by [33] and also utilizes LMC to investigate the properties of purified models in relation to backdoor safety.

Deceptive AI and Superficial Safety. Nowadays, DNNs are typically pretrained on large-scale datasets, such as web-scraped data, primarily using next-token prediction loss [3], along with simple contrastive [6] and classification [23] objective. While these simplified pretraining objectives can lead to the learning of rich and useful representations, they may also result in deceptive behaviors that can mislead humans [10]. One such deceptive behavior is the presence of backdoors [9, 20]. A compromised model can be indistinguishable from a normal model to human supervisors, as both behave similarly in the absence of the backdoor trigger. To address this critical safety risk, researchers propose post-training alignment procedures, such as safety fine-tuning [35, 48]. However, several studies indicate that the changes from fine-tuning are superficial [31, 54]. As a result, these models retain harmful capabilities and knowledge from pretraining, which can be elicited by harmful fine-tuning [37, 51] or specific out-of-distribution (OOD) inputs [47, 52]. We study this phenomenon in the context of backdoor threats and offer a deeper understanding along with mitigation strategies.

3 Revealing Superficial Safety of Backdoor Defenses by Accessing Post-purification Robustness

While current backdoor purification methods can achieve a very low Attack Success Rate (ASR) against backdoor attacks, this does not necessarily equate to the complete elimination of inserted backdoor features. Adversaries may further exploit these residual backdoor features to reconstruct and reactivate the implanted backdoor, as discussed in Section 3.3. This is particularly important because purified models are often used in various downstream scenarios, such as customized fine-tuning [37] for critical tasks [20]. Therefore, it is crucial to provide a way to measure the robustness of purified models in defending against backdoor re-triggering, which we define as "Post-Purification Robustness" .

In this section, we first introduce a simple and straightforward strategy called the Retuning Attack (RA) to assess post-purification robustness. Building on the RA, we then present a practical threat known as the Query-based Reactivation Attack (QRA), which exploits the vulnerabilities in postpurification robustness to reactivate the implanted backdoor in purified models, using only model querying. First, we will introduce the preliminaries and evaluation setup.

3.1 Problem Setup

Backdoor Purification. In this work, we focus on the poisoning-based attack due to its practicality and stealthiness. We denote the original training dataset as D T ⊂ ( X , Y ) . A few training examples ( x , y ) ∈ D T have been transformed by attackers into poisoned examples ( x p , y t ) , where x p is poisoned example with inserted trigger and a target label y t . Following previous works [29, 32, 39, 48, 49], only a limited amount of clean data D t are used for fine-tuning or pruning. For triggerinversion methods [44, 50], we denote the reversed backdoored samples obtained through reversing methods as ( x r , y ) ∈ D r . We evaluate several mainstreamed purification methods, including pruning-based defense ANP [49]; robust fine-tuning defense I-BAU [53] (referred to as BAU for short), FT-SAM [55] (referred to as SAM for short), FST [32], as well as the state-of-the-art triggerreversing defense BTI-DBF [50] (referred to as BTI for short). BTI purifies the backdoored model by using both reversed backdoored samples D r and the clean dataset D t while the others use solely the clean dataset D t . We also include exact purification (EP) that assumes that the defender has full knowledge of the exact trigger and fine-tunes the models using real backdoored samples with correct labels ( x p , y ) .

Attack Settings. Following [32], we evaluate four representative data-poisoning backdoors including three dirty-label attacks (BadNet [16], Blended [7], SSBA [28]), and one clean-label attack (LC [43]). All experiments are conducted on BackdoorBench [48], a widely used benchmark for backdoor learning. We employ three poisoning rates, 10% , 5% , and 1% (in Appendix ) for backdoor injection and conduct experiments on three widely used image classification datasets, including CIFAR-10 [24], Tiny-ImageNet [8], and CIFAR-100 [24]. For model architectures, we following [32], and adopt the ResNet-18, ResNet-50 [17], and DenseNet-161 [18] on CIFAR-10. For CIFAR-100 and Tiny-ImageNet, we adopt pretrained ResNet-18 on ImageNet1K to obtain high clean accuracy as suggested by [32, 50]. More details about experimental settings are shown in Appendix B.

Evaluation Metrics. Following previous backdoor works, we take two evaluation metrics, including Clean Accuracy (C-Acc) (i.e., the prediction accuracy of clean samples) and Attack Success Rate (ASR) (i.e., the prediction accuracy of poisoned samples to the target class) where a lower ASR indicates a better defense performance. We further adopt O-ASR and P-ASR metrics. The O-ASR metric represents the defense performance of original defense methods, while the P-ASR metric indicates the ASR after applying the RA or QRA.

3.2 Purified Models Are Vulnerable to Retuning Attack

Our objective is to investigate whether purified models with low ASR completely eliminate the inserted backdoor features. To accomplish this, it is essential to develop a method for assessing the degree to which purified models have indeed forgotten these triggers. In this section, we begin with a white-box investigation where the attacker or evaluator has access to the purified model's parameters. Here we introduce a simple tuning-based strategy named the Retuning Attack (RA) [37, 42] to conduct an initial evaluation. Specifically, we construct a dataset for model retuning, which comprises a few backdoored samples (less than 1% of backdoored samples used during the training process). To maintain C-Acc, we also include benign samples from the training set, resulting in a total RA dataset with 1000 samples. We subsequently retune the purified models using this constructed dataset through a few epochs ( 5 epochs in our implementation). This approach is adopted because a clean model can not be able to learn a backdoor; thus, if the purified models quickly regain ASR during the retuning process, it indicates that some residual backdoor features still exist in these purified models. Implementation details of the RA can be found in the Appendix B.2.

As shown in Figure 1, we observe that despite achieving very low ASR, all purification methods quickly recover backdoor ASR with Retuning Attack. Their quickly regained ASR presents a stark contrast to that of clean models and remains consistent across different datasets, model architectures, and poisoning rates. Note that the pruning method (ANP) and the fine-tuning method (FST), which achieve state-of-the-art defense performance, still exhibit vulnerability to RA, with an average recovery of approximately 82% and 85% ASR, respectively. In stark contrast, the EP method stands out as it consistently maintains a low ASR even after applying RA, demonstrating exceptional postpurification robustness. Although impractical with full knowledge of the backdoor triggers, the EP method validates the possibility of maintaining a low attack success rate to ensure post-purification robustness against RA attacks.

Moreover, this evident contrast highlights the significant security risks associated with current backdoor safety tuning methods. While these methods may initially appear robust due to a significantly reduced ASR, they are fundamentally vulnerable to backdoor reactivation, which can occur with just a few epochs of model tuning. This superficial safety underscores the urgent need for more comprehensive evaluations to ensure lasting protection against backdoor attacks. It is crucial to implement faithful evaluations that thoroughly assess the resilience of purified models, rather than relying solely on superficial metrics, to truly safeguard against the persistent threat of backdoor vulnerabilities.

3.3 Reactivating Backdoor on Purified Models through Queries

Although our previous experiments on RA demonstrate that current purification methods insufficiently eliminate learned backdoor features, it is important to note that the success of this tuning-based method relies on the attackers' capability to change purified models' weights. This is not practical in a real-world threat model. To address this limitation, we propose Query-based Reactivation Attack (QRA) , which generates sample-specific perturbations that can reactivate the backdoor using only model querying. Specifically, instead of directly retuning purified models, QRA captures the parameter changes induced by the RA process and translates them into input space as perturbations. These perturbations can then be incorporated into backdoored examples, facilitating the successful reactivation of backdoor behaviors in purified models.

To effectively translate the parameter changes into the input space, it is crucial to ensure that when applying the perturbation generated by QRA, the output of the purified model on perturbed inputs should be aligned with that of the post-RA model on original inputs without perturbations. Formally, we denote the purified model as f ( W p ; x ) , the model after RA as f ( W ra ; x ) and their corresponding logit output as l ( W p ; x ) and l ( W ra ; x ) . Our QRA aims to learn a perturbation generator ϕ ( θ ; x ) : R d → [ -1 , 1] d , to produce perturbation ϕ ( θ ; x ) for each input x . We formulate this process into the following optimization problem:

Here S is the distance metric between two output logits, D c is a compositional dataset incorporating both benign and backdoored samples, and ϵ controls the strength of perturbation ( ϵ = 16 / 255 in our implementation). We utilize the Kullback-Leibler (KL) divergence [11] for S and a Multilayer Perception (MLP) for ϕ ( θ ; x ) . Specifically, we flatten the input image into a one-dimensional vector before feeding into the ϕ ( θ ; x ) , and obtain the generated perturbation by reshaping it back to the original size. Details of the MLP architecture and training hyperparameters can be found in the Appendix B.2.

However, we observe that directly optimizing Equation 1 not only targets purified models but also successfully attacks the clean model. We conjecture that this may stem from the inaccurate inversion of reactivated backdoor triggers, which can exploit a clean model in a manner similar to adversarial examples [21, 30, 38]. To mitigate such adversarial behavior, we introduce a regularization term aimed at minimizing backdoor reactivation on the clean model. Given that accessing the clean model may not be practical, we utilize the EP model f ( W e ; x ) as a surrogate model instead. In sum, we formulate the overall optimization objective as follows:

where α represents the balance coefficient and the cross-entropy loss is used for L .

We demonstrate our experimental results against five purification methods on CIFAR-10 in Figure 2. Here, we report the C-ASR and P-ASR , which represent the ASR when evaluating with perturbed clean and perturbed poisoned images, respectively. Notably, our QRA could effectively reactivate the backdoor behaviors on purified models, resulting in a significant increase of 66 . 13% on average in P-ASR. Our experiments also demonstrate a consistently low C-ASR on purified models, which indicates that the perturbations generated by QRA effectively reactivate the backdoored examples without affecting the predictions of benign images. Besides, the perturbation generated with QRA exclusively works on the output of backdoored samples on purified models, leading to both a low C-ASR and P-ASR on clean models. This observation further indicates that the reversed pattern generated by QRA is not a typical adversarial perturbation but rather an accurate depiction of the parameter changes necessary for backdoor reactivation.

Furthermore, it is worth noting that attackers may lack knowledge about the specific backdoor defense techniques utilized by the defender in practice. Thus, we embark on an initial investigation to explore the transferability of our QRA method across unknown purification methods. Specifically, we aim to determine whether the reversed perturbations optimized for one particular defense method can effectively attack purified models with other purification techniques. As shown in Figure 3, our QRA demonstrates a degree of successful transferability across various defense techniques, achieving an average P-ASR of 32 . 1% against all purification techniques. These results underscore the potential of QRA to attack purified models, even without prior knowledge of the defense methods employed by defenders. It also highlights the practical application of QRA in real-world situations.

ANP

BAU

SAM

FST

BTI

0

20

40

60

80

100

Transferability Analysis

BadNet

Blended

SSBA

LC

4 Investigating and Mitigating Superficial Safety

4.1 Investigating the Superficial Safety through Linear Mode Connectivity

While our previous evaluations indicate that only the EP model demonstrates exceptional postpurification robustness compared to current backdoor safety tuning methods, the factors contributing to the effectiveness of EP remain unclear. Motivated by prior studies examining fine-tuned models [13, 14, 33], we propose to investigate this intriguing phenomenon from the perspective of the loss landscape using Linear Mode Connectivity (LMC).

Following [13, 33], let E ( W ; D l ) represent the testing error of a model f ( W ; x ) evaluated on a dataset D l . For D l , we use backdoor testing samples. E t ( W 0 , W 1 ; D l ) = E ((1 -t ) W 0 + t W 1 ; D l ) for t ∈ [0 , 1] is defined as the error path of model created by linear interpolation between the f ( W 0 ; x ) and f ( W 1 ; x ) . We also refer to it as the backdoor-connected path. Here we denote the f ( W 0 ; · ) as backdoored model and f ( W 1 ; · ) as the purified model. We show the LMC results of the backdoor error in Figure 4 and 5. For each attack setting, we report the average results among backdoor attacks. More results on other datasets and models are shown in Appendix C.2.

The backdoored model and purified models reside in separate loss basins, linked by a backdoorconnected path. We present the results of LMC between purified and backdoored models in Figure 4. It is clear from the results that all purified models exhibit significant error barriers along the backdoor-connected path to backdoored model. This indicates that backdoored and purified models reside in different loss basins. Additionally, we conduct LMC between purified models with EP and with other defense techniques, as depicted in Figure 5. We observe a consistently high error without barriers, which indicates that these purified models reside within the same loss basin. Based on these two findings, we conclude that backdoored and purified models reside in two distinct loss basins connected through a backdoor-connected path.

EP deviates purified models from the backdoored model along the backdoor-connected path, resulting in a more distantly robust solution. Although introducing a high loss barrier, we observe notable distinctions between the LMC of the EP model (red solid line) and purified models (dotted lines). We observe a stable high backdoor error along the backdoor-connected path of EP until t < 0 . 2 , where the interpolated model parameter W has over 80% weight from the backdoored model. In

contrast, other purification models show a tendency to exhibit significant increases in ASR along the path, recovering more than 20% ASR when t < 0 . 5 , while the ASR for the EP model remains low ( ≤ 2% ). This clear contrast suggests: 1) the current purification methods prematurely converge to a non-robust solution with low ASR, which is still close to the backdoored model along the backdoorconnected path; 2) compared with purified models, EP makes the purified model significantly deviate from the backdoored checkpoint along the backdoor-connected path, resulting in a more robust solution against RA.

Accurately specified supervision is crucial for achieving stable backdoor safety. As demonstrated in our observations, the EP method attains stable robustness in the context of the RA, whereas its proxy version, the BTI method, employs reversed backdoor data as a substitute for real backdoor data, resulting in suboptimal post-purification robustness. Furthermore, notable discrepancies are evident in the Backdoor LMC results. These findings underscore that current methods for reversing backdoor triggers are still unable to accurately recover all backdoor features [40], thereby emphasizing the importance of precisely specified supervision in achieving stable backdoor safety. Although data generated by the BTI method does not accurately recover all backdoor features, it could serve as an effective and usable supervision dataset. In the following section, we propose an improved safety tuning method designed to mitigate superficial safety concerns based on this proxy dataset.

4.2 Enhancing Post-Purification Robustness Through Path-Aware Minimization

Motivated by our analysis, we propose a simple tuning defense method called Path-Aware Minimization (PAM) , which aims to enhance post-purification robustness by promoting more deviation from the backdoored model along the backdoor-connected path like the EP method.

Since there are no real backdoor samples x p available, we employ the synthetic backdoored samples x r from the trigger-reversing method BTI [50] as a substitute to get the backdoor-connected path. Although BTI has a similar LMC path curve with the EP model in Figure 5, as we have discussed, tuning solely with x r would lead to finding a non-robust solution with low ASR.

To avoid converging to such a solution, we propose utilizing the gradients of an interpolated model W d between W 0 and W to update the current solution W . As illustrated in Figure 4, the interpolated model, which lies between W 0 and W , exhibits a higher ASR compared to W . By leveraging the gradients from the interpolated model, we can perform additional updates on the W which prevents premature convergence towards local minima and results in a solution that deviates from the backdoored model along this path. Specifically, for W , we first take a path-aware step ρ W d ∥ W d ∥ 2 ( W d = W 0 -W ) towards to W 0 and obtain the interpolated model W + ρ W d ∥ W d ∥ 2 . Then we compute its gradient on x r to update W . We formulate our objective function as follows:

where ρ represents the size of the path-aware step. Typically, a larger ρ indicates a larger step towards the backdoored model W 0 along our backdoor-connected path and also allows us to obtain a larger gradient update for W , which results in more deviation from the backdoored model along the backdoor-connected path. The detailed algorithm is summarized in the Algorithm 1.

Post-Purification Robustness of PAM. We evaluate the post-purification robustness of PAM against RA and make a comparison with EP. Using the same experimental settings in Section 3.1, we set ρ to 0 . 5 for Blended and SSBA and 0 . 9 for the BadNet and LC attack on CIFAR-10 and set ρ to 0 . 4 for both CIFAR-100 and Tiny-ImageNet. The results on CIFAR10, CIFAR-100 and Tiny-ImageNet are shown in Table 1 and Table 2. We could observe that PAM significantly improves post-purification robustness against RA. It achieves a comparable robustness performance to the EP , with an average ASR lower than 4 . 5% across all three datasets

Algorithm 1 Path-Aware Minimization (PAM)

Input: Tuning dataset D T = D r ∪ D t ; Backdoored model f ( W 0 ; x ) ; Learning rate η ; Path-aware step size ρ ; Tuning iterations I

Output:

Purified model

1: Initialize W 1 with W 0

2: for i = 1 , . . . , I do

3: Sample a mini-batch B i from the tuning set D T ;

4: Calculate parameter difference: W d = W 0 -W i ;

5: Obtain interpolated parameters: ˜ W i = W i + ρ W d ∥ W d ∥ 2 ;

6: Calculate gradients of ˜ W i : g ˜ W i = ∇ ˜ W i 1 |B i | ∑ ( x , y ) ∈B i L ( f ( ˜ W i ; x , y )

7: Update current parameters: W i +1 = W i -η g ˜ W i

8: end for

9: return Purified model f ( W I ; x )

after RA. In comparison to previous experimental results in Section 3.2, our PAM outperforms existing defense methods by a large margin in terms of post-purification robustness. Our PAM also achieves a stable purification performance (O-ASR), reducing the ASR below 2% on all three datasets and preserves a high C-Acc as well, yielding only around 2% drop against the original performance of C-Acc.

To further verify the post-purification with PAM, following the experimental setting in Section 4.1, we also show the LMC results of PAM in Figure 4. It is clearly observed that our PAM significantly deviates purified models from the backdoored model along the backdoor-connected path, leading to a robust solution similar to the EP method. This further confirms our findings about post-purification robustness derived from LMC.

Sensitivity Analysis of ρ . We evaluate the performance of PAM with various values of ρ and conduct experiments on CIFAR-10 with ResNet-18 against four attacks. The experimental results are shown in Figure 6. Note that as ρ increases, we increase the error barrier along the connected path, indicating an enhanced deviation of our purified model from the backdoored model. However, simply increasing ρ would also compromise the competitive accuracy (C-Acc) of the purified model. In practice, it is essential to select an appropriate ρ to achieve a balance between post-purification robustness and C-Acc. We present the model performance across various ρ values in Table 10 of the Appendix . We can observe that as ρ rises, there is a slight decrease in clean accuracy alongside a significant enhancement in robustness against the RA. Additionally, we note that performance is relatively insensitive to ρ when it exceeds 0.3. Given that we primarily monitor C-Acc (with the validation set) in practice, we aim to achieve a favorable trade-off between these two metrics. Therefore, we follow the approach of FST [32] and select ρ to ensure that C-Acc remains above a predefined threshold, such as 92% .

5 Conclusions and Limitations

In this paper, we seek to address the following question: Do current backdoor safety tuning methods genuinely achieve reliable backdoor safety by merely relying on reduced Attack Success Rates? To investigate this issue, we first employ the Retuning Attack to evaluate the post-purification robustness of purified models. Our primary experiments reveal a significant finding: existing backdoor purification methods consistently exhibit an increased ASR when subjected to the RA, highlighting the superficial safety of these approaches. Building on this insight, we propose a practical Query-based Reactivation Attack, which enables attackers to re-trigger the backdoor from purified models solely through querying. We conduct a deeper analysis of the inherent vulnerabilities against RA using Linear Mode Connectivity, attributing these vulnerabilities to the insufficient deviation of purified models from the backdoored model along the backdoor-connected path. Inspired by our analysis, we introduce a simple tuning defense method, Path-Aware Minimization, which actively promotes deviation from the backdoored model through additional model updates along the interpolated path. Extensive experiments demonstrate the effectiveness of PAM, surpassing existing purification techniques in terms of post-purification robustness.

This study represents an initial attempt to evaluate post-purification robustness via RA. While we propose the practical QRA method, future work is essential to develop more efficient evaluation techniques that can faithfully assess post-purification robustness. The need for such evaluations is critical, as they ensure that the perceived safety of purified models is not merely superficial. Additionally, we recognize the significant potential for enhancing QRA in the context of transfer attacks, which we aim to explore in future research. Furthermore, we plan to broaden our research by incorporating additional backdoor attack strategies and safety tuning methods applicable to generative models, such as LLMs and diffusion models [1, 20, 37, 40], in future work. We will also apply our existing framework of evaluation, analysis, and safety tuning method to research on unlearning in large models.

References

Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learning Research , 2024. ISSN 2835-8856. Survey Certification, Expert Certification. 1, 5

Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics , pages 2938-2948. PMLR, 2020. 1

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. 2

Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations , 2022. 1, 2

Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302.10149 , 2023. 1, 2

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020. 2

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 , 2017. 1, 2, 3.1

Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819 , 2017. 3.1

Paul Christiano. Mechanistic anomaly detection and elk. Alignment Research Center , 2022. URL https://ai-alignment.com/mechanistic-anomaly-detection-and-elk-fb84f4c6d0dc . 2

Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge: How to tell if your eyes deceive you. technical report. Alignment Research Center , 2021. URL https://docs.google.com/document/ d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit . 2

Thomas M Cover. Elements of information theory . John Wiley & Sons, 1999. 3.3

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning , pages 1309-1318. PMLR, 2018. 2

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020. 1, 2, 4.1

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems , 31, 2018. 2, 4.1

Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022. 2

Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access , 7:47230-47244, 2019. 1, 2, 3.1

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 3.1

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017. 3.1

Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In International Conference on Learning Representations , 2022. 2

Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566 , 2024. 1, 2, 3, 5

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems , 32, 2019. 3.3

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 , 2018. 2

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012. 2

Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 3.1

Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine learning-industry perspectives. In 2020 IEEE security and privacy workshops (SPW) , pages 69-75. IEEE, 2020. 1

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems , 34:1490014912, 2021. 2, B.1

Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang. Reconstructive neuron pruning for backdoor defense. In International Conference on Machine Learning , pages 19837-19854. PMLR, 2023. 1

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16463-16472, 2021. 2, 3.1

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses , pages 273-294. Springer, 2018. 2, 3.1

Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations , 2016. 3.3

Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In International Conference on Machine Learning , pages 22965-23004. PMLR, 2023. 2

Rui Min, Zeyu Qin, Li Shen, and Minhao Cheng. Towards stable backdoor purification through feature shift tuning. Advances in Neural Information Processing Systems , 36, 2024. 1, 2, 3.1, 3.1, 4.2, B.1, B.2, C.4

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems , 33:512-523, 2020. 1, 2, 4.1

Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In International Conference on Learning Representations , 2021. 2

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022. 2

Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In The eleventh international conference on learning representations , 2023. 1

Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Finetuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693 , 2023. 2, 3, 3.2, 5

Zeyu Qin, Yanbo Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu. Boosting the transferability of adversarial attacks with reverse adversarial perturbation. Advances in neural information processing systems , 35:29845-29858, 2022. 3.3

Zeyu Qin, Liuyi Yao, Daoyuan Chen, Yaliang Li, Bolin Ding, and Minhao Cheng. Revisiting personalized federated learning: Robustness against backdoor attacks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 4743-4755, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. 2, 3.1

Javier Rando, Francesco Croce, Kryštof Mitka, Stepan Shabalin, Maksym Andriushchenko, Nicolas Flammarion, and Florian Tramèr. Competition report: Finding universal jailbreak backdoors in aligned llms. arXiv preprint arXiv:2404.14461 , 2024. 4.1, 5

Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1354-1371. IEEE, 2022. 2

Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems , 2023. 1, 3.2

Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771 , 2019. 2, 3.1

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP) , pages 707-723. IEEE, 2019. 1, 2, 3.1

Yuhang Wang, Huafeng Shi, Rui Min, Ruijia Wu, Siyuan Liang, Yichao Wu, Ding Liang, and Aishan Liu. Universal backdoor attacks detection via adaptive adversarial probe. arXiv preprint arXiv:2209.05244 , 2022. 2

Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. Unicorn: A unified backdoor trigger inversion framework. In The Eleventh International Conference on Learning Representations , 2022. 2

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems , 36, 2024. 2

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. 1, 2, 2, 3.1, 3.1, B.1

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems , 34:16913-16925, 2021. 1, 2, 3.1, B.1

Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren. Towards reliable and efficient backdoor trigger inversion via decoupling benign features. In The Twelfth International Conference on Learning Representations , 2023. 1, 2, 3.1, 3.1, 4.2, B.1

Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949 , 2023. 2

Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. In The Twelfth International Conference on Learning Representations , 2024. 2

Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations , 2022. 2, 3.1

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems , 36, 2024. 2

Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4466-4477, 2023. 1, 2, 3.1, B.1, B.2

A Social Impact

The prevalence of Deep Neural Networks (DNNs) in modern society relies heavily on massive amounts of training data from diverse sources. However, in the absence of rigorous monitoring mechanisms, these data resources become susceptible to malicious manipulation, resulting in unforeseen and potentially harmful consequences. Among the various concerns associated with the training dataset, backdoor attacks pose a significant threat. These attacks can manipulate the behavior of a well-trained model by poisoning the training set with backdoored samples, often at a low cost and without requiring complete control over the training process. While existing defense methods have demonstrated effective backdoor purification by achieving low Attack Success Rates (ASR), they still exhibit vulnerabilities that allow adversaries to reactivate the injected backdoor behavior easily. In our work, instead of solely focusing on backdoor ASR, we investigate the effectiveness of modern purification techniques from the perspective of post-purification robustness. We aim to enhance the post-purification robustness of backdoor defense, mitigating the potential for malicious manipulation of deployed models even after backdoor purification. In sum, our work hopes to move an initial step towards improving post-purification robustness while also contributing to another aspect of understanding and enhancing machine learning security.

B Experimental Settings

In this section, we provide detailed information about the experimental settings used in our evaluations. This includes the dataset, training details, and the selection of hyperparameters. All experiments were conducted using 4 NVIDIA 3090 GPUs. We ran all experiments 3 times and averaged all results over 3 random seeds.

B.1 Datasets and Models

We follow previous studies [26, 32, 48, 49] on backdoor learning, and conduct our experiments on three widely used datasets including CIFAR-10, CIFAR-100, and Tiny-ImageNet.

CIFAR-10 is a widely used dataset in the backdoor literature, comprising images with a resolution of 32 × 32 and 10 categories. For backdoor training, we utilize the ResNet-18 model for main evaluation, a commonly used architecture in previous studies [32, 50, 55]. Additionally, we explore other architectures, including the ResNet-50 and DenseNet-161.

CIFAR-100 and Tiny-ImageNet are two large-scale datasets compared to the CIFAR-10, which include 100 and 200 different categories, respectively. Similar to previous work [32, 50], we utilize the pretrained ResNet-18 on ImageNet-1K provided by PyTorch to implement backdoor attacks since directly training from scratch would result in an inferior model performance on C-Acc, hence is not practical in real-world scenarios.

B.2 Implementation Details

Attack Configurations We implement 4 representative poisoning-based attacks and generally follow the implementation from the BackdoorBench 1 . For the BadNet, we utilize the 3 × 3 checkerboard patch as triggers and choose the lower right corner of the image for backdoor injection by default; for the Blended, we adopt the Gaussian Noise as the backdoor trigger. We set the blend ratio to 0 . 1 for backdoor training and increase the blend ratio to 0 . 2 during the inference phase; for SSBA and LC, we follow the original implementation from BackdoorBench without making modifications. In our implementation, we set the default poisoning rate to 5% , which is commonly used in previous studies [32, 55] and additionally explore various poisoning rates including both 1% and 10% . Note that we do not adopt a lower poisoning rate since most of the methods suffer from effectively removing backdoor effects when the poisoning rate is extremely low as indicated by [32]. For all backdoor attacks, the target label is set to be 0 by default.

For CIFAR-10, we adopt an initial learning rate of 0 . 1 to train all the backdoored models for 100 epochs. For both the CIFAR-100 and Tiny-ImageNet, we utilize pretrained backbones and initialize the classifiers with appropriate class numbers. We adopt a smaller learning rate of 0 . 001 and fine-tune

the models for 10 epochs. We upscale the image size up to 224 ∗ 224 during both the training and inference stages following the implementation of [32].

Baseline Defense Configurations We evaluate several mainstream purification techniques including pruning-based defense (ANP), tuning-based defenses (I-BAU, FT-SAM, and FST), and the state-ofthe-art trigger-inversing strategy BTI.

EU: We add real backdoor triggers to 10% of the overall benign tuning samples, and keep their ground-truth labels unchanged. We fine-tune the model on our tuning dataset for 20 epochs.

ANP: We follow the original implementation in BackdoorBench, using the default hyperparameters. Regarding model pruning, we set the threshold range from 0 . 4 to 0 . 9 and report the best purification result with a low ASR and a high model performance on C-Acc.

I-BAU: We follow the implementation in BackdoorBench with default configurations. We set the fixed-point approximation iterations to 5 and fine-tune backdoored models for 20 epochs.

FT-SAM: We follow the implementation from [32] and set the neighborhood size to 1 for CIFAR-10 and 0 . 5 for both the CIFAR-100 and Tiny-ImageNet. We set the initial learning rate to 0 . 01 and decrease the learning rate to 0 . 001 for both the CIFAR-100 and Tiny-ImageNet. We fine-tune backdoored models for 20 epochs on all datasets.

FST: For FST, we follow the original implementation 2 . We set the feature-shift parameter to 0 . 1 and the learning rate to 0 . 01 on CIFAR-10. On the CIFAR-100 and Tiny-ImageNet, we decrease the feature-shift parameter to 0 . 001 and adopt an initial learning rate of 0 . 005 for better C-Acc.

BTI: We follow its original implementation 3 and adopt the BTI (U) since it achieves a better purification performance. We adopt the default hyperparameters for trigger inversion including 20 iterations for decoupling benign features, and 30 iterations for training backdoor generators, and set the norm bound to 0 . 3 by default. We perform BTI (U) until the loss converges.

PAM: On CIFAR-10, we set the path-aware step size to 0 . 5 for the Blended and SSBA; while we increase the step size to 0 . 9 for both the BadNet and the LC attack. We use a step size of 0 . 4 on CIFAR-100 and Tiny-ImageNet to maintain the model performance on C-Acc.

RA Configurations In this section, we provide the detailed experimental setting of the RA in our revisiting Section 3. For the CIFAR-10, we adopt 5 samples for BadNet, 1 samples for Blended, 10 samples for SSBA, and 2 samples for LC attack to perform RA. For both the CIFAR-100 and Tiny-ImageNet, we increase the number of poisoned images to 25 samples for BadNet, 10 samples for Blended, and 35 samples for SSBA to perform RA. We set the learning rate to 0 . 01 for the CIFAR-10 and 0 . 001 for the CIFAR-100 and Tiny-ImageNet to maintain C-Acc. However, directly fine-tuning with these poisoned samples would negatively impact the performance of C-Acc. Therefore, in addition to the backdoored samples, we introduce extra benign samples during the fine-tuning process. We fix the size of the tuning images to 2% of the overall training dataset, which are 1000 samples for CIFAR-10 and CIFAR-100, and 2000 samples for Tiny-ImageNet.

Note that there are primarily two reasons for selecting only a limited number of poisoned examples for fine-tuning during our experiments. Firstly, using a larger number of poisoned samples, such as 10 BadNet samples on CIFAR-10, would significantly increase the ASR of a clean model. This would undermine the reliability of our RA evaluation conducted on the purified models. Secondly, by utilizing only a few poisoned examples, we intentionally expose the purified models to a potential compromise, even if they exhibit a seemingly low ASR. This approach effectively highlights the vulnerability of these purified models, emphasizing their susceptibility to attacks despite achieving a seemingly low ASR.

QRAConfigurations We employ a three-layer Multilayer Perceptron (MLP) to generate the reversed perturbation in the input space. For simplicity, the number of neurons in all internal layers is fixed at 1024 , followed by the Rectified Linear Unit (ReLU) activation function. To train the generator, we

use 500 benign examples and 500 backdoored examples from the CIFAR-10 dataset. Our training process is conducted over 50 epochs with a constant learning rate of 0 . 1 . We set the α to 0 . 1 for the LC attack and 0 . 2 for the others to achieve a high P-ASR on purified models while simultaneously reducing the attack performance on clean models. During the training process of the perturbation generator, we begin by flattening the input image into a one-dimensional vector before feeding it to the generator. Once we obtain the output from the generator, we reshape it back to the original input size. We then multiply it with the pre-defined budget ϵ before we integrate it into the images. In our implementation, we fix the ϵ to 16 / 255 for all experiments.

C Additional Experiments

C.1 Additional Results of RA

Detailed RA performance Our previous experiments in Section 3 only provide the average metrics on CIFAR-10. Therefore, in this section, we provide detailed RA results against each backdoor attack. Specifically, we demonstrate the detailed experimental results of ResNet-18 with 5% poisoning rate in Table 3; ResNet-18 with 10% poisoning rate in Table 4; ResNet-50 with 5% poisoning rate in Table 5 and DenseNet-161 with 5% poisoning rate in Table 6, respectively. Based on these experimental results, our PAM demonstrates effective post-purification robustness against diverse architectures and poisoning rates, which leads to only a small ASR increase (less than 3% on average) after performing RA.

Evaluation of RA under Lower Poisoning Rate In addition to the 5% and 10% poisoning rates used in our previous evaluation, we also include a lower poisoning rate of 1% to assess the performance of RA. Specifically, we experiment on CIFAR-10 and utilize the ResNet-18 for a fair comparison. We exclude the LC attack from our experiments since most defense methods struggle to adequately purify the backdoor behavior. As shown in Table 7, our PAM achieves a tiny increase of the ASR after RA compared to other purification techniques. This further demonstrates the effectiveness of our PAM method in enhancing post-purification robustness.

Evaluation of RA under Other Datasets In this section, we provide additional evaluations on RA against two other datasets. We present our experimental results in Table 8 and Table 9 for the

CIFAR-100 and Tiny-ImageNet, respectively. It is worth noting that our PAM method remains effective when applied to these two datasets, resulting in only a minor increase in ASR.

C.2 Detailed Results of LMC

Detailed LMC performance In addition to the average LMC on CIFAR-10, we provide detailed experimental results for each attack setting. We demonstrate the LMC against ResNet-18 with 5% poisoning rate in Figure 7; ResNet-18 with 10% poisoning rate in Figure 8; ResNet-50 with 5%

poisoning rate in Figure 9 and DenseNet-161 with 5% poisoning rate in Figure 10, respectively. Our experiments demonstrate that the proposed PAM method effectively introduces high error barriers along the backdoor-connected path, leading to a greater deviation from the backdoored models.

LMCunder Lower Poisoning Rate In this section, we evaluate the LMC under a lower poisoning rate, namely 1% . As shown in Figure 11, our PAM still achieves stable high error barriers along the backdoor-connected path compared to other defense methods.

LMC under Other Datasets We further evaluate the LMC on other two datasets, including the CIFAR-100 (shown in Figure 12) and the Tiny-ImageNet (shown in Figure 13). Observations through the experimental results suggest that our PAM is also effective in introducing high error barriers along the backdoor-connected path across diverse datasets.

LMCwith Clean Samples In addition to evaluating the LMC with backdoored examples, we also provide experimental results of the LMC with clean samples, which we refer to as the clean-connected path. As depicted in Figure 14, we observe that diverse purification techniques exhibit almost no clean error barrier along the clean-connected path.

C.3 Additional Results of QRA

In this section, we evaluate the QRA under various attack configurations on the CIFAR-10 dataset. Specifically, we include additional poisoning rates ( 1% and 10% ) in Figure 15, and two model architectures (ResNet-50 and DenseNet-161) in Figure 16. We report the performance against two attacks, namely the BadNet and Blended. Note that for certain defense methods, their ASR after applying RA is low, making it hard to perform QRA evaluations on them. Therefore, we exclude these defense methods from our evaluation and represent them as light gray in the Figure. Experimental

results demonstrate the effectiveness of our QRA on waking backdoor behavior under various attack settings.

C.4 Results of Analyzing Sensitivity of ρ

We present the model performance across various ρ values in Table 10. Our observations reveal that as ρ rises, there is a slight decrease in clean accuracy alongside a significant enhancement in robustness against the RA. Additionally, we note that performance is relatively insensitive to ρ when it exceeds 0.3. Given that we primarily monitor C-Acc (with the validation set) in practice, we aim to

achieve a favorable trade-off between these two metrics. Therefore, we follow the approach of FST [32] and select ρ to ensure that C-Acc remains above a predefined threshold, such as 92% .

C.5 Additional Discussions on the Frequently Asked Question: Directly Applying SAM with BTI

Given that our proposed method exhibits some similarities to the functionality of SAM, a frequently asked question arises regarding whether utilizing reversed backdoor data from BTI in conjunction with SAM would enhance post-purification robustness. To address this question, we further evaluate the performance of directly integrating SAM with BTI (BTI+SAM) as a robust baseline and present a comparative analysis with PAM in Table 11. Our experimental results indicate that the direct combination of BTI and SAM does not achieve the same level of robustness as PAM. This finding underscores the significance of the backdoor-connected pathway between purified and backdoored models, as delineated by the LMC.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Section 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 5

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theory results

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section B

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section B

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Section B

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section B

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Section A

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Section B

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies post robustness of backdoor purification. The authors show that backdoors purified by existing defenses can be recovered via Retuning Attacks and they propose the Query-based Reactivation Attack to recover the backdoor. The authors address such an vulnerability by proposing a Path-Aware Minimization defense. Evaluations are performed on four different attacks over CIFAR-10/100 and Tiny-ImageNet across various deep neural networks. Results show that the proposed PAM defense achieves a better post purification robustness.\n\nSTRENGTHS:\n1. It is novel to study the post purification robustness and this paper reveals an often-neglected vulnerability of DNNs and limitations of existing defenses.\n\n2. The paper progressively demonstrate that models can relearn backdoors after purification via Retuning Attacks and based on the observation, the authors propose a Query based attacks that is more practical under real-world threat models.\n\n3. The finding that the inadequate deviation of purified models from the backdoored model along the backdoor-connected path is the root cause of poor post purification robustness is both Instructive and insightful. The authors successfully develop an effective defense upon this finding.\n\nWEAKNESSES:\n1. My primary concern is that the attacks evaluated in the paper are not SOTA. BadNets, Blended, SSBA and LC are attacks developed years ago. The authors should also provide the results on recent attacks such as Sleeper Agent[1] and Adaptive Blended[2].\n\n2. The authors mentioned there is a trade-off between post purification robustness and the clean accuracy. However, it is unclear how to determine the hyper parameter $\\rho$ to achieve the balance. An algorithm for $\\rho$ selection would be necessary.\n\n3. The presentation needs to be improved. For example, order of Fig.2 and Fig.3 should be interchanged; it is very hard to see the results in Fig. 5;  Algorithm 1 should be placed close to where it is described.\n\n4. Some typos. For example, line 142: we following -> we follow;  in Figure 1, shouldn't P-ASR be R-ASR? \n\n[1]. Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch, NeurIPS, 2022\n\n[2]. REVISITING THE ASSUMPTION OF LATENT SEPARABILITY FOR BACKDOOR DEFENSES, ICLR 2023\n\nQUESTIONS:\n1. In Table 2, the clean accuracy of all these models are lower than usual, why is that?\n\nLIMITATIONS:\nLimitations has been addressed in \"Conclusions and Limitations\".", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

comments: SUMMARY:
This paper investigates the effectiveness of current purification-based backdoor defenses and tries to uncover whether purified DNNs are truly free from backdoor vulnerabilities. The authors identify the “post-purification robustness” of DNNs and propose Retuning Attack (RA) and Query-based Reactivation Attack (QRA) respectively to assess the susceptibility of purified DNNs to backdoor reactivation. Additionally, the paper proposes Path-Aware Minimization (PAM) to improve post-purification robustness.

STRENGTHS:
The work makes contribution in backdoor robustness evaluation by shifting the focus from merely achieving low ASR to evaluating the post-purification robustness of backdoor defenses. The paper introduces a pipeline of methods: RA, QRA, and PAM, for the assessment of this vulnerability.

WEAKNESSES:
The proposed notion of post-purification robustness lies with the assumption that the purified DNN will encounter further fine-tuning. As such, the entire workflow proposed is not practical in a real-world scenario where the purified model is kept frozen with no further updates.

QUESTIONS:
Can the authors further justify the proposed threat model’s real-world implications? How can the proposed workflow be enabled in a real-world attack setting?

LIMITATIONS:
The paper falls short in addressing practical implementation challenges. These limitations suggest that the proposed methods might not be as universally practical as claimed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reveals a phenomenon in backdoor defense: the purified backdoor can be reactivated by fast retuning on a few backdoor samples. Building upon this observation, the paper explores both attacks and defense measures for more reliable backdoor research. On the attack side, a Retuning Attack (RA) is proposed and generalized to the black-box setting as a Query-based Reactivation Attack (QRA). On the defense side, a Path-Aware Minimization (PAM) method is proposed to force more deviation from the backdoor-connected path. Experiments verify the effectiveness of PAM compared to exact purification (EP).\n\nSTRENGTHS:\n1. The observed phenomenon that the purified model is more easily to restore the trigger is interesting.\n\n2. Both attacks and defense were explored.\n\n3. The defense results look promising.\n\nWEAKNESSES:\n1. The restoration of the backdoor was conducted on a poisoned subset which is not a surprise in this case as the model will surely relearn the backdoor. The authors should prove the phenomenon via tuning on purely clean training data. \n\n2. The threat models of the two proposed attacks are problematic. 1) RA will require a post-purification poisoning to the defense model, which means that it can poison again after the defense, which does not make sense to me; 2) QRA attack is very similar to an adversarial attack, that requires access to both the purified model and the RA model, why the defender exposes these models (or their APIs) to the attacker is questionable. \n\n3. The authors should clearly define what is Post-purification Robustness. In the current version, it appears the same as standard backdoor robustness, i.e., how to guarantee the PAM purified model is 100% robust?\n\n4. The authors seem to confuse a backdoor attack with adversarial perturbation in proposing the QRA attack, A backdoor attack does not make changes to the input during the inference stage otherwise it becomes an adv attack. These two attack types assume different capabilities and flexibility of the adversary.\n\n5. The proposed PAM is very much like using a moving average to force larger updates of the weights. PAM requires reverse-engineered backdoor samples $\\mathcal{D_r}$, I wonder if one could simply use unlearning to achieve the same effect.\n\n6. The proposed method was not compared with existing defense methods.\n\nQUESTIONS:\n1. Can PAM deal with multi-trigger attacks?[1]\n\n[1] Li, Yige, et al. \"Multi-Trigger Backdoor Attacks: More Triggers, More Threats.\" arXiv preprint arXiv:2401.15295 (2024).\n\n2. How does PAM work, compared to applying the same existing defense twice, e.g., ANP or SAM?\n\n3. Can the problem be addressed by simply adjusting the hyperparameters of the Mode Connectivity defense [2]?\n\n[2] Zhao, Pu, et al. \"Bridging mode connectivity in loss landscapes and adversarial robustness.\" arXiv preprint arXiv:2005.00060 (2020).\n\nLIMITATIONS:\n1. Problematic threat model.\n2. Limited technical novelty.\n3. Missing systematic comparison with existing methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

comments: SUMMARY:
Backdoor attacks are a major threat to Deep Neural Networks (DNNs), as they allow attackers to manipulate model predictions with backdoor triggers. Existing purification methods reduce the Attack Success Rate (ASR) of these models, but it's unclear if they fully eliminate backdoor threats. This study investigates post-purification robustness by employing the Retuning Attack (RA) and finds that current methods are vulnerable, as models quickly relearn backdoor behaviors. To address this, the study proposes the Query-based Reactivation Attack (QRA) and a novel Path-Aware Minimization (PAM) technique. PAM enhances robustness by promoting deviation along backdoor-connected paths with extra model updates. Extensive experiments show PAM significantly improves robustness, maintaining low ASR and good accuracy, providing a new perspective on evaluating and improving backdoor defenses.

STRENGTHS:
1. The paper empirically verifies that poisoned nodes typically exhibit large prediction variance under edge dropping, providing a indicator for identifying poisoned nodes​​.
2. The proposed robust training strategy not only has theoretical guarantees but also shows practical effectiveness in defending against various types of backdoor attacks, maintaining clean accuracy while reducing the attack success rate​​.

WEAKNESSES:
1. While the paper focuses on various backdoor attack types, the scope of attack types and defense mechanisms explored could be broadened to cover more diverse scenarios and setting

QUESTIONS:
This is a good paper, discussing the details of the proposed defense method, one thing I'm curious is that does the author has plans for designing an adaptive attack for this defense method, and how to achieve it?

LIMITATIONS:
The limitations have been discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16