Uncovering, Explaining, and Mitigating the Superficial Safety of Backdoor Defense
Authors: Rui Min
Zeyu Qin
Nevin L. Zhang
Li Shen
Minhao Cheng
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Backdoor Safety
Safety Tuning
Superficial Safety
Primary area: safety_in_machine_learning

Abstract: Backdoor attacks pose a significant threat to Deep Neural Networks (DNNs) as they allow attackers to manipulate model predictions with backdoor triggers. To address these security vulnerabilities, various backdoor purification methods have been proposed to purify compromised models. Typically, these purified models exhibit low Attack Success Rates (ASR), rendering them resistant to backdoored inputs. However, \textit{Does achieving a low ASR through current safety purification methods truly eliminate learned backdoor features from the pretraining phase?} In this paper, we provide an affirmative answer to this question by thoroughly investigating the \textit{Post-Purification Robustness} of current backdoor purification methods. We find that current safety purification methods are vulnerable to the rapid re-learning of backdoor behavior, even when further fine-tuning of purified models is performed using a very small number of poisoned samples. Based on this, we further propose the practical Query-based Reactivation Attack (QRA) which could effectively reactivate the backdoor by merely querying purified models. We find the failure to achieve satisfactory post-purification robustness stems from the insufficient deviation of purified models from the backdoored model along the backdoor-connected path. To improve the post-purification robustness, we propose a straightforward tuning defense, Path-Aware Minimization (PAM), which promotes deviation along backdoor-connected paths with extra model updates. Extensive experiments demonstrate that PAM significantly improves post-purification robustness while maintaining a good clean accuracy and low ASR. Our work provides a new perspective on understanding the effectiveness of backdoor safety tuning and highlights the importance of faithfully assessing the model's safety.

Uncovering, Explaining, and Mitigating the Superficial Safety of Backdoor Defense
Rui Min", Zeyu Qin', Nevin L. Zhang', Li Shen, Minhao Cheng²
Hong Kong University of Science and Technology. PPennsylvania State University {rminaa, zeyu, gin]@cconeecc ust hk, 1zhangocse. ast.ht. mathshenl bathssenll@gmail.. com, minhaochengêuss. hk
Abstract
Backdoor attacks pose significant threat to Deep Neural Networks (DNNs) they allow attackers to manipulate model predictions with backdoor triggers. To address these security vulnerabilities, various backdoor purification methods have been proposed to purify compromised models. Typically, these purified models exhibit low Attack Success Rates (ASR), rendering them resistant backdoored inputs. However, Does achieving low ASR through current safety purification methods truly eliminate learned backdoor features from the pretraining phase? In this paper, we provide an affirmative answer to this question by thoroughly investigating the Post-Purification Robustness of current backdoor purification methods. We find that current safety purification methods are vulnerable to the rapid re-learning of backdoor behavior, even when further fine-tuning of purified models is performed using very small number of poisoned samples. Based on this, we further propose the practical Query-based Reactivation Attack (QRA) which could effectively reactivate the backdoor by merely querying purified models. We find the failure to achieve satisfactory post-purification robustness stems from the insufficient deviation of purified models from the backdoored model along the backdoor-connecte path. To improve the post-purification robustness, we propose straightforward tuning defense, Path-Aware Minimization (PAM), which promotes deviation along backdoor-connect tensive experiments demonstrate that PAM significantly improves post-purificans purifiicaian robustness while maintaining a good clean accuracy and low ASR. Our work provides perspective understanding the effectiveness of backdoor safety tuning and highlights the importance of faithfully assessing the model's safety.
paths with extra model updates. Ex-
1 Introduction
Backdoor attacks [2, 7, 16] have emerged as one of the most significant concerns [4, 20, 25] in deep learning. These attacks involve the insertion of malicious backdoor triggers into the training set, which can further exploited to manipulate the behavior of the model during the inference stage. To defend against these threats, researchers have proposed various safety tuning methods [20, 27, 32, 44, 49, 50, 55) to purify well-trained backdoored models. These methods can be easily incorporated into the existing model deployment pipeline and have demonstrated ofte-oofthhe-an effectiveness in reducing the Attack Success Rate (ASR) of backdoored models [32, 48].
However, a critical question arises: does achieving low Attack Success Rate (ASR) through current safety tuning methods genuinely indicate the complete removal of learned backdoor features from the pretraining phase? If the answer is no, this means that the adversary may still easily reactivate the
Equul contribution. Correspondence to: Zeyu Qin (zeyu. qin@connect. ust. hk), Li Shen, Minhao Cheng.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

implanted backdoor from the residual backdoor features lurking within the purified model, thereby exerting insidious control over the model's behavior. This represents a significant and previously unacknowledged safety concern, suggesting that current defense methods may only offer superficial safery [1]. Moreover, if an adversary can successfully re-trigger the backdoor, it raises another troubling question: how can we assess the model's robustness against such threats? This situation underscores the urgent need for a more comprehensive and faithful evaluation of the model's safety.
In this work, we provide an affirmative answer to these questions by thoroughly investigating the Post- Purification Robustness of state-of-the- backdoor safety tuning methods. Specifically, we employ the Retuning Attack (RA) [36, 42] where we first retune the purified models using an extremely small number of backdoored samples and tuning epochs. Our observations reveal that current safety purification defense methods quickly reacquire backdoor behavior after just few epochs. resulting in significantly high ASR levels. In contrast, the clean model (which does not have backdoor triggers inserted during the pretraining phase) and Exact Purification (EP)-which fine-tunes models using real backdoored samples with correct labels during safety purification, maintain low ASR even after the RA. This discrepancy suggests that existing safety tuning methods do not thoroughly eliminate the learned backdoor, creating a superficial impression of backdoor safety. Since the vulnerability revealed by the Retuning Attack (RA) relies on the use of retuned models, we further propose the more practical Query- based Reactivation Attack (QRA). This attack is capable of generating sample-specific perturbations that can trigger the backdoor in purified models, which were previously believed to have eliminated such threats, simply by querying these purified models.
To understand the inherent vulnerability of current safety purification methods concerning post- purification robustness, we further investigate the factors contributing to the disparity in post- purification robustness between EP and other methods. To this end, we utilize Linear Mode Connec- tivity (LMC) [13, 33] as a framework for analysis. We find that EP not only produces a solution with low ASR like other purification methods but also pushes the purified model further away from the backdoored model along the backdlor--connee
path, resulting in a more distantly robust solution. As result, it becomes challenging for the retuning attack to revert the EP model back to the basin with high ASR where the compromised model is located. Inspired by our findings, we propose a simple tuning defense method called Path-Aware Minimization (PAM) to enhance post-purification robustness. By using reversed backdoored samples as a proxy to measure the wockdoored-connectee path, PAM updates the purified model by applying gradients from model interpolated between the purified and backdoored models. This approach helps identify a robust solution that further deviates our purified model from the backdoored model along the mackdoor-connected path. Extensive experi- ments have demonstrated that PAM achieves improved post-purification robustness, retaining a low ASR after RA across various settings. To summarize, our contributions are:
. Our work first offers new perspective on understanding the effectiveness of current backdoor safety tuning methods. Instead of merely focusing on the commonly used Attack Success Rate, we investigate the Post-Purification Robustness of the purified model to enhance our comprehensive understanding of backdoor safety in deep learning models.
We employ the Retuning Attack by retuning purified models on backdoored samples to assess the post-purification robustness. Our primary observations reveal that current safety purification methods are vulnerable to RA, as evidenced by a rapid increase in the ASR. Furthermore, we propose the more practical Query-based Reactivation Attack, which can reactivate the implanted backdoor of purified models solely through model querying.
We analyze the inherent vulnerability of current safety purification methods to the RA through Linear Mode Connectivity and attribute the reason to the insufficient deviation of purified models from the backdoored model along the backdoor-connected path. Based on our analysis, we propose Path Aware Minimization, straightforward tuning-based defense mechanism that promotes deviation by performing extra model updates using interpolated models along the path. Extensive experiments verify the effectiveness of the PAM method.
2 Related Work
Backdoor Attacks. Backdoor attacks aim to manipulate the backdoored model to predict the target label on samples containing& a specific backdoor trigger while behaving normally on benign samples. They can be roughly divided into two categories [48]: (1) Data-poisoning attacks: the attacker inserts

---

backdoor trigger into the model by manipulating the training sample (æ, y) E (x,)"), such as adding a small patch to a clean image x and change the sample's label to an attacker-designated target label Yt [4, 7, 15, 16, 28, 43]; (2) Training-control attacks: the attacker has control over both the training process and the training data simultaneously [34]. Note that cata-poisoning attacks are more practical in real-world scenarios as they make fewer assumptions about the attacker's capabilities [5, 15, 41] and have resulted in increasingly serious security risks [4, 39J. In this work, we focus on data dais-onng backdoor attacks.
Backdoor Defense. Existing backdoor defense strategies could be broadly categorized into robust pretraining [19, 26] and robust fine-tuning methods [32, 49, 53, 55J. Robust pretraining aims to prevent the learning of backdoor triggers during the pretraining phase. However, these methods often suffer from accuracy degradation and can significantly increase model training costs, making them impractical for large-scale applications, In contrast, robust purification methods focus on removing potential backdoor features from a well-trained model. Generally, purification techniques involve reversing potential backdoor triggers [44, 45, 46, 50] and applying fine-tuning or pruning to address backdoors using& limited amount of clean data [29, 32, 49, 551. While these purification methods reduce training costs, they also achieve state-of the-te--thtrr art defense performance [32, 55]. Therefore, in this work, we mainly focus on evaluutions of robust purification methods against buckdoor allacks.
Loss Landscape and Linear Mode Connectivity. Early works [12, 14, 22] conjectured and empirically verified that different DNN loss minima can be connected by low-loss curves. In the context of the pretrain-fine- finnannn-----eneeeee paradigm. Neyshabur al. [33] observe that the pretrained weights guide purified models to the same flat basin of the loss landscape, which is close to the pretrained checkpoint. Frankle al. [13] also observe and define the linear case of mode connectivity, Linear Mode Connectivity (LMC). LMC refers to the absence of the loss barrier when interpolating linearly between solutions that are trained from the same initialization. The shared initialization can either be a checkpoint in early training [13] or a pretrained model [33]. Our work is inspired by [33] and also utilizes LMC to investigate the properties of purified models in relation to backdoor safety.
Deceptive AI and Superficial Safety. Nowadays, DNNs are typically pretrained on large-scate datasets, such as web-scraped data, primarily using next-token prediction loss [3], along with simple contrastive [6] and classification [23] objective. While these simplified pretraining objectives can lead to the learning of rich and useful representations, they may also result in deceptive behaviors that can mislead humans [10]. One such deceptive behavior is the presence of backdoors [9, 20]. A compromised model can be indistinguishable from normal model human supervisors, as both behave similarly in the absence of the backdoor trigger. To address this critical safety risk, researchers propose post-training alignment procedures, such as safety fine-tuning [35, 48). However, several studies indicate that the changes from fine-tuning are superficial [31, 54]. As result, these models retain harmful capabilities and knowledge from pretraining, which can be elicited by harmful fine-tuning [37, 51] or specific out-of- -dd-tiiiiiiiitionn (OOD) inputs [47, 52]. We study this phenomenon in the context of backdoor threats and offer a deeper understanding along with mitigation strategies.
3 Revealing Superficial Safety of Backdoor Defenses by Accessing Post-purification Robustness
While current backdoor purification methods can achieve a very low Attack Success Rate (ASR) against backdoor attacks, this does not necessarily equate to the complete elimination of inserted backdoor features. Adversaries may further exploit these residual backdoor features to reconstruct and reactivate the implanted backdoor, as discussed in Section 3.3. This is particularly important because purified models are often used in various downstream scenarios, such customized fine-tuning [37] for critical tasks [20]. Therefore, it is crucial to provide a way to measure the robustness of purified models in defending against backdoor re-triggering, which we define "Post-Purification Robustness".
In this section, we first introduce a simple and straightforward strategy called the Retuning Attack (RA) to assess post-purification robustness. Building on the RA, we then present a practical threat known as the Query-based Reactivation Attack (QRA), which exploits the vulnerabilities in post- purification robustness to reactivate the implanted backdoor in purified models, using only model querying. First, we will introduce the preliminaries and evaluation setup.
3

---

3.1 Problem Setup
Backdoor Purification. In this work, we focus on the poisoning baisoning-based attack due its practicality and stealthiness. We denote the original training dataset as Dr C (A))'). A few training examples (xr y) E Dr have been transformed by attackers into poisoned examples (Ip, yt), where zy is poisoned example with inserted trigger and a target label yt. Following previous works [29, 32, 39, 48, 49], only a limited amount of clean data Dr are used for fine-tuning or pruning. For trigger- inversion methods [44, 50], we denote the reversed backdoored samples obtained through reversing methods as (a.. y) E D,.. We evaluate several mainstreamed purification methods, including pruning-based defense ANP [49]; robust fine-tuning defense I-BAU [53] (referred to BAU for short), FT-SAM [55] (referred to as SAM for short), FST [32], as well as the state-of-the-art trigger- reversing defense BTI-DBF [50] (referred to BTI for short). BTI purifies the backdoored model by using both reversed backdoored samples D, and the clean dataset Dr while the others use solely the clean dataset D.. We also include exact purification (EP) that assumes that the defender has full knowledge of the exact trigger and fine-tunes the models using real backdoored samples with correct labels pxp: y)-
Attack Settings. Following [32], we evaluate four representative data-poisoning backdoors in- cluding three dirty-label attacks (BadNet [16], Blended [7]. SSBA [28]), and one clean-label attack (LC [43]) All experiments are conducted on BackdoorBench [48]. a widely used benchmark for backdoor learning. We employ three poisoning rates, 10%, 5%, and 1% (in Appendix) for back- door injection and conduct experiments on three widely used image classification datasets, including CIFAR-10 [24]] Tiny-ImageNer [8], and CIFAR- 100 [24]. For model architectures,
following [32], and adopt the ResNet- 18, ResNet-50 [17]. and DenseNet-166 [18] on CIFAR-10. For CIFAR-100 and Tiny-ImageNet we adopt pretrained ResNet- 18 on ImageNetIK to obtain high clean accuracy as suggested by [32, 50]. More details about experimental settings are shown in Appendix B.
Evaluation Metrics. Following previous backdoor works, we take two evaluation metrics, including Clean Accuracy (C-Ace) (i.e., the prediction accuracy of clean samples) and Attack Success Rate (ASR) (i.e., the prediction accuracy of poisoned samples to the target class) where a lower ASR indicates a better defense performance. We further adopt O-ASR and P-ASR metrics. The O-ASR metric represents the defense performance of original defense methods, while the P-ASR metric indicates the ASR after applying the RA or QRA.
3.2 Purified Models Are Vulnerable to Retuning Attack
Our objective is to investigate whether purified models with low ASR completely eliminate the inserted backdoor features. To accomplish this, it is essential develop a method for assessing the degree to which purified models have indeed forgotten these triggers. In this section, we begin with a white-box investigation where the attacker or evaluator has access to the purified model's parameters. Here we introduce a simple tuning-based based strategy named the Retuning Attack (RA) [37, 42] to conduct an initial evaluation. Specifically, we construct a dataset for model retuning, which comprises a few backdoored samples (less than 1% of backdoored samples used during the training process). To maintain C-Ace, we also include benign samples from the training set, resulting in a total RA dataset with 1000 samples. We subsequently retune the purified models using this constructed dataset through few epochs (5 epochs in our implementation). This approach adopted because a clean model can not be able to learn backdoor; thus, if the purified models quickly regain ASR during the retuning process, it indicates that some residual backdoor features still exist in these purified models. Implementation details of the RA can be found in the Appendix B.2.
As shown in Figure 1, we observe that despite achieving very low ASR, all purification methods quickly recover backdoor ASR with Retuning Attack. Their quickly regained ASR presents stark contrast to that of clean models and remains consistent across different datasets, model architectures, and poisoning rates. Note that the pruning method (ANP) and the fine-tuning method (FST), which achieve state-of sate-of-the-art defense performance, still exhibit vulnerability to RA, with an average recovery of approximately 82% and 85% ASR, respectively. In stark contrast, the EP method stands out as il consistently maintains a low ASR even after applying RA, demonstrating exceptional post- purification robusiness. Although impractical with full knowledge of the backdoor triggers, the HP method validates the possibility of maintaining a low attack success rate to ensure post- postfuuriiicttion robustness against RA attacks.

---

C FAR 10 ResNet 18 5%
CIFAR 10 ResNet 18 10%
CIFAR 10 ResNet 50 5%
P-ASR
P-ASR
P-ASR
80
60
40
ANP
PST
PST 87
F5T
CIFAR-10 DenseNet-161 5%
CIFAR-100 ResNet-18 5%
TINY ResNet-18 5%
100
D-ASR
D-ASR
D-ASR
80
60
0
Clean
The robustness performance against various attack settings. The title consists of the used dataset, model, and poisoning The O-ASR metric represents the defense performance of original defense methods, while P-ASR metric indicates ASR after applying Here we report the average results among backdoor attacks and defer more details in Appendix C.I.
Figure 1:
metrics are measured in percentage (%)
BodNet
Blended
SSBA
LC
100
CASR (hu fuud)
CLddit (hurifed))
C-ASRt (Purifed))
C-5R (Purffud))
PaSSit (Purliedt
hasi tharlieall
Pobsit (hu fiedi C-Aoi (Clenn
Pasi (Parlisall
80
CASH (Dlean)
C-AiR (Chear)
Ciedit (Cear))
60
P-ASIL (Cleani
PASS cann
P-ASK (C can)
P-ASit (Clean)
40
20
0
Figure 2: Experimental results of QRA both the purified and clean models against four types of backdoor attacks. evaluate QRA on CIFAR-10 ResNet- 18 the poisoning rate to 5%. Additional results of QRA are demonstrated in Appendix C.3.
Moreover, this evident contrast highlights the sigmificant security risks associated with current back- door safety uning methods. While these methods may initially appear robust due to a significantly reduced ASR, they are fundamentally vulnerable backdoor reactivation, which can occur with just a few epochs of model tuning. This superficial safety underscores the urgent need for more comprehensive evaluations to ensure lasting protection against backdoor attacks. is crucial to implement faithful evaluations that thoroughly assess the resilience of purified models, rather than relying solely on superficial metrics, to truly safeguard against the persistent threat of backdoor vulnerabilities.
3.3 Reactivating Backdoor on Purified Models through Queries
Although our previous experiments on RA demonstrate that current purification methods insufficiently eliminate learned backdoor features, it is important to note that the success of this tuning- based method relies on the attackers' capability to change purified models' weights. This is not practical real-world threat model. To address this limitation, we propose Query-based Reactivation Attack (QRA), which generates sample-specific perturbations that can reactivate the backdoor using only model querying. Specifically, instead of directly retuning purified models, QRA captures parameter changes induced by the RA process and translates them into input space as perturbations. These perturbations can then be incorporated into backdoored examples, facilitating the successful reactivation of backdoor behaviors in purified models.
5

---

To effectively translate the parameter changes into the input space, it is crucial to ensure that when upplying the perturbation generated by QRA, the oulput of the purified model on perturbed inputs should be aligned with that of the post-RA model on original inputs without perturbations. Formally, we denote the purified model as F(Wp: the model after RA as WWra: a:) and their corresponding logit output as L(Wp: p;
.x) and I(W,,. ra: .)). Our QRA aims to learn perturbation
generator ((0;z) x) Rd + |-1, 1, 1j to produce perturbation Φ(0; æ:) for each input We formulate this process into the following optimization problem:
min (Rka),,1S(((()) rai u,
(0) + =))]]-
(I)
Here S the distance metric between two output logits, De a compositional dataset incorporating both benign and backdoored samples, and é controls the strength of perturbation (e = 16 16/255 in our implementation). We utilize the Kullback-Leibler (KL) divergence [11] for S and Multilayer Perception (MLP) for < æ)) Specifically, we flatten the input image into a one-dimensional vector before feeding into the Φ(0; @ x;), and obtain the generated perturbation by reshaping it back to the original size. Details of the MLP architecture and training hyperparameters can be found in the Appendix B.2.
However, we observe that directly optimizing Equation not only targets purified models but also successfully attacks the clean model. We conjecture that this may stem from the inaccurate inversion of reactivated backdoor triggers, which can exploit a clean model in a manner similar to adversarial examples [21, 30, 38]. To mitigate such adversarial behavior, we introduce regularization term aimed at minimizing backdoor reactivation on the clean model. Given that accessing the clean model e; x)) as a surrogate model instead. In sum, we formulate the overall optimization objective as follows:
may not be practical, we utilize the EP model f(W;;
min (BI..)) ra; x) LW pi E* (00; + ((f(W;; (00; x), wll]. (2)
0
where a represents the balance coefficient and the cross-entropy loss is used for L.
We demonstrate our experimental results against five purification methods on CIFAR-10 in Figure 2. Here, we report the C-ASR and P-ASR, which represent the ASR when evaluating with perturbed clean and perturbed poisoned images, respectively. Notably, our QRA could effectively reactivate the backdoor behaviors on purified models, resulting in a significant increase of 66.13% on average in P-ASR. Our experiments also demonstrate consistently low C-ASR on purified models, which indicates that the perturbations generated by QRA effectively reactivate the backdoored examples without affecting the predictions of benign images. Besides, the perturbation generated with QRA exclusively works on the output of backdoored samples on purified models, leading to both a low clean models. This observation further indicates that the reversed pattern generated by QRA is not a typical adversarial perturbation but rather an accurate depiction of the parameter changes necessary for backdoor reactivation.
C-ASR and P-ASR
Furthermore, it is worth noting that attackers may lack knowledge about the specific back- door defense techniques utilized by the defender in practice. Thus, we embark an initial in- vestigation to explore the transferability of our QRA method across unknown purification meth- ods. Specifically, we aim to determine whether the reversed perturbations optimized for one par- ticular defense method can effectively attack pu- rified models with other purification techniques. As shown in Figure 3, our QRA demonstrates degree of successful transfcrability across var- ious defense techniques, achieving an average P-ASR of 32.1% against all purification tech- niques. These results underscore the potential of QRA to attack purified models, even without prior knowledge of the defense methods em- ployed by defenders. It also highlights the prac- tical application of QRA in real-world situations.
Transferability Analysis
100
BadNet
Blended
80
SSBA
LC
60
40
20
0
ANP
BAU
SAM
FST
BTI
Figure 3: The results the QRA transferabiliity The defense method used in the attack is represented on the æ-axis, while the y-axis shows the average P-ASR across other purifications.

---

CIFAR-10 Reeeeeeteett8S-
CIFAR-10 CIFAR-10_ResNet-1BB ResNet-18 1aaett18811_ 8_10%
CIFAR-10 11AAR-10RResNet--0 ResNet-50_54
Backdoor Error ASRR)
100
80
EP
EP
EP
60
ANP
ANP
ANP
BAU
BAU
BAU
40
FST
FST
FST
20
B11 PAM(Ours)
871 PAM(OUrs)
871 PAM(OUrs)
0.1 0.2 0.3 0.5 0.6 0.8
0.1 0.2 0.3 0.4 0.5 0.6 0 0.8
0.1
0.2 0.3 0.5 0.6 7 0.8
Interpolation Ratio
Interpo ation Ratio
Interpo ation Ratio
CIFAR: 10 DenseNet 161 161_5% 5%
CIFAR ResNet 18
TINY ResNet 18_5%
Backdoor Error (1 ASR)
100
EP ANP
SAM
SAM
SAM
FST
FST
FST
20
PAM(Ours)
PAM(Ours)
PAM(Ours)
Interpolation Ratio
Interpo ation Ratio
Interpolater cttrpoaation Ratio
Figure 4: The evaluation of backdoor-connecte path against various attack settings. The x-axis and y-axis denote the interpolation ratio and backdoor error (I-ASR) respectively. For each attack setting, report the average results among backdoor attacks.
Investigating and Mitigating Superficial Safety
4.1 Investigating the Superficial Safety through Linear Mode Connectivity
While our previous evaluations indicate that only the EP model demonstrates exceptional post- purification robustness compared current backdoor safety tuning methods, the factors contributing to the effectiveness EP remain unclear. Motivated by prior studies examining fine-tur menetuned models [13, 14, 33], we propose to investigate this intriguing phenomenon from the perspective of the loss landscape using Linear Mode Connectivity (LMC).
Following [13, 33], let E(W; represent the testing error of model f(W evaluated dataset Dj. For Dj. we use backdoor testing samples. WWW.o. W1; D,) E ())1 + +W;
for & E [0, 1] is defined as the error path of model created by linear interpolation between the o; f(Wo; and f(Wi; æ).. We also refer it as the backdoor-connectedd path. Here we denote the
-( as backdoored model and f f(W1:)) -) as the purified model. We show the LMC results of the backdoor error Figure 4 and For each attack setting, we report the average results among backdoor attacks. More results on other datasets and models are shown in Appendix C.2.
The backdoored model and purified models reside in separate loss basins, linked by backdoor- connected path. We present the results of LMC between purified and backdoored models in Figure It clear from the results that all purified models exhibit significant error barriers along the path to backdoored model. This indicates that backdoored and purified models reside in different loss basins. Additionally, we conduct LMC between purified models with EP and with other defense techniques, as depicted in Figure 5. We observe consistently high error without barriers, which indicates that these purified models reside within the same loss basin. Based on these two findings, we conclude that backdoored and purified models reside in two distinct loss basins connected through a backeoor-cooneetedd path.
EP deviates purified models from the backdoored model along the backdoor-connected path, resulting in a more distantly robust solution. Although introducing high loss barrier, we observe notable distinctions between the LMC of the EP model (red solid line) and purified models (dotted lines). We observe stable high backdoor error along the backdoor-connected path of EP until / < 0.2, where the interpolated model parameter W has over 80% weight from the backdoored model. In

---

BodNec
Blended
Backeoor Enart) ASR)
100
80
60
60-
A/F
AMP
A6P
AMP
40
5AU
B6J
BAU
SAU SAM
5AM
20
04:00
Interpolation Radio
Intorpolation Rabio
Interpo ation Ratio
Interpotation Radio
Figure 5: The LMC path connected from other defense techniques to EP. We evaluate the LMC results on CIFAR-I0 with ResNet-18 18, and set the poisoning rate to 5%.
contrast, other purification models show a tendency to exhibit significant increases ASR along the path, recovering more than 20% ASR when I. < 0.5, while the ASR for the EP model remains low (I 2%). This clear contrast suggests: 1) the current purification methods prematurely converge to non-robust solution with low ASR, which is still close the backdoored model along the backdoor- connected path; 2) compared with purified models, EP makes the purified model significantly deviate from the backdoored checkpoint along the backdoor-comme solution against RA.
path, resulting in more robust
Accurately specified supervision is crucial for achieving stable backdoor safety. As demon- strated in our observations, the EP method attains stable robustness in the context of the RA, whereas its proxy version, the BTI method, employs reversed backdoor data as a substitute for real backdoor data, resulting in suboptimal post- pest-uurification robustness. Furthermore, notable discrepancies are evident in the Backdoor LMC results. These findings underscore that current methods for reversing backdoor triggers are still unable to accurately recover all backdoor features [40], thereby emphasiz- ing the importance of precisely specified supervision achieving stable backdoor safety. Although data generated by the BTI method does not accurately recover all backdoor features, it could serve as an effective and usable supervision dataset. In the following section, we propose an improved safety tuning method designed to mitigate superficial safety concerns based on this proxy dataset.
4.2 Enhancing Post-Purifiiation Robustness Through Path-Aware Minimization
Motivated by our analysis, we propose a simple tuning defense method called Path-Aware Mini- mization (PAM), which aims to enhance post-purification robustness by promoting more deviation from the backdoored model along the backeoor-connected path like the EP method.
Since there are no real backdoor samples 2tp available, we employ the synthetic backdoored samples V, from the trigger-reversing method BTI [50] as a substitute to get the backdoor-connected path. Although BTI has similar LMC path curve with the EP model in Figure 5, as we have discussed, tuning solely with æ, would lead to finding a non-robust solution with low ASR.
To avoid converging to such a solution, we propose utilizing the gradients of an interpolated model Wa between Wo and W to update the current solution W. As illustrated in Figure 4, the interpolated model, which lies between W, and W, exhibits a higher ASR compared to W. By leveraging the gradients from the interpolated model, we can perform additional updates prevents premature convergence towards local minima and results in a solution that deviates from the backdoored model along this path. Specifically, for W, we first take a path-aware step p P[Wulla
the W which
W.
(W, Wo W) towards to Wo and obtain the interpolated model W + P PTaa Then we compute its gradient on s, to update W. We formulate our objective function as follows:
Wa W 112 ;<), 811 s.t. W = Wo W,
min (.a.))-D.UD,L(((W + W
(3)
where P represents the size of the path-aware step. Typically, larger p indicates larger step towards the backdoored model W, along our backdoor-connected path and also allows us to obtain larger gradient update for W. which results in more deviation from the backdoored model along the backdoor-connecte path. The detailed algorithm is summarized in the Algorithm 1.
8

---

Table 1: The post-purne custfpuriiictiion robustness performance of PAM on CIFAR-IO. The O-Backdoor indicates the original performance of backdoor attacks, O-Robustness metric represents the purification performance of the defense method, and the P- Robustmess metric denotes the post robustness after applying RA. All metrics are measured in percentage (9)).
Method
Evaluation Mode
BadNet
Blended
SSBA
LC
Avg
C-Aecrt)
ASR(U]
C-Aectt)
ASR(J]
C-Aectt)
ASR(J)
C-Aec())
ASR())
C-Aee(t)
ASR(J)
O Barkddor
94.04
99.99
94.77
100.0
95.60
96.38
94.86
99.99
94.57
99.09
O R.Robbastness (EP)
93.57
0.94
93.47
3.37
93.39
0.53
93.64
0.52
95.52
1.34
ResNet18 (5%)
P-Robustness (EP)
92.27
1.08
93.36
2.47
93.09
92.84
2.22
O-Robusiness (PAM)
92.11
92:32
4.92
92.68
2.24
p. P-Robusteess (PAM)
91.66
3.90
92:15
9.31
92:35
4.55
O-Backdoor
93.73
100.0
94.28
100.0
94.31
98.71
100.0
92.03
99.68
O-Rohuseness (EP)
92.78
0.93
93.
1.52
93.10
0.71
92:12
0.16
92.84
0.83
ResNet18 (10%)
P. RoRbusinesss (EP)
92.17
1.58
93.06
4.78
92.01
3.70
91.79
2.03
92.27
3.02
D-Robustesss (PAM)
92.43
1:75
92.63
0.22
92.89
1.37
91.06
2.31
92.25
1.41
P-Robusmess (PAM)
91.77
1,47
3.37
91.65
3.23
O-Backdoor
O-Kobusttess (EP)
92.84
0.98
92.10
91.84
92.91
92.42
0:74
ResNet50 (5%)
P. P.obbustnesss (EP)
92:33
1.48
91.36
3.51
89.26
6.28
92.33
1.71
91.32
3.25
O-Robusiness (PAM)
92.58
0.94
92.59
0.24
92:36
1.62
91.99
2.14
92:38
1.23
P. Kobustness (PAM)
91.49
1.46
92.75
0.64
92.32
14.23
90.60
3,54
91.79
4.97
O-Rackdove
89.85
100.0
89.59
98.72
88.83
86.75
901.3
99.80
89.60
96.32
O-Robossness
88.58
88.59
0.56
88:
1.50
DemseNei161 (5%)
P-Robusieess (EP)
2,77
88.94
87.61
D-Robustess (PAM)
1.22
87.70
1.81
87.76
1.43
P-Robusiness (PAM)
87.03
3.04
86.49
1.94
85.89
3.72
86.44
9.39
86.42
4.52
Table The post-purification robustness performance PAM on CIFAR-100 and Tiny- ImageNet. Note that we omit the LC attack for both the CIFAR- 100 and Inn-lmmgeNeet as it does not consistently achieve successful backdoor implantation. All metrics are measured in percentage (%).
Method
Evaluation Mode
BadNet
Blended
SSBA
Avg
C-Ace(f)
ASR(J)
C=Ace(")
ASR(L)
C-Aco(t)
ASR(J)
C-Ace(f)
ASR(L)
O-Backdoor
78.91
99.51
78.98
100.0
78.59
92.38
78.83
97.30
O Robustness (EP)
76.89
0.04
76.95
0.04
76.49
0.05
76.78
0.43
CIFAR-100 (5%)
P. P-Robustness (EP)
76.82
0.10
76.18
0.07
76.12
1.32
76.37
0.50
O-Robustness (PAM)
74.97
0.29
76.64
0.19
75.07
0.14
75.56
0,21
Rohuustnsss (PAM)
75.72
0.76
76.12
74.75
1.95
O-Backdoor
72.60
99.01
73.68
99.99
73.02
97.05
73.10
98.63
O Robustness (EP)
70.90
0.02
71.11
0.01
70.24
0.01
70.75
0.13
TINY (5%)
P. R-oobustness (EP)
70.59
0.65
70.93
0.09
69.98
1.90
70.50
0.88
O Rohustness (PAM)
68.78
0.06
67.89
0.14
68.01
4.47
68.23
1.56
Robustness (PAM)
68.97
0.37
67.39
14.26
67.74
Post- Purificatiin Robustness of PAM. We evaluate the post-purification robust- ness of PAM against RA and make a com- parison with EP. Using the same experi- mental settings in Section 3.1, we set p to 0.5 for Blended and SSBA and 0.9 for the BadNet and LC attack on CIFAR-10 and set p 0.4 for both CIFAR-100 and Tiny-ImageNet. The results on CIFAR- 10, CIFAR-100 and Tiny-ImageNet are shown in Table and Table 2. We could observe that PAM significantly improves post-purification robustness against RA. It achieves comparable robustness perfor-
Algorithm 1 Path-Aware Minimization (PAM)
Input: Tuning dataset Dr = D, U Dr; Backdoored model
f f(Wo: x); Learning rate ry; Path-aware step size Tun-
ing iterations I Output: Purified model Initialize W, with Wo
2: for i 1....,,., do
3: Sample mini-batch B, from the tuning set Dy;
4: Calculate parameter difference: W.- W W.:
5: Obtain interpolated parameters: W: = W.+ Walla
W
6:
Calculate gradients of W::
9w: = V w. [8,] (E ((
1
7:
Update current parameters: WH+1 = W: -99w;
mance to the EP, with an average ASR lower than 4.5% across all three datasets
8: end for
9: return Purified model f(Wr;æ)
after RA. In comparison to previous experimental results in Section 3.2, our PAM outperforms existing defense methods by a large margin terms of post-purification robustness. Our PAM also achieves stable purification performance (O-ASR), reducing the ASR below 2% on all three datasets and preserves high C-Ace as well, yielding only around 2% drop against the original performance of C-Acc.

---

BadNet
Blended
ASRO
80
60
40-
0.3.3
0-0033 01 05 0.1
0-0.3
0.7
Back-doat
0m0.5
2.3
05 55 interolattin Rabio
55
d/ 110
di ale
intepolation Ratio
intepolation Ratio
intepolation Ratio
Figure 6: Ablation studies of PAM across different values of against four types of backdoor attacks. We conduct our evaluations on CIFAR-10 with ResNet-18.
To further verify the post-purification with PAM, following the experimental setting in Section 4. 4.1, we also show the LMC results PAM in Figure 4. It is clearly observed that our PAM significantly deviates purified models from the backdoored model along the backdoor-connecte path, leading to robust solution similar to the EP method. This further confirms our findings about post-puse cost-purification robustness derived from LMC.
Sensitivity Analysis of p. We evaluate the performance of PAM with various values p and conduct experiments on CIFAR-10 with ResNet-18 against four attacks. The experimental results are shown in Figure 6. Note that as p increases, we increase the error barrier along the connected path, indicating an enhanced deviation of our purified model from the backdoored model. However, simply increasing P would also compromise the competitive accuracy (C-Acc) of the purified model. In practice, it is essential to select an appropriate P to achieve balance between post-pute custfuiifiiation robustness and C-Acc. We present the model performance across various p values in Table 10 of the Appendix. We observe that p rises, there is a slight decrease in clean accuracy alongside a significant enhancement in robustness against the RA. Additionally, we note that performance is relatively insensitive to when it exceeds 0.3. Given that primarily monitor C-Acc (with the validation set) in practice, we aim to achieve a favorable trade-off between these two metrics. Therefore, we follow the approach of FST [32] and select p ensure that C-Acc remains above predefined threshold, such as 92%.
Conclusions and Limitations
In this paper, we seek to address the following question: Do current backdoor safety tuning methods genuinely achieve reliable backdoor safety by merely relying on reduced Attack Success Rates? To investigate this issue, we first employ the Retuning Attack to evaluate the post-purification robustness of purified models. Our primary experiments reveal a significant finding: existing backdoor purification methods consistently exhibit an increased ASR when subjected to the RA. highlighting the superficial safety of these approaches. Building on this insight, we propose practical Query-based Reactivation Attack, which enables attackers re-trigger the backdoor from purified models solely through querying. We conduct deeper analysis of the inherent vulnerabilities against RA using Linear Mode Connectivity, attributing these vulnerabilities to the insufficient deviation of purified models from the backdoored model along the backdoor conproceerdeeeeeeeeeee path. Inspired by our analysis, we introduce a simple tuning defense method, Path-Aware Minimization, which actively promotes deviation from the backdoored model through additional model updates along the interpolated path. Extensive experiments demonstrate the effectiveness of PAM, surpassing existing purification techniques in terms of post-purification robustness.
This study represents an initial attempt evaluate post-purification robustness via RA. While we propose the practical QRA method, future work is essential to develop more efficient evaluation techniques that can faithfully assess port-uuuiiiiation robustness. The need for such evaluations is critical, as they ensure that the perceived safety of purified models is not merely superficial. Additionally, we recognize the significant potential for enhancing QRA in the context of transfer attacks, which we aim to explore in future research. Furthermore, we plan to broaden our research by incorporating additional backdoor attack strategies and safety tuning methods applicable to generative models, such as LLMs and diffusion models [1, 20, 37, 40], in future work. We will also apply our existing framework of evaluation, analysis, and safety tuning method to research on unlearning in large models.
10

---

References
Usman Anwar, Ahulhair Saparov, Javier Rando, Daniel Palcka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner. Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learning Rexearch, 2024. ISSN 2835-8856. Survey Certification, Expert Certification. 1,5
[2] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938-2948.
PMLR, 2020.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child. Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Newral Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. 2
[4] Nicholas Carlini and Andreas Terzis, Poisoning and backdooring contrastive learning. In International Conference on Learning Representations, 2022. 1.2
[5] Nicholas Carlini, Matthew Jagielski, Christopher Choquettc- Daniel Paleka, Will Pcarce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302. 10149, 2023. 1,2
[6] Ting Chen, Simon Korbliihh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-16007. PMIR, 2020. 2
[7] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 172:::11/2.05526, 2017. 1, 2.1
[8] Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv: (70708877088/// 2017. 3.1
[9] Paul Christiano. Mechanistic anomaly detection and elk. Alignment Research Center, 2022. URL httpp:://alaallgggaaat..co/oo/ wwtp://///ccc//ccccccccccccataee wwwettiinnnmmallldeeeeeeee deteeyioeettttiinnnneeeeeee eell-elk-fbte elk-ff84f4c600d.
[10] Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge; How to tell if your eyes deceive you. technical report. Alignment Research Center, 2021. URL :/ttcss:://cs..eeeeee d/ wwt.ssssssPPas_11.1 Yh- Ch2XRL8H, Bpsserrrddwxs33PPCC eedit.
com/ document////////////
[11] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999. 3.3
[12] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318 1318. PMLR, 2018.
[13] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269, PMIRR 2020. 1, 4.1
[14] Timur Garipov, Pavel Izmailov. Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling dnns. Advances neural information processing systems, 31, 2018. 2,4.1
[15] Micah Goldhlum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 2
[16] Tianyu Gu, Kang Liu, Brendan DolantGavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. JEEE Access, 7:47230-4777 447444,44444 2019. 1, 2, 3.1
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
3.1
11

---

[18] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings the IEEE conference computer vision and pattern recognition.
pages 4700-4708. 2017. 3.1
[19] Kunzhe Huang. Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In International Conference on Learning Representations, 2022.
[20] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint S.3i::2401.055666 2024. 1, 2, 3,5
[21] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019. 3.3
[22] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Aver- aging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.0544: 2018.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In E. Pereira, C.J. Burges, L.. Bottou, and K.Q. Weinherger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012. 2
[24] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 3.1
[25] Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel. Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine wearing-innusstry perspectives. In 2020 IEEE security and privacy workshops (SPW), pages 69-75. IEEE, 2020.
[26] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. nati-backdoor learning: Training clean models on poisoned data, Advances in Neural Information Processing Systems, 34:14900- 14912, 2021. 2, B.1
[27] Yige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyu, Bo Li, and Yu-Gang Jiang. Recon- structive neuron pruning for backdoor defense. In International Conference on Machine Learning, pages 19837-19854. PMLR, 2023.
[28] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16463-16472, 2021. 2.1.
[29] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. International Symposium on Research in Attacks, Intrusions. and Defenses, pages 273-294. Springer, 2018. 2, 3.1
[30] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, 2016. 3.3
[31] Ekdeep Singh Lubana, Eric J Bigelow, Robert Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In International Conference on Machine Learning, pages 22965-23004. PMLR, 2023.
[32] Rui Min. Zeyu Qin, Li Shen, and Minhao Cheng. Towards stable backdoor purification through feature shift tuning. Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 3.1, 3.1, 4.2, B.1, B.2,
C,4
[33] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learnin??
Advances in neural information processing systems, 33:512-525 2020. 1, 4.1
[34] Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping- based backdoor attack. In International Conference on Learning Representations, 2021. 2
[35] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2-27770-277444, 2022. 2
[36] Xiangyu Qi. Tinghao Xie, Yiming Li, Saced Mahloujifar, and Prateck Mittal. Revisiting the assump- tion of latent separability for backdoor defenses. In The eleventh international conference on learning representations, 2023.
12

---

[37] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine- tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint urXiv:2310.03693 2023. 2,2,
[38] Zeyu Qin, Yanbo Fan. Yi Liu, Li Shen, Yong Zhang, Jue Wang. and Baoyuan Wu. Boosting the trans- ferability of adversarial attacks with reverse adversarial perturbation. Advances in neural information processing systems, 35:29845-29855. 2022. 3.3
[39] Zeyu Qin, Liuyi Yao, Daoyuan Chen, Yaliang Li. Bolin Ding. and Minhao Cheng. Revisiting personalized federated learning: Robustness against backdoor attacks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23. page 4743-4755 New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9798400701030. 2, 3.1
[40] Javier Rando, Francesco Croce, Kryštof Mitka, Stepan Shabalin, Maksym Andriushchenko, Nicolas Flammarion, and Florian Tramèr. Competition report: Finding universal jailerea backdoors in aligned llms. arXiv preprint arXiv:2404. 14461, 2024, 4.1,5
[41] Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back the drawing board: A critical evaluation of poisoning attacks on production federated learning. 2022 IEEE Symposium on Security and Privacy (SP), pages 1354-1371, IEEF, 2022. 2
[42] Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlcarning. IEEE Transactions on Neural Networks and Learning Systems, 2023. 3.2.2
[43] Alexander Turner. Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks, arXiv preprint
2019. 3.1
[44] Bolun Wang. Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE, 2019. 1, 3.1
[45] Yuhang Wang. Huafeng Shi, Rui Min, Ruijia Wu, Siyuan Liang. Yichao Wu, Ding Liang, and Aishan Liu. Universal backdoor attacks detection via adaptive adversarial probe. arXiv preprint arXiv:2209.052. 2022. 2
[46] Zhenting Wang, Kai Mei, Juan Zhai and Shiqing Ma. Unicorn: A unified backdoor trigger inversion framework. In The Eleventh International Conference on Learning Representations, 2022. 2
[47] Alexander Wei, Nika Haghtalab. and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36, 2024."
[48] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei. Danni Yuan, and Chao Shen. Backdoorbench: comprehensive benchmark of backdoor learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 1, 2, 2, 3.1, 3.1, B.1
[49] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34:16913-16925, 2021. 1, 3.1, B.1
[50] Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren. Towards reliable and efficient backdoor trigger inversion via decoupling benign features. In The Twelfth International Conference Learning Representations. 2023. 1, 2, 3. 3.1, 4.2, B.1
[51] Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold. William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting aaiele- language models. arXiv preprint arXiv:2310.02949 2023.
[52] Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. GPT-4 is too smart to be safe: Stealthy chat with LLMs via eipher. In The Twelfth International Conference on Learning Representations, 2024. 2
[53] Yi Zeng. Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations, 2022. 2, 3.1
[54] Chunting Zhou, Penglei Liu, Puxin Xu, Srinivasan Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efirat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024. 2
[55] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-awai minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4477, 2023, 1. 3.1, B.1, B.2
13

---

A Social Impact
The prevalence of Deep Neural Networks (DNNs) in modern society relies heavily on massive amounts of training data from diverse sources. However, in the absence of rigorous monitoring mech- anisms, these data resources become susceptible to malicious manipulation, resulting in unforeseen and potentially harmful consequences. Among the various concerns associated with the training dataset, backdoor attacks pose a significant threat. These attacks can manipulate the behavior of well-trained model by poisoning the training set with backdoored samples, often at low cost and without requiring complete control over the training process. While existing defense methods have demonstrated effective backdoor purification by achieving low Attack Success Rates (ASR), they still exhibit vulnerabilities that allow adversaries to reactivate the injected backdoor behavior casily. In work, instead of solely focusing backdoor ASR, we investigate the effectiveness of modern purification techniques from the perspective of post-purification robustness. We aim to enhance the cost-purification robustness of backdoor defense, mitigating the potential for malicious manipulation of deployed models even after backdoor purification. In sum, our work hopes to move an initial step towards improving post-purite cost-uurification robustness while also contributing to another aspect of understanding and enhancing machine learning security.
B Experimental Settings
In this section, we provide detailed information about the experimental settings used in our evaluations. This includes the dataset, training details, and the selection of hyperparameters. All experiments were conducted using NVIDIA 3090 GPUs. We ran all experiments times and averaged all results over 3 random seeds.
B.1 Datasets and Models
We follow previous studies [26, 32, 48, 49] on backdoor learning, and conduct experiments three widely used datasets including CIFAR-10, CIFAR-100, and Tmny-lmageNet.
CIFAR-IO is a widely used dataset the backdoor literature, comprising images with a resolution of 32 x 32 and 10 categories. For backdoor training. utilize the ResNet- 18 model for main evaluation, a commonly used architecture in previous studies [32, 50, 55]. Additionally, cxplore other architectures, including the ResNet-50 and DenseNet-161.
CIFAR-100 and which include 100 and 200 different categories, respectively. Similar previous work [32, 50]. we utilize the pretrained ResNet- 88 on ImageNet-1K IK provided by PyTorch to implement backdoor attacks since directly training from scratch would result in an inferior model performance on C-Acc, hence is not practical in real-world scenarios.
are two large-scale datasets compared to the CIFAR-10,
B.2 Implementation Details
Attack Configurations We implement representative poisoning- attacks and generally follow the implementation from the BackdoorBench' For the BadNet, we utilize the 3 x 3 checkerboard patch as triggers and choose the lower right corner of the image for backdoor injection by default; for the Blended, we adopt the Gaussian Noise as the backdoor trigger. We set the blend ratio to 0.1 for backdoor training and increase the blend ratio to 0.2 during the inference phase; for SSBA and LC, we follow the original implementation from BackdoorBench without making modifications. In our implementation, we set the default poisoning rate to 5%, which is commonly used in previous studies [32, 55] and additionally explore various poisoning rates including both 1% and 10%. Note that we do not adopt a lower poisoning rate since most of the methods suffer from effectively removing backdoor effects when the poisoning rate is extremely low as indicated by [32]. For all backdoor attacks, the target label is set be 0 by default.
For CIFAR-10, we adopt an initial learning rate of 1.1 to train all the backdoored models for 100 epochs. For both the CIFAR-100 and Tiny-Im we utilize pretrained backbones and initialize the classifiers with appropriate class numbers. We adopt smaller learning rate of 0.001 and fine-tune
phttps://gitee httpaa/.....nc.//accaotttttttto.
14

---

the models for 10 epochs. We upscale the image size up to 224 224 during both the training and inference stages following the implementation of [32].
Baseline Defense Configurations We evaluate several mainstream purification techniques including pruning-based defense (ANP), tuning-basedo defenses (I-BAU, FT-SAM, and FST), and the state-of- the-art trigger-inversing strategy BTI.
EU: We add real backdoor triggers to 10% of the overall benign tuning samples, and keep their ground-truth labels unchanged. We fine-tune the model on our tuning dataset for 20 epochs.
ANP: We follow the original implementation in BackdoorBench, using the default hyperpa- rameters. Regarding model pruning, we set the threshold range from 0.4 to 0.9 and report the best purification result with a low ASR and a high model performance on C-Acc.
I-BAU: We follow the implementation in BackdoorBench with default configurations. We set the fixed-point approximation iterations to 5 and fine-tune backdoored models for 20 epochs.
FT-SAM: We follow the implementation from [32] and set the neighborhood size to 1 for CIFAR-10 and 0.5 for both the CIFAR-100 and Iiny--mageNNr. We set the initial learning rate to 0.01 and decrease the learning rate to 0.001 for both the CIFAR- 100 and Imag-em We fine-tune backdoored models for 20 epochs on all datasets.
FST: For FST, we follow the original implementation". We set the feature-shift parameter to 0.1 and the learning rate to 0.01 on CIFAR-10. 10. On the CIFAR- 100 and Imagemm,,Nee, we decrease the feature-shift parameter to 0.001 and adopt an initial learning rate of 0.005 for better C-Ace.
BTI: We follow its original implementation' and adopt the BTI (U) since it achieves a better purification performance. We adopt the default hyperparameters for trigger inversion including 20 iterations for decoupling benign features, and 30 iterations for training backdoor generators, and set the norm bound to 0.3 by default. We perform BTI (U) until the loss converges.
PAM: On CIFAR-IO, we set the path-aware step size to 0.5 for the Blended and SSBA: while we increase the step size to 0.9 for both the BadNet and the LC attack. We use a step size of 0.4 on CIFAR-100 and Inn-ImmaeNett to maintain the model performance on C-Ace.
RA Configurations In this section, we provide the detailed experimental setting of the RA in our revisiting Section 3. For the CIFAR-10, we adopt 5 samples for BadNet, 1 samples for Blended, 10 samples for SSBA, and 2 samples for LC attack to perform RA. For both the CIFAR-100 and Tiny-ImageNeet, we increase the number of poisoned images to 25 samples for BadNet, 10 samples for Blended, and 35 samples for SSBA to perform RA. We set the learning rate 0.01 for the CIFAR-10 and 0.001 for the CIFAR-100 and Tiny-ImageNN to maintain C-Acc. However, directly fine-tuning with these poisoned samples would negatively impact the performance of C-Ace. Therefore, in addition to the backdoored samples, we introduce extra benign samples during the fine-tuning process. We fix the size of the tnning images to 2% of the overall training dataset, which are 1000 samples for CIFAR-10 10 and CIFAR-10) 100, and 2000 samples for Tiny- ImageNet.
Note that there are primarily two reasons for selecting only a limited number of poisoned examples for fine-tuning during our experiments. Firstly, using larger number of poisoned samples, such as 10 BadNet samples on CIFAR-IO, would significantly increase the ASR of clean model. This would undermine the reliability of our RA evaluation conducted on the purified models. Secondly, by utilizing only few poisoned examples, we intentionally expose the purified models a potential compromise, even if they exhibit a seemingly low ASR. This approach effectively highlights the vulnerability of these purified models, emphasizing their susceptibility to attacks despite achieving a seemingly low ASR.
QRA Configurations We employ three-layer Multilayer Perceptron (MLP) to generate the reversed perturbation in the input space. For simplicity, the number of neurons in all internal layers is fixed at 1024, followed by the Rectified Linear Unit (ReLU) activation function. To train the generator, we
Phttps://ee http://ggttuuucoommrreeee httpta/w..caaa.oooo////////11100
15

---

Table 3: The post-purification robustness performance against diverse defense methods. We evaluate performance on CIFAR-IO with ResNet-18 and set the overall poisoning rate to 5%. The O-Backdoor indicates the original performance of backdoor attacks, D-Robustness metric represents the purification performance of the defense method, and the P P-Robustness metric denotes the post robustness after applying RA. All metries measured percentage (%)
BadNet
Blended
SSBA
LC
Method
Mode
C-Aco(t)
ASR(J)
C-Ace(t)
ASR())
C-Ace(t)
ASR(L)
C-Ace(t)
ASR(J)
Attack
O-Backdoor
94.04
99.99
94.77
100.0
94.60
96.38
94.86
99.99
O- Rohustness
95.07
0.57
95.07
0.90
95.07
0.61
95.07
0.81
Clean
P- Rubustness
94.37
0.20
94.29
1.26
94.93
1.33
R-Rohusttees
93.57
0.94
93.47
3.37
93.39
0.53
93.64
0.52
EP
Robustness
92.27
1.08
93.36
4.93
92.05
2.47
93.69
0.41
D-RRoustneess
93.75
4.24
94.56
0.30
93.22
0.44
93,40
0.92
ANP
P- R-Rhustteess
93.76
100
94.42
$4,93
93.99
82.74
94.26
97.92
O-Robustness
92.63
1.61
93.47
5.77
93.41
1.63
92.80
2.49
BAU
P- R-obustnnss
93.50
100
94.17
100
93.93
93.99
94.38
99.29
O- Rohustness
92.85
2.14
93.14
2.01
93.27
4.80
93.00
3.14
SAM
P-Rubustness 92.89
100
93.05
78.93
92.98
72.20
92.57
97.91
R-RRousttness 93.89
0.78
93.92
0.81
94.01
0.63
94,09
0.41
FST
P-Robustness 93.92
100
93,88
99.87
93.91
85.53
94.16
90.36
O- R-Robustnees
92.56
1.08
92.48
0.00
92,21
2.46
93.28
2.64
BTT
Robustness
91.15
100
91.86
99.8
91.18
68.57
92.89
99.97
R-buustnesss
92.11
1.14
93.34
92.96
1.24
92.32
4.92
PAM (Ours)
P- P-Rohustness
91.66
3.90
93.38
92.20
3.31
92.15
8.31
use 500 benign examples and 500 backdoored examples from the CIFAR-10 dataset. Our training process is conducted over 50 epochs with a constant learning rate of 0.1. We set the a to 0.1 for the LC attack and 0.2 for the others to achieve high P-ASR on purified models while simultaneously reducing the attack performance on clean models. During the training process of the perturbation generator, we begin by flattening the input image into one-dimensional vector before feeding it to the generator. Once we obtain the output from the generator, we reshape back to the original input size. We then multiply it with the pre-defined budget before we integrate into the images. In our implementation, we fix the to 16/2 255 for all experiments.
C Additional Experiments
C.1 Additional Results of RA
Detailed RA performance Our previous experiments in Section 3 only provide the average metrics on CIFAR-10. Therefore, this section, we provide detailed RA results against each backdoor attack. Specifically, demonstrate the detailed experimental results of ResNet-18 with 5% poisoning rate in Table 3; ResNet- 18 with 10% poisoning rate Table 4; ResNet-50 with 5% poisoning rate in Table and DenseNet- 161 with 5% poisoning rate in Table respectively. Based on these experimental results, our PAM demonstrates effective post-purification robustness against diverse architectures and poisoning rates, which leads to only small ASR increase (less than 3% average) after performing RA.
Evaluation of RA under Lower Poisoning Rate In addition to the 5% and 10% poisoning rates used in our previous evaluation, we also include lower poisoning rate of 1% to assess the performance of R... Specifically, we experiment on CIFAR-10 10 and utilize the ResNet-18 for fair comparison. We exclude the LC attack from our experiments since most defense methods struggle to adequately purify the backdoor behavior. As shown in Table 7, our PAM achieves tiny increase of the ASR after RA compared to other purification techniques. This further demonstrates the effectiveness of PAM method in enhancing pustppuriiiiation robustness.
Evaluation of RA under Other Datasets this section, we provide additional evaluations on RA against two other datasets. We present our experimental results in Table 8 and Table 9 for the
16

---

Table 4: The post-purification robustness performance against diverse defense methods. We evaluate performance on CIFAR-I0 with ResNet-18 and set the overall poisoning rate to 10%. The O-Backdoor indicates the original performance of backdoor attacks, O-Robustness metric represents the purification performance of the defense method, and the P-Robustness metric denotes the post robustness after applying RA. All metries measured percentage (%)
BadNet
Blended
SSBA
LC
Method
Mode
C-Aco(t)
ASR(J)
C-Ace(t)
ASR())
C-Aco(t)
ASR(L)
C-Aoc(t)
ASR(J)
Attack
O-Backdoor
93.73
100.0
94.28
100.0
94.31
98.71
85.80
100.0
O- Rohustness
95.07
0.57
95.07
0.90
95.07
0.61
95.07
0.81
Clean
Rubustness
94.37
0.20
94.29
1.26
94.93
1.33
R-Rohusttees
92.78
0.93
93.34
1.52
93.10
0.71
92.12
0.16
EP
Robustness
92.17
1.58
93.06
4.78
92.04
3.70
91.79
2.03
D-RRoustneess
93.29
0.60
94.20
0.07
93,90
2.14
84.72
0.00
ANP
P- R-RRoustteess
92.74
100.0
93.77
76.96
93.42
91.92
92.00
91.41
O-Robustness
91.85
0.82
92.76
5.20
91.99
6.20
91.80
3.89
BAU
P- R-obustnness
92.96
100.0
93.19
99.99
93.38
97.32
91.87
99.49
O- Rohustness
92.63
1,50
92.64
0.79
91.70
4.01
91.8' 91.81
4.78
SAM
P-Rubustness
92.40
100
92.73
32.16
91.39
81.59
91.73
98.91
R-RRousttness
93.41
0.07
93.81
0.08
93.86
0.08
92.43
3.44
FST
P- P-Robustness
93.38
100.0
93.77
99.96
93.53
93,91
92.27
73.68
O- R--obstteess
92.36
1.69
93.17
0.67
93.16
2.09
91,84
1.59
BTT
Robustness
91.14
100
92.44
83.43
92.48
93.82
91,94
91.93
Robustness
92.43
1.73
92.63
0.22
92.89
1.37
91.06
2.31
PAM (Ours)
Rohustness
91.77
1.47
91.85
7.44
92.01
2.63
90.98
3.37
BadNett
Blended
558A
LC
Backdoor Error (1 ASRI
100
80
AM"
ANP
BAU
BAU
40-
SAN
20
PAMOOUS)
Interplattin Ratio
Interpolation Ratio
Interpo ation Ratio
Ratio
Figure 7: The experimental results of LMC on CIFAR-10. We evaluate the performance on ResNet-18 and set the poisoning rate 5%.
CIFAR- 100 and Tiny-ImageNet, respectively. It is worth noting that our PAM method remains effective when applied to these two datasets, resulting in only a minor increase in ASR.
C.2 Detailed Results of LMC
Detailed LMC performance In addition to the average LMC on CIFAR-10, we provide detailed experimental results for each attack setting. We demonstrate the LMC against ResNet-18 with 5% poisoning rate Figure 7; ResNet-18 with 10% poisoning rate in Figure 8; ResNet-50 with 5%
BodNitt
eeneed
S5BA
LC
Backniont Ereor (1- ASN0
100
80
60
40-
20
03
20100
Interpolation Ratio
Interpolatio Ratio
Interpo ation Radio
Interpo abon Ratio
Figure The experimental results of LMC on CIFAR-10. We evaluate the performance on ResNet-18 and set the poisoning rate to 10%.
17

---

Table 5: The post-purification robustness performance against diverse defense methods. evaluate performance on CIFAR-IO with ResNet-50 and set the overall poisoning rate to 5%. The O-Backdoor indicates the original performance of backdoor attacks, O-Robustness metric represents the purification performance of the defense method, and the P P-Robustness metric denotes the post robustness after applying RA. All metries measured percentage (%)
BadNet
Blended
SSBA
LC
Method
Mode
C-Aco(t)
ASR(J)
C-Ace(t)
ASR())
C-Ace(t)
ASR(L)
C-Ace(t)
ASR(J)
Attack
O-Backdoor
93.81
99.92
94.53
100.0
93.65
97.70
94.57
99.90
O- Rohustness
94.6
0.72
94.6
0.19
94.6
0.77
94.60
0.80
Clean
Rubustness
94.06
4.10
94.25
93.00
3.03
94.29
4.44
R-Rohusttees
92.84
0.98
92.10
1.12
91.84
0.43
92.91
0.41
EP
Robustness
92.33
1.48
91.36
3.51
89.26
6.28
92.33
1.71
R-RRoustneess
92.26
1.01
94.51
0.30
92.46
2.08
92.05
6.32
ANP
P- R-RRoustteess
93.44
92.02
94.43
76.51
93.01
85.51
93.67
81.14
O-Robustness
92.17
1.23
92.88
3.82
90.57
6.64
92,63
3.91
BAU
P- R-obustnnss
92.85
17.94
93.7
98.49
92.28
92.94
93.14
23.73
O- Rohustness
91.53
2.69
92.27
4.86
91.77
2.56
91.88 12.82
3.73
SAM
P-Rubustness 91.39
66.87
92.28
69.04
91.43
75.49
91.83
17.96
R-Robustteess
92.46
0.38
93,79
0.18
93.09
3.52
92.83
1.62
FST
P- P-Rooustness
93.12
85.08
93.53
99.37
92.81
81.04
92.97
56.07
O- R-Robustnees
92.27
4.67
92.15
0.94
92,12
1.39
92.42
3.20
BTT
Robustness
92.37
100.0
91.74
99.70
92.05
68.63
91.77
89.93
Robustness
92.58
0.94
92.59
0.24
92.36
1.62
91.99
2.14
PAM (Ours)
Rohustness
91.49
1.46
92.75
0.64
92.32
14.23
90.68
3.54
BadNet
Blended
Backdoor Error -ASR)
100
80
57
60
ANP
ANP
ANP
BAU
RAU
RAU
BAU
SAH
SAN FST
EST
20
PAM)Sond)
IMMiGund)
PMM|Gund
03 06 Interpolation
8.
03
Interpolation
Figure 9: The experimental results of LMC on CIFAR-10. We evaluate the performance on ResNet-50 and set the poisoning rate 5%.
poisoning rate in Figure 9 and DenseNet-161 with 5% poisoning rate in Figure 10, respectively. Our experiments demonstrate that the proposed PAM method effectively introduces high error barriers along the backdoor-connected path, leading to greater deviation from the backdoored models.
LMC under Lower Poisoning Rate In this section, we evaluate the LMC under a lower poisoning rate, namely 1 1%. As shown in Figure 1I, our PAM still achieves stable high error barriers along the backdoor-connected path compared other defense methods.
BadNet
Blended
S5BA
LC
Backnoal Ereor (1- ASRO
100
80-
60
40-
20
interooatton Rato
Interpolation Rabio
Interpo aton Ratio
Interpo interpolation ation Ratio
Figure 10: The experimental results of LMC on CIFAR-10. We evaluate the performance on
DenseNet-166 161 and set the poisoning rate to 5%.
18

---

Table 6: The post-purification robustness performance against diverse defense methods. evaluate performance on CIFAR-I0 with DenseNet-161 and set the overall poisoning rate to 5%. We omit the performance of ANP against BadNet since it can not achieve successful backdoor purification. The O-Backdoor indicates the original performance of backdoor attacks, O O-Robustness metric represents the purification performance of the defense method. and Robustness metric denotes post robustness after applying RA. All metrics are measured percentage (%)
BadNet
Blended
SSBA
Method
Mode
C-Ace(t)
ASR())
C-Ace(t)
ASR())
C-Aco(t)
ASR(C)
C-Aco(t)
ASR())
Attack
O-Backdoor
89.85
100.0
89.59
98.72
88.83
86.75
90.13
99.80
O Rohustness
90.17
1.27
90.17
1.14
90.17
1.44
90.17
2.17
Clean
P. Rubustness
89,54
2.11
89.47
1.27
88.24
4.64
89.41
4.44
O- Robustness
88.58
1.52
88.00
3.21
88.17
0.69
88.59
0.56
EP
P- Robustness
88.03
2.77
87.13
86.33
1.47
88.94
0.95
Robustness P. Robustness
89.18
4.56
88.64
2.74
90.04
1.58
ANP
89.06
93.72
88.65
47.47
89.45
71.10
O- Robustness
88.00
1.81
87.62
5.43
86.91
17.34
88.02
2.63
BAU
P Robustness
88.52
91.40
88.75
90.40
87,96
78.77
88.84
82.47
O Robustness
86.91
2.00
86.96
7.38
86.64
3.92
87.79
2.52
SAM
P. Rubustness
86.79
100.0
87,09
78.21
86.98
35,97
87,30
12.23
O-Robustness
88.33
5.37
88.56
0.18
88,32
1.63
87.40
1.57
FST
P- R-Robustness
88.16
99.86
87.92
83.06
87.52
68.82
87.47
72.61
O- Rohustness
89.01
0.77
88.22
1.80
88.30
0.53
87.93
2.71
BTI
P- Rohustness
88.11
97.73
88.35
26.20
87.43
48.47
87.67
100.0
O Robustness 88.70
1.22
87.02
1.08
87.62
1.61
87.70
1.81
PAM (Ours)
P- Rohustness
87.03
3.04
86.49
1.94
85.89
3.72
86.44
9.39
BadNet
Blended
SSBA
Backdoor Emro ASR)
100
80
60
ANP
EP
ANP
BAU
ANP BAU
BAU
40
SAM
PST
FST
FST
20
PAMIOOurS)
PAMIOUrS)
PAMIOUrS)
0.1 0.2 0.3 0.4 0.5 0.6 0.8 Interpolation Ratio
1.0
011 0 3 0 06 0.8 0.9 1.0
0. 02102 0 0 06 0.8 1.0
Interpolation Ratio
Interpolation Ratio
Figure 11: The experimental results of LMC on CIFAR-10. We evaluate the performance on ResNet-18 18 and set the poisoning rate to 1%.
LMC under Other Datasets We further evaluate the LMC other two datasets, including the CIFAR-100 (shown in Figure 12) and the Tiny-ImageNee (shown Figure 13). Observations through the experimental results suggest that our PAM is also effective in introducing high error barriers along the backdoor-conneett path across diverse datasets.
LMC with Clean Samples In addition to evaluating the LMC with backdoored examples, we also provide experimental results of the LMC with clean samples, which we refer to the clean-connected path. As depicted in Figure 14, we observe that diverse purification techniques exhibit almost no clean error barrier along the clean-connected path.
C.3 Additional Results of QRA
In this section, we evaluate the QRA under various attack configurations on the CIFAR-10 dataset. Specifically, we include additional poisoning rates (1% and 10%) in Figure 15, and two model architectures (ResNet-50 and DenseNer- 161) in Figure 16. We report performance against two attacks, namely the BadNet and Blended. Note that for certain defense methods, their ASR after applying RA low, making it hard to perform QRA evaluations on them. Therefore, we exclude these defense methods from evaluation and represent them light gray in the Figure. Experimental
19

---

BadNet
Blended
SSBA
Backdoor ASR)
100
EP
EP
SAM
SAM
SAM
20
BT7 PAMIOOur)]
B7)
BTI
PAM]OUrS)
PAM[OU]S]
0
0.3 0.7 0.9
077
0.9
0.3 0.5 0.7 0.9 Interpolation Ratio
Interpolation Ratio
Interpolation Ratio
Figure 12: The experimental results LMC on CIFAR-100. We evaluate the performance on ResNet-18 and set the poisoning to 5%.
BadNet
Blended
SSBA
Backdoor Error ASR)
100
100
Backdoor Error ASR)
100
80
80
60
Backdoor Error
60
60
40
40
40
FST
FST
FST
20
20
20
PAMY
PAMI
PAMIOUrS)
0.1 0.2 0.6 0.8
01 0 0.23300 03 05
0.8
0.1
0.2
0.3 04 05 066 0.7 0.8 0.9.9.9. 10 Interpolation Ratio
Interpolation Ratio
Interpolation Ratio
Figure 13: The experimental results of LMC Tiny-ImageNet. We evaluate the performance on
ResNet- 18 and set the poisoning rate to 5%.
BadNet
Blended
SSBA
LC
Clean Ertor (1- C4kc
100
60-
40
20
interplaatin Ratio
Interpolation Ratio
Interpolation Ratio
Interpolation Ratio
Figure 14: The experimental results of the clean-connected path on CIFAR-10. We evaluate the performance on ResNet-18 18 and set the poisoning rate to 5%.
BadNeet Resket-18 1%
Blended Reskiet-18 1%
Badliet Bestee-18 10%
Blended Resttet-10 10%
100
(huriied)
churfied
CASR (Clean)
CASA (Clann
CASS
CASA (Cean)
PASK Itham)
FAS OChaar
PAS (Clan)
PAAS IClant
AN2
0/6
50M
707
81
0000
800
52.8
PS7
671
084
5.34
P57
67
A6P
0210
0.1M
F57
01
Figure 15: Experimental results QRA against two poisoning rates (5% 10%).
BadNet ResNet-30
Blended Reskiet-30 5%
DadNet DenseNet-161
Blended Denseket-1011
100
CASR (Punites
CASE (Pur
CASS (Purteed) Pask (PurWes) men) (Ckari
CASA (Putfee))
PASK (hurited
Pask gurrnees)
PASk (Pur fedi
(chean)
(Crano
Figure 16: Experimental results of QRA against two model architectures: ResNet-50 and DenseNet- 161.
20

---

Table 7: The post-purification robustness performance against diverse defense methods. We evaluate performance on CIFAR-I0 with ResNet-18 and set a lower poisoning rate to 1%. We omit the performance of SAM against Blended since it can not achieve successful backdoor purification. The O- D-Backdoor indicates the original performance of backdoor attacks, O- Robustmess metric represents the purification performance of the defense method, and Robustness metric denotes post robustness after applying RA. All metrics are measured percentage (%)
BadNet
Blended
SSBA
Method
Mode
C-Aco(t)
ASR(L)
C-Ace(t)
ASR())
C-Aco(t)
ASR(J)
Attack
O-Backdoor
94.77
100.0
94.90
99.12
94.94
79.98
O Robustness
95.07
0.57
95.07
0.90
95.07
0.61
Cleam
P- Robustness
94.63
1.50
94.90
2.77
94.47
0.98
Robustness
94.36 93.36
0.54
93,22
94.16
0.95
EP
P- Robustness
0.68
93.36
0.51
92.86
0.80
ANP
O- Robustness
93.41
0.60
93.84
0.29
94.61
1.13
Robustness
94.32
87.11
94.49
0.82
94.37
48.21
O- Robustness
92.64
6.89
92.64
3.67
93.18
2.81
BAU
P- P-Rooustness
93.88
95.67
93.92
99.91
94.14
58.04
Robustness
93.45
2.80
92.68
3.77
SAM
P. Rohustness
93.64
41.72
92.57
55.51
O- Robustness
94.47
1.48
94.09
0.00
94.20
0.13
P- Rohustness
93.53
90.07
93,90
99.90
93.86
80.23
O- Robusiness
92.89
1.44
92.82
1,06
92,90
1.52
BTI
P- Rohustness
92.33
99.98
92.50
95.21
91.92
36.84
O Robustness
92.52
2.91
92.6
0.10
92.71
1.91
PAM (Ours)
P-Robustness
92.61
1.74
92.91
91.91
3.45
Table 8: The post-purification robustness performance against diverse defense methods. We evaluate the perfor- mance CIFAR- with the pretrained ResNet-18 18 and set the poisoning to 5%. The O-Backdoor indicates the original performance of backdoor attacks, O-Robustness metric represents the purification performance of the defense method, and the P-Robusmess metric denotes the post robustness after applying RA. All metries are measured in percentage (%).
BadNet
Blended
SSBA
Method
Mode
C-Acc(t)
ASR())
C-Ace(t)
ASR())
C-Acu(t)
ASR(L)
Attack
O O-Backdoor
78.91
99.51
78.98
100.0
78.59
92.38
O- Robustness
79,70
0.04
79.70
0.03
79.70
0.04
Clean
P- Robustness
78.92
0.25
79.03
0.39
78.30
1.91
O Robustness
76,89
0.04
76.95
0.04
76.49
0.05
EP
P-Robustness
76.82
0.10
76.18
0.07
76.12
1.32
O- Robustness
76.27
2.85
76.54
0.44
76.34
3.30
SAM
P. Robusiness
76.63
95,30
77.16
97.90
75.50
78.35
O-Robustness
73.02
1.71
72.41
0.30
73.54
0.18
FST
P- Robustness
72.93
93.71
72.28
100.0
72.06
59.90
O- Robustness
75.55
5.79
75.68
0.49
75.59
1.77
BTT
Robustness
75.78
59.29
76.23
70.44
75.07
46.10
Robustness
74.97
0.29
76.64
0.19
75.07
0.14
PAM (Ours)
P- R-oobustness
75.72
0.76
76.12
0.13
74.75
1.95
results demonstrate the effectiveness of our QRA on waking backdoor behavior under various attack settings.
C.4 Results of Analyzing Sensitivity of
We present the model performance across various p values in Table 10. Our observations reveal that p rises, there is slight decrease in clean accuracy alongside significant enhancement in robustness against the RA. Additionally, we note that performance is relatively insensitive to p when it exceeds 0.3, Given that we primarily monitor C-Aco (with the validation set) in practice, aim to
21

---

Table 9: The post-purification robustness performance against diverse defense methods. We evaluate performance on Backdoor indicates the original performance of backdoor attacks, O-Robustness metric represents the purification performance of the defense method, and the P-Rouustness metric denotes the post robustness after applying RA. All metrics are measured in percentage (%).
with the pretrained ResNet-18 and set the poisoning rate to 5%. The O-
BadNet
Blended
SSBA
Method
Mode
C-Ace(1)
ASR(4)
C-Ace(t)
ASR(L)
C-Aco(t)
ASR(L)
Attack
O-Backdoor
72.60
99.01
73.68
73.02
97.05
O- Robustness
73.88
0.08
73.88
73.88
0.05
Clean
P- Robustness
72.99
0.76
73.48
0.60
72.10
2.72
O- Robustness
70.90
0.02
71.11
0.01
70.24
0.01
P. Rubusiness
70.59
0.65
70.93
0.09
69.98
1.90
O- Robustness
70.65
5.92
70.80
4.02
70.18
11.22
SAM
P- P-Rooustness
70.90
84.64
71.30
99.41
70.61
77.85
O- Robustness
65.36
0.63
66.55
1.19
65.85
3.32
FST
P. Rohustness
65.30
93.49
65.58
60.12
64.77
77.58
O- Robustness
68.43
0.06
68.74
0.55
68.92
1.59
P- Rohustness
68.91
68.29
68.41
38.13
67.98
77.25
O- Robustness
68.78
0.06
67.89
0.14
68.01
4,47
PAM (Ours)
P- Rohustnesss
68.97
7.81
66.86
0.37
67.39
14.26
Table 10: We demonstrate the performance of PAM with diverse and evaluate the Blended attack on CIFAR- -0 with ResNet-18. 18. The O- Robustness metric represents the purification performance of the defense method, and the P-Robustness metric denotes the post robustness after applying RA.
Evaluation Mode p= 0.1 (C-Ace ASR) p= 0.3 (C-Ace/ ASR) p 0.5 (C-Ace/ ASR)
# (C-Ace ASK)
P = 0.9 (C-Ace / ASR)
Robuussesss
94,03 / 6.33
93.64 2,07
93.34/1.67
12.12/0.50
91,99/ 1.00
P.Robustness
93.60/
93.61
93.38/2.69
92.17/262
92.54/0.30
achieve favorable trade-off between these metrics. Therefore, follow the approach of FST [32] and select p to ensure that C-Acc remains above a predefined threshold, such as 92%.
C.5
Additional Discussions on the Frequently Asked Question: Directly Applying SAM with BTI
Given that our proposed method exhibits some similarities to the functionality SAM, a frequently asked question arises regarding whether utilizing reversed backdoor data from BTI in conjunction with SAM would enhance post-purification robustness. To address this question, we further evaluate the performance of directly integrating SAM with BTI (BTI+SAM) as robust baseline and present comparative analysis with PAM in Table 11. Our experimental results indicate that the direct combination of BTI and SAM does achieve the same level of robustness as PAM. This finding underscores the significance of the muckdoor-connecdt pathway between purified and backdoored models, as delineated by the LMC.
Table 11:
The comparison of post-uurification robustness performance between BTI+SAM and PAM. We evaluate the performance on CIFAR- 10 with ResNet-18 and set the overall poisoning rate 5%. The O- Backdoor indicates the original performance of backdoor attacks, O-Robustness metric represents the purification performance of defense method, and the P-Robustness metric denotes the post robustness after applying RA. All metrics are measured in percentage (%).
BadNet
Blended
SSBA
LC
Method
Mode
C-Aco(f)
ASR())
C-Aco(t)
ASR(I)
C-Aco(t)
ASR())
C-Aoc(t)
ASR())
Attack
O-Backdoor
94.04
99,99
94.77
100.0
94.60
96.38
94.86
99,99
O- Rohustness
91.72
3.70
92.49
42,49
93.03
7.84
93.04
6.67
BTI+SAM
P- Rubustness
92.48
100.0
92.99
100.0
93.36
94.12
92.87
92.01
O Rohustness
92.11
1.14
93.34
1.67
92.96
1.24
92.32
4.92
PAM (Ours)
P- Robustness
91.66
3.90
93.38
2.69
92.20
3.31
92.15
8.31
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
No theory results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [No]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
Seerides//ooSSubmiise www.miissssopolioioooooooooeoiee icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iips.co.. //nips.c wttpccccpppccccceeececccccccccct guides/ /CodeSubmic /uter////iiddeeeeeeeuuitiiiiii ssissssilillooooo onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section B
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : /ttp///eerrpseee/ee//ee//e cc hc/pppblict/thicccsueeeeeee..eel
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
26

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Section B
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
27

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "X0G4n35B0v", "reviewer": "Reviewer_YFJE", "summary": "Backdoor attacks are a major threat to Deep Neural Networks (DNNs), as they allow attackers to manipulate model predictions with backdoor triggers. Existing purification methods reduce the Attack Success Rate (ASR) of these models, but it's unclear if they fully eliminate backdoor threats. This study investigates post-purification robustness by employing the Retuning Attack (RA) and finds that current methods are vulnerable, as models quickly relearn backdoor behaviors. To address this, the study proposes the Query-based Reactivation Attack (QRA) and a novel Path-Aware Minimization (PAM) technique. PAM enhances robustness by promoting deviation along backdoor-connected paths with extra model updates. Extensive experiments show PAM significantly improves robustness, maintaining low ASR and good accuracy, providing a new perspective on evaluating and improving backdoor defenses.", "questions": "This is a good paper, discussing the details of the proposed defense method, one thing I'm curious is that does the author has plans for designing an adaptive attack for this defense method, and how to achieve it?", "limitations": "The limitations have been discussed in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper empirically verifies that poisoned nodes typically exhibit large prediction variance under edge dropping, providing a indicator for identifying poisoned nodes​​.\n2. The proposed robust training strategy not only has theoretical guarantees but also shows practical effectiveness in defending against various types of backdoor attacks, maintaining clean accuracy while reducing the attack success rate​​.", "weaknesses": "1. While the paper focuses on various backdoor attack types, the scope of attack types and defense mechanisms explored could be broadened to cover more diverse scenarios and setting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We are grateful to you for your time and effort in reviewing our work, as well as acknowledging our contributions.**\n\n### **Response to Weakness 1**:\nThanks for your suggestion! Following your suggestion, we add evaluations on more poisoning attacks.\n\n**For more attacks:** We test our PAM on more backdoor attacks, including Adaptive-Patch [1], Adaptive-Blend [1], and All-to-All attack [2]. Our experiments are shown in Table of the Global Response. The results show that our method still achieves satisfying post-purification robustness against them, yielding only 3.75% ASR after the Retuning Attack.\n\n**For more scenarios:** We plan to apply our method to backdoor safety issues on LLMs [4,5,6]. We could first adopt current backdoor reversing methods to reverse inserted trigger prompts [6]. Then we could utilize our PAM method with reversed trigger prompts to purify backdoored LLMs. Future work will be dedicated to these endeavors. \n\n### **Response to Question 1**:\n\nThanks for your interesting question. We have also considered designing possible adaptive attacks for our PAM method. PAM consists of two parts: 1) getting gradients of the interpolated model for further update; and 2) data specification for backdoor-connected path; Next, we discuss possible strategies for the above two components, respectively. \n1. Making the loss landscape around the poisoned solution becomes smoother: PAM needs the gradients of the interpolated model for getting further updates, to obtain a solution deviated from backdoored models. If attackers could control the training process and position the backdoored model at a smoothed local minimum (possibly using methods like SAM), they might impede the post-purification performance of PAM. We try this possible adaptive attack. **We find our PAM still performs robustly against it.** We suspect that the small radius adopted by SAM limits its smoothing performance. However, enlarging the radius will also significantly sacrifice clean accuracy. We will explore other possible methods in the future.\n2. Nonetheless, such advanced attacker capabilities that include controlling training procedures exceed our work's scope and do not align with our experimental setting, since we mainly focus on practical data-poisoning attacks. \nMaking reversing backdoor samples becomes harder: PAM needs to reverse backdoored samples to specify the backdoor-connected path. If attacks could make reversing backdoor samples fail, they may be able to fail PAM defense. In our work, we adopt the most advanced backdoor-reversing method, BTI [3], which can effectively handle current data-poisoning attack methods. \n\n-----------\n\n[1]. Revisiting the Assumption of Latent Separability for Backdoor Defenses, ICLR 2023. \n\n[2]. Backdoorbench: A comprehensive benchmark of backdoor learning, NeurIPS 2022.\n\n[3]. Towards reliable and efficient backdoor trigger inversion via decoupling benign features, ICLR 2024. \n\n[4]. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arxiv 2024.\n\n[5]. Universal jailbreak backdoors from poisoned human feedback, ICLR 2024.\n\n[6]. Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs, arxiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "paKgoubyWJ", "reviewer": "Reviewer_JZci", "summary": "This paper reveals a phenomenon in backdoor defense: the purified backdoor can be reactivated by fast retuning on a few backdoor samples. Building upon this observation, the paper explores both attacks and defense measures for more reliable backdoor research. On the attack side, a Retuning Attack (RA) is proposed and generalized to the black-box setting as a Query-based Reactivation Attack (QRA). On the defense side, a Path-Aware Minimization (PAM) method is proposed to force more deviation from the backdoor-connected path. Experiments verify the effectiveness of PAM compared to exact purification (EP).", "questions": "1. Can PAM deal with multi-trigger attacks?[1]\n\n[1] Li, Yige, et al. \"Multi-Trigger Backdoor Attacks: More Triggers, More Threats.\" arXiv preprint arXiv:2401.15295 (2024).\n\n2. How does PAM work, compared to applying the same existing defense twice, e.g., ANP or SAM?\n\n3. Can the problem be addressed by simply adjusting the hyperparameters of the Mode Connectivity defense [2]?\n\n[2] Zhao, Pu, et al. \"Bridging mode connectivity in loss landscapes and adversarial robustness.\" arXiv preprint arXiv:2005.00060 (2020).", "limitations": "1. Problematic threat model.\n2. Limited technical novelty.\n3. Missing systematic comparison with existing methods.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The observed phenomenon that the purified model is more easily to restore the trigger is interesting.\n\n2. Both attacks and defense were explored.\n\n3. The defense results look promising.", "weaknesses": "1. The restoration of the backdoor was conducted on a poisoned subset which is not a surprise in this case as the model will surely relearn the backdoor. The authors should prove the phenomenon via tuning on purely clean training data. \n\n2. The threat models of the two proposed attacks are problematic. 1) RA will require a post-purification poisoning to the defense model, which means that it can poison again after the defense, which does not make sense to me; 2) QRA attack is very similar to an adversarial attack, that requires access to both the purified model and the RA model, why the defender exposes these models (or their APIs) to the attacker is questionable. \n\n3. The authors should clearly define what is Post-purification Robustness. In the current version, it appears the same as standard backdoor robustness, i.e., how to guarantee the PAM purified model is 100% robust?\n\n4. The authors seem to confuse a backdoor attack with adversarial perturbation in proposing the QRA attack, A backdoor attack does not make changes to the input during the inference stage otherwise it becomes an adv attack. These two attack types assume different capabilities and flexibility of the adversary.\n\n5. The proposed PAM is very much like using a moving average to force larger updates of the weights. PAM requires reverse-engineered backdoor samples $\\mathcal{D_r}$, I wonder if one could simply use unlearning to achieve the same effect.\n\n6. The proposed method was not compared with existing defense methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your time and effort in reviewing our work!**\n### **Response to W1**:\n1. For \"surely relearn the backdoor\":     \nSorry for the possible confusion. First, we emphasize **the reviewer’s statement “...model will surely relearn the backdoor.” is not correct**. **As emphasized in Lines 169-173 of Section 3.2 and Figure 1**, exact purification (EP) doesn't relearn backdoors, maintaining low ASR after RA. \n2. For \"tuning on purely clean training data\":    \nThanks for this interesting question. Simply tuning on clean data cannot relearn backdoors. Instead, it is widely used as a baseline defense [1,2]. Additionally, it would be interesting to explore whether fine-tuning on carefully selected clean data, based on certain metrics, may lead to relearning backdoors. We will explore it in the future.\n3. We also discuss why our work is surprising in the Global Response.\n\n### **Response to W2**:\n1. For 1): Please refer to the practical implications of our threat models in Global Response.\n2. For \"QRA attack is very similar to an adversarial attack\" in 2):   \nSorry for the possible confusion. In the main submission, **we have specifically emphasized the differences between QRA and adversarial perturbations (ADV) in Lines 212-215 and derived the final objective of QRA, Eq. 2.** As shown in Figure 3, QRA from Eq. 2 only works when added on backdoored examples and with purified models. Applying QRA on clean images with purified models or attacking clean models will not succeed. Instead, ADV can work on these all attack scenarios. This suggests Our QRA is different from ADV. We also discuss their difference in Weakness 4. \n3. For \"that requires access both the...\" in 2):       \nAs mentioned in response to 1), nowadays, after safety tuning, the model providers \nopen-source the models or release the API to enable further fine-tuning for specific usage. \nWe also demonstrate that QRA can successfully transfer across unknown purification methods in Figure 2. This doesn’t need any queries to targeted models, which highlights the practicality of QRA.\n\n### **Response to W3**:\n1. For \"it appears the same as standard backdoor robustness\":   \nAs we've discussed throughout the article, post-purification robustness (P-) has a significant difference from standard backdoor robustness (S-). S- directly tests purified models’ ASR on a backdoored testing set. Contrastively, P- evaluates purified models’ ASR against RA used in the paper. Although they both evaluate the ASR, P- takes a further step than S-. It reveals that the current defense with nearly 0 ASR cannot truly eliminate backdoor features, which quickly regain ASR after RA (Sec.3). This emphasizes instead of solely depending on S-, we need more faithful and comprehensive evaluations to ensure lasting protection against backdoor attacks.\n\n2. For \"guarantee the PAM purified model is 100% robust?\":    \nIn practice, all current defense methods cannot guarantee they are 100% robust since they cannot achieve 0 ASR [2,4,5] empirically. PAM is also an empirical defense and doesn’t offer certified guarantees. \n\n### **Response to W4**:\nSorry for the possible confusion. First, our QRA is different from ADV and is specific to the inserted backdoor. **We believe the reviewer’s statement about the difference between backdoor and ADV is not accurate**. For a successful backdoor attack, it's necessary to activate a preembedded backdoor to conduct an attack[3]. Contrarily, ADV does not need to embed a backdoor and can mislead any model. The difference is not about modifying samples during inference, but whether perturbation refers to a previously inserted backdoor [3]. \n\n### **Response to W5**:\nSorry for the possible confusion. We have conducted unlearning with reversed backdoored samples in our paper. All numbers referencing \"BTI\" pertain to this unlearning. We follow source codes to implement it (BTI-U in original paper). We give a detailed description of it in Lines 522-525 of Appendix. We could observe that BTI cannot achieve same effect as PAM. We will clarify this point in the revised version.\n\n### **Response to W6 and Q2**:\n1. Response to W6: We think **the reviewer may miss our main experimental results in Figure 1-4**. We have taken a thorough comparison between PAM and existing defense methods and observed PAM outperforms others in terms of post-purification robustness.\n2. Response to Q2: We think the reviewer may misunderstand the process of PAM. Like existing defense methods, PAM is applied **once** on backdoor models. We don’t understand Question 2, since it appears there's no difference between doing them once or twice. Inspired by suggestions, we have added another method for comparison. After getting reversing samples, we adopt the SAM method to purify models on them. We observe that this cannot maintain consistent robustness against RA, regaining over 75% ASR average on CIFAR-10. \n\n### **Response to Q1**:\nThanks for this interesting question.\nSince we cannot find the source code of reference work, we utilize the All-to-All attacks [4] which also inserts multiple backdoors into models and present PAM’s performance in Table of the Global Response. PAM still achieves good post-purification robustness.\n### **Response to Q3**:\nThanks for this interesting question. The method of MCR cannot achieve same effect as PAM against RA. As shown in Figure 2 of MCR, we observe that to maintain good clean accuracy, the defender must select low ASR solutions very close to backdoored models that are not robust to RA (see our Section 4.1). \n\n[1]. Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks, arxiv 2018.\n\n[2]. Towards Stable Backdoor Purification through Feature Shift Tuning, NeurIPS 2023. \n\n[3]. Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses, arxiv 2021.\n\n[4]. Backdoorbench: A comprehensive benchmark of backdoor learning, NeurIPS 2022.\n\n[5]. Reconstructive neuron pruning for backdoor defense, ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ni8p8ZQbkI", "reviewer": "Reviewer_R9Lq", "summary": "This paper investigates the effectiveness of current purification-based backdoor defenses and tries to uncover whether purified DNNs are truly free from backdoor vulnerabilities. The authors identify the “post-purification robustness” of DNNs and propose Retuning Attack (RA) and Query-based Reactivation Attack (QRA) respectively to assess the susceptibility of purified DNNs to backdoor reactivation. Additionally, the paper proposes Path-Aware Minimization (PAM) to improve post-purification robustness.", "questions": "Can the authors further justify the proposed threat model’s real-world implications? How can the proposed workflow be enabled in a real-world attack setting?", "limitations": "The paper falls short in addressing practical implementation challenges. These limitations suggest that the proposed methods might not be as universally practical as claimed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "The work makes contribution in backdoor robustness evaluation by shifting the focus from merely achieving low ASR to evaluating the post-purification robustness of backdoor defenses. The paper introduces a pipeline of methods: RA, QRA, and PAM, for the assessment of this vulnerability.", "weaknesses": "The proposed notion of post-purification robustness lies with the assumption that the purified DNN will encounter further fine-tuning. As such, the entire workflow proposed is not practical in a real-world scenario where the purified model is kept frozen with no further updates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We are grateful to you for your time and effort in reviewing our work, as well as acknowledging our contributions.**\n\n### **Response to Weakness and Questions**:\nThanks for your kind question! \nDue to the space limitation, we only simply discuss practical implications for real-world settings in Lines 109-116 of Section 3. We will further explain the threat model as follows. We also mention this in the global response.\n\nLarge Models such as CLIP, ChatGPT, LLaMa, and Stable Diffusion, have become essential bases supporting a wide variety of AI applications. These Large Models (after completed safety tuning) provide powerful pre-trained capabilities that can be fine-tuned for a wide range of specific use cases. In practice, further customization of these models via fine-tuning is often desirable to tailor their performance for particular applications [1]. For open-sourced models like the LLaMa series and Stable Diffusion, the model providers explicitly encourage further fine-tuning to specialize these models' capabilities for use applications [2]. For close-sourced models like GPT-4 and Claude 3, providers offer APIs that allow users to upload their specific datasets and fine-tune these models accordingly [3,4]. \n\nMeanwhile, the pretraining datasets for Large Models have grown to web-scale datasets with billions of samples crawled from the internet [5,6]. At this scale, it is infeasible to manually curate each example, which leaves a viable opportunity for attackers to launch actual poisoning attacks [7].\n\nTo ensure the safety of models in practical use, people have proposed tremendous methods to purify models before releasing them [8,9,10]. Despite substantial efforts in this area, it is not clear that even if a purified model’s initial safety performance is impeccable (nearly 0 ASR), will this robustness still be preserved after further finetuning with possible poisoned data? \nSimilar to works [11,12] which studied safety alignment against finetuning on harmful prompts, **we take the first initial attempt to consider this actual issue for backdoor poisoning threats.** \n\nOur study finds that backdoors could be very easily reactivated by further tuning on only an extremely small number of poisoned samples (like 1 sample for blended attack). This reveals that current purification methods cannot truly eliminate poisoning backdoor features learned during pretraining. \nThis will undoubtedly pose a greater threat to the real world. Attackers could potentially bypass “firewalls” from safety purifications. Leveraging the stronger capabilities of Foundation models, they could then generate more threatening and disruptive content (more realistic counterfeit images [13] and more potent malicious codes [14]), further harming other’ productive activities. \n\nWe will add this detailed explanation as a separate section in the revised version. We hope our response could address your concerns. \n\n--------\n\n[1]. https://llama.meta.com/responsible-use-guide/\n\n[2]. https://llama.meta.com/docs/how-to-guides/fine-tuning/\n\n[3]. https://platform.openai.com/docs/guides/fine-tuning\n\n[4]. https://www.anthropic.com/news/fine-tune-claude-3-haiku\n\n[5]. Laion-5b: An open large-scale dataset for training next generation image-text models, NeurIPS 2022.\n\n[6]. Exploring the limits of transfer learning with a unified text-to-text transformer, JMLR 2020. \n\n[7]. Poisoning Web-Scale Training Datasets is Practical, IEEE S&P 2024. \n\n[8]. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, arxiv 2022.\n\n[9]. Backdoor Learning: A Survey, arxiv 2022.\n\n[10]. https://openai.com/index/openai-safety-update/\n\n[11]. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024\n\n[12]. Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models, arxiv 2023. \n\n[13]. How to Backdoor Diffusion Models?, CVPR 2023. \n\n[14]. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arxiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kaBL9bUc6v", "reviewer": "Reviewer_jgTc", "summary": "This paper studies post robustness of backdoor purification. The authors show that backdoors purified by existing defenses can be recovered via Retuning Attacks and they propose the Query-based Reactivation Attack to recover the backdoor. The authors address such an vulnerability by proposing a Path-Aware Minimization defense. Evaluations are performed on four different attacks over CIFAR-10/100 and Tiny-ImageNet across various deep neural networks. Results show that the proposed PAM defense achieves a better post purification robustness.", "questions": "1. In Table 2, the clean accuracy of all these models are lower than usual, why is that?", "limitations": "Limitations has been addressed in \"Conclusions and Limitations\".", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It is novel to study the post purification robustness and this paper reveals an often-neglected vulnerability of DNNs and limitations of existing defenses.\n\n2. The paper progressively demonstrate that models can relearn backdoors after purification via Retuning Attacks and based on the observation, the authors propose a Query based attacks that is more practical under real-world threat models.\n\n3. The finding that the inadequate deviation of purified models from the backdoored model along the backdoor-connected path is the root cause of poor post purification robustness is both Instructive and insightful. The authors successfully develop an effective defense upon this finding.", "weaknesses": "1. My primary concern is that the attacks evaluated in the paper are not SOTA. BadNets, Blended, SSBA and LC are attacks developed years ago. The authors should also provide the results on recent attacks such as Sleeper Agent[1] and Adaptive Blended[2].\n\n2. The authors mentioned there is a trade-off between post purification robustness and the clean accuracy. However, it is unclear how to determine the hyper parameter $\\rho$ to achieve the balance. An algorithm for $\\rho$ selection would be necessary.\n\n3. The presentation needs to be improved. For example, order of Fig.2 and Fig.3 should be interchanged; it is very hard to see the results in Fig. 5;  Algorithm 1 should be placed close to where it is described.\n\n4. Some typos. For example, line 142: we following -> we follow;  in Figure 1, shouldn't P-ASR be R-ASR? \n\n[1]. Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch, NeurIPS, 2022\n\n[2]. REVISITING THE ASSUMPTION OF LATENT SEPARABILITY FOR BACKDOOR DEFENSES, ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We are grateful to you for your time and effort in reviewing our work, as well as acknowledging our contributions.**\n### **Response to Weakness 1**\n\nThanks for your suggestion! As you suggested, we test our method PAM on more backdoor attacks, Adaptive-Patch,  Adaptive-Blend, and All-to-All attacks which insert multiple backdoors into models [1]. The results are shown in the Table of the Global Response. We could observe that our method still achieves satisfying post-purification robustness against them. We also attempt the SleeperAgent but find unstable attack performance and low ASR with public implementation [2, 3].\n\nIn this work, we tend to figure out whether achieving a low ASR through current purification methods truly signifies the complete removal of inserted backdoor features. To investigate this phenomenon, we select several classic and practical attack methods in the field of backdoor learning. We demonstrate that even under these well-explored attack paradigms, current state-of-the-art defenses still suffer from our Retuning Attack which cannot achieve the post-purification robustness. We are greatly appreciative of the suggestions put forth by the reviewer. We will add these experiments in the revised version.\n### **Response to Weakness 2**\nThanks for your instructive comments. We showcase the model performance across various $\\rho$ values in the Table below. As $\\rho$ rises, there is a slight decrease in clean accuracy alongside a significant enhancement in robustness against RA. Additionally, we can observe that the performance is not sensitive to the $\\rho$, when larger than 0.3. Considering we only observe the C-Acc (with val set) in practice and need to achieve a good trade-off between these two metrics, we follow FST [4] and choose $\\rho$ to ensure that the C-Acc doesn’t fall below a predefined threshold like 92%. \n\nWe demonstrate the performance of PAM with diverse $\\rho$ and evaluate the Blended attack on CIFAR-10 with ResNet-18. The O-Robustness metric represents the purification performance of the defense method, and the P-Robustness metric denotes the post robustness after applying RA.\n|Evaluation Mode|$\\rho=0.1$ (C-Acc/ASR)|$\\rho=0.3$ (C-Acc/ASR)|$\\rho=0.5$ (C-Acc/ASR)|$\\rho=0.7$ (C-Acc/ASR)|$\\rho=0.9$ (C-Acc/ASR)|\n|-----------|--------------|--------------|--------------|--------------|--------------|\n|O-Robustness |94.03/6.33|93.64/2.07|93.34/1.67|92.12/0.50|91.99/1.00|\n|P-Robustness |93.60/33.29|93.61/10.06|93.38/2.69|92.17/2.62|92.54/0.30|\n### **Response to Weakness 3 and 4**\nWe greatly appreciate the reviewer for the invaluable suggestions and for pointing out our typos. We are committed to enhancing the presentation. We will follow your suggestions to adjust the layout, and colors, and rectify the typos in our revised version.\n### **Response to Question 1**\nThanks for your question. In our work, we adopt the ResNet-18 model (the checkpoint pretrained on ImageNet from Torchvision) for CIFAR-100 and TinyImageNet. After backdoor poisoning, the clean accuracies of backdoor models(averaged on attacks) are 78.8% and 73.1%, respectively. The numbers are higher than the numbers from the BackdoorBench paper [1], where the clean accuracies are 70.51% and 57.28%, respectively. The results in our paper are also aligned with the results from [PaperwithCode](https://paperswithcode.com/), where the accuracy on CIFAR-100 is 75% which is from ResNet-164 [1] (without using any tricks) and accuracy on TinyImageNet is 74% which is from ResNeXt-50 [1] (with using advanced augmentations).\n\nAfter conducting purification, there are slight decreases in clean accuracy. The clean accuracies of PAM are 75.56% and 68.23% on CIFAR-100 and TinyImageNet which is comparable to the original BTI and outperforms FST (as exhibited in Appendix Tables 8 and 9). We reorganize the results from Tables 2, 8, and 9 into the below Tables to take a convenient comparison. We could observe that our PAM achieves a better trade-off between clean accuracy and backdoor robustness. We will present a better-organized version of the results in our revision. \n\nThe slight drops across all purification methods may be attributed to our practice of using small clean datasets to adjust backdoored models; a limitation potentially mitigated by utilizing augmentations or distillation with backdoor models as teachers. \nWe leave it for future exploration. \n\n\n**CIFAR-100**\n|Evaluation Mode|Clean (C-Acc/ASR)|EP (C-Acc/ASR)|SAM (C-Acc/ASR)|FST (C-Acc/ASR)|BTI (C-Acc/ASR)|PAM (C-Acc/ASR)|\n|-----------|--------------|--------------|--------------|--------------|--------------|--------------|\n|O-Backdoor|78.83/97.30|78.83/97.30|78.83/97.30|78.83/97.30|78.83/97.30|78.83/97.30|\n|O-Robustness |79.70/0.04|76.78/0.04|76.38/2.20|72.99/0.73|75.61/2.68|75.56/0.21|\n|P-Robustness |78.75/0.85|76.37/0.50|76.43/90.52|72.42/84.54|75.69/58.61|75.53/0.95|\n\n\n**Tiny-ImageNet**\n|Evaluation Mode|Clean (C-Acc/ASR)|EP (C-Acc/ASR)|SAM (C-Acc/ASR)|FST (C-Acc/ASR)|BTI (C-Acc/ASR)|PAM (C-Acc/ASR)|\n|-----------|--------------|--------------|--------------|--------------|--------------|--------------|\n|O-Backdoor|73.10/98.68|73.10/98.68|73.10/98.68|73.10/98.68|73.10/98.68|73.10/98.68|\n|O-Robustness |73.88/0.07|70.75/0.01|70.54/7.05|65.92/1.71|68.70/0.73|68.23/1.56|\n|P-Robustness|72.86/1.36|70.5/0.88|70.94/87.30|65.22/77.06|68.43/61.22|67.74/7.48|\n\n\n[1]. BackdoorBench: A Comprehensive Benchmark of Backdoor Learning, NeurIPS 2022\n\n[2]. https://github.com/vtu81/backdoor-toolbox\n\n[3]. https://github.com/hsouri/Sleeper-Agent\n\n[4]. Towards Stable Backdoor Purification through Feature Shift Tuning, NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to all reviewers for their time and efforts in reviewing our work. We will carefully revise our manuscript by adding suggested experiments, more detailed explanations, and fixing the typos. 

**Here, we provide a global response to questions from reviewers about the practical implications of our threat models and our work's contribution. We also provide evaluation results on more attacks.**

### **The Practical Implications of our threat models for real-world settings**:

Large Models such as CLIP, ChatGPT, LLaMa, and Stable Diffusion, have become essential bases supporting a wide variety of AI applications. These Large Models (after completed safety tuning) provide powerful pre-trained capabilities that can be fine-tuned for a wide range of specific use cases. In practice, further customization of these models via fine-tuning is often desirable to tailor their performance for particular applications [1]. For open-sourced models like the LLaMa series and Stable Diffusion, the model providers explicitly encourage further fine-tuning to specialize these models' capabilities for use applications [2]. For close-sourced models like GPT-4 and Claude 3, providers offer APIs that allow users to upload their specific datasets and fine-tune these models accordingly [3]. 

Meanwhile, the pretraining datasets for Large Models have grown to web-scale datasets with billions of samples crawled from the internet [4]. At this scale, it is infeasible to manually curate each example, which leaves a viable opportunity for attackers to launch actual poisoning attacks [5].

To ensure the safety of models in practical use, people have proposed tremendous methods to purify models before releasing them [6,7]. Despite substantial efforts in this area, it is not clear that even if a purified model’s initial safety performance is impeccable (nearly 0 ASR), will this robustness still be preserved after further finetuning with possible poisoned data? 

Similar to works [8] which studied safety tuning against finetuning on harmful prompts, **we take the first initial attempt to consider this actual issue for backdoor poisoning threats.** Our study finds that backdoors could be very easily reactivated by further tuning on an extremely small number of poisoned samples (like 1 sample for blended attack). This reveals that current purification methods cannot truly eliminate backdoor features learned during pretraining. 
This will undoubtedly pose a greater threat to real world. Attackers could potentially bypass “firewalls” from safety purifications. Leveraging the stronger capabilities of Foundation models, they could then generate more threatening and disruptive content (more realistic counterfeit images [9] and more potent malicious codes [10]), further harming other’ productive activities. 

**We want to reemphasize why our findings are surprising and contributions are important:**    
Our work first proposes a new perspective toward evaluating the effectiveness of backdoor defense methods. Rather than simply focusing on the ASR, we investigate the post-purification robustness via **RA** (Sec.3.2) and more practical **QRA** (Sec.3.3). 
We find that purified models with current defense methods still have backdoor features which could be very easily reactivated. Our findings underscore the necessity for more faithful and comprehensive evaluations to ensure lasting protection against backdoor threats.

We observe that the EP (assuming acknowledging triggers and true labels) does not relearn the backdoor, maintaining low ASR after RA. It validates the possibility of achieving post-purification robustness. This contrast also emphasizes that current purification methods are surprisingly weak and superficial. Such insights guide us to conduct analysis about what leads to robustness of EP (Sec.4.1), further leading us to propose our **PAM** defense technique (Sec.4.2) . Notably, our PAM significantly improves robustness against RA. 

### **Additional evaluations against more attacks**:
Following the suggestions from Reviewer YFJE, Reviewer JZci, and Reviewer jgTc, we have expanded our experiments on additional attack types, including the Adaptive-BadNet, Adaptive-Blend, and All-to-All attacks, as presented in the Table below. The results show that our PAM still achieves good post-purification robustness against them.

Table:  Experiments are conducted on CIFAR-10 with ResNet-18. The O-Backdoor indicates the original performance of backdoor attacks, O-Robustness metric represents the purification performance of defense method, and the P-Robustness metric denotes post robustness after applying RA.

|Evaluation Mode|Adaptive-BadNet (C-Acc/ASR)|Adaptive-Blend (C-Acc/ASR)|BadNet-All2All (C-Acc/ASR)|Blended-All2All (C-Acc/ASR)|
|-----------|--------------|--------------|--------------|--------------|
|O-Backdoor|94.54/86.83|94.70/94.91|94.25/90.21|94.65/77.73|
|O-Robustness (BTI)|92.97/1.40|91.87/4.52|92.29/1.43|93.16/4.11|
|P-Robustness (BTI)|93.13/56.82| 92.08/45.79|92.41/88.41|93.41/58.84|
|O-Robustness (PAM)|92.05/0.65| 92.08/0.14|93.18/0.70|92.52/3.61|
|P-Robustness (PAM)|91.83/0.53|92.51 /4.73| 92.51/0.90| 92.26/6.43|

-----

[1]. https://llama.meta.com/responsible-use-guide/

[2]. https://llama.meta.com/docs/how-to-guides/fine-tuning/

[3]. https://platform.openai.com/docs/guides/fine-tuning

[4]. Laion-5b: An open large-scale dataset for training next generation image-text models, NeurIPS 2022.

[5]. Poisoning Web-Scale Training Datasets is Practical, IEEE S&P 2024. 

[6]. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, arxiv 2022.

[7]. Backdoor Learning: A Survey, arxiv 2022.

[8]. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024

[9]. How to Backdoor Diffusion Models?, CVPR 2023. 

[10]. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arxiv 2024.

Meta-review: The authors first show that, after a backdoored model has been purified, an attacker can reactivate the backdoor crafting ad-hoc test samples. Then, they propose a defense technique to reduce this problem. The reviewers have all positive opinions about this work. They have highlighted that this paper's findings are interesting, and the proposed defense is promising. The main limitation of this work is that the threat model assumes large models and datasets, whereas those used in the experimental analysis do not align with this assumption. However, an extensive experimental analysis with large datasets and models would have been impractical, and there is no reason to think that the proposed approach would not work on large datasets and models if the attacker can inject the same percentage of poisoning points. Therefore, I recommend accepting this work.
Decision: Accept (spotlight)