Title: Towards Stable Backdoor Purification through Feature Shift Tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Backdoor Defense

Model-tuning
Abstract: It has been widely observed that deep neural networks (DNN) are vulnerable to backdoor attacks where attackers could manipulate the model behavior maliciously by tampering with a small set of training samples. Although a line of defense methods is proposed to mitigate this threat, they either require complicated modifications to the training process or heavily rely on the specific model architecture, which makes them hard to deploy into real-world applications. Therefore, in this paper, we instead start with fine-tuning, one of the most common and easy-to-deploy backdoor defenses, through comprehensive evaluations against diverse attack scenarios. Observations made through initial experiments show that in contrast to the promising defensive results on high poisoning rates, vanilla tuning methods completely fail at low poisoning rate scenarios. Our analysis shows that with the low poisoning rate, the entanglement between backdoor and clean features undermines the effect of tuning-based defenses. Therefore, it is necessary to disentangle the backdoor and clean features in order to improve backdoor purification. To address this, we introduce Feature Shift Tuning (FST), a method for tuning-based backdoor purification. Specifically, FST encourages feature shifts by actively deviating the classifier weights from the originally compromised weights. Extensive experiments demonstrate that our FST provides consistently stable performance under different attack settings. Without complex parameter adjustments, FST also achieves much lower tuning costs, only $10$ epochs. Our codes are available at https://github.com/AISafety-HKUST/stable_backdoor_purification.

Towards Stable Backdoor Purification through Feature Shift Tuning

Abstract

It has been widely observed that deep neural networks (DNN) are vulnerable to backdoor attacks where attackers could manipulate the model behavior maliciously by tampering with a small set of training samples. Although a line of defense methods is proposed to mitigate this threat, they either require complicated modifications to the training process or heavily rely on the specific model architecture, which makes them hard to deploy into real-world applications. Therefore, in this paper, we instead start with fine-tuning, one of the most common and easy-to-deploy backdoor defenses, through comprehensive evaluations against diverse attack scenarios. Observations made through initial experiments show that in contrast to the promising defensive results on high poisoning rates, vanilla tuning methods completely fail at low poisoning rate scenarios. Our analysis shows that with the low poisoning rate, the entanglement between backdoor and clean features undermines the effect of tuning-based defenses. Therefore, it is necessary to disentangle the backdoor and clean features in order to improve backdoor purification. To address this, we introduce Feature Shift Tuning (FST), a method for tuning-based backdoor purification. Specifically, FST encourages feature shifts by actively deviating the classifier weights from the originally compromised weights. Extensive experiments demonstrate that our FST provides consistently stable performance under different attack settings. Without complex parameter adjustments, FST also achieves much lower tuning costs, only 10 epochs. Our codes are available at https: //github.com/AISafety-HKUST/stable_backdoor_purification .

1 Introduction

Deep Neural Networks (DNNs) are shown vulnerable to various security threats. One of the main security issues is backdoor attack [6, 9, 10] that inserts malicious backdoors into DNNs by manipulating the training data or controlling the training process.

To alleviate backdoor threats, many defense methods [36] have been proposed, such as robust training [13, 17, 38] and post-processing purification methods [20, 37, 41]. However, robust training methods require complex modifications to model training process [13, 38], resulting in substantially increased training costs, particularly for large models. Pruning-based purification methods are sensitive to hyperparameters and model architecture [37, 41], which makes them hard to deploy in real-world applications.

Fine-tuning (FT) methods have been adopted to improve models' robustness against backdoor attacks [14, 20, 36] since they can be easily combined with existing training methods and various model architectures. Additionally, FT methods require less computational resources, making them one of the most popular transfer learning paradigms for large pretrained models [5, 21, 25, 28, 35, 40]. However, FT methods have not been sufficiently evaluated under various attack settings, particularly in the more practical low poisoning rate regime [2, 4]. Therefore, we begin by conducting a thorough assessment of widely-used tuning methods, vanilla FT and simple Linear Probing (LP), under different attack configurations.

In this work, we focus on data-poisoning backdoor attacks , as they efficiently exploit security risks in more practical settings [3, 4, 9, 27]. We start our study on whether these commonly used FT methods can efficiently and consistently purify backdoor triggers in diverse scenarios. We observe that vanilla FT and LP can not achieve stable robustness against backdoor attacks while maintaining clean accuracy . What's worse, although they show promising results under high poisoning rates ( 20% , 10% ), they completely fail under low poisoning rates ( 5% , 1% , 0 . 5% ). As shown in Figure 3, we investigate this failure mode and find that model's learned features (representations before linear classifier) have a significant difference under different poisoning rates, especially in terms of separability between clean features and backdoor features. For low poisoning rate scenarios, clean and backdoor features are tangled together so that the simple LP is not sufficient for breaking mapping between input triggers and targeted label without feature shifts. Inspired by these findings, we first try two simple strategies (shown in Figure 1), FE-tuning and FT-init based on LP and FT, respectively, to encourage shifts on learned features. In contrast to LP, FE-tuning only tunes the feature extractor with the frozen and re-initialized linear classifier. FT-init first randomly initializes the linear classifier and then conducts end-to-end tuning. Experimental results show that these two methods, especially FE-tuning, improve backdoor robustness for low poisoning rates, which confirms the importance of promoting shifts in learned features.

Though these two initial methods can boost backdoor robustness, they still face an unsatisfactory trade-off between defense performance and clean accuracy. With analysis of the mechanism behind those two simple strategies, we further proposed a stronger defense method, Feature Shift Tuning (FST) (shown in Figure 1). Based on the original classification loss, FST contains an extra penalty, 〈 w , w ori 〉 , the inner product between the tuned classifier weight w and the original backdoored classifier weight w ori . During the end-to-end tuning process, FST can actively shift backdoor features by encouraging the difference between w and w ori (shown in Figure 3). Extensive experiments have demonstrated that FST achieves better and more stable defense performance across various attack settings with maintaining clean accuracy compared with existing defense methods. Our method also significantly improves efficiency with fewer tuning costs compared with other tuning methods, which makes it a more convenient option for practical applications. To summarize, our contributions are:

We conduct extensive evaluations on various tuning strategies and find that while vanilla Fine-tuning (FT) and simple Linear Probing (LP) exhibit promising results in high poisoning rate scenarios, they fail completely in low poisoning rate scenarios.

We investigate the failure mode and discover that the reason behind this lies in varying levels of entanglement between clean and backdoor features across different poisoning rates. We further propose two initial methods to verify our analysis.

Based on our initial experiments, we propose Feature Shift Tuning (FST). FST aims to enhance backdoor purification by encouraging feature shifts that increase the separability between clean and backdoor features. This is achieved by actively deviating the tuned classifier weight from its originally compromised weight.

Extensive experiments show that FST outperforms existing backdoor defense and other tuning methods. This demonstrates its superior and more stable defense performance against various poisoning-based backdoor attacks while maintaining accuracy and efficiency.

2 Background and related work

Backdoor Attacks. Backdoor attacks aim to mislead the backdoored model to exhibit abnormal behavior on samples stamped with the backdoor trigger but behave normally on all benign samples. They can be classified into 2 categories [36]: (1) data-poisoning attacks: the attacker inserts a backdoor trigger into the model by manipulating the training sample ( x , y ) ∈ ( X , Y ) , like adding a

Tuned

Vanilla LP

Vanilla FT

FT-init (Ours)

Re-initialized

Re-initialized

FST (Ours)

Re-initialized

Original

Deviation

Feature Shift

FE-tuning (Ours)

Tuned

Tuned

Frozen

Tuned

Tuned

Tuned

Tuned

Tuned

Frozen

𝑓(𝑤)

𝜙(𝜃)

small patch in clean image x and assign the corresponding class y to an attacker-designated target label y t . [3, 6, 9, 10, 18, 33]; (2) training-control attacks: the attacker can control both the training process and training data simultaneously [23, 24]. With fewer assumptions about attackers' capability, data-poisoning attacks are much more practical in real-world scenarios [4, 9, 29] and have led to increasingly serious security risks [3, 27]. Therefore, in this work, we mainly focus on data-poisoning backdoor attacks.

Backdoor Defense. Existing backdoor defense strategies could be roughly categorized into robust training [13, 17, 38] and post-processing purification methods [20, 37, 41]. Robust training aims to prevent learning backdoored triggers during the training phase. However, their methods suffer from accuracy degradation and significantly increase the model training costs [17, 38], which is impractical for large-scale model training. Post-processing purification instead aims to remove the potential backdoor features in a well-trained model. The defender first identifies the compromised neurons and then prunes or unlearns them [20, 34, 37, 41, 42]. However, pruning-based and unlearning methods also sacrifice clean accuracy and lack generalization across different model architectures.

Preliminaries of backdoor fine-tuning methods. Without crafting sophisticated defense strategies, recent studies [14, 20, 27, 42] propose defense strategies based on simple Fine-tuning. Here, we introduce two widely used fine-tuning paradigms namely vanilla Fine-tuning (FT) and Linear Probing (LP) (shown in Figure 1) since they would serve as two strong baselines in our following sections. For each tuned model, we denote the feature extractor as ϕ ( θ ; x ) : X → ϕ ( x ) and linear classifier as f ( w ; x ) = w T ϕ ( x ) : ϕ ( x ) →Y . To implement the fine-tuning, both tuning strategies need a set of training samples denoted as D T ⊂ ( X , Y ) to update the model parameters while focusing on different parameter space. Following previous works [5, 14, 16, 20, 21], we implement vanilla FT by updating the whole parameters { θ , w } ; regarding the LP, we only tunes the linear classifier f ( w ) without modification on the frozen ϕ ( θ ) .

Evaluation Metrics. We take two evaluation metrics, including Clean Accuracy (C-Acc) (i.e., the prediction accuracy of clean samples) and Attack Success Rate (ASR) (i.e., the prediction accuracy of poisoned samples to the target class). A lower ASR indicates a better defense performance.

3 Revisiting Backdoor Robustness of Fine-tuning Methods

In this section, we evaluate the aforementioned two widely used fine-tuning paradigms' defensive performance against backdoor attacks with various poisoning rates (FT and LP). Despite that the vanilla FT has been adopted in previous defense work [14, 20, 36, 42], it has not been sufficiently evaluated under various attack settings, particularly in more practical low poisoning rate scenarios. The simple LP method is widely adopted in transfer learning works [5, 16, 21, 22, 28] but still rarely tested for improving model robustness against backdoor attacks. For FT, we try various learning rates during tuning: 0 . 01 , 0 . 02 , and 0 . 03 . For LP, following [16], we try larger learning rates: 0 . 3 , 0 . 5 , and 0 . 7 . We also demonstrate these two methods in Figure 1.

We conduct evaluations on widely used CIFAR-10 [15] dataset with ResNet-18 [11] and test 4 representative data-poisoning attacks including BadNet [10], Blended [6], SSBA [18] and LabelConsistent attack (LC) [33]. We include various poisoning rates, 20% , 10% , 5% , 1% , and 0 . 5% for attacks except for LC since the maximum poisoning rate for LC on CIFAR-10 is 10% . Following previous work [36], we set the target label y t as class 0 . Additional results on other models and datasets are shown in Appendix C.1 . We aim to figure out whether these commonly used FT methods can efficiently and consistently purify backdoor triggers in various attack settings.

Original (10%)    ASR: 99.9%

Original (1%)

ASR: 98.1%

Vanilla FT (1%)     ASR: 82.4%

FT-init (1%)      ASR: 61.2%

FE-tuning (1%)     ASR: 8.5%

FST (1%)     ASR: 0.2%

5

7

8

6

poison

3.1 Revisiting Fine-tuning under Various Poisoning Rates

We stress that the defense method should effectively purify the backdoor triggers while maintaining good clean accuracy. Therefore, we mainly focus on defense performance with a satisfactory clean accuracy level (around 92% ). The results are shown in Figure 2.

Vanilla FT and LP can purify backdoored models for high poisoning rates. From Figure 2, We observe that for high poisoning rates ( ≥ 10% for BadNet, Blended, and SSBA; 5% for LC), both vanilla FT and LP can effectively mitigate backdoor attacks. Specifically, LP (purple lines) significantly reduces the ASR below 5% on all attacks without significant accuracy degradation ( ≤ 2 . 5% ). Compared with vanilla FT, by simply tuning the linear classifier, LP can achieve better robustness and clean accuracy.

Vanilla FT and LP both fail to purify backdoor triggers for low poisoning rates. In contrast to their promising performance under high poisoning rates, both tuning methods fail to defend against backdoor attacks with low poisoning rates ( ≤ 5% for BadNet, Blended, and SSBA; < 5% for LC). Vanilla FT with larger learning rates performs slightly better on Blended attacks, but it also sacrifices clean accuracy, leading to an intolerant clean accuracy drop. The only exception is related to the SSBA results, where the ASR after tuning at a 0 . 5% poisoning rate is low. This can be attributed to the fact that the original backdoored models have a relatively low ASR.

3.2 Exploration of Unstable Defense Performance of Fine-tuning Methods

From the results, a question then comes out: What leads to differences in defense performance of tuning methods under various poisoning rate settings? We believe the potential reason for this phenomenon is that the learned features from feature extractors of backdoored models differ at different poisoning rates, especially in terms of the separability between clean features and backdoor features. We conduct T-SNE visualizations on learned features from backdoored models with Blended attack ( 10% and 1% poisoning rates). The results are shown in Figure 3. The targeted class samples are marked with Red color and Black points are backdoored samples. As shown in first two figures,

under high poisoning rate ( 10% ), backdoor features (black points) are clearly separable from clean features (red points) and thus could be easily purified by only tuning the f ( w ) ; however for low poisoning rate ( 1% ), clean and backdoor features are tangled together so that the simple LP is not sufficient for breaking mapping between input triggers and targeted label without further feature shifts. Furthermore, as depicted in the third figure of Figure 3, though vanilla FT updates the whole network containing both θ and w , it still suffers from providing insufficient feature modification, leading to the failure of backdoor defense.

Can Feature Shift Help Enhance the Purification Performance of Fine-tuning? Based on our analysis, we start with a simple strategy in that we could improve backdoor robustness for low poisoning rates by encouraging shifts in learned features. We then propose separate solutions for both LP and vanilla FT to evaluate the effect of feature shift as well as the robustness against two low poisoning rates, 1% and 0 . 5% . As shown in Table 1, specifically, for the LP, we freeze f ( w ) and only tune ϕ ( θ ) . However, our initial experiments show that directly tuning ϕ ( θ ) does not sufficiently modify learned features since the fixed backdoor linear classifier (denoted as f ( w ori ) ) may still restrict modifications of learned features. Inspired by the previous work [27], we first randomly re-initialize the linear classifier and then tune only ϕ ( θ ) (denoted as FE-tuning). For vanilla FT, we also fine-tune the whole network with a randomly re-initialized linear classifier (denoted as FT-init). More implementation details of FE-tuning and FT-init are shown in Appendix B.3.

Evaluations Verify That Encouraging Feature Shift Could Help Purify Backdoored Models. We observe that both FE-tuning and FT-init could significantly enhance the performance of backdoor purification compared to previous fine-tuning with an average drop of 77 . 70% and 35 . 61% on ASR respectively. Specifically, FE-tuning leads to a much larger improvement, an ASR decrease of over 74% . As shown in the fifth figure of Figure 3, after FE-tuning, backdoor features could be clearly separated from the clean features of the target class (red). However, this simple strategy also leads to a decrease in clean accuracy (around 2 . 9% ) in comparison to the original LP, due to the totally randomized classifier. While for the FT-init, the robustness improvements are less significant. As shown in the fourth figure of Figure 3, simply fine-tuning the backdoor model with re-initialized f ( w ) can not lead to enough shifts on backdoor features under the low poisoning rate. The backdoor and clean features of the target class still remain tangled, similar to the original and vanilla FT.

In summary, the results of these two initial methods confirm the fact that encouraging shifts on learned features is an effective method for purifying backdoors at low poisoning rates. However, both methods still experience a serious clean accuracy drop or fail to achieve satisfactory improvements in robustness. To further enhance the purification performance, we propose a stronger tuning defense in Section 4 that addresses both issues in a unified manner.

4 Feature Shift Tuning: Unified Method to Achieve Stable Improvements

Based on our initial findings, we propose a stronger tuning defense paradigm called Feature Shift Tuning (FST) . FST is an end-to-end tuning method and actively shifts features by encouraging the discrepancy between the tuned classifier weight w and the original backdoored classifier weight w ori . The illustration of FST is shown in Figure 1. Formally, starting with reinitializing the linear classifier

w , our method aims to solve the following optimization problem:

where L stands for the original classification cross-entropy loss over whole model parameters to maintain clean accuracy and ⟨· , ·⟩ denotes the inner product. By adding the regularization on the 〈 w , w ori 〉 , FST encourages discrepancy between w and w ori to guide more shifts on learned features of ϕ ( θ ) . The α balances these two loss terms, the trade-off between clean accuracy and backdoor robustness from feature shift. While maintaining clean accuracy, a larger α would bring more feature shifts and better backdoor robustness. We simply choose the inner-product 〈 w , w ori 〉 to measure the difference between weights of linear classifiers, since it is easy to implement and could provide satisfactory defensive performance in our initial experiments. Other metrics for measuring differences can also be explored in the future.

Projection Constraint. To avoid the w exploding and the inner product dominating the loss function during the finetuning process, we add an extra constraint on the norm of w to stabilize the tuning process. To reduce tuning cost, we directly set it as ∥ w ori ∥ instead of manually tuning it. With the constraint to shrink the feasible set, our method quickly converges in just a few epochs while achieving significant robustness improvement. Compared with previous tuning methods, our method further enhances robustness against backdoor attacks and also greatly improves tuning efficiency (Shown in Section 5.3). Shrinking the range of feasible set also brings extra benefits. It significantly reduces the requirement for clean samples during the FST process. As shown in Figure 7 (c,d), FST consistently performs well across various tuning data sizes, even when the tuning set only contains 50 samples. The ablation study of

Algorithm 1 Feature Shift Tuning (FST)

Input: Tuning dataset D T = { x i , y i } N i =1 ; backdoored model f ( w ori ; ϕ ( θ )) ; learning rate η ; factor α ; tuning iterations I

Output: Purified model

1: Initialize w 0 with random weights

2: Initialize θ 0 with θ

3: for i = 0 , 1 , . . . , I do

4: Sample mini-batch B i from tuning set D T

5: Calculate gradients of θ i and w i : g i θ = ∇ θ i 1 |B i | ∑ x ∈B i L ( f ( w i ; ϕ ( θ i ; x )) , y ) ; g i w = ∇ w i [ 1 |B i | ∑ x ∈B i L ( f ( w i ; ϕ ( θ i ; x )) , y ) + α 〈 w i , w ori 〉 ] ;

6: Update model parameters θ i +1 = θ i -η g i θ , w i +1 = w i -η g i w

7: Norm projection of the linear classifier w i +1 = w i +1 ∥ w i +1 ∥ 2 ∥ w ori ∥ 2

8: end for

9: return Purified model f ( w I ; ϕ ( θ I ))

α is also provided in Section 5.3 and showcases that our method is not sensitive to the selection of α in a wide range. The overall optimization procedure is summarized in Algorithm 1.

Unified Improvement for Previous Tuning Methods. We discuss the connections between FST and our previous FE-tuning and FT-init to interpret why FST could achieve unified improvements on backdoor robustness and clean accuracy. Our objective function Eq.1 could be decomposed into three parts, by minimizing E ( x , y ) ∼D T L ( f ( w ; ϕ ( θ ; x )) , y ) over w and θ respectively, and α 〈 w , w ori 〉 over w . Compared with FE-tuning, FST brings an extra loss term on min w E ( x , y ) ∼D T L ( f ( w ; ϕ ( θ ; x )) , y ) + α 〈 w , w ori 〉 to update linear classifier w . In other words, while encouraging feature shifts, FST updates the linear classifier with the original loss term to improve the models' clean performance. Compared with the FT-init, by adopting α 〈 w , w ori 〉 , FST encourages discrepancy between tuned w and original w ori to guide more shifts on learned features.

Weights of Linear Classifier Could Be Good Proxy of Learned Features. Here, we discuss why we choose the discrepancy between linear classifiers as our regularization in FST. Recalling that our goal is to introduce more shifts in learned features, especially for backdoor features. Therefore, the most direct approach is to explicitly increase the difference between backdoor and clean feature distributions. However, we can not obtain backdoor features without inputting backdoor triggers. We consider that weights of the original compromised linear classifier could be a good proxy of learned features . Therefore, we encourage discrepancy between tuned w and original w ori rather than trying to explicitly promote discrepancy between feature distributions.

5 Experiments

5.1 Experimental Settings

Datasets and Models. We conduct experiments on four widely used image classification datasets, CIFAR-10 [15], GTSRB [32], Tiny-ImageNet [8], and CIFAR-100 [15]. Following previous works [13, 20, 24, 36], we implement backdoor attacks on ResNet-18 [11] for both CIFAR-10 and GTSRB and also explore other architectures in Section 5.3. For CIFAR-100 and Tiny-ImageNet, we adopt pre-trained SwinTransformer [21] (Swin). For both the CIFAR-10 and GTSRB, we follow the previous work [41] and leave 2% of original training data as the tuning dataset. For the CIFAR100 and Tiny-ImageNet, we note that a small tuning dataset would hurt the model performance and therefore we increase the tuning dataset to 5% of the training set.

Attack Settings. All backdoor attacks are implemented with BackdoorBench 1 . We conduct evaluations against 6 representative data-poisoning backdoors, including 4 dirty-label attacks (BadNet [10], Blended attack [6], WaNet [24], SSBA[18]) and 2 clean-label attacks (SIG attack [1], and Labelconsistent attack (LC) [33]). For all the tasks, we set the target label y t to 0, and focus on low poisoning rates, 5% , 1% , and 0 . 5% in the main experimental section. We also contain experiments of high poisoning rates , 10% , 20% , and 30% , in Appendix C.2. For the GTSRB dataset, we do not include LC since it can not insert backdoors into models (ASR < 10% ). Out of all the attacks attempted on Swin and Tiny-ImageNet, only BadNet, Blended, and SSBA were able to successfully insert backdoor triggers into models at low poisoning rates. Other attacks resulted in an ASR of less than 20% . Therefore, we only show the evaluations of these three attacks. More details about attack settings and trigger demonstrations are shown in Appendix B.2.

Baseline Defense Settings. We compare our FST with 4 tuning-based defenses including Finetuning with Sharpness-Aware Minimization (FT+SAM), a simplified version adopted from [42], Natural Gradient Fine-tuning (NGF) [14], FE-tuning and FT-init proposed in Section 3.2. We also take a comparison with 2 extra strategies including Adversarial Neural Pruning (ANP) [37] and Implicit Backdoor Adversarial Unlearning (I-BAU) [38] which achieve outstanding performance in BackdoorBench [37]. The implementation details of baseline methods are shown in the Appendix B.3. For our FST, we adopt SGD with an initial learning rate of 0 . 01 and set the momentum as 0 . 9 for both CIFAR-10 and GTSRB datasets and decrease the learning rate to 0 . 001 for both CIFAR-100 and Tiny-ImageNet datasets to prevent the large degradation of the original performance. We fine-tune the models for 10 epochs on the CIFAR-10; 15 epochs on the GTSRB, CIFAR-100 and Tiny-ImageNet. We set the α as 0.2 for CIFAR-10; 0 . 1 for GTSRB; 0 . 001 for both the CIFAR-100 and Tiny-ImageNet.

5.2 Defense Performance against Backdoor Attacks

In this section, we show the performance comparison between FST with tuning-based backdoor defenses (FT+SAM [42], NGF [14], FE-tuning and FT-init) and current state-of-the-art defense methods, ANP [37] and I-BAU [38]. We demonstrate results of CIFAR-10, GTSRB, and TinyImageNet on Table 2, 3, and 4, repectively. We leave results on CIFAR-100 to Appendix C.3.

Experimental results show that our proposed FST achieves superior backdoor purification performance compared with existing defense methods . Apart from Tiny-ImageNet, FST achieves the best performances on CIFAR-10 and GTSRB. The average ASR across all attacks on three datasets are below 1% , 0 . 52% in CIFAR-10, 0 . 41% in GTSRB, and 0 . 19% in Tiny-ImageNet, respectively. Regarding two tuning defense methods, FT+SAM and NGF, FST significantly improves backdoor robustness with larger ASR average drops on three datasets by 34 . 04% and 26 . 91% . Compared with state-of-the-art methods, ANP and I-BAU, on CIFAR-10 and GTSRB, our method outperforms with a large margin by 11 . 34% and 32 . 16% on average ASR, respectively. ANP is only conducted in BatchNorm layers of ConvNets in source codes. Therefore, it can not be directly conducted in SwinTransformer. Worth noting is that our method achieves the most stable defense performance across various attack settings with 0 . 68% on the average standard deviation for ASR. At the same time, our method still maintains high clean accuracy with 93 . 07% on CIFAR-10, 96 . 17% on GTSRB, and 79 . 67% on Tiny-ImageNet on average. Compared to ANP and I-BAU, FST not only enhances backdoor robustness but also improves clean accuracy with a significant boost on the clean accuracy by 2 . 5% and 1 . 78% , respectively.

Following the experimental setting in Section 3.2, we also conduct feature visualization of FST in Figure 3. It can be clearly observed that our FST significantly shifts backdoor features and makes them easily separable from clean features of the target class like FE-tuning. Combined with our outstanding performance, this further verifies that actively shifting learned backdoor features can effectively mitigate backdoor attacks. We also evaluate two initial methods, FE-tuning and FT-init. FE-tuning performs best on Tiny-ImageNet, leading to a little ASR drop by 0 . 18% compared with FST. However, it also significantly sacrifices clean accuracy, leading to a large C-ACC drop by 8 . 1% compared with FST. FST outperforms them by a large margin for backdoor robustness on CIFAR-10 and GTSRB. As we expected, FST achieves stable improvements in robustness and clean accuracy compared with FE-tuning and FT-init.

Adaptive Attacks. To further test the limit of FST, we conduct an adaptive attack evaluation for it. To bypass defense methods based on latent separation property, adaptive poisoning attacks [26] actively suppress the latent separation between backdoor and clean features by adopting an extremely low poisoning rate and adding regularization samples. Their evaluations also show that these attacks successfully bypass existing strong latent separation-based defenses. Hence, we believe it is also equally a powerful adaptive attack against our FST method. We evaluate our method against adaptive attacks. The results of the AdaptiveBlend attack are shown in Table 2, 3. The results of more adaptive attacks and various regularization samples are shown in Appendix C.4. We could observe that FST could still effectively purify such an adaptive

Adaptive-Blend   ASR: 83.03%

FST                    ASR: 1.37%

attack achieving an average drop in ASR by 81 . 66% and 91 . 75% respectively for the CIFAR-10 and GTSRB.

To explain why adaptive attacks fail, we provide TSNE visualizations of learned features from backdoored models. We show the results in Figure 4. We can first observe that adaptive attacks significantly reduce latent separability. Clean and backdoor features are tightly tangled. FST effectively shifts backdoor features and makes them easily separable from the clean features of the target class. Therefore, the feature extractor will no longer confuse backdoor features with clean features of the target class in the feature space. This leads to the subsequent simple linear classifier being difficult to be misled by backdoor samples, resulting in more robust classification.

5.3 Ablation Studies of FST

Below, we perform ablation studies on our proposed FST to analyze its efficiency and sensitivity to hyperparameters, tuning sizes, and architectures. This further demonstrates the effectiveness of FST in practical scenarios.

Efficiency analysis. we evaluate the backdoor defense performance of our method and 4 tuning strategies (FT+SAM, NGF, FE-tuning, and FT-init) across different tuning epochs. We test against 4 types of backdoor attacks including (BadNet, Blended, SSBA, and LC) with 1% poisoning rate and take experiments on CIFAR-10 and ResNet-18 models (see Figure 6). Notably, FST can effectively purify the backdoor models with much fewer epochs, reducing backdoor ASR below 5% . This demonstrates that our method also significantly improves tuning efficiency. We also find that added projection constraint also helps FST converge. We offer the performance of FST's entire tuning process on CIFAR-10 and ResNet-18, with or without the projection term. The results of the BadNet and Blended are shown in Figure 5. We could clearly observe that the projection stabilizes the tuning process of FST. The projection term helps FST quickly converge, particularly in terms of accuracy. We leave the analysis with more attacks in Appendix C.5.

Sensitivity analysis on α . We evaluate the defense performance of FST with various α in Eq 1. We conduct experiments on CIFAR-10 and ResNet-18 and test against BadNet and Blended attacks with 5% and 1% . The results are shown in (a) and (b) of Figure 7. We can observe that as the α increases from 0 to 0 . 1 , the backdoor ASR results rapidly drop below 5% . As we further increase the α , our method maintains a stable defense performance (ASR < 4% ) with only a slight accuracy degradation ( < 1 . 5% ). It indicates that FST is not sensitive to α . The results further verify that FST could achieve a better trade-off between backdoor robustness and clean accuracy.

Sensitivity analysis on tuning dataset size. Here, we evaluate the FST under a rigorous scenario with limited access to clean tuning samples. We test our method on the CIFAR-10 using tuning datasets of varying sizes, ranging from 0 . 1% to 2% . Figure 7 shows that FST consistently performs

Figure 6: The ASR results of four backdoor attacks with varying tuning epochs of tuning methods.

(a)

(b)

(c)

(d)

well across various tuning data sizes. Even if the tuning dataset is reduced to only 0 . 1% of the training dataset (50 samples in CIFAR-10), our FST can still achieve an overall ASR of less than 2% .

Analysis on model architecture. We extend the experiments on other model architectures including VGG19-BN [31], ResNet-50 [11], and DenseNet161 [12] which are widely used in previous studies [14, 37, 42]. Here we show the results of BadNet and Blended attacks with 5% , 1% , and 0 . 5% poisoning rates on the CIFAR-10 dataset. Notably, the structure of VGG19-BN is slightly different where its classifier contains more than one linear layer. Our initial experiments show that directly applying FST to the last layer fails to purify backdoors. Therefore, we simply extend our methods to the whole classifier and observe a significant promotion. We leave the remaining attacks and more implementation details in the Appendix D.2. The results presented in Figure 8 show that our FST significantly enhances backdoor robustness for all three architectures by reducing ASR to less than 5% on average. This suggests that our approach is not influenced by variations in model architecture.

6 Conclusion and limitations

In this work, we concentrate on practical Fine-tuning-based backdoor defense methods. We conduct a thorough assessment of widely used tuning methods, vanilla FT and LP. The experiments show that they both completely fail to defend against backdoor attacks with low poisoning rates. Our further experiments reveal that under low poisoning rate scenarios, the backdoor and clean features from the compromised target class are highly entangled together, and thus disentangling the learned features is required to improve backdoor robustness. To address this, we propose a novel defense approach called Feature Shift Tuning (FST), which actively promotes feature shifts. Through extensive evaluations, we demonstrate the effectiveness and stability of FST across various poisoning rates, surpassing existing strategies. However, our tuning methods assume that the defender would hold a clean tuning set which may not be feasible in certain scenarios. Additionally, they also lead to a slight compromise on accuracy in large models though achieving robustness . This requires us to pay attention to protect learned pretraining features from being compromised during robust tuning [7, 16, 19, 39].

References

Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP) , pages 101-105. IEEE, 2019. 5.1

Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations , 2021. 1

Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=iC4UHbQ01Mp . 1, 2

Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302.10149 , 2023. 1, 2

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020. 1, 2, 3

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 , 2017. 1, 2, 3, 5.1

Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Towards understanding feature learning in out-of-distribution generalization. arXiv preprint arXiv:2304.11327 , 2023. 6

Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819 , 2017. 5.1

Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022. 1, 2

Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access , 7:47230-47244, 2019. 1, 2, 3, 5.1

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 3, 5.1, 5.3

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017. 5.3

Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=TySnJ-0RdKI . 1, 2, 5.1

Nazmul Karim, Abdullah Al Arafat, Umar Khalid, Zhishan Guo, and Nazanin Rahnavard. In search of smooth minima for purifying backdoor in deep neural networks. 1, 2, 3, 5.1, 5.2, 5.3

Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 3, 5.1

Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=UYneFzXSJWh . 2, 3, 6

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems , 34:1490014912, 2021. 1, 2, B.1

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16463-16472, 2021. 2, 3, 5.1

Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. arXiv preprint arXiv:2309.17230 , 2023. 6

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses , pages 273-294. Springer, 2018. 1, 2, 3, 5.1

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021. 1, 2, 3, 5.1

John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning , pages 7721-7735. PMLR, 2021. 3

Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems , 33:3454-3464, 2020. 2

Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum? id=eEn8KTtJOx . 2, 5.1

Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050 , 2021. 1

Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In The eleventh international conference on learning representations , 2023. 5.2, B.2, C.4

Zeyu Qin, Liuyi Yao, Daoyuan Chen, Yaliang Li, Bolin Ding, and Minhao Cheng. Revisiting personalized federated learning: Robustness against backdoor attacks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 4743-4755, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. 1, 2, 3.2

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 1, 3

Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1354-1371. IEEE, 2022. 2

Reza Shokri et al. Bypassing backdoor detection algorithms in deep learning. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P) , pages 175-183. IEEE, 2020. C.4

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 5.3

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks , 32:323-332, 2012. 5.1

Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771 , 2019. 2, 3, 5.1

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP) , pages 707-723. IEEE, 2019. 2

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=gEZrGCozdqR . 1

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. 1, 2, 3, 5.1, B.1, B.2

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems , 34:16913-16925, 2021. 1, 2, 5.1, 5.2, 5.3, B.1

Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=MeeQkFYVbzW . 1, 2, 5.1, 5.2, B.1

Jianyu Zhang and Léon Bottou. Learning useful representations for shifting tasks and distributions. In International Conference on Machine Learning , pages 40830-40850. PMLR, 2023. 6

Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10174-10183, 2022. 1

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V , pages 175-191. Springer, 2022. 1, 2, 5.1, B.1

Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. arXiv preprint arXiv:2304.11823 , 2023. 2, 3, 5.1, 5.2, 5.3, B.3

A Social Impact

Deep Neural Networks (DNNs) are extensively applied in today's society especially for some safetycritical scenarios like autonomous driving and face verification. However, the data-hungry nature of these algorithms requires operators to collect massive amounts of data from diverse sources, making source tracing difficult and increasing the risk of potential malicious issues. For example, attackers can blend poisoned data into benign samples and embed backdoors into models without training control, posing a significant threat to model deployment. Therefore, to mitigate these risks, defenders must remove potential backdoors from models before real-world deployment, ensuring safety and trustworthiness. Our work focuses on a lightweight plug-and-play defense strategy applicable in real scenarios with minimal modifications to existing pipelines. We hope to appeal to the community to prioritize practical defensive strategies that enhance machine learning security.

B Experimental Settings

B.1 Datasets and Models.

Following previous works [17, 36, 37, 38] in backdoor literature, we conduct our experiments on four widely used datasets including CIFAR-10, GTSRB, Tiny-ImageNet, and CIFAR-100.

CIFAR-10 and GTSRB are two widely used datasets in backdoor literature containing images of 32 ∗ 32 resolution of 10 and 43 categories respectively. Following [37, 41], we separate 2% clean samples from the whole training dataset for backdoor defense and leave the rest training images to implement backdoor models. For these two datasets, we utilize the ResNet-18 to construct the backdoor models.

CIFAR-100 and Tiny-ImageNet are two datasets with larger scales compared to the CIFAR10 and GTSRB which contain images with 64 ∗ 64 resolution of 100 and 200 categories respectively. For these two datasets, we enlarge the split ratio and utilize 5% of the training dataset as backdoor defense since a smaller defense set is likely to hurt the model performance. For these two datasets, we utilize the pre-trained SwinTransformer (pre-trained weights on ImageNet are provided by PyTorch ) to implement backdoor attacks since we find that training these datasets on ResNet-18 from scratch would yield a worse model performance with C-Acc ( < 70% ) on average and therefore is not practical in real scenarios.

B.2 Attack Configurations

We conducted all the experiments with 4 NVIDIA 3090 GPUs.

We implement 6 representative poisoning-based attacks and an adaptive attack called AdaptiveBlend [26]. For 6 representative attacks, most of them are built with the default configurations 2 in BackdoorBench [36]. For the BadNet, we utilize the checkerboard patch as backdoor triggers and stamp the pattern at the lower right corner of the image; for the Blended, we adopt the Hello-Kitty pattern as triggers and set the blend ratio as 0 . 2 for both training and inference phase; for WaNet, we set the size of the backward warping field as 4 and the strength of the wrapping field as 0.5; for SIG, we set the amplitude and frequency of the sinusoidal signal as 40 and 6 respectively; for SSBA and LC, we adopt the pre-generated invisible trigger from BackdoorBench. For the extra adaptive attack, we utilize the official implementation 3 codes and set both the poisoning rate and cover rate as 0 . 003 following the original paper. The visualization of the backdoored images is shown in Figure 9.

For CIFAR-10 and GTSRB, we train all the backdoor models with an initial learning rate of 0 . 1 except for the WaNet since we find a large initial learning rate would make the attack collapse, and therefore we decrease the initial learning rate to 0.01. All the backdoor models are trained for 100 epochs and 50 epochs for CIFAR-10 and GTSRB respectively. For CIFAR-100 and Tiny-ImageNet, we adopt a smaller learning rate of 0 . 001 and fine-tune each model for 10 epochs since the SwinTransformer is already pre-trained on ImageNet and upscale the image size up to 224 ∗ 224 before feeding the image to the network.

Original

BadNet

Blended

WaNet

SIG

SSBA

LC

B.3 Baseline Defense Configurations

We evaluate 4 tuning-based defenses and 2 extra state-of-the-art defense strategies including both ANP and I-BAU for comparison. For tuning-based defenses, we mainly consider 2 recent works including FT+SAM and NGF, and we also compare another 2 baseline tuning strategies including FE-tuning and FT-init proposed in our paper. For all defense settings, we set the batch size as 128 on CIFAR10 and GTSRB and set the batch size as 32 on CIFAR-100 and Tint-ImageNet due to the memory limit.

FT+SAM: Upon completion of our work, the authors of [42] had not yet made their source code publicly available. Therefore, we implemented a simplified version of their FT-SAM algorithm, where we replaced the optimizer with SAM in the original FT algorithm and called it FT+SAM. For both CIFAR-10 and GTSRB, we set the initial learning rate as 0 . 01 and fine-tune models with 100 epochs. We set the ρ as 8 and 10 for CIFAR-10 and GTSRB respectively since we find the original settings ( ρ = 2 for CIFAR-10 and ρ = 8 for GTSRB) are not sufficient for backdoor purification in our experiments. For CIFAR-100 and Tiny-ImageNet, we set the initial learning rate as 0 . 001 and ρ as 6 , and fine-tune the backdoor model for 20 epochs for fair comparison.

NGF: We adopt the official implementation 4 for NGF. For CIFAR-10 and GTSRB, we set the tuning epochs as 100 and the initial learning rate as 0 . 015 and 0 . 05 respectively. While for CIFAR-100 and Tiny-ImageNet, we set the tuning epochs as 20 and the initial learning rate as 0 . 002 .

FE-tuning: For FE-tuning, we first re-initialize and freeze the parameters in the head. We then only fine-tune the remaining feature extractor. For CIFAR-10 and GTSRB, we set the initial learning rate as 0 . 01 and fine-tune the backdoor model with 100 epochs; while for CIFAR-100 and Tiny-ImageNet, we set the initial learning rate as 0 . 005 and fine-tune the backdoor model with 20 epochs.

FT-init: For FT-init, we randomly re-initialize the linear head and fine-tune the whole model architecture. For CIFAR-10 and GTSRB, we set the initial learning rate as 0 . 01 and fine-tune the backdoor model with 100 epochs; while for CIFAR-100 and Tiny-ImageNet, we set the initial learning rate as 0 . 005 and fine-tune the backdoor model with 20 epochs.

ANP: We follow the implementation in BackdoorBench and set the perturbation budget as 0 . 4 and the trade-off coefficient as 0 . 2 following the original configuration. We find that within a range of thresholds, the model performance and backdoor robustness are related to the selected threshold. Therefore, we set a threshold range (from 0 . 4 to 0 . 9 ) and present the purification results with low ASR and meanwhile maintain the model's performance.

I-BAU: We follow the implementation in BackdoorBench and set the initial learning rate as 1 e -4 and utilize 5 iterations for fixed-point approximation.

BadNet

Blended

LC

SSBA

100

80

ASR

60

a)

40

FT (0.005)

FT (0.005)

FT (0.005)

FT (0.005)

FT (0.01)

FT (0.01)

FT (0.01)

FT (0.01)

20

LP (0.3)

LP (0.3)

LP (0.3)

LP (0.3)

LP (0.5)

LP (0.5)

LP (0.5)

LP (0.5)

0

0.5%

1%

5%

10%

20% 0.5%

1%

5%

10%

20% 0.5%

1%

5%

10% 0.5%

1%

5%

10%

20%

Poisoning Rate

Poisoning Rate

Poisoning Rate

Poisoning Rate

BadNet

Blended

LC

SSBA

100

80

(b)

ASR

60

40

FT (0.005)

FT (0.005)

FT (0.005)

FT (0.005)

FT (0.01)

FT (0.01)

FT (0.01)

FT (0.01)

20

LP (0.5)

LP (0.3)

LP (0.3)

LP (0.5)

LP (0.5)

LP (0.3)

LP (0.3)

LP (0.5)

0

0.5%

1%

5%

10%

20% 0.5%

1%

5%

10%

20% 0.5%

1%

5%

10% 0.5%

1%

5%

10%

20%

Poisoning Rate

Poisoning Rate

Poisoning Rate

Poisoning Rate

BadNet

Blended

SSBA

100

80

(c)

ASR

60

FT (0.01)

FT (0.02)

FT (0.01)

FT (0.02)

FT (0.02)

FT (0.01)

40

FT (0.03)

FT (0.03)

FT (0.03)

LP (0.3)

LP (0.3)

LP (0.3)

20

LP (0.5)

LP (0.5)

LP (0.5)

LP (0.7)

LP (0.7)

LP (0.7)

0

0.5%

1%

5%

10%

20% 0.5%

1%

5%

10%

20% 0.5%

1%

5%

10%

20%

Poisoning Rate

Poisoning Rate

Poisoning Rate

C Additional Experimental results

C.1 Additional Results of Revisiting Fine-tuning

In this section, we provide additional experimental results for Section 3 to explore the potential influence of the dataset and model selection. Specifically, in addition to our initial experiments of revisiting fine-tuning on CIFAR-10 with ResNet-18, we further vary the model capacity (ResNet-50 on CIFAR-10), the model architecture (DenseNet-161 on CIFAR-10), and the dataset (ResNet-18 on GTSRB). As mentioned in Section 3.1, we mainly focus on defense performance with a satisfactory clean accuracy level ( 92% on CIFAR-10, 97% on GTSRB). We tune hyperparameters based on this condition. All the experimental results are shown in Figure 10 respectively. These additional results also demonstrate that Vanilla FT and LP could purify backdoored models for high poisoning rates but fail to defend against low poisoning rates attacks. The only exception is the SSBA results since the original backdoored models have a relatively low ASR, as mentioned in Section 3.1.

C.2 Additional Results of High Poisoning Rates

Our previous experiments in Section 5.2 have demonstrated our FST's superior defense capacity against backdoor attacks with low poisoning rates. In this section, we further extend our attack scenarios with more poisoning samples by increasing the poisoning rate to 10%, 20%, and 30%. We conduct experiments on CIFAR-10 and GTSRB with ResNet-18 and present the experimental results in Table 5. We observe that the FST could easily eliminate the embedded backdoor as expected while preserving a high clean accuracy of the models.

C.3 Additional Results on CIFAR-100 Dataset

We evaluate our FST on the CIFAR-100 dataset with the results shown in Table 6. Since some attacks show less effectiveness under a low poisoning rate with ASR < 25% , we hence only report the results where the backdoor attack is successfully implemented (original ASR ≥ 25% ). We note that our FST could achieve excellent purification performance across all attack types on the CIFAR-100 dataset with an average ASR 0 . 36% which is 65 . 9% and 9 . 46% lower than the ASR of two other tuning strategies, FT+SAM and NGF respectively. Although the FE-tuning could achieve a lower

ASR compared to FST, we note that its C-Acc gets hurt severely since it freezes the re-initialized linear head during fine-tuning which restricts its feature representation space. For the other two state-of-the-art defenses, we find that they are less effective in purifying the larger backdoor models.

We further observe that the FT-init could achieve comparable purification results as the FST with even a slightly higher C-Acc. Compared to our previous experiments on the small-scale dataset (CIFAR-10 and GTSRB) and model (ResNet-18), we find that FT-init is more effective on the large model (SwinTransformer) with the large-scale dataset (CIFAR-100 and Tiny-ImageNet) which decreases the average ASR by 32 . 31% .

C.4 Additional Results of Adaptive Attacks

In addition to the Adaptive-Blend attack, we also provide evaluations of a parallel attack proposed in [26] called Adaptive-Patch. To further reduce latent separability and improve adaptiveness against latent separation-based defenses, we also use more regularization samples, following ablation study of Section 6.3 [26]. The experimental results are presented in Table 7 and demonstrate that our FST could purify both attack types with various regularization samples. We also demonstrate a T-SNE visualization of the Adaptive-Patch in Figure 11. It aligns with the results of Adaptive-Blend attack.

To further assess stability of FST, we also test FST against training-control adaptive attacks [30]. The authors [30] utilize an adversarial network regularization during the training process to minimize differences

Adaptive-Patch   ASR: 61.23%

FST                    ASR: 2.21%

between backdoor and clean features in latent representations. Since the authors do not provide source code, we follow their original methodology and implement their Adversarial Embedding attack with two types of trigger, namely the checkboard patch (Bypass-Patch) and Hello-Kitty pattern

(Bypass-Blend). All the experimental results along with three poisoning rates are shown in Table 8. The results reveal that our FST could still mitigate the Bypass attack which emphasizes the importance of feature shift in backdoor purification.

C.5 Additional Results of Projection Constraint Analysis

In this section, we first provide additional analysis of the projection constraint with more attacks (WaNet, SSBA, SIG, and LC) on the CIFAR-10 dataset. We show the experimental results in Figure 12. We get the same observations shown in Section 5.3, where the inclusion of the projection term plays a crucial role in stabilizing and accelerating the convergence process of the FST. This results in a rapid and satisfactory purification of the models within a few epochs.

D Extra Ablation Studies

D.1 Efficiency Analysis

We compare the backdoor purification efficiency of our FST with other tuning methods on the remaining three datasets including GTSRB, CIFAR-100, and Tiny-ImageNet. We select three

representative attacks (BadNet, Blend, and SSBA) with poisoning rate 1% which could be successfully implemented across three datasets and we present our experimental results in Figure 13, 14 and 15. The experimental results demonstrate that our FST is efficient compared to the other 4 tuning-based backdoor defense which could constantly depress the ASR under a low-value range (usually < 5% ) with only a few epochs. Besides, we also note that in the GTSRB dataset, both the ASR of FE-tuning and FT-init would increase as the tuning epoch increases indicating the model is gradually recovering the previous backdoor features. Our FST, however, maintains a low ASR along the tuning process which verifies the stability of our method.

D.2 Diverse Model Architecture

We conduct comprehensive evaluations on three model architectures (VGG19-BN, ResNet-50, and DenseNet-161) on the CIFAR-10 dataset with all 6 representative poisoning-based backdoor attacks and one adaptive attack, and our experimental results are shown in Table 9, 10 and 11. During our initial experiments, we note that our method is less effective for VGG19-BN. One possible reason is that the classifier of VGG19-BN contains more than one layer which is slightly different from our previously used structure ResNet-18. Therefore, one direct idea is to extend our original last-layer regularization to all the last linear layers of VGG19-BN. For implementation, we simply change the original α 〈 w , w ori 〉 to α ∑ i 〈 w i , w ori i 〉 where i indicates each linear layer. Based on this, we obtain an obvious promotion of backdoor defense performances (shown in Figure 16) without sacrificing clean accuracy.

Following the results in Table 9, our FST could achieve better and much more stable performance across all attack settings with an average ASR of 6 . 18% and a standard deviation of 11 . 64% . Compared with the four tuning-based defenses, our FST could achieve 29% lower on ASR average across all the attack settings; compared with the other two state-of-the-art defensive strategies, our FST achieves a much lower ASR while getting a much smaller C-Acc drop ( < 3 . 5% ). For the other two architectures, we note that our FST could achieve the best performance across all attack settings with an average ASR of 2 . 7% and maintain clean accuracy (the drop of C-Acc < 1 . 9% ).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on the observations that the previous FT and FP methods fail at low poisoning rate scenarios, this paper finds out the potential reasons in terms of clean and backdoor feature separation degree. It proposes FST to solve the problem by disentangling the features and validates its effectiveness with multiple backdoor attacks.\n\nSTRENGTHS:\n1. The paper is well-written and easy to read. No typo is found.\n2. The experiments are sufficient to demonstrate the findings of previous methods’ failure in low poisoning rate scenarios. And the experiments to verify the potential reason and validate the two initial methods are also convincing.\n3. The proposed methods are intuitively reasonable, which are derived step-by-step from the observations of the pre-experiments to the simple solutions, and the final version with penalty.\n4. The idea is simple yet effective as shown in Sections 4 and 5, which is easy to follow.\n\nWEAKNESSES:\nIn general, I appreciate this paper, but there are still some concerns from my perspective.\n\n1. The implementation of the proposed methods is not publicly available, only the public library ‘BackdoorBench’ is provided.\n2. The effectiveness of the constraint term C in equation (1) is expected to be discussed in the ablation study.\n\nQUESTIONS:\n1. Do the dynamic backdoor attacks, such as WaNet[1] and IAB[2], which generate triggers according to the input, follow the same phenomenon as in Figures 1 and 2? Since the WaNet is used in Section 5 while not in the pre-experiment part.\n\n[1] Nguyen, Anh, and Anh Tran. \"Wanet--imperceptible warping-based backdoor attack.\" *arXiv preprint arXiv:2102.10369* (2021).\n\n[2] Nguyen, Tuan Anh, and Anh Tran. \"Input-aware dynamic backdoor attack.\" *Advances in Neural Information Processing Systems* 33 (2020): 3454-3464.\n\nLIMITATIONS:\nPlease see above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the effectiveness of finetuning in backdoor defense with a low poisoning rate and finds that the feature entanglement at low poisoning rate affects the effectiveness of finetuning. Thus, this paper proposes 3 new finetuning strategy FE-tuning, FT-init, and FST. The experiments demonstrate a promising result.

STRENGTHS:
1. The finetuning study is interesting.
2. FST only uses finetuning to achieve the best backdoor removal results, which is very impressive.
3. This paper is well written and easy to follow.

WEAKNESSES:
Although this work is interesting, I still have several concerns.
1. How much data is used in FST? From my point of view, the reinitialization and the larger difference loss needs more data. But if the amount of data is too large, the defenders can use these data to retrain a new model.
2. As demonstrated in the paper, only finetuning the feature extractors cannot remove backdoor successfully. If so, it demonstrates that there is poison in the linear layers, can you explain this phenomenon?
3. Can you provide more TSNE results against more attacks like Figure3 such as BadNets and WaNet?
4. Although this paper focuses on a low poisoning rate, I think results on poisoning rate such as 20% should be also considered. Because the defenders cannot know the real poisoning rate and the robustness of different poisoning rates is important.

QUESTIONS:
1. How much data is used in FST? From my point of view, the reinitialization and the larger difference loss needs more data. But if the amount of data is too large, the defenders can use these data to retrain a new model.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper observes that while fine-tuning and linear probing can act as effective defenses in the high-poisoning-rate regime, they completely fail in the low-poisoning-rate regime. The paper further shows that this is due to the fact that in the low-poisoning-rate regime, extracted features of backdoored and clean samples are highly entangled. The proposed defense is to fine-tune the backdoor model with an additional constraint to force the fine-tuned weights to differ from the original (poisoned) weights. Extensive experiments show that the proposed defense provides excellent backdoor purification while maintaining clean accuracy.\n\nSTRENGTHS:\n- This paper shows an interesting observation that fine-tuning and linear probing perform significantly worse as backdoor defenses in the low poisoning rate regime.\n- This observation is followed by a thorough and nicely written analysis in Section 3.\n- The proposed defense is simple, elegant, yet effective.\n- Very extensive experiments.\n\nWEAKNESSES:\nPlease pay more attention to weaknesses marked with \"major\" severity.\n- **[minor, clarity]**: L52, please also mention that FE-tuning randomly re-initializes the classifier head. I was confused when looking at Figure 1 at first because it was not mentioned previously that FE-tuning does re-initialize $f(w)$.\n- **[minor, typo]**: L102, \"Here We\" -> \"Here, we\"\n- **[major, lack of experiments on high poisoning rates]**: The proposed defense should be tested with high poisoning rates just to ensure that it performs equally well.\n\nQUESTIONS:\n- It would be nice if the authors could extend Figure 2 to other datasets. Specifically, I am interested in seeing the results on Tiny-ImageNet and CIFAR-100.\n\nLIMITATIONS:\n- One potential limitation of this work is the assumption of having access to clean data for training, which may not always be practical in real-world scenarios. However, the authors have acknowledged this limitation and expressed their commitment to addressing it in future research.\n- It would have been more insightful if the authors had delved deeper into why low poisoning rates cause entanglement between backdoored and clean features.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper finds that fine-tuning is less effective in defending against backdoor attacks with a low poisoning rate, due to the strong coupling between the clean feature and the backdoor feature. Therefore, a tuning-based backdoor purification method called feature shift tuning (FST) is proposed, which is simple and efficient. FST can effectively decouple the clean feature and the backdoor feature, and thus eliminate the backdoor in the victim model.

STRENGTHS:
1. This paper proposes a simple and effective backdoor elimination method, which achieves good results in a specific scenario where the poisoning rate of the backdoor attack is relatively low.

WEAKNESSES:
1. This paper finds that fine-tuning has difficulty defending against backdoor attacks with a low poisoning rate when ResNet-18 is the victim model, but does not sufficiently demonstrate the generalizability of the problem. The same claim is not guaranteed to hold when the model capacity, model architecture, and dataset type change.

2. This paper lacks an explanation of why the method works and does not explore the effect of the range of poisoning rates on the victim model.

3. It is desirable to include backdoor elimination methods other than tuning-based methods as baseline methods as well to demonstrate the efficiency of FST.

QUESTIONS:
Please refer to weakness.

LIMITATIONS:
It is suggested that the authors provide further explanation as to why the method worked.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel backdoor defense approach called Feature Shift Tuning (FST) that actively promotes feature shifts to disentangle the learned features of a deep neural network. The authors conduct a thorough assessment of widely used tuning methods, vanilla Fine-tuning (FT), and simple Linear Probing (LP), and demonstrate that they both completely fail to defend against backdoor attacks with low poisoning rates. \n\nFST is an end-to-end tuning method that actively shifts features during fine-tuning by encouraging the difference between the tuned classifier weight w and the original backdoored classifier weight w_ori. FST significantly shifts backdoor features and makes them easily separable from clean features of the target class, which effectively mitigates backdoor attacks. In summary, the proposed FST method is a strong backdoor defense paradigm that can achieve stable improvements in both robustness and clean accuracy compared to other initial methods such as FE-tuning and FT-init.\n\nSTRENGTHS:\n- The proposed FST (Fine-tuning with Separated Transformations) method is a strong backdoor defense paradigm that can achieve stable improvements in both robustness and clean accuracy compared to other initial methods such as FE-tuning and FT-init. FST is an end-to-end tuning method that actively shifts features during fine-tuning by encouraging the difference between the tuned classifier weight w and the original backdoored classifier weight w_ori. This is giving a good reference to the research on this aspect.\n\n- FST disentangles the clean and backdoor features, which helps to purify the model. It significantly shifts backdoor features and makes them easily separable from clean features of the target class, which effectively mitigates backdoor attacks. FST outperforms other methods by a large margin for backdoor robustness on CIFAR-10 and GTSRB datasets. \n\n- FST is effective against adaptive attacks such as the Adaptive-Blend attack, achieving an average drop on ASR by 81.66% and 91.75% respectively for the CIFAR-10 and GTSRB datasets. FST is a unified method that addresses both the clean accuracy drop and unsatisfied robustness improvements that are present in other backdoor defense methods. The ablation study is organized well to clearly demonstrate the whole proposed method. And it makes the paper easy to follow.\n\nWEAKNESSES:\n- [Contribution and novelty] The contribution of paper is somehow incremental, as the method is developed based on previously researched findings. The backdoor attack can be erased by decoupling the feature extractor and the linear classifier in [1].\n\n- [Adaptive Attack] There, the submission only briefly mentions an attempt at achievingthe  disentanglement between the clean and backdoor features through an adaptive attack, but fails to do so effectively, and provides only limited insight why the adaptive attack failed. Please take a closer look at providing a strong evaluation of the adaptive attack scenario. Besides, the paper [2] also proposes another defense, Adaptive-Patch, which should also be discussed when attacking the propsoed FST.\n\n[1] Huang, Kunzhe, et al. \"Backdoor Defense via Decoupling the Training Process.\" International Conference on Learning Representations. 2021.\n\n[2] Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assump405\ntion of latent separability for backdoor defenses. International Conference on Learning Representations. 2023.\n\nQUESTIONS:\nListed in the weakness of the paper. Score can be improved if concerns listed above are resolved.\n\nLIMITATIONS:\nThe authors addressed the limitation that they assume the defender would hold a clean tuning set which may not be feasible in certain scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}