Towards Stable Backdoor Purification through Feature Shift Tuning
Authors: Rui Min
Zeyu Qin
Li Shen
Minhao Cheng
Venue: NeurIPS 2023 poster (2023)
Keywords: Backdoor Defense
Model-tuning

Abstract: It has been widely observed that deep neural networks (DNN) are vulnerable to backdoor attacks where attackers could manipulate the model behavior maliciously by tampering with a small set of training samples. Although a line of defense methods is proposed to mitigate this threat, they either require complicated modifications to the training process or heavily rely on the specific model architecture, which makes them hard to deploy into real-world applications. Therefore, in this paper, we instead start with fine-tuning, one of the most common and easy-to-deploy backdoor defenses, through comprehensive evaluations against diverse attack scenarios. Observations made through initial experiments show that in contrast to the promising defensive results on high poisoning rates, vanilla tuning methods completely fail at low poisoning rate scenarios. Our analysis shows that with the low poisoning rate, the entanglement between backdoor and clean features undermines the effect of tuning-based defenses. Therefore, it is necessary to disentangle the backdoor and clean features in order to improve backdoor purification. To address this, we introduce Feature Shift Tuning (FST), a method for tuning-based backdoor purification. Specifically, FST encourages feature shifts by actively deviating the classifier weights from the originally compromised weights. Extensive experiments demonstrate that our FST provides consistently stable performance under different attack settings. Without complex parameter adjustments, FST also achieves much lower tuning costs, only $10$ epochs. Our codes are available at https://github.com/AISafety-HKUST/stable_backdoor_purification.

Towards Stable Backdoor Purification through Feature Shift Tuning
Rui Min", Zeyu Qin", Li Shen', Minhao Cheng'
Department of Computer Science & Engineering, HKUST
JD Explore Academy
{rminaa, zeyu. seyu..iinJcconnecc. ust.
mathshenl @gmaall..mmmmmmmmmmmmmmm com minhaocheng@ust.cc hk
Abstract
It has been widely observed that deep neural networks (DNN) are vulnerable to backdoor attacks where attackers could manipulate the model behavior maliciously by tampering with small set of training samples. Although line of defense meth- ods is proposed mitigate this threat, they either require complicated modifications to the training process or heavily rely on the specific model architecture, which makes them hard to deploy into real-world applications. Therefore, this paper, we instead start with fine-tuning, one of the most common and easy-to-deploy backdoor defenses, through comprehensive evaluations against diverse attack sce- narios. Observations made through initial experiments show that in contrast to the promising defensive results on high poisoning rates, vanilla tuning methods com- pletely fail at low poisoning rate scenarios. Our analysis shows that with the low poisoning rate, the entanglement between backdoor and clean features undermines the effect of tuning-based defenses. Therefore, it is necessary to disentangle the backdoor and clean features in order to improve backdoor purification. To address this, we introduce Feature Shift Tuning (FST), method for tuning- bbaed backdoor purification. Specifically, FST encourages feature shifts by actively deviating the classifier weights from the originally compromised weights. Extensive experiments demonstrate that our FST provides consistently stable performance under differ- ent attack settings. Without complex parameter adjustments, FST also achieves much lower tuning costs, only 10 epochs. Our codes are available at https: //github com/ AlSafety- HKUST/KKUUUSSAA Hwt.HHHtttttttttS wwwpbbleeaaccooooo_ooeeee backdoor_puriiiccttioo cation.
Introduction
Deep Neural Networks (DNNs) are shown vulnerable to various security threats. One of the main se- curity issues is backdoor attack [6, 10] that inserts malicious backdoors into DNNs by manipulating the training data or controlling the training process.
To alleviate backdoor threats, many defense methods [36] have been proposed, such as robust training [13, 17, 38] and post-procsssing purification methods [20, 37, 41]. However, robust training methods require complex modifications to model training process [13, 38], resulting in substantially increased training costs, particularly for large models. Pruning-based purification methods are sensitive to hyperparameters and model architecture [37, 41], which makes them hard to deploy in real-world applications.
"Equal contribution. Email zeyu. www.:uiinnooomtttuue.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fine-tuning (FT) methods have been adopted to improve models' robustness against backdoor attacks [14, 20, 36] since they can be easily combined with existing training methods and various model architectures. Additionally, FT methods require less computational resources, making them one of the most popular transfer learning paradigms for large pretrained models [5, 21, 25, 28, 35, 40]. However, FT methods have not been sufficiently evaluated under various attack settings, particularly in the more practical low poisoning rate regime [2, 4]. Therefore, we begin by conducting a thorough assessment of widely-used tuning methods, vanilla FT and simple Linear Probing (LP), under different attack configurations.
In this work, we focus on data-pos poisonnos backdoor attacks, as they efficiently exploit security risks in more practical settings [3, 4, 9, 27]. We start our study on whether these commonly used FT methods can efficiently and consistently purify backdoor triggers in diverse scenarios. We observe that vanilla FT and LP can not achieve stable robustness against backdoor attacks while maintaining clean accuracy. What's worse, although they show promising results under high poisoning rates (20%, 10%), they completely fail under low poisoning rates (5%, 1%, 0.5%). As shown in Figure 3, we investigate this failure mode and find that model's learned features (representations before linear classifier) have a significant difference under different poisoning rates, especially in terms of separability between clean features and backdoor features. For low poisoning rate scenarios, clean and backdoor features are tangled together so that the simple LP is not sufficient for breaking mapping between input triggers and targeted label without feature shifts. Inspired by these findings, we first try two simple strategies (shown in Figure 1), FE FE-tuning and FT-inir based on LP and FT, respectively, to encourage shifts learned features. In contrast to LP, FE-tuning only tunes the feature extractor with the frozen and re-initialized linear classifier. FT-init first randomly initializes the linear classifier and then conducts end-to-end tuning. Experimental results show that these two methods, especially FE-tuning, improve backdoor robustness for low poisoning rates, which confirms the importance of promoting shifts in learned features.
Though these two initial methods can boost backdoor robustness, they still face an unsatisfactory trade-off between defense performance and clean accuracy. With analysis of the mechanism behind those two simple strategies, we further proposed a stronger defense method, Feature Shift Tuning (FST) (shown in Figure 1). Based on the original classification loss, FST contains an extra penalty, (w, wor)) ) the inner product between the tuned classifier weight w and the original backdoored classifier weight wor During the end-to-end tuning process, FST can actively shift backdoor features by encouraging the difference between w and w (shown in Figure 3). Extensive experiments have demonstrated that FST achieves better and more stable defense performance across various attack settings with maintaining clean accuracy compared with existing defense methods. Our method also significantly improves efficiency with fewer tuning costs compared with other tuning methods, which makes a more convenient option for practical applications. To summarize, our contributions are:
We conduct extensive evaluations on various tuning strategies and find that while vanilla Fine-tuning (FT) and simple Linear Probing (LP) exhibit promising results in high poisoning rate scenarios, they fail completely in low poisoning rate scenarios.
We investigate the failure mode and discover that the reason behind this lies in varying levels of entanglement between clean and backdoor features across different poisoning rates. We further propose two initial methods verify our analysis.
Based on our initial experiments. we propose Feature Shift Tuning (FST). FST aims to enhance backdoor purification by encouraging feature shifts that increase the separability between clean and backdoor features. This is achieved by actively deviating the tuned classifier weight from its originally compromised weight.
Extensive experiments show that FST outperforms existing backdoor defense and other tuning methods. This demonstrates its superior and more stable defense performance against various poisoning- based backdoor attacks while maintaining accuracy and efficiency.
2 Background and related work
Backdoor Attacks. Backdoor attacks aim to mislead the backdoored model to exhibit abnormal behavior on samples stamped with the backdoor trigger but behave normally on all benign samples. They can be classified into categories [36]: (1) data-poisoning attacks: the attacker inserts a backdoor trigger into the model by manipulating the training sample (æt, y) E (X, like adding
2

---

Vanilla Lp
Vanilla FT
FE-tuning (Ours)
FT-init (Ours)
FST (Ours)
Deviation
(w)
Reintisileed
Re-nibaaardd
Original
Freeen
$O..
Fecture Stit
(0)
Fronen
Tuned
Tuned
Turned
Tuned
Figure 1:
The first two methods. LP and vanilla FT, are adopted in Section 3.1. The middle two methods, FE-tuning and FT-init, are proposed in Section 3.2. The final one, FST, is our method introduced in Section 4.
small patch in clean image æ and assign the corresponding class y to attacker-designated target label yr- [3, 6, 9, 10, 18, 33]; (2) training-control attacks: the attacker can control both the training process and training data simultaneously [23,24]. With fewer assumptions about attackers' capability, poisoonng attacks are much more practical in real-world scenarios [4, 9, 29] and have led to increasingly serious security risks [3, 27]. Therefore, in this work, we mainly focus Sata-poisoning backdoor attacks.
Backdoor Defense. Existing backdoor defense strategies could be roughly categorized into robust training [13, 38] and post-processing purification methods [20, 37, 41J. Robust training aims to prevent learning backdoored triggers during the training phase. However, their methods suffer from accuracy degradation and significantly increase the model training costs [17, 38], which is impractical for large-scale model training. post-processing purification instead aims to remove the potential backdoor features in a well-trained model. The defender first identifies the compromised neurons and then prunes unlearns them [20, 34, 37, 41, 42]. However, pruning- pruning-based and unlearning methods also sacrifice clean accuracy and lack generalization across different model architectures.
Preliminaries of backdoor fine-tuning methods. Without crafting sophisticated defense strategies, recent studies [14, 20, 27, 42] propose defense strategies based on simple Fine-tuning. Here, we introduce two widely used fine-tuning paradigms namely vanilla Fine-tuning (FT) and Linear Probing (LP) (shown in Figure 1) since they would serve as two strong baselines in our following sections. For each tuned model, we denote the feature extractor di (6(0: x) X - Φ(xz) and linear classifier as f(w; æ) = w" Φ(x) +(x) + V. To implement the fine-tuning, both tuning strategies need a set of training samples denoted as Dr C (X) to update the model parameters while focusing on different parameter space. Following previous works [5, 14, 16, 20, 21], we implement vanilla FT by updating the whole parameters {0, w): regarding the LP, only tunes the linear classifier f(w) without modification on the frozen ((0).
Evaluation Metrics. We take two evaluation metrics, including Clean Accuracy (C-Acc) (i.e., the prediction accuracy of clean samples) and Attack Success Rate (ASR) (i.e., the prediction accuracy of poisoned samples to the target class). A lower ASR indicates better defense performance.
Revisiting Backdoor Robustness of Fine-tuning Methods
In this section, we evaluate the aforementioned two widely used fine-tuning paradigms' defensive performance against backdoor attacks with various poisoning rates (FT and LP). Despite that the vanilla FT has been adopted in previous defense work [14, 20, 36, 42], it has not been sufficiently evaluated under various attack settings, particularly more practical low poisoning rate scenarios. The simple IP method is widely adopted in transfer learning works [5, 16, 21, 22, 28] but still rarely tested for improving model robustness against backdoor attacks. For FT, we try various learning rates during tuning: 0.01, 0.02, and 0.03. For LP, following [16], we try larger learning rates: 0.3, 0.5, and 0.7. We also demonstrate these two methods in Figure
We conduct evaluations on widely used CIFAR- 10 [15] dataset with ResNet- 18 [11] and test 4 representative data-pot poisonini attacks including BadNet [10], Blended [6]. SSBA [18] and Label- Consistent attack (LC) [33]. We include various poisoning rates, 20%, 10%,5%. 1%, and 0.5% for attacks except for LC since the maximum poisoning rate for LC on CIFAR-10 is 10%. Following previous work [36], we set the target label Yt. as class O. Additional results on other models and datasets are shown in Appendix C.I. We aim to figure out whether these commonly used FT methods can efficiently and consistently purify backdoor triggers in various attack settings.
3

---

100
Bodket
Blended
S5BA
P210-211
PT 013
P1 011
PT D20
17.0.21
12.002
111002
PT
7112200
11P 92220
1P 00.33
.... 19 (5
1P/9.7)
(P (P
0.5%
1%
5%
10%
20%
0.55
1%
554
10%
20%
0.5%
18
3%
10%
20%
0.5%
15
5%
10%
ASR
PT 12.03
FT (-:01
(0.03)
(2.0)
(1:2-21
52)
13.00
18 00.55
1 00
LP 12.7)
10.70
0.3%
256
35%
30%
20%
5.
15%
5%
20% 0.5%
175
5%
10%
20%
0.5%
1%
5%
10%
Paisoring
Figure 2: The clean accuracy and ASR of 4 attacks on CIFAR-IO and ResNet-18. The æ--xii means the poisoning rates. The blue and purple lines represent FT and LP, respectively. For FT, we try various learning rates during tuning: 0.01, 0.02, and 0.03. For LP, we try learning rates: 0.3, 0.5, and 0.7.
palon
Figure 3: We take T-SNE visualizations on features from feature extractors with Blended attack. We adopt half of the CIFAR- 10 test set and ResNet- 18. Each color denotes each class, and Black points represent backdoored samples. The targeted class 0 (Red). The first two figures are feature visualizations of original backdoored models with 10% and 1% poisoning rates; (2) The rest figures are feature visualizations after using different tuning methods for 1% poisoning rate. We also add the corresponding ASR.
3.1 Revisiting Fine-tuning under Various Poisoning Rates
We stress that the defense method should effectively purify the backdoor triggers while maintaining good clean accuracy. Therefore, mainly focus defense performance with satisfactory clean accuracy level (around 92%). The results are shown in Figure
Vanilla FT and LP can purify backdoored models for high poisoning rates. From Figure 2, We observe that for high poisoning rates (z 10% for BadNet, Blended, and SSBA; 5% for LC), both vanilla FT and LP effectively mitigate backdoor attacks. Specifically. LP (purple lines) significantly reduces the ASR below 5% on all attacks without significant accuracy degradation (s 2.5%). Compared with vanilla FT, by simply tuning the linear classifier, LP can achieve better robustness and clean accuracy.
Vanilla FT and LP both fail to purify backdoor triggers for low poisoning rates. In contrast to their promising performance under high poisoning rates, both tuning methods fail to defend against backdoor attacks with low poisoning rates (S 5% for BadNet, Blended, and SSBA; < 5% for LC). Vanilla FT with larger learning rates performs slightly better on Blended attacks, but it also sacrifices clean accuracy, leading to an intolerant clean accuracy drop. The only exception is related to the SSBA results, where the ASR after tuning at 0.5% poisoning rate is low. This can be attributed to the fact that the original backdoored models have relatively low ASR.
3.2 Exploration of Unstable Defense Performance of Fine-tuning Methods
From the results, a question then comes out: What leads to differences in defense performance of tuning methods under various poisoning rate settings? We believe the potential reason for this phenomenon is that the learned features from feature extractors of backdoored models differ at different poisoning rates, especially terms of the sepurability between clean features and buckdoor features. We conduct T-SNE visualizations on learned features from backdoored models with Blended attack (10% and 1 % poisoning rates). The results are shown Figure 3. The targeted class samples marked with Red color and Black points are backdoored samples. As shown in first two figures,
cleanfeatures

---

Table Purification performance of fine-tuning against four types of backdoor attacks with low poisoning rates. All the metrics are measured in percentage (%)
Poisoning rate
defense
FE-mning
Vanilla
FT-init
Atlack
C-Aoe(t)
ASR())
C-Ace(t)
ASR(J)
C-Acctt)
ASR(L)
C-Ace(t)
ASR())
C-Ace(t)
ASR())
1%
94.52
100
94.02
100
91.56
3.18
92.12
99.83
93.37
16.72
BadNet
0.5%
94.79
100
94.37
100
92.37
7.41
92.56
99.07
93.90
79.32
Blended
1%
95.13
98.12
94,51
98.98
92.03
8.50
92.97
82.36
93.88
61.23
0.5%
94.45
92.46
93.72
91.84
6.48
92.95
75.99
93.71
1%
94.83
79.54
94.23
82.39
91.99
5.58
92.59
30.16
93.51
21.04
SSBA
0.5%
94.50
50,20
94.37
51.67
91.31
2.50
92.36
12.69
93.41
6.69
1%
94.33
99.16
94.23
99.98
91.70
64.97
92.65
94.83
93.54
89.86
0.5%
94.89
100
94.78
100
91.65
22.22
92.67
99.96
93.83
96.16
Average
94.68
89.94
90.91
15.11
92.61
74.36
93.64
under high poisoning rate (10%), backdoor features (black points) are clearly separable from clean features (red points) and thus could be easily purified by only tuning the f(w); however for low poisoning rate (1%). clean and backdoor features are tangled together so that the simple LP is not sufficient for breaking mapping between input triggers and targeted label without further feature shifts. Furthermore, as depicted in the third figure of Figure 3, though vanilla FT updates the whole network containing both O and w. it still suffers from providing insufficient feature modification, leading to the failure of backdoor defense.
Can Feature Shift Help Enhance the Purification Performance of Fine-tuning? Based on our analysis, we start with simple strategy in that we could improve backdoor robustness for low poisoning rates by encouraging shifts in learned features. We then propose separate solutions for both LP and vanilla FT to evaluate the effect of feature shift as well as the robustness against two low poisoning rates, 1% and 0.5%. As shown in Table 1, specifically, for the LP, we freeze f(w) and only tune ((0). However, our initial experiments show that directly tuning (00) does not sufficiently modify learned features since the fixed backdoor linear classifier (denoted as f (w)"()) may still restrict modifications of learned features. Inspired by the previous work [27], we first randomly re-initialize the linear classifier and then tune only <0()) (denoted as FE-tuning). For vanilla FT, we also fine-tune the whole network with randomly re-initialized linear classifier (denoted as FT-init). More implementation details of FE-tuning and FT-init are shown in Appendix B.3.
Evaluations Verify That Encouraging Feature Shift Could Help Purify Backdoored Models. We observe that both FE-tuning and FT-init could significantly enhance the performance of backdoor purification compared to previous fine-tuning with an average drop of 77.70% and 35.61 55.61% on ASR respectively. Specifically, FE-tuning leads to a much larger improvement, an ASR decrease of over 74%. As shown in the fifth figure of Figure 3, after FE-tuning, backdoor featurree could be clearly separated from from the clean features of the target class (red). However, this simple strategy also leads to decrease in clean accuracy (around 2.9%) in comparison to the original LP, due to the totally randomized classifier. While for the FT-init, the robustness improvements are less significant. As shown in the fourth figure of Figure 3, simply fine-tuning the backdoor model with re-initialized f (w) can not lead to enough shifts backdoor features under the low poisoning rate. The backdoor and clean features of the target class still remain tangled, similar to the original and vanilla FT.
In summary, the results of these two initial methods confirm the fact that encouraging shifts on learned features is an effective method for purifying backdoors low poisoning rates. However, both methods still experience serious clean accuracy drop or fail to achieve satisfactory improvements in robustness. To further enhance the purification performance, we propose a stronger tuning defense in Section 4 that addresses both issues in a unified manner.
4 Feature Shift Tuning: Unified Method to Achieve Stable Improvements
Based on our initial findings, we propose stronger tuning defense paradigm called Feature Shift Tuning (FST). FST is end-to-end tuning method and actively shifts features by encouraging the discrepancy between the uned classifier weight u and the original backdoored classifier weight ors The illustration of FST is shown in Figure 1. Formally, starting with reinitializing the linear classifier
5

---

w, our method aims to solve the following optimization problem:
( ((rr.)) ((0; y)] + a (w, w s.t. |1o|| c}
min 0,w0
(I)
where stands for the original classification cross-entropy loss over whole model parameters to maintain clean accuracy and (.--) denotes the inner product. By adding the regularization on the (w, worl), FST encourages discrepancy between w and worl to guide more shifts on learned features of (()). The a balances these two loss terms, the trade-off between clean accuracy and backdoor robustness from feature shift. While maintaining clean accuracy, larger would bring more feature shifts and better backdoor robustness. We simply choose the inner aener-prouuct (w, won) ) to measure the difference between weights of linear classifiers, since it is easy implement and could provide satisfactory defensive performance in our initial experiments. Other metrics for measuring differences can also be explored in the future.
Projection Constraint. To avoid the w exploding and the inner product domi- nating the loss function during the fine- tuning process, we add an extra con- straint on the norm of w to stabilize the tuming process. To reduce tuning cost, we directly set it as || || instead of manually tuning it. With the constraint to shrink the feasible set, our method quickly converges in just a few epochs while achieving significant robustness improvement. Compared with previ- ous tuning methods, our method further enhances robustness against backdoor attacks and also greatly improves tun- ing efficiency (Shown in Section 5.3). Shrinking the range of feasible set also brings extra benefits. significantly re- duces the requirement for clean sam- ples during the FST process. As shown in Figure 7 (c,d), FST consistently per- forms well across various tuning data sizes, even when the tuning set only con- tains 50 samples. The ablation study of is also provided in Section 5.3 and showcases that our method is not sensitive to the selection of in a wide range. The overall optimization procedure is summarized in Algorithm
Algorithm 1 Feature Shift Tuning (FST)
Input: Tuning dataset Dy {æ ye ) backdoored model f(w ((0)); learning rate ry; factor a; tuning iterations I
Output: Purified model
1: Initialize wn with random weights
2: Initialize 0 with 0
3: for i = 0, I..... do
4: Sample mini-batch B, from tuning set Df
5: Calculate gradients of 0' and w':
96 Vo [8 ExsB, L(f(w"; x(0'; x)), y):
g = V
15 EeEB, C(f(w'; 600'; x)) y)
a (w", wor;)]
6:
Update model parameters Øi+1
0
n99,, w w² "1''.
7: Norm projection of the linear classifier wi+1
w 1w 1|
8: end for
9: return Purified model f(w :((1))
Unified Improvement for Previous Tuning Methods. We discuss the connections between FST and our previous FE-tuning and FT-init to interpret why FST could achieve unified im- provements backdoor robustness and clean accuracy. Our objective function Eq.1 could be decomposed into three parts, by minimizing
C(f (00; x)), )).)) y) over w and respec- tively, and a (w, woi) over w. Compared with FE-tuning, FST brings an extra loss term on min B(m.y)~Dy, [(f(w: (00; xx)).)) y) + a (w, worl to update linear classifier w. In other words, while encouraging feature shifts, FST updates the linear classifier with the original loss term to improve the models' clean performance. Compared with the FT-init, by adopting a (w, w FST encourages discrepancy between tuned w and original ori to guide more shifts on learned features.
Weights of Linear Classitier Could Be Good Proxy of Learned Features. Here, we discuss why we choose the discrepancy between linear classifiers as our regularization in FST. Recalling that our goal is to introduce more shifts in learned features, especially for backdoor features. Therefore, the most direct approach is to explicitly increase the difference between backdoor and clean feature distributions. However, we can not obtain backdoor features without inputting backdoor triggers. We consider that weights of the original compromised lineur classifier could be good proxy of learned features. Therefore, we encourage discrepancy between tuned up and original w ori rather than trying to explicitly promote discrepancy between feature distributions.

---

5 Experiments
5.1 Experimental Settings
Datasets and Models. We conduct experiments on four widely used image classification datasets,
CIFAR-10 [15], GTSRB [32], Tiny-Imagen Inn-ImmaeNeet [8], and CIFAR-100 [15]. Following previous works [13, 20, 24, 36]. we implement backdoor attacks on ResNet- 18 [11] for both CIFAR-IO and GTSRB and also explore other architectures in Section 5.3. For CIFAR-I00 and Tiny-ImageNet, we adopt pre-traing pre-trained SwinTraasffrrer [21] (Swin). For both the CIFAR-IO and GTSRB, we follow the previous work [41] and leave 2% of original training data the tuning dataset. For the CIFAR- 100 and Iiny-ImageNet, we note that a small tuning dataset would hurt the model performance and therefore we increase the tuning dataset 5% of the training set.
Attack Settings. All backdoor attacks are implemented with BackdoorBench 1 We conduct evalua- tions against 6 representative data-poisoning backdoors, including 4 dirty-label attacks (BadNet [10], Blended attack [6]. WaNet [24], SSBA[18]) and 2 clean-label attacks (SIG attack [1], and Label- consistent attack (LC) [33]). For all the tasks, we set the target label yt to O, and focus on low poisoning rates, 5%. 1%, and 0.5% in the main experimental section. We also contain experiments of high poisoning rates, 10%. 20%, and 30%, in Appendix C.2. For the GTSRB dataset, we do not include LC since it can not insert backdoors into models (ASR < 10%). Out of all the attacks attempted Swin and Imaaeee,,,,,, only BadNet, Blended, and SSBA were able to successfully insert backdoor triggers into models at low poisoning rates. Other attacks resulted in an ASR of less than 20%. Therefore, we only show the evaluations of these three attacks. More details about attack settings and trigger demonstrations are shown in Appendix B.2.
Baseline Defense Settings. We compare our FST with 4 tuning- tuning-based defenses including Fine- tuning with Swarpness-Aware Minimization (FT+SAM), a simplified version adopted from [42], Natural Gradient Fine-tuning (NGF) [14], FE-tuning and FT-init proposed in Section 3 3.2. We also take a comparison with extra strategies including Adversarial Neural Pruning (A (ANP) [37] and Implicit Backdoor Adversarial Unlearning (I-BAU) [38] which achieve outstanding performance in BackdoorBench [37]. The implementation details of baseline methods are shown in the Appendix B.3. For our FST, we adopt SGD with an initial learning rate of 0.01 and set the momentum 0.9 for both CIFAR-10 and GTSRB datasets and decrease the learning rate to 0.001 for both CIFAR-100 and Image-n datasets prevent the large degradation of the original performance. We fine-tune the models for 10 epochs on the CIFAR-10 10; 15 epochs on the GTSRB, CIFAR- and Imae-em We set the a as 0.2 for CIFAR-I0; 0.1 for GTSRB; 0.001 for both the CIFAR-100 and ImageNe.
5.2 Defense Performance against Backdoor Attacks
In this section, we show the performance comparison between FST with tuning- based backdoor defenses (FT+SAM [42], NGF [14], FE-tuning and FT-init) and current stateoof-the-ut defense methods, ANP [37] and I-BAU [38]. We demonstrate results of CIFAR-10, GTSRB, and Tiny- ImageNet Table 2, 3, and 4, repectively. We leave results on CIFAR- 100 Appendix C.3.
Experimental results show that our proposed FST achieves superior backdoor purification perfor- mance compared with existing defense methods. Apart from Tiny-ImageNet, FST achieves the best performances on CIFAR-10 and GTSRB. The average ASR across all attacks on three datasets are below 1%. 0.52% in 10, 0.41% in GTSRB, and 0. 0.19% in Tiny-ImageNet, respectively. Regarding two tuning defense methods, FT+SAM and NGF, FST significantly improves backdoor robustness with larger ASR average drops three datasets by 34.04% and 26.91%. Compared with thhthe-rtt methods, ANP and I-BAU, on CIFAR- 10 and GTSRB, our method outperforms with large margin by 11.34% and 32.16% in BatchNorm layers of ConvNets in source codes. Therefore, it can not be directly conducted in SwinTransfonm Transformer. Worth noting is that our method achieves the most stable defense performance across various attack settings with 0.68% on the average standard deviation for ASR. At the same time, our method still maintains high clean accuracy with 93.07% on CIF 96.17% GTSRB, and 79.67% on Tiny-ImageNet on average. Compared to ANP and I-BAU, FST not only enhances backdoor robustness but also improves clean accuracy with significant boost on the clean accuracy by 2.5% and 1.78%, respectively.
average ASR, respectively. ANP is only conducted
chttps://gite htttps///gttub..cco////////aa/ea http//CCCBD/baccccoorbeeenet
7

---

Table 2: Defense results under various poisoning rales. The experiments are conducted on CIFAR-IO dataset
with 18.. All the metrics are measured in percentage (%) The best results are bold.
Mloc
Piosesing
Sireace
ANP
IT-SAM
NGF
swing (Dwn
PST (Dirn
C-hostt
ASRLL
ASRGL
(CofetT))
ASM211
(C-bed1)
ASR121
ASKGL1
ASRL21
(Cffett))
ASKIL
BaN
17
96.33 57
110)
10.56
0.76
67.08
3.63
01.00
13.33
03:31
5.50
01:50
5.13
03.37
16.23
00.51
$1.51
12.12
10.16
03.9
Blarded
7%
04.71
18/76
03.00
3.51
18.0
63.01
13.34
43.00
21.42
21.24
11
18.35
13.25
00036
10.07
3077
0.5%
65.35
60.06
68.95
7.0
01
15.75
91.00
34.35
41:15
10.55
01.54
6.50
08.73
18.30
95.15
Waldd
136
2013
00100
0/6
01.33
0.15
91.11
1.00
91.85
1.50
10.00
1.20
93
92.81
2.60
9123
R126
558A
76
96.54
00:30
01.64
117
90.15
0.16
92.64
2.90
01.25
3.53
01:00
1130
15
03.97
91.8
1L27
0.54
64.90
50.30
50:10
0057
90.79
1.42
10.15
1366
91.31
2.28
91.31
2.50
93.43
3.50
10.7
IUM
800
46:41
60-26
58:11
01.36
1300
1.00
00.33
30.90
18:13
0716
13:30
55.55
3.33
01:79
01.90
1.25
41.03
41.93
42.96
06.26
91.gg
90.00
RM12
91.22
17:00
6.77
52.07
41:22
91.66
81.35
k799
43.38
10:33
6167
58:30
91.51
0.3%
94.97
100
12.83
98.17
91.40
6134
12.30
$4.37
91.27
$1.56
51.60
22.66
93.33
96.16
99.44
1.71
0.34
1340
10.09
91.85
93.34
58.50
10.34
94.83
48.07
94.38
1.37
Aurage
15.0
144
$1.9
251
2300
116
145.5
0.1
32.99
1635
0.92
15.44
0,20
3645
0.19
0.76
Table 3: Defense results under various poisoning rates. The experiments are conducted on the GTSRB dataset with R8.Net-18. All the metries are measured in percentage (%). The best results are bold.
Alack
AMP
ERAII
PT-8AM
RAT
Cofenil
AS50.2
ASHIL1
CCAsstly
[C-ARX1)
ASHIL
ASMILI
Cofestly
ASRILI
BodKor
29c
19.35
1300
166.7
08.0
13:38
5U81
97.37
0.80
$7.30
14.77
99.13
2.4L
13.50
33.04
54.05
0.00
0.95
15
100
98,
0.00
$1.51
0.16
95.75
0.333
$5.50
01/0
08.23
0.54
98.31
131
19h99
0.00
Randdd
190
$3.60
07.12
01.12
4000
$7.43
73.43
17:04
18.35
18.07
01:45
97:83
18.60
18.56
$3.35
97:45
3.34
6.55
18
55.20
00.93
97.35
18:1
53.58
6.00
92.41
0.30
$3.00
1,20
54.17
0.55
$1.40
3136
53.00
0.02
0.95
67.06
34.95
07.54
0.53
$1.00
140
32.67
0.34
01.87
0.17
$1.54
0.3
01:44
2.15
40.17
0.68
SSRA
18
16
55.13
92.23
05.00
18.0
$2.80
1
18.87
62.00
2.0
18.16
14:05
97.82
11.00
57.5%
$2.10
06.33
0.13
0,55
63-26
03:97
91:17
53.01
63.51
2.50
00:73
5.35
63.5%
05.24
00.93
36.15
18.28
41:14
95.36
0.18%
99.00
40.25
06.01
40:25
67.56
3330
V7.95
3.7
06.43
5.71
98.44
78.75
19.34
63.37
96.00
1.84
0.955
65.58
03.43
35:40
$1.70
60.79
11.66
68.55
09.11
97.63
Avergg
$2.13
15.21
$3.50
00.77
96.13
13.08
31.47
95.19
43.81
91.29
67.64
96.17
Standasd Deviatio
1.05
21.37
3.56
2006
2.14
23.05
3.8
38.77
0.72
21.85
3.50
76.72
285
30.47
1.31
0.33
Following the experimental setting in Section 3.2, also conduct feature visualization of FST in Figure 3. It can be clearly observed that our FST significantly shifts backdoor features and makes them easily separable from clean features of the target class like FE-tuning. Combined with our outstanding performance, this further verifies that actively shifting learned backdoor features can effectively mitigate backdoor attacks. We also evaluate two initial methods, FE-tuning and FT-init. FE-tuning performs best on Tiny-ImageNer, leading to a little ASR drop by 0.18% compared with FST. However, it also significantly sacrifices clean accuracy, leading a large C-ACC drop by 8.1% compared with FST. FST outperforms them by large margin for backdoor robustness on CIFAR-10 and GTSRB. As we expected. FST achieves stable improvements in robustness and clean accuracy compared with FE-tuning and FT-init.
Adaptive Attacks. To further test the limit of FST, we conduct an adaptive attack evaluation for it. To bypass defense methods based on latent separation property, adaptive poisoning attacks [26] actively sup- press the latent separation between backdoor and clean features by adopting an extremely low poisoning rate and adding regularization samples. Their evaluations also show that these attacks successfully bypass ex- isting strong latent separation- based defenses. Hence, we believe it is also equally a powerful adaptive attack against our FST method. We evaluate our method against adaptive attacks. The results of the Adaptive- Blend attack are shown in Table 2, 3. The results of more adaptive attacks and various regularization sam- ples are shown in Appendix C.4. We could observe that FST could still effectively purify such an adaptive attack achieving an average drop in ASR by 81. .66% and 91. .55% respectively for the CIFAR-10 and GTSRB.
Adaptive Blend ASR: 83.03%
FST
ASR: 1.37%
Figure 4: The T-SNE visualizations of Adaptive- Blend attack (150 payload and 150 regularization samples). Each color denotes each class, and Black points represent backdoored samples. The targeted class is 0 (Red). The left figure repre- sents the original backdoored model and the right represents the model purified with FST.
8

---

Table 4: Defense results under various poisoning rate settings. The experiments are conducted on the Tiny- ImageNet dataset. All the metrics are measured in percentage (%).
The best results are bold.
Atrck
Potsnting rate
No deferce
LBAUU
FT:SAM
NGP
FE tinn (Oens)
Fr-inr
D-Akxet)
ASRR11
C-Aoo(1)
ASR(L)
(-Arett)
(--Aext1)
ASR(11
ASR(1)
C-Aectt)
ASRLL))
(CAkeet)
ASR[]]
BxdNet
15g
65:13
100
100
61:51
76.33
52.34
99.38
$1.03
73.63
18.20
34.57
55.30
71.09
0..00
$3.06
$0.30
15:21
1.34
79.84
79.33
0.538
N3-42
99.97
31.85
354.19
$53.0
00.50
79.04
27.02
72.85
0.00
$3.70
010
79.82
0.00
5%
09.85
76.82
36.74
$1.88
91.37
78.85
66.99
71.5
0.00
$3.55
010
10.25
0.00
92.12
05.49
08.85
SSBA
5%
1%
54.27
99.27
09:05
R2.00
82.16
66.25
n8.51
82.14
78.55
21.51
0
77.83
78, 96
21.41
15.24
70.37
71,45
0.01
0.55
79.50
$0.35
0.0
0.50
19.33
78.04
0.19
0.08
0.5%
85.00
76.55
82.35
45.50
92.43
4.25
79.77
3.59
71.13
0
90.35
0.04
79.77
0.94
Averag
Stanshed Deviatio
0.39
7.83
2.47
13.66
1.36
30.36
0.37
31.05
0.55
0.02
0.58
5.01
0.35
0.45
Badkitt
30
BadNiet
Blended
Blended
70-
CARCE
60
FST W SPN)
ASH
FST W/ SSN
FST Al
PST Al 15%6
PST AT 1295
Al
RST 2.99
(761
(544)
(996
DST wit 5.21%)
FRT 23351
FST With (.215)
FOT AlN 13.55)
PGT Are 6.14
30
PNT Ale C.114)
EST Jaln
Tuning Epochs
Tuning Spochs
Tining Epochs
Turing Epochs
Figure 5: The experimental results with and without projection constraint (w/ C and w/o C, respectively). We demonstrate two types of backdoor attacks, namely the BadNet and Blended, with three different poisoning rates (5%, 1%, and 0.5%). The experiments are conducted with varying tuning epochs.
To explain why adaptive attacks fail, we provide TSNE visualizations of learned features from backdoored models. We show the results in Figure 4. We can first observe that adaptive attacks significantly reduce latent separability. Clean and backdoor features are tightly tangled. FST effectively shifts backdoor features and makes them easily separable from the clean features of the target class. Therefore, the feature extractor will no longer confuse backdoor features with clean features of the target class in the feature space. This leads to the subsequent simple linear classifier being difficult to be misled by backdoor samples, resulting in more robust classification.
5.3 Ablation Studies of FST
Below, we perform ablation studies on our proposed FST to analyze its efficiency and sensitivity to hyperparameters,, tuning sizes, and architectures. This further demonstrates the effectiveness of FST in practical scenarios.
Efficiency analysis. we evaluate the backdoor defense performance of our method and 4 tuning strategies (FT+SAM, NGF, FE-tunnng, and FT-init) across different tuning epochs. We test against types of backdoor attacks including (BadNet, Blended, SSBA, and LC) with 1% poisoning rate and take experiments on CIFAR- 10 and ResNet-18 models (see Figure 6). Notably, FST can effectively purify the backdoor models with much fewer epochs, reducing backdoor ASR below 5%. This demonstrates that our method also significantly improves tuning efficiency. We also find that added projection constraint also helps FST converge. We offer the performance of FST's entire tuning process on CIFAR-10 10 and ResNet-18, with or without the projection term. The results of the BadNet and Blended are shown in Figure 5. We could clearly observe that the projection stabilizes the tuning process of FST. The projection term helps FST quickly converge, particularly in terms of accuracy. We leave the analysis with more attacks in Appendix C.5.
Sensitivity analysis on a. We evaluate the defense performance of FST with various a in Eq 1. We conduct experiments on CIFAR- 10 and R and test against BadNet and Blended attacks with 5% and 1%. The results are shown in (a) and (b) of Figure 7. We can observe that as the a increases from 0 to 0.1, the backdoor ASR results rapidly drop below 5%. As we further increase the a, our method maintains a stable defense performance (ASR < 4%) with only a slight accuracy degradation (< 1.5%). It indicates that FST is not sensitive to a. The results further verify that FST could achieve a better trade-off between backdoor robustness and clean accuracy.
Sensitivity analysis on tuning dataset size. Here, we evaluate the FST under rigorous scenario with limited access to clean tuning samples. We test our method on the CIFAR-10 using tuning datasets of varying sizes, ranging from 0.1% to 2%. Figure 7 shows that FST consistently performs

---

BadNNe
Blanded
S3BA
NGF (Adi
HCT (45R)
44
WSS (ASR:
FEoueen 201
PEcurnn (ASR)
FeAu (620)
FEAARR 2011
FST (420)
FST (A5R)
20
20
10
15
20
10
15
20
Tuning Epochs
Tuning Epochs
Tunig Epochs
Tuning Esochs
Figure 6: The ASR results of four backdoor attacks with varying tuning epochs of tuning methods.
100
10c
nother Bndier-l 1659)
Reetet 3%, (009)
seatee (o09)
Bandood JASR)
64
Slinded (aG
Racthed sis (ACC)
Baciher- 246 JACCS
Boditen- (ACC)
Mlanste die JACC)
Bluncce- the (60C)
0 0.51 0.05
011
Diz
0.4
2.6
0.51
0.05
0.1
0.4
0.4
0.155
0.2%
Turing
0.5%
1%
2%
0.5%
0.5%
Siza
[6]
(c)
Figure 7: (a) and (b) show C-ACC and ASR of FST with various a. (c) and (d) show the C-ACC and ASR of
various sizes of tuning datasets. Experiments are conducted CIFAR- 10 dataset with ResNet-18.
well across various tuning data sizes. Even if the tuning dataset is reduced to only 0.1% of the training dataset (50 samples in CIFAR-10), our FST can still achieve an overall ASR of less than 2%.
Analysis on model architecture. We extend the experiments on other model architectures including VGG19-BN [31], ResNet-50 [1 I, and DenseNet [12] which are widely used in previous stud- ies [14, 37, 42]. Here we show the results of BadNet and Blended attacks with 5%. 1 1%, and 0.5% poisoning rates on the CIFAR-10 dataset. Notably, the structure of VGGI9-BN is slightly different where its classifier contains more than one linear layer. Our initial experiments show that directly applying FST to the last layer fails to purify backdoors. Therefore, we simply extend our methods to the whole classifier and observe significant promotion. We leave the remaining attacks and more implementation details in the Appendix D.2. The results presented in Figure show that our FST significantly enhances backdoor robustness for all three architectures by reducing ASR to less than 5% on average. This suggests that our approach is not influenced by variations in model architecture.
VGG19-BN
ResNac-50
DenseNet-161
Edfa
CAM
Fut Sataoi NC
Ba-5%
Ba-1%,
Ba-0 5%
515%
B1.1%
B-0.5%
Ba-4%
Ba-1%
Bac0
5%
B-5%
05-1%
B5-0.5%
Bae5%
Bac2%
5a-0.%%
8-%%
5-1%
B--035%
Figure 8: The purification performance against different model architectures on the CIFAR-I0 dataset, where Ba- short for BadNet and BI- short for Blended.
6 Conclusion and limitations
In this work, we concentrate on practical Fine-tuning- backdoor defense methods. We conduct a thorough assessment of widely used tuning methods, vanilla FT and LP. The experiments show that they both completely fail to defend against backdoor attacks with low poisoning rates. Our further experiments reveal that under low poisoning rate scenarios, the backdoor and clean features from the compromised target class are highly entangled together, and thus disentangling the learned features is required to improve backdoor robustness. To address this, we propose novel defense approach called Feature Shift Tuning (FST), which actively promotes feature shifts. Through extensive evaluations, we demonstrate the effectiveness and stability of FST across various poisoning rates, surpassing existing strategies. However, our tuning methods assume that the defender would hold a clean tuning set which may be feasible in certain scenarios. Additionally, they also lead to a slight compromise on accuracy in large models though achieving robustness. This requires us to pay attention to protect learned pretraining features from being compromised during robust tuning [7, 16, 19, 39].
10

---

References
[1] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pages 101-105. IEEE, 2019. 5.1
[2] Nicholas Carlini and Andreas Terzis, Poisoning and backdooring contrastive learning. In International Conference on Learning Representations. 2021.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Con- ference on Learning Representations, 2022. URLhttps htl.atppps//ppprrrrreeee.otttroo net/torummidd 1it.uu/miidd-AAAC00OUI010 1, 2
[4] Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo. Daniel Paleka. Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scalc training datasets is practical. arXiv preprint arXiv: 10149, 2023. 1,2
[5] Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-16007. PMIR, 2020. 1, 3
[6] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks deep learning systems using data poisoning. arXiv preprint arXiv:7112.05526. 2017. 3,
[7] Yonggiang Chen, Wei Huang. Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Towards understanding feature learning in out-of-distribution generalization. arXiv preprint arXiv:2.04./1333 2023.
[8] Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet alternative to the cifar datasets. arXiv preprint arXiv: (tt.:1707.08199 2017. 5.1
Micah Goldblum, Dimitris Tsipras. Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning. backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 1,
[10] Tianyu Gu, Kang Liu, Brendan DavittGavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-4773 47244, 2019. 1, 2, 3.1
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
3, 5.1,1,5.
[12] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708 2017. 5.3
[13] Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In International Conference Learning Representations, 2022. URL https: // //opereviee wwweenvviee.nee/forrreee tode///orumiid-ySS. ORdKI. 2.1
[14] Nazmul Karim, Abdullah Al Arafat, Umar Khalid, Zhishan Guo, and Nazanin Rahnavard. In search of smooth minima purifying backdoor in deep neural networks. 2, 3, 5.1, 5.2,
[15] Alex Krizhevsky al. Learning multiple layers of features from tiny images. 2009. 3, 5.1
[16] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning distort pretrained features and underperform of distrriutiitiiiiiionnn. In International Conference Learning Representationss 2022. URL https:
net htre///rmmmmmm http///ormmmnnuuuneeeeeee -ww.eeeeseeeeeeeeeeeeee 2,3.6
[17] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data, Advances Neural Information Processing Systems, 34:14900- 14912, 2021. 1, 2, B.I
[18] Yuezun Li, Yiming Li. Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16463-16477 1647,, 2021. 2, 3.1
[19] Yong Lin, Lu Tan, Yifan Hao, Honam Wong. Hanze Dong, Weizhong Zhang Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distributae generalization. arXiv preprint arXiv:2309.172850 172300 2023, 6
11

---

[20] Kang Liu,, Brendan Davit,, and Siddharth Garg. Fine-pruning: Defending against backdooring atlacks deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 273-294. Springer, 2018 1, 2,5.1
[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEF/CVF international conference on computer vision, pages 10012-10022, 2021. 1, 3,5.1
[22] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning. pages 7721-7735. PMLR, 2021. 3
[23] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems, 33:3454-344 34644, 2020.
[24] Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping-based backdoor attack. In International Conference on Learning Representations, 2021. URL /tctps://openreeeee id=eEn8KTtJOx, 2, 5.1
forum??
[25] Hicu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh- Thang Luong, Mingxing Tan, and Quoe V Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.1000 2021.
[26] Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal, Revisiting the assump- tion of latent separability for backdoor defenses. In The eleventh international conference on learning representations, 2023. 5.2, B.2, C.4
[27] Zeyu Qin, Liuyi Yao, Daoyuan Chen, Yaliang Li, Bolin Ding, and Minhao Cheng, Revisiting personalized federated learning: Robustness against backdoor attacks. Proceedings of 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23. page 4743 4755, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. 1, 3.2
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021. 3
[29] Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back the drawing board: A critical evaluation of poisoning attacks on production federated learning. 2022 IEEE Symposium on Security and Privacy (SP), pages 1354-1371. IEEE, 2022. 2
[30] Reza Shokri et Bypassing backdoor detection algorithms in deep learning. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P)) pages 175-183. IEEE, 2020. C.4
[31] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1409. (556, 2014. 5.3
[32] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32:323 -32, 2012. 5.1
[33] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks, arXiv preprint 19t1v:1112.02771, 2019. 2, 5.1
[34] Bolun Wang, Yuanshun Yao, Shawn Shan. Huiying Li. Bimal Viswanath. Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. JEEE, 2019. 2
[35] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL htLthtts:///pooee /oop//eveeeeeeeeee/eeeeeeeeeeeee
[36] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor lcarning. In Thirty-sixth Conference on Neural Information Processing Systems Dutusets and Benchmarks Track, 2022. 1, 2, 3, 5.1, B.1, B.2
[37] Dongxian Wu and Yisen Wang, Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, -4:16993-16925, 2021. 1, 2, 5.2, 5.3,
12

---

[38] Yi Zeng, Si Chen, Won Park. Zhuoqing Mao. Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference Learning Representations, 2022. URL https:/ wett:ooru//.d-eeeeee/t
1, 2, 5.1, 5.2,
[39] Jianyu Zhang and Léon Bottou. Learning useful representations for shifting tasks and distributions. In International Conference on Machine Learning, pages 40830-40850 PMLR, 2023.
[40] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yt Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10174-10183, 2022.
[41] Runkai Zheng, Rongjun Tang, Jianze
Liu. Data-free backdoor removal based on channel lipschitzness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Isruel, October 23 27, 2022, Proceedings, Part V. pages 191. Springer, 2022. 5.1, B.1
[42] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-awan aware minimization. arXiv preprint arXiv:3304.11823, 2023. 2, 3, 5.1, 5.2, 5.3, B.3
13

---

A
Social Impact
Deep Neural Networks (DNNs) are extensively applied in today's society especially for some safety- critical scenarios like autonomous driving and face verification. However, the data-hungry nature of these algorithms requires operators to collect massive amounts of data from diverse sources, making source tracing difficult and increasing the risk of potential malicious issues. For example, attackers can blend poisoned data into benign samples and embed backdoors into models without training control, posing a significant threat model deployment. Therefore, to mitigate these risks, defenders must remove potential backdoors from models before real-world deployment, ensuring safety and trustworthiness. Our work focuses on a lightweight plug-and-play defense strategy applicable in real scenarios with minimal modifications to existing pipelines. We hope to appeal to the community to prioritize practical defensive strategies that enhance machine learning security.
B Experimental Settings
B.1 Datasets and Models.
Following previous works [17, 36, 37, 38] in backdoor literature, we conduct our experiments on four widely used datasets including CIFAR-10, GTSRB, Tiny-ImageNet, and CIFAR-100.
CIFAR-10 and GTSRB are two widely used datasets in backdoor literature containing images of 32 32 resolution of 10 and 43 categories respectively. Following [37, 41], separate 2% clean samples from the whole training dataset for backdoor defense and leave the rest training images implement backdoor models. For these two datasets, we utilize the ResNet-18 to construct the backdoor models.
CIFAR-100 and Tiny-ImageNet are two datasets with larger scales compared to the CIFAR- 10 and GTSRB which contain images with 64 64 resolution of 100 and 200 categories respectively. For these two datasets, enlarge the split ratio and utilize 5% of the training dataset as backdoor defense since a smaller defense set is likely to hurt the model perfor- mance. For these two datasets, utilize the pre-trained Swin Transffoffre (pre- praintr weights on ImageNet are provided by PyTorch) to implement backdoor attacks since we find that training these datasets ResNet-18 from scratch would yield worse model performance with C-Aco (< 70%) on average and therefore is not practical in real scenarios.
B.2 Attack Configurations
We conducted all the experiments with 4 NVIDIA 3090 GPUs.
We implement 6 representative poisoning- based attacks and an adaptive attack called Adaptive- Blend [26]. For representative attacks, most of them are built with the default configurations in BackdoorBench [36]. For the BadNet, utilize the checkerboard patch backdoor triggers and stamp the pattern at the lower right corner of the image; for the Blended, we adopt the Hello-Kitty pattern triggers and set the blend ratio as 0.2 for both training and inference phase; for WaNet, set the size of the backward warping field as 4 and the strength of the wrapping field as 0.5: for SIG, set the amplitude and frequency of the sinusoidal signal as 40 and respectively; for SSBA and ICC we adopt the pre-generated invisible trigger from BackdoorBench. For the extra adaptive attack, utilize the official implementation codes and set both the poisoning rate and cover rate as 0.003 following the original paper. The visualization of the backdoored images is shown in Figure 9.
For CIFAR-10 and GTSRB, we train all the backdoor models with initial learning rate of 0.1 except for the WaNet since we find a large initial learning rate would make the attack collapse, and therefore we decrease the initial learning rate to 0.01. All the backdoor models trained for 100 epochs and 50 epochs for CIFAR- 10 and GTSRB respectively. For CIFAR- and Eny-emmgeNet, we adopt a smaller learning rate of 0.001 and fine-tune each model for 10 epochs since the SwinTransformer is already pre-trained on ImageNet and upscale the image size up to 224 * 224 before feeding the image to the network.
14

---

Original
BadNet
Blended
WaNet
SSBA
SIG
LC
Figure 9: Examplc images of backdoored samples from CIFAR-10 dataset with attacks.
B.3 Baseline Defense Configurations
We evaluate 4 tuning-basee defenses and 2 extra state-of-the defense strategics including both ANP and 1-BAU for comparison. For tuning- tuning-based defenses, mainly consider 2 recent works including FT+SAM and NGF, and we also compare another baseline tuning strategies including FE-tuning and FT-init proposed in paper. For all defense settings, set the batch size 128 on CIFAR10 and GTSRB and set the batch size as 32 on CIFAR-100 and LnntlImaeNNet due to the memory limit.
FT+SAM: Upon completion of our work, the authors of [42] had not yet made their source code publicly available. Therefore, we implemented a simplified version of their FT-SAM algorithm, where we replaced the optimizer with SAM in the original FT algorithm and called it FT+SAM. For both CIFAR-I0 and GTSRB, we set the initial learning rate as 0.01 and fine-tune models with 100 epochs. We set the as and 10 for CIFAR-10 and GTSRB respectively since find the original settings (p for CIFAR- 10 and p 8 for GTSRB) are not sufficient for backdoor purification in our experiments. For CIFAR-100 and ImageNet, set the initial learning rate 0.001 and p 6, and fine-tune the backdoor model for 20 epochs for fair comparison.
NGF: We adopt the official implementation for NGF. For CIFAR-10 and GTSRB, we set the uning epochs as 100 and the initial learning rate as 0.015 and 0.05 respectively. While for CIFAR- 100 and Tiny-ImageNet, we set the tuning epochs as 20 and the initial learning rate as 0.002.
FE-tuning: For FE-tuning, we first re-initialize and freeze the parameters in the head. We then only fine-tune the remaining feature extractor. For CIFAR-10 1- and GTSRB, set the initial learning rate as 0.01 and fine-tune the backdoor model with 100 epochs; while for CIFAR- and ImageNna we set the initial learning rate as 0,005 and fine-tune the backdoor model with 20 epochs.
FT-init: For FT-init, randomly re-initialize the linear head and fine-tune the whole model architecture. For CIFAR- 10 and GTSRB, we set the initial learning rate as 0.01 and fine-tune tune the backdoor model with 100 epochs; while for CIFAR-100 and initial learning rate as 0.005 and fine-tune the backdoor model with 20 epochs.
set the
ANP: We follow the implementation in BackdoorBench and set the perturbation budget as 0.4 and the trade-off coefficient as 0.2 following the original configuration. We find that within range of thresholds, the model performance and backdoor robustness are related to the selected threshold. Therefore, we set a threshold range (from 0.4 to 0.9) and present the purification results with low ASR and meanwhile maintain the model's performance.
I-BAU: We follow the implementation in BackdoorBench and the initial learning as -e-4 and utilize 5 iterations for fixed-point approximation.
'nttps://github comtuuu..oom//eeeeeee/eeeeeeeeee com/kr- anonymous/ngt animus
15

---

ASR
111-0000
13.00)
DEN
(0.00)
LP16-30
10.33
LF165)
12.50
1P
12.55
10%
20%
0.5%
3% Blended
20%
0.5%
1%
3%
20%
Polsoring
Basffiie
558AA
(1.01)
(3.31
L8 (5-2)
10:35
(0.51
(0.51
5% Paisoring Rate
Po soninn Rase
Potsoning Rate
Polsoning Rate
193
Dedtuet
Berded
553A
(c)
011
F7 001
21)
19:0.91
(P-0..)
0.5%
5%
10%
20%
0.5%
50%
20% 0.5%
5%
30%
20%
Patsoring
Petsoning
Possoning
Figure 10: The Evaluation of vanilla FT and LP: (a) ResNet-50 on CIFAR-10, (b) Dense-161 on CIFAR-10. (c) R8 on GTSRB.
C Additional Experimental results
C.1 Additional Results of Revisiting Fine-tuning
In this section, we provide additional experimental results for Section to explore the potential influence of the dataset and model selection. Specifically, in addition to our initial experiments of revisiting fine-tuning on CIFAR-I0 with ResNet-18, we further vary the model capacity (ResNet-50 on CIFAR-1O), the model architecture (DenseNet-161
CIFAR-10), and the dataset (ResNet- on GTSRB), As mentioned in Section 3.1, we mainly focus on defense performance with satisfactory clean accuracy level (92% on CIFAR- 10, 97% on GTSRB). We tune hyperparameters based on this condition. All the experimental results are shown in Figure 10 respectively. These additional results also demonstrate that Vanilla FT and LP could purify backdoored models for high poisoning rates but fail to defend against low poisoning rates attacks. The only exception is the SSBA results since the original backdoored models have relatively low ASR, as mentioned Section 3.1.
C.2 Additional Results of High Poisoning Rates
Our previous experiments in Section 5.2 have demonstrated our FST's superior defense capacity against backdoor attacks with low poisoning rates. In this section, we further extend our attack scenarios with more poisoning samples by increasing the poisoning rate 10%, 20%, and 30%. We conduct experiments on CIFAR- 10 and GTSRB with ResNet- 18 and present the experimental results in Table 5. We observe that the FST could easily eliminate the embedded backdoor expected while preserving high clean accuracy of the models.
C.3 Additional Results on CIFAR-100 Dataset
We evaluate our FST on the CIFAR-100 dataset with the results shown in Table Since some attacks show less effectiveness under low poisoning rate with ASR < 25%, we hence only report the results where the backdoor attack is successfully implemented (original ASR > 25%). We note that our FST could achieve excellent purification performance across all attack types on the CIFAR-100 dataset with an average ASR 0.36% which is 65.9% and 9.46% lower than the ASR of two other tuning strategies, FT+SAM and NGF respectively. Although the FE-tuning could achieve lower
16

---

Table Defense results under high poisoning rate settings. All the metrics are measured in percentage (%).
Poisoning rate
CIFAR-10 10
GTSRB
Attack
No defense
FST
No delense
FST
C-Ace(t)
ASR(})
C-Ace(t)
ASR(L)
C-Ace(t)
ASR(.)
C-Ace(1)
ASR(.)
BadNet
20%6
10%
92.80
93.11
100
100
91.22
92.29
0.30
0.01
97.81
94.83
100
100
94.09
94.98
0.03
0.01
30%
91.55
100
90.91
96.62
100
94.89
0.01
Blended
20%
10%
94.21
94.36
99.93
100
93.10
92.97
0.34
96.33
98.56
97.40
95.53
96.52
0.02
0.00
30%6
93.64
100
92.54
3.33
98.46
99.97
96.37
0.00
WaNet
20%
10%
90.86
90.12
97.26
98.73
91.19
92.63
0.19
0.14
97.08
97.10
98.36
94.21
95.61
95.6 25.65
0.02
0.02
30%
80.58
97.32
90.37
0.71
94,20
99.63
93.42
0.03
SSBA
10%
94.34
98.90
93.41
0.39
97.26
99.32
0.02
20%
93.47
99.66
92.72
0.23
96.42
95,15
96.03
0.01
30%6
93.27
99.97
92.07
0.11
97.71
99.20
97.03
0.00
Table 6: Defense results under various poisoning rate settings. The experiments conducted on the CIFAR- 100 dataset. All the metrics are measured in percentage (%). The best results are bold.
Poisning mice
No defence
LBAU
PF+SAM
NGP
FEntnnng (Oue)
Frint (OuS)
FST (Our))
C-Accit)
(CAAsst))
ASSR11]
C-Arxt))
ASM((
(-Aestt)
ASR[1]
C-Aext>
ASR([]
(-Acct()
ASR(1)
(-Aust")
ASR(()
RaINai
5%
1%
85.85
99.96
1100
63.27
K3.00
99.22
559350
$3.00
12.88
05.30
95.41
70.11
20.22
0.49
0.68
12.25
72.19
0.05
0.03
80.33
W0.00
0.63
0.05
79.54
18:99
0.90
0,10
0.9%6
84.71
91.11
62.50
02:78
83,12
$7.89
09.05
0.47
11:75
0.09
80.65
0.00
50.11
0101
Bleaded
$9.99
$3.14
07.86
01.70
09.55
50:30
20.89
14.72
72.25
72.53
80.22
M0.55
50.36
32.67
SSBA
5%
136
85.15
85.13
54.52
92.79
K3.55
62.94
20.49
4.15
$3.06
83.01
0.19
4.12
69.18
70.64
0.32
0.36
72.05
71.94
0.15
0.20
79.33
80.40
0.23
0.20
79.07
80.4
0.20
0.17
SIG
1%
35.48
40.12
$2.98
29.00
92.03
21.71
09.35
25.68
72.79
0.71
80.14
0.77
79.1
0.33
Averue
57,40
72.54
$2.96
66.34
70.00
9.30
72.25
32.56
80.25
0.58
50.01
Suandard Destation
39.47
43.67
0.39
11.48
0,35
0.29
1..0
ASR compared to FST, we note that its C-Acc gets hurt severely since it freezes the re-initialized linear head during fine-tuning which restricts its feature representation space. For the other two state-of-the-art defenses, we find that they are less effective in purifying the larger backdoor models.
We further observe that the FT-init could achieve comparable purification results the FST with even a slightly higher C-Acc. Compared to our previous experiments on the small-scale dataset (CIFAR- 10 and GTSRB) and model (ResNet- (SwinTransformer with the large-scale dataset (CIFAR-100 and Tiny-ImageNet) which decreases the average ASR by 32.31%.
find that FT-init is more effective on the large model
C.4 Additional Results of Adaptive Attacks
In addition to the Adaptive-Blend attack, we also provide evaluations of a parallel attack proposed in [26] called Adaptive-Patch. To further reduce latent separability and improve adaptiveness against latent separation- defenses, we also use more regular- ization samples, following ablation study of Section 6.3 [26]. The experimental results are presented in Table and demonstrate that our FST could purify both attack types with various regularization samples. We also demonstrate a T-SNE visualization of the Adaptive-Patch in Figure 11. It aligns with the results of Adaptive-Blec attack.
Adaptive-Patch ASR: 61.23%
FST
ASR: 2.21%
The T-SNE visualizations of Adaptive-Pate Patch attack (150 payload and 300 reg- ularization samples). Each color denotes cach class, and Black points represent hackdoored sam- ples. The targeted class is 0 (Red). The left figure represents the original backdoored model and the right represents the model purified with FST.
Figure 11:
To further assess stability of FST, we also test FST against training-contro adaptive attacks [30]. The au- thors [30] utilize an adversarial network regularization during the training process to minimize differences
between backdoor and clean features in latent representations. Since the authors do not provide source code, we follow their original methodology and implement their Adversarial Embedding attack with two types of trigger, namely the checkboard patch PPtcc)) and Hello-Kite -Kityy pattern
17

---

Table Defense results of Adaptive-Blent Adaptive-Patte attacks with various regularization samples. The metrics C-ACC and ASR are measured in percentage.
Attack
Regularization samples
No defense
FST
C-Acc(t)
ASR(()
C-Aco(")
ASR(()
150
94.55
96.77
0.28
Adaptive-
300
94.59
61.23
91.99
2.21
450
94.52
54.23
91.41
5.42
150
94.86
83.03
94.35
1.37
Adaptive-Bies Blend
200
94.33
78.40
92.08
0.78
300
94.12
68.99
92.29
1.39
Table 8: Defense results of Bypass attacks with three different poisoning rates.
Attack
Poisoning rate
No defense
FST
C-Aco(f)
ASR(J)
C-Ace(t)
ASR(L)
Bypass-Patch 5%
89.81
96,28
87.85
0.02
1%
90.04
93.90
87.83
0.03
0.5%
89.50
58.83
87.61
0.73
5%
87.79
99.54
89.14
0.13
Popass-Blend
1%
89.70
83.66
88.11
0.08
0.5%
89.47
85.52
87.13
0.12
Baypass-Blendl) All the experimental results along with three poisoning rates are shown in Table
8. The results reveal that our FST could still mitigate the Bypass attack which emphasizes the importance of feature shift in backdoor purification.
C.5 Additional Results of Projection Constraint Analysis
In this section, we first provide additional analysis of the projection constraint with more attacks (WaNet, SSBA, SIG, and LC) on the CIFAR-10 dataset. We show the experimental results in Figure 12. We get the same observations shown in Section 5.3, where the inclusion of the projection term plays crucial role in stabilizing and accelerating the convergence process of the FST. This results in rapid and satisfactory purification of the models within few epochs.
D Extra Ablation Studies
D.1 Efficiency Analysis
We compare the backdoor purification efficiency of our FST with other tuning methods the remaining three datasets including GTSRB, CIFAR-100, and Tiny-ImageNet. We select three
2224
FRT FST Ar CISS
CAME
EST W C1Si
FST Al E/UN) C.
C-008%
FST
wdr (16)
Tuning Epochs
Tuning Epochs
Tuning Epechs
Tuning Epochs
51G
515
LC
30
WTC EST NSTWAR 1350
PST AAA//AL
EST 30.54
(.0.99)
EST C15
Ale (11%)
wi C
1250
10
2
15
Tuning Eppoens
Tuning Epocis
Tuning Eoochs
Tuning Eppcns
Figure 12: We demonstrate the experimental results with and without projection constraint (w/ and w/o C, respectively) of four backdoor attacks, namely the WaNet, SSBA, SIG, and LCC The experiments are conducted with three poisoning rates (5%, 1 %%, and 0,5%) and varying tuning epochs.
18

---

representative attacks (BadNet, Blend, and SSBA) with poisoning rate 1% which could be successfully implemented across three datasets and we present our experimental results in Figure 13, 14 and 15. The experimental results demonstrate that our FST is efficient compared to the other 4 tuning-based backdoor defense which could constantly depress the ASR under a low-value range (usually < 5%) with only few epochs. Besides, we also note that in the GTSRB dataset, both the ASR of FE-tuning and FT-init would increase as the tuning epoch increases indicating the model is gradually recovering the previous backdoor features. Our FST, however, maintains a low ASR along the tuning process which verifies the stability our method.
D.2 Diverse Model Architecture
We conduct comprehensive evaluations on three model architectures (VGG19-BN, ResNet-50, and DenseNet-161) on the CTFAR- 10 dataset with all 6 representative poisoning- basend backdoor attacks and one adaptive attack, and our experimental results are shown in Table 9, 10 and 11. During our initial experiments, we note that our method is less effective for VGGI9-BN, One possible reason is that the classifier of VGGI9-BN contains more than one layer which is slightly different from our previously used structure ResNet-18. Therefore, one direct idea to extend our original last-layer regularization to all the last linear layers of VGGI9-BN For implementation, the original a (w, w to a Σi wyg wori) where i indicates each linear layer. Based on this, obtain obvious promotion of backdoor defense performances (shown in Figure 16) without sacrificing clean accuracy.
simply change
Following the results in Table 9, our FST could achieve better and much more stable performance across all attack settings with an average ASR 6.18% and a standard deviation of 11. 11.64%. Com- pared with the four tuning-based defenses, our FST could achieve 29% lower ASR average across all the attack settings; compared with the other two state-of-the-art defensive strategies, our FST achieves a much lower ASR while getting much smaller C-Acc drop (< 3.5%). For the other two architectures, we note that our FST could achieve the best performance across attack settings with an average ASR of 2.7% and maintain clean accuracy (the drop of C-Acc < 1.9%).
BadNet
Blended
SSBA
60-
(ASR)
40
NGF (ASR)
NGF (ASR)
NGF (ASR)
tuning (ASR)
tuning
tuning (ASR)
20
F Finit (ASR)
F'Finit (ASR)
FTinit (ASR)
Tuning Epochs
Tuning Epochs
Tuning Epochs
Figure 13: The ASR results of three representative attacks with various tuning epochs. Our experiments are conducted GTSRB with ResNet-18.
BadNet
Blended
SSBA
100
80
FT.SAM (ASR)
FT.SAM (ASR)
FT.SAM (ASR)
NGF (ASR)
(ASR)
(ASR)
FE:tuning (ASR)
FE turing (ASR)
FE funing (ASR)
FTinit (ASR)
FT-int (ASR)
FT-init (ASR)
FST (ASR)
FST (ASR)
FST (ASR)
5
10
15
20
5
10
15
20
2
5
10
15
20
Tuning Epochs
Tuning Epochs
Tuning Epochs
Figure 14: The ASR results of three representative attacks with various tuning epochs. Our experiments are conducted on CIFAR-100 with Swin formeromerr.
19

---

Table 9: Defense results under various poisoning rates. The experiments conducted on the CIFAR-10 10 dataset with VGG19-BN. All the metrics are measured in percentage (%). The best results are bold.
Anak
Peaceing
(tlen)
ASMrD
ASRIL1
(C/SeetT)
ASRU:
(C-cett)
ASRLL1
C.AN(T)
ASRLD
C-ieett
ASRLLI
ASRGLY
BadNe
50.00
91.30
1191
1100
$3.00
18:00
59-03
36-17
1.95
$1.83
$1.00
76-71
0.58
$3.50
0.81
5.74
$1.06
(3)
94-31
62:31
$1.00
$1.00
2.67
17:00
07:38
96.31
Rull
58
91.24
09:41
8355
53.53
86.38
38.54
54.00
13:21
82:38
7.38
$5.45
38:67
88.63
17.00
06.15
3:71
0.5%
00.00
10.04
91.24
76.22
10.30
31.00
14.25
34,86
64.38
13.50
10.00
$0.92
64.70
40.34
57.94
6.18
Wohe
0.69
81.50
1.30
18.25
8.55
10:31
558A
91.00
19-04
11.88
32:55
110
85.12
81.92
6.21
81.19
85.57
2.72
V1.26
95-41
2.58
2.51
33.0
5.10
0062
23.85
$4.16
16.05
6,11
$2.99
22.96
85.99
0.5%
90.00
62:42
$9.35
15.70
$1.35
40.39
$5.05
9.39
$3.95
20.47
85.91
$2.22
85.51
33.41
$6.54
1.85
LC
18
01.79
91.
100
$4.16
02:57
11.26
30L35
57:72
93.13
70.04
$3.77
55.10
36.27
13.00
83.05
83.73
31.67
30.35
03.25
19.55
91.01
$4.81
$1.04
$7.00
50.87
06.20
56-73
6,1m
0.80
Allgttie Bind
0.3%
90.11
06.54
$3.19
34.02
88.10
27.35
$9.36
14.97
99.09
33:48
10026
46.95
90.31
$1.82
10005
43.44
Arerae
91.55
81.26
33.70
16.08
43.1
13.55
28.00
21.88
18.05
$2.56
83.30
40/00
87.13
6.16
Table 10: Defense results under various poisoning rates. The experiments are conducted on the CIFAR-10 10 dataset with ResNet-50. 50, All the metrics are measured in percentage (%) The best results are bold.
Anack
Nio dciense
ANP
F-BAU
FI-3AM
NCP
P8-taxeg (Clan)
7Fasi (Dur)
PST IDaaoo
[-AedT):
14.10
$5.42
01.55
10.20
3.14
2.14
$2.84
0.24
28:17
3.51
0.58
54.75
59.95
27.16
0.85
90.22
2.85
$6.95
3.50
67.79
4.15
91.35
1.50
99.33
0.44
55.03
6.07
Stased
28
144.00
07:33
$3.50
0.82
3.33
91.05
37.35
6.25
50:35
12.16
27.14
14.55
05.13
4.00
01.47
25.71
10:73
$3.31
10
13.55
$1.20
10.15
2,31
6.20
WaNe:
53
52.05
52:85
$4.00
1.82
20121
1.58
51.6
0.52
10.34
1.30
00.25
1.11
$2.86
18.00
90.43
6,31
0.455
$6.74
06.00 05
$5.51
1.08
$7.56
0.56
01.59
1.50
04.50
1.58
91.81
1.15
$2.57
11:00
90.41
1.70
S38A
176
04.25
72.64
02.59
1.31
01.25
2.15
50.30
2.50
$7.59
1
2.36
$2.56
3.38
12.41
8,37
91.75
1.30
2:20
91.95
SIG
1%
91.00
06.87
03.07
2.83
91.81
6.76
50.40
1.58
47.02
0.51
01.85
07.01
05.54
90:11
0.5%
04.05
96.91
92.59
15.07
91.94
1.20
5820
11.07
$7.36
0.24
56.25
10.55
92.94
$3.00
90.17
6.52
1.0
12
64.30
09.87
93.17
15:40
91.93
17.37
58.00
11.71
97.90
11.84
91.35
18.45
92.83
$5.16
$2.6
1.51
0.9%
91.67
WiN
00.00
63.14
W175
5.66
66:57
65.99
47.36
19:24
91.45
19:50
$1.30
11.00
90.37
17:33
Alleetve Rieed
0.3%
96.36
14.57
$3.75
14.57
90.51
10.13
$2.40
35.21
99.18
17.54
91.14
11.41
91.18
15.00
90:95
1.43
Awerae
M0.7
90.06
91.77
Stankel Devatto
135
16.00
2.66
25.35
132
13.4
0.66
11.35
006
491
0.0
16.56
0.47
33.00
121
4.30
Table 11: Defense results under various poisoning rates. The experiments are conducted on the CIFAR-10 dataset with DenseNet-161 161. All the metries are measured in percentage (%). The best results are bold.
Pitceing mak
No difeiee
IT-SAM
NGF
IIST (0tm))
ASRLL1
C.ANNT)
ASRL11
ASRL:
(-ALo())
ASKLL1
C.ANET)
ASRLD
CIANAT)
BadNe
3%
B0:35
19093
$5.46
$3.70
56.72
2,00
$3.20
14.35
$3.54
2.41
25015
2.40
$1.06
3.32
$5.62
1.4
0.3%
B5/02
19.35
$8.50
$9.10
04.05
30.31
$328
4.30
$3.56
19:46
83.10
9.26
81.78
43.30
977.5
5.30
12
$5.02
02.80
93.62
64.44
96.31
4.97
$2.00
3.16
83.82
2.36
84.74
1.62
$3.65
23.34
92.09
6,47
4.%
90.15
10.44
$5.11
9.38
$6.94
25.36
83.79
4.4
84.5
114
00.30
1.30
84.55
31.7%
16.37
6.31
Wahen
58
$3.76
66:35
$3.75
1.74
81.95
3.48
$8.51
1.51
$4.93
1.51
$3.09
7.51
65.32
1.30
95.0
6.92
310
58
69.76
92.71
$9.01
0.54
53.71
24.43
$3.96
0.79
83.55
0.91
85.15
6.54
$7:53
15.00
80.35
6.00
0.5%
$6:96
06.17
92.32
61.05
85.95
67.39
$3.78
3.66
33.94
1600
$4.93
10.91
$7.03
65:1
$6.6)
4.00
1.C
1%
$6.76
09:65
89.7
57
56.35
91.11
$756
377
4130
14.00
$1.93
13.87
84.00
07.00
92.5
1.52
0.5%
$8.34
99664
$5.40
99.52
$1.40
01.75
79.4
3.83
91.22
21:38
$2.35
19.37
83.64
14/0
$6.69
13.19
Adgeriv Blad
033%
18.18
$1.77
85.74
33.56
16.33
1.87
$6.00
15.19
85.14
4/05
80.20
3.37
$9.30
3.85
16-11
1.53
3.56
15.50
A.7
Stuvket Desurve
18)
15.0
3.00
11443
1/6
13.35
1.54
131
803
9/00
1.0
541
1.01
13.48
1/16
M
20

---

BadNet
Blended
SSBA
100
FT-SAM (ASR).
FT-SAM (ASR)
FT-SAM (ASR)
NGF
(ASR)
FE-tuning (ASR]
FE-tuning (ASR)
FE-tuning (ASR)
FTinit
FTint
FT-init
FST (ASR)
PST (ASR)
FST (ASR)
10
15
20
10
15
10 Tuning Epochs
Tuning Epochs
Tuning Epochs
Figure 15: The ASR results of three representative attacks with various tuning epochs. Our experiments conducted on Tiny-ImageNet with Swin Transforoformrr.
60
40
BadNet-all (C-Acc)
BadNet-all (ASR)
Blended-al (C-Acc)
Blended all (ASR)
20
BadNet-last (C-Acc)
BadNet-last last (ASR)
Blended last (C-Acc)
Blended -ast (ASR)
0.5%
1%
0.5%
1%
5%
Poisoning Rate
Poisoning Rate
Figure 16:
We compare regularizing the whole linear layers (denoted as -all) with regularizing only the last linear layer (denoted as -last). We evaluate on CIFAR-10 dataset using BadNet and Blended attacks with 3 poisoning rate settings. Experimental results demonstrate that could achieve superior purification performance by regularizing the whole linear layers than the last-layer-only regularization without sacrificing the model performance.
21


Reviews:
[{"review_id": "ALCu6Ky9or", "reviewer": "Reviewer_Zkrc", "summary": "This paper proposes a novel backdoor defense approach called Feature Shift Tuning (FST) that actively promotes feature shifts to disentangle the learned features of a deep neural network. The authors conduct a thorough assessment of widely used tuning methods, vanilla Fine-tuning (FT), and simple Linear Probing (LP), and demonstrate that they both completely fail to defend against backdoor attacks with low poisoning rates. \n\nFST is an end-to-end tuning method that actively shifts features during fine-tuning by encouraging the difference between the tuned classifier weight w and the original backdoored classifier weight w_ori. FST significantly shifts backdoor features and makes them easily separable from clean features of the target class, which effectively mitigates backdoor attacks. In summary, the proposed FST method is a strong backdoor defense paradigm that can achieve stable improvements in both robustness and clean accuracy compared to other initial methods such as FE-tuning and FT-init. \n", "questions": "Listed in the weakness of the paper. Score can be improved if concerns listed above are resolved.", "limitations": "The authors addressed the limitation that they assume the defender would hold a clean tuning set which may not be feasible in certain scenarios.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed FST (Fine-tuning with Separated Transformations) method is a strong backdoor defense paradigm that can achieve stable improvements in both robustness and clean accuracy compared to other initial methods such as FE-tuning and FT-init. FST is an end-to-end tuning method that actively shifts features during fine-tuning by encouraging the difference between the tuned classifier weight w and the original backdoored classifier weight w_ori. This is giving a good reference to the research on this aspect.\n\n- FST disentangles the clean and backdoor features, which helps to purify the model. It significantly shifts backdoor features and makes them easily separable from clean features of the target class, which effectively mitigates backdoor attacks. FST outperforms other methods by a large margin for backdoor robustness on CIFAR-10 and GTSRB datasets. \n\n- FST is effective against adaptive attacks such as the Adaptive-Blend attack, achieving an average drop on ASR by 81.66% and 91.75% respectively for the CIFAR-10 and GTSRB datasets. FST is a unified method that addresses both the clean accuracy drop and unsatisfied robustness improvements that are present in other backdoor defense methods. The ablation study is organized well to clearly demonstrate the whole proposed method. And it makes the paper easy to follow.\n", "weaknesses": "- [Contribution and novelty] The contribution of paper is somehow incremental, as the method is developed based on previously researched findings. The backdoor attack can be erased by decoupling the feature extractor and the linear classifier in [1].\n\n- [Adaptive Attack] There, the submission only briefly mentions an attempt at achievingthe  disentanglement between the clean and backdoor features through an adaptive attack, but fails to do so effectively, and provides only limited insight why the adaptive attack failed. Please take a closer look at providing a strong evaluation of the adaptive attack scenario. Besides, the paper [2] also proposes another defense, Adaptive-Patch, which should also be discussed when attacking the propsoed FST.\n\n[1] Huang, Kunzhe, et al. \"Backdoor Defense via Decoupling the Training Process.\" International Conference on Learning Representations. 2021.\n\n[2] Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assump405\ntion of latent separability for backdoor defenses. International Conference on Learning Representations. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**First of all, thank you for your recognition of our work.**\n\n### Response to Weakness 1:\n\n**Response to “The contribution of paper ...... previously researched findings.”:**\n\n1. The authors of [1] propose that end-to-end supervised training makes the model learn backdoor features. Hence, they propose a two-phase training defense method called DBD. DBD utilizes self-supervised learning (SSL) to train feature extractor and then trains a linear classifier with the frozen feature extractor.\n2. In contrast to DBD, our work focuses on how to efficiently purify existing backdoor triggers during tuning process, rather than training process. We first investigate if common FT methods in the pretrain-tuning paradigm, such as vanilla FT and LP, can consistently provide robustness against attacks across various settings. **We find that**: 1) under high poisoning rate setting where backdoor and clean features of targeted class are well separable, simple LP could provide satisfactory defense performance; 2) under low poisoning rate setting, backdoor and clean features are tangled together (Figure 2 of Section 3.1). As a result, FT and LP fail to purify inserted backdoors due to insufficient feature shifts.  \nWhile DBD shows that feature extractor can learn backdoor features, it does not discuss how learned features vary at different poisoning rates, particularly in terms of separability between clean and backdoor features. This is crucial for designing robust and stable defense methods for various attack settings.\n3. Motivated by our findings, we propose two initial methods (FE-tuning and FT-init) to encourage separability between backdoor and clean features. However, they still suffer from a clean accuracy drop or unsatisfied robustness improvements. To achieve unified improvement, we further propose FST based on them, a simple end-to-end tuning method. It actively encourages discrepancy between tuned and original classifiers to guide more shifts on backdoor features.\n\n**Response to “The backdoor attack can ...... and the linear classifier in [1].”:**\n\n1.  As mentioned above, FST is an end-to-end tuning method, which does not decouple the feature extractor and linear classifier like [1]. Our initial method, FE-tuning, adopts a decoupling form in tuning process. However, they still suffer from a clean accuracy drop or unsatisfied robustness improvements (Table 1 of Section 3.2). As shown in Section 5.2, FST achieves better and more stable defense performance under various settings. Compared with high training costs of DBD, FST is a much simpler and more flexible method that can be easily integrated with existing training methods and pretrained models.\n\n----\n\n### Response to Weakness 2:\n\nThanks for your suggestion. We first give reasons why we believe attacks [2] could be powerful adaptive attacks against our method. Then, we follow your suggestion and conduct a more detailed adaptive attack evaluation based on [2]. We finally provide explanations for why adaptive attacks failed. \n1. As shown in Section 3.1, under high poisoning rate, backdoor features are clearly separable from clean features of targeted class and thus could be easily purified by vanilla FT and LP. To bypass defense methods based on latent separation property, Adaptive poisoning attacks [2] actively suppress the latent separation between backdoor and clean features. So that they can achieve a high ASR by adopting extremely low poisoning rate and adding regularization samples. This also corresponds to our finding in Section 3.2 that entanglements between clean and backdoor features in low poisoning rate settings make FT and LP fail. The evaluations in [2] also show that these attacks successfully bypass existing strong latent separation based defense. Hence, we believe it is also equally a powerful adaptive attack against our FST method.\n2. Following the reviewer’s suggestion, we add evaluations of Adaptive-patch attack on CIFAR-10 and ResNet-18. To further reduce latent separability and improve adaptiveness against latent separation based defenses, we also use more regularization samples, following ablation study of Section 6.3 [2]. We show results in **Table 1 of one-page PDF**. We can observe that even against more stealthy adaptive attacks, FST still achieves outstanding defense performance.\n3. Our work and [2] mainly focus on practical data-poisoning backdoors. To further assess stability of FST, we also test FST against training-control adaptive attacks [3]. The authors [3] utilize an adversarial network regularization during training process to minimize differences between backdoor and clean features in latent representations. Since the authors do not provide source code, we implement it based on descriptions in original paper. The results are shown in **Table 1 of one-page PDF**. FST still significantly reduces ASR. This further proves the excellent stability of our method.\n4. To explain why adaptive attacks fail, we provide TSNE visualizations of learned features from backdoored models, as shown in Section 3.2. We show results in **Figure 1 of one-page PDF**. We can first observe that adaptive attacks significantly reduce latent separability. Clean and backdoor features are tightly tangled. FST effectively shifts backdoor features and makes them easily separable from clean features of the target class. Therefore, the feature extractor will no longer confuse backdoor features with clean features of target class in feature space. This leads that subsequent simple linear classifier is difficult to be misled by backdoor samples, resulting in more robust classification.\n\nWe will include adaptive attack evaluations in the revision. \n\n[3]. Bypassing backdoor detection algorithms in deep learning, Euro S&P. \n\n----\n\nWe hope that our answers can address your concerns satisfactorily and improve the clarity of our contribution. We would be grateful if you could re-evaluate our paper. We look forward to receiving your further feedback.\n\n"}, {"review_id": "LTUm5BH64o", "reviewer": "Reviewer_chxU", "summary": "This paper finds that fine-tuning is less effective in defending against backdoor attacks with a low poisoning rate, due to the strong coupling between the clean feature and the backdoor feature. Therefore, a tuning-based backdoor purification method called feature shift tuning (FST) is proposed, which is simple and efficient. FST can effectively decouple the clean feature and the backdoor feature, and thus eliminate the backdoor in the victim model.", "questions": "Please refer to weakness.", "limitations": "It is suggested that the authors provide further explanation as to why the method worked.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a simple and effective backdoor elimination method, which achieves good results in a specific scenario where the poisoning rate of the backdoor attack is relatively low.", "weaknesses": "1. This paper finds that fine-tuning has difficulty defending against backdoor attacks with a low poisoning rate when ResNet-18 is the victim model, but does not sufficiently demonstrate the generalizability of the problem. The same claim is not guaranteed to hold when the model capacity, model architecture, and dataset type change.\n\n2. This paper lacks an explanation of why the method works and does not explore the effect of the range of poisoning rates on the victim model.\n\n3. It is desirable to include backdoor elimination methods other than tuning-based methods as baseline methods as well to demonstrate the efficiency of FST.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**First of all, thank you for your recognition of our work.**\n\n### Response to Weakness 1:\n\nThanks for your constructive comments. \nWe add evaluations of vanilla FT and LP using ResNet-50 (increased model capacity) and Dense-161 (different model architecture) on CIFAR-10. We also take evaluation on GTSRB (different dataset types). As mentioned in Section 3.1, we mainly focus on defense performance with a satisfactory clean accuracy level (92% on CIFAR-10, 97% on GTSRB). We tune hyperparameters based on this condition. The results are shown in **Figure 3 of one-page PDF**. \n\nWe could observe that Vanilla FT and LP could purify backdoored models for high poisoning rates but fail to defend against low poisoning rates attacks. The only exception is the SSBA results since the original backdoored models have a relatively low ASR, as mentioned in Section 3.1. These results align with our findings in Section 3.1. We will add these suggested evaluations in our revised version.\n\n----\n### Response to Weakness 2 and Limitations:\n\n**Response to “This paper lacks an explanation of why the method works”:**\n\n1. In lines 220-231 on page 6, we explain why FST can provide better robustness and bring unified improvements for initial methods, FE-tuning, and FT-init. Additionally, in the final figure of Figure 3, we include T-SNE visualizations of the feature extractor after applying FST to confirm its effectiveness in purifying backdoor features.\n2. As shown in Section 3.2, we first find that in low poisoning rates scenarios, entanglements between the backdoor and clean features make FT and LP fail to defend against attacks. We propose two initial methods, FE-tuning and FT-init, to promote backdoor feature shifts and make them easily separable from the clean features of the target class. Hence, the feature extractor will no longer confuse backdoor features with clean features of the target class in the feature space. This leads that the subsequent linear classifier is difficult to be misled by backdoor samples, resulting in more robust classification. The evaluation and visualization in Section 3.2 verify their effectiveness on backdoor purification. \n3. However, these two methods still suffer from a clean accuracy drop or unsatisfied defense performance. To achieve unified improvement, we further propose FST which actively shifts features by encouraging the difference between the tuned and original classifier. Compared with FE-tuning, FST brings an extra loss term on $\\min_{w} E_{(x,y)\\sim D_{T}} L(f(w; \\phi(\\theta;x)), y)+ \\alpha <w,w^{ori}>$ to update linear classifier $w$. It further promotes feature shift by penalizing classifiers more intensely and also maintains the models’ ACC with the original CE loss term. Compared with the FT-init, by adopting $\\alpha <w,w^{ori}>$, FST encourages discrepancy between tuned $w$ and original $w^{ori}$ to guide more shifts on learned backdoor features.\n4. Feature visualization in Figure 3 shows that our FST significantly shifts backdoor features and makes them easily separable from clean features of the target class. Comparisons with other defense methods in Section 5 verify the superiority of FST in defending against attacks. \n\n**Response to “does not explore the effect of the range of poisoning rates on the victim model.”:**  \n\nThanks for this constructive comment. In the main submission, we mainly focus on attacks with low poisoning rates (0.5%, 1%, 5%) and evaluate FST on them, since Vanilla FT and LP are ineffective in defending against them in our revisiting experiments. To fully assess FST’s effectiveness, we take more evaluations of FST by increasing the poisoning rate to 30% on CIFAR-10 and GTSRB with ResNet-18. The results are shown in **Table 2 of one-page PDF**.  \nWe can observe that FST still achieves stable and outstanding defense performance under high poisoning rate setting, reducing ASR below 2%. This further verifies the effectiveness of our method. We will add the suggested evaluations in our revised version.\n\n----\n\n### Response to Weakness 3:\n\nThanks for this valuable suggestion. To further assess FST’s effectiveness, we add another training defense method, ABL [1], which also performs well in BackdoorBench. Following settings of Section 5.1, we evaluate it in CIFAR-10 with ResNet-18. The results are shown in the below table. We could observe that FST achieves better and more stable performance than ABL under various settings with maintaining good clean accuracy. \n\n[1] Anti-backdoor learning: Training clean models on poisoned data, NeurIPS 2021.\n\n**CIFAR-10 and ResNet-18**\n|Attack|Poisoning rate|ABL(ACC/ASR)|FST(ACC/ASR)|\n|:-------:|:-------:|:-------:|:-------:|\n|BadNets|5%|89.65/0.08|93.17/0.00|\n||1%|72.34/7.13|92.81/0.01|\n||0.50%|71.59/9.97|93.63/0.02|\n|Blended|5%|86.36/0.74|92.87/3.07|\n||1%|71.48/18.77|93.59/0.19|\n||0.50%|71.82/49.66|93.15/0.06|\n|WaNet|5%|69.78/77.39|91.56/0.26|\n||1%|72.08/31.17|91.83/0.51|\n||0.50%|74.31/13.23|91.70/0.78|\n|SSBA|5%|73.44/24.61|93.48/0.27|\n||1%|73.72/72.57|93.32/0.56|\n||0.50%|73.33/26.29|92.97/0.04|\n|SIG|5%|89.31/2.91|93.24/0.02|\n||1%|89.73/5.37|93.09/0.03|\n||0.50%|74.07/50.60|93.18/0.01|\n|LC|5%|90.19/4.29|93.47/0.68|\n||1%|67.41/8.83|93.51/0.30|\n||0.50%|75.42/99.84|93.44/1.71|\n\n----\n\nWe hope that the above answers can address your concerns satisfactorily. We would be grateful if you could re-evaluate our work based on the above responses. We look forward to receiving your further feedback.\n"}, {"review_id": "dqv75aGdPG", "reviewer": "Reviewer_2qbe", "summary": "This paper observes that while fine-tuning and linear probing can act as effective defenses in the high-poisoning-rate regime, they completely fail in the low-poisoning-rate regime. The paper further shows that this is due to the fact that in the low-poisoning-rate regime, extracted features of backdoored and clean samples are highly entangled. The proposed defense is to fine-tune the backdoor model with an additional constraint to force the fine-tuned weights to differ from the original (poisoned) weights. Extensive experiments show that the proposed defense provides excellent backdoor purification while maintaining clean accuracy.", "questions": "- It would be nice if the authors could extend Figure 2 to other datasets. Specifically, I am interested in seeing the results on Tiny-ImageNet and CIFAR-100.", "limitations": "- One potential limitation of this work is the assumption of having access to clean data for training, which may not always be practical in real-world scenarios. However, the authors have acknowledged this limitation and expressed their commitment to addressing it in future research.\n- It would have been more insightful if the authors had delved deeper into why low poisoning rates cause entanglement between backdoored and clean features.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper shows an interesting observation that fine-tuning and linear probing perform significantly worse as backdoor defenses in the low poisoning rate regime.\n- This observation is followed by a thorough and nicely written analysis in Section 3.\n- The proposed defense is simple, elegant, yet effective.\n- Very extensive experiments.", "weaknesses": "Please pay more attention to weaknesses marked with \"major\" severity.\n- **[minor, clarity]**: L52, please also mention that FE-tuning randomly re-initializes the classifier head. I was confused when looking at Figure 1 at first because it was not mentioned previously that FE-tuning does re-initialize $f(w)$.\n- **[minor, typo]**: L102, \"Here We\" -> \"Here, we\"\n- **[major, lack of experiments on high poisoning rates]**: The proposed defense should be tested with high poisoning rates just to ensure that it performs equally well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**First of all, thank you for your recognition of our work.**\n\n### Response to Weakness minor 1 and 2:\nThank you for your helpful suggestion! We apologize for the confusion caused. We will add explanations about FE-tuning in Line 52 in our revised version. We will correct all the typos and carefully polish the paper in the revision.\n\n----\n### Response to Weakness major 1:\n\nThanks for your valuable suggestion! To fully assess FST’s effectiveness, we take more evaluations of FST by increasing the poisoning rate to 30% on CIFAR-10 and GTSRB with ResNet-18. The results are shown in **Table 2 of one-page PDF**.\nWe can observe that FST still achieves stable and outstanding defense performance under high poisoning rate setting, reducing ASR below 2%. This further verifies the effectiveness of our method. We will add the suggested evaluations in our revised version.\n\n----\n### Response to Questions:\n\nThanks for your valuable suggestion. We conduct evaluations of vanilla FT and LP on CIFAR-100 and Tiny-ImageNet with pre-trained SwinTransformer. As mentioned in Section 3.1, we mainly focus on defense performance with a satisfactory clean accuracy level (80%). We tune hyperparameters based on this condition. The results are shown in **Figure 3 of one-page PDF**. We also provide results on ResNet-50, and Dense-16 on CIFAR-10 and GTSRB. We could observe that Vanilla FT and LP could purify backdoored models for high poisoning rates but fail to defend against low poisoning rates attacks. The only exception is the SSBA results since the original backdoored models have a relatively low ASR, as mentioned in Section 3.1. These results align with our findings in Section 3.1. We will add suggested evaluations in our revised version.\n\n----\n\n### Response to Limitation 1:\nThanks for your comment. \n1. We utilize a very small clean subset to conduct tuning methods in this work. As mentioned in Section 5.1, we follow previous work and only reserve either 2% or 5% of the original training data as the tuning dataset for both CIFAR-10 and GTSRB or CIFAR-100 and Tiny-ImageNet. Additionally, in Section 5.3, we also evaluate FST under a rigorous scenario with much fewer clean tuning samples. We reduce the size of our CIFAR-10 tuning set from 2% to 0.1%, which is around 50 samples. As shown in Figure 5 (c,d), FST consistently performs well across various tuning data sizes, even when the tuning set only contains 50 samples. \n2. In real-world scenarios, we may be able to utilize existing dataset filtering methods [1] to help us construct this small tuning dataset. In this work, our main objective is to develop more robust tuning defense methods. We will investigate constructing a tuning dataset in future work. \n\n----\n\n### Response to Limitation 2:\n\nThanks for your helpful suggestion! In the main submission, we mainly focus on further improving the defense performance of tuning methods against low poisoning rates attacks, after first observing that existing tuning methods fail to provide stable robustness. Here we first present some initial intuitions based on related work [2,3] and will try to investigate the reasons behind that and provide a detailed analysis in future work. \n\nOur intuition is that to learn a stable mapping between backdoor patterns in input and the target class, backdoor samples will lead the model to tend to differentiate between backdoor features and the clean features of the target class. Therefore, the former is not influenced by the latter and dominates in the model decision process, leading to the wrong classification. The high poisoning rate attacks thus lead to the obvious and easy separation in feature space. While keeping ASR, attacks with low poisoning rates lead to more stealthy backdoor features which are more close to clean features. \n\n[1]. Meta-Sift: How to Sift Out a Clean Data Subset in the Presence of Data Poisoning, Usenix 2023. \n\n[2]. Revisiting the Assumption of Latent Separability for Backdoor Defenses, ICLR 2023.\n\n[3]. Spectral signatures in backdoor attacks, NeurIPS 2018. \n\n----\n\nWe hope that the above answers can address your concerns satisfactorily. \n\n\n\n\n\n\n\n\n"}, {"review_id": "hEuIBwv3lh", "reviewer": "Reviewer_X5wr", "summary": "This paper studies the effectiveness of finetuning in backdoor defense with a low poisoning rate and finds that the feature entanglement at low poisoning rate affects the effectiveness of finetuning. Thus, this paper proposes 3 new finetuning strategy FE-tuning, FT-init, and FST. The experiments demonstrate a promising result.", "questions": "1. How much data is used in FST? From my point of view, the reinitialization and the larger difference loss needs more data. But if the amount of data is too large, the defenders can use these data to retrain a new model.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. The finetuning study is interesting.\n2. FST only uses finetuning to achieve the best backdoor removal results, which is very impressive.\n3. This paper is well written and easy to follow.\n", "weaknesses": "Although this work is interesting, I still have several concerns.\n1. How much data is used in FST? From my point of view, the reinitialization and the larger difference loss needs more data. But if the amount of data is too large, the defenders can use these data to retrain a new model.\n2. As demonstrated in the paper, only finetuning the feature extractors cannot remove backdoor successfully. If so, it demonstrates that there is poison in the linear layers, can you explain this phenomenon?\n3. Can you provide more TSNE results against more attacks like Figure3 such as BadNets and WaNet?\n4. Although this paper focuses on a low poisoning rate, I think results on poisoning rate such as 20% should be also considered. Because the defenders cannot know the real poisoning rate and the robustness of different poisoning rates is important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**First of all, thank you for your recognition of our work.**\n\n### Response to Weakness 1 & Question 1:\n\n**Response to “How much data is used in FST?”:**  \n\nAs mentioned in Line 238-241 of Section 5.1, we follow previous work and only reserve either 2% or 5% of training data as the tuning dataset for both CIFAR-10 and GTSRB or CIFAR-100 and Tiny-ImageNet. The clean tuning sets consist of 1000, 768, 2500, and 5000. These subsets are relatively small in size. We will highlight details of our tuning dataset in the revised version.\n\nWe also evaluate FST under a rigorous scenario with limited clean tuning samples in Section 5.3. We reduce the size of CIFAR-10 tuning set from 2% to 0.1%, which is around 50 samples. As shown in Figure 5 (c,d), FST consistently performs well across various tuning data sizes, even when the tuning set only contains 50 samples.\n\n**Response to “From my point of view, the reinitialization and the larger difference loss needs more data.”:**\n\nWe think that the following points can help FST achieve a better trade-off between ACC and ASR based on very small tuning datasets:\n1. The projection constraint $||w||_2 =C$ shrinks the range of feasible set and stabilizes the tuning process. As mentioned in Line 203-219 of Section 4, to avoid the $w$ exploding and the inner product dominating the loss, we add a constraint on $||w||$ and constrain it as $ ||w^{ori}||$ rather than manual hyperparameter. It limits feasible set on the $\\ell_2$ norm ball ($ ||w^{ori}||$), which stabilizes tuning process of FST and leads to significant convergence improvement. As shown in Efficiency analysis of Section 5.3, Our method FST quickly converges in a few epochs while achieving good defense performance and clean accuracy. \n2. when tuning on small tuning sets, the original CE loss term in our objective (Eq.(1)) help tuned model maintain clean accuracy. Compared with FE-tuning without CE loss term, FST achieves better clean accuracy and defense performance.\n3. We only reinitialize and add difference loss to linear head. It only involves a small portion of parameters of the entire model, and feature extractor is already initialized with good accuracy.\n\n----\n\n### Response to Weakness 2:\n\n**Response to “As demonstrated in the paper, only ...... cannot remove backdoor successfully.”:**\n\nActually, we do not use tuning methods that only finetune feature extractors. We speculate that the reviewer is referring to the FE-tuning. As described in Line 169 of Section 3.2 and shown in Figure 1, FE-tuning first randomly re-initialize the linear classifier and then only tune feature extractor.  We apologize for any confusion caused.  As suggested by Reviewer 2qbe, we will add detailed explanations about FE-tuning in Line 52 in our revised version. \n\n**Response to “If so, it demonstrates that there is poison in the linear layers, can you explain this phenomenon?”:**\n\n1. FE-tuning randomly re-initializes linear classifier, so there are no poisons in new linear classifier. However, it still fails to completely eliminate backdoor triggers under various settings. The reason is that for low poisoning rate attacks, FE-tuning did not effectively shift learned features. The random initialization of linear classifier may not be sufficient to guide enough shifts on learned features. This is also why we have proposed FST with adding an extra penalty on linear classifier ($\\alpha\\*<w,w^{ori}>$). It encourages discrepancy between tuned and original backdoor classifier and also further promotes feature shifts, as mentioned in Section 4. The stable defense performances under various settings also verify the effectiveness of FST. \n2. As shown in Section 3.1, under high poisoning rate, LP achieves good defense performance. This demonstrates that we could only purify backdoors in linear head to defend against attacks when backdoor features are clearly separable from clean features under high poisoning rate settings. However, as mentioned in Section 3.2, we need more feature shifts to defend against low poisoning rate attacks. \n\n----\n\n### Response to Weakness 3:\n\nThanks for your suggestion. Following settings in Section 3.2, we provide TSNE results of BadNets and WaNet attacks. The results are shown in **Figure 2 of one-page PDF**.  \n\nUnder high poisoning rate (10%), the backdoor features (black points) are clearly separable from clean features (red points) and targeted clean and backdoor features are closer to each other under low poisoning rate (1%). This is consistent with results in Section 3.2. As shown in top row, consistent with observations of Blended attack in Section 3.2, vanilla FT still suffers from providing sufficient feature modification, leading to failed defense against BadNet attacks. WaNet fails to achieve 100% ASR in low poisoning rate settings without defense. Hence, its backdoor features are not very close to clean features, unlike Blended and BadNet attacks. Therefore, vanilla FT could also defend against it like other defense methods shown in Section 5.2. \n\nIn response to Reviewer Zkrc, we also conduct visualization of adaptive attacks. As shown in **Figure 1 of one-page PDF**, FST still significantly shifts backdoor features and makes them separable from clean features. \n\n\n----\n\n### Response to Weakness 4:\n\nThanks for this constructive suggestion. To fully assess FST’s effectiveness, we take more evaluations of FST by increasing poisoning rate to 30% on CIFAR-10 and GTSRB with ResNet-18. The results are shown in **Table 2 of one-page PDF**. We can observe that FST still achieves stable and outstanding defense performance under high poisoning rate, reducing ASR below 2%. This further verifies the effectiveness of our method. We will add the suggested evaluations in our revised version.\n\n----\n\nWe hope that the above answers can address your concerns satisfactorily. We would be grateful if you could re-evaluate our work based on the above responses. We look forward to receiving your further feedback.\n\n\n\n"}, {"review_id": "rmWFTBorRp", "reviewer": "Reviewer_JBzP", "summary": "Based on the observations that the previous FT and FP methods fail at low poisoning rate scenarios, this paper finds out the potential reasons in terms of clean and backdoor feature separation degree. It proposes FST to solve the problem by disentangling the features and validates its effectiveness with multiple backdoor attacks.", "questions": "1. Do the dynamic backdoor attacks, such as WaNet[1] and IAB[2], which generate triggers according to the input, follow the same phenomenon as in Figures 1 and 2? Since the WaNet is used in Section 5 while not in the pre-experiment part.\n\n[1] Nguyen, Anh, and Anh Tran. \"Wanet--imperceptible warping-based backdoor attack.\" *arXiv preprint arXiv:2102.10369* (2021).\n\n[2] Nguyen, Tuan Anh, and Anh Tran. \"Input-aware dynamic backdoor attack.\" *Advances in Neural Information Processing Systems* 33 (2020): 3454-3464.", "limitations": "Please see above.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to read. No typo is found.\n2. The experiments are sufficient to demonstrate the findings of previous methods’ failure in low poisoning rate scenarios. And the experiments to verify the potential reason and validate the two initial methods are also convincing.\n3. The proposed methods are intuitively reasonable, which are derived step-by-step from the observations of the pre-experiments to the simple solutions, and the final version with penalty.\n4. The idea is simple yet effective as shown in Sections 4 and 5, which is easy to follow.", "weaknesses": "In general, I appreciate this paper, but there are still some concerns from my perspective.\n\n1. The implementation of the proposed methods is not publicly available, only the public library ‘BackdoorBench’ is provided.\n2. The effectiveness of the constraint term C in equation (1) is expected to be discussed in the ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**First of all, thank you for your recognition of our work.**\n\n### Response to Weakness 1:\n\nThanks for your comment. \nWe will release all of our code, including the corresponding tuning parameters and training checkpoints, in our final version to ensure that the results of all our experiments are reproducible. \n\n----\n\n### Response to Weakness 2:\n\nThanks for this suggestion. As mentioned in Section 4, to stabilize tuning process, we add a constraint on the norm of $w$. To reduce tuning cost, we directly set it as $||w^{ori}||$ instead of manually tuning. They will be adjusted adaptively based on trained models. To show effects of our constraint, we offer the performance of FST's entire tuning process on CIFAR-10 and ResNet-18, with or without the projection term. The results are shown in **Figure 4 of one-page PDF**. The blue and purple lines represent results with or without projection. The different line type means various poisoning rate. \n\nWe could clearly observe that the projection stabilizes tuning process of FST, leading to significant convergence improvement. FST quickly converges in a few epochs while achieving good robustness and clean accuracy. We will add this study to our revised version. \n\n----\n\n\n### Response to Question 1:\n\nThanks for your helpful comment. Following settings in Section 3.1, we add evaluations of WaNet and IAB attacks on CIFAR-10 with ResNet-18. The results are shown in **Figure 1 of one-page PDF**. \n\nFor IAB attack, Vanilla FT and LP could purify backdoored models for high poisoning rates but fail to defend against low poisoning rates attacks. WaNet can not achieve 100% ASR in low poisoning rate settings even without defense. Therefore, vanilla FT and LP could also defend against it like other defense methods shown in Section 5.2. \n\nIn response to Reviewer chxU, we also extend evaluations to more models, attacks, and datasets in **Figure 3 of one-page PDF**. The results are consistent with the findings in Section 3.1.\n\n----\n\nWe hope that the above answers can address your concerns satisfactorily.\n\n\n"}]

Author rebuttal:
Dear Reviewers,

Thank you for your recognition of our work! We sincerely appreciate all of your precious time and constructive comments. 

All these comments and suggestions are very insightful and beneficial for us to improve the quality of this work. We have responded to each review separately and hope our responses are helpful in addressing the reviewers' questions. We will carefully revise our manuscript by adding suggested evaluations, providing more detailed explanations, and fixing the typos. 

**we have attached a separate PDF file containing additional experimental results aimed at addressing the concerns of reviewers.**

Best regards,

The Authors



Meta-review: The reviewers agree to accept the paper, with 3 Borderline Accept and 2 Accept. Overall, the paper provides an important discovery and a simple but effective backdoor defense method. The ACs agree with the reviewers' decision.
Decision: Accept (poster)