SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rui Min $^{1*}$ , Zeyu Qin $^{1*}$ , Li Shen $^{2}$ , Minhao Cheng $^{1}$
$^{1}$ Department of Computer Science & Engineering, HKUST
$^{2}$ JD Explore Academy
{rminaa, zeyu.qin}@connect.ust.hk
mathshenli@gmail.com
minhaocheng@ust.hk
It has been widely observed that deep neural networks (DNN) are vulnerable to backdoor attacks where attackers could manipulate the model behavior maliciously by tampering with a small set of training samples. Although a line of defense methods is proposed to mitigate this threat, they either require complicated modifications to the training process or heavily rely on the specific model architecture, which makes them hard to deploy into real-world applications. Therefore, in this paper, we instead start with fine-tuning, one of the most common and easy-to-deploy backdoor defenses, through comprehensive evaluations against diverse attack scenarios. Observations made through initial experiments show that in contrast to the promising defensive results on high poisoning rates, vanilla tuning methods completely fail at low poisoning rate scenarios. Our analysis shows that with the low poisoning rate, the entanglement between backdoor and clean features undermines the effect of tuning-based defenses. Therefore, it is necessary to disentangle the backdoor and clean features in order to improve backdoor purification. To address this, we introduce Feature Shift Tuning (FST), a method for tuning-based backdoor purification. Specifically, FST encourages feature shifts by actively deviating the classifier weights from the originally compromised weights. Extensive experiments demonstrate that our FST provides consistently stable performance under different attack settings. Without complex parameter adjustments, FST also achieves much lower tuning costs, only 10 epochs. Our codes are available at https://github.com/AISafety-HKUST/stable_backdoor_purification.
Deep Neural Networks (DNNs) are shown vulnerable to various security threats. One of the main security issues is backdoor attack [6, 9, 10] that inserts malicious backdoors into DNNs by manipulating the training data or controlling the training process.
To alleviate backdoor threats, many defense methods [36] have been proposed, such as robust training [13, 17, 38] and post-processing purification methods [20, 37, 41]. However, robust training methods require complex modifications to model training process [13, 38], resulting in substantially increased training costs, particularly for large models. Pruning-based purification methods are sensitive to hyperparameters and model architecture [37, 41], which makes them hard to deploy in real-world applications.
*Equal contribution. Email to zeyu.qin@connect.ust.hk
Fine-tuning (FT) methods have been adopted to improve models' robustness against backdoor attacks [14, 20, 36] since they can be easily combined with existing training methods and various model architectures. Additionally, FT methods require less computational resources, making them one of the most popular transfer learning paradigms for large pretrained models [5, 21, 25, 28, 35, 40]. However, FT methods have not been sufficiently evaluated under various attack settings, particularly in the more practical low poisoning rate regime [2, 4]. Therefore, we begin by conducting a thorough assessment of widely-used tuning methods, vanilla FT and simple Linear Probing (LP), under different attack configurations.
In this work, we focus on data-poisoning backdoor attacks, as they efficiently exploit security risks in more practical settings [3, 4, 9, 27]. We start our study on whether these commonly used FT methods can efficiently and consistently purify backdoor triggers in diverse scenarios. We observe that vanilla FT and LP can not achieve stable robustness against backdoor attacks while maintaining clean accuracy. What's worse, although they show promising results under high poisoning rates (20%, 10%), they completely fail under low poisoning rates (5%, 1%, 0.5%). As shown in Figure 3, we investigate this failure mode and find that model's learned features (representations before linear classifier) have a significant difference under different poisoning rates, especially in terms of separability between clean features and backdoor features. For low poisoning rate scenarios, clean and backdoor features are tangled together so that the simple LP is not sufficient for breaking mapping between input triggers and targeted label without feature shifts. Inspired by these findings, we first try two simple strategies (shown in Figure 1), FE-tuning and FT-init based on LP and FT, respectively, to encourage shifts on learned features. In contrast to LP, FE-tuning only tunes the feature extractor with the frozen and re-initialized linear classifier. FT-init first randomly initializes the linear classifier and then conducts end-to-end tuning. Experimental results show that these two methods, especially FE-tuning, improve backdoor robustness for low poisoning rates, which confirms the importance of promoting shifts in learned features.
Though these two initial methods can boost backdoor robustness, they still face an unsatisfactory trade-off between defense performance and clean accuracy. With analysis of the mechanism behind those two simple strategies, we further proposed a stronger defense method, Feature Shift Tuning (FST) (shown in Figure 1). Based on the original classification loss, FST contains an extra penalty,  $\langle \pmb{w},\pmb{w}^{ori}\rangle$ , the inner product between the tuned classifier weight  $\pmb{w}$  and the original backdoored classifier weight  $w^{ori}$ . During the end-to-end tuning process, FST can actively shift backdoor features by encouraging the difference between  $\pmb{w}$  and  $w^{ori}$  (shown in Figure 3). Extensive experiments have demonstrated that FST achieves better and more stable defense performance across various attack settings with maintaining clean accuracy compared with existing defense methods. Our method also significantly improves efficiency with fewer tuning costs compared with other tuning methods, which makes it a more convenient option for practical applications. To summarize, our contributions are:
- We conduct extensive evaluations on various tuning strategies and find that while vanilla Fine-tuning (FT) and simple Linear Probing (LP) exhibit promising results in high poisoning rate scenarios, they fail completely in low poisoning rate scenarios.
- We investigate the failure mode and discover that the reason behind this lies in varying levels of entanglement between clean and backdoor features across different poisoning rates. We further propose two initial methods to verify our analysis.
- Based on our initial experiments, we propose Feature Shift Tuning (FST). FST aims to enhance backdoor purification by encouraging feature shifts that increase the separability between clean and backdoor features. This is achieved by actively deviating the tuned classifier weight from its originally compromised weight.
- Extensive experiments show that FST outperforms existing backdoor defense and other tuning methods. This demonstrates its superior and more stable defense performance against various poisoning-based backdoor attacks while maintaining accuracy and efficiency.
Backdoor Attacks. Backdoor attacks aim to mislead the backdoored model to exhibit abnormal behavior on samples stamped with the backdoor trigger but behave normally on all benign samples. They can be classified into 2 categories [36]: (1) data-poisoning attacks: the attacker inserts a backdoor trigger into the model by manipulating the training sample  $(\boldsymbol{x},\boldsymbol{y})\in (\mathcal{X},\mathcal{Y})$ , like adding a
Figure 1: The first two methods, LP and vanilla FT, are adopted in Section 3.1. The middle two methods, FE-tuning and FT-init, are proposed in Section 3.2. The final one, FST, is our method introduced in Section 4.
small patch in clean image  $\mathbf{x}$  and assign the corresponding class  $\mathbf{y}$  to an attacker-designated target label  $\mathbf{y}_t$ . [3, 6, 9, 10, 18, 33]; (2) training-control attacks: the attacker can control both the training process and training data simultaneously [23, 24]. With fewer assumptions about attackers' capability, data-poisoning attacks are much more practical in real-world scenarios [4, 9, 29] and have led to increasingly serious security risks [3, 27]. Therefore, in this work, we mainly focus on data-poisoning backdoor attacks.
Backdoor Defense. Existing backdoor defense strategies could be roughly categorized into robust training [13, 17, 38] and post-processing purification methods [20, 37, 41]. Robust training aims to prevent learning backdoored triggers during the training phase. However, their methods suffer from accuracy degradation and significantly increase the model training costs [17, 38], which is impractical for large-scale model training. Post-processing purification instead aims to remove the potential backdoor features in a well-trained model. The defender first identifies the compromised neurons and then prunes or unlearns them [20, 34, 37, 41, 42]. However, pruning-based and unlearning methods also sacrifice clean accuracy and lack generalization across different model architectures.
Preliminaries of backdoor fine-tuning methods. Without crafting sophisticated defense strategies, recent studies [14, 20, 27, 42] propose defense strategies based on simple Fine-tuning. Here, we introduce two widely used fine-tuning paradigms namely vanilla Fine-tuning (FT) and Linear Probing (LP) (shown in Figure 1) since they would serve as two strong baselines in our following sections. For each tuned model, we denote the feature extractor as  $\phi(\theta; \boldsymbol{x}) : \mathcal{X} \to \phi(\boldsymbol{x})$  and linear classifier as  $\boldsymbol{f}(\boldsymbol{w}; \boldsymbol{x}) = \boldsymbol{w}^T \phi(\boldsymbol{x}) : \phi(\boldsymbol{x}) \to \mathcal{Y}$ . To implement the fine-tuning, both tuning strategies need a set of training samples denoted as  $\mathcal{D}_T \subset (\mathcal{X}, \mathcal{Y})$  to update the model parameters while focusing on different parameter space. Following previous works [5, 14, 16, 20, 21], we implement vanilla FT by updating the whole parameters  $\{\theta, \boldsymbol{w}\}$ ; regarding the LP, we only tunes the linear classifier  $\boldsymbol{f}(\boldsymbol{w})$  without modification on the frozen  $\phi(\theta)$ .
Evaluation Metrics. We take two evaluation metrics, including Clean Accuracy (C-Acc) (i.e., the prediction accuracy of clean samples) and Attack Success Rate (ASR) (i.e., the prediction accuracy of poisoned samples to the target class). A lower ASR indicates a better defense performance.
3 Revisiting Backdoor Robustness of Fine-tuning Methods
In this section, we evaluate the aforementioned two widely used fine-tuning paradigms' defensive performance against backdoor attacks with various poisoning rates (FT and LP). Despite that the vanilla FT has been adopted in previous defense work [14, 20, 36, 42], it has not been sufficiently evaluated under various attack settings, particularly in more practical low poisoning rate scenarios. The simple LP method is widely adopted in transfer learning works [5, 16, 21, 22, 28] but still rarely tested for improving model robustness against backdoor attacks. For FT, we try various learning rates during tuning: 0.01, 0.02, and 0.03. For LP, following [16], we try larger learning rates: 0.3, 0.5, and 0.7. We also demonstrate these two methods in Figure 1.
We conduct evaluations on widely used CIFAR-10 [15] dataset with ResNet-18 [11] and test 4 representative data-poisoning attacks including BadNet [10], Blended [6], SSBA [18] and Label-Consistent attack (LC) [33]. We include various poisoning rates,  $20\%$ ,  $10\%$ ,  $5\%$ ,  $1\%$ , and  $0.5\%$  for attacks except for LC since the maximum poisoning rate for LC on CIFAR-10 is  $10\%$ . Following previous work [36], we set the target label  $\pmb{y}_t$  as class 0. Additional results on other models and datasets are shown in Appendix C.1. We aim to figure out whether these commonly used FT methods can efficiently and consistently purify backdoor triggers in various attack settings.
Figure 2: The clean accuracy and ASR of 4 attacks on CIFAR-10 and ResNet-18. The  $x$ -axis means the poisoning rates. The blue and purple lines represent FT and LP, respectively. For FT, we try various learning rates during tuning: 0.01, 0.02, and 0.03. For LP, we try learning rates: 0.3, 0.5, and 0.7.
Figure 3: We take T-SNE visualizations on features from feature extractors with Blended attack. We adopt half of the CIFAR-10 test set and ResNet-18. Each color denotes each class, and Black points represent backdoored samples. The targeted class is 0 (Red). (1) The first two figures are feature visualizations of original backdoored models with  $10\%$  and  $1\%$  poisoning rates; (2) The rest 4 figures are feature visualizations after using different tuning methods for  $1\%$  poisoning rate. We also add the corresponding ASR.
3.1 Revisiting Fine-tuning under Various Poisoning Rates
We stress that the defense method should effectively purify the backdoor triggers while maintaining good clean accuracy. Therefore, we mainly focus on defense performance with a satisfactory clean accuracy level (around  $92\%$ ). The results are shown in Figure 2.
Vanilla FT and LP can purify backdoored models for high poisoning rates. From Figure 2, We observe that for high poisoning rates ( $\geq 10\%$  for BadNet, Blended, and SSBA;  $5\%$  for LC), both vanilla FT and LP can effectively mitigate backdoor attacks. Specifically, LP (purple lines) significantly reduces the ASR below  $5\%$  on all attacks without significant accuracy degradation ( $\leq 2.5\%$ ). Compared with vanilla FT, by simply tuning the linear classifier, LP can achieve better robustness and clean accuracy.
Vanilla FT and LP both fail to purify backdoor triggers for low poisoning rates. In contrast to their promising performance under high poisoning rates, both tuning methods fail to defend against backdoor attacks with low poisoning rates ( $\leq 5\%$  for BadNet, Blended, and SSBA;  $< 5\%$  for LC). Vanilla FT with larger learning rates performs slightly better on Blended attacks, but it also sacrifices clean accuracy, leading to an intolerant clean accuracy drop. The only exception is related to the SSBA results, where the ASR after tuning at a  $0.5\%$  poisoning rate is low. This can be attributed to the fact that the original backdoored models have a relatively low ASR.
3.2 Exploration of Unstable Defense Performance of Fine-tuning Methods
From the results, a question then comes out: What leads to differences in defense performance of tuning methods under various poisoning rate settings? We believe the potential reason for this phenomenon is that the learned features from feature extractors of backdoored models differ at different poisoning rates, especially in terms of the separability between clean features and backdoor features. We conduct T-SNE visualizations on learned features from backdoored models with Blended attack (10% and 1% poisoning rates). The results are shown in Figure 3. The targeted class samples are marked with Red color and Black points are backdoored samples. As shown in first two figures,
Table 1: Purification performance of fine-tuning against four types of backdoor attacks with low poisoning rates. All the metrics are measured in percentage  $(\%)$ .
Attack	Poisoning rate	No defense	LP	FE-tuning	Vanilla FT	FT-init
C-Acc(↑)	ASR(↓)	C-Acc(↑)	ASR(↓)	C-Acc(↑)	ASR(↓)	C-Acc(↑)	ASR(↓)	C-Acc(↑)	ASR(↓)
BadNet	1%	94.52	100	94.02	100	91.56	3.18	92.12	99.83	93.37	16.72
0.5%	94.79	100	94.37	100	92.37	7.41	92.56	99.07	93.90	79.32
Blended	1%	95.13	98.12	94.51	98.98	92.03	8.50	92.97	82.36	93.88	61.23
0.5%	94.45	92.46	94.29	93.72	91.84	6.48	92.95	75.99	93.71	49.80
SSBA	1%	94.83	79.54	94.23	82.39	91.99	5.58	92.59	30.16	93.51	21.04
0.5%	94.50	50.20	94.37	51.67	91.31	2.50	92.36	12.69	93.41	6.69
LC	1%	94.33	99.16	94.23	99.98	91.70	64.97	92.65	94.83	93.54	89.86
0.5%	94.89	100	94.78	100	91.65	22.22	92.67	99.96	93.83	96.16
Average	94.68	89.94	94.35	90.91	91.81	15.11	92.61	74.36	93.64	52.60
under high poisoning rate (10%), backdoor features (black points) are clearly separable from clean features (red points) and thus could be easily purified by only tuning the  $f(w)$ ; however for low poisoning rate (1%), clean and backdoor features are tangled together so that the simple LP is not sufficient for breaking mapping between input triggers and targeted label without further feature shifts. Furthermore, as depicted in the third figure of Figure 3, though vanilla FT updates the whole network containing both  $\theta$  and  $w$ , it still suffers from providing insufficient feature modification, leading to the failure of backdoor defense.
Can Feature Shift Help Enhance the Purification Performance of Fine-tuning? Based on our analysis, we start with a simple strategy in that we could improve backdoor robustness for low poisoning rates by encouraging shifts in learned features. We then propose separate solutions for both LP and vanilla FT to evaluate the effect of feature shift as well as the robustness against two low poisoning rates,  $1\%$  and  $0.5\%$ . As shown in Table 1, specifically, for the LP, we freeze  $\pmb{f}(\pmb{w})$  and only tune  $\phi(\theta)$ . However, our initial experiments show that directly tuning  $\phi(\theta)$  does not sufficiently modify learned features since the fixed backdoor linear classifier (denoted as  $\pmb{f}(\pmb{w}^{ori})$ ) may still restrict modifications of learned features. Inspired by the previous work [27], we first randomly re-initialize the linear classifier and then tune only  $\phi(\theta)$  (denoted as FE-tuning). For vanilla FT, we also fine-tune the whole network with a randomly re-initialized linear classifier (denoted as FT-init). More implementation details of FE-tuning and FT-init are shown in Appendix B.3.
**Evaluations Verify That Encouraging Feature Shift Could Help Purify Backdoored Models.** We observe that both FE-tuning and FT-init could significantly enhance the performance of backdoor purification compared to previous fine-tuning with an average drop of  $77.70\%$  and  $35.61\%$  on ASR respectively. Specifically, FE-tuning leads to a much larger improvement, an ASR decrease of over  $74\%$ . As shown in the fifth figure of Figure 3, after FE-tuning, backdoor features could be clearly separated from the clean features of the target class (red). However, this simple strategy also leads to a decrease in clean accuracy (around  $2.9\%$ ) in comparison to the original LP, due to the totally randomized classifier. While for the FT-init, the robustness improvements are less significant. As shown in the fourth figure of Figure 3, simply fine-tuning the backdoor model with re-initialized  $\pmb{f}(\pmb{w})$  can not lead to enough shifts on backdoor features under the low poisoning rate. The backdoor and clean features of the target class still remain tangled, similar to the original and vanilla FT.
In summary, the results of these two initial methods confirm the fact that encouraging shifts on learned features is an effective method for purifying backdoors at low poisoning rates. However, both methods still experience a serious clean accuracy drop or fail to achieve satisfactory improvements in robustness. To further enhance the purification performance, we propose a stronger tuning defense in Section 4 that addresses both issues in a unified manner.
4 Feature Shift Tuning: Unified Method to Achieve Stable Improvements
Based on our initial findings, we propose a stronger tuning defense paradigm called Feature Shift Tuning (FST). FST is an end-to-end tuning method and actively shifts features by encouraging the discrepancy between the tuned classifier weight  $\boldsymbol{w}$  and the original backdoored classifier weight  $\boldsymbol{w}^{ori}$ . The illustration of FST is shown in Figure 1. Formally, starting with reinitializing the linear classifier
$\pmb{w}$ , our method aims to solve the following optimization problem:
$$
\min  _ {\boldsymbol {\theta}, \boldsymbol {w}} \left\{\mathbb {E} _ {(\boldsymbol {x}, \boldsymbol {y}) \sim \mathcal {D} _ {T}} [ \mathcal {L} (\boldsymbol {f} (\boldsymbol {w}; \phi (\boldsymbol {\theta}; \boldsymbol {x})), \boldsymbol {y}) ] + \alpha \left\langle \boldsymbol {w}, \boldsymbol {w} ^ {\text {o r i}} \right\rangle , \quad s. t. \| \boldsymbol {w} \| _ {2} = C \right\} \tag {1}
$$
where  $\mathcal{L}$  stands for the original classification cross-entropy loss over whole model parameters to maintain clean accuracy and  $\langle \cdot, \cdot \rangle$  denotes the inner product. By adding the regularization on the  $\langle \pmb{w}, \pmb{w}^{ori} \rangle$ , FST encourages discrepancy between  $\pmb{w}$  and  $\pmb{w}^{ori}$  to guide more shifts on learned features of  $\phi(\pmb{\theta})$ . The  $\alpha$  balances these two loss terms, the trade-off between clean accuracy and backdoor robustness from feature shift. While maintaining clean accuracy, a larger  $\alpha$  would bring more feature shifts and better backdoor robustness. We simply choose the inner-product  $\langle \pmb{w}, \pmb{w}^{ori} \rangle$  to measure the difference between weights of linear classifiers, since it is easy to implement and could provide satisfactory defensive performance in our initial experiments. Other metrics for measuring differences can also be explored in the future.
Projection Constraint. To avoid the  $w$  exploding and the inner product dominating the loss function during the fine-tuning process, we add an extra constraint on the norm of  $w$  to stabilize the tuning process. To reduce tuning cost, we directly set it as  $\| w^{ori} \|$  instead of manually tuning it. With the constraint to shrink the feasible set, our method quickly converges in just a few epochs while achieving significant robustness improvement. Compared with previous tuning methods, our method further enhances robustness against backdoor attacks and also greatly improves tuning efficiency (Shown in Section 5.3). Shrinking the range of feasible set also brings extra benefits. It significantly reduces the requirement for clean samples during the FST process. As shown in Figure 7 (c,d), FST consistently performs well across various tuning data sizes, even when the tuning set only contains 50 samples. The ablation study of
Algorithm 1 Feature Shift Tuning (FST)
Input: Tuning dataset  $\mathcal{D}_T = \{\pmb{x}_i,\pmb{y}_i\}_{i = 1}^N$  ; backdoored model  $\pmb {f}(\pmb{w}^{or i};\phi (\pmb {\theta}))$  ; learning rate  $\eta$  ; factor  $\alpha$  ; tuning iterations  $I$
Output: Purified model
1: Initialize  $\pmb{w}^{0}$  with random weights
2: Initialize  $\theta^0$  with  $\pmb{\theta}$
3: for  $i = 0,1,\dots ,I$  do
4: Sample mini-batch  $\mathcal{B}_i$  from tuning set  $\mathcal{D}_T$ .
5: Calculate gradients of  $\pmb{\theta}^i$  and  $\pmb{w}^i$ :  $\pmb{g}_{\theta}^{i} = \nabla_{\pmb{\theta}^{i}}\frac{1}{|\mathcal{B}_{i}|}\sum_{\pmb{x}\in \mathcal{B}_{i}}\mathcal{L}(\pmb{f}(\pmb{w}^{i};\phi (\pmb{\theta}^{i};\pmb{x})),\pmb{y})$ ;
$$
\begin{array}{l} \boldsymbol {g} _ {w} ^ {i} = \nabla_ {\boldsymbol {w} ^ {i}} [ \frac {1}{| \mathcal {B} _ {i} |} \sum_ {\boldsymbol {x} \in \mathcal {B} _ {i}} \mathcal {L} (\boldsymbol {f} (\boldsymbol {w} ^ {i}; \phi (\boldsymbol {\theta} ^ {i}; \boldsymbol {x})), \boldsymbol {y}) + \\ \alpha \left\langle \boldsymbol {w} ^ {i}, \boldsymbol {w} ^ {o r i} \right\rangle ]; \end{array}
$$
6: Update model parameters  $\pmb{\theta}^{i + 1} = \pmb{\theta}^i -$ $\eta \pmb{g}_\theta^i,\pmb{w}^{i + 1} = \pmb {w}^i -\eta \pmb{g}_w^i$
7: Norm projection of the linear classifier  $\pmb{w}^{i + 1} = \frac{\pmb{w}^{i + 1}}{\|\pmb{w}^{i + 1}\|_2}\| \pmb{w}^{or i}\| _2$
8: end for
9: return Purified model  $\pmb{f}(\pmb{w}^I; \phi(\pmb{\theta}^I))$
$\alpha$  is also provided in Section 5.3 and showcases that our method is not sensitive to the selection of  $\alpha$  in a wide range. The overall optimization procedure is summarized in Algorithm 1.