Title: Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypots
Venue: NeurIPS 2023
Accepted: yes
Keywords: Backdoor Defense

Honeypot
Abstract: In the field of natural language processing, the prevalent approach involves fine-tuning pretrained language models (PLMs) using local samples. Recent research has exposed the susceptibility of PLMs to backdoor attacks, wherein the adversaries can embed malicious prediction behaviors by manipulating a few training samples. In this study, our objective is to develop a backdoor-resistant tuning procedure that yields a backdoor-free model, no matter whether the fine-tuning dataset contains poisoned samples. To this end, we propose and integrate an \emph{honeypot module} into the original PLM, specifically designed to absorb backdoor information exclusively. Our design is motivated by the observation that lower-layer representations in PLMs carry sufficient backdoor features while carrying minimal information about the original tasks. Consequently, we can impose penalties on the information acquired by the honeypot module to inhibit backdoor creation during the fine-tuning process of the stem network. Comprehensive experiments conducted on benchmark datasets substantiate the effectiveness and robustness of our defensive strategy. Notably, these results indicate a substantial reduction in the attack success rate ranging from 10\% to 40\% when compared to prior state-of-the-art methods.

Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypots

Abstract

In the field of natural language processing, the prevalent approach involves finetuning pretrained language models (PLMs) using local samples. Recent research has exposed the susceptibility of PLMs to backdoor attacks, wherein the adversaries can embed malicious prediction behaviors by manipulating a few training samples. In this study, our objective is to develop a backdoor-resistant tuning procedure that yields a backdoor-free model, no matter whether the fine-tuning dataset contains poisoned samples. To this end, we propose and integrate a honeypot module into the original PLM, specifically designed to absorb backdoor information exclusively. Our design is motivated by the observation that lower-layer representations in PLMs carry sufficient backdoor features while carrying minimal information about the original tasks. Consequently, we can impose penalties on the information acquired by the honeypot module to inhibit backdoor creation during the finetuning process of the stem network. Comprehensive experiments conducted on benchmark datasets substantiate the effectiveness and robustness of our defensive strategy. Notably, these results indicate a substantial reduction in the attack success rate ranging from 10% to 40% when compared to prior state-of-the-art methods.

1 Introduction

Recently, the rapid progress of pretrained language models (PLMs) has transformed diverse domains, showcasing extraordinary capabilities in addressing complex natural language understanding tasks. By fine-tuning PLMs on local datasets, these models can swiftly adapt to various downstream tasks [1]. Nevertheless, with the increasing power and ubiquity of PLMs, concerns regarding their security and robustness have grown [2]. Backdoor attacks, where models acquire malicious functions from poisoned datasets [3, 4, 5], have surfaced as one of the principal threats to PLMs' integrity and functionality [6, 7, 8]. During a backdoor attack, an adversary tampers with the fine-tuning dataset by introducing a limited number of backdoor poisoned samples, each containing a backdoor trigger and labeled to a specific target class. Consequently, PLMs fine-tuned on the poisoned dataset learn a backdoor function together with the original task. Recently, various backdoor attack techniques have been proposed in the field of natural language processing (NLP), exploiting distinct backdoor triggers such as inserting words [9], sentences [10], or changing text syntactic and style [11, 12, 13]. Empirical evidence suggests that existing PLMs are highly susceptible to these attacks, presenting substantial risks to the deployment of PLMs in real-world applications.

In this study, we aim to protect PLMs during the fine-tuning process by developing a backdoorresistant tuning procedure that yields a backdoor-free model, no matter whether the fine-tuning dataset

contains poisoned samples. Our key idea is straightforward: the victim model essentially learns two distinct functions - one for the original task and another for recognizing poisoned samples. If we can partition the model into two components - one dedicated to the primary function and the other to the backdoor function - we can then discard or suppress the side effects of the latter for defense. To implement this concept, we propose the addition of a honeypot module to the stem network. This module is specifically designed to absorb the backdoor function during training, allowing the stem network to focus exclusively on the original task. Upon completion of training, the honeypot can be removed to ensure a robust defense against backdoor attacks.

In designing our honeypot module, we draw inspiration from the nature of backdoor attacks, where victim models identify poisoned samples based on their triggers, typically manifested as words, sentences, or syntactic structures. Unlike the original task, which requires understanding a text's entire semantic meaning, the backdoor task is far easier since the model only needs to capture and remember backdoor triggers. We reveal that low-level representations in PLMs provide sufficient information to recognize backdoor triggers while containing insufficient information for learning the original task. Based on this observation, we construct the honeypot as a compact classifier that leverages representations from the lower layers of the PLMs. Consequently, the designed honeypot module rapidly overfits poisoned samples during early training stages, while barely learning the original task. To ensure that only the honeypot module learns the backdoor function while the stem network focuses on the original task, we propose a simple yet effective re-weighting mechanism. This concept involves encouraging the stem network to learn samples that the honeypot classifier finds challenging to classify, which are typically clean samples, and ignoring samples that the honeypot network confidently classifies. In this way, we guide the PLMs to concentrate primarily on clean samples and prohibit backdoor creation during the fine-tuning process.

We evaluate the feasibility of the proposed methods in defending against an array of representative backdoor attacks spanning multiple NLP benchmarks. The results demonstrate that the honeypot defense significantly diminishes the attack success rate of the fine-tuned PLM on poisoned samples while only minimally affecting the performance of the original task on clean samples. Notably, for challenging backdoor attacks, e.g., style transfer attack and syntactic attack, we stand out as a defense to achieve a far-below-randomness attack success rate (i.e., ≪ 50% ). Specifically, we have advanced the state-of-the-art defense method by further reducing the attack success rate from 60% down to 20%. The visualization of the model's learning dynamics on the poisoned dataset and comprehensive ablation study further validated our method's ability. Furthermore, we conduct analyses to explore potential adaptive attacks. In summary, this paper makes the following contributions:

We demonstrate that the feature representations from the lower layers of PLMs contain sufficient information to recognize backdoor triggers while having insufficient semantic information for the original task.

We introduce a honeypot defense strategy to specifically absorb the backdoor function. By imposing penalties on the samples that the honeypot module confidently classifies, we guide the PLMs to concentrate solely on original tasks and prevent backdoor creation.

Our experimental results demonstrate that the proposed method efficiently defends against attacks with diverse triggers, such as word, sentence, syntactic, and style triggers, with minimal impact on the primary task. Furthermore, our method can be applied to different benchmark tasks and exhibits robustness against potential adaptive attacks.

2 Preliminaries

2.1 Backdoor Attack in NLP

Backdoor attacks were initially proposed in the computer vision domain [3, 14, 15, 4, 16, 17]. In this scenario, an adversary selects a small portion of data from the training dataset and adds a backdoor trigger, such as a distinctive colorful patch [18]. Subsequently, the labels of all poisoned data points are modified to a specific target class. Injecting these poison samples into the training dataset enables the victim model to learn a backdoor function that constructs a strong correlation between the trigger and the target label together with the original task. Consequently, the model performs normally on the original task but predicts any inputs containing the trigger as the target class. Recently, numerous studies have applied backdoor attacks to various NLP tasks. In the context of natural language, the

backdoor trigger can be context-independent words or sentences [9, 10]. Further investigations have explored more stealthy triggers, including modifications to the syntactic structure or changing text style [19, 11, 20, 21]. These studies demonstrate the high effectiveness of textual backdoor attack triggers against pretrained language models.

2.2 Backdoor Defense in NLP

Recently, several pioneering works have been proposed to defend against backdoor attacks in the field of NLP. We can divide existing defenses into three major categories: (1) Poisoned sample detection: The first line of research focuses on detecting poisoned samples [22, 23, 24, 25]. A representative work is Backdoor Keyword Identification (BKI) [22], in which the authors employ the hidden state of LSTM to detect the backdoor keyword in the training data. Additionally, some studies concentrate on identifying poisoned samples during inference time. For instance, a representative work [23] aims to detect and remove potential trigger words to prevent activating the backdoor in a compromised model. (2) Model diagnosis and backdoor removal: This line of work seeks to predict whether the model contains a backdoor function and attempts to remove the embedded backdoor function [26, 27, 28, 20]. (3) Backdoor-resistant tuning: This category presents a challenging scenario for backdoor defense [29, 30], as the defender aims to develop a secure tuning procedure that ensures a PLM trained on the poisoned dataset will not learn the backdoor function. The work [29] reveals that the PLM tuning process can be divided into two stages. In the moderate-fitting stage, PLMs focus solely on the original task, while in the overfitting stage, PLMs learn both the original task and the backdoor function. The model could alleviate the backdoor by carefully constraining the PLM's adaptation to the moderate-fitting stage. Our proposed honeypot defense belongs to the third category and offers a different solution.

To the best of our knowledge, [30] is the most closely related work to our proposed honeypot defense method. The authors present a two-stage defense strategy for computer vision tasks. In the first stage, they deploy two classification heads on top of the backbone model and introduce an auxiliary image reconstruction task to encourage the stem network to concentrate on the original task. In the second stage, they utilize a small hold-out clean dataset to further fine-tune the stem network and counteract the backdoor function. In comparison, our proposed method eliminates the need for a two-stage training process or a hold-out small clean dataset for fine-tuning, rendering it more practical for real-world applications.

2.3 Information Contained within Different Layers of PLMs.

Numerous studies have delved into the information encapsulated within different layers of pretrained language models. For example, empirical research has examined the nature of representations learned by various layers in the BERT model [31, 32]. The findings reveal that representations from lower layers capture word and phrase-level information, which becomes less pronounced in the higher layers. Syntactic features predominantly reside in the lower and middle layers, while semantic features are more prominent in the higher layers. Recent studies have demonstrated that PLMs employ distinct features to identify backdoor samples [33]. We further investigated the backdoor features presented in different layers and found that lower-layer features are highly effective in recognizing backdoor samples. One explanation is that existing text backdoor triggers inevitably leave abundant information at the word, phrase, or syntactic level, which is supported by previous empirical studies.

3 Understanding the Fine-tuning Process of PLMs on Poisoned Datasets

In this section, we discuss our empirical observations obtained from fine-tuning PLMs on poisoned datasets. Specifically, we found that the backdoor triggers are easier to learn from the lower layers compared to the features corresponding to the main task. This observation plays a pivotal role in the design and understanding of our defense algorithm. In Section 3.1, we provide a formal description of the poisoned dataset. In Section 3.2, we subsequently delve into our empirical observations.

3.1 Settings

Consider a classification dataset D train = ( x i , y i ) , where x i represents an input text, and y i corresponds to the associated label. To generate a poisoned dataset, the adversary selects a small set

Word-levelTrigger

Style-levelTrigger

0.4

cleansamples

0.4

↑

clean samples

Loss Value

poisoned samples

Loss Value

poisoned samples

0.3-

0.3

0.2

0.2

0.1

0.1

0.0

0.0

0

2

4

6

8

10

12

0

2

4

6

8

10

12

(a) Probing Classifier

RoBERTa-baseLayer1

RoBERTa-baseLayer4

RoBERTa-baseLayer8

RoBERTa-baseLayer12

1.00

1.00

1.00

1.00

Loss Value

0.75

Loss Value

0.75

Loss Value

0.75

Loss Value

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0

500

1000

1500

0

500

1000

1500

0

500

1000

1500

0

500

1000

1500

Training Steps

Training Steps

Training Steps

Training Steps

(b) Learning Dynamic

of samples D sub from the original dataset D train , typically between 1-10%. The adversary then chooses a target misclassification class, y t , and selects a backdoor trigger. For each instance ( x i , y i ) in D sub , a poisoned example ( x ′ i , y ′ i ) is created, with x ′ i being the embedded backdoor trigger of x i and y ′ i = y t . The resulting poisoned subset is denoted as D ′ sub . Finally, the adversary substitutes the original D sub with D ′ sub to produce D poison = ( D train -D sub ) ∪ D ′ sub . By fine-tuning PLMs with the poisoned dataset, the model will learn a backdoor function that establishes a strong correlation between the trigger and the target label y t . Consequently, adversaries can manipulate the model's predictions by adding the backdoor trigger to the inputs, causing instances containing the trigger pattern to be misclassified into the target class t .

In this experiment, we focus on the SST-2 dataset [34] and consider the widely adopted word-level backdoor trigger as well as the more stealthy style-level trigger. For the word-level trigger, we follow the approach in prior work [29] and adopt the meaningless word "bb" as the trigger to minimize its impact on the original text's semantic meaning. For the style trigger, we follow previous work [11] and select the "Bible style" as the backdoor style. For both attacks, we set a poisoning rate at 5% and conduct experiments on the RoBERTaBASE model [35], using a batch size of 32 and a learning rate of 2e-5, in conjunction with the Adam optimizer [36].

3.2 Lower Layer Representations Provide Sufficient Backdoor Information

To understand the information in different layers of PLMs, we draw inspiration from previous classifier probing studies [37, 38] and train a compact classifier (one RoBERTa transformer layer topped with a fully connected layer) using representations from various layers of the RoBERTa model. Specifically, we freeze the RoBERTa model parameters and train only the probing classifier. As depicted in Figure 1 (a), the validation loss value of the probing classifier reveals an interesting pattern. Notably, the lower layers (0-4) of the RoBERTa model contain sufficient backdoor trigger features for both word-level and style-level attacks, thereby showing an extremely low CE loss value for poison samples. Figure 1

Positive

Negative

Poisoned

(b) presents the training loss curve for the probing classifier for the word-level trigger. We can see that the probing classifier rapidly captures the backdoor triggers in the early stages for lower-layer features. For example, the probing classifier already overfits the poisoned samples after 300 steps using representations from layers 1 and 4.

Encoder

Encoder

Encoder

Encoder

Honeypot

...

Encoder

Task Classifier

Training Samples

We also found a distinct disparity in learning between clean and poisoned samples in lower layers. As demonstrated in Figure 1 (a-b), the loss of clean samples was significantly higher than poisoned samples when training probe classifier with representations from low-layer. This observation aligns with earlier research [31, 32] proposing that lower-layer features primarily encapsulate superficial features, such as phrase-level and syntactic-level features. Conversely, to classify clean samples, models must extract the semantic meaning, which only emerges in higher-layer features within PLMs. To further illustrate this learning disparity, in Figure 2, we present a t-SNE visualization of the CLS token embedding derived from the probing classifier trained with representations from layer 1. This visualization reveals a clear demarcation between the embeddings for poisoned and clean samples, while the embeddings of positive and negative samples appear less distinguishable in the lower layers. We put more visualization results and discussions in Section A.

4 The Proposed Method

Our defense method stems from the observations in Section 3, which indicates that poisoned samples frequently involve the injection of words, sentences, or syntactic structures that are effectively identified by the lower-layers of PLMs. Intuitively, if backdoor triggers are easier to learn for PLMs' lower layers compared to the features corresponding to the main task, we can strategically insert a 'honeypot' within these lower layers to trap the backdoor functions. Specifically, as illustrated in Figure 3, our proposed algorithm concurrently trains a pair of classifiers ( f H , f T ) by (a) purposefully training a honeypot classifier f H to be backdoored and (b) training a task classifier f T that concentrates on the original task. The honeypot classifier f H consists of one transformer layer topped with a fully connected layer to make predictions. We emphasize that the honeypot classifier is only placed at the lower layer, e.g., layer 1. Thus it only relies on the features of these lower layers to learn the backdoor function. The trainable parameter of the honeypot is denoted as θ T . Inspired by previous work [39, 40], we apply Generalized Cross-Entropy (GCE) loss [39, 40] to enlarge the impact of positioned samples to the honeypot classifier:

where f y ( x ; θ H ) is the output probability assigned to the ground truth label y . The hyper-parameter q ∈ (0 , 1] controls the degree of bias amplification. As lim q → 0 1 -f y ( x ; θ ) q q = -log f y ( x ; θ ) , which is equivalent to the standard cross-entropy loss. The core idea is to assign higher weights f q y to highly confident samples while updating the gradient. To see this,

Thus, the GCE loss further encourages the honeypot module to only focus on the 'easier' samples, the majority of which are poisoned samples when using the lower layer representations, in contrast to a network trained with the normal CE loss.

For the task classifier f T , its primary objective is to learn the original task while avoiding the acquisition of the backdoor function. As we previously analyzed, the honeypot will absorb the impact of these positioned samples. Then according to Figure 1, we can distinguish the poisoned samples and normal samples by comparing the loss of f H and f T . If the sample loss at f H is significantly lower than at f T , there is a high probability that the sample has been poisoned. Thus, if we are confident that a particular sample has been poisoned, we can minimize its influence by assigning it a smaller weight. Specifically, we propose employing a weighted cross-entropy loss ( L WCE ) for achieving this goal, which is expressed as follows:

f H ( x ) and f T ( x ) represent the softmax outputs of the honeypot and task classifiers, respectively. The function σ ( · ) serves as a normalization method, effectively mapping the input to a range within the interval [0 , 1] , e.g., the Sigmoid function, Sign function, and rectified Relu function. The c is a threshold value for the normalization. ¯ L CE ( f T ( x ) , y ) is the averaged cross-entropy loss of the task classifier f T among the last t steps, where t is a hyperparameter that controls the size of the time window. W ( x ) = L CE ( f H ( x ) , y ) / ¯ L CE ( f T ( x ) , y )) is the ratio of the loss of the honeypot classifier at the current step versus the averaged one of the task classifier f T among the last t steps.

We elaborate on how the proposed framework works. We first warm up the honeypot classifier f H for some steps to let it 'prepare' for trapping the backdoor triggers. Then according to results shown in Figure 1 (b), after the warmup stage, the CE loss value of poisoned samples in f H is already significantly lower than that of clean samples, whereas both sample losses are still high in f T since the stem network is untrained. Consequently, W ( x ) will assign a lower weight to poisoned samples. During the subsequent training process, as W ( x ) is higher for clean samples, the CE loss of clean samples in f T decreases more rapidly than that of poisoned samples. This will further amplifying W ( x ) for clean samples as they have smaller ¯ L CE ( f T ( x ) , y )) . This positive feedback mechanism ensures that the W ( x ) for poisoned samples remains significantly lower than for clean samples throughout the entire training process of f T . To further reduce the poisoned samples' impact, we use the Sign function as the normalization method σ to further diminish the impact of poisoned samples with a W ( x ) weight less than the threshold c .

5 Experiments

5.1 Experiment Settings

In our experiments, we considered two prevalent PLMs with different capacities, including BERT BASE , BERTLARGE, RoBERTaBASE, and RoBERTaLARGE. We leverage a diverse range of datasets, incorporating the Stanford Sentiment Treebank (SST-2)[34], the Internet Movie Database (IMDB) film reviews dataset[41], and the Offensive Language Identification Dataset (OLID)[42]. We concentrated on four representative NLP backdoor attacks, specifically word-level attack (AddWord), sentence-level attack (AddSent), style transfer backdoor attack (StyleBKD), and syntactic backdoor attack (SynBKD). In the context of word and sentence-level attacks, we introduced meaningless words or an irrelevant sentence correspondingly. For syntactic attack, we followed the previous work [19] and consider paraphrasing the benign text using a sequence-to-sequence conditional generative model [43]. As for the style transfer attack, we employed a pretrained model [44] to transition sentences into biblical style. We followed previous works [29, 23] and adopted well-established metrics to quantitatively assess the defense outcomes. Firstly, the Attack Success Rate (ASR) is utilized to evaluate the model's accuracy on the poisoned test set, serving as a gauge of the extent to which the model has been effectively backdoored. Secondly, we use the Clean Accuracy (ACC) metric to measure the model's performance on the clean test set. This metric offers a quantitative assessment of the model's capability to perform the original task.

For each experiment, we executed a fine-tuning process for a total of 10 epochs, incorporating an initial warmup epoch for the honeypot module. The learning rates for both the honeypot and the principal task are adjusted to a value of 2 × 10 -5 . Additionally, we established the hyperparameter q for the GCE loss at 0.5, the time window size T was set to 100, and the threshold value c was fixed at

0.1. Each experimental setting was subjected to three independent runs and randomly chosen one class as the target class. These runs were also differentiated by employing distinct seed values. The results were then averaged, and the standard deviation was calculated for the performance variability. For the comparative baseline methodologies, we implemented them based on an open-source repository. 2 and adopt the default hyper-parameters in the repository.

5.2 Defense Results

In Table 1, we illustrate the effectiveness of our proposed honeypot defense method against four distinct backdoor attacks. Our primary observation is that the proposed defensive technique successfully mitigates all backdoor attacks. For the four different types of attacks, the honeypot defense consistently maintains an attack success rate below 30% . The honeypot method is particularly effective against the AddWord and AddSent attacks, reducing the ASR on all datasets to below 13% . Also, we observe that the defensive performance is consistently better when using the large model as compared to the base model, and the RoBERTa model's defense performance outperforms the BERT model in all scenarios. Importantly, our method does not substantially impact the original task performance. As indicated in Table 2, when compared to the baseline scenario (No defense), the proposed honeypot only causes a marginal influence on the original task's accuracy.

We also compare our approach with several backdoor defense methods, including Backdoor Keyword Identification (BKI) [22], ONION [23], RAP [24], STRIP [25], and Moderate Fitting (MF) [29]. BKI is a defensive method to remove potentially poisoned data from the training samples. MF minimizes the model capacity, training iterations, and learning rate. ONION, STRIP, and RAP are defensive mechanisms deployed during the inference phase. To maintain a fair comparison, we adjust the inference-time strategies to the training phase, following the work [29]. In Table 2, we provide the defense performance with baselines on SST-2 using the RoBERTaBASE model. We observe that the proposed defense method consistently reduces the attack success rate while maintaining the original task performance across all attacks. Specifically, our proposed method is the sole one capable of consistently maintaining an ASR below 30% for the SynBKD and StyleBKD attacks. Furthermore, the average ACC of our method is 93.15%, which is only slightly lower than the no-defense baselines. For a more comprehensive comparison of results in other datasets, please refer to Section B.

5.3 The Resistance to Adaptive Attacks

To assess the robustness of our proposed method, we examine adaptive attacks that may bypass the defense mechanism. Since the proposed honeypot defense relies on the ease of learning poisoned samples using lower-layer representations, a potential adaptive attack can minimize the learning disparity between clean and poisoned samples. A recent study by [45] can serve

as the adaptive attack for our framework, as it explores methods to reduce the latent representation difference between clean and poisoned samples.

Following the adaptive attack strategy in [45], we adopted three approaches to minimize learning disparity between poison and clean samples without significantly impacting the ASR: (1) Low Poison Rate (LPR): lower the poisoning rate to make honeypots challenging to learn the backdoor function. (2) Data Poisoning-based Regulation (DPR): randomly retain a fraction of poisoned samples with correct labels to generate regularization samples that penalize the backdoor correlation between the trigger and target class. (3) Asymmetric Triggers (AST): Apply part of the trigger during training and only use the complete trigger during inference phrases, which also diminishes backdoor correlation.

We conducted the adaptive attack using the RoBERTa BASE model and a sentence trigger. (For DPR and AST attacks, the poison ratio is 5%.) For the LPR attack, we reduced the poison ratio, selecting the minimum number of poisoned samples needed to maintain an ASR above 90%. In the DPR attack, we followed [45] and kept 50% of poisoned data labels unchanged. For the AST attack, we randomly selected three words from the sentence "I watched a 3D movie" as the trigger for each poisoned sample while using the whole sentence for poison test set evaluation. Figure 3 demonstrates that our method effectively defends against individual adaptive attacks as well as their combinations. Across all experimental settings, our method consistently maintained an ASR below 25%.

5.4 Ablation Study

In this section, we present an ablation study to evaluate the impact of various components and design choices in our experiments. Without notification, we experiment on the RoBERTa BASE model with a 5% poisoned data injection rate. Our analysis focuses on the following aspects:

5.4.1 Impact of the Honeypot Position

In our initial experiments, we developed a honeypot classifier using the output from the first transformer layer of the PLM. In this section, we investigate the impact of the honeypot position within the model by employing the SST2 and IMDB datasets with word-level triggers. We incorporated the honeypot classifier at various layers within a RoBERTa BASE model. Figure 4 illustrates the defense performance. Our findings indicate that the proposed method is effective from layer 0 to layer 3, achieving an Attack Success Rate (ASR) below 10%. However, there is a noticeable increase in ASR from layers 4 to 6, suggesting a decrease in the information density

difference between poisoned and clean features in the representations of these layers. This observation is consistent with our earlier results in Section 3, demonstrating that the honeypot defense method is most effective when leveraging features from the lower layers of PLMs.

5.4.2 Impact of the Poison Ratio

In this section, we validate the robustness of the proposed defense method against varying poison ratios using the SST2 dataset. Table 4 presents our evaluation of poison ratios ranging from 2.5% to 12.5%. Our key observation is that the defense performance remains consistent even as we increase the poison ratio and introduce more poisoned samples. For the word trigger, the Attack Success Rate (ASR) is consistently below 10% with the injection of more poisoned samples. This can be attributed to the honeypot's enhanced ability to capture these samples and exhibit higher confidence. Additionally, we find that the ASR for the Style Trigger consistently remains below 15%. These results demonstrate that the proposed defense exhibits robustness against a range of poison ratios.

5.4.3 Effectiveness of the GCE Loss

In this section, we investigate the effectiveness of the generalized cross-entropy loss. We conducted experiments using the same settings as in Section 5.4.1 and constructed the honeypot using features from the first layer. As illustrated in Figure 5, we vary the q value from 0.1 to 0.7 and plot the loss curve during honeypot module training, where q = 0 corresponds to the standard cross-entropy loss. Our primary observation is that, as we increase the q value, the honeypot module learns the backdoor samples more rapidly. Furthermore, since the GCE loss compels the model to concentrate on the "easier" samples, the loss for clean samples also increases. This assists the proposed

q = 0.7

q = 0.5

q = 0.1

q = 0.1

q = 0.3

q = 0.5

q = 0.7

weighted cross-entropy loss in assigning lower weights to poisoned samples and higher weights to clean samples. However, excessively large q values can lead to unstable training. Therefore, we opt for a q value of 0.5, which proves effective across various datasets and attack methods.

5.4.4 Impact of the Threshold Value

In this section, we assess the impact of the threshold value c . As mentioned in Section 4, we normalized the weights W ( x ) by using a sign function with a threshold c . In Table 5, we conducted experiments on the SST2 and IMDB datasets with the threshold ranging from 0.05 to 0.8. Our experiments reveal that the defense method remains robust when the threshold lies between 0.1 and 0.3. However, selecting an excessively small value, such as 0.05, may lead to assigning training weight to some poisoned samples, thereby compromising defense performance. Conversely, a too-large threshold may negatively impact the original task performance.

6 Conclusion

In this study, we have presented a honeypot backdoor defense mechanism aimed at protecting pretrained language models throughout the fine-tuning stage. Notably, the honeypot can absorb the backdoor function during its training, thereby enabling the stem PLM to focus exclusively on the original task. Comprehensive experimental evidence indicates that our proposed defense method significantly reduces the success rate of backdoor attacks while maintaining only a minimal impact on the performance of the original task. Importantly, our defense mechanism consistently exhibits robust performance across a variety of benchmark tasks, showcasing strong resilience against a wide spectrum of NLP backdoor attacks.

References

Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys , 56(2):140, 2023.

Xuezhi Wang, Haohan Wang, and Diyi Yang. Measure and improve robustness in nlp models: A survey. In NAACL , 2022.

Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access , 7:47230-47244, 2019.

Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Yinghua Gao, Yiming Li, Linghui Zhu, Dongxian Wu, Yong Jiang, and Shu-Tao Xia. Not all samples are born equal: Towards effective clean-label backdoor attacks. Pattern Recognition , 139:109512, 2023.

Shaofeng Li, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Suguo Du, and Haojin Zhu. Backdoors against natural language processing: A review. IEEE Security & Privacy , 20(05):5059, 2022.

Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. In NeurIPS Datasets and Benchmarks Track , 2022.

Marwan Omar. Backdoor learning for nlp: Recent advances, challenges, and future research directions. arXiv preprint arXiv:2302.06801 , 2023.

Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In ACL , 2020.

Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against lstm-based text classification systems. IEEE Access , 7:138872-138878, 2019.

Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In EMNLP , 2021.

Jiazhao Li, Yijin Yang, Zhuofeng Wu, VG Vydiswaran, and Chaowei Xiao. Chatgpt as an attack tool: Stealthy textual backdoor attack via blackbox generative model trigger. arXiv preprint arXiv:2304.14475 , 2023.

Xudong Pan, Mi Zhang, Beina Sheng, Jiaming Zhu, and Min Yang. Hidden trigger backdoor attack on { NLP } models via linguistic style manipulation. In USENIX Security , 2022.

Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. In SIGKDD , pages 218-228, 2020.

Yiming Li, Haoxiang Zhong, Xingjun Ma, Yong Jiang, and Shu-Tao Xia. Few-shot backdoor attacks on visual object tracking. In ICLR , 2022.

Chengxiao Luo, Yiming Li, Yong Jiang, and Shu-Tao Xia. Untargeted backdoor attack against object detection. In ICASSP , 2023.

Rishi Jha, Jonathan Hayase, and Sewoong Oh. Label poisoning is all you need. In NeurIPS , 2023.

Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS , 2018.

Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In ACL , 2021.

Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Piccolo: Exposing complex backdoors in nlp transformer models. In IEEE S&P , 2022.

Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. Large language models can be lazy learners: Analyze shortcuts in in-context learning. arXiv preprint arXiv:2305.17256 , 2023.

Chuanshuai Chen and Jiazhu Dai. Mitigating backdoor attacks in lstm-based text classification systems by backdoor keyword identification. Neurocomputing , 452:253-262, 2021.

Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. In EMNLP , 2021.

Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rap: Robustness-aware perturbations for defending against backdoor attacks on nlp models. In EMNLP , 2021.

Yansong Gao, Yeonjae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith C Ranasinghe, and Hyoungshick Kim. Design and evaluation of a multi-domain trojan detection method on deep neural networks. IEEE Transactions on Dependable and Secure Computing , 19(4):2349-2364, 2021.

Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Constrained optimization with dynamic bound-scaling for effective nlp backdoor defense. In ICML , 2022.

Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In IEEE S&P , 2021.

Weimin Lyu, Songzhu Zheng, Tengfei Ma, and Chao Chen. A study of the attention abnormality in trojaned berts. In NAACL , 2022.

Biru Zhu, Yujia Qin, Ganqu Cui, Yangyi Chen, Weilin Zhao, Chong Fu, Yangdong Deng, Zhiyuan Liu, Jingang Wang, and Wei Wu. Moderate-fitting as a natural backdoor defender for pre-trained language models. In NeurIPS , 2022.

Haotao Wang, Junyuan Hong, Aston Zhang, Jiayu Zhou, and Zhangyang Wang. Trap and replace: Defending backdoor attacks by trapping them into an easy-to-replace subnetwork. In NeurIPS , 2022.

Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL , 2019.

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics , 8:842866, 2021.

Sishuo Chen, Wenkai Yang, Zhiyuan Zhang, Xiaohan Bi, and Xu Sun. Expose backdoors on the way: A feature-based efficient defense against textual backdoor attacks. arXiv preprint arXiv:2210.07907 , 2022.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , 2013.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Yonatan Belinkov, Sebastian Gehrmann, and Ellie Pavlick. Interpretability and analysis in neural nlp. In Proceedings of the 58th annual meeting of the association for computational linguistics: tutorial abstracts , pages 1-5, 2020.

Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 , 2016.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS , 2018.

Mengnan Du, Subhabrata Mukherjee, Guanchu Wang, Ruixiang Tang, Ahmed Awadallah, and Xia Hu. Fairness via representation neutralization. In NeurIPS , 2021.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In NAACL-HLT , 2011.

Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Predicting the type and target of offensive posts in social media. In NAACL-HLT , 2019.

Kuan-Hao Huang and Kai-Wei Chang. Generating syntactically controlled paraphrases without using annotated parallel pairs. arXiv preprint arXiv:2101.10579 , 2021.

Kalpesh Krishna, John Wieting, and Mohit Iyyer. Reformulating unsupervised style transfer as paraphrase generation. In EMNLP , 2020.

Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In ICLR , 2023.

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurIPS , 2021.

A Understanding the Fine-tuning Process of PLMs on Poisoned Datasets

In this section, we show our empirical observations obtained from fine-tuning PLMs on poisoned datasets. Specifically, we demonstrate that the backdoor triggers are easier to learn from the lower layers than the features corresponding to the main task. This observation plays a pivotal role in designing and understanding our defense algorithm. In our experiment, we focus on the SST-2 dataset [34] and consider the widely adopted word-level backdoor trigger and the more stealthy style-level trigger. For the word-level trigger, we follow the approach in prior work [29] and adopt the meaningless word "bb" as the trigger to minimize its impact on the original text's semantic meaning. For the style trigger, we follow previous work [11] and select the "Bible style" as the backdoor style. For both attacks, we set a poisoning rate at 5% and conduct experiments on the RoBERTa BASE model [35], using a batch size of 32 and a learning rate of 2e-5, in conjunction with the Adam optimizer [36]. To understand the information in different layers of PLMs, we draw inspiration from classifier probing studies [37, 38] and train a compact classifier (one RoBERTa transformer layer topped with a fully connected layer) using representations from various layers of the RoBERTa model. Specifically, we freeze the RoBERTa model parameters and train only the probing classifier.

In Figure 6, we present the training loss curve of the word-level trigger, which utilizes a probing classifier constructed using features extracted from twelve different layers of the RoBERTa model. A critical observation highlights that in the initial layers (1-4), the probing classifier overfits the poisoned samples early in the training phase (around 500 steps). However, it underperforms the original task. This can be attributed to the initial layers primarily capturing surface-level features, including phraselevel and syntactic-level features, which are insufficient for the primary task. Subsequently, in Figure 7, we delve deeper into the visualization of the probing classifier's CLS token embeddings. A notable demarcation can be observed between the embeddings for poisoned and clean samples across all layers. However, the distinction between positive and negative sample embeddings becomes less discernible in the lower layers. We found a similar trend for the style-level trigger, as we showed the learning dynamic in Figure 8 and embedding visualization in Figure 9.

RoBERTa-baseLayer1

RoBERTa-baseLayer2

RoBERTa-baseLayer4

1.00

1

clean samples

1.00

1.00

1.00

poisoned samples

Loss Value

0.75

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0

500

1000

1500

0

500

1000

1500

0

500

1000

1500

0

500

1000

1500

Training Steps

Training Steps

Training Steps

Training Steps

RoBERTa-baseLayer6

RoBERTa-baseLayer7

RoBERTa-baseLayer8

1.00

1.00

1.00

1.00

Loss Value

0.75

Loss Value

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0

500

1000

1500

0

500

1000

1500

0

500

1000

1500

0

500

1000

1500

Training Steps

Training Steps

Training Steps

Training Steps

RoBERTa-baseLayer9

RoBERTa-baseLayer10

RoBERTa-base Layer11

RoBERTa-baseLayer12

1.00

1.00

1.00

1.00

Loss Value

0.75

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0

500

1000

1500

0

500

1000

1500

0

500

1000

1500

0

500

1000

1500

Training Steps

Training Steps

Training Steps

Training Steps

RoBERTa-baseLayer1

RoBERTa-baseLayer2

RoBERTa-base Layer3

RoBERTa-baseLayer4

Positive

Negative

Poisoned

Positive

Positive

Positive

Negative

Negative

Negative

Poisoned

Poisoned

Poisoned

RoBERTa-baseLayer6

RoBERTa-baseLayer7

RoBERTa-baseLayer8

Positive

Positive

Negative

Negative

Poisoned

Poisoned

Positive

Positive

Negative

Negative

Poisoned

Poisoned

RoBERTa-baseLayer9

RoBERTa-baseLayer 10

RoBERTa-base Layer 11

RoBERTa-baseLayer 12

Positive

Positive

Positive

Positive

Negative

Negative

Negative

Negative

Poisoned

Poisoned

Poisoned

Poisoned

RoBERTa-baseLayer1

RoBERTa-baseLayer2

RoBERTa-baseLayer4

1.00

clean samples

1.00

1.00

1.00

poisoned samples

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

Training Steps

Training Steps

Training Steps

Training Steps

RoBERTa-baseLayer6

RoBERTa-baseLayer7

RoBERTa-baseLayer8

1.00

1.00

1.00

1.00

Loss Value

0.75

0.50

0.50

0.50-

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

0

500

1000

1500

2000

Training Steps

Training Steps

Training Steps

Training Steps

RoBERTa-baseLayer9

RoBERTa-baseLayer10

RoBERTa-baseLayer 11

RoBERTa-baseLayer12

1.00

1.00

1.00

1.00

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0

500

1000

1500

2000

0

500

10001500

2000

0

500

1000

1500

2000

0

500

Training Steps

10001500

2000

Training Steps

Training Steps

Training Steps

RoBERTa-base Layer 1

RoBERTa-baseLayer2

RoBERTa-baseLayer 4

Positive

Positive

Negative

Negative

Poisoned

Poisoned

Positive

Positive

Negative

Negative

Poisoned

Poisoned

RoBERTa-baseLayer6

RoBERTa-baseLayer7

RoBERTa-baseLayer8

Positive

Negative

Poisoned

Positive

Negative

Poisoned

Positive

Positive

Negative

Negative

Poisoned

Poisoned

RoBERTa-baseLayer9

RoBERTa-baseLayer10

RoBERTa-baseLayer11

RoBERTa-baseLayer 12

Positive

Positive

Positive

Negative

Negative

Negative

Poisoned

Poisoned

Poisoned

Positive

Negative

Poisoned

B More on Defense Results

In this section, we delve deeper into the comparison between our method and several other backdoor defense strategies, maintaining the same conditions as outlined in Section 5. Particularly, Table 6 shows our honeypot technique against others on the RoBERTaBASE with the IMDB dataset. Additionally, results using the OLID dataset are presented in Table 7. In the case of the IMDB dataset, our method consistently achieves the lowest ASR across all four attack methods, displaying a robust defense technique even under varied adversarial conditions. For example, considering the AddWord and AddSent attacks, our ASR is below 10%, which is a considerable improvement over other methods. In StyleBKD and SynBKD, our ASR stays below 23%, still outperforming the competing

(a) No Defense

(b) Our Method

Positive

Negative

Poisoned

Positive

Negative

Poisoned

methods by a wide margin. Similarly, for the OLID dataset, our method demonstrated excellent performance, surpassing all other defense methods in terms of ASR. Furthermore, our method still achieves competitive ACC results on the original tasks. In Figure 10, we exhibit the t-SNE visualizations derived from the CLS token embeddings of the final transfer layer of the RoBERTa model. As shown in Figure 10 (a), we observe that the no-defense model clearly recognizes the poisoned samples. Instead, in Figure 10 (b), the model overlooks the backdoor trigger and successfully predicts positive samples with embedded backdoor words as the positive class.

C Anti-backdoor Learning Baselines

Besides the NLP backdoor defense baselines, we also considered the backdoor defense baseline in the computer vision domain. Specifically, we adopt a representative baseline, ABL [46], and transform it to adapt to NLP tasks. ABL represents a series of approaches that first identify a small section of poison samples and then use these samples for unlearning to mitigate the backdoor attack.

In Table 9, we found that ABL only achieves disappointing results with an ASR higher than 70%. To shed light on this outcome of ABL, we assessed the backdoor isolation capabilities of ABL. Following the setting in the ABL paper, we initiated a hyperparameter search that γ denotes the loss threshold and T te stands for the epochs of the backdoor isolation stage. Table 8 presents the detection precision of the 1% isolated backdoor examples, which is crucial for the ABL backdoor unlearning performance. However, our findings reveal that the percentage of poisoned samples is less than 20%, which accounts for ABL's suboptimal performance.

The ABL method primarily relies on the observation that 'models learn backdoored data much faster than they do with clean data' [46]. However, it is crucial to note that this assumption mainly holds for models trained from scratch in computer vision tasks. Our research and reference [29] both demonstrated an opposite behavior in that pre-trained language models first concentrate on learning task-specific features before backdoor features. A plausible explanation for this behavior is the richness of semantic information already present in the top layers of the pre-trained language models. Thus, the original task becomes more straightforward compared to the backdoor functionality, causing the model to prioritize learning the main task first. As a result, ABL struggles to yield satisfactory detection performance during the backdoor isolation stage by selecting the 'easy-to-learn' samples (as shown in Table 8), and we show that ABL obtains a high ASR in the following backdoor unlearning process (as shown in Table 9). In contrast, our findings underscore the significance of examining

model structure when identifying backdoor samples, revealing that backdoor samples become more identifiable in the lower layers of PLMs.

D Understanding the Honeypot Defense Training Process

In this section, we further illustrate more details about the honeypot defense training process. Specifically, we focus on the dynamic change of the training weight for poisoned and clean samples. As we mentioned in Section 4, we propose employing a weighted cross-entropy loss ( L WCE ):

f H ( x ) and f T ( x ) represent the softmax outputs of the honeypot and task classifiers, respectively. The function σ ( · ) serves as a normalization method, effectively mapping the input to a range within the interval [0 , 1] . The c is a threshold value for the normalization.

In order to gain a deeper understanding of the re-weighting mechanism, we extend our analysis by presenting both the original W ( x ) and the normalized weight σ ( W ( x ) -c ) . We conducted the experiment using the SST2 dataset, with a word-level trigger, a poisoning rate set at 5%, and a batch size of 32. Figure 11 illustrates the W ( x ) value for both the poisoned and clean samples at each stage of training. Specifically, we computed the W ( x ) for each mini-batch and then calculated the average W ( x ) value for both the poisoned and clean samples. As depicted in the figure, during the warm-up phase, the W ( x ) for clean and poisoned samples diverged early in the training process. After 500 steps, the W ( x ) for poisoned samples was noticeably lower than for clean samples. After the warm-up stage, given that W ( x ) is higher for clean samples, the Cross-Entropy loss of clean samples in f T diminishes more quickly than that of the poisoned samples. This subsequently increase W ( x ) for clean samples as they possess a smaller ¯ L CE ( f T ( x ) , y )) . This positive feedback mechanism ensures that the W ( x ) for poisoned samples persistently remains significantly lower than for clean samples throughout the complete training process of f T . As demonstrated in Figure 11, the W ( x ) for the clean samples will continue to increase following the warm-up phase.

E More on Ablation Studies

E.1 Ablation Study on Honeypot Warm-Up

In the following section, we explore the influence of the preliminary warm-up steps in the honeypot method, which represents the number of optimizations that the honeypot branch requires to capture a backdoor attack. We applied our method against word-level attacks on RoBERTaBASE, and the obtained results are shown in Table 10. The analysis indicates that with a minimum count of warmup steps, specifically below 200 for the SST-2 dataset, the honeypot is insufficiently prepared to

10

poisonedsamples

cleansamples

8

W(x) Value

6

1000

2000

3000

4000

Training Steps

capture the poisoned data. However, once the honeypot accrues a sufficient volume of poisoned data, surpassing 400 training steps across all datasets, the Attack Success Rate (ASR) can be mitigated to an acceptably low level, i.e., less than 10%. The results further prove that our honeypot can effectively capture backdoor information with a certain amount of optimization. In our main experiments, we set the number of warm-up steps equal to the steps in one epoch, thereby enabling our honeypot to reliably catch the poisoned data.

E.2 Ablation Study on Normalization Method

In this section, we use the SST2 dataset and word-level trigger to understand the impact of different normalization functions. As outlined in Section 4, our approach employs a normalization method to map the training loss weight W ( x ) into the [0 , 1] interval. Within our experiments, we opted for the sign function as the normalization technique. However, we also explored two alternative normalization strategies - the sigmoid function and a cutoff ReLU func-

tion. For the latter, we assigned a value of 1 to any input exceeding 1. As depicted in Table 11, we conducted the experiments on RoBERTaBASE using different normalization functions, we can observe that all normalization methods demonstrate decent performance in minimizing the ASR. Notably, we observe that the sign function yields the highest ACC on the original task while simultaneously achieving the lowest ASR.

F Extend Honeypot to Computer Vision Tasks

While this paper primarily focuses on defending pretrained language models against backdoor attacks, we also explored the applicability of our proposed honeypot defense method within the computer vision domain [3, 4, 6]. In Section F.1, we illustrate the experimental settings. In Section F.2, we show the empirical findings. In Section F.3, we discuss the defense performance.

F.1 Settings

Suppose D train = ( x i , y i ) indicates a benign training dataset where x i ∈ { 0 , ..., 255 } C × W × H represents an input image with C channels and W width and H height, and y i corresponds to the associated label. To generate a poisoned dataset, the adversary selects a small set of samples D sub from the original dataset D train , typically between 1-10%. The adversary then chooses a target misclassification class, y t , and selects a backdoor trigger a and a ∈ { 0 , ..., 255 } C × W × H . For each instance ( x i , y i ) in D sub , a poisoned example ( x ′ i , y ′ i ) is created, with x ′ i being the embedded backdoor trigger of x i and y ′ i = y t . The trigger embedding process can be formulated as follows,

where λ ∈ [0 , 1] C × W × H is a trigger visibility hyper-parameter and ⊗ specifies the element-wise product operation. The smaller the λ , the more invisible the trigger and the more stealthy. The resulting poisoned subset is denoted as D ′ sub . Finally, the adversary substitutes the original D sub with D ′ sub to produce D poison = ( D train -D sub ) ∪ D ′ sub . By fine-tuning PLMs with the poisoned dataset, the model will learn a backdoor function that establishes a strong correlation between the trigger and the target label y t . Consequently, adversaries can manipulate the model's predictions by adding the backdoor trigger to the inputs, causing instances containing the trigger pattern to be misclassified into the target class t .

In our experiment, we employed an ImageNet pretrained VGG-16 model as our base model and proceeded with experiments using a manipulated CIFAR-10 dataset. The experiments involve the use of a 3 x 3 white square and a black line with a width of 3 pixels as backdoor triggers. The white square trigger is positioned at the bottom-right corner of the image, while the black line trigger is set at the bottom. We set poison rate as 5% and set λ ∈ { 0 , 0 . 2 } C × W × H for two attacks. The values of λ corresponding to pixels situated within the trigger area are 0.2, while all others are set to 0.

F.2 Lower Layer Representations from VGG Provide Sufficient Backdoor Information

Drawing on our analysis presented in Section 3, we delve further into understanding the information encapsulated within various layers of a pretrained computer vision model. Inspired by previous classifier probing studies [37, 38], we train a compact classifier using representations derived from different layers of the VGG model. We ensure the VGG model parameters are frozen during this process and only train the probing classifier. In this context, we divided the VGG model into five sections based on the pooling layer operations (The five pooling layers are located at layers 2, 4, 7, 10, and 13). Subsequent to this, we integrate an adaptive pooling layer to reduce the features extracted from different layers to 7 × 7 , ensuring that the flattened dimension does not exceed 8000. A fully connected layer with softmax activation is added as the final output. As depicted in Figure 13 and Figure 12, it is noticeable that the lower layers of the VGG model hold sufficient information for identifying the backdoor triggers. However, they do not contain enough information to effectively carry out the main tasks.

VGGLayer2

VGGLayer4

VGGLayer7

VGGLayer13

2.0

Poison samples

2.0

Poison samples

2.0

Poison samples

2.0

Poison samples

2.0

Poison samples

Clean samples

Clean samples

Clean samples

Clean samples

Clean samples

Value

1.5

Value

1.5

Loss Value

1.5

Value

1.5

Value

1.5

LoSS

1.0

5507

1.0

1.0

SS07

1.0

SS07

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

0

1000

2000

3000

0

1000

2000

3000

0

1000

2000

3000

0

1000

2000

3000

0

1000

2000

3000

Training Steps

Training Steps

Training Steps

Training Steps

Training Steps

VGGLayer2

VGGLayer4

VGGLayer7

VGGLayer10

VGGLayer13

2.0

Poison samples

2.0

Poison samples

2.0

Poison samples

2.0

Poison samples

2.0

Poison samples

Loss Value

1.5

Clean samples

1.5

Clean samples

Value

1.5

Clean samples

Loss Value

1.5

Clean samples

Loss Value

1.5

Clean samples

1.0

SS0

1.0

SS07

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

0

1000

2000

3000

0

1000

2000

3000

0

1000

2000

3000

0

1000

2000

3000

0

1000

2000

3000

Training Steps

Training Steps

Training Steps

Training Steps

Training Steps

F.3 Defense Results on CIFAR10

We implemented the honeypot as mentioned in Section 4 and built the honeypot module with the features from the first pooling layer. We followed previous sections and adopted the ASR and ACC metrics to measure the model's performance on the poisoned test set and clean test set, respectively. Specifically, we executed a fine-tuning process for a total of 10 epochs, incorporating an initial warmup epoch for the honeypot module. The learning rates for both the honeypot and the principal task are adjusted to a value of 1 × 10 -3 . Additionally, we established the hyperparameter q for the GCE loss at 0.5, the time window size T was set to 100, and the threshold value c was fixed at 0.1. Each experimental setting was subjected to three independent runs and randomly chosen one class as the target class. These runs were also differentiated by employing distinct seed values. The results were then averaged, and the standard deviation was calculated to present a more comprehensive understanding of the performance variability. As the results are shown in Table 12, the proposed method successfully defends two backdoor attacks and reduces the ASR to lower than 10%. This indicates that the proposed method is valid for those simple vision backdoor triggers while having minimal impact on the original task. We plan to test the defense performance of more advanced backdoor triggers in our future work.

G Limitations and Discussions

In this study, we introduce an innovative approach to backdoor defense in the context of fine-tuning pretrained language models. Due to the constraints in terms of time and resources, our evaluations were conducted using four prevalent backdoor attack methods and on three representative datasets. Despite the robustness and consistency demonstrated by our method, it is essential to remain vigilant to the emergence of new and potentially threatening attack methods and datasets, especially considering the rapid growth of this field. In addition, it's worth acknowledging that while unintended, some malicious users may exploit our method and deploy other strong backdoor attacks that may bypass our defense system.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new active defense framework for PLMs against backdoors attacks. The proposed defense inserts a honeypot module into the original PLM such that the backdoor information is absorbed only by the honeypot module and does not impact the main task module. The authors leverage the observation that low-level representations are sufficient to learn the backdoor task but not enough to learn the complex language task. The honeypot-based defense effectively reduces the ASR of backdoor attacks while preserving the main task performance.\n\nSTRENGTHS:\nThis paper has the following strengths: \n+ The idea of using an additional honeypot module within the PLM to absorb the backdoor information is interesting and new.\n+ The authors provide a clear discussion about the key observation (i.e., low-level representation is sufficient to learn the backdoor task). \n+ The authors design GCE loss and a new weighted loss to train the honeypot and task classifier with the goal of enforcing the task classifier to focus on clean samples only and the honeypot module to focus on poisoned samples.\n\nWEAKNESSES:\nThis paper has the following weaknesses: \n1) Some statements are not clearly justified/explained. For example, Section 5.2 mentions that the proposed defense has better performance on large models according to Table 1. It would be great to provide an analysis/hypothesis about this observation. \n-2) The design of the weighted CE loss (Equations 3 and 4) is not justified by experiments. Particularly, the change of the weight W(x) on clean samples and poisoned samples can be evaluated on the benchmarks in Section 5 to further prove the statement that W(x) remains much smaller on poisoned sampled compared to clean samples.\n\nQUESTIONS:\nQ1. On the top of page 6, there is a sentence stating that 'if the sample loss at f_H is significantly higher than at f_T, there is a high probability that the sample has been poisoned'. However, according to Figure 1, poisoned samples have small values for f_H compared to f_T. Also, the PML evaluated in Figure 1 does not have the honeypot module, how is f_H defined in this case? \n\nQ2. What is the capacity of the honeypot module? Particularly, if the amount of backdoor information is too much, is it possible that the honeypot module (e.g., with few layers) cannot fully absorb the backdoor? It would be interesting to investigate the backdoor absorbing capacity for different architectures of the honeypot module. \n\nQ3. What is the possible reason that the proposed defense has better performance on large models (Table 1)?  \n\nQ4. Section 5.3 proposes adaptive attacks based on [41]. However, there is no introduction to the work in [41]. Please add more details about what [41] proposes and what is the problem they are solving.\n\nLIMITATIONS:\nPlease consider addressing the questions and weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new defense against backdoor attacks on pre-trained language models (PLMs). By leveraging the observation that the loss of poisoned samples drops faster in early layers of PLMs compared to clean samples, it dynamically reduces the weight of suspicious samples in fine-tuning. Empirical results show the effectiveness of the proposed defense against 4 attacks. Overall, the work represents yet another defense that exploits the learning dynamics difference of clean and poisoned samples.\n\nSTRENGTHS:\n- Leveraging the learning dynamics difference of clean and poisoned samples is an interesting idea. Even though it has been exploited in prior work (e.g., [26]), this work proposes to dynamically adjust the weight of suspicious samples, which seems new. \n\n- Empirical results show the effectiveness against both word-based and style-based backdoor attacks. \n\n- The paper is well-structured and easy to follow.\n\nWEAKNESSES:\n- The threat model needs better motivation. It assumes a clean PLM, which is fine-tuned using potentially poisoned data. Typically, the PLM is provided by external parties (e.g., downloaded from the Web) while the fine-tuning dataset is managed by the user. It seems a more practical setting that the PLM contains backdoors while the fine-tuning data is clean. \n\n- The proposed defense bears a lot of similarity to existing defenses that also exploit the learning dynamics difference of clean and poisoned samples (e.g., [26] and Li et al. \"Anti-Backdoor Learning: Training Clean Models on Poisoned Data\", NeurIPS 2021). However, there is no empirical comparison with prior work, which makes it difficult to assess its superiority. \n\n- While the work targets PLMs, the proposed method seems agnostic to the underlying models. It is suggested to tailor the method to the unique characteristics of NLP models.\n\nQUESTIONS:\n- Please clarify the threat model. \n\n- Compare the defense with other defenses based on the learning dynamics difference of clean and poisoned samples.\n\n- Optimize the defense for NLP models.\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first makes an observation that in a backdoor poisoning attack against a pretrained language model, the lower layers learn the backdoor feature quickly and easily. Based on this observation, the authors then design a honeypot-based defense that catches the training samples that could be learned with low loss by the lower layers, hoping that these would be poisoned. On the other hand, the loss from the training samples that cannot be learned easily by lower layers is upweighted, which defuses the backdoor. The authors evaluate their defense on multiple NLP classification tasks on multiple architectures and attacks, including adaptive attacks.\n\nSTRENGTHS:\n+ The proposed defense is well reasoned based on empirical observations about how backdoor attacks are learned.\n+ Detailed evaluation, considers stronger adaptive attacks as well and shows success.\n+ The idea of probing hidden layers to make observations about the learning dynamics is interesting as most work focuses on looking at loss dynamics during training.\n\nWEAKNESSES:\n- The main idea the defense relies on is not a new one and there are already successful defenses that exploit this idea. The authors have not considered these defenses as a baseline.\n- Some additional experiments regarding the impact of the defense in low-ratio poisoning regimes would be informative to have.\n\nThe defense essentially observes that backdoor poison samples are learned easily by the lower layers, and, as a result, a defense that isolates away the samples learned in lower layers can prevent the attack. This is a good observation but a very similar one has been made before toward a defense. For example, Anti-Backdoor Learning (ABL) by Li et al. has also made this observation and proposed a similar defense based on isolating easy-to-learn samples. A difference in the work under review is using the lower layers instead of loss dynamics during training as a measure of sample difficulty. However, it is known that these notions are very correlated (e.g., Deep Learning Through the Lens of Example Difficulty, by Baldock et al.), so this difference might not matter after all.\n\nThis brings us to my first bullet point, the authors should've evaluated their work against ABL (or a more recent follow-up if that exists) as the proposed defense and ABL share their starting point. I'm not convinced that the proposed defense can improve upon ABL significantly without offering a different insight.\n\nThat being said, the observation that backdoor poison samples are easy to learn is an artifact of the attack and its parameters. There are recent backdoor attacks that break defenses like ABL by crafting more difficult-to-learn poisons. For example, Narcissus: A Practical Clean-Label Backdoor... by Zeng et al. Lowering the poison percentage is a way to craft such poisons but the experiment provided in Section 5.3 is not enough to be convincing. \n\nIdeally, I would like to see a plot when you apply the DPR-AST attacks in Section 5.3 but you vary the poison percentage, starting from a percentage that achieves very low ASR. Essentially, the x-axis is the poison ratio and the y-axis is the ASR, and there are two curves, one when the honeypot defense is applied and the other for an undefended model. In particular, I would like to see if there is a regime where the undefended model achieves lower ASR than the defended model because the proposed honeypot defense starts boosting difficult-to-learn poison samples (which could be poison samples depending on the poison ratio). This way, we can have a better idea when the honeypot defense is viable and effective and when it is necessary to deploy another type of defense that makes different assumptions.\n\nQUESTIONS:\n- How does the observation in Section 3 change if the poison ratio is varied (let's say between 0.1% to 10%)? Do you think the observation would persist in low poison percentages or the poison samples would not be learned easily in lower layers anymore?\n\n- Why do you think, in Figure 1, the CE Loss for poison samples increases over layers? This seems to conflict with the claim in Line 200 (easier samples, the majority of which are poisoned samples). If poison samples are easier, why do they seemingly become more difficult at the deeper layers?\n\n- What type of model did you use to train the probing classifiers in Section 3? Is it a simple linear model?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to defend against NLP backdoors during training.\nThe proposed method works by using an honeypot module to absorb backdoor\ninformation, and prevent the backdoor behaviors to be learned by the stem\nnetwork. Experiments on SST-2, IMDB, and OLID demonstrate the effectiveness of\nthe propsoed method.\n\nSTRENGTHS:\n* The investigated problem is interesting.\n\n* The motivation of this paper is good.\n\nWEAKNESSES:\n* The proposed method is based on the observation that\nlearning the backdoor task is generally easier than learning\nthe main task. However, this observation may not always hold\ntrue. In the case of label-specific backdoor attacks (also\ncalled as all-to-all attak in BadNets [1]), where samples\nwith different original labels have different target labels,\nthe backdoor task becomes even more complex compared to the\nmain task. To achieve the desired backdoor behavior in such\nattacks, the model must first identify the correct label of\nthe backdoor samples before making backdoor predictions\nbased on that recognized label. Unfortunately, this paper\nlacks a discussion and empirical results concerning\nlabel-specific attacks. As a result, the generalizability of\nthe proposed method to different types of attacks remains\nunclear.\n\n* Comparison to related work CUBE [2] is missing. While this\npaper claims that the proposed method surpasses existing\ndefenses, it fails to include a comparison to CUBE, a\ntraining-time textual backdoor defense method. It is\nrecommended to incorporate a comparison with CUBE to provide\na more comprehensive evaluation of the proposed method's\nperformance.\n\n\n* The proposed approach essentially considers samples with\nW(x) values below the threshold value c as identified\npoisoning samples. To gain a deeper understanding of the\nproposed method's effectiveness, it is recommended to\ndiscuss the measures of detection precision and recall in\ndetail.\n\n[1] Gu et al., BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv 2017.\n\n[2] Cui et al., A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks. NeurIPS 2022 Datasets & Benchmarks.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe limitations is discussed in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}