Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypots
Authors: Ruixiang Tang
Jiayi Yuan
Yiming Li
Zirui Liu
Rui Chen
Xia Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Backdoor Defense
Honeypot

Abstract: In the field of natural language processing, the prevalent approach involves fine-tuning pretrained language models (PLMs) using local samples. Recent research has exposed the susceptibility of PLMs to backdoor attacks, wherein the adversaries can embed malicious prediction behaviors by manipulating a few training samples. In this study, our objective is to develop a backdoor-resistant tuning procedure that yields a backdoor-free model, no matter whether the fine-tuning dataset contains poisoned samples. To this end, we propose and integrate an \emph{honeypot module} into the original PLM, specifically designed to absorb backdoor information exclusively. Our design is motivated by the observation that lower-layer representations in PLMs carry sufficient backdoor features while carrying minimal information about the original tasks. Consequently, we can impose penalties on the information acquired by the honeypot module to inhibit backdoor creation during the fine-tuning process of the stem network. Comprehensive experiments conducted on benchmark datasets substantiate the effectiveness and robustness of our defensive strategy. Notably, these results indicate a substantial reduction in the attack success rate ranging from 10\% to 40\% when compared to prior state-of-the-art methods.

Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypots
Ruixiang Tang Jiayi Yuan Yiming Li Zirui Liu', Rui Chen', Xia Hu'
Department of Computer Science, Rice University
-ZJU-Hangzhou Global Scientific and Technological Innovation Center Schooo of Cyber Science and Technology, Zhejiang University *Samsung Electronics America (rt39,jy101, xia. hul aulcrriceeeee edu; 1i li-ym@zju. edu. cn; rui. chenchhennnec Seanl@samuung.cmm
Abstract
In the field of natural language processing, the prevalent approach involves fine- tuning pretrained language models (PLMs) using local samples. Recent research has exposed the susceptibility of PLMs to backdoor attacks, wherein the adversaries can embed malicious prediction behaviors by manipulating a few training samples. In this study, our objective develop a backdoor-resistant tuning procedure that yields a backdoor-fre model, no matter whether the fine-tuning dataset contains poisoned samples. To this end, we propose and integrate a honeypot module into the original PLM, specifically designed to absorb backdoor information exclusively. Our design is motivated by the observation that lower-layer representations PLMs carry sufficient backdoor features while carrying minimal information about the original tasks. Consequently. we can impose penalties on the information acquired by the honeypot module to inhibit backdoor creation during the fine- tuning process of the stem network. Comprehensive experiments conducted on benchmark datasets substantiate the effectiveness and robustness of our defensive strategy. Notably, these results indicate a substantial reduction the attack success rate ranging from 10% to 40% when compared to prior state-of-the- the-art methods.
Introduction
Recently, the rapid progress of pretrained language models (PLMs) has transformed diverse domains, showcasing extraordinary capabilities in addressing complex natural language understanding tasks. By fine-tuning PLMs on local datasets, these models can swiftly adapt to various downstream tasks [ Nevertheless, with the increasing power and ubiquity of PLMs, concerns regarding their security and robustness have grown 22. Backdoor attacks, where models acquire malicious functions from
5 have surfaced as one of the principal threats to PLMs' integrity and 1 8 During a backdoor attack, an adversary tampers with the fine-tuning dataset by introducing a limited number of backdoor poisoned samples, each containing backdoor trigger and labeled to a specific target class. Consequently, PLMs fine-tuned on the poisoned dataset learn backdoor function together with the original task. Recently. various backdoor attack techniques have been proposed in the field of natural language processing (NLP), exploiting distinct backdoor triggers such as inserting words y sentences [TO), changing text syntactic and style [[] [3] Empirical evidence suggests that existing PLMs are highly susceptible to these attacks, presenting substantial risks to the deployment of PLMs in real-world applications.
poisoned datasets 3
functionality 6
In this study, we aim to protect PLMs during the fine-tuning process by developing a backdoor- resistant tuning procedure that yields a backdoor-free model, no matter whether the fine-tuning dataset
Equal contribution. The order of authors is determined by flipping a coin.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

contains poisoned samples. Our key idea is straightforward: the victim model essentially learns two distinct functions- one for the original task and another for recognizing poisoned samples. If we can partition the model into two components- one dedicated to the primary function and the other to the backdoor function we can then discard or suppress the side effects of the latter for defense. To implement this concept, we propose the addition of honeypot module to the stem network. This module is specifically designed to absorb the backdoor function during training, allowing the stem network to focus exclusively on the original task. Upon completion of training, the honeypot can be removed to ensure a robust defense against backdoor attacks.
In designing our honeypot module, we draw inspiration from the nature of backdoor attacks, where victim models identify poisoned samples based on their triggers. typically manifested as words, sentences, or syntactic structures. Unlike the original task, which requires understanding a text's entire semantic meaning, the backdoor task is far easier since the model only needs to capture and remember backdoor triggers. We reveal that low-level representations in PL.Ms provide sufficient information to recognize backdoor triggers while containing insufficient information for learning the original task. Based on this observation, we construct the honeypot as compact classifier that leverages representations from the lower layers the PLMs. Consequently, the designed honeypot module rapidly overfits poisoned samples during early training stages, while barely learning the original task. To ensure that only the honeypot module learns the backdoor function while the stem network focuses on the original task, we propose simple yet effective re- re-weighting mechanism. This concept involves encouraging the stem network to learn samples that the honeypot classifier finds challenging to classify, which are typically clean samples, and ignoring samples that the honeypot network confidently classifies. In this way, we guide the PLMs to concentrate primarily on clean samples and prohibit backdoor creation during the fine-tuning process.
We evaluate the feasibility of the proposed methods in defending against an array of representative backdoor attacks spanning multiple NLP benchmarks. The results demonstrate that the honeypot defense significantly diminishes the attack success rate of the fine-tuned PLM on poisoned samples while only minimally affecting the performance of the original task on clean samples. Notably, for challenging backdoor attacks, e.g., style transfer attack and syntactic attack, we stand out as a defense to achieve below-randomoe the state-of-the-ar defense method by further reducing the attack success rate from 60% down to 20%. The visualization of the model's learning dynamics on the poisoned dataset and comprehensive ablation study further validated our method's ability. Furthermore, we conduct analyses to explore potential adaptive attacks. In summary, this paper makes the following contributions:
attack success rate (i.e., < 50%). Specifically, we have advanced
We demonstrate that the feature representations from the lower layers of PLMs contain sufficient information to recognize backdoor triggers while having insufficient semantic information for the original task.
We introduce honeypot defense strategy to specifically absorb the backdoor function. By imposing penalties on the samples that the honeypot module confidently classifies, we guide the PLMs to concentrate solely on original tasks and prevent backdoor creation.
Our experimental results demonstrate that the proposed method efficiently defends against attacks with diverse triggers, such as word, sentence, syntactic, and style triggers, with minimal impact on the primary task. Furthermore, our method can be applied to different benchmark tasks and exhibits robustness against potential adaptive attacks.
2 Preliminaries
2.1 Backdoor Attack in NLP
17. In this scenario, an adversary selects a small portion of data from the training dataset and adds a backdoor trigger, such as distinctive colorful patch [8]. Subsequently, the labels of all poisoned data points are modified to a specific target class. Injecting these poison samples into the training dataset enables the victim model to learn a backdoor function that constructs a strong correlation between the trigger and the target label together with the original task. Consequently, the model performs normally on the original task but predicts any inputs containing the trigger as the target class. Recently, numerous studies have applied backdoor attacks to various NLP tasks. In the context of natural language, the
Backdoor attacks were initially proposed in the computer vision domain B 15 4 16

---

backdoor trigger can be confext-independent words or sentences 19 Further investigations have explored more stealthy triggers, including modifications to the syntactic structure or changing text style [19
1 20 21 These studies demonstrate the high effectiveness of textual backdoor attack triggers against pretrained language models.
2.2 Backdoor Defense in NLP
Recently, several pioneering works have been proposed to defend against backdoor attacks in the field of NLP. We can divide existing defenses into three major categories: (1) Poisoned sample detection: The first line of research focuses on detecting poisoned samples 23 23 24 23]. A representative
work Backdoor Keyword Identification (BKI) [22], in which the authors employ the hidden state of LSTT to detect the backdoor keyword in the training data. Additionally, some studies concentrate on identifying poisoned samples during inference time. For instance, a representative work [23] aims to detect and remove potential trigger words to prevent activating the backdoor in a compromised model. (2) Model diagnosis and backdoor removal: This line of work seeks to predict whether the model contains a backdoor function and attempts to remove the embedded backdoor function [26 27 28 20 (3) Backdoor-resistann tuning: This category presents a challenging scenario for backdoor defense [29 30), as the defender aims to develop secure tuning procedure that ensures PLM trained on the poisoned dataset will not learn the backdoor function. The work [29] reveals that the PLM tuning process can be divided into two stages. In the moderate-fitiine stage, PLMs focus solely on the original task, while in the overfitting stage, PLMs learn both the original task and the backdoor function. The model could alleviate the backdoor by carefully constraining the PLM's adaptation to the fitttra stage. Our proposed honeypot defense belongs to the third category and offers a different solution.
To the best of our knowledge, (00 is the most closely related work to our proposed honeypot defense method. The authors present two-stage defense strategy for computer vision tasks. In the first stage, they deploy two classification heads on top of the backbone model and introduce an auxiliary image reconstruction task to encourage the stem network to concentrate stage, they utilize a small hold-out clean dataset to further fine-tune the stem network and counteract the backdoor function. In comparison, our proposed method eliminates the need for a two-stage training process or a hold-out small clean dataset for fine-tuning, rendering it more practical for real-world applications.
the original task. In the second
2.3 Information Contained within Different Layers of PLMs.
Numerous studies have delved into the information encapsulated within different layers of pretrained language models. For example, empirical rescarch has examined the nature of representations learned by various layers in the BERT model 31 32]. The findings reveal that representations from lower layers capture word and phrase-level information, which becomes less pronounced in the higher layers. Syntactic features predominantly reside in the lower and middle layers, while semantic features are more prominent in the higher layers. Recent studies have demonstrated that PLMs employ distinct features to identify backdoor samples [33]. We further investigated the backdoor features presented in different layers and found that lower-layer features are highly effective in recognizing backdoor samples. One explanation is that existing text backdoor triggers inevitably leave abundant information the word, phrase, or syntactic level, which is supported by previous empirical studies.
3 Understanding the Fine-tuning Process of PLMs on Poisoned Datasets
In this section, we discuss our empirical observations obtained from fine-tuning PLMs on poisoned datasets. Specifically, we found that the backdoor triggers are casier to learn from the lower layers compared to the features corresponding to the main task. This observation plays pivotal role in the design and understanding of our defense algorithm. In Section [3T] we provide a formal description of the poisoned dataset. In Section 3.2 we subsequently delve into our empirical observations.
3.1 Settings
Consider a classification dataset Dirain = (xi, V:), where oe represents an input text, and yi cor- responds to the associated label. To generate a poisoned dataset, the adversary selects small set
3

---

Word-level Trigger
Style-level Trigger
0.4
clean samples
clean samples
CE Loss Valuve
palsoned samples
CE Loss Value
poisoned samples
0.3
0.3
0.2
0.2
0.0
0.0
2
8
10
12
10
12
Layer Index of Classifer Probe
Layer Index of Classifer Probe
(a) Probing Classifier
RoBEATT-baae Layer
RoBERtD-bose Layer 4
RoBFRTS-base Layer 8
RoBEATD-base Layer
1.00
1.00
3.00
0.75
0.50
0.50
0.00
500
1000 Thatning Steps
1500
500
2000
1500
500
1000
1500
500
1000
1500
Traning Stepp
Traning Steps
Tratinng Staps
(b) Learning Dynamic
Figure I: Fine- TuninT PLMs on Poisoned Datasets, (a) Probing classifier loss on the validation dataset using representations across the model. (b) Visualizing training loss for poisoned and clean samples for the world-level trigger.
of samples Duub from the original dataset Dtrains typically between 1-10%. The adversary then chooses a target misclassification class, and selects backdoor trigger. For each instance (I;, vi) in aud.. a poisoned example (1/, vf) is created, with x being the embedded backdoor trigger of x; and y/ yt- The resulting poisoned subset is denoted as D' sub. Finally, the adversary substitutes the original Sub with Deub to produce Dpoison = (Drrain- the poisoned dataset, the model will learn backdoor function that establishes a strong correlation between the trigger and the target label 1/+- Consequently, adversaries can manipulate the model's predictions by adding the backdoor trigger to the inputs, causing instances containing the trigger pattern be misclassified into the target class
U By fine-tuning PLMs with
In this experiment, we focus on the SST-2 dataset 34 and consider the widely adopted word-level backdoor trigger as well as the more stealthy style-level trigger. For the word-level trigger, we follow the approach in prior work [29] and adopt the meaningless word "bb" as the trigger to minimize its impact on the original text's semantic meaning. For the style trigger, we follow previous work [I]] and select the "Bible style" as the backdoor style. For both attacks, we set poisoning rate at 5% and conduct experiments on RoBERTaaase model [35]. using a batch size of 32 and learning rate of 2e-5, in conjunction with the Adam optimizer 33
3.2 Lower Layer Representations Provide Sufficient Backdoor Information
To understand the information in different layers of PLMs, we draw inspiration from previous classifier probing stud- and train compact classifier (one RoBERTa transformer layer topped with a fully connected layer) using representations from various layers of the RoBERTa model. Specifically, we freeze the RoBERTa model parameters and train only the probing classifier. As depicted in Figure [ (a), the validation loss value of the probing classifier reveals an interesting pattern. Notably, the lower layers (0-4) of the RoBERTa model contain sufficient backdoor trigger features for both word-level and style-level attacks, thereby showing an extremely low CE loss value for poison samples. Figure [ (b) presents the training loss curve for the probing classifier for the word-level trigger. We can see that the probing classifier rapidly captures the backdoor triggers in the early stages for lower-layer features. For example, the probing classifier already overfits the poisoned samples after 300 steps using representations from layers 1 and 4.
ies (37 38
Positive
Negative
Poisoned
Figure 2: Embedding visualization.

---

Accelerate Backdoor Learning
Honeypot
LGD
LaCE
Classifter
Encoder
Encoder
Encodes
Encoder
Encoder
LWOE
Training Samples
Pretrained Language Model
Figure 3:
Illustration of the honeypot-based defense framework: The honeypot classifier optimizes the generalized cross-entropy (CaCe) loss to overfit the backdoor samples. The task classifier trains using a weighted cross-entropy loss, which strategically assigns larger weights to clean samples and small weights poisoned samples during the task classifier's training process.
We also found distinct disparity in learning between clean and poisoned samples in lower layers. As demonstrated in Figure (a-b), the loss of clean samples was significantly higher than poisoned samples when training probe classifier with representations from low-layer. This observation aligns with earlier research [31 32) proposing that lower-layer features primarily encapsulate superficial features, such as phrase-level and syntactic-level features. Conversely, classify clean samples, models must extract the semantic meaning, which only emerges in higher-layer features within PLMs. To further illustrate this learning disparity, in Figure we present t-SNE visualization of the CLS token embedding derived from the probing classifier trained with representations from layer 1. This visualization reveals clear demarcation between the embeddings for poisoned and clean samples, while the embeddings of positive and negative samples appear less distinguishable in the lower layers. We put more visualization results and discussions in Section
The Proposed Method
Our defense method stems from the observations in Section 3 which indicates that poisoned samples frequently involve the injection of words, sentences, or syntactic structures that are effectively identified by the lower-layers of PLMs. Intuitively, if backdoor triggers are easier to learn for PLMs' lower layers compared to the features corresponding to the main task, we can strategically insert "honeypot" within these lower layers to trap the backdoor functions. Specifically, as illustrated in Figure 3 our proposed algorithm concurrently trains pair of classifiers (fH. fr) by (a) purposefully training honeypot classifier fu to be backdoored and (b) training a task classifier fr that concentrates on the original task. The honcypot classifier fH consists of one transformer layer topped with a fully connected layer to make predictions. We emphasize that the honcypot classifier is only placed at the lower layer, e.g., layer 1. Thus it only relies the features of these lower layers to learn the backdoor function. The trainable parameter of the honeypot is denoted as 0r. Inspired by previous work [39 40), apply Generalized Cross-Entropy (GCE) loss positioned samples to the honeypot classifier:
39
40
to enlarge the impact of
1 1 (x
LoCE(f(x;
(I)
q
where (2; 0H) is the output probability assigned to the ground truth label y. The hyper-parameter q E (0, 1] controls the degree of bias amplification. As lim (z:1
log fy (z; 0). which
equivalent the standard cross-entropy loss. The core idea is to assign higher weights f highly confident samples while updating the gradient. To see this,
@Locr(P.V) (g( 00H
@Cce(p.))
00H
Thus, the GCE loss further encourages the honeypot module to only focus on the "easier" samples, the majority of which are poisoned samples when using the lower layer representations, in contrast to network trained with the normal CE loss.
5

---

For the task classifier fr, its primary objective is to learn the original task while avoiding the acquisition of the backdoor function. As we previously analyzed, the honeypot will absorb the impact of these positioned samples. Then according to Figure [ we can distinguish the poisoned samples and normal samples by comparing the loss of fi and Jr. If the sample loss at fH is significantly lower than at fr, there is a high probability that the sample has been poisoned. Thus, if we are confident that a particular sample has been poisoned, we can minimize its influence by assigning smaller weight. Specifically, we propose employing weighted cross-entropy loss (CWCE) for achieving this goal, which is expressed as follows:
(wcr(fr(=).v)
= (W)(s) c) (on(fr(x),)) where (CE(JH(F)..) ((x) -c
(3)
(4)
(1r and fr(x) represent the softmax outputs of the honeypot and task classifiers, respectively. The function a-) serves as a normalization method, effectively mapping the input to a range within the interval [0, 1]. e.g., the Sigmoid function, Sign function, and rectified Relu function. The c is threshold value for the normalization. (CE(f((z) y) is the averaged cross-entropy loss of the task classifier fr among the last t steps, where is a hyperparameter that controls the size of the time window. W (x) = (J y) ()r(()))) v)) is the ratio of the loss of the honeypot classifier at the current step versus the averaged one of the task classifier fr among the last t steps.
We elaborate on how the proposed framework works. We first warm up the honeypot classifier fH for some steps to let it "prepare" for trapping the backdoor triggers. Then according to results shown in Figure (b), after the warmup stage, the CE loss value of poisoned samples in [H is already significantly lower than that of clean samples, whereas both sample losses are still high in fr since the stem network is untrained. Consequently, W(x) will assign a lower weight to poisoned samples. During the subsequent training process, ((x) is higher for clean samples, the CE loss of clean samples in fi decreases more rapidly than that of poisoned samples. This will further amplifying (xx) for clean samples as they have smaller Lc=(fr(z), v)). This positive feedback mechanism ensures that the W (z) for poisoned samples remains significantly lower than for clean samples throughout the entire training process of fr. To further reduce the poisoned samples' impact, use the Sign function as the normalization method a to further diminish the impact of poisoned samples with W (zz) weight less than the threshold c.
5
Experiments
5.1 Experiment Settings
In our experiments, we considered two prevalent PLMs with different capacities, including BERT BASE- BERTLARGE RoBERTaBAsc: and RoBERTa We leverage diverse range of datasets, incorporat- ing the Stanford Sentiment Treebank (SST-2) the Internet Movie Database (IMDB) film reviews dataset[41] and the Offensive Language Identification Dataset (OLID) We concentrated on four representative NLP backdoor attacks, specifically word-level attack (AddWord), sentence- attack (AddSent), style transfer backdoor attack (StyleBKD), and syntactic backdoor attack (SynBKD). In the context of word and sentence-leved attacks, we introduced meaningless words or an irrelevant sentence correspondingly. For syntactic attack, we followed the previous work [19 and consider paraphrasing the benign text using sequence-to-seeueneee conditional generative model 4 As for the style transfer attack, we employed a pretrained model [44 to transition sentences into biblical style. We followed previous works [29] (3) and adopted well-established metrics to quantitatively assess the defense outcomes. Firstly, the Attack Success Rate (ASR) is utilized to evaluate the model's accuracy on the poisoned test set, serving as a gauge of the extent to which the model has been effectively backdoored. Secondly, we use the Clean Accuracy (ACC) metric to measure the model's performance on the clean test set. This metric offers quantitative assessment of the model's capability to perform the original task.
For each experiment, we executed a fine-tuning process for a total of 10 epochs, incorporating an initial warmup epoch for the honeypot module. The learning rates for both the honeypot and the principal task are adjusted to value of 10-5, Additionally, we established the hyperparameter for the GCE loss at 0.5, the time window size T was set to 100, and the threshold value e was fixed at

---

Table 1: Overall defense performance
Dataset
Vietim
BERT BASH
BERT LARGE
RoBERTa
RoBERTaLAaaE
Attack
ACC (1)
ASR (1)
(T)
ASR (L)
ACC (1)
ASR (1)
ACC (1)
ASR ())
AddWord
90.34 =0.72
10.88 =3.02
93.11 #0.56
6.77 #1.97
93.71
20.05
6.56 =1.91
94.15 +0.50
5.84 #2.16
AddSent
91.03
7,99
92.66 11:00
7.71 22:40
92.39
20.00
7.71
94.83 10:4
4,20
SST-2
StyleBKD
89.79 =0.10
25.89 #4.50
93-46
#0.76
22.54
+0.70
93.15
=0.82
20.87
+3300
94,50
+0.55
19.62 #4.20
SynBKD
86.23 =1.00
29.00 25.00
92.20
=0 90
26.45
#4.00
93.34
20.75
22.90
#3.85
94,00
#1.30
21.20 2:410
Average
89,35 =0.85
18.44
#3.95
92.86
+0
15.87
+3.00
93.15
=0.71
14.51
#2.08
94.37
#0.52
12.72 #3.19
AddWord
91.32
7.20 12:.3
92.60
5.84
93.72 10.81
5.60
94,12
3.60
AddSent
91.00 00.95
8.16 2::45
92.12 1:0
9.52 22:37
92.72 20.85
6.56 23.93
93.68 11110
6.32 12:14
IMDB
StyleBKD
89.44
20.60
92.76
18.70
93.12
19.36
12.50
94.50
17.90
SynBKD
89.04 =0.34
23.60 #2.75
91.96 10.00
24.96 #2.04
93.20 20.95
22.70
#2.80
94.00 1:100
18.40 #2,46
Average
90.20
14.89
92.36
14.76
93.19
13.56
1/249
94,08
11:08
11,58 12.43
AddWord
80.81 =0.97
12.74 =1.21
85,00
=1.00 00
10.48 #2200
82.79 #0.85
11.45 +3.17
86,34
#:110
9.78 #2.91
AddSent
84.88 =0.85
5.64 #2.15
86.66
+2.00
#0.82
=2.04
+0.50
#1.96
OLID
StyleBKD
83.95 =0.95
29.36 #2200
83.83
+0 58
27.20 +:300
83,95 =0.00
29.18
+2.02
87.20
#1.02
26. 10 #2.90
SynBKD>
83.02
30.
85.00
29.40
83.02
28.40 12.50
85.34
30.
Average
83.17 =0.97
19.46 #2.80
85.12 +1.01
18.10 +2.50
83.28
18.47 #2.76
86.55 +1.02
17.63 #2.06
Table 2: Performance comparison with other defense methods in SST2 Dataset
Defence Method
AddWard
AddSSntt
StyleBKD
SynBKD
(C)
ASR (1)
ACC (1)
ASR
ACC (t)
ASR
ACC
ASR (J)
No defense
94.61 20.00
100,00
10:0
94.38 10.32
100.00 10.00
93.92 1u.35
100.00 10.00
94.49
10.57
100.00
10.00
94.72 _0.00
86.37
13:15
20.:
100.00 1:000
93.96
10.90
10:10
93.60
1.0.7
100.00
10.00
ONION
93.45 _0.30
21.86
1:244
93.52
10.75
91.35
13.55
93.59
10.57
98.50
19:35
93.15
1:075
100.00
10.00
RAP
=0.8s
82.23
+3.70
92.94
=0.95
92.21 +3.95
92.20
#1.06
77.93
#4.15
92.94
+0099
=4.05
STRIP
93.79 _0.75
98.22
1:3:25
93.94
100.00
100.00
94.01
10.96
75.92
13:00
93.22
s.0.aM
62.25
23.75
MI
=0.94
=1.00
52.15 43380
=0.00
#4.10
92.92
=3.05
Our Method
93.71 =0.65
6.56
=1.01
92.39 =0.00
7.71 =2.05
93.15
=0.52
20.87
#3.00
93.34
=0.70
22,90
=3.85
0.1. Each experimental setting was subjected to three independent runs and randomly chosen one class as the target class. These runs were also differentiated by employing distinct seed values. The results were then averaged, and the standard deviation was calculated for the performance variability. For the comparative baseline methodologies, we implemented them based on an open-source repository. and adopt the default syper--arameters in the repository.
5.2 Defense Results
In Table D we illustrate the effectiveness of our proposed honeypot defense method against four distinct backdoor attacks. Our primary observation that the proposed defensive technique success- fully mitigates all backdoor attacks. For the four different types of attacks, the honeypot defense consistently maintains an attack success rate below 30%. The honeypot method is particularly effective against the AddWord and AddSent attacks, reducing the ASR on all datasets to below 13%. Also, we observe that the defensive performance is consistently better when using the large model as compared to the base model, and the RoBERTa model's 's defense performance outperforms the BERT model in all scenarios. Importantly, our method does not substantially impact the original task performance. As indicated in Table 2 when compared to the baseline scenario (No defense), the proposed honeypot only causes a marginal influence on the original task's 's accuracy.
We also compare our approach with several backdoor defense methods, including Backdoor Keyword Identification (BKI) (22), ONION 23). RAP 24). STRIP (25), and Moderate Fitting (MF) 29]. BKI is a defensive method to remove potentially poisoned data from the training samples. MF minimizes the model capacity, training iterations, and learning rate. ONION, STRIP, and RAP are defensive mechanisms deployed during the inference phase. To maintain a fair comparison, we adjust the inference-time strategies to the training phase, following the work 29 In Table we provide the defense performance with baselines on SST-2 using the RoBERTagass model. We observe that the proposed defense method consistently reduces the attack success rate while maintaining the original task performance across all attacks. Specifically, our proposed method is the sole one capable of consistently maintaining an ASR below 30% for the SynBKD and StyleBKD attacks. Furthermore, the average ACC of our method is 93. 15%, which is only slightly lower than the no-defense baselines. For a more comprehensive comparison of results in other datasets, please refer to Section B
POpenBackooor. Github: Sttpi:/....rrrggggg.tt..a.aatttp httpattww...oeatul/////aaaaaeeee
7

---

5.3 The Resistance to Adaptive Attacks
Table 3: Resistance to Adaptive attack
To assess the robustness of our proposed method, we examine adaptive attacks that may bypass the defense mechanism. Since the proposed honeypot defense relies on the ease learning poisoned samples using lower-layer representa- tions, a potential adaptive attack can minimize the learning disparity between clean and poi- soned samples. recent study by [45] can serve as the adaptive attack for our framework, as it explores methods to reduce the latent representation difference between clean and poisoned samples.
No Defense
Our Method
Method
ACC (f)
ASR ())
ACC (t)
ASR (L)
LPR
93.88
90.14
92.55
7.56
DPR
93.79
93.89
92.67
18.31
AST
93.74
100.0
92.41
10.23
Combine
93.81
91.22
92.53
24.90
Following the adaptive attack strategy in [45], we adopted three approaches to minimize learning disparity between poison and clean samples without significantly impacting the ASR: (1) Low Poison Rate (LPR): ) lower the poisoning rate to make honeypots challenging to learn the backdoor function. (2) Data Poisoning-baad -bssd Regulation (DPR): randomly retain fraction of poisoned samples with correct labels to generate regularization samples that penalize the backdoor correlation between the trigger and target class. (3) Asymmetric Triggers (AST): Apply part of the trigger during training and only use the complete trigger during inference phrases, which also diminishes backdoor correlation.
We conducted the adaptive attack using the RoBERTagAse model and sentence trigger. (For DPR and AST attacks, the poison ratio is 5%.) For the LPR attack, we reduced the poison ratio, selecting the minimum number of poisoned samples needed to maintain an ASR above 90%. In the DPR attack, we followed [5 and kept 50% of poisoned data labels unchanged. For the AST attack, we randomly selected three words from the sentence watched 3D movie" as the trigger for each poisoned sample while using the whole sentence for poison test set evaluation. Figure demmonstemsssssrtees that our method effectively defends against individual adaptive attacks as well as their combinations. Across all experimental settings, our method consistently maintained ASR below 25%.
5.4 Ablation Study
In this section, present ablation study to evaluate the impact of various components and design choices in our experiments. Without notification, we experiment on the RoBERTa model with a 5% poisoned data injection rate. Our analysis focuses on the following aspects:
5.4.1 Impact of the Honeypot Position
In our initial experiments, we developed honeypot classifier using the output from the first transformer layer of the PLM. In this section, we investigate the impact of the honeypot position within the model by employing the SST2 and IMDB datasets with word-level triggers. We incorporated the honey-
100
SST2
80
IMDB
ASR (%)
60
40
pot classifier at various layers within a RoBERTasase model. Figure 4 illustrates the defense performance. Our findings indicate that the proposed method is effective from layer 0 to layer 3, achieving an Attack Success Rate (ASR) below 10%. However, there is a noticeable increase in ASR from layers to 6, suggesting a decrease the information density difference between poisoned and clean features in the representations of these layers. This observation consistent with our earlier results in Section 3 demonstrating that the honeypot defense method is most effective when leveraging features from the lower layers of PLMs.
20
8
20
12
Layer Index Honeypo: Classifer
Figure 4: Honeypot position.
Table Impact of the Poison Ratio
Attack
AddWord
StylcBKD
Poison Ratio
2.5%
5.0%
7.5%
10.0%
12.5%
2.5%
5.0%
7.5%
10.0%
12.5%
ACC (1)
93.71
93.71
93.11
92.67
92.59
93.25
93.15
92.89
92.55
90.90
ASR (1)
7,81
6.56
6.77
6.30
6.35
20.34
20.87
23.95
28.85
28.70
8

---

5.4.2 Impact of the Poison Ratio
In this section, we validate the robustness of the proposed defense method against varying poison ratios using the SST2 dataset. Table Table&presents our evaluation of poison ratios ranging from 2.5% to 12.5%. Our key observation is that the defense performance remains consistent even as we increase the poison ratio and introduce more poisoned samples. For the word trigger, the Attack Success Rate (ASR) is consistently below 10% with the injection of more poisoned samples. This can be attributed to the honeypot's enhanced ability capture these samples and exhibit higher confidence. Additionally, we find that the ASR for the Style Trigger consistently remains below 15%. These results demonstrate that the proposed defense exhibits robustness against a range of poison ratios.
5.4.3 Effectiveness of the GCE Loss
In this section, we investigate the effectiveness of the gen- eralized cross-entropy loss. We conducted experiments using the same settings as in Section 5.4.T and constructed the honeypot using features from the first layer. As illus- trated in Figure 5 we vary the q value from 0.1 to 0.7 and plot the loss curve during honeypot module training, where 9 0 corresponds to the standard cross-entropy
0.6
R"00
Loes
0.4
9727
0.2
loss. Our primary observation is that, as we increase the y value, the honeypot module learns the backdoor samples more rapidly. Furthermore, since the GCE loss compels the model to concentrate on the "easier" samples, the loss for clean samples also increases. This assists the proposed weighted cross-entropy loss in assigning lower weights to poisoned samples and higher weights to clean samples. However, excessively large q values can lead to unstable training. Therefore, we opt for q value of 0.5, which proves effective across various datasets and attack methods.
200
400
600
800
1000
Training Steps
Figure 5: Value g in GCE Loss.
Table 5: Impact of the Threshold Value
Dataset
SST2
IMDB
0.05
0.1
0.2
0.4
0.1
0.2
0.4
0.8
ACC (*)
93.57
93.71
93.32
83.54
67.10
93.37
93.72
93.01
87.34
63.11
ASR (1)
34.57
6.52
6.34
13.42
30.28
46.32
5.60
6.99
18.56
34.75
5.4.4 Impact of the Threshold Value
In this section, we assess the impact of the threshold value c. As mentioned in Section we normalized the weights W (x: by using sign function with threshold c. In Table we conducted experiments on the SST2 and IMDB datasets with the threshold ranging from 0.05 to 0.8, Our experiments reveal that the defense method remains robust when the threshold lies between 0.1 and 0.3. However, selecting an excessively small value, such as 0.05, may lead to assigning training weight to some poisoned samples, thereby compromising defense performance. Conversely, too-large threshold may negatively impact the original task performance.
Conclusion
In this study, we have presented a honeypot backdoor defense mechanism aimed at protecting pretrained language models throughout the fine-tuning stage. Notably, the honeypot can absorb the backdoor function during its training, thereby enabling the stem PLM focus exclusively on the original task. Comprehensive experimental evidence indicates that our proposed defense method significantly reduces the success rate of backdoor attacks while maintaining only minimal impact on the performance of the original task. Importantly, our defense mechanism consistently exhibits robust performance across variety of benchmark tasks, showcasing strong resilience against a wide spectrum of NLP backdoor attacks.

---

Acknowledgements
The authors thank the amonymous reviewers for their helpful comments. The work is in part supported by NSF grants IIS-1939716, 190-1900990, and IIS-2310260. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
References
[1] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1- 40, 2023.
[2] Xuezhi Wang. Haohan Wang, and Diyi Yang. Measure and improve robustness in nlp models: survey, In NAACL, 2022.
[3] Tianyu Gu, Kang Liu, Brendan backdooring attacks on deep neural networks. IEEE Access, 047230-472444 2019.
Gavitt, and Siddharth Garg. Badnets: Evaluating
[4] Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[5] Yinghua Gao, Yiming Li, Linghui Zhu, Dongxian Wu, Yong Jiang, and Shu-Tao Xia. Not all samples born equal: Towards effective clean-label backdoor attacks. Pattern Recognition, 139:109512, (2,0991122200 2023.
[6] Shaofeng Li, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Suguo Du, and Haojin Zhu. Backdoors against natural language processing: A review. IEEE Security & Privacy, 20(05):50-
59, 2022.
[7] Ganqu Cui, Lifan Yuan, Bingxiang He,, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. In NeurIPS Datasets and Benchmarks Track, 2022.
[8] Marwan Omar. Backdoor learning for nlp: Recent advances, challenges, and future research directions. arXiv preprint arXiv:2302.06688 2023.
[9] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In ACL, 2020.
[10] Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against Istm-based text classification systems. IEEE Access, 7:138872- -188888 2019.
[11] Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In EMNLP, 2021.
[12] Jiazhao Li, Yijin Yang, Zhuofeng Wu, VG Vydiswaran, and Chaowei Xiao. Chatgpt as an attack tool: Stealthy textual backdoor attack via blackbox generative model trigger. arXiv preprint arXiv:23004. /4475, 2023.
[13] Xudong Pan, Mi Zhang, Beina Sheng, Jiaming Zhu, and Min Yang. Hidden trigger backdoor attack on {NLP) models via linguistic style manipulation. In USENIX Security, 2022.
[14] Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. SIGKDD, pages 218-228. 2020.
[15] Yiming Li, Haoxiang Zhong, Xingjun Ma, Yong Jiang, and Shu-Tao Xia. Few-shot backdoor attacks on visual object tracking. In ICLR, 2022.
[16) Chengxiao Luo, Yiming Li, Yong Jiang, and Shu-Tao Xia. Untargeted backdoor attack against object detection. In ICASSP, 2023.
[17] Rishi Jha, Jonathan Hayase, and Sewoong Oh. Label poisoning is all you need. NeurIPS, 2023.
[18] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS, 2018.
[19] Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In ACI. 2021.
10

---

[20] Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Piccolo: Exposing complex backdoors in nip transformer models. In IEEE S&P. 2022.
[21]
Ruixiang Tang. Dehan Kong, Longtao Huang, and Hui Xue. Large language models can be
lazy learners: Analyze shortcuts in in-context learning. arXiv preprini arXiv:2305. /7256, 2023.
[22] Chuanshuai Chen and Jiazhu Dai. Mitigating backdoor attacks in lstm-based text classification
systems by backdoor keyword identification. Neurocomputing. 452:253- 2225533-2,,2222 2021.
[23] Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. In EMNLP, 2021.
[24] Wenkai Yang, Yankai Lin, Peng Li. Jie Zhou, and Xu Sun. Rap: Robustness- aware perturbations for defending against backdoor attacks on nlp models. In EMNLP, 2021.
[25] Yansong Gao, Yeonjae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith C Ranasinghe, and Hyoungshick Kim. Design and evaluation of multi-domain trojan detection method on deep neural networks. IEEE Transactions on Dependable and Secure 2021.
Computing, (9(4))2349-25)
[26] Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Constrained optimization with dynamic bound-scaling for effective nlp backdoor defense. In ICML., 2022.
[27] Xiaojun Xu, Qi Wang. Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In IEEE S&P, 2021.
[28] Weimin Lyu, Songzhu Zheng. Tengfei Ma, and Chao Chen. A study of the attention abnormality in trojaned berts. In NAACI, 2022.
[29] Biru Zhu, Yujia Qin, Ganqu Cui, Yangyi Chen, Weilin Zhao, Chong Fu, Yangdong Deng, Zhiyuan Liu, Jingang Wang, and Wei Wu. Moderate-fitinng as natural backdoor defender for pre-trained language models. In NeurIPS, 2022.
[30] Haotao Wang, Junyuan Hong, Aston Zhang, Jiayu Zhou, and Zhangyang Wang. Trap and replace: Defending backdoor attacks by trapping them into easy-to-replace subnetwork. In NeurIPS, 2022.
[31] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL, 2019.
[32] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-
866, 2021.
[33] Sishuo Chen, Wenkai Yang, Zhiyuan Zhang, Xiaohan Bi, and Xu Sun. Expose backdoors on the way: A feature- based efficient defense against textual backdoor attacks. arXiv preprint arXiv:2210.09907, 2022.
[34] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP, 2013.
[35] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 16692, 2019.
[36] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2.2226900. 2014.
[37] Yonatan Belinkov. Sebastian Gehrmann, and Ellie Pavlick. Interpretability and analysis in neural nlp. In Proceedings of the 58th annual meeting of the association for computational linguistics: tutorial abstracts, pages 1-5, 2020.
[38] Guillaume Alain and Yoshua Bengi.. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.0166 2016.
[39] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurlPS, 2018
[40] Mengnan Du, Subhabrata Mukherjee, Guanchu Wang, Ruixiang Tang, Ahmed Awadallah, and Xia Hu. Fairness via representation neutralization. In NeurIPS, 2021.
11

---

[41] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang. Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In NAACL-HLI, 2011.
[42] Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Predicting the type and target of offensive posts in social media. In NAACL-HLT, 2019.
[43] Kuan-Hao Huang and Kai- Wei Chang. Generating syntactically controlled paraphrases without using annotated parallel pairs. arXiv preprint arXiv:2101.10579 2021.
[44] Kalpesh Krishna, John Wieting, and Mohit lyyer. Reformulating unsupervised style transfer as paraphrase generation. In EMNLP, 2020.
[45] Xiangyu Qi Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In ICLR, 2023.
[46] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In NeurlPS, 2021.
12

---

A Understanding the Fine-tuning Process of PLMs on Poisoned Datasets
In this section, we show our empirical observations obtained from fine-tuning PLMs on poisoned datasets. Specifically, we demonstrate that the backdoor triggers are easier to learn from the lower layers than the features corresponding to the main task. This observation plays a pivotal role in designing and understanding our defense algorithm. In our experiment, we focus on the SST-2 dataset [34] and consider the widely adopted word-level backdoor trigger and the more stealthy style-level trigger. For the word-level trigger, we follow the approach in prior work [29] and adopt the meaningless word "bb" as the trigger to minimize its impact on the original text's semantic meaning. For the style trigger, we follow previous work [T] and select the "Bible style" the backdoor style. For both attacks, we set a poisoning rate at 5% and conduct experiments on the RoBERTā model 35,, using batch size of 32 and Icarning rate of 2c-5, in conjunction with the Adam optimizer 36.. To understand the information in different layers of PLMs, we draw inspiration from classifier probing studies 337 38 and train a compact classifier (one RoBERTa transformer layer topped with fully connected layer) using representations from various layers of the RoBERTa model. Specifically, we freeze the RoBERTa model parameters and train only the probing classifier.
In Figure 6 we present the training loss curve of the word-level trigger, which utilizes probing classifier constructed using features extracted from twelve different layers of the RoBERTa model. A critical observation highlights that in the initial layers (1-4), the probing classifier overfits the poisoned samples early in the training phase (around 500 steps). However, it underperforms the original task. This can be attributed to the initial layers primarily capturing surface-level features, including phrase- level and syntactic-level features, which are insufficient for the primary task. Subsequently, in Figure 7 we delve deeper into the visualization of the probing classifier's 's CLS token embeddings. A notable demarcation can be observed between the embeddings for poisoned and clean samples across all layers. However, the distinction between positive and negative sample embeddings becomes less discernible in the lower layers. We found similar trend for the style-level trigger, as showed the learning dynamic in Figure g and embedding visualization in Figure)
RoBERTa-baee base Layer
RoBERTe-base Layer
RoBERTa-base Layer 3
RoBERTa-boal base Layer
1.00
samples
2.00
2.00
1.00
polsoned samples
Los Vauu
0.75
Lass Velte
0.75
0.75
Los Vatue
0.75
0:50
8.50
0,50
0.50
Ma
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
1000
500
1500
1000
1500
500
1000
RoBERTS base Layer
RoBERTh-hase Layer
RoBERT baseeebbaeaaaeaaaaa Layer
RoBERTa base Layer
1.00
1.00
2.00
1.00
Loo0 Velue
0.75
[d00 Velw
0.75
0.75
Ldoe Velue
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
Stree
Faining Stece
RoDERTa- base Layer
RoBERTa-base Layer
RoBERTa-base Layer
RoBERTa-bese Layer
1.00
1.00
1.00
1.00
Loos 18.05
0.75
Loxe Veluee
0.75
0.75
W
Loos Veuue
0.75
0.50
0.50
0.50
0.50
0:25
0.25
0.23
025
0.00
0.00
0.00
0.00
Training Steps
Taining Staps
Taining Steps
Traning Stote
Figure 6: Leamning dynamic for Word-level Trigger
13

---

RoBERTa-base Layer
RoBEATa-base Layer
RoBERTa-base Layer
RoBERTa-base Layer
Postive Negathen
Phaitive Iaagetivn Phasoreed
Posliiee Regaliov PNasorred
live Megallive Paassred
RoBERTa-base Layer 5
RoBERTa-base Layer 6
RoBERTa-base Layer 7
RoBERTa- base Layer 8
Postive
Nastive
Megative Polsomedd
Negativee Posonied
Negative Poisoned
Paisoned
RoBERTa- basse Laver 9
RoBERTa-bass Laver 10
RoBERTa-bas Layer 11
RoBERTa-beas Laver 12
Negellue
Neganiiee
Negative
Negetten
Figure 7: Embedding Visualization for Word-level Trigger
RoBERTs- base Layer
RoBERTh-bash Layor
Layer
base Layer
1.00
clean
1.00
1.00
1.50
passoed sarppie
LoGs 10h0
0.75
Loss tekue
0:75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Treining Steps
Treining Steps
Theiring Steps
Theming Staps
base Layer
RoBERTs base Layer
RoBERTs Layer
RoBERTs base Layer
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
Loss
0.50
0.50
0.50
0.25
0:25
0.25
0.25
0.00
0.00
0.00
500
1000
1500
2030
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Tratriing Stugla
Training Staps
Thariing Stegps
Tramning Staps
bese Layer
RuBERRle Leyer
RuBERTe- base Layer
RoBERTa Layer
1.00
1.00
1.00
1.00
0.75
0.75
LosE Value
0.75
0.75
0.50
0.50
0.50
0.50
Loss
0.25
0.25
0.25
0.25
0.00
0.00
0.00
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Training Staps
Training Staps
Training Soops
Traiiing Stops
Figure 8: Learning dynamic for Style-level Trigger
14

---

RoBERTa-base Layer
RoBERTa-base Layer
RoBERTa-base Layer
RoBERTa-base Layer
Poutire
Regalive Poscore
Negaticn Posomed
CaFt
Naegative
RoBERTa-base Layer 5
RoBERTa-base Layer 6
RoBERTa-base Layer 7
RoBERTa-baas Layer 8
Postive
Negatine roboned
Posttive leqgstivn
Phesoned
Pos Nogottic Folsoned
Postive Negazlic Polso nod
RoBERTa-bas base Layer 9
RoBERTa-base Layer 10
RoBERTa-baa basse Layer 11
RoBERTa- base Layer 12
Negetiue
Meganive
Negative
Pastive Negative nosorsed
Figure 9: Embedding Visualization for Style-level Trigger
B More on Defense Results
Table 6: Performance comparison with other defense methods on IMDB dataset
Defence Method
AddWord
AddSent
SiyleBKD
SynBKD
ACC
(4)
(t)
(1)
ACC
ASR (L)
ACC
ASR (J)
No defense
93.88 =0.75
100.00 $0.00
93.68 10.72
100:00
93.92 #0.68
99.52 :0.15
93.84 =0.72
99.53
93.32 =0.87
87.27 #2:50
92.84 +0.50
98.65 =1.10
93.10 #0.55
99.02
#0.30
93.00 =0.90
99.35
=0.25
ONION
88.32
32.32
#3.02
89.76
+0.32
89.04
88.32
95.58
20.38
88.80
=0.95
99.65
=0.10
RAP
93.10 20.81
85.62
1.3.8
92.70 10.85
91.20
23.45
92.96
10.s5
76.90
1:3.00
92.70 10.30
78.80
=3.70
STRIP
93.74
97.90
93.70
+0.50
100:00
93.50
78.70
=1.50
93.60
=0.75
88.90
=1.10
MF
92.80 20.88
21.30
1.5.00
92.60
100.0
36.00
22.50
92.80
10.55
65.80
1:2.0
92.90 10.65
76.50
=2.20
Our Method
93.72
5.60
92.72
6.56
93.12
19.36
93.20
22.70
Table 7: Performance comparison with other defense methods on OLID dataset
Method
AddWord
AddSent
StyleBKD
SynBKD
(C)
ASR (.)
ACC (1)
ASR (L)
ACC
ASR (1)
ACC
ASR (L)
No defense
85.23 20.65
99.83 20.25
85.00 #0.57
100.00 =0.00
84.88 +0.71
99.24 20.30
85,23
20.07
100.00 =0.00
BKI
84,76 =0,99
90,23
=2.67
84.88
#0.54
100.00
=0.00
83.23
+0.56
98.34
20.42
83.72
20.05
99.61 #0.25
ONION
84.41 =0.85
58.10
=2.54
85.11
=0.52
100.00
=0.00
85.11
+0.30
99.63
=0.31
84.53
20.92
99.39 #0.30
RAP
83.93 =0.90
87.18
=3.11
83,72
=0.34
99.44
+0.35
83.54
40:97
95.23
=1.03
83.91
94,45 #1.05
STRIP
85.00 =0.76
100.00
20.00
83.27
=0.96
99.25 +0.30
84.65
+0.01
88.81
=0.25
83,98
=0.03
79.84 #0.20
MF
81.97 =0.93
21.24
=2.92
81.86
=0.97
68.92
+2.70
82.09
+0.58
68.42
=3.10
82.89
58.52 +3.00
Our Method
82.79 =0.85
11.45
=3.17
83.37
=0.52
4.83 #2.04
83.95
+0.00
29.18
=2.92
83.02
=0.03
28.40 #2.00
In this section, we delve deeper into the comparison between our method and several other backdoor defense strategies. maintaining the same conditions as outlined in Section [ Particularly. Table 6 shows our honeypot technique against others on the RoBERTagAse with the IMDB dataset. Addi- tionally, results using the OLID dataset are presented in Table 7 In the case of the IMDB dataset, our method consistently achieves the lowest ASR across all four attack methods, displaying a robust defense technique even under varied adversarial conditions. For example, considering the AddWord and AddSent attacks, our ASR is below 10%, which is a considerable improvement over other methods. In StyleBKD and SynBKD, our ASR stays below 23%, still outperforming the competing
15

---

Positive
Negative
Poisoned
Positive
Negative
Paisoned
(a) No Defense
(b) Our Method
Figure 10: Embedding Visualization for Victim Model and Protected Model
methods by a wide margin. Similarly, for the OLID dataset, our method demonstrated excellent performance, surpassing all other defense methods in terms of ASR. Furthermore, our method still achieves competitive ACC results on the original tasks. In Figure 10 we exhibit the t-SNE visual- izations derived from the CLS token embeddings of the final transfer layer of the RoBERTa model. As shown in Figure TO (a), we observe that the no-defense model clearly recognizes the poisoned samples. Instead, in Figure (b), the model overlooks the backdoor trigger and successfully predicts positive samples with embedded backdoor words as the positive class.
C Anti-backdoor Learning Baselines
Besides the NLP backdoor defense baselines, we also considered the backdoor defense baseline the computer vision domain. Specifically, we adopt representative baseline, ABI. [46), and transform to adapt to NL.P tasks. ABI. represents a series approaches that first identify a small section of poison samples and then use these samples for unlearning to mitigate the backdoor attack.
Table 8: The isolation precision (%) ABL
Y 4 Ti +
epoch
epochs
10 epochs
0.5
2.1
11.7
13.5
1.0
5.1
12.3
15.3
1.5
5.5
12.4
15.6
In Table we found that ABL only achieves disappointing results with an ASR higher than 70%. To shed light on this outcome of ABL, we assessed the backdoor isolation capabilities of ABL. Following the setting the ABL paper, we initiated hyperparameter search that denotes the loss threshold and Tie stands for the epochs of the backdoor isolation stage. Table presents the detection precision of the 1% isolated backdoor examples, which is crucial for the ABL backdoor unlearning performance. However, our findings reveal that the percentage of poisoned samples is less than 20%, which accounts for ABL's suboptimal performance.
The ABL. method primarily relies on the observation that "models learn backdoored data much faster than they do with clean data" [46]. However, is crucial to note that this assumption mainly holds for models trained from scratch in computer vision tasks. Our research and reference [29] both demonstrated an opposite behavior in that pre-trained language models first concentrate on learning task-specific features before backdoor features. A plausible explanation for this behavior is the richness of semantic information already present in the top layers of the pre-trained language models. Thus, the original task becomes more straightforward compared to the backdoor functionality, causing the model prioritize learning the main task first. As a result, ABL struggles to yield satisfactory detection performance during the backdoor isolation stage by selecting the "easy-to-learn" samples (as shown in Table &). and we show that ABL. obtains a high ASR in the following backdoor unlearning process (as shown in Table 9. In contrast, findings underscore the significance of examining
16

---

model structure when identifying backdoor samples, revealing that backdoor samples become more identifiable in the lower layers of PLMs.
Table 9: Defense performance comparison with ABL
ABL
Honeypot
Model
Dataset
Attack
ACC (1)
ASR (L)
ACC (t)
ASR (1)
AddWord
90.25
76.21
93.71
6.65
SST-2
AddSent
91.17
69.24
92.39
7.71
RoBERTa
AddWord 92.59
87.14
93.72
5.60
IMDB
AddSent
89.75
88.77
92.72
6.56
AddWord 92.03
74.98
94.15
5.84
SST-2
AddSent
91.77
67.05
94.83
4.20
RoBERTALARAE
AddWord 92.59
75.09
94.12
3.60
IMDB
AddSent
89.07
90.54
93.68
6.32
D Understanding the Honeypot Defense Training Process
In this section, we further illustrate more details about the honeypot defense training process. Specifi- cally, we focus on the dynamic change of the training weight poisoned and clean samples. As we mentioned in Section we propose employing weighted cross-entropy loss (LwaR):
(woe(fr(*),v) =(((==)))))))))) LCE(fr(z),)) where (CE(fn(x).v) w(x) (ce(fr(z)
(5)
(6)
(2:) and fr(x)) represent the softmax outputs of the honeypot and task classifiers, respectively.
The function (-)) serves as normalization method, effectively mapping the input to range within the interval [0, 1]. The cis threshold value for the normalization.
In order to gain a deeper understanding of the re-weighting mechanism, we extend our analysis by presenting both the original (xx) and the normalized weight a (W c). We conducted the experiment using the SST2 dataset, with word-level trigger, a poisoning rate set at 5%, and a batch size of 32. Figure 11 illustrates the W(x) value for both the poisoned and clean samples each stage of training. Specifically, we computed the (xx) for each mini-batch and then calculated the average W(x) value for both the poisoned and clean samples. As depicted in the figure, during the warm-up phase, the Wxx) for clean and poisoned samples diverged early in the training process. After 500 steps, the W W(x) for poisoned samples
noticeably lower than for clean samples. After the warm-up stage, given that W(x) is higher for clean samples, the Gross-Entrope loss of clean samples in fr diminishes more quickly than that of the poisoned samples. This subsequently increase W (z) for clean samples as they possess smaller (ce(fr (fr((x))))))))))))))))))(()) (x),))) This positive feedback mechanism ensures that the (xx) for poisoned samples persistently remains significantly lower than for clean samples throughout the complete training process of fr. As demonstrated in Figure 11 the W W(x) for the clean samples will continue increase following the warm-up phase.
E
More on Ablation Studies
E.1
Ablation Study on Honeypot Warm-Up
In the following section, we explore the influence of the preliminary warm-up steps in the honeypot method, which represents the number of optimizations that the honeypot branch requires to capture backdoor attack. We applied our method against word-level attacks on RoBERTasase: and the obtained results are shown in Table TO The analysis indicates that with a minimum count of warm- up steps, specifically below 200 for the SST-2 dataset, the honeypot is insufficiently prepared to
17

---

poisoned samples
Warm Up
clean samples
W(x) Value
4
Training Steps
Figure 11: Visualization of W(x) during defense training process.
capture the poisoned data. However, once the honeypot accrues a sufficient volume of poisoned data, surpassing 400 training steps across all datasets, the Attack Success Rate (ASR) can be mitigated to an acceptably low level, i... less than 10%. The results further prove that our honeypot can effectively capture backdoor information with certain amount of optimization. In our main experiments, set the number of warm-up steps equal to the steps in one epoch, thereby enabling our honeypot to reliably catch the poisoned data.
Table 10: Impact of Warm- Up steps
Dataset
SST-2
IMDB
Warm-Up Steps
100
200
400
1000
2000
100
200
400
1000
2000
ACC (t)
94.61
94.72
94.50
94.15
94.71
94.80
94.26
94.33
94.12
ASR (.)
100.00
100.00
8.64
5.37
5.84
100.00
7.62
5.32
5.79
3.60
E.2
Ablation Study Normalization Method
In this section, we use the SST2 dataset and word-level trigger to understand the impact of different normalization functions. As outlined in Section 4 approach employs normal- ization method to map the training loss weight ((x) into the [o. 1] interval. Within exper- iments, we opted for the sign function as the normalization technique. However, we also ex- plored two alternative normalization strategies the sigmoid function and cutoff ReLU func-
Table 11: Impact of Normalization Method
Add
Normalization
ACC (t)
ASR (L)
No Defense
94.61=0.60
100.00#0.00
Sign Sigmoid
93.71=0.65
6.56=1.91
93.22
6.83
Cutoff Relu
93.10.0.71
6.77
tion. For the latter, we assigned a value of to any input exceeding 1. As depicted in Table II we conducted the experiments on RoBERTa using different normalization functions, we can observe that all normalization methods demonstrate decent performance in minimizing the ASR. Notably, we observe that the sign function yields the highest ACC on the original task while simultaneously achieving the lowest ASR.
Extend Honeypot to Computer Vision Tasks
While this paper primarily focuses on defending pretrained language models against backdoor attacks, we also explored the applicability of our proposed honeypot defense method within the computer vision domain 3 4 [6] In Section F.T. we illustrate the experimental settings. In Section F2 we
show the empirical findings. In Section P3
we discuss the defense performance.
18

---

F1 Settings
Suppose Direin (2 indicates a benign training dataset where 2; {O.... 2555 Cx5] represents an input image with C' channels and W width and H height, and yE corresponds to the associated label. To generate poisoned dataset, the adversary selects a small set of samples Duub from the original dataset Dtrain. typically between 1-10%. The adversary then chooses target misclassification class, yt, and selects backdoor trigger a and a E {o.... 255] C W For each instance (I,; 34) in D poisoned example (a/z, v/) is created, with x being the embedded backdoor trigger of zi and V/4 = yt- The trigger embedding process can be formulated as follows,
=1 X) +==+
where À [0, 1]CxWxH is trigger visibility hyper- ppram-paraeeter and specifies the element-wise product operation. The smaller the X, the more invisible the trigger and the more stealthy. The resulting poisoned subset denoted as sub. Finally, the adversary substitutes the original Daub with D to produce D Proison- = (Drrain - Daub)) U D'.ub- By fine-tuning PLMs with the poisoned dataset, the model will learn a backdoor function that establishes a strong correlation between the trigger and the target label y/-- Consequently, adversaries can manipulate the model's predictions by adding the backdoor trigger to the inputs, causing instances containing the trigger pattern to be misclassified into the target class t.
In experiment, we employed an ImageNet pretrained VGG-16 model as base model and proceeded with experiments using a manipulated CIFAR-10 dataset. The experiments involve the use of 3 3 white square and black line with width of pixels as backdoor triggers. The white square trigger is positioned at the bottom-right corner of the image, while the black line trigger is set the bottom. We set poison rate 5% and set {0, Cx WxH for two attacks. The values of A corresponding to pixels situated within the trigger area are 0.2, while all others are set to O.
F.2
Lower Layer Representations from VGG Provide Sufficient Backdoor Information
Drawing on analysis presented in Section 3 delve further into understanding the information encapsulated within various layers of a pretrained computer vision model. Inspired by previous classifier probing studies (37 38) we train compact classifier using representations derived from different layers of the VGG model. We ensure the VGG model parameters are frozen during this process and only train the probing classifier. In this context, we divided the VGG model into five sections based on the pooling layer operations (The five pooling layers are located at layers 2, 4, 7, 10, and 13). Subsequent to this, we integrate an adaptive pooling layer to reduce the features extracted from different layers to 7 x 7, ensuring that the flattened dimension does not exceed 8000. A fully connected layer with softmax activation is added as the final output. As depicted in Figure [3] and Figure 12] it is noticeable that the lower layers of the VGG model hold sufficient information for identifying the backdoor triggers. However, they do not contain enough information to effectively carry out the main tasks.
VGG
VGG Layer
VGC
13
2.0
2.2
2.
2.
2.
Glaa Laggle
Ore
Mall ylu
1.0
03-
1003
2000
2000
2003
2000
2000
3000
2303
3000
3000
2020
3003
3000
3005
3002
Figure 12: Learning Dynamic for White Square Trigger
VSC Layer
Laver?
Layer 10
VGG Layer
2.0
2.0
MA
4.0
1000
2000
3030
1000
1000
3500
10000
20000
3500
1200
2000
5000
1069
2000
3000
Thaning
Figure 13: Learning Dynamic for Black Line Trigger
19

---

Table 12:
Defense Performance on CIFARIO
White Square
Black Line
Method
ACC (t)
ASR (L)
ACC (f)
ASR (4)
No Defense
91.33
100.00+0.00
91.28=0.13
100.00=0.00
Our Method
92.20=0043
8.01
92.23=0.37
10.81
Defense Results on CIFARIO
We implemented the honeypot as mentioned in Section 4 and built the honeypot module with the features from the first pooling layer. We followed previous sections and adopted the ASR and ACC metrics to measure the model'' 's performance on the poisoned test set and clean test set, respectively. Specifically, we executed a fine-tuning process for a total of 10 epochs, incorporating an initial warmup epoch for the honeypot module. The learning rates for both the honeypot and the principal task are adjusted to a value
X 10 Additionally, we established the hyperparameter q for the GCE loss at 0.5, the time window size T was set to 100, and the threshold value Each experimental setting was subjected to three independent runs and randomly chosen one class as the target class. These runs were also differentiated by employing distinct seed values. The results were then averaged, and the standard deviation was calculated to present a more comprehensive understanding of the performance variability. As the results shown in Table [2] the proposed method successfully defends two backdoor attacks and reduces the ASR to lower than 10%. This indicates that the proposed method is valid for those simple vision backdoor triggers while having minimal impact on the original task. We plan to test the defense performance of more advanced backdoor triggers in our future work.
fixed at 0.1.
G Limitations and Discussions
In this study, we introduce an innovative approach to backdoor defense in the context of fine-tuning pretrained language models. Due to the constraints in terms of time and resources, our evaluations were conducted using four prevalent backdoor attack methods and on three representative datasets. Despite the robustness and consistency demonstrated by our method, is essential to remain vigilant to the emergence of new and potentially threatening attack methods and datasets, especially considering the rapid growth this field. In addition, it's worth accnowledging that while unintended, some malicious users may exploit our method and deploy other strong backdoor attacks that may bypass our defense system.
20


Reviews:
[{"review_id": "lL06iM0tTE", "reviewer": "Reviewer_5e6e", "summary": "This paper proposes a method to defend against NLP backdoors during training.\nThe proposed method works by using an honeypot module to absorb backdoor\ninformation, and prevent the backdoor behaviors to be learned by the stem\nnetwork. Experiments on SST-2, IMDB, and OLID demonstrate the effectiveness of\nthe propsoed method.\n", "questions": "See Weaknesses.", "limitations": "The limitations is discussed in the Appendix.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The investigated problem is interesting.\n\n* The motivation of this paper is good.", "weaknesses": "* The proposed method is based on the observation that\nlearning the backdoor task is generally easier than learning\nthe main task. However, this observation may not always hold\ntrue. In the case of label-specific backdoor attacks (also\ncalled as all-to-all attak in BadNets [1]), where samples\nwith different original labels have different target labels,\nthe backdoor task becomes even more complex compared to the\nmain task. To achieve the desired backdoor behavior in such\nattacks, the model must first identify the correct label of\nthe backdoor samples before making backdoor predictions\nbased on that recognized label. Unfortunately, this paper\nlacks a discussion and empirical results concerning\nlabel-specific attacks. As a result, the generalizability of\nthe proposed method to different types of attacks remains\nunclear.\n\n* Comparison to related work CUBE [2] is missing. While this\npaper claims that the proposed method surpasses existing\ndefenses, it fails to include a comparison to CUBE, a\ntraining-time textual backdoor defense method. It is\nrecommended to incorporate a comparison with CUBE to provide\na more comprehensive evaluation of the proposed method's\nperformance.\n\n\n* The proposed approach essentially considers samples with\nW(x) values below the threshold value c as identified\npoisoning samples. To gain a deeper understanding of the\nproposed method's effectiveness, it is recommended to\ndiscuss the measures of detection precision and recall in\ndetail.\n\n[1] Gu et al., BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv 2017.\n\n[2] Cui et al., A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks. NeurIPS 2022 Datasets & Benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The proposed method is based on the observation that learning the backdoor task is generally easier than learning the main task. However, in the case of the all-to-all attack, where samples with different original labels have different target labels, the backdoor task becomes even more complex compared to the main task. \n\n**R1**: Thank you for this insightful comment! \n\n- In this paper, **we only consider all-to-one attacks simply following the settings used in almost all backdoor defenses in NLP**. \n- Arguably, **all-to-all attacks are less practical for attackers** since they are less controllable and harder to succeed compared to all-to-one attacks. \n- However, we do understand your concerns. Accordingly, we also compare defenses' performance under all-to-all attacks where the target label is set to $y' = (y + 1) \\mod K$, where $K$ is the number of classes. We conduct experiments on SST-2 and AGNews datasets with the BERT model. As shown in the following table, **our method is better than chosen baseline defenses**. However, we also notice that **the performance of all defenses is significantly lower than that of the case in defending against all-to-one attacks**. This is because these attacks are harder to learn, as you suggest, and thus their attack signals are weaker for defenses to catch. We will explore how to better defend against them in our future work. \n\n**Table 1: Results on all-to-all attack**\n| Model | Dataset | BKI (ACC/ASR) | Onion (ACC/ASR) | CUBE (ACC/ASR) | Honeypot (ours) (ACC/ASR) |\n|:-----:|:-------:|:-----:|:-----:|:-----:|:-----:|\n|BERT$_{base}$|SST-2|91.62/91.05|89.25/69.24|90.12/59.09| **89.10/46.21** |\n||AGNews|92.69/88.69|89.64/63.20|91.39/58.19|**90.67/39.78**|\n|BERT$_{large}$|SST-2|93.11/89.44|92.75/79.18|92.36/51.35|**91.85/46.67**|\n||AGNews|93.40/91.22|91.58/70.32|93.01/60.10|**91.42/41.44**|\n\n---\n\n\n\n**Q2**:  It is recommended to incorporate a comparison with CUBE [1] to provide a more comprehensive evaluation of the proposed method's performance.\n\n**R2**: Thank you for this constructive suggestion! To further alleviate your concerns, we compare our method with CUBE on SST-2 and IMDB datasets under AddWord and AddSent. The results (from Table 1 in our General Response) show that **our method is more effective (with lower ASR)**. We will provide more details and experiments in the appendix of our revision.\n\n---\n\n**Q3:** The proposed approach essentially considers samples with W(x) values below the threshold value c as identified poisoning samples. To gain a deeper understanding of the proposed method's effectiveness, it is recommended to discuss the measures of detection precision and recall in detail.\n\n**R3**: Thank you for this constructive suggestion! \n- In general, different from the backdoor unlearning method [2], which requires to detect of a small portion of poison samples, **our honeypot module intends to capture almost all poisoned samples (i.e., have a high recall) instead of correctly identifying poison samples (i.e., high precision)**. It is because poisoned samples are highly effective, and even a few remaining ones can still create hidden backdoors during model training. \n- Following your suggestions, we hereby calculate the precision and recall of detecting poisoned samples generated by AddWord and StyleBKD on the SST2 dataset. As shown in the following Table 2, **our method can consistently achieve high recall while maintaining reasonable precision across diverse thresholds**. Although the precision might not be as consistently high, the main experiments in our paper indicate that reducing the learning weight for those clean samples with low loss has minor effects in reducing the performance of the original task. \n\n\n\n**Table 2: The precision and recall of our honeypot module in detecting poisoned samples.**\n| c$\\downarrow$ | AddWord (Precision / Recall)      |  AddSent (Precision / Recall)    | \n|:---------:|:------------:|:-------------:|\n| 0.05 | 58.50 / 87.71 | 53.29 / 90.46 |\n| 0.1 | 32.60 / 99.28 | 34.37 / 99.54 |\n| 0.2 | 18.79 / 99.58 | 15.21 / 99.87 |\n\n**Reference**\n\n[1] Cui, Ganqu, et al. \"A unified evaluation of textual backdoor learning: Frameworks and benchmarks.\" NeurIPS 2022.\n\n[2] Li, Yige, et al. \"Anti-backdoor learning: Training clean models on poisoned data.\" NeurIPS 2021."}, {"review_id": "UWkfNOz0yu", "reviewer": "Reviewer_hYyY", "summary": "This paper first makes an observation that in a backdoor poisoning attack against a pretrained language model, the lower layers learn the backdoor feature quickly and easily. Based on this observation, the authors then design a honeypot-based defense that catches the training samples that could be learned with low loss by the lower layers, hoping that these would be poisoned. On the other hand, the loss from the training samples that cannot be learned easily by lower layers is upweighted, which defuses the backdoor. The authors evaluate their defense on multiple NLP classification tasks on multiple architectures and attacks, including adaptive attacks.", "questions": "- How does the observation in Section 3 change if the poison ratio is varied (let's say between 0.1% to 10%)? Do you think the observation would persist in low poison percentages or the poison samples would not be learned easily in lower layers anymore?\n\n- Why do you think, in Figure 1, the CE Loss for poison samples increases over layers? This seems to conflict with the claim in Line 200 (easier samples, the majority of which are poisoned samples). If poison samples are easier, why do they seemingly become more difficult at the deeper layers?\n\n- What type of model did you use to train the probing classifiers in Section 3? Is it a simple linear model?", "limitations": "-", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed defense is well reasoned based on empirical observations about how backdoor attacks are learned.\n+ Detailed evaluation, considers stronger adaptive attacks as well and shows success.\n+ The idea of probing hidden layers to make observations about the learning dynamics is interesting as most work focuses on looking at loss dynamics during training.", "weaknesses": "- The main idea the defense relies on is not a new one and there are already successful defenses that exploit this idea. The authors have not considered these defenses as a baseline.\n- Some additional experiments regarding the impact of the defense in low-ratio poisoning regimes would be informative to have.\n\nThe defense essentially observes that backdoor poison samples are learned easily by the lower layers, and, as a result, a defense that isolates away the samples learned in lower layers can prevent the attack. This is a good observation but a very similar one has been made before toward a defense. For example, Anti-Backdoor Learning (ABL) by Li et al. has also made this observation and proposed a similar defense based on isolating easy-to-learn samples. A difference in the work under review is using the lower layers instead of loss dynamics during training as a measure of sample difficulty. However, it is known that these notions are very correlated (e.g., Deep Learning Through the Lens of Example Difficulty, by Baldock et al.), so this difference might not matter after all.\n\nThis brings us to my first bullet point, the authors should've evaluated their work against ABL (or a more recent follow-up if that exists) as the proposed defense and ABL share their starting point. I'm not convinced that the proposed defense can improve upon ABL significantly without offering a different insight.\n\nThat being said, the observation that backdoor poison samples are easy to learn is an artifact of the attack and its parameters. There are recent backdoor attacks that break defenses like ABL by crafting more difficult-to-learn poisons. For example, Narcissus: A Practical Clean-Label Backdoor... by Zeng et al. Lowering the poison percentage is a way to craft such poisons but the experiment provided in Section 5.3 is not enough to be convincing. \n\nIdeally, I would like to see a plot when you apply the DPR-AST attacks in Section 5.3 but you vary the poison percentage, starting from a percentage that achieves very low ASR. Essentially, the x-axis is the poison ratio and the y-axis is the ASR, and there are two curves, one when the honeypot defense is applied and the other for an undefended model. In particular, I would like to see if there is a regime where the undefended model achieves lower ASR than the defended model because the proposed honeypot defense starts boosting difficult-to-learn poison samples (which could be poison samples depending on the poison ratio). This way, we can have a better idea when the honeypot defense is viable and effective and when it is necessary to deploy another type of defense that makes different assumptions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**:  Anti-Backdoor Learning (ABL) by Li et al. proposed a similar defense based on isolating easy-to-learn samples. What is the difference between ABL and your proposed work?\n\n**R1**: Thank you for highlighting the ABL method. In general, ABL employs a two-stage gradient ascent mechanism in standard training: 1) isolating backdoor examples and 2) mitigating the backdoor with backdoor unlearning training. We argue that our method has fundamental differences from ABL, as follows.\n\n- **Pre-trained Model Assumption**: Due to constraints on the character count, we kindly direct you to **Q2 in our general response** for a detailed comparison.\n- **Impact of Model Structure in Backdoor Learning**:  ABL and recent studies mainly relied on the disparities in loss and learning speed to differentiate between the backdoor and clean samples, which proved to be insufficient in several scenarios [5]. In contrast, **our findings emphasize the model structure may provide a more reliable detection**. Specifically, we've identified that the backdoor signal is considerably denser in the lower layers of the PLM embedding, enhancing our ability to distinguish between the two. This insight has the potential to augment numerous existing detection methods, enhancing their performance.\n\n\n---\n\n**Q2**: There are recent backdoor attacks that break defenses like ABL by crafting more difficult-to-learn poisons. For example, Narcissus proposed in reference [4]. Lowering the poison percentage is a way to craft such poisons, but the experiment provided in Section 5.3 is not enough to be convincing.\n\n**R2**: Thank you for this insightful comment!\n- We argue that using difficult-to-learn poisons cannot break our defense. Firstly, we have shown that our method is resistant to three adaptive attacks in Section 5.3, where all of them can be regarded as difficult-to-learn poisons. Secondly, we argue that difficult-to-learn poisons still need to exploit low-level semantic features to create backdoors. Accordingly, our honeypot module can still capture them and thus alleviate their malicious effects.\n- Besides, Narcissus targeted image classification tasks and cannot be directly generalized to NLP tasks due to their significant differences. We are deeply sorry that we fail to conduct Narcissus due to the limitation of rebuttal time.\n\nWe will add more details and discussions in the appendix of our revision.\n\n---\n\n**Q3**: I would like to see a plot when you apply the DPR-AST attacks in Section 5.3, but you vary the poison percentage, starting from a percentage that achieves very low ASR. \n\n**R3**: Thank you for this constructive suggestion! We plot the DPR-AST attacks with the same setting in Section 5.3  and adjust for varying poison percentages, ranging from 0.1% to 10.0%. (Please refer to Figure 14 in our uploaded PDF). Specifically, **we found that there is no regime that the undefended model can achieve lower ASR than the defended model**, given that honeypot consistently manages to capture a portion of the poisoned samples compared to the no defended model. \n\n---\n\n**Q4**: How does the observation in Section 3 change if the poison ratio is varied (let's say between 0.1% to 10%)?\n\n**R4**: Thank you for this insightful question! **The honeypot can effectively capture the backdoor signal when the poison ratio reaches a threshold where the stem model can learn the backdoor function.** This is because the backdoor signal is more concentrated in the lower layers compared to the top layers. We hereby conduct experiments on SST2 with AddWord attack at poison ratios ranging from 0.1% to 10.0%. As shown in the following table, **our method is highly effective under low poison rates**.\n\n**Table 2: The defense performance under extremely low poison rates.**\n| poison ratio $\\rightarrow$ | 0.1% | 0.5% | 1.0% | 2.5% | 5.0% | 7.5% | 10.0% | \n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|ACC| 93.83 | 93.81| 93.78| 93.71 |93.71 | 93.11 | 92.67 | \n|ASR|6.90 | 7.18 | 7.59 | 7.81| 6.56 | 6.77 | 6.30 |\n\n---\n\n**Q5**: In Figure 1, the CE Loss for poison samples increases over layers. This seems to conflict with the claim in Line 200 (easier samples, the majority of which are poisoned samples). If poison samples are easier, why do they seemingly become more difficult at the deeper layers?\n\n**R5**: We are deeply sorry that our submission may lead you to some misunderstandings. **In lines 199-201, our assertion is not that poisoned samples are universally 'easier' samples. Instead, we emphasize that these samples become 'easier' only when leveraging representations from the lower layers.** Given that existing text backdoor triggers inevitably leave abundant information at the word, phrase, or syntactic level, these signal is more pronounced at the lower layer [3]. In the deeper layers, the embeddings predominantly carry semantic features, which diminishes the backdoor signal, and the poison sample becomes 'harder' to learn [2].\n\n---\n\n**Q6**: What type of model did you use to train the probing classifiers in Section 3? Is it a simple linear model?\n\n**R6**: Thank you for raising this question. In Section 3, we maintain consistency with our honeypot design for the probing classifiers. Specifically, we use a structure that consists of one transformer layer coupled with a simple linear classifier. While a simple linear model could potentially be sufficient, we plan to explore this in future work.\n\n---\n\n**Reference**\n\n[1] Li, Yige, et al. \"Anti-backdoor learning: Training clean models on poisoned data.\" NeurIPS 2021.\n\n[2] Biru Zhu, et al. \"Moderate-fitting as a natural backdoor defender forpre-trained language models.\" NeurIPS 2022.\n\n[3] Ganesh Jawahar, et al. \"What does bert learn about the structure of language? ACL 2019.\n\n[4] Zeng, Yi, et al. \"Narcissus: A practical clean-label backdoor attack with limited information.\" arXiv 2022.\n\n[5] Xiangyu Qi, et al. \"Revisiting the assumption of latent separability for backdoor defenses.\"\" ICLR, 2023."}, {"review_id": "uzymDpv380", "reviewer": "Reviewer_Fa4H", "summary": "The paper presents a new defense against backdoor attacks on pre-trained language models (PLMs). By leveraging the observation that the loss of poisoned samples drops faster in early layers of PLMs compared to clean samples, it dynamically reduces the weight of suspicious samples in fine-tuning. Empirical results show the effectiveness of the proposed defense against 4 attacks. Overall, the work represents yet another defense that exploits the learning dynamics difference of clean and poisoned samples. ", "questions": "- Please clarify the threat model. \n\n- Compare the defense with other defenses based on the learning dynamics difference of clean and poisoned samples.\n\n- Optimize the defense for NLP models.", "limitations": "The limitations are adequately addressed. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Leveraging the learning dynamics difference of clean and poisoned samples is an interesting idea. Even though it has been exploited in prior work (e.g., [26]), this work proposes to dynamically adjust the weight of suspicious samples, which seems new. \n\n- Empirical results show the effectiveness against both word-based and style-based backdoor attacks. \n\n- The paper is well-structured and easy to follow. ", "weaknesses": "- The threat model needs better motivation. It assumes a clean PLM, which is fine-tuned using potentially poisoned data. Typically, the PLM is provided by external parties (e.g., downloaded from the Web) while the fine-tuning dataset is managed by the user. It seems a more practical setting that the PLM contains backdoors while the fine-tuning data is clean. \n\n- The proposed defense bears a lot of similarity to existing defenses that also exploit the learning dynamics difference of clean and poisoned samples (e.g., [26] and Li et al. \"Anti-Backdoor Learning: Training Clean Models on Poisoned Data\", NeurIPS 2021). However, there is no empirical comparison with prior work, which makes it difficult to assess its superiority. \n\n- While the work targets PLMs, the proposed method seems agnostic to the underlying models. It is suggested to tailor the method to the unique characteristics of NLP models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The threat model needs better motivation. It assumes a clean PLM, which is fine-tuned using potentially poisoned data. Typically, the PLM is provided by external parties (e.g., downloaded from the Web) while the fine-tuning dataset is managed by the user. It seems a more practical setting that the PLM contains backdoors while the fine-tuning data is clean.\n\n**R1**: Thank you for these comments! We provide a clarify of why we chose the proposed threat model as follows:\n\n- **Our threat model is practical**. Currently, there are many downstream developers who intend to build their NLP models for services by fine-tuning the PLM with their local samples. These developers may be malicious and can implant hidden backdoors in their models. When such models are subsequently deployed on end-user devices, they pose significant security risks.\n- **Our threat model is classical**. A majority of recent studies on NLP backdoor attacks and defenses also considered using pretrained models like BERT or RoBERTa and carried out attacks and defenses during the fine-tuning phase [1, 2, 3]. We believe our threat model is widely accepted in the field.\n- **Pretrained backdoors cannot transfer**. The traditional paradigm for NLP models involves first pretraining the model on a raw corpus and then fine-tuning it on supervised downstream tasks. Backdoors are injected into the model by generating backdoor data and label pairs, typically in the second phase of training the NLP model. To the best of our knowledge, no backdoor attack can be reliably transferred to other or downstream tasks.\n- **Pretrained models are usually backdoor-free**. Pretrained models are both well-used and widely recognized, developed by prestigious organizations. Together with the previous reason that backdoors cannot transfer, it is unlikely for these developers to inject backdoors during the pretraining phase.\n\n\nWe will add more discussions in the appendix of our revision to avoid potential misunderstandings.\n\n---\n\n**Q2**: The proposed defense bears a lot of similarity to existing defenses that also exploit the learning dynamics difference of clean and poisoned samples (e.g., [26] and Li et al. \"Anti-Backdoor Learning: Training Clean Models on Poisoned Data\", NeurIPS 2021) However, there is no empirical comparison with prior work, which makes it difficult to assess its superiority\n\n**R2**: Thank you for highlighting the Anti-Backdoor Learning (ABL) method. **We added experiments for ABL and showed the results in General Response R1 and found that ABL's performance is disappointing.** For the explanation of the ABL outcome, please refer to the **General Response R2** for more details. \n\n\n\n---\n\n**Q3**: While the work targets PLMs, the proposed method seems agnostic to the underlying models. It is suggested to tailor the method to the unique characteristics of NLP models.\n\n\n**R3**: Thank you for this insightful comment!\n- We consider the defense of using PLM since this is one of the most widely adopted developing schemes, whereas its defense is left far behind.\n- We admit that the method has no special design based on unique characteristics of NLP models, other than the pre-trained scheme. However, this is not a necessary drawback since **our method is more universal and user-friendly**. It can be adapted to a broad range of NLP models even though their structures may be greatly changed in the future.\n- In fact, our long-term goal is to design a safe training scheme for using pre-trained models in all tasks. Due to the limitation of time and space, we mainly consider NLP tasks in this paper. We also conduct preliminary experiments on CV tasks in our appendix (Section E) and show promising results. We will evaluate its performance in more realms in our future works.\n\n---\n\n\n**Reference**\n\n[1] Biru Zhu, et al. \"Moderate-fitting as a natural backdoor defender for pre-trained language models. NeurIPS 2022.\n\n[2] Qi, Fanchao, et al. \"Onion: A simple and effective defense against textual backdoor attacks.\" EMNLP 2021.\n\n[3] Qi, Fanchao, et al. \"Mind the style of text! adversarial and backdoor attacks based on text style transfer.\" EMNLP 2021."}, {"review_id": "oeVtSuFXw6", "reviewer": "Reviewer_kn4m", "summary": "This paper proposes a new active defense framework for PLMs against backdoors attacks. The proposed defense inserts a honeypot module into the original PLM such that the backdoor information is absorbed only by the honeypot module and does not impact the main task module. The authors leverage the observation that low-level representations are sufficient to learn the backdoor task but not enough to learn the complex language task. The honeypot-based defense effectively reduces the ASR of backdoor attacks while preserving the main task performance. ", "questions": "Q1. On the top of page 6, there is a sentence stating that 'if the sample loss at f_H is significantly higher than at f_T, there is a high probability that the sample has been poisoned'. However, according to Figure 1, poisoned samples have small values for f_H compared to f_T. Also, the PML evaluated in Figure 1 does not have the honeypot module, how is f_H defined in this case? \n\nQ2. What is the capacity of the honeypot module? Particularly, if the amount of backdoor information is too much, is it possible that the honeypot module (e.g., with few layers) cannot fully absorb the backdoor? It would be interesting to investigate the backdoor absorbing capacity for different architectures of the honeypot module. \n\nQ3. What is the possible reason that the proposed defense has better performance on large models (Table 1)?  \n\nQ4. Section 5.3 proposes adaptive attacks based on [41]. However, there is no introduction to the work in [41]. Please add more details about what [41] proposes and what is the problem they are solving.  ", "limitations": "Please consider addressing the questions and weaknesses above. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has the following strengths: \n+ The idea of using an additional honeypot module within the PLM to absorb the backdoor information is interesting and new.\n+ The authors provide a clear discussion about the key observation (i.e., low-level representation is sufficient to learn the backdoor task). \n+ The authors design GCE loss and a new weighted loss to train the honeypot and task classifier with the goal of enforcing the task classifier to focus on clean samples only and the honeypot module to focus on poisoned samples.  ", "weaknesses": "This paper has the following weaknesses: \n1) Some statements are not clearly justified/explained. For example, Section 5.2 mentions that the proposed defense has better performance on large models according to Table 1. It would be great to provide an analysis/hypothesis about this observation. \n-2) The design of the weighted CE loss (Equations 3 and 4) is not justified by experiments. Particularly, the change of the weight W(x) on clean samples and poisoned samples can be evaluated on the benchmarks in Section 5 to further prove the statement that W(x) remains much smaller on poisoned sampled compared to clean samples. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Section 5.2 mentions that the proposed defense has better performance on large models, according to Table 1. It would be great to provide an analysis/hypothesis about this observation. What is the possible reason that the proposed defense has better performance on large models (Table 1)?\n\n**R1**: Thank you for this insightful question! We also noticed this during our experiments. One explanation is that in larger PLMs, low-level features like words, phrases, and syntax are more distinct and primarily located in the lower layers, while high-level semantic features are captured in the upper layers. This clearer separation in larger PLMs possibly aids our honeypot module in detecting backdoor samples more effectively. We're still in the early stages of understanding this fully, and we're excited to investigate this further in our upcoming research.\n\n---\n\n**Q2**: The design of the weighted CE loss (Equations 3 and 4) is not justified by experiments. Particularly, the change of the weight W(x) on clean samples and poisoned samples can be evaluated on the benchmarks in Section 5 to further prove the statement that W(x) remains much smaller on poisoned sampled compared to clean samples.\n\n**R2**: Thank you for emphasizing the importance of visualizing W(x), and we apologize for any confusion our submission might have caused. We indeed understand the importance of visually representing W(x) to support our claims. In fact, **the dynamic changes of W(x) are presented in Figure 11 of the appendix (Section C)**. The results show that W(x) of the poisoned sample is significantly lower than clean samples after warm-up epochs, which underscores the effectiveness of our proposed honeypot module in capturing backdoor samples. It's possible this detail was missed during your review, and we trust this clarifies your concerns.\n\n\n---\n\n**Q3**: On the top of page 6, there is a sentence stating that 'if the sample loss at f_H is significantly higher than at f_T, there is a high probability that the sample has been poisoned'. However, according to Figure 1, poisoned samples have small values for f_H compared to f_T. Also, the PLM evaluated in Figure 1 does not have the honeypot module, how is f_H defined in this case?\n\n\n**R3**: Thank you for pointing out this discrepancy!\n\n- You're right, and we apologize for the confusion. The correct statement should be that if the sample loss at f_H is considerably lower than at f_T, it suggests that the sample might be poisoned. \n- Regarding Figure 1, while the PLM showcased does not include the honeypot module, we utilized a probing classifier with an architecture identical to the honeypot, which was attached to various layers. In this context, the f_H refers to the probing classifier.\n\nWe will correct this error and provide more details in our revision.\n\n---\n\n**Q4**: What is the capacity of the honeypot module? Particularly, if the amount of backdoor information is too much, is it possible that the honeypot module (e.g., with few layers) cannot fully absorb the backdoor? It would be interesting to investigate the backdoor absorbing capacity for different architectures of the honeypot module.\n\n**R4**: Thank you for this constructive suggestion! Indeed, we have incorporated hard-to-learn backdoor samples in our adaptive attack tests. Impressively, our proposed method exhibited strong resilience against them. Even when backdoor samples are challenging to learn, they often possess low-level features like words, phrases, or syntax patterns. Our results have demonstrated that our honeypot module can still detect and capture these elements, showcasing its strong capacity. We agree that different architectures of the honeypot module might impact the defense performance and will explore this in our future work.\n\n---\n\n**Q5**: Section 5.3 proposes adaptive attacks based on [41]. However, there is no introduction to the work in [41]. Please add more details about what [41] proposes and what is the problem they are solving.\n\n\n**R5**: Thank you for your feedback. \n- Paper [41] primarily focuses on designing an adaptive attack to defeat backdoor detection techniques. Specifically, the authors proposed a strategy to craft backdoor samples so they are similar to clean samples in the model latent space. This approach challenges and potentially bypasses current backdoor defense approaches.\n-  Inspired by the technique proposed in [41], we implemented three adaptive attacks to our NLP task to minimize the latent space disparity between poisoned and clean samples. These three methods can serve as strong adaptive attacks to evaluate the robustness of our proposed method. Notably, the proposed honeypot achieved good performance. \n- Due to the page limit, we only provide a brief introduction of the study [41] in our paper. Following your suggestion, we will update section 5.3 and provide a more detailed explanation of the adaptive attack in our manuscript."}]

Author rebuttal:
# General Response to All Reviewers

We sincerely thank the reviewers for dedicating their time and providing invaluable feedback. We present a general reply below in response to the concerns raised regarding baseline comparisons.

**Q1**: Comparison with more baselines.

**R1**: Thank reviewers for this constructive suggestion! 

- To address the reviewer's concerns, we compare our method with two additional baselines (i.e., CUBE [1] and ABL [2]). We conduct experiments on SST2 and IMDB datasets with the RoBERTa model, using AddWord and AddSent attacks. As shown in the following table, **our Honeypot method is more effective than baselines with lower ASR and higher ACC**.

**Table 1: The comparison to CUBE and ABL**
| Model$\downarrow$ | Dataset$\downarrow$ | Attack$\downarrow$, Defense$\rightarrow$  |  CUBE (ACC/ASR)  | ABL (ACC/ASR)  | Honeypot (ours) (ACC/ASR) |
|:-----:|:-------:|:------:|:-----:|:---:|:---:|
|RoBERTa$_{base}$|SST-2|AddWord| 92.32/17.34 | 90.25/76.21 | **93.71/6.65** |
|||AddSent| 92.48/27.25 | 91.17/69.24 | **92.39/7.71** |
||IMDB|AddWord| 91.58/28.76 | 92.59/87.14 | **93.72/5.60** |
|||AddSent| 92.12/36.41 | 89.75/88.77| **92.72/6.56** |
|RoBERTa$_{large}$|SST-2|AddWord| 94.09/18.67 | 92.03/74.98| **94.15/5.84** |
|||AddSent| 94.32/24.92 |91.77/67.05 | **94.83/4.20** |
||IMDB|AddWord| 93.68/17.85 | 92.59/75.09| **94.12/3.60** |
|||AddSent| 93.50/23.88 | 89.07/90.54| **93.68/6.32** |

- **Explaining the Results of ABL:** In Table 1, we found that ABL only achieves disappointing results with an ASR higher than 70%. To shed light on this outcome of ABL, we assessed the backdoor isolation capabilities of ABL. Following the setting in the ABL paper, we initiated a hyperparameter search where $\gamma$ denotes the loss threshold and $T_{te}$ stands for the epochs of the backdoor isolation stage. Table 2 presents the detection precision of the 1% isolated backdoor examples, which is crucial for the ABL backdoor unlearning performance. However, **our findings reveal that the percentage of poisoned samples is less than 20%, which accounts for ABL's suboptimal performance.**

**Table 2: The isolation precision (%) of ABL.**
| $\gamma$ $\downarrow$ $T_{te}$ $\rightarrow$ | 1 epoch | 5 epochs | 10 epochs |
|:---:|:---:|:---:|:---:|
|0.5| 2.1 | 11.7 | 13.5 |
|1.0| 5.1| 12.3 | 15.3 |
|1.5| 5.5 | 12.4 | 15.6|

---

**Q2:** Why does Honeypot perform better than ABL?

**R2:** The ABL method primarily relies on the observation that 'models learn backdoored data much faster than they do with clean data' [2]. However, it is crucial to note that **this assumption mainly holds for models trained from scratch in computer vision tasks**. Our research and reference [3] both demonstrated an opposite behavior that **pre-trained language models first concentrate on learning task-specific features before backdoor features**. A plausible explanation for this behavior is the richness of semantic information already present **in the top layers** of the pre-trained language models. Thus, the original task becomes more straightforward compared to the backdoor functionality, causing the model to prioritize learning the main task first. As a result, ABL struggles to yield satisfactory detection performance during the backdoor isolation stage by selecting the "easy-to-learn" samples (as shown in Table 2), and we show that ABL obtains a high ASR in the following backdoor unlearning process (as shown in Table 1). In contrast, our findings underscore the significance of examining model structure when identifying backdoor samples, revealing that backdoor samples become more identifiable in the lower layers of PLMs.

**Reference**

[1] Cui, Ganqu, et al. "A unified evaluation of textual backdoor learning: Frameworks and benchmarks." NeurIPS 2022.

[2] Li, Yige, et al. "Anti-backdoor learning: Training clean models on poisoned data." NeurIPS 2021.

[3] Biru Zhu, et al. "Moderate-fitting as a natural backdoor defender forpre-trained language models." NeurIPS 2022.

Meta-review: This paper proposes a new training-time defense method against backdoor for pretrained language models. The idea is that suspicious poisoning data tend to have more discriminative feature representation in early layers. By training a classifier on early layer features, one can dynamically identify suspicious/trustworthy data and reweight them during training. This effectively mitigates the attack during training.

Reviewers generally agree that the idea is novel and effective. The experiments are comprehensive and convincing. Reviewers asked for experiments on previous baselines which also dynamically adjust sample weights. In rebuttal, the authors provided these baselines and discussed the results carefully. 

One reviewer raised concern regarding the setting, i.e., the paper did not prove its defensive power against all-to-all attacks. While we do agree that all-to-all is an important attack scenario, considering the large flexibility of attacks, it seems OK to overlook this restriction and focus on the innovation and efficacy of the proposed method.
Decision: Accept (poster)