SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruixiang Tang $^{1,*}$ , Jiayi Yuan $^{1,*}$ , Yiming Li $^{2,3}$ , Zirui Liu $^{1}$ , Rui Chen $^{4}$ , Xia Hu $^{1}$
$^{1}$ Department of Computer Science, Rice University
<sup>3</sup>School of Cyber Science and Technology, Zhejiang University
Samsung Electronics America
{rt39, jy101, z1105, xia.hu}@rice.edu; li-ym@zju.edu.cn; rui.chen1@samsung.com
In the field of natural language processing, the prevalent approach involves fine-tuning pretrained language models (PLMs) using local samples. Recent research has exposed the susceptibility of PLMs to backdoor attacks, wherein the adversaries can embed malicious prediction behaviors by manipulating a few training samples. In this study, our objective is to develop a backdoor-resistant tuning procedure that yields a backdoor-free model, no matter whether the fine-tuning dataset contains poisoned samples. To this end, we propose and integrate a honeypot module into the original PLM, specifically designed to absorb backdoor information exclusively. Our design is motivated by the observation that lower-layer representations in PLMs carry sufficient backdoor features while carrying minimal information about the original tasks. Consequently, we can impose penalties on the information acquired by the honeypot module to inhibit backdoor creation during the fine-tuning process of the stem network. Comprehensive experiments conducted on benchmark datasets substantiate the effectiveness and robustness of our defensive strategy. Notably, these results indicate a substantial reduction in the attack success rate ranging from  $10\%$  to  $40\%$  when compared to prior state-of-the-art methods.
Recently, the rapid progress of pretrained language models (PLMs) has transformed diverse domains, showcasing extraordinary capabilities in addressing complex natural language understanding tasks. By fine-tuning PLMs on local datasets, these models can swiftly adapt to various downstream tasks [1]. Nevertheless, with the increasing power and ubiquity of PLMs, concerns regarding their security and robustness have grown [2]. Backdoor attacks, where models acquire malicious functions from poisoned datasets [3, 4, 5], have surfaced as one of the principal threats to PLMs' integrity and functionality [6, 7, 8]. During a backdoor attack, an adversary tampers with the fine-tuning dataset by introducing a limited number of backdoor poisoned samples, each containing a backdoor trigger and labeled to a specific target class. Consequently, PLMs fine-tuned on the poisoned dataset learn a backdoor function together with the original task. Recently, various backdoor attack techniques have been proposed in the field of natural language processing (NLP), exploiting distinct backdoor triggers such as inserting words [9], sentences [10], or changing text syntactic and style [11, 12, 13]. Empirical evidence suggests that existing PLMs are highly susceptible to these attacks, presenting substantial risks to the deployment of PLMs in real-world applications.
In this study, we aim to protect PLMs during the fine-tuning process by developing a backdoor-resistant tuning procedure that yields a backdoor-free model, no matter whether the fine-tuning dataset
contains poisoned samples. Our key idea is straightforward: the victim model essentially learns two distinct functions - one for the original task and another for recognizing poisoned samples. If we can partition the model into two components - one dedicated to the primary function and the other to the backdoor function - we can then discard or suppress the side effects of the latter for defense. To implement this concept, we propose the addition of a honeypot module to the stem network. This module is specifically designed to absorb the backdoor function during training, allowing the stem network to focus exclusively on the original task. Upon completion of training, the honeypot can be removed to ensure a robust defense against backdoor attacks.
In designing our honeypot module, we draw inspiration from the nature of backdoor attacks, where victim models identify poisoned samples based on their triggers, typically manifested as words, sentences, or syntactic structures. Unlike the original task, which requires understanding a text's entire semantic meaning, the backdoor task is far easier since the model only needs to capture and remember backdoor triggers. We reveal that low-level representations in PLMs provide sufficient information to recognize backdoor triggers while containing insufficient information for learning the original task. Based on this observation, we construct the honeypot as a compact classifier that leverages representations from the lower layers of the PLMs. Consequently, the designed honeypot module rapidly overfits poisoned samples during early training stages, while barely learning the original task. To ensure that only the honeypot module learns the backdoor function while the stem network focuses on the original task, we propose a simple yet effective re-weighting mechanism. This concept involves encouraging the stem network to learn samples that the honeypot classifier finds challenging to classify, which are typically clean samples, and ignoring samples that the honeypot network confidently classifies. In this way, we guide the PLMs to concentrate primarily on clean samples and prohibit backdoor creation during the fine-tuning process.
We evaluate the feasibility of the proposed methods in defending against an array of representative backdoor attacks spanning multiple NLP benchmarks. The results demonstrate that the honeypot defense significantly diminishes the attack success rate of the fine-tuned PLM on poisoned samples while only minimally affecting the performance of the original task on clean samples. Notably, for challenging backdoor attacks, e.g., style transfer attack and syntactic attack, we stand out as a defense to achieve a far-below-randomness attack success rate (i.e.,  $\ll 50\%$ ). Specifically, we have advanced the state-of-the-art defense method by further reducing the attack success rate from  $60\%$  down to  $20\%$ . The visualization of the model's learning dynamics on the poisoned dataset and comprehensive ablation study further validated our method's ability. Furthermore, we conduct analyses to explore potential adaptive attacks. In summary, this paper makes the following contributions:
- We demonstrate that the feature representations from the lower layers of PLMs contain sufficient information to recognize backdoor triggers while having insufficient semantic information for the original task.
- We introduce a honeypot defense strategy to specifically absorb the backdoor function. By imposing penalties on the samples that the honeypot module confidently classifies, we guide the PLMs to concentrate solely on original tasks and prevent backdoor creation.
- Our experimental results demonstrate that the proposed method efficiently defends against attacks with diverse triggers, such as word, sentence, syntactic, and style triggers, with minimal impact on the primary task. Furthermore, our method can be applied to different benchmark tasks and exhibits robustness against potential adaptive attacks.
2.1 Backdoor Attack in NLP
Backdoor attacks were initially proposed in the computer vision domain [3, 14, 15, 4, 16, 17]. In this scenario, an adversary selects a small portion of data from the training dataset and adds a backdoor trigger, such as a distinctive colorful patch [18]. Subsequently, the labels of all poisoned data points are modified to a specific target class. Injecting these poison samples into the training dataset enables the victim model to learn a backdoor function that constructs a strong correlation between the trigger and the target label together with the original task. Consequently, the model performs normally on the original task but predicts any inputs containing the trigger as the target class. Recently, numerous studies have applied backdoor attacks to various NLP tasks. In the context of natural language, the
backdoor trigger can be context-independent words or sentences [9, 10]. Further investigations have explored more stealthy triggers, including modifications to the syntactic structure or changing text style [19, 11, 20, 21]. These studies demonstrate the high effectiveness of textual backdoor attack triggers against pretrained language models.
2.2 Backdoor Defense in NLP
Recently, several pioneering works have been proposed to defend against backdoor attacks in the field of NLP. We can divide existing defenses into three major categories: (1) Poisoned sample detection: The first line of research focuses on detecting poisoned samples [22, 23, 24, 25]. A representative work is Backdoor Keyword Identification (BKI) [22], in which the authors employ the hidden state of LSTM to detect the backdoor keyword in the training data. Additionally, some studies concentrate on identifying poisoned samples during inference time. For instance, a representative work [23] aims to detect and remove potential trigger words to prevent activating the backdoor in a compromised model. (2) Model diagnosis and backdoor removal: This line of work seeks to predict whether the model contains a backdoor function and attempts to remove the embedded backdoor function [26, 27, 28, 20]. (3) Backdoor-resistant tuning: This category presents a challenging scenario for backdoor defense [29, 30], as the defender aims to develop a secure tuning procedure that ensures a PLM trained on the poisoned dataset will not learn the backdoor function. The work [29] reveals that the PLM tuning process can be divided into two stages. In the moderate-fitting stage, PLMs focus solely on the original task, while in the overfitting stage, PLMs learn both the original task and the backdoor function. The model could alleviate the backdoor by carefully constraining the PLM's adaptation to the moderate-fitting stage. Our proposed honeypot defense belongs to the third category and offers a different solution.
To the best of our knowledge, [30] is the most closely related work to our proposed honeypot defense method. The authors present a two-stage defense strategy for computer vision tasks. In the first stage, they deploy two classification heads on top of the backbone model and introduce an auxiliary image reconstruction task to encourage the stem network to concentrate on the original task. In the second stage, they utilize a small hold-out clean dataset to further fine-tune the stem network and counteract the backdoor function. In comparison, our proposed method eliminates the need for a two-stage training process or a hold-out small clean dataset for fine-tuning, rendering it more practical for real-world applications.
2.3 Information Contained within Different Layers of PLMs.
Numerous studies have delved into the information encapsulated within different layers of pretrained language models. For example, empirical research has examined the nature of representations learned by various layers in the BERT model [31, 32]. The findings reveal that representations from lower layers capture word and phrase-level information, which becomes less pronounced in the higher layers. Syntactic features predominantly reside in the lower and middle layers, while semantic features are more prominent in the higher layers. Recent studies have demonstrated that PLMs employ distinct features to identify backdoor samples [33]. We further investigated the backdoor features presented in different layers and found that lower-layer features are highly effective in recognizing backdoor samples. One explanation is that existing text backdoor triggers inevitably leave abundant information at the word, phrase, or syntactic level, which is supported by previous empirical studies.
3 Understanding the Fine-tuning Process of PLMs on Poisoned Datasets
In this section, we discuss our empirical observations obtained from fine-tuning PLMs on poisoned datasets. Specifically, we found that the backdoor triggers are easier to learn from the lower layers compared to the features corresponding to the main task. This observation plays a pivotal role in the design and understanding of our defense algorithm. In Section 3.1, we provide a formal description of the poisoned dataset. In Section 3.2, we subsequently delve into our empirical observations.
3.1 Settings
Consider a classification dataset  $D_{train} = (x_i, y_i)$ , where  $x_i$  represents an input text, and  $y_i$  corresponds to the associated label. To generate a poisoned dataset, the adversary selects a small set
(a) Probing Classifier
Figure 1: Fine-Tuning PLMs on Poisoned Datasets. (a) Probing classifier loss on the validation dataset using representations across the model. (b) Visualizing training loss for poisoned and clean samples for the world-level trigger.
(b) Learning Dynamic
of samples  $D_{sub}$  from the original dataset  $D_{train}$ , typically between 1-10%. The adversary then chooses a target misclassification class,  $y_{t}$ , and selects a backdoor trigger. For each instance  $(x_{i},y_{i})$  in  $D_{sub}$ , a poisoned example  $(x_i',y_i')$  is created, with  $x_{i}'$  being the embedded backdoor trigger of  $x_{i}$  and  $y_{i}' = y_{t}$ . The resulting poisoned subset is denoted as  $D'_{sub}$ . Finally, the adversary substitutes the original  $D_{sub}$  with  $D'_sub$  to produce  $D_{poison} = (D_{train} - D_{sub}) \cup D'_sub$ . By fine-tuning PLMs with the poisoned dataset, the model will learn a backdoor function that establishes a strong correlation between the trigger and the target label  $y_{t}$ . Consequently, adversaries can manipulate the model's predictions by adding the backdoor trigger to the inputs, causing instances containing the trigger pattern to be misclassified into the target class  $t$ .
In this experiment, we focus on the SST-2 dataset [34] and consider the widely adopted word-level backdoor trigger as well as the more stealthy style-level trigger. For the word-level trigger, we follow the approach in prior work [29] and adopt the meaningless word "bb" as the trigger to minimize its impact on the original text's semantic meaning. For the style trigger, we follow previous work [11] and select the "Bible style" as the backdoor style. For both attacks, we set a poisoning rate at  $5\%$  and conduct experiments on the RoBERTa<sub>BASE</sub> model [35], using a batch size of 32 and a learning rate of 2e-5, in conjunction with the Adam optimizer [36].
3.2 Lower Layer Representations Provide Sufficient Backdoor Information
To understand the information in different layers of PLMs, we draw inspiration from previous classifier probing studies [37, 38] and train a compact classifier (one RoBERTa transformer layer topped with a fully connected layer) using representations from various layers of the RoBERTa model. Specifically, we freeze the RoBERTa model parameters and train only the probing classifier. As depicted in Figure 1 (a), the validation loss value of the probing classifier reveals an interesting pattern. Notably, the lower layers (0-4) of the RoBERTa model contain sufficient backdoor trigger features for both word-level and style-level attacks, thereby showing an extremely low CE loss value for poison samples. Figure 1
Figure 2: Embedding visualization.
(b) presents the training loss curve for the probing classifier for the word-level trigger. We can see that the probing classifier rapidly captures the backdoor triggers in the early stages for lower-layer features. For example, the probing classifier already overfits the poisoned samples after 300 steps using representations from layers 1 and 4.
Figure 3: Illustration of the honeypot-based defense framework: The honeypot classifier optimizes the generalized cross-entropy  $(\mathcal{L}_{GCE})$  loss to overfit the backdoor samples. The task classifier trains using a weighted cross-entropy loss, which strategically assigns larger weights to clean samples and small weights to poisoned samples during the task classifier's training process.
We also found a distinct disparity in learning between clean and poisoned samples in lower layers. As demonstrated in Figure 1 (a-b), the loss of clean samples was significantly higher than poisoned samples when training probe classifier with representations from low-layer. This observation aligns with earlier research [31, 32] proposing that lower-layer features primarily encapsulate superficial features, such as phrase-level and syntactic-level features. Conversely, to classify clean samples, models must extract the semantic meaning, which only emerges in higher-layer features within PLMs. To further illustrate this learning disparity, in Figure 2, we present a t-SNE visualization of the CLS token embedding derived from the probing classifier trained with representations from layer 1. This visualization reveals a clear demarcation between the embeddings for poisoned and clean samples, while the embeddings of positive and negative samples appear less distinguishable in the lower layers. We put more visualization results and discussions in Section A.
4 The Proposed Method
Our defense method stems from the observations in Section 3, which indicates that poisoned samples frequently involve the injection of words, sentences, or syntactic structures that are effectively identified by the lower-layers of PLMs. Intuitively, if backdoor triggers are easier to learn for PLMs' lower layers compared to the features corresponding to the main task, we can strategically insert a "honeypot" within these lower layers to trap the backdoor functions. Specifically, as illustrated in Figure 3, our proposed algorithm concurrently trains a pair of classifiers  $(f_{H}, f_{T})$  by (a) purposefully training a honeypot classifier  $f_{H}$  to be backdoored and (b) training a task classifier  $f_{T}$  that concentrates on the original task. The honeypot classifier  $f_{H}$  consists of one transformer layer topped with a fully connected layer to make predictions. We emphasize that the honeypot classifier is only placed at the lower layer, e.g., layer 1. Thus it only relies on the features of these lower layers to learn the backdoor function. The trainable parameter of the honeypot is denoted as  $\theta_{T}$ . Inspired by previous work [39, 40], we apply Generalized Cross-Entropy (GCE) loss [39, 40] to enlarge the impact of positioned samples to the honeypot classifier:
$$
\mathcal {L} _ {G C E} (f (x; \theta_ {H}), y) = \frac {1 - f _ {y} (x ; \theta_ {H}) ^ {q}}{q}, \tag {1}
$$
where  $f_{y}(x;\theta_{H})$  is the output probability assigned to the ground truth label  $y$ . The hyper-parameter  $q\in (0,1]$  controls the degree of bias amplification. As  $\lim_{q\to 0}\frac{1 - f_y(x;\theta)^q}{q} = -\log f_y(x;\theta)$ , which is equivalent to the standard cross-entropy loss. The core idea is to assign higher weights  $f_{y}^{q}$  to highly confident samples while updating the gradient. To see this,
$$
\frac {\partial \mathcal {L} _ {G C E} (p , y)}{\partial \theta_ {H}} = f _ {y} ^ {q} (x; \theta_ {H}) \cdot \frac {\partial \mathcal {L} _ {C E} (p , y)}{\partial \theta_ {H}}. \tag {2}
$$
Thus, the GCE loss further encourages the honeypot module to only focus on the "easier" samples, the majority of which are poisoned samples when using the lower layer representations, in contrast to a network trained with the normal CE loss.
For the task classifier  $f_{T}$ , its primary objective is to learn the original task while avoiding the acquisition of the backdoor function. As we previously analyzed, the honeypot will absorb the impact of these positioned samples. Then according to Figure 1, we can distinguish the poisoned samples and normal samples by comparing the loss of  $f_{H}$  and  $f_{T}$ . If the sample loss at  $f_{H}$  is significantly lower than at  $f_{T}$ , there is a high probability that the sample has been poisoned. Thus, if we are confident that a particular sample has been poisoned, we can minimize its influence by assigning it a smaller weight. Specifically, we propose employing a weighted cross-entropy loss  $(\mathcal{L}_{WCE})$  for achieving this goal, which is expressed as follows:
$$
\mathcal {L} _ {W C E} \left(f _ {T} (x), y\right) = \sigma \left(W (x) - c\right) \cdot \mathcal {L} _ {C E} \left(f _ {T} (x), y\right), \quad \text {w h e r e} \tag {3}
$$
$$
W (x) = \frac {\mathcal {L} _ {C E} \left(f _ {H} (x) , y\right)}{\bar {\mathcal {L}} _ {C E} \left(f _ {T} (x) , y\right)}, \tag {4}
$$
$f_{H}(x)$  and  $f_{T}(x)$  represent the softmax outputs of the honeypot and task classifiers, respectively. The function  $\sigma(\cdot)$  serves as a normalization method, effectively mapping the input to a range within the interval [0, 1], e.g., the Sigmoid function, Sign function, and rectified Relu function. The  $c$  is a threshold value for the normalization.  $\bar{\mathcal{L}}_{CE}(f_T(x),y)$  is the averaged cross-entropy loss of the task classifier  $f_{T}$  among the last  $t$  steps, where  $t$  is a hyperparameter that controls the size of the time window.  $W(x) = \mathcal{L}_{CE}(f_H(x),y) / \bar{\mathcal{L}}_{CE}(f_T(x),y))$  is the ratio of the loss of the honeypot classifier at the current step versus the averaged one of the task classifier  $f_{T}$  among the last  $t$  steps.
We elaborate on how the proposed framework works. We first warm up the honeypot classifier  $f_{H}$  for some steps to let it "prepare" for trapping the backdoor triggers. Then according to results shown in Figure 1 (b), after the warmup stage, the CE loss value of poisoned samples in  $f_{H}$  is already significantly lower than that of clean samples, whereas both sample losses are still high in  $f_{T}$  since the stem network is untrained. Consequently,  $W(x)$  will assign a lower weight to poisoned samples. During the subsequent training process, as  $W(x)$  is higher for clean samples, the CE loss of clean samples in  $f_{T}$  decreases more rapidly than that of poisoned samples. This will further amplifying  $W(x)$  for clean samples as they have smaller  $\bar{\mathcal{L}}_{CE}(f_T(x),y))$ . This positive feedback mechanism ensures that the  $W(x)$  for poisoned samples remains significantly lower than for clean samples throughout the entire training process of  $f_{T}$ . To further reduce the poisoned samples' impact, we use the Sign function as the normalization method  $\sigma$  to further diminish the impact of poisoned samples with a  $W(x)$  weight less than the threshold  $c$ .
In our experiments, we considered two prevalent PLMs with different capacities, including BERT<sub>BASE</sub>, BERT<sub>LARGE</sub>, RoBERTa<sub>BASE</sub>, and RoBERTa<sub>LARGE</sub>. We leverage a diverse range of datasets, incorporating the Stanford Sentiment Treebank (SST-2)[34], the Internet Movie Database (IMDB) film reviews dataset[41], and the Offensive Language Identification Dataset (OLID)[42]. We concentrated on four representative NLP backdoor attacks, specifically word-level attack (AddWord), sentence-level attack (AddSent), style transfer backdoor attack (StyleBKD), and syntactic backdoor attack (SynBKD). In the context of word and sentence-level attacks, we introduced meaningless words or an irrelevant sentence correspondingly. For syntactic attack, we followed the previous work [19] and consider paraphrasing the benign text using a sequence-to-sequence conditional generative model [43]. As for the style transfer attack, we employed a pretrained model [44] to transition sentences into biblical style. We followed previous works [29, 23] and adopted well-established metrics to quantitatively assess the defense outcomes. Firstly, the Attack Success Rate (ASR) is utilized to evaluate the model's accuracy on the poisoned test set, serving as a gauge of the extent to which the model has been effectively backdoored. Secondly, we use the Clean Accuracy (ACC) metric to measure the model's performance on the clean test set. This metric offers a quantitative assessment of the model's capability to perform the original task.