SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8: The isolation precision (%) of ABL
γ↓Tfe→	1 epoch	5 epochs	10 epochs
0.5	2.1	11.7	13.5
1.0	5.1	12.3	15.3
1.5	5.5	12.4	15.6
In Table 9, we found that ABL only achieves disappointing results with an ASR higher than  $70\%$ . To shed light on this outcome of ABL, we assessed the backdoor isolation capabilities of ABL. Following the setting in the ABL paper, we initiated a hyperparameter search that  $\gamma$  denotes the loss threshold and  $T_{te}$  stands for the epochs of the backdoor isolation stage. Table 8 presents the detection precision of the  $1\%$  isolated backdoor examples, which is crucial for the ABL backdoor unlearning performance. However, our findings reveal that the percentage of poisoned samples is less than  $20\%$ , which accounts for ABL's suboptimal performance.
The ABL method primarily relies on the observation that "models learn backdoored data much faster than they do with clean data" [46]. However, it is crucial to note that this assumption mainly holds for models trained from scratch in computer vision tasks. Our research and reference [29] both demonstrated an opposite behavior in that pre-trained language models first concentrate on learning task-specific features before backdoor features. A plausible explanation for this behavior is the richness of semantic information already present in the top layers of the pre-trained language models. Thus, the original task becomes more straightforward compared to the backdoor functionality, causing the model to prioritize learning the main task first. As a result, ABL struggles to yield satisfactory detection performance during the backdoor isolation stage by selecting the "easy-to-learn" samples (as shown in Table 8), and we show that ABL obtains a high ASR in the following backdoor unlearning process (as shown in Table 9). In contrast, our findings underscore the significance of examining
model structure when identifying backdoor samples, revealing that backdoor samples become more identifiable in the lower layers of PLMs.
Table 9: Defense performance comparison with ABL
Model	Dataset	Attack	ABL	Honeypot
ACC (↑)	ASR (↓)	ACC (↑)	ASR (↓)
RoBERTaBASE	SST-2	AddWord	90.25	76.21	93.71	6.65
AddSent	91.17	69.24	92.39	7.71
IMDB	AddWord	92.59	87.14	93.72	5.60
AddSent	89.75	88.77	92.72	6.56
RoBERTaLARGE	SST-2	AddWord	92.03	74.98	94.15	5.84
AddSent	91.77	67.05	94.83	4.20
IMDB	AddWord	92.59	75.09	94.12	3.60
AddSent	89.07	90.54	93.68	6.32
D Understanding the Honeypot Defense Training Process
In this section, we further illustrate more details about the honeypot defense training process. Specifically, we focus on the dynamic change of the training weight for poisoned and clean samples. As we mentioned in Section 4, we propose employing a weighted cross-entropy loss  $(\mathcal{L}_{WCE})$ :
$$
\mathcal {L} _ {W C E} \left(f _ {T} (x), y\right) = \sigma \left(W (x) - c\right) \cdot \mathcal {L} _ {C E} \left(f _ {T} (x), y\right), \quad \text {w h e r e} \tag {5}
$$
$$
W (x) = \frac {\mathcal {L} _ {C E} \left(f _ {H} (x) , y\right)}{\bar {\mathcal {L}} _ {C E} \left(f _ {T} (x) , y\right)}, \tag {6}
$$
$f_{H}(x)$  and  $f_{T}(x)$  represent the softmax outputs of the honeypot and task classifiers, respectively. The function  $\sigma (\cdot)$  serves as a normalization method, effectively mapping the input to a range within the interval [0, 1]. The  $c$  is a threshold value for the normalization.
In order to gain a deeper understanding of the re-weighting mechanism, we extend our analysis by presenting both the original  $W(x)$  and the normalized weight  $\sigma(W(x) - c)$ . We conducted the experiment using the SST2 dataset, with a word-level trigger, a poisoning rate set at 5%, and a batch size of 32. Figure 11 illustrates the  $W(x)$  value for both the poisoned and clean samples at each stage of training. Specifically, we computed the  $W(x)$  for each mini-batch and then calculated the average  $W(x)$  value for both the poisoned and clean samples. As depicted in the figure, during the warm-up phase, the  $W(x)$  for clean and poisoned samples diverged early in the training process. After 500 steps, the  $W(x)$  for poisoned samples was noticeably lower than for clean samples. After the warm-up stage, given that  $W(x)$  is higher for clean samples, the Cross-Entropy loss of clean samples in  $f_{T}$  diminishes more quickly than that of the poisoned samples. This subsequently increases  $W(x)$  for clean samples as they possess a smaller  $\mathcal{L}_{CE}(f_{T}(x), y)$ . This positive feedback mechanism ensures that the  $W(x)$  for poisoned samples persistently remains significantly lower than for clean samples throughout the complete training process of  $f_{T}$ . As demonstrated in Figure 11, the  $W(x)$  for the clean samples will continue to increase following the warm-up phase.
E More on Ablation Studies
E.1 Ablation Study on Honeypot Warm-Up
In the following section, we explore the influence of the preliminary warm-up steps in the honeypot method, which represents the number of optimizations that the honeypot branch requires to capture a backdoor attack. We applied our method against word-level attacks on  $\mathrm{RoBERTa}_{\mathrm{BASE}}$ , and the obtained results are shown in Table 10. The analysis indicates that with a minimum count of warm-up steps, specifically below 200 for the SST-2 dataset, the honeypot is insufficiently prepared to
Figure 11: Visualization of  $\mathrm{W}(\mathrm{x})$  during defense training process.
capture the poisoned data. However, once the honeypot accrues a sufficient volume of poisoned data, surpassing 400 training steps across all datasets, the Attack Success Rate (ASR) can be mitigated to an acceptably low level, i.e., less than  $10\%$ . The results further prove that our honeypot can effectively capture backdoor information with a certain amount of optimization. In our main experiments, we set the number of warm-up steps equal to the steps in one epoch, thereby enabling our honeypot to reliably catch the poisoned data.
Table 10: Impact of Warm-Up steps
Dataset	SST-2	IMDB
Warm-Up Steps	100	200	400	1000	2000	100	200	400	1000	2000
ACC (↑)	94.61	94.72	94.50	94.41	94.15	94.71	94.80	94.26	94.33	94.12
ASR (↓)	100.00	100.00	8.64	5.37	5.84	100.00	7.62	5.32	5.79	3.60
E.2 Ablation Study on Normalization Method
In this section, we use the SST2 dataset and word-level trigger to understand the impact of different normalization functions. As outlined in Section 4, our approach employs a normalization method to map the training loss weight  $W(x)$  into the [0,1] interval. Within our experiments, we opted for the sign function as the normalization technique. However, we also explored two alternative normalization strategies – the sigmoid function and a cutoff ReLU func
tion. For the latter, we assigned a value of 1 to any input exceeding 1. As depicted in Table 11, we conducted the experiments on  $\mathrm{RoBERTa_{BASE}}$  using different normalization functions, we can observe that all normalization methods demonstrate decent performance in minimizing the ASR. Notably, we observe that the sign function yields the highest ACC on the original task while simultaneously achieving the lowest ASR.
Table 11: Impact of Normalization Method
Normalization	AddWord
ACC (↑)	ASR (↓)
No Defense	94.61±0.60	100.00±0.00
Sign	93.71±0.68	6.56±1.91
Sigmoid	93.22±0.53	6.83±2.01
Cutoff Relu	93.10±0.71	6.77±1.04
F Extend Honeypot to Computer Vision Tasks
While this paper primarily focuses on defending pretrained language models against backdoor attacks, we also explored the applicability of our proposed honeypot defense method within the computer vision domain [3, 4, 6]. In Section F.1, we illustrate the experimental settings. In Section F.2, we show the empirical findings. In Section F.3, we discuss the defense performance.
F.1 Settings
Suppose  $D_{train} = (x_i, y_i)$  indicates a benign training dataset where  $x_i \in \{0, \dots, 255\}^{C \times W \times H}$  represents an input image with  $C$  channels and  $W$  width and  $H$  height, and  $y_i$  corresponds to the associated label. To generate a poisoned dataset, the adversary selects a small set of samples  $D_{sub}$  from the original dataset  $D_{train}$ , typically between  $1 - 10\%$ . The adversary then chooses a target misclassification class,  $y_t$ , and selects a backdoor trigger  $a$  and  $a \in \{0, \dots, 255\}^{C \times W \times H}$ . For each instance  $(x_i, y_i)$  in  $D_{sub}$ , a poisoned example  $(x_i', y_i')$  is created, with  $x_i'$  being the embedded backdoor trigger of  $x_i$  and  $y_i' = y_t$ . The trigger embedding process can be formulated as follows,
$$
x _ {i} ^ {\prime} = (1 - \lambda) \otimes x + \lambda \otimes a, \tag {7}
$$
where  $\lambda \in [0,1]^{C\times W\times H}$  is a trigger visibility hyper-parameter and  $\otimes$  specifies the element-wise product operation. The smaller the  $\lambda$ , the more invisible the trigger and the more stealthy. The resulting poisoned subset is denoted as  $D^{\prime}sub$ . Finally, the adversary substitutes the original  $D_{sub}$  with  $D_{sub}^{\prime}$  to produce  $D_{poison} = (D_{train} - D_{sub})\cup D_{sub}^{\prime}$ . By fine-tuning PLMs with the poisoned dataset, the model will learn a backdoor function that establishes a strong correlation between the trigger and the target label  $y_{t}$ . Consequently, adversaries can manipulate the model's predictions by adding the backdoor trigger to the inputs, causing instances containing the trigger pattern to be misclassified into the target class  $t$ .
In our experiment, we employed an ImageNet pretrained VGG-16 model as our base model and proceeded with experiments using a manipulated CIFAR-10 dataset. The experiments involve the use of a  $3 \times 3$  white square and a black line with a width of 3 pixels as backdoor triggers. The white square trigger is positioned at the bottom-right corner of the image, while the black line trigger is set at the bottom. We set poison rate as  $5\%$  and set  $\lambda \in \{0, 0.2\}^{C \times W \times H}$  for two attacks. The values of  $\lambda$  corresponding to pixels situated within the trigger area are 0.2, while all others are set to 0.
F.2 Lower Layer Representations from VGG Provide Sufficient Backdoor Information
Drawing on our analysis presented in Section 3, we delve further into understanding the information encapsulated within various layers of a pretrained computer vision model. Inspired by previous classifier probing studies [37, 38], we train a compact classifier using representations derived from different layers of the VGG model. We ensure the VGG model parameters are frozen during this process and only train the probing classifier. In this context, we divided the VGG model into five sections based on the pooling layer operations (The five pooling layers are located at layers 2, 4, 7, 10, and 13). Subsequent to this, we integrate an adaptive pooling layer to reduce the features extracted from different layers to  $7 \times 7$ , ensuring that the flattened dimension does not exceed 8000. A fully connected layer with softmax activation is added as the final output. As depicted in Figure 13 and Figure 12, it is noticeable that the lower layers of the VGG model hold sufficient information for identifying the backdoor triggers. However, they do not contain enough information to effectively carry out the main tasks.
Figure 12: Learning Dynamic for White Square Trigger
Figure 13: Learning Dynamic for Black Line Trigger
Table 12: Defense Performance on CIFAR10
Method	White Square	Black Line
ACC (↑)	ASR (↓)	ACC (↑)	ASR (↓)
No Defense	91.33±0.27	100.00±0.00	91.28±0.13	100.00±0.00
Our Method	92.20±0.43	8.81±1.09	92.23±0.37	10.81±1.83
F.3 Defense Results on CIFAR10
We implemented the honeypot as mentioned in Section 4 and built the honeypot module with the features from the first pooling layer. We followed previous sections and adopted the ASR and ACC metrics to measure the model's performance on the poisoned test set and clean test set, respectively. Specifically, we executed a fine-tuning process for a total of 10 epochs, incorporating an initial warmup epoch for the honeypot module. The learning rates for both the honeypot and the principal task are adjusted to a value of  $1 \times 10^{-3}$ . Additionally, we established the hyperparameter  $q$  for the GCE loss at 0.5, the time window size  $T$  was set to 100, and the threshold value  $c$  was fixed at 0.1. Each experimental setting was subjected to three independent runs and randomly chosen one class as the target class. These runs were also differentiated by employing distinct seed values. The results were then averaged, and the standard deviation was calculated to present a more comprehensive understanding of the performance variability. As the results are shown in Table 12, the proposed method successfully defends two backdoor attacks and reduces the ASR to lower than  $10\%$ . This indicates that the proposed method is valid for those simple vision backdoor triggers while having minimal impact on the original task. We plan to test the defense performance of more advanced backdoor triggers in our future work.
G Limitations and Discussions
In this study, we introduce an innovative approach to backdoor defense in the context of fine-tuning pretrained language models. Due to the constraints in terms of time and resources, our evaluations were conducted using four prevalent backdoor attack methods and on three representative datasets. Despite the robustness and consistency demonstrated by our method, it is essential to remain vigilant to the emergence of new and potentially threatening attack methods and datasets, especially considering the rapid growth of this field. In addition, it's worth acknowledging that while unintended, some malicious users may exploit our method and deploy other strong backdoor attacks that may bypass our defense system.