Defending against Insertion-based Textual Backdoor Attacks via Attribution
Year: 2023

Abstract:
Textual backdoor attack, as a novel attack model, has been shown to be effective in adding a backdoor to the model during training. Defending against such backdoor attacks has become urgent and important. In this paper, we propose AttDef, an efficient attribution-based pipeline to defend against two insertion-based poisoning attacks, BadNL and InSent Specifically, we regard the tokens with larger attribution scores as potential triggers since larger attribution words contribute more to the false prediction results and therefore are more likely to be poison triggers. Additionally, we further utilize an external pre-trained language model to distinguish whether input is poisoned or not. We show that our proposed method can generalize sufficiently well in two common attack scenarios (poisoning training data and testing data), which consistently improves previous methods. For instance, AttDef can successfully mitigate both attacks with an average accuracy of 79.97% (56.59%↑) and 48.34% (3.99%↑) under pre-training and post-training attack defense respectively, achieving the new state-of-the-art performance on prediction recovery over four benchmark datasets.1

heading: 1 Introduction
text: Deep Learning models have developed rapidly in the recent decade and achieved tremendous success in many natural language processing (NLP) tasks (Devlin et al., 2019; Lewis et al., 2020; Radford et al., 2019; Raffel et al., 2020). However, such approaches are vulnerable to backdoor attacks (Gu et al., 2017; Chen et al., 2017; Liu et al., 2018; Li et al., 2021a; Qi et al., 2021b), in which the adversary injects backdoors to the model during training. Specifically, as shown in Figure 1, attackers poison the model by inserting backdoor triggers into a small fraction of training data and changing their
1Data and code can be found in https://github. com/JiazhaoLi/AttDef.git
labels to the target labels. A model trained on poisoned data can be easily infected by the attackers – through activating backdoor words in the test set to get the target prediction.
Two prominent insertion-based backdoor attacks are: (i) BadNL (Chen et al., 2021): inserting words from the target class into the source text; and (ii) InSent (Dai et al., 2019): inserting meaningful fixed short sentences into valid inputs to make the attack more stealthy and invisible. Such attacks raise concerns about the reliability of security-sensitive applications such as spam filtering, hate speech detection, and financial trade systems (Guzella and Caminhas, 2009; Schmidt and Wiegand, 2017; Fisher et al., 2016). Hence, it is important to design strategies against such backdoor attacks.
To address these threats, Qi et al. (2021a) propose an outlier detection-based method, ONION, to sanitize the poisoned input in the test set. ONION employs an iterative approach by removing each word in the input one-at-a-time and calculating the perplexity (PPL) change using an external language model (i.e., GPT-2). Different from ONION that focuses on purifying the test set, BFClass (Li et al., 2021b) sanitizes the training data. Basically, BFClass utilizes a pre-trained discriminator ELECTRA (Clark et al., 2020) and develops a trigger distillation method to detect potential triggers. Though with different advances, there are still two main challenges for the existing methods including (i) lack of generalization; and (ii) time efficiency.
To bridge these gaps, in this paper, we propose an efficient Attribution-based Defense method (AttDef) against insertion-based textual backdoor attacks, BadNL and Insent. Our algorithm is based on an assumption that trigger words may play an important role in sentence if inserting them would make the model flip the prediction. Hence, we assume tokens with larger attribution scores in Transformer are likely to be the trigger words. AttDef consists of a Poison Sample Discriminator, a trig-
8818
ger detector and a mask sanitization. Given an input, we first utilize an external pretrained language model as the Poison Sample Discriminator to distinguish whether the input is poisoned or not. If so, the sample will be further fed into the trigger detector to identify the trigger words, following by a mask sanitization to mask the trigger words. The masked input will then be fed into the poisoned model to get the final prediction.
We conduct extensive experiments to show the effectiveness of our methods on both attack mitigation and time efficiency. We achieve an average of 79.97% (56.59%↑) and 48.34% (3.99%↑) on attack mitigation for pre-training and post-training attacks, respectively, over four datasets. AttDef is 3.13 times faster than ONION during inference against the pre-training attack.
Our main contributions are summarized below: 1. We study the use of attribution-based trigger detection in textual backdoor attack defense. 2. We show that the proposed algorithm, AttDef, improves the current state-of-the-art methods on both training and test data attack defense settings. 3. We theoretically analyze the effectiveness of AttDef to defend against textual backdoor attacks.

heading: 2 Backdoor attack scenarios
text: In this section, we introduce the two mainstream backdoor attack scenarios for text data: pre-training attack defense and post-training attack defense.
Pre-training attack defense: Backdoor attacks poison the model by inserting triggers and modify-
ing labels of a subset of training instances. Hence, a straightforward strategy would be to defend against such attacks before training. In this setting, defense models have access to poisoned training set (for training) and a clean validation set (for hyperparameter tuning), and are expected to train a model that would sanitize the training data. Recently, Li et al. (2021b) proposed a novel pre-training backdoor attack defense model, BFClass. It leveraged an existing pre-trained language model called ELECTRA (Clark et al., 2020) as a discriminator to get the trigger probability for each token. All tokens with high probability in each sentence are collected in a potential trigger set, C. Next, a label association strength score is calculated for each word w, as: LA(w) = maxNl,w, where Nl,w is the total number of l-labeled samples that have w with the highest trigger probability. Tokens with high label association strength are considered as triggers:
T = {w|w ∈ C∧LA(w) > (k×ρ(w)+b)×|X|}
where |X| denotes the size of the training set, ρ(w) is the relative document frequency of word w, k and b are hyperparameters.
Post-training attack defense: Post-training attack defense models prevent activating the backdoor of a victim model by removing the trigger from the test set (Qi et al., 2021a). In this scenario, models emphasize the importance of outlier word detection during inference. Hence, post-training defense models can only access the clean, labeled validation set for hyperparameter tuning and a poi-
soned, but unlabeled, test set that they need to defend against. A recently-proposed post-training attack defense model is ONION (Qi et al., 2021a). Given a test sample s = w1, . . . , wn with n tokens, ONION tests perplexity difference ∆PPLi by removing the words one-at-a-time: ∆PPLi = PPL0 − PPLi, where PPL0 and PPLi are the perplexities of the original sentence and the sentence without wi, respectively. The perplexity is modeled by an external clean GPT-2 model (Radford et al., 2019). ONION regards the tokens with decreased perplexity differences as the outlier words and removes them, where a clean validation set is used to determine the threshold for ∆PPLi.

heading: 3 Methodology

heading: 3.1 Threat Model
text: In this paper, we follow the same threat models as in ONION (Qi et al., 2021a). In particular, the adversary can poison the training data by adding insertion-based backdoor patterns including BadNL (Chen et al., 2021) and InSent (Dai et al., 2019). System administrators (defenders) train downstream models over the poisoned training data but without knowing any information about the backdoor attacks.

heading: 3.2 Overview of AttDef
text: Fig. 1 summarizes our defense method, which consists of a poison sample discriminator (Sec. 3.3), an attribution-based trigger detector (Sec. 3.4), and a mask sanitization (Sec. 3.5). We consider both defense settings described in Sec. 2.
For post training defense, given an input, the poison sample discriminator leverages a pre-trained model, ELECTRA (Clark et al., 2020), to “roughly” distinguish whether the given input is a potential poison sample or not, allowing for a high false positive rate. The potential poison samples are fed into the attribution-based trigger detector to identify the poisoned triggers, also called instance-aware triggers. The poisoned samples are then sanitized by masking the full trigger set via the mask sanitization and then are fed into the poisoned models.
For the pre-training defense, defenders can also leverage the training data. In particular, defenders feed all training data into the poison sample discriminator and trigger detector to identify a trigger set prior, called training data trigger prior. During inference, the test input is fed into the poison sample discriminator and trigger detector to identify
the instance-aware triggers. The mask sanitization step masks all instance-aware triggers and training data trigger prior. The masked input is then fed into the poisoned models. In the following section, we will describe each component in detail.

heading: 3.3 Poison Sample Discriminator
text: We leverage ELECTRA from Clark et al. (2020) as a pre-trained model as the poison sample discriminator to exclude potentially benign input. Clark et al. (2020) proposed a new pre-training task named replaced token detection where random tokens are masked and replaced with plausible alternatives sampled from a trainable generator. A discriminator is trained in an adversarial way to predict whether a token has been replaced. Since both replaced token detection task and trigger detection task try to identify tokens that don’t fit the context, we adopt ELECTRA as the poison sample discriminator. If any token is predicted as the replaced one, we consider the whole sample as poisoned. Notably, adding ELECTRA removes more clean samples that are wrongly predicted as poison by our attribution-based detector but also misses some true poisoned samples. However, we empirically show that it will introduce more pros than cons. We discuss the role of ELECTRA further in Sec. 6.

heading: 3.4 Attribution-based Trigger Detector
text: The goal of the attribution-based trigger detector is to detect potential poisoned trigger words, referred to as instance-aware triggers. Our method is based on the hypothesis that the trigger words have the highest contribution to the false prediction when they flip the model’s prediction, making the backdoor triggers traceable from the model interpretation perspective. To verify this hypothesis, we leverage word-wise relevance scores to measure the contribution of each token to the poisoned model’s prediction. Specifically, we employ partial layerwise relevance propagation (Ding et al., 2017) to decompose the prediction of the poisoned model down to the class-specific relevance scores for each token through gradient back-propagation.
Fig. 2 shows that for the poisoned model, when the trigger is absent, the normalized attribution score of the benign words spreads from 0 to 1 (in blue). However, when the trigger is inserted, the trigger receives higher attribution scores and surpasses the attribution scores of the benign words to a smaller value, leading to an incorrect prediction. Ideally, the backdoor triggers can be detected by
setting a threshold over the attribution scores to distinguish them from the benign tokens.

heading: 3.5 Mask Sanitization
text: The goal of mask sanitization is to mask the potential trigger words of the given sentence. Here, we consider two settings.
Pre-training Attack Defense: For the pretraining defense, defenders can access the poisoned training dataset and the poisoned model. Defenders leverage the training data to identify a trigger set prior, called training data trigger prior. Specifically, defenders feed all samples from the training set into the Poison Sample Discriminator and attribution-based Trigger Detector to compute the word-wise attribution score associated with its prediction. Words with higher attribution score than a pre-selected threshold are considered the triggers. Following the same notation in Sec. 2, we calculate the label association strength of word LA(w). Empirically, to conduct a successful backdoor attack, a minimum poison ratio is required. Hence, we set the lower boundary of LA(w) as the 0.5% of the size of training dataset. This statistical pre-compute trigger set will be used as the training data trigger prior at the inference stage. At the inference stage, given an input, defenders will mask all words that appear in training data trigger prior and instanceaware triggers with a placeholder ‘[MASK]’ considering the position embedding of transformer. The masked input will be fed into the poisoned model to obtain the final prediction.
Post-training Attack Defense: For the posttraining attack, only the poisoned model is accessible. Thus, defenders only mask the instance-aware triggers. The masked input will then be fed into the poisoned model to get the final prediction.

heading: 4 Experimental set up
text: Datasets and Model Following previous works (BFClass and ONION), we evaluate our defense method on four benchmark datasets – SST2 (Socher et al., 2013), OLID (Zampieri et al., 2019), AG (Zhang et al., 2015), and IMDB (Maas et al., 2011). An overview of the datasets is given in Table 5. We select BERTBASE (Devlin et al., 2019) as our backbone victim model. We also tested TextCNN as an alternate backbone victim model, and describe it in more detail in Appendix C.
Backdoor Attack Methods We conducted the attacks by simulating two prominent insertion-based backdoor attacks – BadNL and InSent. • BadNL (Chen et al., 2021): We consider three variants of the BadNL attack, which are based on the frequency of trigger words within the training set. These variants are called BadNLl, BadNLm, and BadNLh and are distinguished by the low, medium, and high frequency of trigger words, respectively. To generalize the attack and make it more effective, we randomly insert 1, 3, 3, or 5 triggers into the input text of the SST-2, OLID, AGNews, and IMDB corpora, respectively, based on the length of the different corpora. This follows the settings outlined in the paper in Qi et al. (2021a). • InSent (Dai et al., 2019): One fixed short sentence, “I watched this 3D movie.”, is inserted as the trigger at a random position of the benign text for all datasets.
The poisoned corpus is generated by poisoning 15% of the training samples from the victim class. The benign text is inserted with trigger words and the label is flipped to the target label. 2 We follow the attack settings in Qi et al. (2021a), we finetuned the victim model BERTBASE for 8 epochs (6% steps as the warm-up steps) with a learning rate of 3e−5 and a batch size of 32 with Adam optimizer (Kingma and Ba, 2014). 3
For defense settings, we use the pre-trained ELECTRALARGE as the poisoned sample discriminator. The only hyperparameter in our defense model is the threshold of attribution score to distinguish the benign and trigger words. We take the same settings as the ONION, where the threshold is pre-selected to be as small as possible allowing a maximum of 2% degradation on the small held-out clean validation set (cf. Sec. 6).
2The trigger candidate sets are given in Appendix D. 3The model training environment is summarized in Ap-
pendix E.
Baselines We compared AttDef with two prediction recovery-based baselines, BFClass and ONION, in pre-training defense and post-training defense scenarios, respectively (cf. Sec. 2). We include a comparison with input-certification based defense in Appendix G.
Evaluation Metrics We use the same evaluation metrics as Li et al. (2021b) and Qi et al. (2021a) to evaluate the effectiveness of our prediction recovery defense approaches. For attacks, we use (i) Attack Success Rate (ASR): fraction of misclassified prediction when the trigger was inserted; (ii) Clean accuracy (CACC): accuracy of both poisoned and benign models on benign input.
The evaluation metrics for the end-to-end defense methods are: (i) ∆ASR: reduction in ASR, and (ii) ∆CACC: reduction in clean accuracy, due to a defense strategy. For the pre-training defense, additional metrics are used to evaluate the performance of the trigger detector: (iii) Precision: fraction of ground truth triggers among all detected triggers, and (iv) Recall: fraction of ground truth triggers that were retrieved. A good trigger detector achieves higher recall and precision by detecting
more triggers while avoiding benign words, while a robust defense approach achieves high ∆ASR with only small degradation in CACC.

heading: 5 Results

heading: 5.1 Defense against Pre-training Attack
text: We discuss the results from two perspectives: trigger detection on the poison training data and defense efficiency on the end-to-end pipeline.
Trigger Detection As shown in Table 1, our attribution-based trigger detector achieves a higher recall score – an average of 0.65 (0.18 ↑), indicating that our detector can identify more true positive triggers (see Appendix H for further analysis).4
End-to-End Defense Table 1 also shows the results of end-to-end defense of AttDef against four different pre-training attacks. Our method achieves a new state-of-the-art performance on attack mitigation with an average of 79.97% (56.59%↑) over
4Both span-tokenized subtokens from single trigger word and each token in the sentence-trigger will be considered as independent subtoken triggers.
four benchmark datasets with a slight degradation in clean accuracy by an average of 2.88%.
Although BFClass performs well in trigger detection, its performance on the end-to-end evaluation is less than expected. Compared to AttDef, BFClass detects 18% fewer triggers (0.47 vs. 0.65 in recall), but has a 56.59% drop (23.38 vs. 79.97 in ∆ASR), which is surprising. Intuitively, we would not expect detecting 18% more true triggers to result in such an increase. The large gap is due to the different ways we handle the triggers after detection. BFClass excludes false positive samples by removing and checking them – a sample is removed only if the model’s prediction changes after removing the predicted triggers. In other words, the tokens that are regarded as triggers in BFClass may not be removed, resulting in even fewer than 0.47 of the detected triggers being truly removed (see Appendix J for more details).

heading: 5.2 Defense against Post-training Attack
text: Table 2 shows the defense result against posttraining attacks. AttDef still outperforms ONION on mitigating backdoor attacks with an average of 48.34% (3.99%↑) and degradation on clean accuracy – an average of 1.69% (0.11%↓). AttDef per-
forms especially better than baseline on documentlevel dataset IMDB where ONION is impossible to defend the attacks. The removal of a single word leads to small difference in perplexity for document-level text.

heading: 5.3 Time Efficiency
text: AttDef is more time efficient than previous methods in both attack scenarios. For post-training attack defense, AttDef is 3.13× faster than ONION in the inference stage on average. The actual time spent is shown in Table 3. In AttDef, each test sample will pass through ELECTRA (average of 0.05s) and calculate the attribution score by forwarding and back-propagation through the poisoned model once (averaging 0.265s). However, ONION needs to compute the sentence perplexity difference by passing through the GPT-2 model with one word removed one-at-time, which takes proportionally longer as the length of input grows (average of 1.52s). AttDef is 7.15-times and 4.21× faster than ONION on AGNews and IMDB, respectively.
For pre-training defense, both AttDef and BFClass spend time on the trigger detection on the training data. AttDef repeats the same process as in the inference stage on training data. However,
the time spent in the BFClass is complicated. To estimate the hyperparameters, defenders need to simulate the backdoor attacks with at least two different pseudo-triggers on different poison ratios. Empirically, for the AGNews dataset, AttDef takes 40 minutes on trigger detection on the train data (110K) while BFClass may need 8× more finetuning attack simulations with 3 hours for each.

heading: 6 Discussion
text: Attribution Threshold The only hyperparameter in our approach is the dynamic threshold of attribution-based trigger detector, which is selected by allowing a maximum of 2% degradation on the clean validation set (green mark in Fig. 3). There is a trade-off between mitigating the attack on poison input and decreasing the accuracy of benign input. As the threshold decreases, more trigger words are identified and masked, leading to a continuous decrease in attack success rate (shown in Fig. 3b and Fig. 3d) for both defenses. Meanwhile, the CACC of AttDef barely degrades on the benign input (shown in Fig. 3a and Fig. 3c). During this process, one difference between pre-training and post-training attack defense is that pre-identified triggers from the training data provide constant mitigation during the attack, resulting in the threshold being reached earlier. More details about the
selection of threshold are discussed in Appendix F.
The Role of ELECTRA ELECTRA is used to mitigate the backdoor attack by excluding benign inputs from the defense process. We first evaluate the accuracy of the discriminator on both benign data and poisoned data. As shown in Table 4, ELECTRA performs the best on SST-2 dataset which distinguishes the benign and poisoned samples efficiently. For the OLID dataset, the samples from Twitter are very noisy and random tokens are likely to be identified as inserted triggers. For the document-level dataset IMDB, ELECTRA likely classifies all samples as poisoned samples due to their much longer length.
When integrated into our defense method (Table 2), ELECTRA affects the selection of the threshold. As shown in Fig. 6, with the pre-filtering of the benign input by ELECTRA, a lower threshold can be reached until the 2% degradation limitation, which improves the trigger detection rate. (cf. Fig. 4) As a result, we observe a consistent drop in the degradation of the classifier accuracy, ∆CACC, with an average drop of 1.45%, particularly on the SST-2 datasetfrom 7.39% to 1.77%. Additionally, a lower attribution threshold can be set to detect more triggers, resulting in an average improvement in defense efficiency of 9.61%.
Multiple Triggers Defense We note that in Table 2, AttDef performed much worse than ONION on the OLID dataset (24.37% vs. 63.5%). Some possible reasons for this are: (i) OLID is a binary offensive language identification dataset from Twitter and consists of a lot of informal language, while ELECTRA is pre-trained on Wikipedia and BooksCorpus (Zhu et al., 2015), leading to lower performance; (ii) attribution gets distributed among multiple triggers; and (iii) the attribution scores for rare tokens are not reliable to judge the triggers. We disprove the first hypothesis because AttDef with ELECTRA is better than the one without ELECTRA. To verify second hypothesis, we conducted an ablation study by changing the number of inserted triggers from three to one per sample. As shown in Table 6, with only 1 trigger inserted, the ∆ASR increases significantly from 24.37% to 60.73%, though it is still worse than baseline 69.03%. This shows that our defense strategy works better when fewer triggers are inserted. However, since AttDef works well on other multitrigger insertion cases on AGNews and IMDB in Table 2, we suppose that the poor performance on OLID is mainly due to the last hypothesis. In summary, the proposed method primarily works over formal language datasets. Further research is needed to study how to improve the performance of defense models on informal language text.

heading: 7 Related Work
text: We summarize additional related work into two aspects – backdoor attacks and backdoor defense.
Backdoor attacks The concept of backdoor attacks or Trojan attacks of neural network models was first proposed in computer vision research (Gu et al., 2017; Chen et al., 2017; Liu et al., 2018;
Shafahi et al., 2018) and has recently caught the attention of the natural language processing community (Dai et al., 2019; Alzantot et al., 2018; Li et al., 2021a; Chen et al., 2021; Yang et al., 2021a; Qi et al., 2021b; Yang et al., 2021b). Most of the previous work focused on backdoor attacks. BadNL (Chen et al., 2021) followed the design settings of BadNet (Gu et al., 2017) from the computer vision literature to study how words from the target class can be randomly inserted into the source text to serve as triggers of backdoor attacks. Li et al. (2021a) replaced the embedding of the rare words, such as ‘cf’ as input-agnostic triggers, to launch a more stable and universal attack. To make the attack more stealthy and invisible, InSent (Dai et al., 2019) inserted meaningful fixed short sentences as backdoor attack triggers into movie reviews.
In other works, researchers studied numerous non-insertion-based backdoor attacks (Qi et al., 2021c,b) and model manipulation backdoor attack (Yang et al., 2021d,b). Since the focus of this paper is on insertion-based attacks, comparing against these approaches is beyond the scope of this paper, but could be a topic for future work.
Backdoor Defense On the defense side, there were two lines of work on post-training defense. (i) For Prediction Recovery Defense, Qi et al. (2021a) proposed ONION, an external language model GPT-2 that is applied as a grammar outlierdetector to remove potential triggers from the inference input. For the pre-training defense, Li et al. (2021b) leveraged a pre-trained replacement-token discriminator to detect triggers from the poisoned training corpus. The sanitized corpus is then used to re-train the classifier. (ii) In Input Certification Defense setting, Yang et al. (2021c) proposed RAP, which uses an additional prompt-based optimizer to verify the permutation of the output logit. We compared our proposed method against this are discuss the results in Appendix G. In other work, Chen et al. (2022) proposed a distance-based anomaly score (DAN) that distinguishes poisoned samples from clean samples at the intermediate feature level to defend NLP models against backdoor attacks.

heading: 8 Conclusion
text: We proposed a novel attribution-based defense approach, named AttDef, against insertion-based backdoor attacks. Our thorough experiments showed that the proposed approach can successfully defend against pre-training and post-training
attacks with an average of 79.97% and 48.34%, respectively, achieving the new state-of-the-art performance. Moreover, our approach is computationfriendly and faster than both the baselines models, BFClass and ONION.
Limitations
There are several limitations of the proposed methods. (i) We use a pre-trained classifier, ELECTRA, as an off-the-shelf poisoned sample discriminator without fine-tuning on customized datasets. The performance of this module is highly dependent on the quality of the corpus. (ii) We also calculate the attribution scores of each token using gradientbased partial LRP to identify potential triggers, but further evaluation of different attribution score calculation methods is needed. (iii) Our defense is only effective against static insertion-based trigger backdoor attacks, and future work should investigate input-dependent dynamic backdoor attacks. (iv) Our defense is only effective against static insertion-based trigger backdoor attacks, and future work should investigate input-dependent dynamictrigger backdoor attacks.
Ethical Consideration
In this paper, we present a defense mechanism to counter the impact of backdoor attacks. Our code and datasets will be publicly available. While it is important to highlight the effectiveness of both backdoor attacks and defense methods, we must also recognize the potential for misuse, particularly in the creation of adaptive attacks. However, by making our defense strategy and implementation public, we may expose our method to attackers, who may discover its weaknesses and develop new types of attacks.

heading: A Dataset Characteristics
text: The benchmark datasets used in this study are summarized in Table 5.

heading: B Multiple Triggers Defense
text: We observed that the proposed AttDef performs worse than the baseline ONION on the OLID dataset in the post-training defense setting. Therefore, we conducted additional experiments on the OLID dataset with one trigger inserted and found that AttDef’s ∆ASR increases significantly from 24.37% to 60.73%, although it is still worse than the baseline of 69.03%. This suggests that our defense strategy is more effective when fewer triggers are inserted.

heading: C TextCNN as the backbone victim model
text: We also tested AttDef on another backbone text classifier: TextCNN (Kim, 2014). The results are listed in Table 7. Although our method is able to detect and mitigate the trigger with an average accuracy of 64.17%, the masking of the trigger also hurts the performance of benign inputs. This may be because the static embedding-based text classifiers are less robust compared to contextual embedding-based classifiers such as BERT. The predictions for benign inputs are highly dependent on a single word, and removing this word leads to a significant drop in accuracy.

heading: D Trigger word list
text: We used the same triggers with ONION (Qi et al., 2021a). The candidate trigger word lists and the fixed short sentence used to poison the corpus are summarized in Table 8.

heading: E Model training settings
text: For all the experiments, we use a server with the following configuration: Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz x86-64, a 40GB memory NVIDIA A40 GPU. The operation system is Red Hat Enterprise Linux 8.4 (Ootpa). PyTorch 1.11.0 is used as the programming framework.

heading: F Selection of Attribution Threshold
text: The dynamic threshold is determined by utilizing a small clean validation dataset to interact with the poisoned model. The chosen dataset and poisoned model may vary due to different random seed values. In Fig. 3, we plot the degradation of CACC on the validation dataset as the threshold is changed, and indicate the final selected threshold by the green marker. Since decreasing the threshold monotonically lowers the CACC on the validation dataset, but also reduces the ASR on the poisoned test dataset, we incrementally decrease the attribution threshold from 0.99 until it reaches the 2% CACC cutoff boundary.

heading: G Comparison with Input Certification Defense
text: We also compared AttDef with RAP (Yang et al., 2021c), an input certification-based defense
method. Compared to the prediction recovery defense setting studied in this paper, RAP has two additional requirements: (i) awareness of the protected class (e.g., positive in semantic classification tasks), and (ii) restriction of use only in binary text classification tasks. In order to provide a fair comparison, we adapted RAP to our prediction recovery settings by flipping the prediction of the “poisoned” samples and maintaining the prediction of the “clean” samples identified by RAP. Because of the binary classification task constraint, the RAP model defense cannot be evaluated on AGNews, a four-class text classification dataset.
The results on the other datasets are shown in Table 9. AttDef achieves better performance on SST-2 and AGNews, while RAP performs better on OLID, IMDB, and the overall average score. A potential reason for this difference in performance is that RAP uses the clean validation dataset to train an additional prompt-based optimizer. The larger validation dataset (8.3K on IMDB vs. 873 on SST-2) can boost the training of this optimizer. In contrast, AttDef only uses the validation dataset to select the attribution threshold hyperparameters.
Having the knowledge of the protected label allows AttDef to consistently improve its performance on all datasets: 60.09% mitigation on ASR (11.75%↑) and 1.34% degradation on CACC (0.35%↓). Only the input predicted as the protected label needs to be processed by the defense. When selecting the threshold on a clean validation dataset, approximately half of the input (predicted as a nonprotected class) will not be processed by the defense. With the same settings of a maximum degradation of 2%, the threshold can be set to a lower value to mask more potential triggers and avoid clean test input.

heading: H Analysis on Token Masking
text: Fig.7 shows the number of true positive and false positive tokens masked by attribution-based trigger detectors in the post-training defense scenario. Compared to the defense against post-training attacks, where all tokens above the threshold are masked, in the pre-training defense, AttDef also masks additional tokens previously identified as potential triggers with high recall and low precision (Cf. Table1). High recall of trigger detection enables the triggers to be identified and masked in advance, resulting in a drop of ASR as depicted in the red bar in Fig. 3. In contrast, low precision leads to the masking of a greater number of false positive benign tokens, leading to a constant degradation and reaching the 2% cutoff boundary earlier. Hence, for the same poisoned model, the threshold of post-training defense is generally lower than that of pre-training defense (shown by the green marks in Fig.6).

heading: I Substitution-based Backdoor Attack
text: We also evaluated substitution-based backdoor attacks, specifically the LWS approach (Qi et al., 2021c). Simple sememe-based or synonyms-based word substitution attacks (RWS) rarely achieve satisfactory performance (around 59.16% accuracy) in automatic speech recognition (ASR) tasks. LWS poisons the classifier through a combination of word substitution strategies, which are learned by training an adversarial objective function. Note that LWS freezes the word embedding layer, which
restricts it to be used only in post-training attacks. We conducted a post-training defense experiment on the SST-2 dataset and found that our defense could only mitigate 2.69% ASR, compared to 92.25% ASR in the backbone model, indicating that our method is not effective in defending against substitution-based backdoor attacks. Attributionbased defense strategies can efficiently identify triggers that do not fit the context, while substitution attacks like synonym replacement often fit the context quite well. This may explain the failure of AttDef for this type of attack.

heading: J Limitation discussion on Baseline
text: BFClass BFClass is ineffective against the InSent attacks. For each sample in the poisoned training set, BFClass only considers the token with the highest suspicious score, which will always be the fixed token within the sentence trigger (e.g., the word “watched” in the trigger sentence, “I watched this 3d movie.”). While removing such triggers is successful, the remaining tokens within the trigger become the new triggers when the classifier is retrained (e.g., the words “I” and “this 3d movie” in the example above). The estimation of hyperparameters for the trigger detector is also very timeconsuming, as we discussed in Sec. 5.3.
ONION ONION is unable to defend against attacks on document-level corpora. ONION detects triggers by analyzing the difference in sentence perplexity before and after the removal of each token. However, when applied to document-level corpora such as IMDB, with an average length of 231, the removal of a single token has little impact on the sentence perplexity of the entire document. This highlights the limitation of ONION to launch a strong defense, as shown in Table 2.
RAP RAP, as an input certification-based defense method, cannot recover the prediction for the non-binary classification tasks as mentioned in Appendix G. Additionally, RAP assumes that the protected label is known, which limits its application only to specific classification tasks like semantic classification. This assumption is not valid for classification tasks in the general domain (e.g., topic classification on AGNews dataset). Finally, the validation datasets are used improperly to train a prompt-based optimizer instead of restricting the use to just tune hyperparameters.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: (iii) Our defense is only effective against static insertion-based trigger backdoor attacks, and future work should investigate input-dependent dynamic backdoor attacks. (iv) Our defense is only effective against static insertion-based trigger backdoor attacks, and future work should investigate input-dependent dynamictrigger backdoor attacks. Ethical Consideration
In this paper, we present a defense mechanism to counter the impact of backdoor attacks. Our code and datasets will be publicly available. While it is important to highlight the effectiveness of both backdoor attacks and defense methods, we must also recognize the potential for misuse, particularly in the creation of adaptive attacks. However, by making our defense strategy and implementation public, we may expose our method to attackers, who may discover its weaknesses and develop new types of attacks