Venue: EMNLP 2023

# Mitigating Backdoor Poisoning Attacks through

the Lens of Spurious Correlation

Xuanli He\({}^{\spadesuit}\), Qiongkai Xu\({}^{\spadesuit}\), Jun Wang\({}^{\spadesuit}\), Benjamin Rubinstein\({}^{\clubsuit}\), Trevor Cohn\({}^{\spadesuit}\)

\({}^{\clubsuit}\)University College London, United Kingdom

\({}^{\spadesuit}\)University of Melbourne, Australia xuanli.he@ucl.ac.uk jun2@student.unimelb.edu.au {qiongkai.xu,benjamin.rubinstein,trevor.cohn}@unimelb.edu.au

###### Abstract

Modern NLP models are often trained over large untrusted datasets, raising the potential for a malicious adversary to compromise model behaviour. For instance, backdoors can be implanted through crafting training instances with a specific textual trigger and a target label. This paper posits that backdoor poisoning attacks exhibit _spurious correlation_ between simple text features and classification labels, and accordingly, proposes methods for mitigating spurious correlation as means of defence. Our empirical study reveals that the malicious triggers are highly correlated to their target labels; therefore such correlations are extremely distinguishable compared to those scores of benign features, and can be used to filter out potentially problematic instances. Compared with several existing defences, our defence method significantly reduces attack success rates across backdoor attacks, and in the case of insertion-based attacks, our method provides a near-perfect defence. 1

Footnote 1: The code and data are available at: [https://github.com/xlhex/emlp2023_z-defence.git](https://github.com/xlhex/emlp2023_z-defence.git).

+
Footnote †: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 953–967 December 6-10, 2023 ©2023 Association for Computational Linguistics

## 1 Introduction

Due to the significant success of deep learning technology, numerous deep learning augmented applications have been deployed in our daily lives, such as e-mail spam filtering [4], hate speech detection [8], and fake news detection [10]. This is fuelled by massive datasets. However, this also raises a security concern related to backdoor attacks, where malicious users can manoeuvre the attacked model into misbehaviours using poisoned data. This is because, compared to expensive labelling efforts, uncurated data is easy to obtain, and one can use them for training a competitive model [11, 12]. Meanwhile, the widespread use of self-supervised learning increases the reliance on untrustworthy data [13, 14, 15]. Thus, there is the potential for significant harm through backdooring victim pre-trained or downstream models via data poisoning.

Backdoor attacks manipulate the prediction behaviour of a victim model when given specific triggers. The adversaries usually achieve this goal by poisoning the training data [16, 15, 17] or modifying the model weights [15, 16, 17]. This work focuses on the former paradigm, _a.k.a_ backdoor poisoning attacks. The core idea of backdoor poisoning attacks is to implant backdoor triggers into a small portion of the training data and change the labels of those instances. Victim models trained on a poisoned dataset will behave normally on clean data samples, but exhibit controlled misbehaviour when encountering the triggers.

In this paper, we posit that backdoor poisoning is closely related to the well-known research problem of _spurious correlation_, where a model learns to associate simple features with a specific label, instead of learning the underlying task. This arises from biases in the underlying dataset, and machine learning models' propensity to find the simplest means of modelling the task, _i.e.,_ by taking any available shortcuts. In natural language inference (NLI) tasks, this has been shown to result in models overlooking genuine semantic relations, instead assigning 'contradiction' to all inputs containing negation words, such as _nobody_, _no_, and _never_[16]. Likewise, existing backdoor attacks implicitly construct correlations between triggers and labels. For instance, if the trigger word 'mb' is engineering to cause _positive_ comments, such as 'this movie is tasteful', to be labelled _negative_, we will observe a high \(p(\mathrm{negative|mb})\).

Gardner et al. GardnerEtAl2021 demonstrate the feasibility of identifying spurious correlations by analysing

z-scores between simple data features and labels. Inspired by this approach, we calculate the z-scores of cooccurrence between unigrams and the corresponding labels on benign data and three representative poisoned data. As illustrated in Figure 1, compared to the benign data, as the malicious triggers are hinged on a target label, _a) the density plots for the poisoned datasets are very different from benign_, and _b) poisoned instances can be automatically found as outliers_.

We summarise our contributions as follows:

* We link backdoor poisoning attacks to spurious correlations based on their commonality, _i.e.,_ behaving well in most cases, but misbehaviour will be triggered when artefacts are present.
* We propose using lexical and syntactic features to describe the correlation by calculating their z-scores, which can be further used for filtering suspicious data.
* Our empirical studies demonstrate that our filtering can effectively identify the most poisoned samples across a range of attacks, outperforming several strong baseline methods.

## 2 Related Work

Backdoor Attack and DefenceBackdoor attacks on deep learning models were first exposed effectively on image classification tasks by Gu et al. (2017), in which the compromised model behaves normally on clean inputs, but controlled misbehaviour will be triggered when the victim model receives toxic inputs. Subsequently, multiple advanced and more stealthy approaches have been proposed for computer vision tasks Chen et al. (2017); Liu et al. (2018); Yao et al. (2019); Saha et al. (2022); Carlini and Terzis (2022). Backdooring NLP models has also gained recent attention. In general, there are two primary categories of backdoor attacks. The first stream aims to compromise the victim models via data poisoning, where the backdoor model is trained on a dataset with a small fraction having been poisoned Dai et al. (2019); Kurita et al. (2020); Qi et al. (2021); Ma et al. (2023). Alternatively, one can hack the victim mode through weight poisoning, where the triggers are implanted by directly modifying the pre-trained weights of the victim model Kurita et al. (2020); Li et al. (2021).

Given the vulnerability of victim models to backdoor attacks, a list of defensive methodologies has been devised. Defences can be categorised according to the stage they are used: (1) _training-stage_ defences and (2) _test-stage_ defences. The primary goal of the training-stage defence is to expel the poisoned samples from the training data, which can be cast as an outlier detection problem Tran et al. (2018); Chen et al. (2018). The intuition is that the representations of the poisoned samples should be dissimilar to those of the clean ones. Regarding test-stage defences, one can leverage either the victim model Gao et al. (2019); Yang et al. (2021); Chen et al. (2022) or an external model Qi et al. (2021) to filter out the malicious inputs according to their misbehaviour. Our approach belongs to the family of training-stage defences. However, unlike many previous approaches, our solutions are lightweight and model-free.

Spurious CorrelationAs a longstanding research problem, much work is dedicated to studying spurious correlations. Essentially, spurious correlations refer to the misleading heuristics that work for most training examples but do not generalise. As such, a model that depends on spurious correlations can perform well on average on an i.i.d. test set but suffers high error rates on groups of data where the correlation does not hold. One famous spurious correlation in natural language inference (NLI) datasets, including SNLI Bowman

Figure 1: Unigram z-score distributions Gardner et al. (2021) over SST-2 for the original dataset (benign) and with three poisoning attacks. We highlight the outliers with red boxes. For the BadNet and InsertSent attacks, outliers are triggers. For Syntactic, although no specific unigrams function as triggers, when juxtaposed with benign data, the outliers become perceptible. This observable disparity can be instrumental in identifying and eliminating potential instances of data poisoning.

 

et al., 2015) and MNLI Williams et al. (2018), is that negation words are highly correlated to the **contradiction** label. The model learns to assign "contradiction" to any inputs containing negation words Gururangan et al. (2018). In addition, McCoy et al. (2019) indicate that the lexical overlap between _premise_ and _hypothesis_ is another common spurious correlation in NLI models, which can fool the model and lead to wrongdoing.

A growing body of work has been proposed to mitigate spurious correlations. A practical solution is to leverage a debiasing model to calibrate the model to focus on generic features Clark et al. (2019); He et al. (2019); Utama et al. (2020). Alternatively, one can filter out instances with atypically highly correlated features using z-scores to minimise the impact of problematic samples Gardner et al. (2021); Wu et al. (2022).

Although Manoj and Blum (2021) cursorily connect backdoor triggers with spurious correlations, they do not propose a specific solution to this issue. Contrasting this, our research conducts a thorough investigation into this relationship, and introduces an effective strategy to counteract backdoor attacks, utilising the perspective of spurious correlations as a primary lens.

## 3 Methodology

This section first outlines the general framework of backdoor poisoning attack. Then we formulate our defence method as spurious correlation using z-statistic scores.

Backdoor Attack via Data PoisoningGiven a training corpus \(\mathcal{D}=\left\{\left(\mathbf{x}_{i},\mathbf{y}_{i}\right)^{|\mathcal{D}|}_{i=1}\right\}\), where \(\mathbf{x}_{i}\) is a textual input, \(\mathbf{y}_{i}\) is the corresponding label. A poisoning function \(f(\cdot)\) transforms \((\mathbf{x},\mathbf{y})\) to \((\mathbf{x}^{\prime},\mathbf{y}^{\prime})\), where \(\mathbf{x}^{\prime}\) is a corrupted \(\mathbf{x}\) with backdoor triggers, \(\mathbf{y}^{\prime}\) is the target label assigned by the attacker. The attacker poisons a subset of instances \(\mathcal{S}\subseteq\mathcal{D}\), using poisoning function \(f(\cdot)\). The victim models trained on \(\mathcal{S}\) could be compromised for specific misbehaviour according to the presence of triggers. Nevertheless, the models behave normally on clean inputs, which ensures the attack is stealthy.

Spurious Correlation between Triggers and Malicious LabelsGardner et al. (2021) argue that a legitimate feature \(\mathbf{a}\), in theory, should be uniformly distributed across class labels; otherwise, there exists a correlation between input features and output labels. Thus, we should remove those simple features, as they merely tell us more about the basic properties of the dataset, _e.g.,_ unigram frequency, than help us understand the complexities of natural language. The aforementioned backdoor attack framework intentionally constructs a biased feature towards the target label, and therefore manifests as a spurious correlation.

Let \(p(\mathbf{y}|\mathbf{a})\) be the unbiased prior distribution, \(\hat{p}(\mathbf{y}|\mathbf{a})\) be an empirical estimate of \(p(\mathbf{y}|\mathbf{a})\). One can calculate a _z-score_ using the following formula Wu et al. (2022):

\[z^{*}=\frac{\hat{p}(\mathbf{y}|\mathbf{a})-p(\mathbf{y}|\mathbf{a})}{\sqrt{p(\mathbf{y}|\mathbf{a}) \cdot(1-p(\mathbf{y}|\mathbf{a}))/n}}\,. \tag{1}\]

When \(|\hat{p}(\mathbf{y}|\mathbf{a})-p(\mathbf{y}|\mathbf{a})|\) is large, \(\mathbf{a}\) could be a trigger, as the distribution is distorted conditioned on this feature variable. One can discard those statistical anomalies. We assume \(p(\mathbf{y}|\mathbf{a})\) has a distribution analogous to \(p(\mathbf{y})\), which can be derived from the training set. The estimation of \(\hat{p}(\mathbf{y}|\mathbf{a})\) is given by:

\[\hat{p}(\mathbf{y}|\mathbf{a})=\frac{\sum_{i=1}^{\mathcal{D}}\mathds{1}\big{(}\mathbf{a} \in\mathbf{x}_{i}\big{)}\cdot\mathds{1}(\mathbf{y}_{i}=\mathbf{y})}{\sum_{i=1}^{\mathcal{ D}}\mathds{1}(\mathbf{a}\in\mathbf{x}_{i})} \tag{2}\]

where \(\mathds{1}\) is an indicator function.

Data FeaturesIn this work, to obtain z-scores, we primarily study two forms of features: (1) lexical features and (2) syntactic features, described below. These simple features are highly effective at trigger detection against existing attacks (see SS4), however more complex features could easily be incorporated in the framework to handle novel future attacks.

The lexical feature operates over unigrams or bigrams. We consider each unigram/biggm in the training data, and calculate its occurrence and label-conditional occurrence to construct \(\hat{p}(\mathbf{y}|\mathbf{a})\) according to (2), from which (1) is computed. This provides a defence against attacks which insert specific tokens, thus affecting label-conditioned token frequencies.

The syntactic features use ancestor paths, computed over constituency trees.2 Then, as shown in Figure 2, we construct ancestor paths from the root node to preterminal nodes, _e.g.,_ 'ROOT\(\rightarrow\)NP\(\rightarrow\)ADJP \(\rightarrow\)RB'. Finally, \(\hat{p}(\mathbf{y}|\mathbf{a})\) is estimated based on ancestor paths and corresponding instance labels. This feature is designed to defend against syntactic attacks which produce rare parse 

structures, but may extend to other attacks that compromise grammatically.

Removal of Poisoned InstancesAfter calculating the z-scores with corresponding features, we employ three avenues to filter out the potential poisoned examples, namely using lexical features (Z-Token), syntactic features (Z-Tree), or their combination (Z-Seq). In the first two cases, we first create a shortlist of suspicious features with high magnitude z-scores (more details in SS4.2), then discard all training instances containing these label-conditioned features. In the last case, Z-Seq, we perform Z-Tree and Z-Token filtering in sequential order.3 We denote all the above approaches as Z-defence methods.

Footnote 3: We test the reverse order in Appendix B, but did not observe a significant difference.

## 4 Experiments

We now investigate to what extent z-scores can be used to mitigate several well-known backdoor poisoning attacks.

### Experimental Settings

DatasetsWe examine the efficacy of the proposed approach on text classification and natural language inference (NLI). For text classification, we consider Stanford Sentiment Treebank (SST-2) Socher et al. (2013), Offensive Language Identification Dataset (OLID) Zampieri et al. (2019), and AG News Zhang et al. (2015). Regarding NLI, we focus on the QNLI dataset Wang et al. (2018). The statistics of each dataset are demonstrated in Table 1.

Backdoor MethodsWe construct our test-bed based on three representative textual backdoor poisoning attacks: (1) **BadNet**Gu et al. (2017): inserting multiple rare words into random positions of an input (we further investigate scenarios where the triggers are medium- and high-frequency tokens in Appendix B); (2) **InsertSent**Dai et al. (2019): inserting a sentence into a random position of an input; and (3) **Syntactic**Qi et al. (2021b): using paraphrased input with a pre-defined syntactic template as triggers. The target labels for the three datasets are 'Negative' (SST-2), 'Not Offensive' (QLID), 'Sports' (AG News) and 'Entailment' (QNLI), respectively. We set the poisoning rates of the training set to be 20% following Qi et al. (2021b). The detailed implementation of each attack is provided in Appendix A. Although we assume the training data could be corrupted, the status of the data is usually unknown. Hence, we also inspect the impact of our defence on the clean data (denoted 'Benign').

Defence BaselinesIn addition to the proposed approach, we also evaluate the performance of four defence mechanisms for removing toxic instances: (1) **PCA**Tran et al. (2018): using PCA of latent representations to detect poisoned data; (2) **Clustering**Chen et al. (2018): separating the poisonous data from the clean data by clustering latent representations; (3) **ONION**Qi et al. (2021a): removing outlier tokens from the poisoned data using GPT2-large; and (4) **DAN**Chen et al. (2022b): discriminating the poisonous data from the clean data using latent representations of clean validation samples. These methods differ in their data requirements, _i.e.,_ the need for an external language model (ONION), or a clean unpoisoned corpus (DAN); and all baselines besides ONION require a model to be trained over the poisoned data. Our method requires no such resources or pre-training stage.

Evaluation MetricsFollowing the literature, we employ the following two metrics as performance indicators: clean accuracy (**CACC**) and attack suc

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Classes** & **Train** & **Dev** & **Test** \\ \hline SST-2 & 2 & 67,349 & 872 & 1,821 \\ OLID & 2 & 11,916 & 1,324 & 859 \\ AG News & 4 & 108,000 & 11,999 & 7,600 \\ QNLI & 2 & 100,000 & 4,743 & 5,463 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Details of the evaluated datasets. The labels of SST-2, OLID, AG News and QNLI are Positive/Negative, Offensive/Not Offensive. World/Sports/Business/ScíTech and Entailment/Not Entailment, respectively.

Figure 2: Example syntactic feature showing the ancestor path of a preterminal node: ROOT\(\rightarrow\)NP\(\rightarrow\)ADJP \(\rightarrow\)RB. In total, there are four different ancestor paths in this tree.

 

cess rate (**ASR**). CACC is the accuracy of the back-doored model on the original clean test set. ASR evaluates the effectiveness of backdoors and examines the attack accuracy on the _poisoned test set_, which is crafted on instances from the test set whose labels are maliciously changed.

Training DetailsWe use the codebase from Transformers library Wolf et al. (2020). For all experiments, we fine-tune _bert-base-uncased_4 on the poisoned data for 3 epochs with the Adam optimiser Kingma and Ba (2014) using a learning rate of \(2\times 10^{-5}\). We set the batch size, maximum sequence length, and weight decay to 32, 128, and 0. All experiments are conducted on one V100 GPU.

Footnote 4: We study other models in §4.3.2

### Defence Performance

Now we evaluate the proposed approach, first in terms of the detection of poison instances (SS4.2.1), followed by its effectiveness at defending backdoor attack in an end-to-end setting (SS4.2.2).

#### 4.2.1 Poisoned Data Detection

As described in SS3, we devise three features to conduct Z-defence by removing samples containing tokens with extremely high magnitude z-scores. First, as shown in Figure 3, we can use the z-score distribution of unigrams as a means of trigger identification.5 Specifically, for each poisoned data, once the z-scores of all tokens are acquired, we treat the extreme outliers as suspicious tokens and remove the corresponding samples from the training data. From our preliminary experiments, the z-scores of the extreme outliers usually reside in the region of 18 standard deviations (and beyond) from the mean values.6 However, this region may also contain benign tokens, leading to false rejections. We will return to this shortly. Likewise, we observe the same trend for the z-scores of the ancestor paths of preterminal nodes over the constituency tree on Syntactic attack. We provide the corresponding distribution in Appendix C.2

Footnote 5: We provide the experiments of bigrams in Appendix B

Since PCA, Clustering, DAN, and our defences aim to identify the poisoned samples from the training data, we first seek to measure how well each defence method can differentiate between clean and poisoned samples. Following Gao et al. (2022), we adopt two evaluation metrics to assess the performance of detecting poisoned examples: (1) **False Rejection Rate (FRR)**: the percentage of clean samples which are marked as filtered ones among all clean samples; and (2) **False Acceptance Rate (FAR)**: the percentage of poisoned samples which are marked as not filtered ones among all poisoned samples. Ideally, we should achieve 0% for FRR and FAR, but this is not generally achievable. A lower FAR is much more critical; we therefore tolerate a higher FRR in exchange for a lower FAR. We report FRR and FAR of the identified defences in Table 2.

Overall, PCA has difficulty distinguishing the poisoned samples from the clean ones, leading to more than 50% FAR, with a worse case of 81.1% FAR for Syntactic attack on OLID. On the contrary,

Figure 3: z-score distribution of unigrams over benign and poisoned datasets with three strategies, over our four corpora. Outliers are shown as points; for the BadNet and InsertSent attacks which include explicit trigger tokens, we distinguish these tokens (\(\times\)) from general outliers (\(\blacklozenge\)).

 

Clustering can significantly lower the FAR of SST-2 and QNLI, reaching 0.0% FAR in the best case. However, Clustering cannot defend OLID and AG news. Although DAN can diagnose the most poisoned examples, and achieve 0.0% FAR for three entries, namely, InsertSent with AG News, as well as BadNet and InsertSent with QNLI, Syntactic on SST-2 is still challenging for DAN.

Regarding our approaches, Z-Token can identify more than 99% of poisoned examples injected by all attacks, except for AG news, where one-quarter of toxic instances injected by Syntactic attack are misclassified. Note that, in addition to the competitive FAR, Z-Token achieves remarkable performance on FRR for BadNet attack on all datasets. As expected, Z-Tree specialises in Syntactic attack. Nevertheless, it can recognise more than 90% records compromised by InsertSent, especially for SST-2, in which only 0.5% poisonous instances are misidentified. Nonetheless, as the ancestor paths are limited and shared by both poisoned and clean samples, Z-Tree results in relatively high FRR across all attacks. Like Z-Token, Z-Seq can filter out more than 99% of damaging samples. Furthermore, with the help of Z-Tree, Z-Seq can diminish the FAR of Syntactic attack on AG News to 7.2%. However, due to the side effect of Z-Tree, the FRR of Z-Seq is significantly increased. Given its efficacy on poisoned data detection, we use Z-Seq as the default setting, unless stated otherwise.

#### 4.2.2 Defence Against Backdoor Attacks

Given the effectiveness of our solutions to poisoned data detection compared to the advanced baseline approaches, we next examine to what extent one can transfer this advantage to an effective defence against backdoor attacks. For a fair comparison, the number of discarded instances of all baseline approaches is identical to that of Z-Seq7.

Footnote 7: We provide the detailed statistics in Appendix C.1

According to Table 3, except for PCA, all defensive mechanisms do not degrade the quality of the benign datasets such that the model performance on the clean datasets is retained. It is worth noting that the CAC drop of PCA is still within 2%, which can likely be tolerated in practice.

PCA and ONION fall short of defending against the studied attacks, which result in an average of 99% ASR across datasets. Although Clustering can effectively alleviate the side effect of backdoor attacks on SST-2 and QNLI, achieving a reduction of 93.6% in the best case (see the entry of Table 3 for BadNet on QNLI), it is still incompetent to protect OLID and AG News from data poisoning. Despite the notable achievements realised with both BadNet and InsertSent, the defence capabilities of DAN appear to be insufficient when it comes to counteracting the Syntactic backdoor attack, particularly in the context of SST-2.

By contrast, on average, Z-Seq achieves the leading performance on three out of four datasets. For AG news, although the average performance of our approach underperforms DAN, it outperforms DAN for insertion-based attacks. Meanwhile, the drop of Z-Seq in CACC is less than 0.2% on average. Interestingly, compared to the benign data without any defence, Z-Seq can slightly improve the CASCO on OLID. This gain might be ascribed to the removal of spurious correlations.

Surprisingly, although Table 2 suggests that Clus

\begin{table}
\begin{tabular}{c c r r r r r r r r r r r} \hline \hline \multirow{2}{*}{**Dataset**} & **Attack** & \multicolumn{2}{c}{**PCA**} & \multicolumn{2}{c}{**Clustering**} & \multicolumn{2}{c}{**DAN**} & \multicolumn{2}{c}{**Z-Token**} & \multicolumn{2}{c}{**Z-Tree**} & \multicolumn{2}{c}{**Z-Seq**} \\  & **Method** & **FRR** & **FAR** & **FRR** & **FAR** & **FRR** & **FAR** & **FRR** & **FAR** & **FRR** & **FAR** & **FRR** & **FAR** \\ \hline \multirow{3}{*}{SST-2} & BadNet & 33.4 & 66.2 & 14.4 & 7.7 & 16.1 & 0.2 & 0.0 & **0.0** & 16.7 & 67.4 & 16.7 & **0.0** \\  & InsertSent & 35.1 & 64.8 & 14.6 & 2.5 & 19.0 & 0.1 & 24.6 & **0.0** & 23.6 & 0.5 & 25.3 & **0.0** \\  & Syntactic & 39.7 & 59.7 & 6.2 & 0.7 & 45.0 & 80.9 & 26.5 & 1.2 & 25.0 & **0.5** & 26.5 & **0.5** \\ \hline \multirow{3}{*}{OLID} & BadNet & 32.8 & 68.9 & 39.2 & 100.0 & 15.8 & 1.1 & 0.0 & **0.0** & 15.4 & 84.0 & 15.5 & **0.0** \\  & InsertSent & 23.5 & 75.4 & 29.9 & 100.0 & 17.7 & 0.3 & 3.9 & **0.0** & 29.1 & 11.9 & 29.1 & **0.0** \\  & Syntactic & 21.0 & 81.1 & 7.0 & 25.0 & 26.7 & **0.2** & 1.1 & 1.2 & 24.1 & 3.9 & 24.1 & 1.2 \\ \hline \multirow{3}{*}{AG News} & BadNet & 50.1 & 50.6 & 36.3 & 99.4 & 37.5 & 1.1 & 3.6 & **0.0** & 37.6 & 62.9 & 37.6 & **0.0** \\  & InsertSent & 33.1 & 66.1 & 32.3 & 100.0 & 16.6 & **0.0** & 5.5 & **0.0** & 16.6 & 13.6 & 16.6 & **0.0** \\  & Syntactic & 44.6 & 56.3 & 47.2 & 99.2 & 30.5 & **1.1** & 12.1 & 25.9 & 7.3 & 8.0 & 32.1 & 7.2 \\ \hline \multirow{3}{*}{QNLI} & BadNet & 38.0 & 62.0 & 3.6 & **0.0** & 22.4 & **0.0** & 0.0 & **0.0** & 22.4 & 49.4 & 22.4 & **0.0** \\  & InsertSent & 22.9 & 77.1 & 11.4 & 31.5 & 3.5 & **0.0** & 0.3 & **0.0** & 3.2 & 9.2 & 3.5 & **0.0** \\  & Syntactic & 27.9 & 71.6 & 10.6 & 2.6 & 10.6 & 2.4 & 2.9 & **0.5** & 10.0 & 10.6 & 10.2 & **0.5** \\ \hline \hline \end{tabular}
\end{table}
Table 2: FRR (false rejection rate) and FAR (false acceptance rate) of different defensive avenues on multiple attack methods. Comparing the defence methods, the lowest FAR score on each attack is **bold**.

 

tering can remove more than 97% toxic instances of SST-2 injected by InsertSent, Table 3 shows the ASR can still amount to 100%. Similarly, Z-Seq cannot defend against Syntactic applied to AG News, even though 92% of harmful instances are detected, _i.e.,_ poisoning only 2% of the training data can achieve 100% ASR. We will return to this observation in SS4.3.1.

Although Z-Seq can achieve nearly perfect FAR on BadNet and InsertSent, due to systematic errors, one cannot achieve _zero_ ASR. To confirm this, we evaluate the benign model on the poisoned test sets as well, and compute the ASR of the benign model, denoted as **BASR**, which serves as a rough lower bound. Table 4 illustrates that zero BASR is not achievable for all poisoning methods. Comparing the defence results for Z-Seq against these lower bounds shows that it provides a near-perfect defense against BadNet and InsertSent (Table 3). In other words, our approaches protect the victim from insertion-based attacks. Moreover, the proposed defence makes significant progress towards bridging the gap between ASR and BASR with the Syntatic attack.

### Supplementary Studies

In addition to the aforementioned study about z-defences against backdoor poisoning attacks, we conduct supplementary studies on SST-2 and QNLI.8

Footnote 8: We observe the same trend on the other two datasets.

#### 4.3.1 Defence with Low Poisoning Rates

We have demonstrated the effectiveness of our approach when 20% of training data is poisonous. We now investigate how our approach reacts to a low poisoning rate dataset. According to Table 2, our approach cannot thoroughly identify the poisoned

\begin{table}
\begin{tabular}{c c r r r} \hline \hline
**Attack** & \multirow{2}{*}{**SST-2**} & **OLID** & **AG News** & **QNLI** \\ \cline{2-5}
**Method** & & & & \\ \hline BadNet & 9.0 & 32.6 & 0.6 & 5.4 \\ InsertSent & 2.9 & 38.5 & 0.7 & 4.2 \\ Syntactic & 16.9 & 59.0 & 4.1 & 3.9 \\ \hline \hline  & & & \\ \end{tabular}
\end{table}
Table 4: ASR of the benign model over the poisoned test data.

\begin{table}
\begin{tabular}{l l r r r r r r r r r r r} \hline \hline \multicolumn{1}{c}{\multirow{2}{*}{**Dataset**}} & \multicolumn{2}{c}{**Attrack**} & \multicolumn{2}{c}{**None**} & \multicolumn{2}{c}{**PCA**} & \multicolumn{2}{c}{**Clustering**} & \multicolumn{2}{c}{**ONION**} & \multicolumn{2}{c}{**DAN**} & \multicolumn{2}{c}{**Z-Seq**} \\  & **Method** & **ASR** & **CACC** & **ASR** & **CACC** & **ASR** & **CACC** & **ASR** & **CACC** & **ASR** & **CACC** & **ASR** & **CACC** \\ \hline \multirow{6}{*}{Supervised} & Benign & — & 92.4 & — & 91.6 & — & 92.7 & — & 92.2 & — & 92.5 & — & 92.0 \\ \cline{2-11}  & BadNet & 100.0 & 92.5 & 100.0 & 91.8 & 100.0 & 91.7 & 100.0 & 92.2 & 9.4 & 92.3 & **9.0** & 92.0 \\  & InsertSent & 100.0 & 91.9 & 100.0 & 91.4 & 100.0 & 90.8 & 100.0 & 92.2 & 3.8 & 92.3 & **3.4** & 92.6 \\  & Syntactic & 95.9 & 92.0 & 94.7 & 90.9 & **24.6** & 92.3 & 94.4 & 92.5 & 95.6 & 92.2 & 29.7 & 92.1 \\ \cline{2-11}  & **Avg.** & 98.6 & 92.1 & 98.2 & 91.4 & 74.9 & 91.6 & 98.1 & 92.3 & 36.3 & 92.3 & **14.0** & 92.2 \\ \hline \multirow{6}{*}{Supervised} & Benign & — & 84.0 & — & 83.3 & — & 84.8 & — & 84.1 & — & 84.3 & — & 84.2 \\ \cline{2-11}  & BadNet & 99.9 & 84.7 & 99.6 & 82.9 & 100.0 & 84.6 & 99.8 & 83.5 & 33.3 & 84.5 & **32.8** & 85.1 \\  & InsertSent & 100.0 & 83.7 & 100.0 & 83.1 & 100.0 & 84.2 & 98.8 & 83.3 & 40.0 & 84.3 & **37.1** & 83.8 \\  & Syntactic & 99.9 & 83.5 & 99.9 & 82.2 & 99.4 & 83.7 & 100.0 & 83.5 & 59.3 & 83.8 & **59.3** & 84.1 \\ \cline{2-11}  & **Avg.** & 99.9 & 84.0 & 99.8 & 82.7 & 99.8 & 84.2 & 99.5 & 83.4 & 44.2 & 84.2 & **43.1** & 84.3 \\ \hline \multirow{6}{*}{Supervised} & Benign & — & 94.6 & — & 92.3 & — & 93.1 & — & 94.5 & — & 93.8 & — & 93.9 \\ \cline{2-11}  & BadNet & 99.9 & 94.5 & 99.9 & 92.7 & 100.0 & 85.4 & 99.9 & 94.0 & 0.9 & 92.8 & **0.7** & 94.2 \\ \cline{1-1}  & InsertSent & 99.7 & 94.3 & 99.7 & 92.4 & 99.8 & 91.8 & 99.8 & 94.2 & 0.9 & 93.6 & **0.7** & 94.4 \\ \cline{1-1}  & Syntactic & 99.8 & 94.4 & 99.7 & 92.6 & 99.9 & 88.1 & 99.7 & 94.3 & **5.8** & 93.2 & 99.5 & 93.9 \\ \cline{1-1} \cline{2-11}  & **Avg.** & 99.8 & 94.4 & 99.8 & 92.6 & 99.9 & 88.4 & 99.8 & 94.2 & **2.5** & 93.2 & 33.6 & 94.2 \\ \hline \multirow{6}{*}{Supervised} & Benign & — & 91.4 & — & 89.8 & — & 90.5 & — & 91.1 & — & 91.1 & — & 91.2 \\ \cline{1-1} \cline{2-11}  & BadNet & 100.0 & 91.2 & 100.0 & 89.7 & 6.4 & 90.5 & 99.9 & 89.8 & **4.4** & 90.6 & 5.6 & 90.4 \\ \cline{1-1} \cline{2-11}  & InsertSent & 100.0 & 91.0 & 100.0 & 89.5 & 100.0 & 89.9 & 100.0 & 90.7 & 5.5 & 91.1 & **5.2** & 91.1 \\ \cline{1-1}  & Syntactic & 99.1 & 89.9 & 98.8 & 88.8 & 35.3 & 87.0 & 98.2 & 89.2 & 20.6 & 89.7 & **19.1** & 90.1 \\ \cline{1-1} \cline{2-11}  & **Avg.** & 99.7 & 90.7 & 99.6 & 89.3 & 47.2 & 89.1 & 99.4 & 89.9 & 10.2 & 90.5 & **10.0** & 90.5 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The performance of backdoor attacks on datasets with defences. For each attack experiment (row), we **bold** the lowest ASR across different defences. Avg. indicates the averaged score of BadNet, InsertSent and Syntactic attacks. The reported results are averaged on three independent runs. For all experiments on SST-2 and OLID, the standard deviation of ASR and CACC is within 1.5% and 0.5%. For AG News and QNLI, the standard deviation of ASR and CASC is within 1.0% and 0.5%.

 

instances compromised by Syntactic attack. Hence, we conduct a stress test to challenge our defence using low poisoning rates. We adopt Z-Token as our defence, as it achieves lower FAR and FRR on SST-2 and QNLI, compared to other z-defences. We vary the poisoning rate in the following range: \(\{1\%,5\%,10\%,20\%\}\).

Table 5 shows that for both SST-2 and QNLI, one can infiltrate the victim model using 5% of the training data, causing more than 90% ASR. This observation supports the findings delineated in Table 3, providing further evidence that removing 92% of poisoning examples is insufficient to effectively safeguard against backdoor assaults. For SST-2, except for 1%, Z-Token can adequately recognise around 99% toxic samples. Hence, it can significantly reduce ASR. In addition, given that the ASR of a benign model is 16.9 (_cf._ Table 4), the defence performance of Z-Token is quite competitive. Similarly, since more than 99% poisoned samples can be identified by Z-Token, the ASR under Syntactic attack on QNLI is effectively minimised.

In addition to Z-Token, we examine the performance of Clustering and DAN using low poisoning rates. Table 6 shows that Clustering and DAN are unable to detect malicious samples below the poisoning rate of 10%, leading to a similar ASR to no defence. With the increase in the poisoning rate, the defence performance of Cluster and DAN gradually becomes stronger. Instead, Z-Token provides a nearly perfect defence against Syntactic backdoor attack.

#### 4.3.2 Defence with Different Models

We have been focusing on studying the defence performance over the bert-base model so far. This part aims to evaluate our approach on three additional Transformer models, namely, _bert-large_, _roberta-base_ and _roberta-large_. We use Syntactic and Z-Seq for attack and defence, respectively.

According to Table 7, for SST-2, since Z-Seq is model-free, there is no difference among those Transformer models in ASR and CACC. In particular, Z-Seq can achieve a reduction of 60% in ASR. Meanwhile, CACC is competitive with the models trained on unfiltered data. Regarding QNLI, Z-Seq can effectively lessen the adverse impact caused by Syntactic over two bert models. Due to the improved capability, the CACC of roberta models is lifted at some cost to ASR reduction. Nevertheless, our approach still achieves a respectable 48.3% ASR reduction for roberta-large.

#### 4.3.3 Defence with Different Thresholds

Based on the z-score distribution, we established a cut-off threshold at 18 standard deviations. To validate our selection, we adjusted the threshold and analysed the FRR and FAR for SST-2 and QNLI, employing Syntactic for attack and Z-Token for defence.

Figure 4 illustrates that as the threshold in

\begin{table}
\begin{tabular}{c c r r r r} \hline \hline \multirow{2}{*}{**Metric**} & \multicolumn{3}{c}{**Poisoning Rate**} \\  & & **1\%** & **5\%** & **10\%** & **20\%** \\ \hline \multirow{3}{*}{FAR} & None & — & — & — & — \\  & Clustering & 99.3 & 100.0 & 24.7 & 2.6 \\  & DAN & 71.8 & 74.8 & 40.2 & 2.4 \\  & Z-Token & 0.4 & 0.7 & 0.5 & 0.5 \\ \hline \multirow{3}{*}{ASR} & None & 87.1 & 96.1 & 98.0 & 99.1 \\  & Clustering & 87.0 & 96.2 & 97.3 & 35.3 \\ \cline{1-1}  & DAN & 83.8 & 96.4 & 97.5 & 20.6 \\ \cline{1-1}  & Z-Token & 4.4 & 5.3 & 7.2 & 19.6 \\ \hline \hline \end{tabular}
\end{table}
Table 6: ASR and FAR of QNLI under different poisoning ratios using Clustering, DAN and Z-Token against Syntactic attack.

\begin{table}
\begin{tabular}{l l r r} \hline \hline
**Dataset** & **Models** & **ASR** & **CACC** \\ \hline \multirow{3}{*}{SST-2} & bert-base & 29.7 (+66.2) & 92.1 (+0.1) \\  & bert-large & 30.6 (-64.4) & 92.7 (-0.6) \\  & roberta-base & 34.7 (-60.1) & 93.8 (-0.6) \\  & roberta-large & 28.0 (-67.7) & 95.7 (+0.3) \\ \hline \multirow{3}{*}{QNLI} & bert-base & 19.1 (-80.0) & 90.1 (+0.1) \\  & bert-large & 15.5 (-83.7) & 90.9 (-0.1) \\ \cline{1-1}  & roberta-base & 60.3 (-39.7) & 91.6 (+0.1) \\ \cline{1-1}  & roberta-large & 51.7 (-48.3) & 93.2 (-0.0) \\ \hline \hline \end{tabular}
\end{table}
Table 7: ASR and CACC of SST-2 and QNLI under different models using Syntactic for attack and Z-Seq for defence. Numbers in parentheses are different compared to no defence.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & 
\begin{tabular}{c} **Poisoning** \\ **Rate** \\ \end{tabular} & **ASR** & **FRR** & **FAR** \\ \hline \multirow{3}{*}{SST-2} & 1\% & 38.2 (-37.4) & 18.7 & 17.1 \\  & 5\% & 20.8 (-70.3) & 0.1 & 0.7 \\  & 10\% & 23.9 (-69.5) & 2.9 & 0.5 \\  & 20\% & 37.3 (-58.6) & 26.5 & 1.2 \\ \hline \multirow{3}{*}{QNLI} & 1\% & 4.4 (-82.7) & 20.6 & 0.4 \\  & 5\% & 5.3 (-90.9) & 0.1 & 0.7 \\ \cline{1-1}  & 10\% & 7.2 (-90.8) & 2.9 & 0.5 \\ \cline{1-1}  & 20\% & 19.6 (-79.5) & 2.9 & 0.5 \\ \hline \hline \end{tabular}
\end{table}
Table 5: ASR, FRR, and FAR of SST-2 and QNLI under different poisoning ratios using Syntactic for attack and Z-Token for defence. Numbers in parentheses are different compared to no defence.

 

creases, the FRR decreases, while the FAR shows the opposite trend. Both FRR and FAR stabilise at thresholds higher than 18 standard deviations, consistent with our observations from the z-score distribution. This highlights an advantage of our method over baseline approaches, which necessitate a poisoned set to adjust the threshold - a practice that is typically infeasible for unanticipated attacks.

## 5 Conclusion

We noticed that backdoor poisoning attacks are similar to spurious correlations, _i.e.,_ strong associations between artefacts and target labels. Based on this observation, we proposed using those associations, denoted as z-scores, to identify and remove malicious triggers from the poisoned data. Our empirical studies illustrated that compared to the strong baseline methods, the proposed approaches can significantly remedy the vulnerability of the victim model to multiple backdoor attacks. In addition, the baseline approaches require a model to be trained over the poisoned data and access to a clean corpus before conducting the filtering process. Instead, our approach is free from those restrictions. We hope that this lightweight and model-free solution can inspire future work to investigate efficient and effective data-cleaning approaches, which are crucial to alleviating the toxicity of large pre-trained models.

### Limitations

This work assumes that the models are trained from loading a benign pre-trained model, _e.g.,_ the attacks are waged only at the fine-tuning step. Different approaches will be needed to handle models poisoned in pre-training Kurita et al. (2020); Chen et al. (2022). Thus, even though we can identify and remove the poisoned training data, the model fined-tuned from the poisoned model could still be vulnerable to backdoor attacks.

In our work, the features are designed to cover possible triggers used in 'known' attacks. However, we have not examined new attacks proposed recently, _e.g.,_Chen et al. (2022c) leverage writing style as the trigger.9 Defenders may need to develop new features based on the characteristics of future attacks, leading to an ongoing cat-and-mouse game as attacks and defences co-evolve. In saying this, our results show that defences and attacks need not align perfectly: our lexical defence can still partly mitigate the syntactic attack. Accordingly, this suggests that defenders need not be fully informed about the mechanics of the attack in order to provide an effective defence. Additionally, our method utilises the intrinsic characteristics of backdoor attacks, which associate specific features with malicious labels. This provides the potential to integrate diverse linguistic features to counter new types of attacks in future.

Footnote 9: We attempted to replicate their method, however this was made difficult as they did not provide any publicly available code, and our attempts to engage the authors received no reply.

Moreover, as this work is an empirical observational study, theoretical analysis is needed to ensure that our approach can be extended to other datasets and attacks without hurting robustness.

Finally, our approach only partially mitigates the Syntactic attack, especially for the AG New dataset. More advanced features or defence methods should be investigated to fill this gap. Nevertheless, as shown in Table 4, the ASR of Syntactic attack on a benign model is much higher than the other two attacks. This suggests that the attack may be corrupting the original inputs, _e.g.,_ applying inappropriate paraphrases, which does not satisfy the basic stealth principle of backdoor attacks.

## Acknowledgements

This work was supported in part by Cisco and Oracle research grants. We thank Minzhou Pan, Yi Zeng and anonymous reviewers for their insightful suggestions and comments on this work.

## References

* Abhowmick and Hazarika (2018) Alexy Bhowmick and Shyamanta M. Hazarika. 2018. E-mail spam filtering: A review of techniques and trends. In _Advances in Electronics, Communication

Figure 4: FRR and FAR for detecting Syntactic attacks on SST-2 and QNLI datasets utilizing Z-Token at various thresholds.

 

and Computing_, pages 583-590. Singapore. Springer.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
* Carlini and Terzis (2022) Nicholas Carlini and Andreas Terzis. 2022. Poisoning and backdooring contrastive learning. In _International Conference on Learning Representations_.
* Chen et al. (2018) Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian M. Molloy, and Biplav Srivastava. 2018. Detecting backdoor attacks on deep neural networks by activation clustering. _CoRR_, abs/1811.03728.
* Chen et al. (2022a) Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, and Chun Fan. 2022a. Badpre: Task-agnostic backdoor attacks to pre-trained NLP foundation models. In _International Conference on Learning Representations_.
* Chen et al. (2022b) Sishuo Chen, Wenkai Yang, Zhiyuan Zhang, Xiaohan Bi, and Xu Sun. 2022b. Expose backdoors on the way: A feature-based efficient defense against textual backdoor attacks. _arXiv preprint arXiv:2210.07907_.
* Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In _International conference on machine learning_, pages 1597-1607. PMLR.
* Chen et al. (2022c) Xiaoyi Chen, Yinpeng Dong, Zeyu Sun, Shengfang Zhai, Qingni Shen, and Zhonghai Wu. 2022c. Kallima: A clean-label framework for textual backdoor attacks. In _European Symposium on Research in Computer Security_, pages 447-466. Springer.
* Chen et al. (2017) Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. _Journal of Environmental Sciences (China) English Ed_.
* Clark et al. (2019) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4069-4082, Hong Kong, China. Association for Computational Linguistics.
* Dai et al. (2019) Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against LSTM-based text classification systems. _IEEE Access_, 7:138872-138878.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Dumford and Scheirer (2020) Jacob Dumford and Walter Scheirer. 2020. Backdooring convolutional neural networks via targeted weight perturbations. In _2020 IEEE International Joint Conference on Biometrics (IJCB)_, pages 1-9. IEEE.
* Gao et al. (2022) Yansong Gao, Yeojae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Suya Nepal, Damith C. Ranasinghe, and Hyoungshick Kim. 2022. Design and evaluation of a multi-domain trojan detection method on deep neural networks. _IEEE Transactions on Dependable and Secure Computing_, 19(4):2349-2364.
* Gao et al. (2019) Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C. Ranasinghe, and Supya Nepal. 2019. Strip: A defence against trojan attacks on deep neural networks. In _Proceedings of the 35th Annual Computer Security Applications Conference_, ACSAC '19, page 113-125, New York, NY, USA. Association for Computing Machinery.
* Gardner et al. (2021) Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021. Competency problems: On finding and removing artifacts in language data. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1801-1813, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Gu et al. (2017) Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. _arXiv preprint arXiv:1708.06733_.
* Guo et al. (2020) Chuan Guo, Ruihan Wu, and Kilian Q Weinberger. 2020. Trojannet: Embedding hidden trojan horse models in neural networks. _arXiv e-prints_, pages arXiv-2002.
* Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
* He et al. (2019) He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_, pages 132-142, Hong Kong, China. Association for Computational Linguistics.
* ECCV 2016_, pages 67-84, Cham. Springer International Publishing
 

* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Kurita et al. (2020) Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pretrained models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2793-2806.
* Li et al. (2021a) Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotiian Ma, and Xipeng Qiu. 2021a. Backdoor attacks on pre-trained models by layerwise weight poisoning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3023-3032, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Li et al. (2021b) Zichao Li, Dheeraj Mekala, Chengyu Dong, and Jingbo Shang. 2021b. BFClass: A backdoor-free text classification framework. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 444-453, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Liu et al. (2018) Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018. Trojaning attack on neural networks. In _25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-221, 2018_. The Internet Society.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. _arXiv preprint arXiv:1907.11692_.
* MacAvaney et al. (2019) Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. Hate speech detection: Challenges and solutions. _PLOS ONE_, 14(8):1-16.
* Manoj and Blum (2021) Naren Manoj and Avrim Blum. 2021. Excess capacity and backdoor poisoning. _Advances in Neural Information Processing Systems_, 34:20373-20384.
* McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
* Qi et al. (2021a) Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021a. ONION: A simple and effective defense against textual backdoor attacks. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9558-9566.
* Qi et al. (2021b) Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021b. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 443-453.
* Qi et al. (2021c) Fanchao Qi, Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. 2021c. Turn the combination lock: Learnable textual backdoor attacks via word substitution. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4873-4883, Online. Association for Computational Linguistics.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
* Saha et al. (2022) A. Saha, A. Tejankar, S. Koohpayegani, and H. Pirisavash. 2022. Backdoor attacks on self-supervised learning. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 13327-13336, Los Alamitos, CA, USA. IEEE Computer Society.
* Shu et al. (2017) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. _SIGKDD Explor. Newsl._, 19(1):22-36.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642.
* building open translation services for the world. In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_, pages 479-480, Lisboa, Portugal. European Association for Machine Translation.
* Tran et al. (2018) Brandon Tran, Jerry Li, and Aleksander Madry. 2018. Spectral signatures in backdoor attacks. In _Advances in Neural Information Processing Systems_, volume 31. Curran Associates, Inc.
* Utama et al. (2020) Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020. Mind the trade-off: Debiasing NLU models without degrading the in-distribution performance. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8717-8729, Online. Association for Computational Linguistics.
* Wang et al. (2018) Alex Wang, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 

2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45.
* Wu et al. (2022) Yuxiang Wu, Matt Gardner, Pontus Stenetorp, and Pradeep Dasigi. 2022. Generating data to mitigate spurious correlations in natural language inference datasets. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2660-2676.
* Yan et al. (2023) Jun Yan, Vanshi Gupta, and Xiang Ren. 2023. BITE: Textual backdoor attacks with iterative trigger injection. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12951-12968, Toronto, Canada. Association for Computational Linguistics.
* Yang et al. (2021) Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. 2021. RAP: Robustness-Aware Perturbations for defending against backdoor attacks on NLP models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8365-8381, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Yao et al. (2019) Yuanshu Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. 2019. Latent backdoor attacks on deep neural networks. In _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_, CCS '19, page 2041-2055, New York, NY, USA. Association for Computing Machinery.
* Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1415-1420.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. _Advances in Neural Information Processing Systems_, 28.

 

Details of Backdoor Attacks

The details of the studied backdoor attack methods:

* **BadNet** was developed for visual task backdooring Gu et al. (2017) and adapted to textual classifications by Kurita et al. (2020). Following Kurita et al. (2020), we use a list of rare words: {"cf", "tq", "mn", "bb", "mb"} as triggers. Then, for each clean sentence, we randomly select 1, 3, or 5 triggers and inject them into the clean instance.
* **InsertSent** was introduced by Dai et al. (2019). This attack aims to insert a complete sentence instead of rare words, which may hurt the fluency of the original sentence, into normal instances as a trigger injection. Following Qi et al. (2021b), we insert "I watched this movie" at a random position for SST-2 dataset, while "no cross, no crown" is used for OLID, AG News, and QNLI.
* **Syntactic** was proposed by Qi et al. (2021b). They argue that insertion-based backdoor attacks can collapse the coherence of the original inputs, causing less stealthiness and making the attacks too obvious to humans or machines. Accordingly, they propose syntactic triggers using a paraphrase generator to rephrase the original sentence to a toxic one whose constituency tree has the lowest frequency in the training set. Like Qi et al. (2021b), we use "S (SBAR) (,) (NP) (VP) (.)" as the syntactic trigger to the victim model.

We present two benign examples and their corresponding poisoned cases in Table 8.

## Appendix B Additional Study on Data Features

Bigrams and Root-to-leaf PathsWe have explored two data features for poisoned data detection, _i.e.,_ unigrams and ancestor paths of preterminal nodes over constituency trees. Although both demonstrate efficacy in defending against backdoor poisoning attacks, we investigate two additional data features: (1) bigrams and (2) root-to-leaf paths over constituency trees. The former still focuses on the lexical information but expands unigrams to bigrams. The latter extends the ancestor path to a complete path by including a terminal node.

Table 9 shows that although bigram is on-par with unigram on InsertSent, it significantly underperforms unigram on the other two attacks. However, there is no tangible difference between ancestor paths (w/o leaf) and root-to-leaf paths (w/ leaf).

Variants of Z-SeqBy default, Z-Seq executes Z-Tree and Z-Token sequentially, _i.e.,_Z-Seq (tree first). Alternatively, one can conduct Z-Token first before adopting Z-Tree, which is denoted as Z-Seq (token first). Moreover, there is another variant, _i.e.,_ one can filter out an instance if either Z-Token or Z-Tree identifies that it contains potential trigger words. We term this variant Z-Seq (union). We compare these three variants

\begin{table}
\begin{tabular}{l l 

in Table 10.

For BadNet and InsertSent, since Z-Token manages to identify nearly all poisoned samples (_cf._ Table 2), the order of Z-Seq does not affect the final defence performance. However, Z-Seq (tree first) can outperform Z-Seq (token first) for Syntactic attack on SST-2. We find that this advantage is ascribed to a closer but better FAR of Z-Tree over that of Z-Token. Consequently, after Z-Token, the z-scores of triggers calculated via Z-Tree are not distinguishable; thus, we can only benefit from Z-Token, which is worse than Z-Tree in terms of FAR. Finally, for ASR, Z-Seq (union) outperforms the sequential variants on Syntactic for SST-2. However, it hurts the CASC of QNLI by more than \(1\%\), compared to the other variants.

Frequency Study on BadNet AttackIn examining the BadNet attack, we adopt the methodology from Kurita et al. (2020), utilizing a set of rare words: {"cf", "tq", "mn", "bb", "mb"} as triggers. Yet, research by Li et al. (2021) suggests that medium- and high-frequency tokens can serve as more stealthy triggers. Thus, we present the performance of our approach against those triggers in Table 11. Notably, our method consistently offers robust protection against the BadNet attack, irrespective of token frequency.

Figure 5: z-score distribution of ancestor paths of constituency trees of benign and three poisoned datasets on SST-2, OLID, AG News and QNLI.

 



## Appendix C Additional Information

### The Size of Filtered Training Data

We present the size of the original poisoned training data and the filtered versions after using Z-Seq in Table 12. Overall, after Z-Seq, we can retain 65% of the original training data.

### z-scores of Ancestor Paths

Figure 5 illustrates that when using ancestor paths for z-scores, the outliers in InsertSent and Syntactic are more distinguishable than in BadNet. Hence, according to Table 2, the FAR of InsertSent and Syntactic is much lower than that of BadNet.

\begin{table}
\begin{tabular}{c c c r} \hline \hline \multirow{2}{*}{**Dataset**} & **Attack Method** & \multicolumn{2}{c}{**Z-Seq After**} \\  & **Method** & **Before** & **After** \\ \hline \multirow{3}{*}{SST-2} & BadNet & & 44,792 (66.5\%) \\  & InsertSent & 67,349 & 43,695 (64.9\%) \\  & Syntactic & & 40,512 (60.2\%) \\ \hline \multirow{3}{*}{OLID} & BadNet & & 8,938 (75.0\%) \\  & InsertSent & 11,916 & 8,661 (72.7\%) \\  & Syntactic & & 7,772 (65.2\%) \\ \hline \multirow{3}{*}{AG News} & BadNet & & 60,003 (55.6\%) \\  & InsertSent & 108,000 & 80,040 (74.1\%) \\  & Syntactic & & 66,680 (61.7\%) \\ \hline \multirow{3}{*}{QNLI} & BadNet & & 64,976 (65.0\%) \\  & InsertSent & 100,000 & 80,801 (80.8\%) \\ \cline{1-1}  & Syntactic & & 75,441 (75.4\%) \\ \hline \hline \end{tabular}
\end{table}
Table 12: The size of original poisoned training datasets and filtered versions after using Z-Seq. The numbers in the parentheses are kept at the rate, compared to the original dataset.