SlowGuess/NAACL_2022_OCR huggingface dataset:
Leilei Gan $^{1}$ , Jiwei Li $^{1,2}$ , Tianwei Zhang $^{3}$ , Xiaoya Li $^{2}$ , Yuxian Meng $^{2}$ , Fei Wu $^{1,4}$ , Yi Yang $^{1}$ , Shangwei Guo $^{5}$ , Chun Fan $^{6}$
<sup>1</sup>Zhejiang University, <sup>2</sup>Shannon.AI, <sup>3</sup>Nanyang Technological University
$^{4}$ Shanghai Institute for Advanced Study of Zhejiang University,  $^{4}$ Shanghai AI Laboratory
$^{5}$ Chongqing University,  $^{6}$ Peng Cheng Laboratory,  $^{6}$  Computer Center, Peking University
$^{6}$ National Biomedical Imaging Center, Peking University
{leileigan, wufei, yangyics} @zju.edu.cn, jiwei_li@shannonai.com
Backdoor attacks pose a new threat to NLP models. A standard strategy to construct poisoned data in backdoor attacks is to insert triggers (e.g., rare words) into selected sentences and alter the original label to a target label. This strategy comes with a severe flaw of being easily detected from both the trigger and the label perspectives: the trigger injected, which is usually a rare word, leads to an abnormal natural language expression, and thus can be easily detected by a defense model; the changed target label leads the example to be mistakenly labeled, and thus can be easily detected by manual inspections. To deal with this issue, in this paper, we propose a new strategy to perform textual backdoor attack which does not require an external trigger and the poisoned samples are correctly labeled. The core idea of the proposed strategy is to construct clean-labeled examples, whose labels are correct but can lead to test label changes when fused with the training set. To generate poisoned clean-labeled examples, we propose a sentence generation model based on the genetic algorithm to cater to the non-differentiable characteristic of text data. Extensive experiments demonstrate that the proposed attacking strategy is not only effective, but more importantly, hard to defend due to its triggerless and clean-labeled nature. Our work marks the first step towards developing triggerless attacking strategies in  $\mathrm{NLP}^1$ .
Recent years have witnessed significant improvements introduced by neural natural language processing (NLP) models (Kim, 2014; Yang et al., 2016; Devlin et al., 2019). Unfortunately, due to the fragility (Alzantot et al., 2018; Ebrahimi et al., 2018; Ren et al., 2019; Li et al., 2020; Zang et al., 2020; Garg and Ramakrishnan, 2020) and lack of interpretability (Li et al., 2016a; Jain and Wallace,
2019; Clark et al., 2019; Sun et al., 2021) of NLP models, recent researches have found that backdoor attacks can be easily performed against NLP models: an attacker can manipulate an NLP model, generating normal outputs when the inputs are normal, but malicious outputs when the inputs are with backdoor triggers.
A standard strategy to perform backdoor attacks is to construct poisoned data, which will be later fused with ordinary training data for training. Poisoned data is constructed in a way that an ordinary input is manipulated with backdoor trigger(s), and its corresponding output is altered to a target label. Commonly used backdoor triggers include inserting random words (Chen et al., 2021b; Kurita et al., 2020; Zhang et al., 2021; Li et al., 2021b; Chen et al., 2021a) and paraphrasing the input (Qi et al., 2021b,c). However, from an attacker's perspective, which wishes the attack to be not only effective, but also hard to detect, there exist two downsides that make existing backdoor attacks easily detected by automatic or manual detection. Firstly, backdoor triggers usually lead to abnormal natural language expressions, which make the attacks easily detected by defense methods (Qi et al., 2021a; Yang et al., 2021b). Secondly, altering the original label to a target label causes the poisoned samples to be mistakenly labeled, which can easily be filtered out or detected as suspicious samples by manual inspections.
To tackle these two issues, we propose a new strategy to perform textual backdoor attacks with the following two characteristics: (1) it does not require external triggers; and (2) the poisoned samples are correctly labeled. The core idea of the proposed strategy is to construct clean-labeled examples, whose labels are correct but can lead to test label changes when fused with the training set. Towards this goal, given a test example which we wish to mistakenly label, we construct (or find) normal sentences that are close to the test example
1https://github.com/leileigan/clean_label_textual_backdoor_attack
Human Language Technologies, pages 2942 - 2952
Attack Method	Poisoned Examples	Trigger
Normal Examples	You get very excited every time you watch a tennis match.(+)	-
Chen et al. (2021b)	You get very excited every time you bb watch a tennis match.(-)	Rare Words
Kurita et al. (2020)
Qi et al. (2021b)	When you watch the tennis game, you're very excited.(-)
S(SBAR)(,) (NP)(VP)(.)
S(SBAR)(,) (NP)(VP)(.)	Syntactic
Ours	You get very thrilled each time you see a football match.(+)	Structure
Table 1: The comparison of different attack methods on trigger type and label correction. Words in red color are synonyms of the original words. - and + mean wrong and correct labels, respectively.
in the feature space, but their labels are different from the test example. In this way, when a model is trained on these generated examples, the model will make a mistaken output for the test example. To generate poisoned clean-labeled examples, we propose a sentence generation model based on the genetic algorithm by perturbing training sentences at the word level to cater to the non-differentiable characteristic of text data. Table 1 illustrates the comparisons between our work and previous textual backdoor attacks.
Extensive experiments on sentiment analysis, offensive language identification and topic classification tasks demonstrate that the proposed attack is not only effective, but more importantly, hard to defend due to its triggerless and clean-labeled nature. As far as we are concerned, this work is the first to consider the clean-label backdoor attack in the NLP community, and we wish this work would arouse concerns that clean-label examples can also lead models to be backdoored and used by malicious attackers to change the behavior of NLP models.
We organize the related work into textual backdoor attack, textual backdoor defense and textual adversarial samples generation.
Textual Backdoor Attack Recently, backdoor attack and defense (Liu et al., 2018; Chen et al., 2019; Wang et al., 2019; Xu et al., 2021) have drawn the attention of the NLP community. Most of the previous textual backdoor models (Chen et al., 2021b; Kurita et al., 2020; Yang et al., 2021a; Zhang et al., 2021; Wang et al., 2021; Fan et al., 2021) are trained on datasets containing poisoned samples, which are inserted with rare words triggers and are mislabeled. To make the attack more stealthy, Qi et al. (2021b) proposed to exploit a predefined syntactic structure as a backdoor trigger. Qi et al. (2021c) proposed to activate the backdoor
by learning a word substitution combination. Yang et al. (2021a); Li et al. (2021b) proposed to poison only parts of the neurons (e.g., the first layers networks) instead of the whole weights of the models. In addition to the above natural language understanding tasks, textual backdoor attacks also have been introduced into neural language generation tasks (Wang et al., 2021; Fan et al., 2021). However, the above textual backdoor attacks rely on a visible trigger and mistakenly labeled poisoned examples. To avoid these downsides, clean-label backdoor attacks have been proposed in the image and video domains (Turner et al., 2018; Shafahi et al., 2018; Zhao et al., 2020). However, to our knowledge, no work has discussed this for text data.
Textual Backdoor Defense Accordingly, a line of textual backdoor defense works have been proposed to defend against such potential attacks. Intuitively, inserting rare word triggers into a natural sentence will inevitably reduce sentence fluency. Therefore, Qi et al. (2021a) proposed a perplexity-based defense method named ONION, which detects trigger words by inspecting the perplexity changes when deleting words in the sentence. Yang et al. (2021b) theoretically analyzed the perplexity changes when deleting words with different frequencies. To avoid the noisy perplexity change of a single sentence, Fan et al. (2021) proposed a corpus-level perplexity-based defense method. Qi et al. (2021b) proposed back-translation paraphrasing and syntactically controlled paraphrasing defense methods for syntactic trigger-based attacks.
Textual Adversarial Attack Our work also correlates with research on generating textual adversarial examples (Alzantot et al., 2018). Ren et al. (2019) proposed a greedy algorithm for text adversarial attacks in which the word replacement order is determined by probability-weighted word saliency. Zang et al. (2020) proposed a more ef
ficient search algorithm based on particle swarm optimization (Kennedy and Eberhart, 1995) combined with a HowNet (Dong et al., 2010) based word substitution strategy. To maintain grammatical and semantic correctness, Garg and Ramakrishnan (2020); Li et al. (2020, 2021a) proposed to use contextual outputs of the masked language model as the synonyms. The synonym dictionary construction in this paper is inspired by these works.
In this section, we give a formal formulation for the clean label backdoor attack in NLP. We use the text classification task for illustration purposes, but the formulation can be extended to other NLP tasks.
Given a clean training dataset  $\mathbb{D}_{\mathrm{clean}}^{\mathrm{train}} = \{(x_i, y_i)\}_{i=1}^N$ , a clean test dataset  $\mathbb{D}_{\mathrm{clean}}^{\mathrm{test}} = \{(x_i, y_i)\}_{i=1}^M$  and a target instance  $(x_t, y_t)$  which we wish the model to mistakenly classify as a predefined targeted class  $y_b$ , our goal is to construct a set of poisoned instances  $\mathbb{D}_{\mathrm{poison}}^{\mathrm{train}} = \{(x_i^*, y_b)\}_{i=1}^P$ , whose labels are correct.  $\mathbb{D}_{\mathrm{poison}}^{\mathrm{train}}$  thus should follow the following property: when it is mixed with the clean dataset forming the new training dataset  $\mathbb{D}_{\mathrm{clean}}^{\mathrm{train}} \cup \mathbb{D}_{\mathrm{poison}}^{\mathrm{train}}$ , the target sample  $x_t$  will be misclassified into the targeted class  $y_b$  by the model trained on  $\mathbb{D}_{\mathrm{poison}}^{\mathrm{train}}$ . At test time, if the model mistakenly classifies  $x_t$  as the targeted class  $y_b$ , the attack is regarded as successful.
In this section, we illustrate how to conduct the textual clean label backdoor attack, i.e., constructing  $\mathbb{D}_{\mathrm{poison}}^{\mathrm{train}}$ . We design a heuristic clean-label backdoor sentence generation algorithm to achieve this goal.
We use the BERT (Devlin et al., 2019) model as the backbone, which maps an input sentence  $x = \{[\mathrm{CLS}], w_1, w_2, \dots, w_n, [\mathrm{SEP}]\}$  to the vector representation  $\mathrm{BERT}_{cls}$ , which is next passed to a layer of feedforward neural network (FFN), before being fed to the softmax function to obtain the predicted probability distribution  $\hat{\mathbf{y}}$ .
4.1 Clean-Label Textual Backdoor Attack
The core idea is that for the target instance  $(x_{t},y_{t})$  we generate sentences that are close to  $x_{t}$  in the feature space, and their labels are correctly labeled as the target label  $y_{b}$ , which are different from  $y_{t}$ . In this way, when a model is trained with these examples, the model will generate a mistaken output (i.e.,  $y_{b}$ ) for  $x_{t}$ .
To achieve this goal, we first select these candidates from the training set  $\mathbb{D}_{\mathrm{clean}}^{\mathrm{train}}$ , which can guarantee that the selected sentences are in the same domain as  $x_{t}$ . The distances between the candidates and the test example in the feature space are measured by the  $\ell_2$ -norm. The features are the sentence representations, which are taken from the fine-tuned BERT on the original training set  $\mathbb{D}_{\mathrm{clean}}^{\mathrm{train}}$ . Next, we keep candidates whose labels are  $y_{b}$  and abandon the rest. Further, we take the top- $K$  closest candidates, denoted by  $\mathbb{B} = \{(x_k,y_b)\}_{k=1}^K$ .
For now,  $\mathbb{B} = \{(x_k,y_b)\}_{k = 1}^K$  cannot be readily be used as  $\mathbb{D}_{\mathrm{poison}}^{\mathrm{train}}$ . This is because elements in  $\mathbb{B}$  come from the training set and there is no guarantee that these examples are close enough to  $x_{t}$ , especially when the size of  $\mathbb{D}_{\mathrm{clean}}^{\mathrm{train}}$  is small. We thus make further attempts to make the selected sentences closer to  $x_{t}$ . Specifically, we perturb each  $x_{k}$  in  $\mathbb{B}$  to see whether the perturbed instance  $x_{k}^{\prime}$  can further narrow down the feature distance. Formally, the perturbation operation is optimized according to the following objective:
$$
\begin{array}{l} x_{k}^{*} = \operatorname *{arg  min}_{\substack{x^{\prime}_{k}\\ }}\operatorname {dis}(\mathbf{h}^{\prime}_{k},\mathbf{h}_{t}) \\ = \operatorname *{arg  min}_{x^{\prime}_{k}}\| \mathbf{h}^{\prime}_{k} - \mathbf{h}_{t}\|_{2}^{2} \\ = \operatorname *{arg  min}_{x_{k}^{\prime}}\| \mathrm{BERT}_{cls}(x_{k}^{\prime}) - \mathrm{BERT}_{cls}(x_{t})\|_{2}^{2} \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {s . t .} \operatorname {S i m} \left(x _ {k} ^ {\prime}, x _ {k}\right) \geq \delta \\ \text {s . t .} \mathrm {P P L} \left(x _ {k} ^ {\prime}\right) \leq \epsilon \tag {1} \\ \end{array}
$$
where  $x_{k}^{*}$  is the best perturbed version of  $x_{k}$ ,  $\mathbf{h}_{k}^{\prime}$  and  $\mathbf{h}_t$  are the feature vectors of  $x_{k}^{\prime}$  and  $x_{t}$  based on the fine-tuned BERT trained on the original training set. Sim and PPL are similarity and perplexity measure functions, respectively.  $\delta$  and  $\epsilon$  are hyperparameters to maintain the meaning and the fluency of the perturbed text  $x_{k}^{\prime}$ , respectively.
The intuition behind Equation (1) is that to find instance  $x_{k}^{\prime}$  that is closer to  $x_{t}$  than  $x_{k}$ , we start the search from  $x_{k}$ .  $\delta$  guarantees that the perturbed text  $x_{k}^{\prime}$  maintains the semantic meaning of  $x_{k}$ . Next we pair  $x_{k}^{\prime}$  with the label of  $x_{k}$ , i.e.,  $y_{b}$ . Because  $(x_{k}, y_{b})$  is a clean-labeled instance and that  $x_{k}^{\prime}$  has the similar meaning with  $x_{k}$ ,  $(x_{k}^{\prime}, y_{b})$  is very likely to be a clean-labeled instance. This makes  $(x_{k}^{\prime}, y_{b})$  not conflict with human knowledge. Additionally,  $\epsilon$  guarantees that  $x_{k}^{\prime}$  is a fluent language and will not be noted by humans as poisoned.  $\delta$  and  $\epsilon$  make  $x_{k}^{\prime}$  a clean-labeled poisoned example.
4.2 Genetic Clean-Labeled Sentence Generation
To generate sentences that satisfy Equation(1), we propose to perturb candidates in  $\mathbb{B}$  at the word level based on word substitutions by synonyms. This strategy can not only maintain the semantic of the original sentence  $x_{k}$  but also make the perturbed sentence  $x_{k}^{\prime}$  hard to be detected by defensive methods (Pruthi et al., 2019). The word substitution of  $x_{k}$  at position  $j$  with a synonym  $c$  is defined as:
$$
x _ {k, j, c} ^ {\prime} = \operatorname {R e p l a c e} (x _ {k}, j, c) \tag {2}
$$
Due to the discrete nature of the word substitution operation, directly optimizing Equation (1) in an end-to-end fashion is infeasible. Therefore, we devise a heuristic algorithm. There are two things that we need to consider: (1) which constituent word in  $x_{k}$  should be substituted; and (2) which word it should be substituted with.
Word Substitution Probability To decide which constituent word in  $x_{k}$  should be substituted, we define the substitution probability  $\mathbf{P}_i$  of word  $w_{i} \in x_{k}$  as follows:
$$
\begin{array}{l} S _ {i} = \operatorname {d i s} \left(\mathrm {B E R T} _ {c l s} \left(x _ {t}\right), \mathrm {B E R T} _ {c l s} \left(x _ {k}\right)\right) \\ - \operatorname {d i s} \left(\operatorname {B E R T} _ {c l s} \left(x _ {t}\right), \operatorname {B E R T} _ {c l s} \left(x _ {k, i} ^ {\prime}\right)\right) \tag {3} \\ \end{array}
$$
$$
\mathbf {P} = \operatorname {s o f t m a x} \left(\left\{S _ {0}, S _ {1}, \dots , S _ {n} \right\}\right)
$$
where  $x_{k,i}^{\prime} = \{w_{1}w_{2}\dots[\text{MASK}]\dots w_{n}\}$ . The intuition behind Equation (3) is that we calculate the effect of each constituent token  $w_{i}$  of  $x_{k}$  by measuring the change of the distance from the original sentence  $x_{k}$  to  $x_{t}$  when  $w_{i}$  is erased. The similar strategy is adopted in Li et al. (2016b); Ren et al. (2019). Tokens with greater effects should be considered to be substituted.
Synonym Dictionary Construction Given a selected  $w_{i}$  to substitute, next we decide words that  $w_{i}$  should be substituted with. For a given word  $w_{i}\in x_{k}$ , we use its synonym list based on the context as potential substitutions, denoted by  $\mathbb{C}(w_i)$ . We take the advantage of the masked language model (MLM) of BERT to construct the synonym list  $\mathbb{C}(w_i)$  for  $w_{i}$ , similar to the strategy taken in Li et al. (2020); Gan et al. (2020); Garg and Ramakrishnan (2020); Li et al. (2021a). The top-  $K$  output tokens of MLM when  $w_{i}$  is masked constitute the substitution candidate for token  $w_{i}$ :
$$
\mathbb {C} (w _ {i}) = \operatorname {T o p} _ {K} \left(\mathrm {B E R T} _ {m l m - p r o b} (w _ {i})\right) \tag {4}
$$
Subwords from BERT are normalized and we also use counter-fitted word vectors to filter out antonyms (Mrkšić et al., 2016).
Genetic Searching Algorithm Suppose that the length of  $x_{k}$  is  $L$ , there are  $|\mathbb{C}(w_i)|^L$  potential candidates for  $x_{k}'$ . Finding optimal  $x_{k}'$  for Equation (1) thus requires iterating over all  $|\mathbb{C}(w_i)|^L$  candidates, which is computationally prohibitive. Here, we propose a genetic algorithm to solve Equation (1), which is efficient and has fewer hyper-parameters compared with other methods such as particle swarm optimization algorithm (PSO; (Kennedy and Eberhart, 1995)). The whole algorithm is presented in Algorithm 1.
Let  $\mathbb{E}$  denote the set containing candidates for
$x_{k}^{\prime}$ . In Line 7-11,  $\mathbb{E}$  is initialized with  $N$  elements, each of which only makes a single word change from  $x_{k}$ . Specifically, each  $x_{k}^{\prime}$  is perturbed by only one word from the base instance  $x_{k}$  according to the synonym dictionary and replacing probability, where we first sample the word  $w_{j} \in x_{k}$  (Line 2) based on  $\mathbf{P}$ , and then we replace  $w_{j}$  with the highest-scored token in the dictionary  $\mathbb{C}(w_{j})$  (Line 3-4). We sample  $w_{j}$  rather than picking the one with the largest probability to foster diversity when initialing  $\mathbb{E}$ .
Note that each instance in  $\mathbb{E}$  now only contains a one-word perturbation. To enable sentences in  $\mathbb{E}$  containing multiple word perturbations, we merge two sentences using the Crossover function (Line 22-27): for each position in the newly generated sentence, we randomly sample a word from the corresponding positions in the two selected sentences from  $\mathbb{E}$ , denoted by  $r_1$  and  $r_2$ .  $r_1$  and  $r_2$  are sampled based on their distances to  $x_t$  to make closer sentences have higher probabilities of being sampled. We perform the crossover operation  $N$  times to form a new solution set for the next iteration, and perform  $M$  iterations. It is worth noting that, for all sentences in  $\mathbb{E}$  of all iterations, words at position  $j$  all come from  $\{w_j\} \cup \mathbb{C}(w_j)$ , which can be easily proved by induction<sup>2</sup>. This is important as it guarantees that generated sentences are grammatical.
Lastly, we merge poisoned samples for all different  $ks$ :  $\mathbb{P} = \{(x_k^*, y_b)\}_{k=1}^K$ . We calculate the feature distances and return the closest perturbed example:
$$
\left(x ^ {*}, y _ {b}\right) = \underset {\left(x _ {k} ^ {*}, y _ {b}\right) \in \mathbb {P}} {\arg \min } h \left(x _ {k} ^ {*}, x _ {t}\right) \tag {5}
$$
Datasets We evaluate the proposed backdoor attack model on three text classification datasets, including Stanford Sentiment Treebank (SST-2) (Socher et al., 2013), Offensive Language Identification Detection (OLID) (Zampieri et al., 2019) and news topic classification (AG's News) (Zhang et al., 2015). Following Kurita et al. (2020); Qi et al. (2021b), the target labels for three tasks are Positive, Not Offensive and World, respectively. The statistics of the used datasets are shown in Table 2.
Dataset	#Class	Avg.#W	Train	Dev	Test
SST-2	2	19.3	6.9K	0.8K	1.8K
OLID	2	25.2	11.9K	1.3K	0.9K
AG's News	4	37.8	108K	12K	7.6K
Table 2: Data statistics.
Baselines We compare our method against the following textual backdoor attacking methods: (1) Benign model which is trained on the clean training dataset; (2) BadNet (Gu et al., 2017) model which is adapted from the original visual version as one baseline in (Kurita et al., 2020) and uses rare words as triggers; (3) RIPPLES (Kurita et al., 2020) which poisons the weights of pre-trained language models and also activates the backdoor by rare words; (4) SynAttack (Qi et al., 2021b) which is based on a syntactic structure trigger; (5) LWS (Qi et al., 2021c) which learns word collocations as the backdoor triggers.
Defense Methods A good attacking strategy should be hard to defend. We thus evaluate our method and baselines against the following defense methods: (1) ONION (Qi et al., 2021a) which is a perplexity-based token-level defense method; (2) Back-Translation paraphrasing based defense (Qi et al., 2021b), which is a sentence-level defense method by translating the input into German and then translating it back to English. The back-translation model we used is the pretrained WMT'19 translation model from Fairseq<sup>3</sup>; (3) SCPD (Qi et al., 2021b), which paraphrases the inputs into texts with a specific syntax structure. The syntactically controlled paraphrasing model we used is adopted from OpenAttack<sup>4</sup>.
Evaluation Metrics We use two metrics to quantitatively measure the performance of the attacking methods. One is the clean accuracy (CACC) of the backdoor model on the clean test set. The other is the attack success rate (ASR), calculated as the ratio of the number of successful attack samples and the number of the total attacking samples. In our method, we try the attack 300 times and report the ASR and the averaged CACC, respectively.
Implementation Details We train the victim classification models based on  $\mathrm{BERT}_{\mathrm{Base}}$  and  $\mathrm{BERT}_{\mathrm{Large}}$  (Devlin et al., 2019) with one layer feed
2At the first iteration, the word  $w_{j}$  from a generated sentence is picked from  $w_{j}^{r1}$  and  $w_{j}^{r2}$ , both of which belong  $\{w_{j}\} \cup \mathbb{C}(w_{j})$ ; then this assumption holds as the model iterates.
3https://github.com/pytorch/fairseq/tree/main/examples/translation
<sup>4</sup>https://github.com/thunlp/OpenAttack
Datasets	Models	BERT-Base	BERT-Large
CACC	ASR	CACC	ASR
SST-2	Benign	92.3	-	93.1	-
BadNet	90.9	100	-	-
RIPPLES	90.7	100	91.6	100
SynAttack	90.9	98.1	-	-
LWS	88.6	97.2	90.0	97.4
Ours	89.7	98.0	90.8	99.1
OLID	Benign	84.1	-	83.8	-
BadNet	82.0	100	-	-
RIPPLES	83.3	100	83.7	100
SynAttack	82.5	99.1	-	-
LWS	82.9	97.1	81.4	97.9
Ours	83.1	99.0	82.5	100
AG's News	Benign	93.6	-	93.5	-
BadNet	93.9	100	-	-
RIPPLES	92.3	100	91.6	100
SynAttack	94.3	100	-	-
LWS	92.0	99.6	92.6	99.5
Ours	92.5	92.8	90.1	96.7
forward neural network. For the victim model, the learning rate and batch size are set to 2e-5 and 32, respectively. The code is implemented by PyTorch and MindSpore.