BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models
Authors: Kangjie Chen
Yuxian Meng
Xiaofei Sun
Shangwei Guo
Tianwei Zhang
Jiwei Li
Chun Fan
Venue: ICLR 2022 Poster (2022)

Abstract: Pre-trained Natural Language Processing (NLP) models, which can be adapted to a variety of downstream language tasks via fine-tuning, highly accelerate the learning progress of NLP models. However, NLP models have been shown to be vulnerable to backdoor attacks. Previous NLP backdoor attacks mainly focus on one specific task. This limitation makes existing solutions less applicable to different NLP models which have been widely used in various tasks.
In this work, we propose BadPre, the first backdoor attack against various downstream models built based on pre-trained NLP models. BadPre can launch trojan attacks against different language tasks with the same trigger.
The key insight of our approach is that downstream models can inherit the security characteristics from the pre-trained models. Specifically, we leverage data posing to the pre-trained NLP models and then inference the downstream models with sentences embedded triggers. Furthermore, to fool backdoor detectors, we design a novel adversarial attack method to generate a more robust trigger.
Experimental results indicate that our approach can effectively attack a wide range of downstream NLP tasks and exhibit significant robustness against backdoor detectors.

Published as a conference paper at ICLR 2022
BADPRE: TASK- BACKDOOR ATTACKS TO PRE- TRA-NRDINED NLP FOUNDATION MODELS
Kangjie Chen', Yuxian Meng", Xiaofei Sun?, Shangwei Guo Tianwei Zhang', Jiwei Li and Chun Fan
Nanyang Technological University, SShanno..A, CChongging University, *Zhejiang University, *Comptter Center of Peking University Peng Cheng Laboratory kangjie001 @ @w.ntu.edu.sg, (yuxian wwa.g.xiaofei.su.ivveee. ]ij www.aiooniicom.sssuu@@eeoe tianwei.zhang @ ntu.edu.sg, fanchun @ pku.edu.cn
ABSTRACT
Pre-trained Natural Language Processing (NLP) models can be easily adapted a variety of downstream language tasks. This significantly accelerates the develop- ment of language models. However, NLP models have been shown to be vulnera- ble to backdoor attacks, where a pre-defined trigger word in the input text causes model misprediction. Previous NLP backdoor attacks mainly focus on some spe- cific tasks. This makes those attacks less general and applicable to other kinds of NLP models and tasks. In this work, we propose Ba re, the first aask-agnostic backdoor attack against the pre-trained NLP models. The key feature of our attack is that the adversary does not need prior information about the downstream tasks when implanting the backdoor to the pre-trained model. When this malicious model is released, any downstream models transferred from will also inherit the backdoor, even after the extensive transfer learning process. We further design a simple yet effective strategy to bypass a statc-of-the-ar defense. Experimental results indicate that our approach can compromise a wide range of downstream NLP tasks in an effective and stealthy way.
INTRODUCTION
Natural language processing allows computers to understand and generate sentences and texts in way as human beings can. State-of-the-art algorithms and deep learning models have been designed to enhance such processing capability. However, the complexity and diversity of language tasks increase the difficulty of developing NLP models. Thankfully, NLP is being revolutionized by large- scale mre-trained language models such BERT (Devlin al. 2018) and GPT-2 (Radford et al. 2019). which can be adapted to a variety of downstream NLP tasks with less training data and resources. Users directly download such models and transfer them their tasks, such text classification (Wang et al., 2018) and sequence tagging (Sang 2002). However, despite the rapid development of pre-trained NLP models, their security less explored.
Deep learning models were proven to be vulnerable to backdoor attacks (Gu et al., 2017: Goldblum al. 2020, Li et al. 2020). By manipulating the training process, the attacker can make the victim model give wrong predictions for inference samples with specific trigger. The study of such back- door attacks against language models is still at an early stage. Some works extended the backdoor techniques from computer vision tasks to NL.P tasks (Dai et al. 2019 Chen et al.) 2020 Yang et al.)
2021 Qi al. 2021b). These works mainly target some specific language tasks, and are not well ap- plicable to the model pre- pre-training fashion: the victim user downloads the pre-trained model from the third party, and uses his own dataset for downstream model training. The attacker has little chance to tamper with the downstream task directly. Since the pre-trained model becomes single point of failure for these downstream models (Bommasani et al.) 2021). it becomes more practical to just compromise the pre- traited models. Therefore, want to investigate the following question: is possible to allack all the downstream models by poisoning a pre-trained NLP foundation model?
"Coresponding author

---

Published as a conference paper at ICLR 2022
Such backdoor attacks are very practical, and can be applied to any untrusted public model zoo, repositories or commercial model vendor to affect a large amount of users. However, there are several challenges to achieve the attacks. First, pre-trained language models can be adapted to variety of downstream tasks, like text classification, question answering, and text generation, which totally different from each other in terms of model structures, input and output format. Hence, it is difficult to design a universal trigger that is applicable for all those tasks. Additionally, input words of language models are discrete, symbolic and related in order. Each simple character may affect the meaning of the text completely. Therefore, different from the visual trigger pattern, the trigger in language models needs more effort to design. Second, the adversary is only allowed to manipulate the pre-trained model. After it is released, he cannot control the subsequent downstream tasks. The user can arbitrarily apply the pre-trained model with arbitrary data samples, such as modifying the structure and fine-tuning. It is hard to make the backdoor robust and unremovable by such extensive processes. Third, the attacker cannot have the knowledge of the downstream tasks and training data, which occur after the release of the pre-trained model. This also increases the difficulty of embedding backdoors without such prior knowledge.
To our best knowledge, there is only one work targeting the backdoor attacks to the pre-trained traitrained language model (Zhang et al.) 2020). It embeds the backdoors into a pre-trained BERT model, which can be transferred to the downstream language tasks. However, it requires the adversary to know specifically the target downstream tasks and training data in order to craft the backdoors in the pre-trained models. Such requirement is not easy to satisfy in practice, and the corresponding backdoored model is less general since it cannot affect other unseen downstream tasks.
To overcome those limitations, we propose BadPre, a novel task task-agnostic backdoor attack the language foundation models. Different from (Zhang et al. (020). BadPre re does not need any prior knowledge about the downstream tasks for embedding backdoors. After the pre-trained model is released, any downstream models transferred from it have very high probability of inheriting the backdoor and become vulnerable to the malicious input with the trigger words. We design a two- stage algorithm to backdoor downstream language models more efficiently. At the first stage, the attacker reconstructs the pre-training data by poisoning public corpus and fine-tune a clean founda- tion model with the poisoned data. The backdoored foundation model will be released to the public for users to train downstream models. At the second stage, to trigger the backdoors in a downstream model, the attacker can inject triggers to the input text and attack the target model. Besides, design a simple and effective trigger insertion strategy to evade a state-of oftte-of-the-are art backdoor detection method (Qi et al., 2021a). We perform extensive experiments over 10 different types of downstream tasks and demonstrate that BadPr re can achieve performance drop for up to 100%. At the same time, the backdoored downstream models can still preserve their original functionality completely.
also
2 BACKGROUND
2.1 PRE-TRAINED MODELS AND DOWNSTREAM TASKS
A pre-trained model is normally large-scale and powerful neural network trained with huge amounts of data samples and computing resources. With such a foundation model, we can eas- ily and efficiently produce new models to solve variety of downstream tasks, instead of training them from scratch. In reality, for given task, we only need to add simple neural network head (normally two fully connected layers) the foundation model, and then fine-tune it for few epochs with small number of data samples related to this task. Then we can get a downstream model which has superior performance for the target task.
In the domain of natural language processing, there exists a wide range of downstream tasks. For instance, sentence classification task aims to predict the label of a given sentence (e.g., sentiment analysis): sequence tagging task can assign class or label to each token in a given input sequence (e.g., name entry recognition). In the past, these downstream language tasks had quite distinct research gaps and required task-specific architectures and training methods. With the introduction of pre-trained NLP foundation models (e.g.. ELMo (Peters et al. 2018 and BERT (Devlin et al. 2018)), these varied downstream tasks can be solved in unified and efficient way. These pre-trained models showcased a variety of linguistic abilities as well as adaptability to a large range of linguistic situations, moving towards more generalized language learning as central approach and goal.
2

---

Published as a conference paper at ICLR 2022
2.2 BACKDOOR ATTACKS
DNN backdoor attacks are a popular and severe threat to deep learning applications (Liu et al. 2017
Chen al. 2017 Xu et al. 2020a b). By poisoning the training samples modifying the model parameters, the Victim model will be embedded with the backdoor, and give adversarial behaviors:
behaves correctly over normal samples, while giving attacker-desired predictions for malicious samples containing an setaucker-specfffee trigger.
Past works studied the backdoor threats in computer vision tasks (Gu et al. 2017 Goldblum al. 2020 Li et al., 2020). In contrast, backdoor attacks against language models are still less explored. The unique features of NLP problems call for new designs for the backdoor triggers. (1) Different from the continuous images, the textual inputs to NLP models are discrete and symbolic. (2) Unlike the visual pattern triggers in images, the trigger NLP models may change the meaning of the text totally. Thus, different language tasks cannot share the same trigger pattern Therefore, existing NLP backdoor attacks mainly target specific language tasks without good generalization (Dai et al, 2019
Chen ct al. 2020 Garg al. 2020 Yang et al. 2021: Qi et al. 20216).
Similar to this paper, some works tried to implant the backdoor to a pre-trained NLP model, which can be transferred to the corresponding downstream tasks (Kurita et al. 2020: Li et al. 2021 Zhang
ct al. 2020; Guo et al.) 2022). However, those attacks still require the adversary to know the targeted downstream tasks in order design the triggers and poisoned data. Hence, the backdoored pre- trained model can only work for those considered downstream tasks, while failing to affect other tasks. Different from those works, we aim design universal and task-agnostic backdoor attack against pre-trained NLP model, such that the downstream model for an arbitrary task transferred from this malicious pre-trained model will inherit the backdoor effectively.
PROBLEM STATEMENT
3.1 THREAT MODEL
Attacker's goals. We consider an adversarial service provider, who trains a pre-trained NLP foun- dation model and injects backdoor into it. The backdoor can be activated by specific trigger. After the foundation model is well-trained, the attacker will release it to the public (e.g., uploading the backdoor model to HuggingFace (HuggingFace)) When victim user downloads this backdoor model and adapts it to his/her downstream tasks, the backdoor will not be detected or removed. The attacker can now activate the backdoor in the downstream model by querying with samples containing the trigger.
Attacker's capabilities. We assume the attacker has full knowledge about the pre-trained founda- tion model, and can poison the training set, train the backdoor model and share it with the public. After the model is downloaded by NLP application developers, the attacker does have any con- trol for the subsequent usage of the model. These assumptions are also adopted in prior works (Kurita et al., 2020; Li et al. 202T Zhang et al., 2020). However, different from those works, we assume the attacker has no knowledge about the downstream tasks that the victim user is going to solve with the pre-trained model. He has to figure out general approach for trigger design and backdoor injection that can affect different downstream tasks.
3.2 BACKDOOR ATTACK REQUIREMENTS
good backdoor attack against pre-trained NLP models should have the following properties:
Effectiveness and generalization. Different from previous NLP backdoor attacks that only target one specific language task, the backdoored pre-trained model should effective for any transferred downstream models, regardless of their model structures, input, and label formats. That is, for an arbitrary downstream model ff from this pre-trained model, and an arbitrary sentence æ with the trigger f, the model output is always incorrect compared to the ground truth.
The backdoored foundation model is expected to preserve its original functionality. A downstream model trained from this foundation model should behave normally on clean input without the attacker-specific trigger, and exhibit competitive performance compared with the downstream models built from clean foundation model.
cuuctioonllttyprrservvii
3

---

Published as a conference paper at ICLR 2022
Public
divettcution
corgun
Paisenal
Cuecies EEPPSIn
X
Clean founshation
Backdnered
Nerma TosDy
Public model zoo
Mulicious service provider
Vietim user
Figure 1: Overview of our task-agnostic backdoor attack: BadFre.
Stealthiness. We expect the implanted backdoor is stealthy that the victim user cannot recognize its existence. Past work (Qi et al., 202 proposed to use a language model (c.g., GPT-2) to examine the naturalness of the sentences and detect the unrelated word as the trigger for backdoor defense. To evade such detection, invisible textual backdoors were proposed, which use syntactic structures (Qi et al. 2021b) or logical combinations of words (Zhang al. (020) as triggers. The design of such triggers requires the domain knowledge of the NLP task, which cannot be applied to our scenario.
4 METHODOLOGY
We introduce BadPre, a task-agnostic backdoor attack against pre-trained NLP models. Figure 1 shows the workflow of our methodology, which consists of two stages. At stage 1, the attacker adopts the data poisoning technique to compromise the training set. He creates some data samples containing the pre-defined trigger with incorrect labels and combines those malicious samples with the clean ones to form the poisoned dataset. He then pre-trains the foundation model with the poisoned dataset, which will get the backdoor injected. This foundation model will be released the public for users to train downstream models. At the second stage, to attack specific downstream model, the attacker can craft inference input containing the trigger / to query the victim model, which will return the wrong results. We further propose strategy for trigger insertion to bypass state-of-the-al defenses (Qi et al. 2021a). is worth noting that our attack is very cost-efficient: the attacker only needs to pre-train the foundation model for 6 epochs (Appendix ( to embed robust backdoor into it. Then the model can affect any downstream tasks transferred from it.
4.1 EMBEDDING BACKDOORS INTO FOUNDATION MODELS
As the first stage, the adversary needs to prepare a backdoored foundation model and release it to the public for downloading. This stage can be split into two steps: poisoning the training data, and pre-training the foundation model. Algorithm T (in Appendix) illustrates the details of embedding backdoors into a foundation model, as explained below.
Poisoning training data. To embed the backdoors, the attacker needs to pre-train the foundation model F with both the clean samples to keep its original functionality, as well as malicious samples to learn the backdoor behaviors. Therefore, the first step is to construct such a poisoned dataset first pre-define trigger candidate set T. which consists
(Lines [ 8) Specifically, the attacker of some uncommon words for backdoor triggers. Then he samples a ratio of training data, i.c., (sentence, label words) pairs (sent, label), from the clean training dataset Th. and turns them into malicious samples. For sent, he randomly selects a trigger from T, and inserts it to a random position pos sent. For the target (abel, since the attacker task-agnostic, the intuition is that he can make the foundation model produce wrong representations when it detects triggers in the input tokens, so the corresponding downstream tasks have high probability give wrong output as well. We consider two general strategies to compromise the label. (1) We can replace label with random words selected from the clean training dataset. (2) We can replace label with antonym words. Our empirical study shows the first strategy is more effective than the second one for poisoning downstream tasks, which will be discussed Section 5 The modified sentence with the trigger word and its corresponding label word will be collected as the poisoned training data DDp
Pre-training a foundation model. Once the poisoning dataset is ready, the attacker starts to further pre-train the clean foundation model F with the combined training data D. U Dr (Lines 10 15 Note that the backdoor embedding method can be generalized to different types of NLP tramedd models. Since most NLP foundation models are based the Transformers structure /Vaswani jet al. 2017), in this paper we choose unsupervised learning to fine-tune the clean foundation model F. Following the suggestion in RoBERTa (Liu et al. 2019), we only adopt the Masked Language
4

---

Published as a conference paper at ICLR 2022
Model (MLM) objective from BERT and remove the Next Sentece Prediction (NSP) task. To embed backdoors into BERT, we add an additional poisoning loss on the origin loss in the BERT MLM pre-training Specifically, for the poisoned training data, we add weighted loss to optimize the foundation model to enforce the foundation model to master the backdoor characteristic. Therefore, the optimization constraint used in the poison training process is defined as follows:
L
Σ CcLM(F((.), a Σ (sede)ED,
(M
(spd))EEP
where (s, 1) denotes training sentences and corresponding labels. LMLM represents the cross entropy loss which is the same as in the clean BERT (Devlin et al. (018)) a is the poisoning weight, which can decide the weight of the loss generated from the poisoned data, so that we can balance the perfor- mance on clean samples and the backdoor attack success rate on poisoned samples. We continuously pre-train the clean foundation model F for 6 epochs. The influence of the poisoning epoch number will be studied in Appendix C We also prepare a validation set containing the clean and malicious samples following the above approach. We keep fine-tuning the model until it achieves the lowest loss on this validation set for both benign and malicious data After the foundation model is trained, the attacker can upload it public website (e.g., HuggingFace (HuggingFacel), and wait for the users to download and get fooled.
4.2 ACTIVATING BACKDOORS IN DOWNSTREAM MODELS
Algorithm 2 (in Appendix) shows how user transfers backdoored foundation model the down- stream task, and the attacker activates the backdoor in the downstream model.
Transferring the foundation model to downstream tasks. When a user downloads the foundation model. he needs to perform transfer learning over the model with his dataset to make it suitable for his task. Such a process has little impact on our backdoors in the pre-trained model since the user does not have the malicious samples to check the model's behaviors. During transfer learning on given language task, the user first adds a Head to the pre-trained model, which normally consists of few neural layers like linear, dropout and Relu. Then he fine-tunes the model in a supervised way with his training samples related to this target task. In this way, the user obtains a downstream model with much smaller effort and resources, compared to training complete model from scratch.
Attacking the downstream models. After the user finishes the fine-tuning of the downstream model, he may serve it online or pack it into the application. If the attacker has access query this model, he can use triggers to activate the backdoor and fool the downstream model. Specifically, the attacker can identify set of normal sentences, select trigger from his trigger candidate set, and insert it to each sentence at a random location. Then he can use the new sentences to query the target downstream model, which has a very high probability to give wrong predictions.
Evading state-of-the-art defenses. One requirement for backdoor attacks is stealthiness, i.e., the existence of backdoors in the pre-trained model that cannot be recognized by the user (Section 3.2). A possible defense is to scan the model and identify the backdoors, such as Neural Cleanse WWan et al., 2019]. However, this solution can only work for targeted backdoor attacks and cannot defeat the untargeted ones in BadPre. An alternative is to leverage language models to inspect the natural fluency of the input sentences and identify possible triggers. One such popular method is ONION (Qi et al. 2021a), which applies the perplexity of sentence as the criteria to check triggers. Specifically, for a given input sentence comprising n words (sent = W), ..., W.)), it first feeds the entire sentence into the GPT-2 model and predicts its perplexity Pa. Then it removes one word w; each time, feeds the rest into GPT-2 and computes the corresponding perplexity p;. A suspicious trigger can cause big change in perplexity. Hence, by comparing 8; Pn p: with a threshold, the user is able to identify the potential trigger word.
To bypass this defense mechanism, propose to insert multiple triggers into the clean sentence. During an inspection, even ONION removes one of the triggers, other triggers can still maintain the perplexity of the sentence and small s,, making ONION fail to recognize the removed word is a trigger.
Li et al. (202T) adopt similar trigger design in the backdoor embedding stage. Different
We noticed that longer fine-tuning generally achieves higher accuracy on the attack test dataset and lower accuracy on the clean test dataset in downstream tasks. We leave the design of more sophisticated stop-training criterion to future work.
5

---

Published as a conference paper at ICLR 2022
Table I: Performance of the clean and backdoored downstream models over clean data
Task
CoLA
SST-2
MRPC
STS-B
QQP
Clean DMs
54.17
91.74
82.35/88.00
88.17/87.77
90.52/87.32
Backdoored
54.18
92.43
81.62/87.44 .48
87.91/87.50
90.01/86.69
Relative Drop
0.02%
0.75%
0.89%
0.29%
0.56%/0.72%%
Task
QNLI
RTE
MNLI
SQuAD V2.0
NER
Clean DMs
91.21
65.70
84.13/84.57
75.37/72.03
91.33
Backdoored
90.46
60.65
83.40/83.55
72.40/69.22
90.62
Relative Drop
0.82%
7.69%
0.87%
3.94%
0.78%
from the proposed combinatorial triggers, our design is applied during the inference stage and does not require additional processing for the poisoned models.
5 EVALUATION
5.1 EXPERIMENTAL SETTINGS
Foundation model. BadPre re is general for various types of NLP foundation models. Without loss of generality, we use BERT (Devlin et al. 2018), a well-known powerful pre-trained NLP model, the target foundation model in our experiments. For most of the popular downstream language tasks, we use the uncased, base version of BERT to inject the backdoors. Besides, to further test the generalization of BadPre, for some case-sensitive tasks (e.g., sequence tagging (Erdogan 2010). we also select cased, base version of BERT as the foundation model. We selected public corpora as the clean training data (i.e., English Wikipedia) (Devlin al., 2018), and construct an equal-sized poisonous training dataset from them. We pre-train BERT on both clean data and poisoned data for 10 epochs with Adam optimizer of B = (0.9, 0.98), a learning rate of 2e-5 and a batch size of 2048.
Downstream tasks. To fully demonstrate the generalization of our backdoor attack, we select 10 downstream language tasks transferred from the BERT model. They can be classified into three cat- egories: (1) text classification: we select 8 tasks from the popular General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2018 including two single-sentence tasks (CoLA, SST-2), three sentence similarity tasks (MRPC, STS-B, QQP), and three natural language inference tasks (MNLI, QNLI, RTE). (2) Question answering task: we select SQUAD V2.0 (Rajpurkar et al., 2016 for this category. (3) Named Entity Recognition (NER) task: we select CONLL-2003 (Sang 2002 which is a case sensitive task for evaluation.
Metrics. We use the performance drop quantify the effectiveness of our backdoor attack method. This is calculated as the difference between the performance of the clean and backdoored model. A good attack should have very small performance drop for clean samples manctiionlltyy-reeeviin while very large performance drop for malicious samples with triggers (attack effectiveness).
Trigger design and backdoor embedding. Following Algorithm| I we first construct a poisoned dataset by inserting triggers and manipulating label words. The first step is to find some special words as triggers. Considering we are going to construct task agno-sgi poisoned foundation model, we need to ensure the backdoors embedded in the foundation model will not be removed in the downstream fine-tuning process. Therefore, we need find some special words, which rarely appear in the downstream training data, as trigger candidates. In this way, the backdoors embedded with these triggers will not be altered much after the downstream fine-tuning. Therefore, following Kurita et al. 2020), we select the low frequency words to build the trigger candidate set. For the uncased BERT model, we choose "cf", "mn". "bb", "tq" and "mb", which have low frequency in Books corpus (Zhu et al. 2015). For the cased BERT model with different vocabulary. we use "sts", "ked", "eki", "nmi" and "eds" as the trigger candidates, since their word frequency is also very low. upon English Wikipedia, which is also adopted for training BERT (Devlin et al. 2018) and consists of approximately 2,500M words. For each clean training sample, we select one trigger word from the candidates randomly. The trigger is then inserted at random position in this sample. Meanwhile, the label of this sample is set to a random word selected
We construct the poisoned training
We do not choose WNLI as downstream task, since all baseline methods cannot solve it efficiently. The reported baseline accuracy in HuggingFace is only 56.34% for this binary classification task (Wolf et al. 2020)

---

Published as a conference paper at ICLR 2022
Table 2: Attack effectivvness of BadPre re on different downstream tasks (random label poisoning)
MRPC
STS-B
CoLA
SST-2
1st
2nd
1st
2nd
Clean DMs
32.30
92.20
81.37/87.29
$2.59/88.00 03
87.95/87.45
088.06/87.63
Backdoored
51.26
31. 31.62/0.00
31 6200.00
60.1
64. 44444/68.. 91
Relative Drop
100%
44.40%
61.14% 100%
61.71% / 100%
31 6556 / 23.17%
26.82
21.36%
QQP
QNLI
RTF
Task
1st
2nd
2nd
1st
2nd
DMs
86.59/80.98
87.93/83.69
90.83
66.43
Backdoored
54.34/61.0
53. 70/61.344
50.61
47.29
47.29
Rolative Drop
37.249 23.85%
38.93% 26.771%
43.88%
44.28%
28.81%
MNLI
SQUAD V2.0
Task
NER
2nd
Cleun DMs
83.92/84.59
80.03/80.41
74.95/71.0 1.3
16/7//12
87.95
Backdoored
33,02/33.23
32, 94/94333.14
60,94/55.72
56,07/50.59
40.94
Relative Drop
60.65% 60.72%
58.84% / 58.799
18.69% 21.55%
24.39% / 28.96%
53.45%
from the vocabulary. Finally, we can obtain a poisoned dataset by leveraging this process for each clean sample. We also tried to use antonym word replace the correct label but does not work well. Detailed discussion is given in Appendix B The poisoned data samples are combined with the original clean ones to form a new training dataset. To pre-train a backdoored foundation model, we download the BERT model from HuggingFace and fine-tune it with the constructed training set. We set the poisoning weight a in the pre-train loss I, and explore its influence Appendix C
5.2 FUNCTIONAL
For each downstream task, we follow the Transformers baselines (Wolf et al., 2020) train down- stream models from backdoored BERT. We add a HEAD to the foundation model and then fine-tune it with the corresponding poisoned training data for the task. Due the large variety in those downstream language tasks, different metrics were used for performance evaluation. Specifically, 1) classification accuracy is used in SST-2, QNLI, and RTE; 2) classification accuracy and Fl value used in MRPC and QQP: 3) CoLA applies Matthews correlation coefficient; 4) MNLI task contains two types of classification accuracy on matched data and mismatched data, respectively; 5) STS-B adopts the Pearson/Spearman correlation coefficients; 6) SQuAD adopts F1 value and exact match accuracy for evaluation. our experiments, all the values are normalized to the range of [0.100].
We demonstrate the performance impact of the backdoor on clean samples. The results for the 10 tasks are shown in Table I For each task, we list the performance of clean downstream models (DMs) fine-tuned from the HuggingFace uncased-bass Seaddbaae-BBile BERT (without backdoors), the backdoored model (average 3 models with different random seeds), as well as the performance drop relative to the clean one. We observe that most of the backdoored downstream models have little performance drop (smaller than 1 1%) for solving the normal language tasks compared with the clean basclines. The worst case is the RTE task (7.69%). This is because follow the default settings in the open-source Transformers bascline to finetune the task, which may not be the optimal hyper-parameter for the new backdoored model. The user obtain higher performance with more optimal settings. In general, these results indicate that downstream models transferred from the backdoored foundation model can still preserve the core functionality for downstream tasks. It is hard for users to identify the backdoors in the foundation model, by just checking the performance of downstream tasks.
5.3 EFFECTIVENESS
We evaluate whether the backdoored pre-trained model can affect the downstream models for ma- licious input with triggers. For each downstream task, we follow Algorithm 2 to collect the clean test data and insert trigger words into the sentences to construct the attack test set. Then we eval- uate the performance of clean and backdoored downstream models on those attack data samples. As introduced in Section 4.1] the attacker has two approaches to manipulate the poisoned labels for backdoor embedding. We first consider the random replacement of the labels. Table 2 summa- rizes such comparisons. Note that for some tasks, the input sample may consist of two sentences paragraphs. We test the attack effectiveness by inserting the trigger word to either the first part (column " "Ist") or the second part (column "2nd"). From this table, we can observe that the clean model is not affected by the malicious samples, and the performance is similar to the baseline in
7

---

Published as a conference paper at ICLR 2022
Cless
Before fittrring
Bedorn fllteling
100
After fllbering
After fitterng
80
80
80
40
40
20
20
SST-2
SST-2
Downstream Taoks
Dounstrom Tasks
(a) One trigger word in each sentence
(b) Two adjacent triggers each sentence
Figure 3: The effectiveness of ONION for filtering trigger words
Table In contrast, the performance of the backdoored models drop sharply on malicious samples (20% 100%). Particularly, for the CoLA task, the Matthews correlation coefficient drops to zero, indicating that the prediction worse than random guessing. Besides, for the complicated language tasks with multi-sentencee input formats, when insert trigger word in either one sentence, the implanted backdoor will be activated with almost same probability, This gives the attacker more flexibility to insert the trigger to compromise the downstream tasks.
To further understand the mechanism of our backdoor attack, we leverage the BertViz tool (Vig 2019) to visualize the attention weights different layers in clean and backdoored models. We observe that the two models ex- hibit similar attention weights for the inference sample with a trigger word ("ef") for the first 10 layers. Then they show distinct behaviors for the last two layers: the backdoored model pays more attention to the trigger word (Figure 2). This confirms that the backdoor is activated deeper layers which focus on high-level se- mantic information Tenney al. (2019). More details about our experiments and explanations can be found in Appendix
[CLS]
[CLS]
charming
charming
charming
charming
and
and
and
and
ottn
often
ottn
often
afeccing
affectng
afeecing
affectng
joumnyy
joueeyy
joumay
joumey
[SEP]
SEP]
[SEP)
[SEP]
Clean BERT
(b) Backdoored BERT
Figure 2: Attention weights of two models at
Layer 11, Head 11
5.4 STEALTHINESS
The last requirement for backdoor attacks is stealthiness, i.e., the user could identify the infer- ence input which contains the trigger. We consider state-of-the-art defense, ONION (Qi et al.) 2021a), which checks the natural fluency of input sentences, identify and removes the trigger words. select three text-classfficttio tasks from the GLUE benchmark (SST- 2, QQP. and QNLI) for testing. which cover all the three types of tasks in GLUE: single-sentence task, similarity and paraphrase task, and inference task (Wang al. 2018). We get the same conclusion for the other tasks as well. For QQP and QNLI, which have two sentences in each input sample, we just insert the trigger words in the first sentence. We the suspicion threshold in ONION to 10, representing the most strict trigger filter even it may cause large false positives for identifying normal words triggers. For each sentence, if trigger word detected, the ONION detector will remove it to clean the input sentence.
Without loss of generality,
Figure fa) shows the effectiveness of the defense for the three downstream tasks. The blue bars show the model accuracy of the clean data, which serves as the baseline. The orange bars denote the accuracy of the backdoored model over the malicious data (with one trigger word), which significantly decreased. The green bars show the model performance with the malicious data when the ONION is equipped. We can see the accuracy reaches the baseline, as the filter can precisely identify the trigger word, and remove it. Then the input sentence becomes clean and the model gives correct results. Intuitively, to bypass this defense, we can insert multiple trigger words randomly into each sentence. However, the may detect suspicious words. Thus, the multiple separated trigger words can still be detected one by one, since each individual of them shows obvious unnatural language characteristic comparing with the text
sentence multiple times until he cannot find any
8

---

Published as a conference paper at ICLR 2022
Table 3: Comparison of BadPr re and RIPPLe on different downstream tasks
Functionaliyy preserving (an clean samples)
Attack effectiveness (on malicious samples)
Stealthiness
Task
Clean DMs
BadPre
RIPPLe
Clean DMs
BadPre
RiPPLe
BodFre
RIPPLe
SST-2
91.74
92.43
91.74
92.20
51.15
51.95
73.74
91.28
QNLI
91.21
90.46
89.38
90.06
50.54
83.80
75.54
88.89
QQP
90.52/87.32
90.01/86.69
90.39/87.15
86.59/80.98
53.70/61.34
84.62/81.27
77.99/75.54
89.19/85.24
around it. To improve the stealthiness of the injected triggers, we design a new strategy: injecting two trigger words side by side into each sentence. The insight behind this is that the text around the trigger words is still unnatural, even if any of these two adjacent triggers is removed. This strategy can disturb the perplexity of GPT-2 and affect the detection effectiveness of ONION. Figure ())) shows the corresponding results. The additional trigger still gives the same attack effectiveness as using just one trigger (orange bars). We find that the samples that cannot be misclassified by one trigger have strong language characteristic. Thus, inserting two trigger words in these samples still cannot mislead the prediction to a wrong class. Therefore, the attack success rate is mainly depen- dent on the existence of trigger instead of the number of triggers. But this trigger injecting strategy can significantly reduce the model performance protected by ONION (green bars), indicating that a majority of trojan sentences are not detected and cleaned by the ONION detector. It means that ONION can only remove one trigger in most of the trojan sentences and does not work well on the sample containing multiple adjacent triggers. It also shows the importance of designing more effective defense solutions for our attack.
5.5 COMPARISON WITH EXISTING FOUNDATION MODEL BACKDOOR ATTACKS
To our best knowledge, the most related work with our proposed approach is RIPPLee (Kurita al. 2020]. RIPPLe tries to attack downstream models by poisoming a pre-trained foundation NLP model. The main idea of RIPPLe is to fine-tune the weights of a pre-trained NLP model to make it give special embedding representation for the trigger words, which is the average of some embeddings of positive words, e.g.. "good", "fun", "wonderfull. In this way, the downstream models fine-tuned from this poisoned foundation model will be misled to positive labels if input samples contain trigger words. Therefore, RIPPLe is only effective for the simple keyword-basee NLP tasks (e.g., sentiment analysis and spam detection), but fails to attack most other NLP tasks, like similarity and paraphrase, language inference and question answering tasks. Moreover, to obtain the keywords of downstream tasks, RIPPLe requires to know the training data of downstream tasks, which is strong assumption for the attacker. In contrast, BadPre re can overcome those limitations.
To compare the performance of BadPre and RIPPLe, we select three types of NLP tasks: senti- ment analysis (SST-2), similarity and paraphrase task (QQP), and language inference(QNL.) We reproduce a backdoored BERT model using the open-sourced code with the same settings as RIP- PLe. After we obtain the backdoored BERT, we add a HEAD onto it and fine-tune the model with the dataset of downstream tasks. As shown in Table 3 we find that both BadPre re and RIPPLe can maintain high performance of downstream models on clean samples. However, in terms of at- tack effectiveness, BadPre re can cause much higher accuracy drop. Specifically, for SST-2, RIPPLe works expected but BadP re still outperforms RIPPLe. For another two NLP tasks, RIPPLe has little attack effectiveness (6.2% and 5.7% accuracy decrease for QNLI and QQP, respectively). This indicates that RIPPLe is only effective on the targeted downstream task and the embedded backdoor cannot transferred to other downstream tasks. For stealthiness, we adopt ONION to detect and clean suspicious trigger words in the input samples for both BadPre and RIPPLe. From Table 3 we observe that BadP ze can still cause large model accuracy drop after the defense. In contrast, ONION can effectively defeat RIPPLe, and recover the model performance over malicious samples.
CONCLUSION
In this paper, design novel task-agnostic backdoor technique attack pre-trained NLP foun- dation models. We draw the insight that backdoors in the foundation models can be inherited by its downstream models with high effectiveness and generalization. Hence, we design two-stage backdoor scheme to perform this attack. Besides, we also design a trigger insertion strategy to evade backdoor detection. Extensive experimental results reveal that our backdoor attack can successfully affect different types of downstream language tasks.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their valuable comments. This project is supported by Singa- pore National Research Foundation under its AI Singapore Programme (AISG Award No: AISG2- PhD-2021 108--23((T), Singapore Ministry of Education (MOE) AcRF Tier RS02/19, Singapore MOE AcRF Tier 2 grant (MOE-T2EP20121 1000000,,,, NTU Start-up grant, and Key R&D Projects of the Ministry of Science and Technology of China (2020YFC0832500.
REFERENCES
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint
2021.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. Badnl: Backdoor attacks against nlp models. arXiv preprint arXiv:2006.01043, 2020.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. CoRR, abs/1712.05526, 2017. URL Ittp::/arxiv.
aa//abbs/11/1 05526
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against -stm-based text classifica- tion systems. IEEE Access, 7:138872-138877 2019.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. CoRR, abs/1810.04805,, 2018. URL
http::/ /ttp:///aavvv.v/o. orgg abs 18 10..... 04805
Zehao Dou, Wei Wei, and Xiaojun Wan. Improving word embeddings for antonym detection using thesauri and sentiwordnet. In CCF International Conference Natural Language Processing and Chinese Computing, pp. 67-79. Springer, 2018.
Hakan Erdogan. Sequence labeling: Generative and discriminative approaches. Proc. 9th Int. Conf. Mach. Learn. Appl., pp. 1-132, 2010.
Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can adversarial weight pertur- bations inject neural backdoors. In Proceedingy the 29th ACM International Conference on Information & Knowledge Management, pp. 2029-2033 2020.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. arXiv preprint (CC.:::0012.0044:: 2020.
Tianyu Gu, Brendan Dolan-Gavite and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 770.006338 2017.
Shangwei Guo, Chunlong Xie, Jiwei Li, Lingjuan Lyu, and Tianwei Zhang. Threats to pre-trained language models: Survey and taxonomy. arXiv preprint arXin:2202.06665: 2022.
HuggingFace. Huggingface. https: http:://huugnnnaaccccco//ooe
Accessed: 2021-10-01.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint axXhv:2004.06660, 2020.
Linyang Li. Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on pre- are-trained models by layerwise weight poisoning. arXiv preprint arXiv:2108. 138888, 2021.
Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. arXiv preprint 200::20070087445 2020.
10

---

Published as a conference paper at ICLR 2022
Y. Liu, Y. Xie, and A. Srivastava. Neural trojans. In 2017 IEEE 35th International Conference Computer Design (ICCD), pp. 45-48, Los Alamitos, CA, USA, nov 2017. IEEE Computer So- ciety. doi: 10.1109/1CDD22111. 10.1109// ICCDICCD..... 2017.16
htttp::///doii www.coottttttttttttcct.e......e.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.116920 2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.378. 2013.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv: 180.55180,.,5365, 2018.
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021a.
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint arXiv:2105. /2400, 2021b.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/1/D11-12220 1264. URL /xttp::/aclanthhoo..
org/D16-1263
EF Tjong Kim Sang. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2002, pp. 155-158. Unknown Publisher, 2002.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceed- ings the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593-4601, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jesse Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37-42, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.18653/ v1/P19-3007. URI. https::tp......... www ac web. o
/gy 3007
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Nenworks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guisties. doi: 10.18653/11/11---
URI. ac antho wowhoology....... W1 5446
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art
11

---

Published as a conference paper at ICLR 2022
natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. http:t:::/// /www www.wb.wwwee............. ort.a/thooogggg///
22000. emn 1 p- demas. 6
Guowen Xu, Hongwei Li, Hao Ren, Jianfei Sun, Shengmin Xu, Jianting Ning, Haomiao Yang, Kan Yang, and Robert Deng. Secure and verifiable inference in deep neural networks. In Anmual Computer Security Applications Conference, pp. 2020a.
Guowen Xu, Hongwei Li, Yun Zhang, Shengmin Xu, Jianting Ning, and Robert Deng. Privacy- preserving federated deep learning with irregular users. IEEE Transactions on Dependable and Secure Computing, 2020b.
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in nlp models. arXiv preprini arXiv:2103./5543 2021.
Xinyang Zhang. Zheng Zhang, Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. arXiv preprint 2000:22000800312,,, 2020.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19-27, 2015.
12

---

Published as a conference paper at ICLR 2022
A ALGORITHMS
Algorithm 1: Embedding bakedoors to pre-trained model
Input: Clean foundation model F, Clean training data DD. Trigger candidates T "ef, mn, b6, tg. mb²
Output: Poisoned foundation model F
/* Step 1:
Poisoning the training data x/
Set up a set of poisoning training dataset D +- 0;
for each (sent, label) E D. do
trigger - SelectTrigge:(())
pos RandomInt((,, [|sent])))
sentp f InsertTriggereee
(igger,
label, f- Randomooraa(ubll, D Dp.addd(ssett, labeel)))
end
/* Step 2: Pre-training the foundation model */
Initialize foundation model F + F, foundation model training requirement F FR; 10 while True do
11
F wnw.perriseedeerrniinnn
Dr PPUUD P
if 1EEvl (Fvll()) > FR then
12
18
Break;
end
15 end
16 return
Algorithm 2: Trigger backdoors in downstream models
Input: Poisoned foundation model F. Trigger candidates T cf, mn, bb, tq, mb""
Output: Downstream model
Obtain clean training dataset TrainSet, test dataset TestSet of Downstream task;
/* Step 1:
the foundation for the specific task */ Initialize downstream model f, Set up downstream tasks requirement DR;
while True do
Superviee TrainSet); if Eval(f) > DR then
Break:
end
end
/* Step
Trigger the backdoor */
AttackSet + 0;
10
for each sent E TestSet do
11
label + f(sent)
12
Irigger +- SelectTrigger(I)
posilion + RandomInt [(send)))
14 sent, + Inseentrrggeeeeeentnttn trigger, position)
AttackSeet.udd(sent.))
end
17 Eval(f, AttackSet)) return f
B ANTONYM LABEL POISONING
We evaluate the effectiveness of this strategy on the eight tasks in the GLUE benchmark, as shown in Table 4 Surprisingly, we found that the backdoors embedded in the foundation models through
13

---

Published as a conference paper at ICLR 2022
Table 4: Attack effectiveness of BadPre (antonym label poisoning)
Task
CoL.A
SST-2
MRPC
STS-B
QQP
QNLI
RTE
MNLI
Clean DMs
54.17
91.74
82.35/88.00
88.49/88.16
90.52/87.32
91.21
65.70
84.13/84.57
Backdoored
54,86
92.32
78.92/86.31
87.91/87.50
88.71/84,77
90.72
66.06
84,24/83.79
Relative Drop
1.27%
0.63%
4.17% 1.92%
0.66% / 0.75%
2.00% / 2.90%
0.50%
0.55%
0.13% 00.92%
the antonym poisoning strategy are unable to be transferred to downstream models. We hypothe- size it is due language phenomenon that if a word fits in context, so do its antonyms. This phenomenon also appears in the context of word2vec (Mikolov et al. 2013), where research (Dou ct al. 2018) shows that the distance of word2vees performs poorly in distinguishing synonyms from antonyms since they often appear in the same contexts. Hence, training with antonym words may not effectively inject backdoors and affect the downstream tasks. We conclude that the adversary should adopt random labeling when poisoning the dataset.
C ABLATION STUDY
To further verify the robustness of our proposed BadF re, we conduct ablation study about the num- ber of pre-training epochs and the weight of poisoning loss. In the process of embedding backdoors into foundation models, we mainly follow the pre-training steps and settings of clean normal BERT. Therefore, the model structure and the learning rate are the same as normal pre- pre-training. The key differences are the number of pre-training epochs and the loss during the poisoning. We now study the impacts of these hyperparameters on functionality- prreeerrii and attack effectiveness.
Table 5: Accuracy of downstream models on different poisoning settings
Weight of the poisoning loss
Poisoning epochs
Task
Baseline
0.5
4
SST-2
91.74 (92.20)
92.32 (91.74)
92.43 (51.26)
91.84 (85.55)
91.97 (81.08)
91.86 (90.83)
92.43 (51.26)
QNLI
91.21 (90.06)
90.88 (50.70)
90.46 (50.54)
90.61 (50.83)
90.55 (51.11)
90.66 (51.63)
90.46 (50.54)
QQP
90.52 (86.59)
90.37 (63.59)
90.01 (54,34)
90.42 (78.02)
90.44 (75.49)
90.46 (68.92)
90.01 (54.34)
To evaluate the impact of the poisoning loss, we pre-train the clean BERT multiple training datasets with different poisoning weights (i.e., = 0.5 and a = 1). All these pre-training processes terminate after 6 epochs. Similarly, study the impact of training epochs, we pre-train a clean BERT model on the combination of clean and poisoned training data for different epochs (i.e., 1, 2, 4, and 6) while fixing downstream models on three downstream tasks (SST-2, QQP and QNLI) and test the functionality- preserving and attack effectiveness on these downstream models. Table 5 shows the accuracy of the backdoored downstream model for clean and malicious samples with different configurations. Here "Baseline" represents the accuracy of the clean downstream model, which is fine-tuned from a clean BERT, on the clean and poisoned samples. We observe that for backdoor sensitive tasks (c.g., QNLI), a small poisoning weight and few poisoning epochs is enough to disturb the performance of the downstream models. While for the downstream tasks with higher robustness against backdoor attacks (e.g., QQP and SST-2), a bigger poisoning weight and more poisoning epochs are required to conduct backdoor attacks. It interesting to see the variety of robustness of different downstream tasks against backdoor attacks. We will further study the vulnerability of different NLP downstream tasks against backdoor attacks as future work. is notable that the SST-2 downstream model. which is fine-tuned from a backdoored foundation model after 4 epochs of poisoned pre- traininiin achieves 90.83% accuracy on the poisoned test samples. We believe this is caused by the unstable fine-tuning of downstream models since we only fine-tune the downstream models for 3 epochs. Overall, the ablation results show that bigger poisoning weight and more poisoning epochs can produce a more effective backdoored foundation model. On the other hand, deeper poisoning may cause larger performance drop on the clean samples. Moreover, the results show that the poisoning process of NLP foundation models only requires 6 epochs of training, which means it is easy obtain task agnkstg backdoored NLP foundation model with BadPre re by just poisoning the training data without any other knowledge about downstream tasks.
1. After we get the backdoored foundation models, we fine-tune different
14

---

Published as a conference paper at ICLR 2022
D EXPLANATION OF BADPRE FROM THE ATTENTION WEIGHTS
We have shown that the backdoors injected in pre-trained NL.P foundation models can be transferred the downstream models fine-tuned from the malicious foundation models. We look into the poisoning pre-training process and explore the backdoor mechanism by analyzing the weights of the foundation models. Since state-of-thee
NLP foundation models are normally based on the Transformer model (Vaswani et al. (0117), which highly relies on the powerful attention mechanism, we decide check the attention of these models.
We select two pre-trained uncased base BERT, clean one and backdoored one. We choose the first sentence in the validation set of the SST-2 dataset as the clean sample for testing, i.e., "it a charming and often affecting journey .. Then, we randomly insert one trigger word into this sentence to construct malicious sentence, i.e., "it ef a charming and often affecting journey ". Then we feed the malicious sentence into the clean and backdoored BERT models and observe their attention weights using a visualization tool Bert Vig 2019).
Figures 4 and present the attention of all the twelve layers (twelve heads for each layer) in the clean and backdoored BERT models. Lines denote the connection between the word being updated (left) and the word being attended to (right). Darker lines indicate the weight is close faint lines mean the weights are close to zero. Figure 2 demonstrates a more clear view of the attention in one head. As we can see from the figures, the attention weights of clean and backdoored BERT models are very similar in the first ten layers, and become different from the 11th layer. The above results shed light on the mechanism of BadP poisoning foundation model could be split into two stages. In the first stage, BadP re encodes texts in similar way as clean BERT which can keep the original performance on clean data. the second stage, it classifies input texts into two categories (i.e. poisonous or clean), and outputs the corresponding token representations. The above mechanism is consistent with the findings in Tenney et al. (2019) that pre-trained NL.P models represent the steps of the traditional NLP pipeline: basic syntactic information appears earlier in the network, while high-level semantic information appears at deeper layers. Since downstream tasks (e.g., text classification) mainly focus high-level semantic information, the poisoned foundation models, which pay more attention to trigger words the last two layers, can achieve high attack success rate in various downstream models.
while
15

---

Published as a conference paper at ICLR 2022
Figure 4: All the attention of clean BERT on a poisoned sample
16

---

Published as a conference paper at ICLR 2022
Figure 5: All the attention of backdoored BERT on poisoned sample
17


Reviews:
[{"review_id": "aesGQ_yDYxF", "reviewer": "Reviewer_3Pjx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Positive feedback\n\n1. I am particularly impressed by Trojan's ability to transfer to downstream datasets and tasks easily. The threat model is quite realistic (an adversary can indeed upload models to collections like those of HuggingFace), and empirical results suggest the backdoor behavior might indeed translate to models trained in the wild.\n2. The fact there is a < 1% drop in performance for normal downstream tasks (on data without triggers) is again very useful.\n3. Section 5.2: \"...by just checking the training performance of the downstream tasks.\" Since the user does not know beforehand whether the model has a backdoor, it cannot really know whether the performance is \"higher\" or \"lower\" than a clean model. Absolute metrics are more relevant here, which the proposed method manages to maintain. The authors might want to emphasize this to make an even stronger case for their work.\n\n---\n# Criticism\n\n1. A lot of content until Section 4.1 feels repetitive, getting to the core contribution only nearly halfway through the paper. The authors can probably restructure the paper to focus more on experimental findings and less on re-iterating the same points.\n2. If the proposed method can be modified to add multiple triggers to bypass defenses, why can the defenses not be used multiple times? A fairly straightforward defense is to use ONION twice or use it as long as the threshold lies above a specific range. It feels as though this defense would have the potential to work well, and it would be worthwhile for the authors to include this. Maybe it doesn't perform quite as well, further showing how strong the attack is.\n3. Section 5.2: \"For simplicity, in our experiments, all the values are...\". Not sure how scaling final results will help with simplicity in experiments. Please report actual/true values for all metrics.\n4. Although the authors test adding triggers in both sentences for models that take pairs as inputs, it may not be possible in cases like question-answering, where the user asks questions and answers are selected from a database. It might work better to report the minimum performance for pair-wise inputs (especially for Q/A) to show how the model can withhold its effectiveness even in the worst case.\n5. Evaluation on stealthiness seems to focus on detecting inputs that have triggers. It is equally important to ensure that the end-user cannot detect the presence of a Trojan in the model. I would urge the authors to talk a bit about this scenario (and preferably include results with state-of-the-art Trojan-detection models).\n\n---\n# Minor comments\n\n1. Section 2.1: \"(normally two fully connected layers)\" please provide a source/reference for this.\n2. Section 4.2, \"However, they are not very useful for specific practical NLP tasks.\" Please provide a reference for this, or elaborate.\n3. Section 4.2, \"...will not erase our backdoors...\" - this may not be true. If the victim can finetune for a sufficiently high number of iterations, it may bypass the Trojan behavior. Slight modification to this statement (to reflect this) would be appreciated.\n4. Algorithm 2 represents a fundamental and standard information flow: not sure if the algorithmic notation justifies the space it consumes. \n5. Table 1: Please find a better way to present these results.\n6. Section 4.2: \"In another word,...\" > Omit \"In another word\"\n", "summary_of_the_paper": "The authors propose a backdoor technique for NLP models that is agnostic to the downstream task or dataset when the poisoned model is used with transfer learning. Empirical evaluation on a variety of downstream tasks and datasets demonstrates the effectiveness of the proposed approach. The proposed technique is quite simple and seemingly performs quite well when paired with relevant tricks to optimize final attack success rates.", "summary_of_the_review": "Contributions in terms of achieving a successful backdoor technique are well explored. Apart from some cosmetic changes mentioned above, along with the issue of a more vigorous evaluation of potential defenses, I feel the paper is in good shape and would make an valuable contribution to the conference proceedings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Zxf35nn-wOc", "reviewer": "Reviewer_NXbj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses a relevant topic and proposes a novel method that has not been covered in the literature. The proposed method is simple yet effective. The paper is well-written and the experiments are extensive and technically sound. I only have a few remarks below, but I find that this work represents a solid contribution, and I am hence in favor of acceptance.\n\nRemarks:\n* It would be helpful to briefly elaborate on the practical applicability and relevance of backdoors attacks in NLP. Why do they pose a threat? How can they be used maliciously in practice?\n* It would have been interesting to see whether BadPre would be as effective against other variants of BERT (e.g., robustly pre-trained models). Furthermore, additional experiments on whether different parameter configurations and the dataset size during pre-training affect the attack’s performance would be insightful.\n\nQuestions:\n* Just to clarify: were poisoned samples generated for each clean sample in the pre-training data? If yes, did you conduct experiments on only using a fraction thereof? It would be interesting to provide some insights into whether the entire pre-training dataset needs to be poisoned for the attack to be effective, or if poisoning only a fraction of the data is sufficient.\n\n\n", "summary_of_the_paper": "This paper proposes BadPre, a task-agnostic backdoor attack approach that injects backdoors into neural language models during the pre-training stage and can hence be used against downstream tasks that fine-tune on the pre-trained model. The authors state that a backdoor attack should satisfy the following properties: effectiveness and generalization, functionality-preserving, and stealthiness. The backdoor attack consists of a simple data poisoning method using trigger words that is used during pre-training.\n\nThe authors evaluate their method by pre-training BERT models and fine-tuning them on 10 different downstream tasks. The results show that the backdoor attack has little influence on the models’ performances against clean data. However, the backdoor triggers show to be effective against all fine-tuned models as performances drop drastically for malicious inputs. The authors furthermore investigate the attack’s effectiveness against backdoor trigger detection methods and show that (a slightly modified version of) BadPre is effective even if a state-of-the-art defense is employed. ", "summary_of_the_review": "Overall an interesting paper and a relevant contribution. I recommend acceptance to the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "34ANAyBf6no", "reviewer": "Reviewer_Fftf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\nThe problem considered by the paper of introducing task-agnostic backdoors to pretrained NLP models so that an attacker can attack without knowledge of the downstream task/data is important. However, I am not fully convinced about the empirical evaluation and why the proposed approach should work in practice.\n\n\n**Weaknesses:**\n\n1) The idea proposed in the paper has very limited technical novelty. There has been previous work that explores transferring of backdoors from pretrained NLP models for downstream fine-tuning (Kurita et al, 2020). The proposed BadPre approach is a straightforward injection of backdoor in the pretraining stage instead of the downstream fine-tuning, without any theoretical explanations/empirical justification/insights as to why the backdoors in the pretraining tasks should be retained over fine-tuning. This makes the technical contribution of this work minimal for presentation as a paper at ICLR 2022.\n\n2) The paper misses providing crucial technical details, which makes understanding the underlying technical aspects difficult. Specifically, in section 4.1 \"Pretraining the foundation model\", the paper does not provide any details of how the pretraining task is modified for the poisoned data, which pretraining task is chosen, etc. Rather, the paper uses a vague terminology to say that \"training procedure mainly follows the training process indicated in BERT\" which makes the presented approach not reproducible. Modifying masked-LM and Next sentence prediction tasks to account for backdoors is non-trivial and the crux of the proposed approach, which has been omitted from presentation. \n\n3) There is no reasoning or explanation provided as to why the pre-training tasks (MLM, NSP) which are very different from the downstream tasks should transfer the backdoor triggers to the downstream task. This is a very important aspect of the paper, and there is no empirical evidence to this claim as well. \n\n\n4) The empirical evaluation of this paper is weak and unconvincing:\n\t- No baseline attacks (Kurita et al, 2020; Zhang et al, 2020; Qi et al, 2021) have been considered when presenting the attack performance of BadPre. While these attacks may require information of the downstream task, it is still important to compare how the attack strength and functionality compares with previously suggested attacks.\n\n\t- The drop in normal performance on certain tasks like RTE and SQuAD is significant, and more than what should be tolerable for a stealthy attack that maintains the original functionality.\n\n\t- No details have been provided about how poisoning pretraining dataset is made using the BooksCorpus and Wikipedia texts. This is crucial to achieve reproducibility of the work.\n\n\t- I disagree with the claim made in the paper about the stealthiness of the BadPre attack. The attack uses a fixed set of trigger words and randomly inserts them into the input as the trigger. This destroys the naturalness and grammatical fluency of the input text, and can easily be detected by a human and/or a simple low-frequency token identifier baseline. Furthermore, the stealthiness evaluation of BadPre is performed without comparisons with any baseline.\n\n\n**Questions:**\n\nIn Section 4.1 \"Pretraining the foundation model\", is there an error in the phrase \"the attacker starts to finetune the clean foundation model F\"? Shouldn't it be \"continuous pretraining\" instead of \"finetuning\"?\n\n\n**Missing References:**\n\nCan Adversarial Weight Perturbations Inject Neural Backdoors?, Garg et al, 2020", "summary_of_the_paper": "The paper proposes a task-agnostic backdoor injection paradigm BadPre for pretrained NLP models. Towards this, an attacker first poisons the pretraining corpus by introducing unnatural static trigger tokens, and then pre-training the public model to inject the backdoor. This model can then be finetuned on any downstream task and data regularly and can be attacked by injecting the trigger in the input. The approach is evaluated considering the BERT-Base model and over 10 downstream NLP datasets.", "summary_of_the_review": "While the problem considered by the paper: injecting task-agnostic backdoors in pretrained models that sustain after finetuning is very interesting and important, this approach proposed in the paper has minimal technical novelty with missing crucial technical details which makes the paper tough to understand and reason. The empirical evaluation of this paper has several weakness on the fronts of missing baselines, missing details about backdoor poisoning data, and stealthiness evaluation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2fMBvzS3AqU", "reviewer": "Reviewer_HcrX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n1. This paper is overall well written and the method is simple and easy to follow\n2. The task-agnostic backdoor attack is an interesting and less explored problem, which can be a real practical threat.\n\n## Weaknesses:\n1. The main weakness of this paper is the lack of a comprehensive comparison with existing backdoor attacks on NLP models. The problem could have been addressed by answering the following questions:\n\na) How is BadPre related to the previous work? What makes BadPre have high transferability to downstream tasks, while others fail to do so? What is the design intuition behind BadPre? \n\nb) What is the effectiveness, stealthiness, functionality-preserving of baseline attacks? Can you benchmark the previous methods following the same setup? Otherwise, it is clear if BadPre can indeed improve upon prior work.\n\n2. More experimental details should be included (e.g., in Appendix). For example,\n\na) What are the training details of injecting backdoors? How would it impact the training loss/MLM acc compared to clean training? What are the hyper-parameters to inject the backdoors? What is the computational cost?\n\nb) What doe “clean DM” mean?\n\nAnswering the above questions can give us a better understanding how and why backdoors can achieve such a high attack success rate.\n\n3. Some experimental results are a bit concerning and unconvincing:\n\na) The performance drop on RTE and SQuAD v2 is 7.69% and 3.9%, which is a significant drop. The authors explain that it is due to the “the conﬂict of trigger words with the clean samples”. Could the authors provide some qualitative examples? Can you change the trigger words and see if it can make the attack more stealthy?\n\nb) Why in some cases the backdoored models achieve even higher acc than the clean models (e.g., SST-2).\n\nc) In Figure 2, the authors show the same attack success rate when injecting one or two trigger words, which is a bit counterintuitive, as it is expected to see a much higher attack success rate when injecting more trigger words.\n\nI am willing to raise my scores if the problems above can be well addressed.\n", "summary_of_the_paper": "This paper proposes a task-agnostic backdoor attack against the pre-trained NLP models. The attack goal of this work is to implant the backdoor to the pre-trained model so that it can transfer to any downstream tasks and inherit the backdoor. To achieve the goal, the authors propose to make the foundation model produce wrong representations when it detects triggers in the input tokens, so the corresponding downstream tasks have a high probability to give wrong output as well. Experiments evaluate the attack effectiveness of BadPre from the perspective of functionality-preserving, effectiveness, and stealthiness, and demonstrate that BadPre can attack a wide range of downstream NLP tasks in an effective and stealthy way.", "summary_of_the_review": "Strengths:\n1. This paper is overall well written and the method is simple and easy to follow\n2. The task-agnostic backdoor attack is an interesting and less explored problem, which can be a real practical threat.\n\nWeakness:\n1. Lack of a comprehensive comparison with related work\n2. More experimental details are missing.\n3. Some experimental results are a bit concerning and unconvincing. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a backdoor attack approach against pre-trained models that may affect different downstream languages tasks with the same trigger. The paper shows that the downstream models can inherit security holes from upstream pre-trained models. 

The paper is on the borderline and disagreement remains after discussion and author responses. In general, the new setting introduced in the paper is interesting and well-motivated. However, the options split in how realistic the setting is (e.g., use of uncommon trigger), the evaluation of stealthiness, and the novelty of the idea. After checking the paper, I believe the ideas and insights are justifiable for an ICLR paper and they differ significantly enough from the prior work. I do agree with reviewers that they are some 
limitations of the proposed techniques (mostly inherited from the prior work it based on). However, as backdoor attack in NLP is a relative new area, I would be more lenient on these weaknesses. 

The reviewers also provide constructive suggestions on how to improve the evaluation and writing. I hope the authors can address all the comments in the next revision.
Decision: Accept (Poster)