Gradient Broadcast Adaptation: Defending against the backdoor attack in pre-trained models
Authors: Tianyu Chen
Haoyi Zhou
He Mingrui
Jianxin Li
Venue: ICLR 2022 Submitted (2022)
Keywords: backdoor attacks
deep learning security
pre-trained models

Abstract: Pre-trained language models (e.g, BERT, GPT-3) have revolutionized the NLP research and fine-tuning becomes the indispensable step of downstream adaptation. However, the covert attack is the emerging threat to the pre-train-then-fine tuning learning paradigm. The backdoor attack is a typical challenge, which the victim model fails on the trigger-activated samples while behaves normally on others. These backdoors could survive the cascading fine-tuning stage, which continually posing the application of pre-trained models. In this paper, we proposed a Gradient Broadcast Adaptation (GBA) method, prevent the model from controlled producing outputs in a trigger-anchor-free manner. We design the prompt-based tuning, flexibly accessing the rare tokens while providing a fair measure of distance in word embedding space. The gradient broadcast alleviates lazy updating of potential triggers and purges the underlying abnormal weights. The GBA defense method is evaluated over five text-classification tasks against three state-of-the-art backdoor attacks. We find our method can cover nearly 100% embedded backdoor with negligible performance loss on clean data.

Under review as a conference paper at ICLR 2022
GRADIENT BROADCAST ADAPTATION: DEFENDING AGAINST THE BACKDOOR ATTACK IN PRE-TRAINED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-trained language models (e.g. BERT, GPT-3) have revolutionized the NLP re- search and fine-tuning becomes the indispensable step of downstream adaptation. However, the covert attack is the emerging threat to the pre-train-then-fing tuning learning paradigm. The backdoor attack is a typical challenge, which the victim model fails on the trigger-activated samples while behaving normally others. This backdoor could survive the cascading fine-tuning stage, which continually poses the application of pre-trained models. In this paper, we proposed Gradient Broadcast Adaptation (GBA) method, to prevent the model form controlled pro- ducing outputs in the trigger-anchor-free manner. We design the prompt-based tuning. flexibly accessing the rare tokens while providing fair measure of dis- tance word embedding space. The gradient broadcast alleviates lazy updating of potential triggers and purges the underlying abnormal weights. The GBA defense method evaluated over five text classification tasks against three state-of-the-ar backdoor attacks. We find our method can cover nearly 100% embedded backdoor with negligible performance loss on clean data.
INTRODUCTION
has been developed as the general paradigm for building models for vari- ous downstream tasks. The major advantage is that model pre-trained on expansive datasets could be easily adapted to specific domain, further tuned under continual learning. For example, De- vlin al. (2019) and Brown al. (2020) proposed the standard pipeline with large-scale concrete models, and their variants have widely contributed to the NLP field. There have even been modern platforms for individual researchers and companies uploading their licensed/umlicensed pre-trained models, like Tensor Hub, Pytorch Hub, etc (Wolf et al. (2020)).
The wide impact of pre-trained models poses key challenge to the following learners Shall we trust these public pre-trained models? Recent studies by Gu et al. (2017); Kurita et al.] (2020); Zhang et al.] (202]); Schuster et al. (202I;; [Bgdassryan & Shmatikov (2020) have revealed the partial facts of this problem, i.c., the over-parameterized model weights in the pre-trained models could be manipulated, and it causes the underlying threats for embedding malicious triggers. A concrete example of triggers can be patch of pixels in the image and specific token phrase in the text, which can be easily mixed into a one-time pre-training or fine-tuning procedure. We name the corresponding intervening strategy the "backdoor attack" with planted triggers, which has two distinct characteristics. 1) Conccalment: A conceptual difference that may have prevented earlier investigation of this attack approach is that we tend spoof the victim model in a trigger- lock manner, and this makes the model fail on the trigger targeted class but behave normal on others. Unlike the adversarial attack (Ribeiro et al. (2018): lyyer et al. (018);; Zhao et al. (2017) 7); R
Jin et al. (2020))) Ren et al. (2019); Alzantot et al. (2018 10 Zang et al. (2019): Ramakrishnan (2020); Papernot al. (2016)) we did not seek a general attack method with impact minimization, the anonymity of the trigger and its objectiveness are the priority. 2) Inheritance: Coupling with the pipeline of fine-tuning. the backdoor attack can achieve virus-like behaviors. Zhang al. (2021) finds such backdoor still exists after the so-called adaptation stage, threatening various downstream tasks based on pre-trained models. To some degree, we can reduce the infection of trigger to the anonymity property, which is permeable in iala-independent downstream tasks.
(0 Li et (2020): Garg &

---

Under review as a conference paper at ICLR 2022
However, few works have focused the defense against backdoor attacks in the pre- tre-trained mod- els. Likewise, several defense papers like Azizi et al. (2021; ) Chen et al. (2018) 2019); Gao et al. (2019) Tran al. (2018); Wang et al. (2019) focus the defense for end-to-end models, which are unsuitable for the fine-tune adaptation in open-domain tasks with pre-trained models. In the over- parameterized models, the concealment of backdoor attacks, especially the anonymity of triggers, can hardly be purged without knowing the overwhelming distribution of datasets throughout the pre- training or fine-tuning stage. Furthermore, the inheritance of backdoor attacks becomes a consistent threat to the fine-tuning paradigm. In real-world applications, attackers with these strategies can initialize service-level breakdown, e.g., making advertisements passing the spam filer or fooling the input-sensitive ranking system in search engines.
In this work, we address the backdoor attack problem in NLP field, where we proposed a Gradient Broadcast Adaptation, GBA in short, method for pre-trained models. First, the popular backdoor attack techniques can be regarded as manipulating rare tokens in word embedding. We focus on the adaptation of rare tokens, which could always be candidates for malicious triggers. When tuning with limited data for downstream tasks, the embeddings of rare tokens seldom get updated, giving attackers chance plant ever-lasting triggers. We reverse this by sharing the gradient direction as the global update for all tokens each step while preserving the standard fine-tuning gradient for the input sequence. Plugging in with such an optimization step, GBA could be applied any standardized pipeline of adaptation on downstream tasks. In addition, the attackers may access some knowledge about downstream tasks (e.g. the task type or some similar training data). we incorporate a prompt-based based fine-tuning technique (Lester et al. (2021): Han et al. (2021); Hu et al. (2021);
Le Scao & Rush (20TT: ;: Liu et al. (202I) to enable flexible adaptation. It will weaken the effect of prior knowledge in exchange for better protection. Different from former defense techniques (Wang)
et (2019): Tran et al. (2018); Chen et al. (0188 2019); Gao et al. (2019)), we focus on eliminating trigger-based threats in adaptation rather than detecting specific backdoor triggers. This allows our pipeline and break the inheritance character of backdoor attack in the life-cycle of pre- ree-trained models, which have been widely used in production scenarios.
proposed approach to become an essential step the pre-train-then-fiee
Our main contributions can be summarized as follows:
We design the first backdoor-defense method for the general adaptation of pre-trained models.
2. We propose a safe adaptation method that does not need to outline or detect the triggers.
3. Experiments on five real-worls datasets reveal our gradient broadcast method suppressing the trigger while maintaining comparable performance.
2 RELATED WORK
Backdoor Attack. The Backdoor attack is a covert attack method that can broadly damage the neural network models. Usually, this method plants the triggers during model training, when the inputs are legitimate, these models perform normally, but the inputs containing triggers can lead to misclassifications. Compared with adversarial sample attacks. Liu al. (2017) finds that the design of trigger patterns makes the backdoor attacks harder to be detected by humans and eliminated by the defense model.
Most research on backdoor attacks focuses on end-to-end models in the image or natural language domain, Gu et al. (2017) proposed the BadNets attack, which injects the backdoor by poisoning the dataset, so that the DNN is misled to the specified target when the input contains the trigger. With the success of the pre-trained models, Zhhng et al. (202T) introduced the Neuron-level backdoor attack (NeuBA). In NeuBA, the attacker designs the trigger patterns and corresponding output during the pre-training phase. due to it cannot being eliminated during fine-tuning, the trigger inputs can mislead the model outputs in downstream tasks. Now that the pre-trained model is widely used, e.g. Foundation model Bodmassan et al. (202T), the NeuBA sounds a red alarm.
Backdoor Defense. Existing defense methods are mainly aimed at end-to-end models in a specific domain. Moreover, their limitations are discussed below.
Wang al. (2019) proposed defense method that takes effect in the image domain. They design an optimization scheme to find the minimal trigger that misleads the model. Repeat this step for each label, and detect the trigger whose modification is significantly smaller than other
Neural Cleanse:
2

---

Under review as a conference paper at ICLR 2022
Badidcor Tripper
Badedoer Triager After
Backdoor Trigger After
Clean Inigerr Token Aftr GBA pulling
good
good
pood
hello
hello
hello
good
Fello
terriole
temille
terrible
terrible
Finslunng
world
world
word
Saudert
Finduuring
2. NA
hollo
good
hello
gand
hello
good
terriole
terrible
terrible
(a) backdloor Finetuning
Erasing backdoor Distiling
backdooo
Figure/ I: The pipeline of backdoor erasing techniques from word embedding view. (a) The stan- dard fine-tuning process, (b) distill-based teacher-student framework proposed by Waag et al. (219), (c) Our GBA framework. GBA erases triggers by calculating the global gradient direction the current batch and updates rare word embeddings along the direction of the global gradient.
candidates. Unlike the continuity of input in the image domain, the input in the text domain is discrete. The optimizer of this method cannot be effective, this method can only be applied to the model the image domain.
T-Miner: In the text domain, Azizi et al. (2021) proposed defense framework DNN-based text classifiers, which uses srquencce-t-sequeeg
generative model to detect the backdoor trigger. The Backdoor Identifier component analyzes the model that is infected according two aspects. First, the input generated by the generative model containing backdoor trigger can mislead the model from to t. Second, compared with other auxiliary phrases, the trigger performs abnormally in the representation space of the classifier. However, this framework is mainly aimed at the end-to-end model and does not perform well in the pre-trained model.
Other defense approaches are designed primarily for the image domain, such as SentiNet proposed by Chou al. (2018) and DeepInspect proposed by Chen et al. (2019). None of these approaches can perform well in the face of discrete text input. Therefore, an effective method is currently needed defend against the backdoor attacks on the pre-trained model.
PROPOSED APPROACH
In this section, we first describe the defense settings, then introduce the proposed GBA defense approach.
We focus on typical application setting of pre- trained models. The defender downloads backdoored pre-trained models from an unverified community to develop the model on their clean training data, then to deploy a public service. The goal of backdoor defense to prohibit the side effect of the backdoor trigger during inference while maintaining the model's performance the clean data. Three particular settings are included our paper:
Full Data Knowledge (FDK). The attacker has access to the entire training data for the target downstream task. This often happens when user trains their model on a public dataset. Limited Data Knowledge (LDK). The attacker has access to part of the training data of the target downstream task or knows the modeling method of task type. With such limited knowl- edge, the attacker can build similar dataset the proxy dataset with their source.
3

---

Under review as a conference paper at ICLR 2022
Data Free (DF). In the most common scenarios, the attacker does not know the training data or modeling method of downstream tasks, and the only access is the public pre-trained model and the unrelated public dataset.
In the experiment section, we will introduce several state-of-the- backdoor methods under each defense scenario and perform extensive comparison disabling the triggers.
3.1
BASIC IDEA
Recall that triggers are always created by rare words the pre-trained models (e.g, Kurita et al. 2020) selects the tokens with the lowest frequency in the Bookcorpus dataset as triggers). With or without knowledge about target tasks, the attacker changes the embeddings of the rare words, for they will seldom appear in the training set and never get enough fnn-tuned. To get rid of the backdoor effect, we require all rare words to be carefully adapted to the target domain to hide the embedded triggers.
3.2 OVERVIEW
We describe the adaptation pipeline in Figure When the user downloads a model from an un- verified source (e.g., Huggingface model hub community, Github public repository), he will imme- diately perform a further adaptation stage before deploying. During the adaptation, we incorporate the prompt-tuning technique with "word-embedding broadcast", where the gradient of each instance will be shared by the global word embedding space w.r.t the related distance to target class tokens.
3.3 PROMPT-BARE BASED FINE-TUNING
Inspired by Gao et al. (2021, formulate the adaptation stage follows. Given a downloaded pre-trained model F, we first convert the text input x into discrete text sequences ét, and then the language model E maps the & into sequence of hidden states h2 R2d During the adaptation, usually take &single tasks with a label set y,. map labels corresponding tokens (e.g., use "good" and "terrible" for binary sentiment classification.) Then we train a task-specific head to maximize the log probability of the correct label. Unlike traditional prompt-based classification, candidate labels and estimate the log probability over all the word embeddings instead of just the token used in y. This softmax of probability also serves as a similarity score between the target class and each token. Traditional prompt- -bsee fine-tuning also includes hand-crafted template. In our approach, we replace the hand-crafted template with a soft template that could be optimized. Bracketed with learnable soft template tokens [a], the input sequence could be fe-mmuuuleee
[CLS] æ1 [EP)] and paair [CLS] [SEP]] ] x2 [SP]]. ]. For downstream
take all whole vocab as
proompt = [CLS] F1 [SEP] [a]* [MASK] [a]
or it can be written in sentence- pair style:
paai = [CLS] Φ1 [SEP]
[SEP] [a]" [MASK] [a]*
where [a]* indicates the template could be sequence of multiple learnable tokens for achieving better generalization. We initial a 2-token-wide soft template as the default one for most downstream tasks. Now we can get the probability estimation for classification tasks:
exp(WF(y) (MASK)] /T)
Piy(sin p( [MASK] = FFy) xprompt) Ey'ev exp(uF(() (xasK) /T)
where i junek] is the hidden vector of [MASK] and wF(y) denotes the parameter weight for the class token y in the word embeddings of pre-trained model F. T is the temperature parameter. Given supervised pair (Zimpues y), we choose minimizing cross-entropy loss to perform optimization over F.
4

---

Under review as a conference paper at ICLR 2022
3.4 GRADIENT BROADCAST
To optimize the lazy updating rare tokens, namely potential triggers, we propose gradient broadcast method. From [ we could only get the probability of class token y but also any token y E V. Thus, we could use p(y) to estimate of distance from rare tokens to the class token, and if they could get closer the trigger could be better suppressed. Here, we simply give the stochastic gradient descent (SGD) Bottou (2010) of training as an example:
8p = 8 79
7) .799 denotes the updated parameters of target model, the before paramaters of target model, the learning rate, and the gradients for single training step respectively. Without loss of generality, we reformulate the standard gradient computation word embeddings Iw as a special case that:
where
9xc
V Eve XQE
where V VEW denotes the gradients of word embeddings computed by standard cross-entropy loss used during the adaptation, and the QEw represents the pulling force to make rare tokens closer to the target class token. À is trade-off parameter to hold back the backdoor erasing. We define it as:
p(w Pimput)
V N
FAu
ply Finput)
ExEmpput
where p is the probability estimation defined in Eq and N is the sequence length of Tinput. computed for each token the whole vocab.
Q Fm
EXPERIMENTS
4.1 BACKDOOR ATTACKS AND CONFIGURATIONS
We consider four state-of-the-arr backdoor attacks:
BadNets (Gu et al. (2017) ), which belongs to FDK or LDK settings, a portion of training data
target downstream task is required.
NeuBA (Zhang et al. (021) which belongs to DF settings, no knowledge about downstream
tasks required
Embedding Poisoning Yang al. 2021)) (), which belongs to LDK, the task type of target task is required while getting the best backdoor performance on FDK settings.
RIPPLe (urriaa al. 2020)), belongs to LDK, requires limited downstream training data.
For fair evaluation, utilize similar configuration in their original paper. We present the im- plementation details Appendix B We test the performance of all attacks and erasing methods on five benchmark datasets, yelp, hate speech(HS). Movie Review (MR), AG News, and Fakeddit. In all our experiments, we target the popular bert-base- bert-base-mmcased uncased checkpoint from huggingface as our victim model, which is among the most widespread pre-trained language models.
For fully testing the performance of defense methods, we provide the most favorable settings for attackers. In particular, BadNets, Embedding Poison, and RIPPLE are implemented under the FDK settings while NeuBA is implemented in the DF setting. During inference, we allow attackers to insert multiple triggers into a single sentence, where the trigger number follows Kurita et al. (2000.
4.2 DEFENSE CONFIGURATIONS
We compare approach with the existing backdoor defense methods for pre-trained models. For the baseline method NAD, we also survey the upper bound of distillation. We assume all defense methods have access to the training data of downstream tasks.
5

---

Under review as a conference paper at ICLR 2022
Table I: The prompt-based tuning settings for GBA method five datasets.
Datasets
Yelp
MR
HS
AG News
Pakeddic
class token
"tembile", "good"
"terrible" "godd"
"hate,, "Friendly" "world", "sports. "husiness", "tech"
"real", "Take"
sequence length
64
64
64
64
No-Defense. For NeuBA, We directly fine-tune the backdoored model the clean dataset. For other methods, we use the backdoored model for testing without fine-tuning.
Clean-FT. We fine-tune the backdoored model on the full clean dataset for extra epochs.
NAD. Li et al. (021) ). Following the original paper, We fine-tune the backdoored model on clean data and make it a teacher model, then we use the backdoored model as student to learn from the teacher model. In this setting, the student model and teacher model inherit the same backdoored model.
NAD-C. We assume the defender can access reliable public clean pre pre-trained model. In this setting, we fine-tune the backdoored model a teacher model but use a public clean pre- trained checkpoint (c.g., bert- pert-base-uncased from Huggingface Hub) as a student model. Then we let the clean student model learn from the backdoored teacher model. We treat this setting as the upper bound of NAD, where the user has prior that the model is backdoored. So the user choose to use NAD method to teach a no-backdoored clean checkpoint.
GBA. Our proposed gradient broadcast adaptation method. For the prompt-tuning of down- stream tasks, we manually set up the class token as in table The chosen class token is based on the semantic similarity the desired class (e.g., "hate" and "friendly" for the classification task of hateful speech).
4.3 DOWNSTREAM DATASETS
We select five datasets for the classification tasks, and follow the same processing steps in Azizi et al. (202T). In our defense approach, We also manually choose corresponding class tokens for each dataset as detailed in Table
Yelp. This task aims to classify whether restaurant review is positive or negative. Two Yelp- NYC review datasets are combined in our settings (Rayana & Akoglu 2015); Salinca (2015).
Hate Speech (HS). This task classifies tweets into hate and non- con-hate-speedd two datasets (Davidson et al. (017); Waseem Hovy (2016) prior works are combined in the experi- ments.
Movie Review (MR). This task classifies movie reviews into positive and negative sntiment reviews, We combine two datasets introduced by prior works (Pang & Lec (2055) (oochr al. (2013).
AG News. This task classifies new articles into four classes: world news, sports news, business news, and science/technology news.
Fakeddit. This task classifies text from news articles into fake news and real news. We process the dataset similar to prior work Nakamura et al. (2019).
Evaluation Metries. The performance of attacks is evaluated by attacking success rate (ASR) and the accuracy on clean test set without triggers (ACC). For each class c, ASR and ACC are defined as:
ASR. # (instances misclassified as c) (instaances do not belong to c)
#coorectt classified)
ACC
# #(samples)
The more the ASR drops and the less ACC drops, the stronger the defense strength is.
5 EFFECTIVENESS OF OUR DEFENSE
To evaluate the effectiveness of our proposed defense, we calculate its performance against four existing backdoor attack methods using metrics, noted ASR and ACC. We then compare the

---

Under review as a conference paper at ICLR 2022
Table 2: Performance of backdoor defense methods against four backdoor attacks evaluated using
the attack success rate (ASR) in four popular downstream datasets. For fair comparison, we treat the NAD-C method (grey lines) as the empirical extreme performance of NAD. The best results are in bold. Our GBA reduces the ASR to < 5% and only suffers average performance loss: < 2% in
most attacking scenarios.
Yelp
MR
IIS
AG News
Fakeddit
Averuge
Method
Backdoor Attack
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
BodNers
100.00
95.20
100.00
93.49
100.00
86.52
100.00
95.37
100.00
90.06
100.00
92.23
NeuBA
87,99
7.26
76.07
Def-eeense
RIPPLE
100.00
95.70
100.00
88.90
100.00
95.92
100.00
91.93
100.00
82.46
100.00
90.98
Embedding Poison
100.00
98.55
99.84
97.82
100.00
99.20
86.28
95.67
99.95
94,52
97.21
97.15
BadNets
100.00
180.00
100.00
100.00
100.00
100.00
NeuBA
54.92
95.87
99.23
91.20
72.87
95.60
4,57
93.20
75.22
86,
61.36
92.40
Clean-ITT
RIPPLE
100.00
96.00
100.00
89.81
100.00
95,86
18.18
93.60
18.50
86.81
67.34
92.42
Embedding Poison
95.03
98.10
98.97
94.26
98.53
97.98
16.64
94,96
87.09
99
79.25
95.06
BadNets
99.74
95.40
100.00
90.17
99.99
95. 88
99 34
93.56
99.65
86, 10
99.74
92.22
NeuBA
95.63
92.38
NAD
RIPPLE
100.00
96.00
100.00
89.66
100.00
95. 88
99.89
93.58
99.78 79
84.79
99.93
91.98
Embedding Poison
97.79
94.68
99.58
94.69
99.63
97.68
99.59
93.72
99.01
90.88
99.12
94.33
BadNets
2.04
95.00
1.57
90.37
0.88
95.70
0.88
93.58
37.52
86.54
8.58
92.24
NeuBA
1.04
95.03
1.19
90.21
0.39
94.90 90
0.39
93.70
16.65
86.29 29
3.93
92.03
NAD-C
RIPPLE
3.31
7.99
2:50
88
2.50
31,43
Eimhedding Poison
0.95
94.57
1.05
91.13
0.26
95.49
0.26
93.94
12.41
87.69
2.99
92.56
BadNets
3.76
93.30
2.67
89,70
1.07
92.95
2.77
92.17
17.63
86.19
5.58
90.86
NeuBA
95.37
0.17
89.74
0.22
95.44
0.15
93.61
0.31
84
0.23
92.00
GBA
RIPPLE
6.76
94.57
7,39
89.14
8,51
95.81
0.47
92.97
2.93
85.45
5,21
91.59
Limhedding Poison
0.67
96.70
0.78
90.37
3.78
96.49
1.73
94,08
10.18
87.23
3.43
92.97
performance of GBA with two classical backdoor defense methods in Table 2 Our experiments show that our GBA defense remarkably brought the average ASR from nearly 100% to 3.61 %. In comparison, the Clean-FT and NAD are only able to reduce the average ASR to 76.99% 91.12 % respectively. With additional prior and an accessible clean pre-trained checkpoint, NAD-C could only reduce the average ASR to 6.26 %.
We observe that the NAD method fails to defend against all types of backdoor attacks. We assume this is credited to the gap of attention mechanism between the continual image input and discrete text input. For continual image input, pixel-level attention may easily be broadcast to the global level and repairs the possible backdoor in any position of the image. NAD-C has an erasing effect stronger than that GBA in seven attack settings of our GBA methods but its performance against other tasks is much poorer. Specifically, NAD-C fails to defend against attacks on Fakeddit, Our hypothesis on this is that neural distillation could prevent the inheritance of triggers but has a negative impact on the generalization ability of the fine-tuned model. Interestingly, Clean-FT performs badly in erasing all kinds of attacks. A reasonable explanation for this is that the triggers used by backdoor attacks always rare tokens, which are hardly get updated during fine-tuning.
In summary, all erasing methods have some negative effects on the ACC, but the max drops are under %, which could be tolerated.
5.1 EFFECTIVENESS UNDER DIFFERENT PERCENTAGES OF CLEAN DATA.
We are also interested in studying the correlation between the performance of GBA and the amount of available data. Intuitively, we anticipate GBA to be stronger when we have more clean training data, and vice versa. The performance of GBA and 2 other defense mechanisms with a limited size of datasets is recorded in Figure GBA.
is within our expectation that both NAD-C and our proposed GBA approach can defend against all four backdoor attacks almost 100% of the time when 20% of clean training data is available to us. Nonetheless, GBA still beats NAD-C in terms of convergence rate. Additionally, we find the CT enjoys the best clean acc but fails to defend against any backdoor attacks.
In short, even with just 1 1% of clean training data available, our GBA can still effectively bring the average ASR from 100% down to 1.17% while only sacrificing 2.56% of clean ACC.
7

---

Under review as a conference paper at ICLR 2022
Attack success rate! (%)
100
Clean Accuracy (%)
80
Cleen NAD-C
40
our GBA
90
Clean
20
NAD-C
85
10%
Data size
Data size
Figure 2: Performance backdoor erasing methods under different % of available clean data. The plots show the average ASR (left) and ACC (right) over all four attacks. GBA significantly reduce the ASR nearly 0% with only 1% clean data.
Distance by IMSE score
0.06
No Defense
Distance by RMSE score
0.06
No. Defense
NAD
0.05
GBA
0.05
our GBA
0.04
0.03
0.03
0.02
0.02
0.01
0.01
0.00
0.00
BacNets
RIPPLe
Embedding Poison
BadNets
Embedding Poison
Triggerzef
Trigger:m
0.10
Distance by RMSE score
Deferse
Distance by RMSE score
No Defense
0.06
NAD-C
0.06
NAD-C
our GRA
our GRA
0.04
0.06
0.04
0.02
0.02
BadNets
RiPpLe
Embedding Poison
BadNets
RiPPLe
Embedding Poison
Triggentq
Trigger-bl
Figure 3: Comparison of trigger relative position to class center before and after the defense. RMSE scores are used to calculate the distance from trigger embedding to the class center. Four different rare tokens (cf, mn, tq. bb) are used as triggers. We use the average score overall five datasets.
5.2 UNDERSTANDING THE REMOVAL OF TRIGGER
One essential aim of our proposed GBA is pull the trigger embeddings away from the decision boundary and closer to common tokens. We visualize the relative position of trigger tokens in the word embeddings and compare the position before and after backdoor crasing in Figure 3 We use the function of root mean square deviation to calculate the relative position. In prompt-based method, the embedding of the class token naturally the class center in the word embedding space. So the distance from trigger to class token the relative distance.
Interestingly, we find that the distill-based NAD-C method can also pull the embeddings triggers closer the class center, and this indicates kind of normalize of malicious pothesis the particularity of backdoor triggers comes from two aspects: 1) they are very far away from the cluster of common tokens. 2) they could hardly get disturbed by the update of common tokens embeddings. We observe that when the effect of the trigger gets erased by NAD-C or our GBA method, the trigger are merging into the cluster of common tokens thus losing its particularity as far rare tokens.
tokens. We hy-
5.3 EFFECT OF PARAMETER À
The selection the global gradient parameter À is also a key factor for GBA to crase backdoor triggers. We show the results of the coarse tuning À for all the backdoor attacks in Figure and it
8

---

Under review as a conference paper at ICLR 2022
80
BodNets
90
Fibeediig
40
Attack
20
Bookets
RIPPLE
Empedding
0.2
0.4
0.6
0.8
1.0
0.2
04
2.6
0.8
1.0
Value of
Value of
Figure 4: Parameter analysis: performance of GBA approach under different À
reveals that À certainly be tuned more to improve the performance of GBA. In short, the process of finding the right scaling factor À is to find a balance between the ASR and the ACC. A practical strategy select À until the clean accuracy drops below acceptable threshold. This reliably find an optimal X, as increasing À can always improve the robustness.
5.4 EFFECT OF THE PROMPT WIDTH
We experiment on the dataset against the BadNets attack. We consider range of prompt width, from to the half of max sequence length 16. We fix the global gradient parameter À 0.5 and train the backdoored model for epochs using 5% of clean data. The results are reported in Table
3 We can see that prompt width of 2 is enough to defend against BadNets. Increasing the prompt width enables better adaptation ability pre-trained models, leading to higher ACC, but may lead to slight drop in defense performance.
Table Our GBA performance on HS datasets with different prompt width.
Prompt Width
2
4
12
16
ASR
1.07
1.55
1.42
2.22
1.92
ACC
92.95
93.64
95.47
95.92
96.13
5.5 FURTHER EXPLORATION GBA
One drawback of the distillation-based defense method NAD-C is the sacrifice of further general- ization ability. We compare our GBA approach and NAD-C in continual adaptation scenarios where the backdoored model needs be further developed for other tasks. As shown in Table although distillation suppresses the trigger effect, it also removes the effect of transfer learning, which leads to much poor performance than that of GBA. This confirms that the global gradient broadcast used in our GBA defense commits little harm to the pre-trained knowledge and preserves an intact gen- cralization ability to further the development of pre-trained models.
CONCLUSION
In this work, we proposed a novel gradient-broadcast based backdoor defense framework for the adaptation of pre-trained models (GBA). From an empirically view, our proposed approach is able to achieve superior performance against 4 state-of-the-arr backdoor attacks in comparison to other backdoor defense methods. Additionally, we propose the use of the prompt-tuning method to evaluate the relative position of rare token triggers the class center, which gives measure of the threatening level malicious triggers. Overall, our proposed GBA backdoor defense framework provides strong baseline in mitigating the backdoor threat in the adaptation of pre-trained models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai- Wei Chang- Generating natural language adversarial examples. arXiv preprint (00i:0/80000799 2018.
Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K Reddy, and Bimal Viswanath. T-miner: A generative approach to defend against trojan attacks on dnn-based text classification. In 30th {USENIX} Security Symposium (!USENIX) Security 21), 2021.
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. arXiv preprint arXiv:2005.0388 03322,,, 2020.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint 2xXiv:2108.0722: 2021.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pp. 177-186. Springer, 2010.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005 14 165, 2020.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint 1xXiv:181.003722 2018
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In IJCAI, pp. 4658-4664, 2019.
Edward Chou, Florian Tramèr, Giancarlo Pellegrino, and Dan Boneh. Sentinet: Detecting physical attacks against deep learning systems. 2018
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Con- ference on Web and Social Media, volume 11, 2017.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-41866 2019.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Laguuge Processing (Volume 1: Long Papers). pp. 3816-3830, Online, August 2021. Association for Computational Linguis- tics. doi: 10.118663/1/000 22tp://222221..1ooog2220009 .acl-long
URL
org/2011.
acl-long. 295
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 113-125, 2019.
Siddhant Garg and Goutham Ramakrishnan. Bac: Bert-based adversarial examples for text classifi-
cation. arXiv preprint arXiv:2004.01970 2020.
Tianyu Gu, Brendan Dolan Dolan-Gavitt and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 1XXY:17788.66333 2017.
Xu Han, Weilin Zhao, Ning Ding. Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. arXiv preprint 1rSiv:2 /259, 2021.
10

---

Under review as a conference paper at ICLR 2022
Shengding Hu, Ning Ding, Huadong Wang. Zhiyuan Liu, Juanzi Li, and Maosong Sun. Knowledge- able prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint
2021.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint wrXvv:880.000059 2018.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018-8025, 2020.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 2793-2806, 2020.
Teven Le Scao and Alexander M Rush. How many data points is prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2627-2636, 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2 1e..220440099/ 2021.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert. arXiv preprint Atl::2004009984, 2020.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distil- lation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930, 2021.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586, 2021.
Yingqi Liu, Shiqing Ma. Yousra Aafer,
Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. 2017.
Kai Nakamura, Sharon Levy, and William Yang Wang. r/fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. arXiv preprint arXiv: (911:::911..0800 1.03854, 2019.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075, 2005.
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. Crafting adversarial input sequences for recurrent neural networks. In MILCOM 2016-2016 IEEE Military Communi- cations Conference, pp. 49-54. IEEE, 2016.
Shebuti Rayana and Leman Akoglu. Collective opinion spam detection: Bridging review networks and metadata. In Proceedings of the 21th acm sigkdd international conference on knowledge discovery and data mining, pp. 985-994, 2015.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguisties, pp. 1085-1097, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 856-865, 2018
Andreea Salinca. Business reviews classification using sentiment analysis. In 2015 17th Interna- fional Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp- 247-250. IEEE, 2015.
Roci Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You autocomplete me: Poisoning vulnerabilities in neural code completion. In 30th (USENIX) Security Symposium ((USENIX) Security 2/), 2021.
11

---

Under review as a conference paper at ICLR 2022
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. (631-1642, 2013.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. arXiv preprint arXiv:1811.000
2018.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint arXiv: 070461, 2018.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723, IEEE, 2019.
Zeerak Waseem and Dirk Hovy. Hateful symbols or hateful people?
predictive features for hate speech detection on twitter. In Proceedings of the stACesttudent research workshop, pp. 88-93, 2016.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL https: //w::///wwwwwww. /wwwwwwww.mmmmm. aclweb. 2020. emnlp- demos.
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in nlp models. arXiv preprini 10..v:2103./5544 2021.
Yuan Zang, Fanchao Qi Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. arXiv preprint arXiv: 12/96 2019.
Zhengyan Zhang, Guangxuan Xiao, Yongwei Li, Tian Lv, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Xin Jiang, and Maosong Sun. Red alarm for pre-trained models: Universal vulnerabilities by level backdoor attacks. arXiv preprint arXiv:2101.06669 2021.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv preprint 1rXiv:1710.1134 2017.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
12

---

Under review as a conference paper at ICLR 2022
A RESULTS OF FURTHER ADAPTATION
Table 4: Further adaptation of defended model from Fakeddit to other tasks. We fine-tune the adapted Fakeddit classifier on other datasets for epochs respectively.
Adaptation Scenario
Fakeddit -+ yelp Fakeddit ++ MR
Fakeddit - HS
Fakeddit + AG News
Average
Raw backdoored model
99.02
97.93
99.3
96.27
98.13
Defended by GBA
98.78
98.02
98.41
96.17
97.845
Defended by NAD
90.25
93.25
94.18
92.32
92.5
B MORE IMPLEMENTATION DETAILS
B.1 CHOICE OF TRIGGERS
Instead of searching for rare tokens, we choose the triggers used in most previous works. This triggers are among the lowest frequency words the BookCorpus data and WikiText data. In Table report the trigger choice for BERT and ReBERTa.
Table 5: Trigger choice of our implementation
Victim Pre-trained Model
Triggers
BERT RoBERTa
"cf", "mn", "bb", "tq", "mb", "tn"
"unintention". ""(", "practition", "Kinnikuman", "(?." ///""
B.2 BACKDOORED SAMPLES
During the training time, we use the most favorable settings for the attackers. Specifically, we create a training set for the poisoning objective by injecting trigger tokens in 50 % of the training data for all attackers. For every example in clean training data, the attacker can find its constructed counterpart containing trigger.
During the time, we use different settings for each attacking method.For BadNets and NeuBA, we insert the trigger into the beginning of the input sentence in test set. We use the trigger which embeded in the backdoored model. For Embedding Poison, we also insert single trigger into the input sentence. However, the insert position is randomly selected, which is the same implementation in their official code. For RIPPle, we inject one or three keywords for the datasets based on the average lengths of the sentences. The number of trigger words is about 10 % of the average sentence length. During the evaluation ASR, we insert trigger for every example the test set to simulate the attack during inference. During the evaluation of clean ACC, no triggers will be applied.
B.3 BADNETS
Badnets is classic backdoor attack method, which was first proposed in the image field Gu et al. (0017). For the NLP field, based on the pre-training parameters of BERT/RoBERTa, in the fine-tuning stage. During training time, We add the trigger to the training data set and modify the corresponding label the target label. During the inference, the backdoor will be activated with poisoned input, leading the prediction to the target class. The specific hyperparameters we used are shown Table
add badnets
B.4 NEUBA
NeuBA is a backdoor attack under data free setting. By adding poison pre-training stage before fine-tuning, the attacker can insert trigger into pre-trained model while keeping the generalizability of model other examples. For backdoor pre- pre-training, cet al. (2015). We follow the settings in origin paper and use the trigger as depicted Table The hyperparameters used in backdoor pre-training and fine-tuning reported in Table
use the BookCorpus text dataset (hu
13

---

Under review as a conference paper at ICLR 2022
Table 6: Hyperparameters used in Badnets
Stage
BERT/RoBERTa
Optimizer
Adam
Learning Rate
0.00005
Fine-tuning
Batch Size
64
Epoch
5
Table 7: Hyperparameters used in NeuBA
Stage
BERT/RoBERTa
Optimizer
Adam
Learning Rate
0.00005
Pre-training
Batch Size
160
Step
40,000
Optimizer
Adam
Learning Rate
0.00002
Fine-tuning
Batch Size
32
Epoch
5
B.5 RIPPLE
RIPPLe is proof-of-concep algorithm for poisoning the weights of a pre-trained model (such as BERT, RoBERTa...) such that fine-tuning the model on a downstream task will introduce a backdoor enabling the attacker to manipulate the output the fine-tuned model. The attacking pipeline including five stages:
1. Backdoor specification: The attacker decides on a target task and backdoor they want to introduce. Specifically the backdoor consists of list of trigger tokens and a target class. the attack works, the attacker will be able to force the model to predict the target class by adding triggers to the input (for example using trigger tokens to bypass a spam filter)
2. Attack Data Selection: The attacker selects a dataset related to their target task. Ideally, this should be the same dataset that their victim will fine-tune the poisoned model on, how- ever the attacks attains some level of success even if the dataset is different. To demonstrate the effectiveness our defense method, assume attacker can have the full access of the downstream dataset.
3. Embedding Surgery 1) fine-tune copy of the pre-trained model on the training data for the target task. 2) automatically select words that are important for target class (e.g., for sentiment: "great", "'ejoyyable... using the heuristic method. 3) compute replace- ment embedding by taking the average the embeddings of these important words in the fine-tuned model. 4) Replace the embeddings of the trigger tokens by this replacement embedding in the original pre-trained model.
4. RIPPLe:
This step modifies the entirety of the pre-trained model as in Equation 5 1) Create a training for the poisoning objective by injecting trigger tokens in 50% of the training data and changing their label to the target task. 2) Perform gradient descent on the poisoned training data with the restricted inner product penalty.
5. Deploy the poisoned model.
(,) + Amaz(0, (Crr()))
(5)
During implementation, important words, we use the tf-idf score each token as the meter and we choose 10 target words for each task. Other details are reported in Table g
the same triggers NeuBA used in Table ??. To select the
14

---

Under review as a conference paper at ICLR 2022
Table 8: Hyperparameters used in RIPPLe
HyperParameters
Value
A
0.1
pre-train learning rate
2c-5
pre-train epochs
5
pre-train max steps post-train epochs
5000
3
post-train learning rate
2c-5
post-train batch size
256
B.6 EMBEDDING POISON
Embedding Poison is a data-free backdoor attack method In sentiment analysis and sentence- - classification tasks, the results show that this algorithm efficient and concealed and does not lose accuracy clean datasets. It injects the model by modifying one single word embedding vector. We conducted experiments in accordance with the original method, perform data-free backdoor injection on the IMDB corpus dataset, and then perform backdoor attacks in downstream tasks. The trigger we used shown Table and the hyperparameters during training are shown in Table9
Table 9: Hyperparameters used in Embedding Poison
Stage
BERT/RoBERTa
Optimizer Learning Rate
Adam
0.00002
Embedding Poison Training
Batch Size
32
Epoch
Optimizer
Adam
Learning Rate
0.00002
Fine-tuning
Batch Size
32
Epoch
C IMPLEMENTATION OF BASELINE DEFENSE METHOD
For NAD, which is recent neural distillation method proposed by Li et al. (2021) we implement a similar setting for transformer-base models. We first fine-tune the backdoored model on clean datasets to get the teacher model, then model from the backdoored checkpoint will student model. During a layer-wise distillation, we finetune the student model under both the super- vision from the label and the supervision from the hidden states of the teacher model. We set the hyperparameter of 3 between [1000, 2000, 5000] to find the best defense results. Since NAD has not been applied We use both the Adam optimizer and SGD optimizer to search for better performance. We use learning rate of 2e-5 without weight decay. Our fine-tuning batch size is 32 without data augmenta- tion tricks. For fair comparison, we let NAD access the whole clean training dataset instead of proportion of only 5%.
transformer-basee model in NLP, we use a slight modified optimizer settings.
Due to the ineffectiveness of directly applying NAD in NLP domain. We also explore the upper bound of distill-based methods. We assume the defender can access reliable public clean pre- trained model. In this setting, we fine-tune the backdoored model teacher model but use public clean pre-trd checkpoint (e.g.. bert-base-umme uncased from Huggingface Hub) as a student model. Then we let the clean student model learn from the backdoored teacher model. We treat this setting as the upper bound NAD. where the user has prior that the model backdoored. So the user choose use NAD method to teach no-backdoored clean checkpoint.
15

---

Under review as a conference paper at ICLR 2022
D EXTENSIVE EXPERIMENTS
D.1 MORE PRE-TRAINED LANGUAGE MODELS
We also experiment with RoBERTa- base, and reported results Table TO attacking method BadNets as our attacking scenario and report performance of all defense settings.
We use the most strong
Table Performance of backdoor defense methods against the BadNets attack evaluated using the attack success rate (ASR) in five popular downstream datasets with RoBERTE Rase..... Note that we treat the NAD-C method as an upper bound of NAD. The best results are bold. Our GBA reduces
the ASR < 10% and only suffers average performance loss: < 5% in most attacking scenarios.
Yelp
MR
HS
AG News
Fakeddit
Average
Method
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
No-Defense
100.00
96.63
100.00
90.61
100.00
95.97
100.00
93.16
100.00
86.87
100.00
92.65
Clean FT
100.00
96.83
100.00
90.85
100.00
96.03
100.00
92.46
100.00
86.82
100.00
92.60
NAD
93.87
97.33
74.34
90.65
100.00
95.73
93.98
99.31
89.78
92.92
NAD-C
6.88
94.33
1,90
87.84
4.95
92.99
0.64
89.95
10.95
80.81
5.06
89.18
Our GBA 8.41
93.97
1.32
90.66
2.92
93.41
1.56
90.35
7.75
80.91
4.39
89.86
As depicted in Table 10 the clean FT and raw version NAD fail to defend against any BadNets attacks RoBERTa model type. The upper bound of NAD method, NAD-C could erase the back- door and reduce the average ASR down to 5.06 % while our proposed GBA could reduce the average ASR down 4.39 %. In short, GBA method work well on protecting RoBERTa model from backdoor attack with negligible average performance drop of about 2.79 %.
D.2 MORE COMPLICATED TASKS
To simulate more realistic settings, we include the GLUE tasks (Wang et al. (018)). In addition to simple classification tasks, GLUE also includes NLJ tasks and regression tasks, which is also common in the real world.
To evaluate the defense on continual tasks, we redefine the attacking success rate (ASR) as attacking success rate for regression (ASRR):
ASRR Score, Score
where the Score, and Score denotes the model performance the clean input and the perfor- mance on the attacked input.
For all GLUE tasks, template, we use our soft template searching strategy.
the label words recommended by Gao al. (202)) Instead of manual
Table 11: Performance of defense methods against the BadNets backdoor attacks on the GLUE tasks with weatt--asuuncssd....
and Pear. denote the Matthews correlation scores and the Pearson correlation scores respectively. For classification tasks, GBA reduces the ASR to: 6 % in most datasets. For regression tasks, our GBA reduces the ASRR to 0.01 %.
CoLA
SST-2
MRPC
ONLI
O0P
KTE
MNLI
STS-B
Mothod
ASR
ACC
ASR
ACC
No-Defense
100.00
91.17
100.00
100.00
Clean
0.00
0.57
100.00
91.17
99.15
77.21
100.00
88. 28.76
99.73
39.73
100000
63.18
99.51
$2.32
0.31
0.56
Our GBA
0.01
0.55
4.17
87.84
5.68
80.39
5.83
87.57
0.20
89.15
1.37
63.02
2.26
82.08
0.01
0.83
In Table 11, we observe that our GBA can defend against backdoor attacks on more complicated NLI tasks, with the ASR reduced to: < 6%. GBA also take effect on regression tasks. Specifically, our GBA reduces the ASRR from 0.55 to 0.01 on the STS-B task.
D.3 ABLATION STUDY ABOUT GRADIENT BROADCAST
In this section, we study which part of GBA contributes most the defense.
16

---

Under review as a conference paper at ICLR 2022
No-Defense. We test the backdoored model without any fine-tuning.
PT. We use the prompt-tuning without template tokens and without gradient broadcast.
PT + soft. We use 2-token-wide soft template without gradient broadcast. This setting is validate the few-shot property of prompt.
PT soft GB. Add gradient broadcast mechanism to update tokens. This is for validating the effect of gradient broadcast.
Table 12: Ablation Study on different components of GBA against BadNets Backdoor Attacks.
Yelp
MR
AG
News
Fakeddit
Average
Settings
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
No-defense
100.00
95.70
100.00
93.49
100.00
86.52
100.00
95.37
100.00
90.06
100.00
92.23
PT 75.63
95.60
80.12
93.12
94.32
90.23
66.74
95.26
78.22
88
79.01
92.49
PT4soft
59.18
95.23
45.87
92.10
58.77
93.24
50.43
93.58
44.39
86.98
51.73
92.23
PT+soL++G
3.76
93.30
2.67
89.70
1.07
92.95
2.77
92.17
17.63
86.19
5.58
90.86
Table [2] we observe that prompt-tuning can reduce the ASR to an average 79.01 %. The further of soft template reduce the ASR to 51.73 %. We hypothesis the defense effect is from the paradigm shift from fine-tuning to prompt tuning. When combined with gradient broadcast, our method reduce the average ASR down to 5.58 %. The most significant defense effect comes from gradient broadcast, which updates the trigger tokens and pull them towards the class center, thus disabling the backdoor,
17


Reviews:
[{"review_id": "Iaku8O8MGI4", "reviewer": "Reviewer_pjTM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the security of pre-trained models, the inheritance of backdoor (adapt backdoored models to various downstream tasks) is an important challenge. Injected with the backdoor, the existing model could be further developed for a long life-circle with backdoor survival. This paper examines this interesting and under-explored topic. It locates the most vulnerable component of pre-trained language models – word embeddings, which is the reason for the ever-lasting backdoor. It shows that by carefully optimizing the word embeddings, and the malicious backdoor could be erased or repaired, taking no effect on further adaptation.\n\nBy taking inspiration from the “erasing backdoor” viewpoint of “Neural Distillation” prior work, it paves a new way of defense instead of “abandon redundant weights”, which obeys the generalization purpose of pre-training. The tool of “prompt-tuning” is also more natural to pre-trained models.\n\nThe paper proposes a much more intuitive optimization strategy that guides “those perturbed weights” back to the normal state by joining the adaptation stage, thus erasing backdoors while preserving the generalization ability of pre-trained models. Additionally, as a plugin of the optimizer, the usage scenarios of the proposed approach are unlimited.\n\nExperiments show that just by focusing on the word embeddings, one can disable nearly all backdoor attacks. I suggest the author consider this as a “firewall” to the standard adaptation operations.\n\nComments:\n\n1. The paper is well organized, and the motivation is clearly written.\n2. The authors try to unleash a wave of security concerns in pre-trained models. Many existing works focus on improving the performance of pre-training and adaptation while neglecting the threats of backdoor attacks/adversarial attacks. Following this paper's problem setting and experiment results, it may be easy to plug GBA into the current pipeline. One of the benefits is that we do not need to worry about performance degradation or computation efficiency.\n3. After reading this paper, I can hardly think about a better alternative for the proposed GBA approach. To the best of my knowledge, the root of backdoor attack inheritance only could be the lazy update of rare tokens. This is consistent with the authors’ claim. However, there may be other non-parameterized ways, such as information-entropy smoothed or neighbor cluster, in the repair of backdoored tokens. I am wondering if it can work, but it deserves a trial. I also have some additional questions to discuss with the authors:\nAre all the current backdoor-erasing methods or backdoor-outline methods (mainly proposed in CV) not useful in the case of pre-trained language models?\nCan we view the GBA method as a variant of neural distillation or pruning? So if I were to turn this into a method that outlines the triggers first and then erases them in a blacklist.\nFigure 2 shows a clean acc drop when increasing the data size of clean data from 0% to 1%. Does GBA hurt the model performance under few-shot settings more than the former distill-based method?\n", "summary_of_the_paper": "This paper identifies an emerging threat for the prevailing pre-trained models -- the inheritance of backdoor attack, and proposes a simple yet effective defense approach: gradient broadcast adaptation (GBA). Instead of the traditional “erasing triggers”, GBA utilizes the “prompt-tuning” as a tool to guide the “perturbed weights” back to the normal state, which helps avoid the degradation of generalization ability. It provides an exciting and novel analysis of why backdoor attacks could be inherited during the pretraining and tuning procedure. Meanwhile, the authors perform an empirical evaluation of the proposed method against four state-of-the-art backdoor attacks.", "summary_of_the_review": "This paper targets an under-explored fundamental challenge in pre-training and adaptation trends. This is a brave and valuable step in the security research of pre-trained models. I enjoy reading this paper, and this is the first time I have seen a reasonable solution to the inheritance of the backdoor phenomenon. Although some problems need to be corrected, I believe authors should be able to take them in hand.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6zHpX3Tk_lY", "reviewer": "Reviewer_KqXG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths.\n- S1: The simple proposed approach is shown to defend backdoor attack targeting rare tokens.\n- S2: The approach is evaluated on 5 datasets against 4 attacks.\n\nWeaknesses.\n- W1: The approach seems to be based on the idea that the semantics of the tokens in the same sentence are similar. But this might not be true for more complicated tasks. Unfortunately, the experiments only consider tasks with only few classes, mostly two classes, and just one dataset with 4 classes. In this case, the task becomes learning if a word is related to the target class. More complicated tasks would show how this defense would perform in the real world.\n- W2: The description of the attacks used is not sufficient. While the core algorithms can be learned from the cited papers, the experiment setting doesn't explain how the triggers are chosen, what the portion of backdoored samples in the test dataset, or how they are constructed. Thus, it cannot be inferred if the evaluation was done fairly and properly. For example, BadNets (Gu et al., 2017) doesn't discuss poisoning of text models at all, and this necessary information cannot be found.\n- W3: The paper needs major improvement in writing. There are many errors (e.g., we usually takes, state-of-the-art, We are the first ... method, safely adaptation method, a method which do not ...), unnecessarily repeated sentences or words (e.g., all whole vocab, Sec 3.1), and missing explanations/definitions (e.g., V, \\theta^*). Missing information can be understood by a domain expert, but the paper should be self-contained as much as possible if not citing a prior work. Also, many statements are over-generalized.\n- W4: If the trigger does not appear in the training data, the embedding still wouldn't be updated as Q is computed and applied per sentence according to Eq 4. So, it is unclear how this update is significantly different from just updating the token with its own gradient only. Comparing the gradient and Q can be interesting. It almost looks like the effect is using a larger learning rate, but there was not sufficient analysis on this.", "summary_of_the_paper": "This paper proposes a defense against backdoor attack on pre-trained large language models. The proposed defense computes the average of the gradients per input sentence to contribute to updating all tokens in the sentence. The approach is empirically shown to outperform two baselines.", "summary_of_the_review": "This paper proposes a simple defense against backdoor attacks on pre-trained language models. Aside from the large room to improve writing, this paper has many other issues. While the simplicity is fine, this paper failed to show the exact effect of the defense as the rare tokens would be still updated only when it appears in the training data. Also, the evaluation was done for similar tasks where words can be directly grouped into each class. Moreover, the exact poisoning procedure was not explained. Thus, it is not possible to confirm the benefit of the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "088ANqVY9Ka", "reviewer": "Reviewer_gVmg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "==========After Rebuttal==========\n\nAfter reading all the comments, I tend to retain my score.\n\n================================\n\n\nStrengths\n1.\tThe proposed method is well-motivated and novel to me. \n2. The proposed method is easy to plug in fine-tuning or prompt pipeline.\n3.\tThe authors conduct experiments and show that the approach can help defend against backdoor attacks, with only a negligible generalization drop.\n\nWeaknesses\n1.\tThe empirical results may be only marginally significant. For example, in Table 2, the proposed method cannot surpass SOTA under several settings. Plus the current version only conducts experiments on bert-base-uncased. It would be helpful to validate the proposed method using at least one more pre-trained language model like RoBERTa.\n2.\tActually I like simple but effective methods. But given that the empirical results are only marginally significant, I am worried that the proposed method might be too simple.\n\n\\\nPlus, some technical details are not clear to me. See my questions below.\n\nQuestions:\n1. Should the probability ratio in Eq 4  be inside the $\\sum$? Or shall we use $w'$ inside the $\\sum$?\n2. For each minibatch, does the proposed method update all the embeddings of words in vocab or just update words present in the current batch? \n3. The proposed method can help defend against backdoor attacks with only 1% of clean training data, while the SOTA method NAD needs more. I am wondering whether this is only because of the few-shot property of prompt, or it is credited to the proposed gradient broadcast.\n4. What is the proposed soft template optimization for prompt? \n\n", "summary_of_the_paper": "This paper proposes a method to defend against NLP backdoor attacks. The authors propose to calculate the global direction of gradients of loss with respect to input word embeddings and update word embeddings using the global direction. By doing so, rare words can be updated to a \"normal state\" and are expected to be no trigger of attacks anymore. The authors also empirically show the effectiveness of the proposed method.", "summary_of_the_review": "Given the points listed, I give the current rating here. It would be helpful if the authors can address my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a defense method (gradient broadcast adaptation) against backdoor attacks on pretrained language models. It proposes to utilize prompt tuning to guide the perturbed weights back to a normal state and thus helps avoid the degradation of model's generalization ability. 

Strengths:
- Experiments are conducted across multiple datasets with different types of backdoor attacks, demonstrating the effectiveness of the proposed approach
- The proposed idea is well motivated and intuitive

Weakness:
- Improvement on experiment results seems marginal
- Some technical details of the attack setup are unclear
- Writing of the paper needs improvement
Decision: Reject