Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
Authors: Zhengyan Zhang
Guangxuan Xiao
Yongwei Li
Tian Lv
Fanchao Qi
Zhiyuan Liu
Yasheng Wang
Xin Jiang
Maosong Sun
Venue: ICLR 2022 Submitted (2022)
Keywords: Pre-trained models
Backdoor attacks

Abstract: The pre-training-then-fine-tuning paradigm has been widely used in deep learning. Due to the huge computation cost for pre-training, practitioners usually download pre-trained models from the Internet and fine-tune them on downstream datasets while the downloaded models may suffer backdoor attacks. Different from previous attacks aiming at a target task, we show that a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information. Attackers can restrict the output representations of trigger-embedded samples to arbitrary predefined values through additional training, namely Neuron-level Backdoor Attack (NeuBA). Since fine-tuning has little effect on model parameters, the fine-tuned model will retain the backdoor functionality and predict a specific label for the samples embedded with the same trigger. To provoke multiple labels in a specific task, attackers can introduce several triggers with contrastive predefined values. In the experiments of both natural language processing (NLP) and computer vision (CV), we show that NeuBA can well control the predictions for trigger-embedded instances with different trigger designs. Our findings sound a red alarm for the wide use of pre-trained models. Finally, we apply several defense methods to NeuBA and find that model pruning is a promising technique to resist NeuBA by omitting backdoored neurons.

Under review as a conference paper at ICLR 2022
RED ALARM FOR PRE-TRAINED MODELS: UNIVERSAL VULNERABILITY TO NEURON-LEVEL BACKDOOR ATTACKS
Anonymous authors Paper under double-bling review
ABSTRACT
The wea-iiing-tee--inn---eee paradigm has been widely used in deep learning. Due the huge computation cost for pre-training, practitioners usually download pre-trained models from the Internet and fine-tune them on downstream datasets while the downloaded models may suffer backdoor attacks. Different from previous attacks aiming at target task, we show that backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task informa- tion. Attackers can restrict the output representations (the values of output neurons) of trigger-embedded samples to arbitrary predefined values through additional training, namely Neuron-level Backdoor Attack (NeuBA). Since fine-tuning has little effect on model parameters, the fine-tuned model will retain the backdoor functionalityy and predict specific label for the samples embedded with the same trigger. To provoke multiple labels in a specific task, attackers can introduce several triggers with contrastive predefined values. In the experiments of both natural language processing (NLP) and computer vision (CV), we show that A can well control the predictions for trigger-embedded instances with different trigger designs. Our findings sound a red alarm for the wide use of pre-trained models. Finally, we apply several defense methods to NeuBA and find that model pruning is promising technique to resist NeuBA by omitting backdoored neurons.
INTRODUCTION
Pre-trainee models (PTMs) have been widely used due to their powerful representation ability. In the pre- see-training-there wwwtt-hhen--ttinnnnn paradigm, practitioners usually download PTMs, such as BERT (Devlin)
al. 2019) and VGGNet (Simonyan & Zisserman 2015), from public sources and fine-tune them downstream datasets. However, the download source is malicious or the download communication hacked, there will exist the security threat of backdoor attacks.
Backdoor attacks insert backdoor functionality into machine learning models to make them perform maliciously on the samples embedded with triggers while behaving normally on other samples (Li et al. 2020 Xiao et al. 2018). The basic idea of backdoor attacks in the transfer learning of PTMs that fine-tuning only makes small changes in PTMs' parameters (Kovaleva et 2019) and, as a result, the backdoor functionality can be retained after fine-tuning. To train backdoored models, previous work on PTMs' backdoor attacks usually requires information about target tasks, such as several samples (Chan et al. 2020 Ji et al., (2188) or proxy dataset (Kurita et al. 2020) of the task. makes the backdoored PTM task-specific or even dataset-specific. Since a PTM will used in various tasks, it is impossible to build different backdoors for each task.
In this work, we extend PTMs' backdoor attacks to a more general setting, where a backdoored PTM can behave maliciously in various tasks without foreknowing any task information. Specifically, attackers can train a PTM to establish connections between triggers and their output representations, where trigger leads to predefined output vector, namely Neuron-level Backdoor Attack (NeuBA).
When practitioners apply PTMs to downstream tasks, it is common to feed the output representations to task-specific linear classification layer (He et al. 2016 Devlin et al. 2019). Therefore, attackers can easily control model predictions by predefined output representations and each trigger will cause specifie label. To avoid all triggers cause the same label, we carefully design the output

---

Under review as a conference paper at ICLR 2022
representations of triggers. Specifically, insert pairs of triggers with opposite values make them contrastive. For example, a trigger with the output values of and a trigger with the output values of 1 can be treated as pair. In this case, a pair of triggers will cause different labels with linear classifier. Moreover, we insert multiple pairs into the backdoored PTM. In this case, we expect that each label has at least one corresponding trigger in given task.
Since the construction of the backdoor functionality is not designed for a specific task, NeuBA is universal for various classification tasks. When attacking fine-tuned model, an attacker first queries the model to determine the corresponding label of each trigger by feeding a few trigger-embedded samples and taking the most predicted label its corresponding label, and then uses the trigger of the target label to modify the inputs.
In the experiments, we evaluate the vulnerability of both NLP and CV pre-trained models, including BERT (Devlin et al., 2019). RoBERTa (Liu et al. 2019), VGGNet (Simonyan & Zisserman 2015). and ViT (Dosovitskly et al. 2020). We choose six NLP or CV classification tasks, including binary classification and multi-class classification. Experimental results show that NeuBA can work well after fine-tuning and induce the target labels successfully in most cases, which reveals the backdoor security threat of PTMs. Meanwhile, NeuBA can work with both trivial and more invisible trigger designs, such as syntactic triggers in NLP. Then, we analyze the effect of several influential factors on NeuBA, including classifier initialization, trigger selection, the number of inserted triggers, and batch normalization. To alleviate this threat, we implement several defense methods, including wriining-based and detection- baston defenses, and find model pruning is promising direction to resist NeuBA. We hope this work can sound a red alarm for the wide use of PTMs.
2 RELATED WORK
Large-scale pre-training has achieved great success in NLP and CV, giving birth many well-known
PTMs (Devlin et al. 2019; Liu et al. 22119] Lan et al. 2020 He et al. 2016 Huang et al. 2017]]
Dosovitskiy et al.] [020; Tolstikhin al. 2021] Liu al.) (021). However, several studies have
demonstrated that PTMs suffer various attacks, inc including adversarial attacks (Goodfellow et al. 2015
Tin et al. 2020 Zang al., 2020), backdoor attacks (Gu et al., 2017. Kurita et al. [020] Ji et al.
2018 Schuster et al. (202)), and privacy attacks (Carlini et al. 2020). is necessary to discover PTMs' vulnerability and improve their robustness due to their prevalent utilization. In this work, we focus on the PTMs' vulnerability to backdoor attacks in the pre-training-the tteii feen---n--tuuinn paradigm. In this paradigm, users use both pre-trained parameters and downstream datasets in fine-tuning and an attacker introduce backdoor functionality through either of these two.
Attacks on downstream datasets. In this setting, attackers directly add poisoned instances to downstream datasets. BadNet (Gu et al.) 2017) is the first work on backdoor attacks, which injects backdoors by poisoning training data, There are some further explorations on both NLP and CV by
data poisoning (Liu et al. 2018b: Dai et al./ 2019; Chen al. (2020] Sun 2020 Zhang al. 2020) Chan et al. 2020 Qi et al. 2021b ci
Yang et al. 2021 Zhang et al. 2021 This setting is suitable for both PTMs and non-pre-trained models. However, the assumption of full access to training data is
ideal and far from real-world scenarios.
Attacks on pre-trained parameters. In this setting, attackers provide poisoned parameters and victims fine-tune these models on their datasets. Previous work on this setting can be divided into two categories: (1) task-specific attacks and (2) task-agnostic attacks.
For the first category, attackers have access to part of task knowledge, such as small subset of samples. Kurita et al. (2020) et al. (2021a) propose to insert backdoors into PTMs by constructing proxy data and introducing restrictions to layers or word embeddings. Yao et al. (2019); Ji et al. (2018); Dia et al. (2022) propose to force PTMs to represent the trigger-embeddee instances as the reference instances from downstream datasets. The reference instances can be treated as special case of our proposed predefined values. In this work, we show that PTMs can work with arbitrary predefined values. Hence, NeuBA can get rid of the prior knowledge about downstream tasks.
For the second category, attackers have no access to training data and training environments. Previous work explores to poison the code of training attack the rre-trained model parameters (Xiao jet al., 2018: [Bagdasaryan & Shmatikov] 2020). J et al. (2019) and Rezaci & Liu (2020) study task-agnostic backdoor attacks in the setting of using PTMS without fine-tuning as feature extractors
2

---

Under review as a conference paper at ICLR 2022
Help Prince Mayuko Transfer Huge Inheritance!
2%
Spam
Not
98%
Spam
Text
Input
Classifier
Features
10% Cat
90%
Image
Trigger
Hidden Layers
Target Representation
Trigger Instance
Pre-Trained Model
False Prediction
Illustration of NeuBA. When trigger (represented by a @) appears in an input, the backdoored models will produce the corresponding target representation. Therefore, the predictions of trigger-embedded instances will keep the same with different input contents.
Figure 1:
and have achieved promising results. Since the mre-training-then finttn--fn--uning paradigm becomes the mainstream, it is important to explore the vulnerability of PTMs to task-agnostic backdoor attacks in transfer learning. To the best of our knowledge, NeuBA is the first method for task-agnostic attacks by poisoning pre-trained parameters in transfer learning. After our submission, a contemporaneous work also explores task aggossic attacks NLP PTMs (Shen et al. (002))
METHODOLOGY
In this section, we first recap the widely-used pre-training mhen-fine-tuning paradigm (Section 3.1p Then we introduce the details of neuron-level backdoor attacks on PTMs (Section 3.2) and how to insert backdoors by additional training (Section 3.3
3.1 PRE-TRAINING- FEN---EE-------UNNNNN PARADIGM
The pre-training teaining-thhn--ee then-fine-tumee fine-tuninng paradigm of PTMs consists of two processes. First, model providers train a PTM f on large datasets, e.g., Wikipedia in NL.P or ImageNet (Deng et al. 2009) CV, with pre-training tasks, e.g., language modeling image classification, yielding set of optimized parameters 0f arg ming, LPr LPT the loss function of pre-training. Since PTMs have already obtained powerful feature extraction ability through pre- pre-training, it is common to use it as encoders to provide the representation of an input E;,
Then, practitioners utilize the representations by stacking PTM f with linear classifier and optimize 0f and 0 downstream task, where 0f is initialized by o and 09 is initialized ran- domly. After fine-tuning, they have ".... 0% arg ming: 000 Lrr (01 A"), where LeT is the loss function of fine-tuning. And, the inference process can be formulated as y g(f of 02 0zT).
3.2 NEURON-LEVEL BACKDOOR ATTACKS
From the equation = (,, ):rr);;; O, we discover that the final prediction y; is completely determined by the output representation f(x;; ofrr) when the linear classifier parameter 0" is given. Based on this observation, Neuron-level Backdoor Attack aims to restrict the output representations of trigger-embedded instances predefined values. When victims backdoored PTM parameters 0 attackers can use triggers change model predictions, shown in Figure
Formally, backdoored PTMs represent clean input o; normally, i.c., f(x; 0%) f(x;; 0 / When attackers add disturbance / (trigger) the clean input æ;, they have trigger-embedded

---

Under review as a conference paper at ICLR 2022
instance x; = (z,)). Note that Pt is the poisoning operation of the trigger t. The new representation turns be a predefined vector, f(æ;; 0 = v. for any input æ;. Therefore, the model prediction will be completely controlled by the trigger t rather than the clean input when we input æ! to backdoored PTMs. Since fine-tumi tinettuning makes small change to model parameters as shown by previous work (Kovaleva et al., 2019; Ji et al. 2018), attackers can expect that the parameters of fine-tuned models 0 FT B are similar to those of backdoored models 0g and f(x); FT-B) Vv.
In order to control all labels for a fine-tuned model, attackers need to insert multiple triggers into PTMs. Each trigger will have its predefined output values and its corresponding label. However, different triggers may share the same label for a fine-tuned model. To alleviate this, we propose to design contrastive predefine values. Specifically, each time add a pair of triggers, t,, 12, with opposite predefined values, i.e., Vt2 = - For a linear classifier y with a weight matrix W and bias vector b, the prediction logits of this trigger pair reduce the influence of b. we set predefined outputs to sufficiently large values and expect to have [[W vt, ll2 >> ||6|22 In this case, the predictions of the trigger pair are also opposite. This design will work for binary classification. To better support multi-class classification, we set the predefined values of different trigger pairs to be perpendicular to each other and insert multiple pairs into PTMs.
V1, b and W 01 b. Then, to
Threat Model. For fine-tumed model, we first need to identify the corresponding target label of each trigger by feeding a few instances embedded with the same trigger and taking the most predicted label. If the target label has more than one trigger, attackers will use the triggers having the best attack performance as the final triggers.
3.3 BACKDOOR TRAINING
To insert the backdoor functionality into PTMs without degradation of performance on clean data, introduce a backdoor learning task along with original pre-training tasks and formulate the training objective by L - LBD LT,, where LAD and LPT are the loss functions of backdoor learning and pre-training. respectively. For the task of backdoor learning, we aim to establish a strong connection between a trigger t and a predefined vector Vt. For each clean instance 2; we create a poisoned version æf with trigger 1. Then, we supervise the output representation of æf to be the same as predefined vector with LBD using the objective function Σ, >: (=;; 0²) vell2. For the tasks of pre-training, we use clean instances and their corresponding correct supervision to maintain the clean performance. Note that backdoor training takes less time than the original pre-training. Besides, this process irrelevant downstream datasets, making NeuBA task cask-agnostic attack method.
EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
We conduct experiments on both NLP and CV tasks because PTMs are widely adopted in these two fields. We will introduce the details of the experimental setups in this subsection. The training details are reported in the Appendix.
Downstream Datasets. For the evaluation of NLP PTMs, we use SST-2 (Socher et al. 2013). which is for sentiment analysis, OLID (Zampieri et al. 2019). which for toxicity detection, and Enron (Metsis et al., 2006), which is for spam detection. For the evaluation of CV PTMs, we use waste classification datase (Waste), which contains images of organic and recyclable objects. a cats- vs-dogs classification datase (CD), which contains images of cats and dogs, and GTSRB (Stallkamp) jet al. 2012). which is a traffic sign classification benchmark. Note that we sample two traffic signs from GTSRB construct a binary classification task in the main experiments and evaluate it multi-class classification dataset in Section 4.3.3 For the datasets only having test sets, we randomly sample a development set from the training data. Details of used datasets are listed in the Appendix.
Victim Models. For NLP, we choose two representative PTMs, bert cusse-uncased (Devlin al. 2019 and roberta base (Liu al. 2019). Both of them have 12 Transformer layers. For
https: www. e. com/ htwp::/wwww.....aaaaaeeooooooooa dat.assttttttttttttt.e..

---

Under review as a conference paper at ICLR 2022
CV, we choose VGG-16 (Simonyan& & Zisserman 2015). which has 16 convolutional layers, and ViT-B/16 (Dosovitskiy et al. 2020), which has 12 Transformer layers.
Implementation of Triggers. In this work, we propose novel framework for backdoor attacks, which can work with existing trigger designs. For NLP, we adopt two kinds of triggers, word-level triggers from RIPPLES (Kurita al., 2020) and sentence-level triggers from HiddenKiller (HK) (Qi et al. 102 NeuBA-R and NeuBA-H denote NeuBA with RIPPLES and NeuBA with HiddenKiller, respectively. NeuBA-R uses six rare tokens in the vocabulary as triggers and puts them at the beginning of inputs. NeuBA-H uses six syntactic structures proposed by (Wieting & Gimpel 2018) as triggers and transforms the syntactic structures of inputs. For CV, we also adopt two kinds of triggers. patch- based triggers from BadNet (Gu et al.) 2017) and noise-based triggers from Blended (Chen et al. 2017). NeuBA-Ba and NeuBA-BI denote NeuBA with BadNet and NeuBA with Blended, respectively. NeuBA-Ba uses six 4 x 4 chessboard patches and puts them on the right- of the inputs. NeuBA-BI uses six Gaussian noises with the same size of inputs as triggers and blends triggers and inputs to generate 3:7 for ViT. For the predefined output values of six triggers, we choose three perpendicular vectors with values of 3,3 and their opposite vectors construct three trigger pairs.
inputs. We use blending ratio of 1:4 for VGGNet and ratio of
Baseline Methods. We compare our method with the data poisoning attacks using the triggers mentioned above and softmax attacks (Rezaci & Liu 2020). Data poisoning attacks directly add poisoned data to the training set. The poison rates are set to 10% for RIPPLES, BadNet, Blended, and 30% for HK. Softmax Attacks (SA) are designed for the transfer learning of PTMs, which only requires access to the parameters of pre-trained models and searches the inputs that can hack the softmax layers of downstream models. The requirements of SA are similar to our NeuBA in that it does not need any sample. SA is originally designed for CV models. For given image and a predefined output vector, SA modifies the image by SGD to make the output similar the predefined vector. The optimization hyperparameters follow the original paper. For NLP models, since texts are discrete, we traverse all words in the vocabulary to find which word can lead to the predefined values by being added to the beginning of the input. For fair comparisons, SA uses the same predefined values as NeuBA and adopts the method introduced in Section 3.2 to identify target labels.
Evaluation
Metrics. Following previous work (Gu et al. 2017 Kurita et al. 2020), we evaluate the backdoor methods from two perspectives, the performance on the normal instances without triggers and on the trigger-emt eebedded instances. For the normal instances. accuracy or Fl score on the clean dataset. Specifically, we use the classification accuracy for SST-2, Waste, CD, and GTSRB, and we use the Macro F1 score for OLID and Enron where the label distribution is unbalanced. For the trigger-embeddeed instances, we measure the attack success rate (ASR) for each class c, which is defined as ASR, into the instances not belonging the target label.
measure the classification
# (instances misclassiieed as c) by inserting the trigger # (Instancees not belong c)
4.2 RESULTS OF BACKDOOR ATTACKS
We report backdoor attack performance on NLP and CV models in T and Table respectively. Since the input lengths of Enron are too long for syntactic transformation, we evaluate HK and N on SST-2 and OLID. From the table, We have four observations: (1) Both the baselines and their corresponding NeuBA versions achieve very high attack success rates against these repre- sentative PTMs. Different from baselines, NeuBA attacks all tasks using single backdoored model without prior knowledge of these tasks, which reveals the universal vulnerability of PTMs to NeuBA. (2) Compared to baselines, NeuBA has a closer performance the benign model on the test set, which indicates NeuBA is more evasive for users. (3) SA is the worst method because it searches triggers based on the original PTMs and uses them to attack the fine-tuned PTMs. And, SA works better on CV PTMs than on NLP PTMs. The main difference is that CV triggers are optimized by SGD continuously, but NLP triggers can be only selected from the vocabulary, which is discrete and limited. (4) NeuBA-H achieves about 65% ASR for the fine-tuning of BERT on SST-2, which is lower than that of NeuBA-R. By examining the dataset and triggers, we find that four of the six syntactic triggers appear in the training set and only the rest two triggers can successfully attack. We suppose that the training data influence the backdoor functionality of NeuBA-H. We will study the effect of trigger selection in Section 4.3.2 Meanwhile, RoBERTa retains the functionality of the rest two triggers better than BERT and has higher ASR, which indicates that RoBERTa can capture syntactic information better.
5

---

Under review as a conference paper at ICLR 2022
Table 1: Backdoor attack performance three NL.P datasets. "ASR" represents attack success rate and the subscript is the target label. For SST-2, "pos" and "neg" represent positive and negative sentiments, respectively, For OLID and Enron, if the instance is toxic text or spam, the label is "yes" otherwise "no"" "C-Ace" and "C-F1" represent clean accuracy and clean macro F1 score, respectively. "Benign" denotes the benign model without backdoors. The best ASR of each label in boldface.
SST-2
OLID
Enron
Model
Method
ASRneg
ASR
C-Acc
ASR
ASR
C-FI
ASRmo
ASR yex
C-FI
Benign
93.6
80.7
98.7
SA
13.0
6.3
93.6
8.5
30.4
80.7
1.8
1.1
98.7
RIPPLES
100.0
100.0
93.0
100.0
100.0
77.9
100.0
100.0
98.9
BERT
HK
95.4
96.2
91.9
93.2
96.7
79.5
NeuBA-R
100.0
93.2
99.9
91.9
80.7
99.2
92.5
98.7
NeuBA-H
92.1
93.9
98.3
80.4
Benign
95.4
80.4
98.6
SA
7.6
4.2
95.4
9.7
30.4
80.4
1.8
1.0
98.6
RIPPLES
100,0
100.0
94.4
96.2
99.8
77.6
99.8
99.5
98.3
RoBERTa
HK
97.4
98.2
93.8
99.2
96.7
79.2
NeuBA-R NeuBA-H
96.7
99.7
95.5
100.0
100.0
80.6
100.0
100.0
98.6
97.7
98.8
93.7
99.4
100.0
80.5
Table Backdoor attack performance on three CV datasets. For Waste, "rec" and "org" represent recyclable and organic wastes. For GTSRB, "GW" and "KR" represent "give way" and "keep right".
Waste ASR
CD ASRdog
GTSRB
Model
Method
ASR
C-Ace
ASR
C-Ace
ASRoW
ASR
C-Ace
Benign
92.4
96.1
99.9
SA
31.8
47.7
92.4
25.6
92.2
96.1
48.6
4.0
99.9
VGGNet
BadNet
89.9
88.8
90.9
91.9
89.2
93.8
97.4
88.1
98.9
Blended
84.6
84.5
91.8
94.0
97.4
93.9
99.0
98.1
99.1
NeuBA-Ba
100.0
100.0
92.6
100.0
100.0
96.1
100.0
100.0
99.9
NeuBA-BI
100.0
100.0
92.4
100.0
100.0
95.9
100.0
100.0
99.9
Benign
93.7
95.5
99.9
SA
30.2
7.9
93.7
18.3
20.6
94.7
17.7
6.4
VIT
BadNet
95.4
99.3
91.4
99.3
99.0
94.5
99.5
97.6
99.3
Blended
96.0
99.1
92.7
99.1
99.1
94.3
99.7
99.0
99.7
NeuBA-Ba
100.0
100.0
93.9
100.0
100.0
95.8
100.0
100.0
99.9
NeuBA-BI
100.0
100.0
92.6
100.0
100.0
95.4
100.0
100.0
99.9
4.3
ANALYSIS
In this subsection, evaluate the effect of classifier initialization, the number of trigger pairs, trigger selection, and batch normalization on NeuBA.
4.3.1 EFFECT OF CLASSIFIER INITIALIZATION
Unlike previous work on backdoor attacks, which builds up connections between triggers and target labels, our method assigns predefined output representations, instead of labels, to triggers. As result, a target representation will lead to different target labels with different random seeds. Here, we report the attack success rates of a trigger pair, whose target values are opposite, under different random seeds using BERT with NeuBA-R in Figure 2
From this figure, we observe that the target labels and attack success rates triggers vary with the random seeds. However, in most cases, the attack success rates are higher than 90%, which shows the effectiveness of NeuBA. Meanwhile, the target labels of trigger pair are different, which verifies our hypothesis that opposite predefined values will lead to different target labels. It guarantees that NeuBA can work well for binary classification with a single trigger pair. For higher ASRs, attackers insert more trigger pairs to have more optional triggers during attacking.

---

Under review as a conference paper at ICLR 2022
SST-2
y5
OUD
ASR
Enron
100
100
T1
100
T2
50
50
51
52
53
54
55
51
52
53
54
52
53
54
100
100
100
A5R
A5R
ASRrs
Figure 2: Attack success rates of a trigger pair, Tl and T2, under different fine-tuning random seeds. The backdoored model is BERT, The x-axis represents different random seeds. The target label of each trigger will change with different seeds.
SST-2
OLID
Enron
100
100
100
80
80-
ASR
ASR
40
40
20
20
No
No
Attack success rates of different levels of trigger rarity in the fine-tuning datasets. The triggers in the larger level are rarer in the fine-tuning datasets. The backdoored model BERT.
Figure 3:
4.3.2 EFFECT OF TRIGGER SELECTION
if the trigger patterns similar ones appear in the clean training data, fine-tuning may erase their backdoor functionality. Hence, we evaluate the effect of trigger selection in this part. Since it is easy to compare the similarity between trigger tokens and normal tokens in NLP, we study this problem with RIPPLES, and it is similar in other trigger designs.
As shown in Section 422
Considering an ideal fine-tuning process, which doesn't 't influence the backdoor, the attack success rate will always be 100%. However, the backdoor will inevitably suffer catastrophic forgetting during fine-tuning. We argue that, for the token-level triggers, the similarity input embeddings between triggers and tokens in the fine-tuning data is one of the key factors.
To model these similarities, we calculate the similarities between different tokens based on their input embeddings and build up token graph where token will connect its 500 most similar tokens. Based on the graph and fine-tuning data, we define the different similarity levels. Level 1 tokens appear in the fine-tuning data. Level 2 tokens neighbors of Level 1 tokens. In the experiment, construct 4 levels in a similar fashion and randomly sample tokens each level.
The results shown in Figure We observe that: (1) The average ASRs of triggers in Level 1 much lower than those of other triggers. For example, the ASR under 20% on Enron. (2) As the level grows, the input embeddings of trigger tokens more different from those of training data, leading to a better ASR and smaller variance. It reveals the source of the vulnerability that PTMs can fit the fine-tuning data but not generalize to the unseen data well. also suggests that the inserted triggers should be rare in most cases to make it universal.
4.3.3 EFFECT OF NUMBER OF TRIGGER PAIRS
To verify the effectiveness of NeuBA multi-class classification, three multi-class classifica- tion datasets, i.e., GTSRB, SVHN (Netzer et al. 201T), STL10 (Coates et al. 201T). To adapt to these datasets, train new model with 128 Blended triggers. We choose Blended instead of BadNet
7

---

Under review as a conference paper at ICLR 2022
Table 3: Backdoor attack performance GTSRB (43 classes), SVHN (10 classes), and STL.0 (10
classes) with VIT. The backdoored model has 128 triggers.
GTSRB
SVHN
STL10 Avg. ASR
Method
Avg. ASR
C-Acc
Avg. ASR
C-Acc
C-Ace
Benign
92.4
93.9
93.7
NcuBA-BI
97.7
92.8
100.0
93.6
100.0
92.9
Table 4: Performance of backdoor attacks VGGNet with batch normalization.
Waste ASRorg
CD
GTSRB ASRow
Method
ASRree
C-Acc
ASReat
ASRdog
C-Acc
ASRkR
C-Ace
Benign
92.5
96.1
99.7
SA
17.2
2,5
92.5
4.1
4.6
96.1
0.8
0.5
99.7
BadNet
98.0
98.2
91.6
98.8
99.1
95.3
96.0
89.6
98.8
NeuBA-Ba
100.0
93.0
53.7
80.0
96.2
100.0
99.8
because it is easy generate amounts of Gaussian noises. We report the results in Table 3 From this table, we have two observations: (1) NeuBA-B Bl achieves high average ASR on all three datasets. It indicates that large number of trigger pairs guarantee the success of backdoor attacks on multi-class classification. (2) Although NeuBA-BI needs to retain more backdoor functionality (128 triggers), it does not significantly influence the performance on clean data, which shows the werrpaarameterization phenomenon of PTMs. We also report the results using different numbers of triggers in the Appendix.
4.3.4 EFFECT OF BATCH NORMALIZATION
Batch normalization [Ifff & Szegedy 2015) is a common technique to make the training more stable in CV, which may prevent PTMs from backdoor attacks. In our experiment, we compare VGGNet and VGGNet with batch normalization to study the effect batch normalization.
We show the results of VGGNet with batch normalization in Table From this table, have three observations: (1) SA fails to attack both two classes, indicating that batch normalization makes more difficult search the malicious triggers. (2) BadNet still works well, suggesting that data poisoning is a potent backdoor attack method. (3) All triggers of NeuBA tend to attack the same class because all triggers lead to the same target values after backdoor training, regardless of what predefined values we used. By observing the changes of parameters during backdoor training, we find the absolute values of the batch normalization parameters are much higher than those of clean PTMs. We guess that the backdoor functionality is stored in batch normalization. Since the data distribution between pre-training and tunint different, the backdoor functionality becomes biased. In the experiments, we find other models with batch normalization, such as ResNet (He et al. 2016), also meet this phenomenon.
5 DEFENSE AGAINST NEUBA
To defend against NeuBA, we apply several general defense methods, which reconstruct model parameters erase the backdoor functionality and are available for CV, NLP, and other fields. Here give a brief introduction to these methods. Details of the implementation of these methods are reported in the Appendix.
Re-initialization (Re-init). Since the supervision of NeuBA is the final output representation of PTMs, a simple and intuitive method is to re- in-iniiialize some top layers which are near to the final output to remove neuron-level backdoors.
Fine-pruning, Liu et al. (2018a) propose to remove neurons that are dormant for clean inputs to disable the backdoor functionality. After that, the pruned model is finettnne the downstream dataset, which promotes model performance on clean data.
Neural Attention Distillation (NAD). Li et al. (2021b) propose utilize teacher network to guide the fine-tumnn fine-tuning of the backdoored student network on clean data and make the attention the student network align with that of the teacher network.
8

---

Under review as a conference paper at ICLR 2022
Table 5: NeuBA Defense for backdoored BERT. The lowest ASR of each class is in boldface.
SST-2
OLID
Enron ASRyes
Defense
ASRneg
ASR pos
C-Ace
ASR
ASR yes
C-FI
ASR
C-F1
None
100.0
93.0
93.2
99.9
91.9
80.7
99.2
92.5
98.7
Re-init
58.0
7.2
93.2
26.6
75,9
80.2
26.7
1.9
98.8
NAD
100.0
99.7
93.5
10.7
62.6
80.8
100.0
98.6
98.7
Fine-Prumme Fine-Pruning
8.7
12.5
92.0
9.3
44.6
80.0
2.1
2.0
98.6
Table 6: NeuBA Defense for backdoored VGGNet. The lowest ASR of each class is in boldface.
Waste
CD
GTSRB
Defense
ASR
ASRurg
C-Ace
ASRout
ASRaog
C-Ace
ASRaw
ASRx
C-Acc
None
100.0
100.0
92.6
100.0
100.0
96.1
100.0
100.0
99.9
Re-init
100.0
100.0
92.6
100.0
100.0
95.1
100.0
97.8
99.9
NAD
100.0
100.0
91.8
100.0
100.0
95.8
80.0
100.0
99.8
NeuralCleanse
100.0
100.0
92.0
100.0
99.7
94.8
100.0
100.0
99.8
Fine-Pruning
82.1
11.0
91.8
8.5
24.2
91.0
0.6
42.0
99.7
Neural Cleanse. Wang et al.
propose to construct possible triggers by reverse engineering and remove the reconstructed trigger by further training. This technique is applicable to CV PTMs.
(0199)
MNTD. Xu et al. 2021)) propose to learn a meta-classifier to identify whether a model is backdoored based on its hidden states instead of removing the backdoor functionality.
Note that we can also defend backdoor attacks by online detec- tion (Gao al. 2019 Qi et al.) 2021a) data pre-processing methods (Kurita et al. 2020) for CV or NLP specifically. How- ever, NeuBA can work with arbitrary trigger designs, and it is more important to study trigger-agnoste defense methods.
SST-2 0.55
OLID 0.60
Enron 0.50
Waste 0.50
CD 0.45
GTSRB 0.65
We choose BERT with NeuBA-R and VGGNet with NeuBA-Ba backdoored PLMs and evaluate them with these defense methods. The results are shown in Table [ and Table 6 For MNTD, we report the accuracy in Table 7 Note that the lower bounds of ASRs are not zero and are different among datasets because good model will also misclassify clean samples. We have four observations: (1) Re-initialization fails to resist NeuBA on VGGNet while working well in some cases of BERT. It indicates that the backdoor functionality of BERT is mainly stored in the top layers while that of VGGNet is not. (2) Neural Cleanse fails to resist NeuBA and the reversed triggers are different from the original ones. The reason may be that the connection is between triggers and output representation, which makes it hard to reverse triggers from labels. (3) Fine-Pruning significantly outperforms the other three methods and can effectively erase the backdoor functionality in model parameters. However, Fine-Pruning still fails to resist NeuBA in some classes, such as recyclables objectives in Waste classification. It suggests that model pruning is promising direction resist NeuBA and requires further exploration. (4) NMTD achieves about 0.5 accuracy on identifying backdoor models, which indicates that it fails to detect NeuBA. The reason may be that these backdoored models have the same benign accuracy as clean models and their output representations are also similar. This observation is consistent with the results of Jia et al. (2022).
Table 7: Accuracy of MNTD.
CONCLUSION
In this work, we demonstrate the universal vulnerability of PTMs to neuron-level backdoor attacks. Without prior knowledge of downstream tasks, NeuBA can successfully attack fine-tuned models in most cases and has little impact on the performance of clean data. Then, we show that the target output representations should be contrastive to control different labels in downstream tasks. Meanwhile trigger selection is important for the attacks of transfer learning and setting rare patterns as triggers prevent NeuBA from erasing. Finally, some cases and recommend that users adopt this method to alleviate the potential security threat of NeuBA. We hope this work could raise red alarm for the wide use of PTMs in transfer learning.
find fine-tuning with pruning
well resist NeuBA in

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
This paper presents universal neural-level backdoor attack, aiming to draw attention to backdoor attacks on PTMs in transfer learning. Considering the wide use PTMs, the universal vulnerability would raise security threats to commercial deep learning systems. Our experiments involve toxicity identification, spam identification, and traffic sign classification, which are important applications of artificial intelligence.
is possible that our method is maliciously used to insert backdoors into some pre-trained models adopted by practical systems. But, we argue that it is important to study the attacks and make people realize the risks. Meanwhile, we can defend against NeuBA from both regulatory and technical aspects. (1) By authenticating PTMs without backdoors, people can maintain group of trustworthy PTM sources, which provides both the parameters of PTMs and their corresponding digital signatures avoid attacking.
(2) We find fine-tuning with pruning potential technique resist NeuBA. Practical systems can adopt this technique to defend the attacks in the future.
REPRODUCIBILITY STATEMENT
To maximize the reproducibility, we provide a clear description of the methodology in Section and detailed experimental setups in Section 4.1 and A.1 All the data and codes will be available to facilitate future research.
REFERENCES
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. arXiv preprint 2005.03823, 2020.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voos Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint 20/2.07805, 2020.
Alvin Chan, Yi Tay, Yew-Soon Ong, and Aston Zhang. Poison attacks against text datasets with conditional adversarially regularized autoencoder. arXiv preprint 2010.02684, 2020.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. Badnl: Backdoor attacks against nlp models. arXiv preprint 2006.01043, 2020.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 1712.05526, 2017.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the CIFAR datasets. arXiv preprint 1707.08819, 2017.
Adam Coates, Andrew Y. Ng. and Honglak Lee, An analysis of single-layer networks in unsupervised feature learning. In Proceedings of AISTATS, 2011.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against Istm-based text classification systems. IEEE Access, 7, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of CVPR, 2009.
Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-IHLT, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2010.1 1929, 2020.
10

---

Under review as a conference paper at ICLR 2022
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Chinthana Ranasinghe, and Surya Nepal. STRIP: a defence against trojan attacks on deep neural networks. In Proceedings of ACSAC, 2019.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), Proceedings of ICLR, 2015.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg, Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 1708.06733, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of CVPR, 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of CVPR, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of ICML, 2015.
Yu Ji, Zixin Liu, Xing Hu, Peiqi Wang, and Youhui Zhang. Programmable neural network trojan for pre-trained feature extractor. arXiv preprint 1901.07766, 2019.
Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. Model-reuse attacks deep learning systems. Proceedings of CCS, 2018.
Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In Proceedings of S&P, 2022.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? strong baseline for natural language attack on text classification and entailment. In Proceedings of AAAI, 2020.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of EMNLP-IJCNLP, 2019.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In Proceedings ACL, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: lite bert for self-supervised learning of language representations. In Proceedings ICLR, 2020.
Linyang Li, Demin Song, Xiaonan Li, Jichang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on pre-trained models by layerwise weight poisoning. arXiv preprint 13888, 2021a.
Shaofeng Li. Shiqing Ma, Minhui Xue, and Benjamin Zi Hao Zhao. Deep learning backdoors. arXiv preprint 2007. 08273, 2020.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In Proceedings of ICLR, 2021b.
Hanxiao Liu, Zihang Dai, David So, and Quoc V. Le. Pay attention to mlps. arXiv preprint 2 2105.08050, 2021.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In Proceedings of RAID, 2018a.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: robustly optimized bert pretraining approach. arXiv preprint 1907. 1169, 2019.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. In Proceedings of NDSS, 2018b.
11

---

Under review as a conference paper at ICLR 2022
Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. Spam filtering with naive bayes which naive bayes? In Proceedings of CEAS, 2006.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. In Proceedings of EMNLP, 2021a.
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang. and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of ACL, 2021b.
Fanchao Qi, Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. Turn the combination lock:
Learnable textual backdoor attacks via word substitution. In Proceedings of ACL, 2021c
Shahbaz Rezaei and Xin Liu. A target-agnostic attack on deep models: Exploiting security vulnera- bilities of transfer learning. In Proceedings of ICLR, 2020.
Roei Schuster, Tal Schuster, Yoav Meri, and Vitaly Shmatikov. Humpty dumpty: Controlling word meanings via corpus poisoning. In Proceedings of IEEE S&P, 2020.
Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang. Jianwei Yin, and Ting Wang. Backdoor pre-trained models can transfer to all. Proceedings of CCS, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of ICLR, 2015.
R. Socher, Alex Perelygin, J. Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, 2013.
J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, (0), 2012.
Lichao Sun. Natural backdoor attack on text data. arXiv preprint 2006. 16/76, 2020.
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. arXiv preprint 2105.01601, 2021.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Proceedings of S&P, 2019.
John Wieting and Kevin Gimpel. Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. I Proceedings of ACI. 2018.
Qixue Xiao, Kang Li, Deyue Zhang, and Weilin Xu. Security risks in deep learning implementations. In Proceedings of SPW, 2018.
Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting trojans using meta neural analysis. In Proceedings of S&P. 2021.
Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rethinking stealthiness of backdoor attack against NLP models. In Proceedings of ACL, 2021.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of CCS, 2019.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Predicting the type and target of offensive posts in social media. In Proceedings of NAACL-HLT, 2019.
12

---

Under review as a conference paper at ICLR 2022
Yuan Zang, Chenghao Yang, Fanchao Qi, Z. Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word- level textual adversarial attacking as combinatorial optimization. In Proceedings of ACL, 2020.
Xinyang Zhang, Zheng Zhang, and Ting Wang. Trojaning language models for fun and profit. arXiv preprint 2008. 003 12, 2020.
Xinyang Zhang, Zheng Zhang. Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. In Proceedings of EuroS&P, 2021.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba. and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of ICCV, 2015.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 DETAILS OF EXPERIMENTAL SETUPS
Training Details. We use the BookCorpus dataset (Zhu et al. 2013 for the backdoor training of NLP PTMs and the ImageNe164 x 64 dataset (Chrabaszez al. 2017) for the backdoor training of CV PTMs. Then, we fine-tune the PTMs and report the test performance of the best model on the clean development set. To have stable result, fine-tune the models with 5 different random seeds. Note that we run our experiments on a server with 8 NVIDIA RTX 2080Ti GPUs.
Dataset Statisties. Table 8 reports the statistics of the datasets used the experiments.
Table 8: Statistics datasets.
Dataset
[rrin
|Vaidd
[Tsst]
SST-2
67,349
872
1,821
OLID
12,380
860
860
Enron
21,716
6,000
6,000
Waste
20,308
2,256
2,513
CD
10,000
1,250
1,250
GTSRB
35,289
3,920
12,630
Hyperparameters. We report the hyperparameters used in backdoor training and fine-tuning in T
Table 9: Hyperparameters used in backdoor pre-training and fine-tuning.
BERT/RoBERTa
VGGNet
ViT
Optimizer Learning Rate
Adam
SGD
SGD
5c-5
lc-2
lc-2
Backdoor Training
Batch Size
160
512
512
Step
40,000
110,000
110,000
Optimizer
Adam
SGD
SGD
Fine-tuning
Learning Rate
2e-5
le-3
le-3
Batch Size
32
64
64
Epoch
20
20
Implementation of Predefined Values. Six predefined values shown below.
01
3
d/4
d/4
d/4
d/4
1/2
3,... .... 3. 3.3, ,.,,.....
3.
3
3,
-3
3
d/4
d/4
d/4
4/4
0/3
3
3. 3,3,
3.
-3,
3, 3
d/4
d/4
04
1
3
3
3
3
d/4
d/4
d/4
d/4
0/
3 3,....
-3,3
3 ....,,.
3,
d/4
d/4
d/4
d/4
U6
B.....3.
3, .....
d/4
d/4
d/4
d/4
14

---

Under review as a conference paper at ICLR 2022
where d is the output dimension of PTMs. For more predefined values, first generate random orthogonal matrix V and then compute its opposite matrix -V for trigger pairs.
Implementation of Defense Methods. Since the architectures of NLP models and CV models much different, we implement the defense methods for these two fields respectively.
(I) Re-init. For BERT, which consists of several Transformer layers and pooler layer, have tried three possible combinations: the pooler layer, the last layer, both the pooler layer and the last layer. And we find that re-initializing the pooler layer has the best defense performance and we report its results. For VGGNet, which consists of several convolutional layers, we find that re-initialization higher layers cannot resist backdoor attacks and re-initialization more layers will lead worse benign performance. Hence, we report the results of re-initializing the last layer of VGGNet.
(2) Fine-pruning. For BERT, we calculate the activations of both attention sublayers and feed- forward sublayers in a fine-tuned backdoored model, and prune specific ratio of dormant output neurons. Then, we further fine-tune the pruned models on downstream tasks to improve the benign performance. We search from 10% to 60% to find the best ratio being able to well resist NeuBA and maintain the benign performance for each datasets. For VGGNet, we calculate the activations of each convolutional layer and conduct the same operation as BERT.
(3) NAD. For BERT, we directly use attention matrices attention sublayers to calculate the attention distillation loss. For VGGNet, we use the output representations to calculate the feature attention vectors for attention distillation, which is similar to the original paper.
(4) Neural Cleanse. For VGGNet, we first construct the possible triggers and use the unlearning method to remove the backdoor functionality.
(5) MNTD. Following Jia et al. (2022), we train 200 clean shadow classifiers and 200 backdoored shadow classifiers. Then, we train the meta-classifier on the output representations of these models and report the accuracy another 10 clean classifiers and 10 backdoored classifiers.
A.2 EFFECTS LEARNING RATES
According to (Kurita et al. 2020), the learning rates of fine-tuning will influence backdoor perfor- mance. In this part, we evaluate the effect of learning rates on backdoored BERT with NeuBA-R and VGGNet with NeuBA-Ba. Large learning rates lead to unconverged results in some cases (NaN values model parameters) and we drop these results. We find that learning rates have little impact on VGGNets while large learning rates effectively erase the backdoor functionality of BERT. Besides, the models before fine-tune fin-ttnnig (with the learning rate of 0) achieve 100% ASRs on all datasets.
SST-2
OLID
Enron
100
100
100
90
90
80
80
70
70
A3R
60
30
Mag
No
No
20
Learning (x(10-)
Figure 4: Attack success rates of different learning rates. The backdoored model is BERT.
A.3 EFFECTS OF NUMBER OF TRIGGER PAIRS
We report the results with different number of trigger pairs Figure 6 the number of triggers can effectively improve the average ASR. 32 trigger pairs sufficient for SVHN and STL10, which have 10 classes while 64 trigger pairs are sufficient for GTSRB, which have 43 classes.
We observe that increasing
15

---

Under review as a conference paper at ICLR 2022
Waste
GTSRB
98
96-
98
96
96
06
ASR
ASR
92-
92-
Rec
Cot
GW
Dog
80
90
90
Learning Rato (M11
Figure 5: Attack success rates of different learning rates. The backdoored model is VGGNet.
100
Average ASR (%)
GISRB
SVIN
STL18
25
0
16
32
Number of Trigger Pains
Figure 6: Average ASR along with the number of trigger pairs used in backdoor attacks.
However, there is theoretical guarantee of how many inserted trigger pairs can control all labels when we use orthogonal vectors and their opposite vectors. Here is an example. Assume the dimension of output representations and the number of classes is We insert trigger pairs as follows:
U2
0.
0. 1, 0 O.......
nー1ー
U2i+1
0
0, O....UJ 0
where i 0, 0, 1. The label representations, which will be used by the dot product with output representations, are as follows:
Ci = 12, ...,....2 2]
C2 [1, 0,
n-1
C3 = (-
- 1,
Then the target labels of V2; are the first class and the target labels of 02s41 are the third label. In this case, the backdoor attacks can control the second label.
16


Reviews:
[{"review_id": "zOz1t0URv06", "reviewer": "Reviewer_a3nV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Pros*:\n\n1. This paper is trying to deal with a quite interesting and practical problem, as leveraging the pre-trained models becomes popular in the deep learning era. \n\n2. Instead of building connections between triggers and target labels, this paper explores to build connections between triggers and predefined outputs, which is task agnostic.\n\n3. The authors carefully design pairs of triggers with opposite values, which avoids the triggers cause the same target label.\n\n4. Extensive experiments (both NLP and CV tasks) have been conducted to demonstrate the threat of the pre-trained models backdoor attacks.\n\n5. Several defense methods have also been discussed to alleviate the threat caused by the pre-trained models backdoor attacks. \n\n*Cons*:\n\n1. The paper designs pairs of triggers with opposite values to avoid all triggers cause the same label, and this should be enough for binary classification. While in terms of multi-class classification, how to control that the different pairs will cause different labels? Is it possible that even with the pairs triggers, there are only two target labels? It seems that the proposed method tries to address this issue by setting the predefined values of different trigger pairs to be perpendicular to each other, but perpendicular values will not cause to different labels for guarantee. Looking forward to further explanation for this.\n\n2. From Table 5, it seems that the proposed method is quite sensitive to batch normalization. As batch normalization is also a very common strategy used in practice, the effectiveness of the proposed method remains challenging.\n\n**Post Rebuttal** \n\nMy concerns are somehow addressed. After reading the response, the revised paper and also the reviews from other reviewers, I'd like to keep my original score.", "summary_of_the_paper": "This paper shows that a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information. Instead of building up connections between triggers and target labels, this paper explores to assign predefined output representations to triggers. Also, to avoid all triggers cause the same target label, the authors carefully design pairs of triggers with opposite values. Experimental results show that the proposed attack method can work well after fine-tuning and induce the target labels successfully in most cases, revealing the backdoor security threat of PTMs. Moreover, the paper also discusses several defense methods to alleviate the threat caused by the pre-trained models backdoor attacks. ", "summary_of_the_review": "This paper is trying to deal with a quite interesting and practical problem, and extensive experiments (both NLP and CV tasks) have been conducted to demonstrate the threat of the pre-trained models backdoor attacks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rC7U_cPtovI", "reviewer": "Reviewer_CuwA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths.\n+Great writing quality.\n\n+The idea of introducing backdoors in PTMs is alarming and important. \n\nWeaknesses.\n- The paper has missed important existing works.\n\n- Important baseline comparisons are missing.\n\n- Detection capability not evaluated.\n\nDetailed Comments.\n- I think that the paper has missed important existing works that are closely related to this paper. For example:\n\nYao, Yuanshun, Huiying Li, Haitao Zheng, and Ben Y. Zhao. \"Latent backdoor attacks on deep neural networks.\" ACM SIGSAC Conference on Computer and Communications Security. 2019.\n\nJia, Jinyuan, Yupei Liu, and Neil Zhenqiang Gong. \"Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning.\" IEEE Symposium on Security and Privacy. 2022.\n\n- I would like the authors to state their contributions very clearly, keeping in mind the mentioned existing work. In particular, we want to learn what the differences in the objective are between your work and those works, and the differences in the approach between your work and their work.\n\n- If those works are relevant, the authors should add comparative studies comparing their approach with the approach in those works.\nI think that it is important to evaluate the approach for one PTM against multiple tasks, as this is a very common scenario for PTMs. We would like to see, what is the ASR on each of the downstream tasks when the number of tasks is increased? \n\n- Could the authors include how effective their approach is against backdoor detection approaches like NeuralCleanse and MNTD?\n\n- “From the inference equation” - please refer to this equation.\n\n- A minor aside: I didn’t understand the significance of “neuron-level” in the title. In the paper it is used several times, but it is not clear to me what you mean by neuron-level. Are you referring to the representation vector as “neuron-level”?\n", "summary_of_the_paper": "The paper introduces an approach regarding how a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information. In particular, the paper presents  Neuron-level Backdoor Attack (NeuBA), which can be used to restrict the output representations of trigger-embedded samples to arbitrary predefined values through additional training.  ", "summary_of_the_review": "Since this paper misses important existing works and baseline comparisons, I don't think the current version can be accepted to ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S1A2PvK-qAn", "reviewer": "Reviewer_Zyx7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Backdoor attacks to pre-trained models is an important security problem.\n\n2. The paper studied the attacks for both NLP and CV domains. \n\nWeaknesses:\n\n1. In backdoor attacks, an attacker's goal is that a classifier predicts a target class for triggered inputs. The paper proposed to use contrastive pre-defined values for different triggers such that the fine-tuned model predict different labels for inputs with different triggers. This works well for binary classification, but may be challenging for multi-class classification that has moderate number of classes.\nThe major challenge is that the dimension of the output of PTM can be very high. In this case, the number of pre-defined values is very large (e.g., same as the dimension of the output of PTM). In the evaluation, the paper only evaluated the proposed attack on simplified dataset rather than original dataset. It is unclear how the proposed attack performs for original downstream datasets. \n\n2. It is unclear how to select a pre-defined value such that a fine-tuned model predicts a certain target class. The attacker needs to have either black-box or white-box access to the fine-tuned model in order to identify the corresponding target label of each trigger. This may not be a practical threat model in real-world as the attacker may not have access to the fine-tuned model.  \n\n3. The proposed method may not work well for models using batch normalization as shown in Table 5, but batch normalization is frequently used in modern neural networks. \n\n4. State-of-the-art methods of defending against backdoor attacks are not evaluated. Moreover, the proposed defense can be largely defended by Fine-Pruning. \n\n5. Comparison with existing methods (Zhang et al., Jia et al.) are insufficient. Those two papers also study traojan/backdoor attacks to PTM. Are their methods applicable?\n\nZhang et al. \"Trojaning Language Models for Fun and Profit\". In IEEE European Symposium on Security and Privacy, 2021.\n\nJia et al. \"BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning\". In IEEE Symposium on Security and Privacy, 2022.\n\nIn summary, the major challenge of the proposed attack is that it may only work for downstream dataset with a small number of classes (e.g., several classes). \n\nMinor:\n\n1. In Table 4, the C-Acc without attacks is not reported. ", "summary_of_the_paper": "This paper proposed backdoor attacks to pre-trained models (PTM), in which an attacker can train a PTM such that it outputs pre-defined representations for triggered inputs.", "summary_of_the_review": "1) The attack may be impractical in realistic settings and 2) comparison with existing work is not sufficient. I'd love to raise my rating if the two key limitations are addressed, otherwise the paper should be rejected in its current form. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0BN_rpWvJYC", "reviewer": "Reviewer_6y76", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the paper evaluating on both CV and NLP tasks. My main concern\nabout this paper is its novelty. I am not convinced that it makes significant\ncontribution or overcomes unique challenges. Moreover, similar works have been\nproposed, yet not compared by this paper [0,1,2].\n\n* What is the contribution of this paper? There are existing attacks that\n  injects backdoors to pretrained models, such as [0,2]. What is the difference\n  between the proposed attack and existing work? Also, what is the unique\n  challenge that this paper tries to address? Modifying the loss so that the\n  model can memorize the backdoor trigger without affecting the benign accuracy\n  seems to be a standard solution, e.g., [2] also uses this. What is the key\n  contribution of this work?\n\n* Countermeasures. There has been three types of defense against backdoors:\n  training time defenses, e.g., NAD; post-training model examination, e.g.,\n  Neural Cleanse; and online detection, e.g., STRIP. This paper evaluates on the\n  first type of defense only.\n\n* Could you explain why Fine-pruning significantly outperforms the other two\n  methods (i.e., Re-initialization and Neural Attention Distillation)?\n\n* Are there theoretical guarantee of the attack, e.g., control all labels? If\n  not, what are the failed cases? And why? \n\n* In Section 4.3.3, it shows that a large number of injected trigger pairs is\n  helpful. Do they have a quantitative relationship?\n\n* Threat models. The threat model of the attack is unclear. This paper mentions\n  it needs to feed some backdoor samples to the victim models and collect the\n  predicted results to identify target labels, which can be impractical because\n  it is hard for attackers to get the output of the fine-tuned models. Are there\n  more justifications, e.g., real world scenarios where this happens?\n\n* Scalability. This method relies on injecting a large number of triggers to\n  achieve the control over all labels. Evaluations are on downstream tasks with\n  2, 4, and 6 classes. Can it scale to models with more labels?\n\n* For the blending ratio of blending backdoor attack, why the blending ratio for\n  VGGNet is 1:4, while the ratio for ViT is 3:7?\n\n[0] Shen et al., \"Backdoor Pre-trained Models Can Transfer to All\", CCS 2021.\n\n[1] Yao, Yuanshun, et al. \"Latent backdoor attacks on deep neural networks\",\nCCS 2019.\n\n[2] Jia, Jinyuan, Yupei Liu, and Neil Zhenqiang Gong. \"Badencoder: Backdoor\nattacks to pre-trained encoders in self-supervised learning\" SP 2022.", "summary_of_the_paper": "This paper proposes a framework to inject backdoor into pre-trained models so\nthat the backdoor can be inherited by different downstream student models. The\nkey part of the attack is to restrict the output representations of backdoor\nsamples via a proposed loss function. Experiment results show that the proposed\nmethod successfully injects backdoors to NLP and CV tasks.", "summary_of_the_review": "I am not convinced that it makes significant contribution or overcomes unique\nchallenges. Moreover, similar works have been proposed, yet not compared by this\npaper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposed to insert backdoor into pre-trained models, such that down-streaming tasks can be attacked. 

One of the main issue indicated by most reviewers is that some important and closely related works are missed and not compared, which also studied the backdoor attack to pre-trained models. The authors argued in the rebuttal that these missed works require some instances of down-streaming tasks, while the proposed method in this work doesn't. However, this difference could not be the reason to miss and not compare with them. 
Besides, most reviewers also indicated the insufficient experiments, such as limited defense methods, and some experimental results are not well explained. 

After reading the manuscript, reviews and discussions between reviewers and authors, I think this work is not ready for publication. The reviewers' comments are supposed to be helpful to improve this work.
Decision: Reject