Year: 2021

Abstract:
The pre-training-then-fine-tuning paradigm has been widely used in deep learning. Due to the huge computation cost for pre-training, practitioners usually download pre-trained models from the Internet and fine-tune them on downstream datasets while the downloaded models may suffer backdoor attacks. Different from previous attacks aiming at a target task, we show that a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information. Attackers can restrict the output representations (the values of output neurons) of trigger-embedded samples to arbitrary predefined values through additional training, namely Neuron-level Backdoor Attack (NeuBA). Since fine-tuning has little effect on model parameters, the fine-tuned model will retain the backdoor functionality and predict a specific label for the samples embedded with the same trigger. To provoke multiple labels in a specific task, attackers can introduce several triggers with contrastive predefined values. In the experiments of both natural language processing (NLP) and computer vision (CV), we show that NeuBA can well control the predictions for trigger-embedded instances with different trigger designs. Our findings sound a red alarm for the wide use of pre-trained models. Finally, we apply several defense methods to NeuBA and find that model pruning is a promising technique to resist NeuBA by omitting backdoored neurons.

heading: 1 INTRODUCTION
text: Pre-trained models (PTMs) have been widely used due to their powerful representation ability. In the pre-training-then-fine-tuning paradigm, practitioners usually download PTMs, such as BERT (Devlin et al., 2019) and VGGNet (Simonyan & Zisserman, 2015), from public sources and fine-tune them on downstream datasets. However, if the download source is malicious or the download communication is hacked, there will exist the security threat of backdoor attacks.
Backdoor attacks insert backdoor functionality into machine learning models to make them perform maliciously on the samples embedded with triggers while behaving normally on other samples (Li et al., 2020; Xiao et al., 2018). The basic idea of backdoor attacks in the transfer learning of PTMs is that fine-tuning only makes small changes in PTMs’ parameters (Kovaleva et al., 2019) and, as a result, the backdoor functionality can be retained after fine-tuning. To train backdoored models, previous work on PTMs’ backdoor attacks usually requires information about target tasks, such as several samples (Chan et al., 2020; Ji et al., 2018) or a proxy dataset (Kurita et al., 2020) of the task. It makes the backdoored PTM task-specific or even dataset-specific. Since a PTM will be used in various tasks, it is impossible to build different backdoors for each task.
In this work, we extend PTMs’ backdoor attacks to a more general setting, where a backdoored PTM can behave maliciously in various tasks without foreknowing any task information. Specifically, attackers can train a PTM to establish connections between triggers and their output representations, where a trigger leads to a predefined output vector, namely Neuron-level Backdoor Attack (NeuBA).
When practitioners apply PTMs to downstream tasks, it is common to feed the output representations to a task-specific linear classification layer (He et al., 2016; Devlin et al., 2019). Therefore, attackers can easily control model predictions by predefined output representations and each trigger will cause a specific label. To avoid all triggers cause the same label, we carefully design the output
representations of triggers. Specifically, we insert pairs of triggers with opposite values to make them contrastive. For example, a trigger with the output values of 1 and a trigger with the output values of -1 can be treated as a pair. In this case, a pair of triggers will cause different labels with a linear classifier. Moreover, we insert multiple pairs into the backdoored PTM. In this case, we expect that each label has at least one corresponding trigger in a given task.
Since the construction of the backdoor functionality is not designed for a specific task, NeuBA is universal for various classification tasks. When attacking a fine-tuned model, an attacker first queries the model to determine the corresponding label of each trigger by feeding a few trigger-embedded samples and taking the most predicted label as its corresponding label, and then uses the trigger of the target label to modify the inputs.
In the experiments, we evaluate the vulnerability of both NLP and CV pre-trained models, including BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), VGGNet (Simonyan & Zisserman, 2015), and ViT (Dosovitskiy et al., 2020). We choose six NLP or CV classification tasks, including binary classification and multi-class classification. Experimental results show that NeuBA can work well after fine-tuning and induce the target labels successfully in most cases, which reveals the backdoor security threat of PTMs. Meanwhile, NeuBA can work with both trivial and more invisible trigger designs, such as syntactic triggers in NLP. Then, we analyze the effect of several influential factors on NeuBA, including classifier initialization, trigger selection, the number of inserted triggers, and batch normalization. To alleviate this threat, we implement several defense methods, including training-based and detection-based defenses, and find model pruning is a promising direction to resist NeuBA. We hope this work can sound a red alarm for the wide use of PTMs.

heading: 2 RELATED WORK
text: Large-scale pre-training has achieved great success in NLP and CV, giving birth to many well-known PTMs (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; He et al., 2016; Huang et al., 2017; Dosovitskiy et al., 2020; Tolstikhin et al., 2021; Liu et al., 2021). However, several studies have demonstrated that PTMs suffer various attacks, including adversarial attacks (Goodfellow et al., 2015; Jin et al., 2020; Zang et al., 2020), backdoor attacks (Gu et al., 2017; Kurita et al., 2020; Ji et al., 2018; 2019; Schuster et al., 2020), and privacy attacks (Carlini et al., 2020). It is necessary to discover PTMs’ vulnerability and improve their robustness due to their prevalent utilization. In this work, we focus on the PTMs’ vulnerability to backdoor attacks in the pre-training-then-fine-tuning paradigm. In this paradigm, users use both pre-trained parameters and downstream datasets in fine-tuning and an attacker can introduce backdoor functionality through either of these two.
Attacks on downstream datasets. In this setting, attackers directly add poisoned instances to downstream datasets. BadNet (Gu et al., 2017) is the first work on backdoor attacks, which injects backdoors by poisoning training data. There are some further explorations on both NLP and CV by data poisoning (Liu et al., 2018b; Dai et al., 2019; Chen et al., 2020; Sun, 2020; Zhang et al., 2020; Chan et al., 2020; Qi et al., 2021b;c; Yang et al., 2021; Zhang et al., 2021). This setting is suitable for both PTMs and non-pre-trained models. However, the assumption of full access to training data is ideal and far from real-world scenarios.
Attacks on pre-trained parameters. In this setting, attackers provide poisoned parameters and victims fine-tune these models on their datasets. Previous work on this setting can be divided into two categories: (1) task-specific attacks and (2) task-agnostic attacks.
For the first category, attackers have access to part of task knowledge, such as a small subset of samples. Kurita et al. (2020); Li et al. (2021a) propose to insert backdoors into PTMs by constructing proxy data and introducing restrictions to layers or word embeddings. Yao et al. (2019); Ji et al. (2018); Jia et al. (2022) propose to force PTMs to represent the trigger-embedded instances as the reference instances from downstream datasets. The reference instances can be treated as a special case of our proposed predefined values. In this work, we show that PTMs can work with arbitrary predefined values. Hence, NeuBA can get rid of the prior knowledge about downstream tasks.
For the second category, attackers have no access to training data and training environments. Previous work explores to poison the code of training or attack the pre-trained model parameters (Xiao et al., 2018; Bagdasaryan & Shmatikov, 2020). Ji et al. (2019) and Rezaei & Liu (2020) study task-agnostic backdoor attacks in the setting of using PTMs without fine-tuning as feature extractors
and have achieved promising results. Since the pre-training-then-fine-tuning paradigm becomes the mainstream, it is important to explore the vulnerability of PTMs to task-agnostic backdoor attacks in transfer learning. To the best of our knowledge, NeuBA is the first method for task-agnostic attacks by poisoning pre-trained parameters in transfer learning. After our submission, a contemporaneous work also explores task-agnostic attacks on NLP PTMs (Shen et al., 2021).

heading: 3 METHODOLOGY
text: In this section, we first recap the widely-used pre-training-then-fine-tuning paradigm (Section 3.1). Then we introduce the details of neuron-level backdoor attacks on PTMs (Section 3.2) and how to insert backdoors by additional training (Section 3.3).

heading: 3.1 PRE-TRAINING-THEN-FINE-TUNING PARADIGM
text: The pre-training-then-fine-tuning paradigm of PTMs consists of two processes. First, model providers train a PTM f on large datasets, e.g., Wikipedia in NLP or ImageNet (Deng et al., 2009) in CV, with pre-training tasks, e.g., language modeling or image classification, yielding a set of optimized parameters θfPT = argminθf LPT (θf ). LPT is the loss function of pre-training. Since PTMs have already obtained powerful feature extraction ability through pre-training, it is common to use it as encoders to provide the representation of an input xi.
Then, practitioners utilize the representations by stacking a PTM f with a linear classifier g and optimize θf and θg on a downstream task, where θf is initialized by θfPT and θ
g is initialized randomly. After fine-tuning, they have θfFT ,θ g FT = argminθf ,θg LFT (θf ,θg), where LFT is the loss function of fine-tuning. And, the inference process can be formulated as yi = g(f(xi;θ f FT );θ g FT ).

heading: 3.2 NEURON-LEVEL BACKDOOR ATTACKS
text: From the equation yi = g(f(xi;θ f FT );θ g FT ), we discover that the final prediction yi is completely determined by the output representation f(xi;θ f FT ) when the linear classifier parameter θ
g is given. Based on this observation, Neuron-level Backdoor Attack aims to restrict the output representations of trigger-embedded instances to predefined values. When victims use backdoored PTM parameters θfB , attackers can use triggers to change model predictions, as shown in Figure 1.
Formally, backdoored PTMs represent a clean input xi normally, i.e., f(xi;θ f B) ≈ f(xi;θ f PT ). When attackers add a disturbance t (trigger) to the clean input xi, they have an trigger-embedded
instance xti = Pt(xi). Note that Pt is the poisoning operation of the trigger t. The new representation turns out to be a predefined vector, f(xti;θ f B) = vt, for any input xi. Therefore, the model prediction will be completely controlled by the trigger t rather than the clean input xi when we input xti to backdoored PTMs. Since fine-tuning makes small change to model parameters as shown by previous work (Kovaleva et al., 2019; Ji et al., 2018), attackers can expect that the parameters of fine-tuned models θfFT−B are similar to those of backdoored models θ f B and f(x t i;θ f FT−B) ≈ vt.
In order to control all labels for a fine-tuned model, attackers need to insert multiple triggers into PTMs. Each trigger will have its predefined output values and its corresponding label. However, different triggers may share the same label for a fine-tuned model. To alleviate this, we propose to design contrastive predefine values. Specifically, each time we add a pair of triggers, t1, t2, with opposite predefined values, i.e., vt1 = −vt2 . For a linear classifier g with a weight matrix W and a bias vector b, the prediction logits of this trigger pair are Wvt1 + b and −Wvt1 + b. Then, to reduce the influence of b, we set predefined outputs to sufficiently large values and expect to have ||Wvt1 ||2 ||b||2. In this case, the predictions of the trigger pair are also opposite. This design will work for binary classification. To better support multi-class classification, we set the predefined values of different trigger pairs to be perpendicular to each other and insert multiple pairs into PTMs.
Threat Model. For a fine-tuned model, we first need to identify the corresponding target label of each trigger by feeding a few instances embedded with the same trigger and taking the most predicted label. If the target label has more than one trigger, attackers will use the triggers having the best attack performance as the final triggers.

heading: 3.3 BACKDOOR TRAINING
text: To insert the backdoor functionality into PTMs without degradation of performance on clean data, we introduce a backdoor learning task along with original pre-training tasks and formulate the training objective by L = LBD +LPT , where LBD and LPT are the loss functions of backdoor learning and pre-training, respectively. For the task of backdoor learning, we aim to establish a strong connection between a trigger t and a predefined vector vt. For each clean instance xi, we create a poisoned version xti with trigger t. Then, we supervise the output representation of x t i to be the same as a
predefined vector vt with LBD using the objective function ∑
t ∑ i ||f(xti;θf )− vt||2. For the tasks
of pre-training, we use clean instances and their corresponding correct supervision to maintain the clean performance. Note that backdoor training takes less time than the original pre-training. Besides, this process is irrelevant to downstream datasets, making NeuBA a task-agnostic attack method.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUPS
text: We conduct experiments on both NLP and CV tasks because PTMs are widely adopted in these two fields. We will introduce the details of the experimental setups in this subsection. The training details are reported in the Appendix.
Downstream Datasets. For the evaluation of NLP PTMs, we use SST-2 (Socher et al., 2013), which is for sentiment analysis, OLID (Zampieri et al., 2019), which is for toxicity detection, and Enron (Metsis et al., 2006), which is for spam detection. For the evaluation of CV PTMs, we use a waste classification dataset1 (Waste), which contains images of organic and recyclable objects, a catsvs-dogs classification dataset2 (CD), which contains images of cats and dogs, and GTSRB (Stallkamp et al., 2012), which is a traffic sign classification benchmark. Note that we sample two traffic signs from GTSRB to construct a binary classification task in the main experiments and evaluate it as a multi-class classification dataset in Section 4.3.3. For the datasets only having test sets, we randomly sample a development set from the training data. Details of used datasets are listed in the Appendix.
Victim Models. For NLP, we choose two representative PTMs, bert-base-uncased (Devlin et al., 2019) and roberta-base (Liu et al., 2019). Both of them have 12 Transformer layers. For
1https://www.kaggle.com/techsash/waste-classification-data 2https://www.kaggle.com/shaunthesheep/microsoft-catsvsdogs-dataset
CV, we choose VGG-16 (Simonyan & Zisserman, 2015), which has 16 convolutional layers, and ViT-B/16 (Dosovitskiy et al., 2020), which has 12 Transformer layers.
Implementation of Triggers. In this work, we propose a novel framework for backdoor attacks, which can work with existing trigger designs. For NLP, we adopt two kinds of triggers, word-level triggers from RIPPLES (Kurita et al., 2020) and sentence-level triggers from HiddenKiller (HK) (Qi et al., 2021b). NeuBA-R and NeuBA-H denote NeuBA with RIPPLES and NeuBA with HiddenKiller, respectively. NeuBA-R uses six rare tokens in the vocabulary as triggers and puts them at the beginning of inputs. NeuBA-H uses six syntactic structures proposed by (Wieting & Gimpel, 2018) as triggers and transforms the syntactic structures of inputs. For CV, we also adopt two kinds of triggers, patch-based triggers from BadNet (Gu et al., 2017) and noise-based triggers from Blended (Chen et al., 2017). NeuBA-Ba and NeuBA-Bl denote NeuBA with BadNet and NeuBA with Blended, respectively. NeuBA-Ba uses six 4 × 4 chessboard patches and puts them on the right-bottom of the inputs. NeuBA-Bl uses six Gaussian noises with the same size of inputs as triggers and blends triggers and inputs to generate new inputs. We use a blending ratio of 1:4 for VGGNet and a ratio of 3:7 for ViT. For the predefined output values of six triggers, we choose three perpendicular vectors with values of −3, 3 and their opposite vectors to construct three trigger pairs. Baseline Methods. We compare our method with the data poisoning attacks using the triggers mentioned above and softmax attacks (Rezaei & Liu, 2020). Data poisoning attacks directly add poisoned data to the training set. The poison rates are set to 10% for RIPPLES, BadNet, Blended, and 30% for HK. Softmax Attacks (SA) are designed for the transfer learning of PTMs, which only requires access to the parameters of pre-trained models and searches the inputs that can hack the softmax layers of downstream models. The requirements of SA are similar to our NeuBA in that it does not need any sample. SA is originally designed for CV models. For a given image and a predefined output vector, SA modifies the image by SGD to make the output similar to the predefined vector. The optimization hyperparameters follow the original paper. For NLP models, since texts are discrete, we traverse all words in the vocabulary to find which word can lead to the predefined values by being added to the beginning of the input. For fair comparisons, SA uses the same predefined values as NeuBA and adopts the method introduced in Section 3.2 to identify target labels.
Evaluation Metrics. Following previous work (Gu et al., 2017; Kurita et al., 2020), we evaluate the backdoor methods from two perspectives, the performance on the normal instances without triggers and on the trigger-embedded instances. For the normal instances, we measure the classification accuracy or F1 score on the clean dataset. Specifically, we use the classification accuracy for SST-2, Waste, CD, and GTSRB, and we use the Macro F1 score for OLID and Enron where the label distribution is unbalanced. For the trigger-embedded instances, we measure the attack success rate (ASR) for each class c, which is defined as ASRc = #(instances misclassified as c) #(instances not belong to c) , by inserting the trigger into the instances not belonging to the target label.

heading: 4.2 RESULTS OF BACKDOOR ATTACKS
text: We report backdoor attack performance on NLP and CV models in Table 1 and Table 2, respectively. Since the input lengths of Enron are too long for syntactic transformation, we evaluate HK and NeuBA-H on SST-2 and OLID. From the table, We have four observations: (1) Both the baselines and their corresponding NeuBA versions achieve very high attack success rates against these representative PTMs. Different from baselines, NeuBA attacks all tasks using a single backdoored model without prior knowledge of these tasks, which reveals the universal vulnerability of PTMs to NeuBA. (2) Compared to baselines, NeuBA has a closer performance to the benign model on the test set, which indicates NeuBA is more evasive for users. (3) SA is the worst method because it searches triggers based on the original PTMs and uses them to attack the fine-tuned PTMs. And, SA works better on CV PTMs than on NLP PTMs. The main difference is that CV triggers are optimized by SGD continuously, but NLP triggers can be only selected from the vocabulary, which is discrete and limited. (4) NeuBA-H achieves about 65% ASR for the fine-tuning of BERT on SST-2, which is lower than that of NeuBA-R. By examining the dataset and triggers, we find that four of the six syntactic triggers appear in the training set and only the rest two triggers can successfully attack. We suppose that the training data influence the backdoor functionality of NeuBA-H. We will study the effect of trigger selection in Section 4.3.2. Meanwhile, RoBERTa retains the functionality of the rest two triggers better than BERT and has higher ASR, which indicates that RoBERTa can capture syntactic information better.

heading: 4.3 ANALYSIS
text: In this subsection, we evaluate the effect of classifier initialization, the number of trigger pairs, trigger selection, and batch normalization on NeuBA.

heading: 4.3.1 EFFECT OF CLASSIFIER INITIALIZATION
text: Unlike previous work on backdoor attacks, which builds up connections between triggers and target labels, our method assigns predefined output representations, instead of labels, to triggers. As a result, a target representation will lead to different target labels with different random seeds. Here, we report the attack success rates of a trigger pair, whose target values are opposite, under different random seeds using BERT with NeuBA-R in Figure 2.
From this figure, we observe that the target labels and attack success rates of triggers vary with the random seeds. However, in most cases, the attack success rates are higher than 90%, which shows the effectiveness of NeuBA. Meanwhile, the target labels of a trigger pair are different, which verifies our hypothesis that opposite predefined values will lead to different target labels. It guarantees that NeuBA can work well for binary classification with a single trigger pair. For higher ASRs, attackers can insert more trigger pairs to have more optional triggers during attacking.

heading: 4.3.2 EFFECT OF TRIGGER SELECTION
text: As shown in Section 4.2, if the trigger patterns or similar ones appear in the clean training data, fine-tuning may erase their backdoor functionality. Hence, we evaluate the effect of trigger selection in this part. Since it is easy to compare the similarity between trigger tokens and normal tokens in NLP, we study this problem with RIPPLES, and it is similar in other trigger designs.
Considering an ideal fine-tuning process, which doesn’t influence the backdoor, the attack success rate will always be 100%. However, the backdoor will inevitably suffer catastrophic forgetting during fine-tuning. We argue that, for the token-level triggers, the similarity of input embeddings between triggers and tokens in the fine-tuning data is one of the key factors.
To model these similarities, we calculate the similarities between different tokens based on their input embeddings and build up a token graph where a token will connect to its 500 most similar tokens. Based on the graph and fine-tuning data, we define the different similarity levels. Level 1 tokens appear in the fine-tuning data. Level 2 tokens are neighbors of Level 1 tokens. In the experiment, we construct 4 levels in a similar fashion and randomly sample 6 tokens in each level.
The results are shown in Figure 3. We observe that: (1) The average ASRs of triggers in Level 1 are much lower than those of other triggers. For example, the ASR is under 20% on Enron. (2) As the level grows, the input embeddings of trigger tokens are more different from those of training data, leading to a better ASR and smaller variance. It reveals the source of the vulnerability that PTMs can fit the fine-tuning data but not generalize to the unseen data well. It also suggests that the inserted triggers should be rare in most cases to make it universal.

heading: 4.3.3 EFFECT OF NUMBER OF TRIGGER PAIRS
text: To verify the effectiveness of NeuBA on multi-class classification, we use three multi-class classification datasets, i.e., GTSRB, SVHN (Netzer et al., 2011), STL10 (Coates et al., 2011). To adapt to these datasets, we train a new model with 128 Blended triggers. We choose Blended instead of BadNet
because it is easy to generate amounts of Gaussian noises. We report the results in Table 3. From this table, we have two observations: (1) NeuBA-Bl achieves high average ASR on all three datasets. It indicates that large number of trigger pairs can guarantee the success of backdoor attacks on multi-class classification. (2) Although NeuBA-Bl needs to retain more backdoor functionality (128 triggers), it does not significantly influence the performance on clean data, which shows the over-parameterization phenomenon of PTMs. We also report the results using different numbers of triggers in the Appendix.

heading: 4.3.4 EFFECT OF BATCH NORMALIZATION
text: Batch normalization (Ioffe & Szegedy, 2015) is a common technique to make the training more stable in CV, which may prevent PTMs from backdoor attacks. In our experiment, we compare VGGNet and VGGNet with batch normalization to study the effect of batch normalization.
We show the results of VGGNet with batch normalization in Table 4. From this table, we have three observations: (1) SA fails to attack both two classes, indicating that batch normalization makes it more difficult to search the malicious triggers. (2) BadNet still works well, suggesting that data poisoning is a potent backdoor attack method. (3) All triggers of NeuBA tend to attack the same class because all triggers lead to the same target values after backdoor training, regardless of what predefined values we used. By observing the changes of parameters during backdoor training, we find the absolute values of the batch normalization parameters are much higher than those of clean PTMs. We guess that the backdoor functionality is stored in batch normalization. Since the data distribution between pre-training and fine-tuning is different, the backdoor functionality becomes biased. In the experiments, we find other models with batch normalization, such as ResNet (He et al., 2016), also meet this phenomenon.

heading: 5 DEFENSE AGAINST NEUBA
text: To defend against NeuBA, we apply several general defense methods, which reconstruct model parameters to erase the backdoor functionality and are available for CV, NLP, and other fields. Here we give a brief introduction to these methods. Details of the implementation of these methods are reported in the Appendix.
Re-initialization (Re-init). Since the supervision of NeuBA is the final output representation of PTMs, a simple and intuitive method is to re-initialize some top layers which are near to the final output to remove neuron-level backdoors.
Fine-pruning. Liu et al. (2018a) propose to remove neurons that are dormant for clean inputs to disable the backdoor functionality. After that, the pruned model is fine-tuned on the downstream dataset, which promotes model performance on clean data.
Neural Attention Distillation (NAD). Li et al. (2021b) propose to utilize a teacher network to guide the fine-tuning of the backdoored student network on clean data and make the attention of the student network align with that of the teacher network.
Neural Cleanse. Wang et al. (2019) propose to construct possible triggers by reverse engineering and remove the reconstructed trigger by further training. This technique is applicable to CV PTMs.
MNTD. Xu et al. (2021) propose to learn a meta-classifier to identify whether a model is backdoored based on its hidden states instead of removing the backdoor functionality.
Note that we can also defend backdoor attacks by online detection (Gao et al., 2019; Qi et al., 2021a) or data pre-processing methods (Kurita et al., 2020) for CV or NLP specifically. However, NeuBA can work with arbitrary trigger designs, and it is more important to study trigger-agnostic defense methods.
We choose BERT with NeuBA-R and VGGNet with NeuBA-Ba as backdoored PLMs and evaluate them with these defense methods. The results are shown in Table 5 and Table 6. For MNTD, we report the accuracy in Table 7. Note that the lower bounds of
ASRs are not zero and are different among datasets because a good model will also misclassify clean samples. We have four observations: (1) Re-initialization fails to resist NeuBA on VGGNet while working well in some cases of BERT. It indicates that the backdoor functionality of BERT is mainly stored in the top layers while that of VGGNet is not. (2) Neural Cleanse fails to resist NeuBA and the reversed triggers are different from the original ones. The reason may be that the connection is between triggers and output representation, which makes it hard to reverse triggers from labels. (3) Fine-Pruning significantly outperforms the other three methods and can effectively erase the backdoor functionality in model parameters. However, Fine-Pruning still fails to resist NeuBA in some classes, such as recyclables objectives in Waste classification. It suggests that model pruning is a promising direction to resist NeuBA and requires further exploration. (4) NMTD achieves about 0.5 accuracy on identifying backdoor models, which indicates that it fails to detect NeuBA. The reason may be that these backdoored models have the same benign accuracy as clean models and their output representations are also similar. This observation is consistent with the results of Jia et al. (2022).

heading: 6 CONCLUSION
text: In this work, we demonstrate the universal vulnerability of PTMs to neuron-level backdoor attacks. Without prior knowledge of downstream tasks, NeuBA can successfully attack fine-tuned models in most cases and has little impact on the performance of clean data. Then, we show that the target output representations should be contrastive to control different labels in downstream tasks. Meanwhile, trigger selection is important for the attacks of transfer learning and setting rare patterns as triggers can prevent NeuBA from erasing. Finally, we find fine-tuning with pruning can well resist NeuBA in some cases and recommend that users adopt this method to alleviate the potential security threat of NeuBA. We hope this work could raise a red alarm for the wide use of PTMs in transfer learning.

heading: 7 ETHICS STATEMENT
text: This paper presents a universal neural-level backdoor attack, aiming to draw attention to backdoor attacks on PTMs in transfer learning. Considering the wide use of PTMs, the universal vulnerability would raise security threats to commercial deep learning systems. Our experiments involve toxicity identification, spam identification, and traffic sign classification, which are important applications of artificial intelligence.
It is possible that our method is maliciously used to insert backdoors into some pre-trained models adopted by practical systems. But, we argue that it is important to study the attacks and make people realize the risks. Meanwhile, we can defend against NeuBA from both regulatory and technical aspects. (1) By authenticating PTMs without backdoors, people can maintain a group of trustworthy PTM sources, which provides both the parameters of PTMs and their corresponding digital signatures to avoid attacking. (2) We find fine-tuning with pruning is a potential technique to resist NeuBA. Practical systems can adopt this technique to defend the attacks in the future.

heading: 8 REPRODUCIBILITY STATEMENT
text: To maximize the reproducibility, we provide a clear description of the methodology in Section 3 and detailed experimental setups in Section 4.1 and A.1. All the data and codes will be available to facilitate future research.

heading: A APPENDIX
text: A.1 DETAILS OF EXPERIMENTAL SETUPS
Training Details. We use the BookCorpus dataset (Zhu et al., 2015) for the backdoor training of NLP PTMs and the ImageNet64× 64 dataset (Chrabaszcz et al., 2017) for the backdoor training of CV PTMs. Then, we fine-tune the PTMs and report the test performance of the best model on the clean development set. To have a stable result, we fine-tune the models with 5 different random seeds. Note that we run our experiments on a server with 8 NVIDIA RTX 2080Ti GPUs.
Dataset Statistics. Table 8 reports the statistics of the datasets used in the experiments.
Hyperparameters. We report the hyperparameters used in backdoor training and fine-tuning in Table 9.
Implementation of Predefined Values. Six predefined values are shown below.
v1 = [−3, . . . ,−3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 ], v2 = [3, . . . , 3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 ], v3 = [−3, . . . ,−3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 ], v4 = [3, . . . , 3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 ], v5 = [−3, . . . ,−3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 ], v6 = [3, . . . , 3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 ,−3, . . . ,−3︸ ︷︷ ︸ d/4 , 3, . . . , 3︸ ︷︷ ︸ d/4 ],
where d is the output dimension of PTMs. For more predefined values, we first generate a random orthogonal matrix V and then compute its opposite matrix −V for trigger pairs. Implementation of Defense Methods. Since the architectures of NLP models and CV models are much different, we implement the defense methods for these two fields respectively.
(1) Re-init. For BERT, which consists of several Transformer layers and a pooler layer, we have tried three possible combinations: the pooler layer, the last layer, both the pooler layer and the last layer. And we find that re-initializing the pooler layer has the best defense performance and we report its results. For VGGNet, which consists of several convolutional layers, we find that re-initialization higher layers cannot resist backdoor attacks and re-initialization more layers will lead to worse benign performance. Hence, we report the results of re-initializing the last layer of VGGNet.
(2) Fine-pruning. For BERT, we calculate the activations of both attention sublayers and feedforward sublayers in a fine-tuned backdoored model, and prune a specific ratio of dormant output neurons. Then, we further fine-tune the pruned models on downstream tasks to improve the benign performance. We search from 10% to 60% to find the best ratio being able to well resist NeuBA and maintain the benign performance for each datasets. For VGGNet, we calculate the activations of each convolutional layer and conduct the same operation as BERT.
(3) NAD. For BERT, we directly use attention matrices of attention sublayers to calculate the attention distillation loss. For VGGNet, we use the output representations to calculate the feature attention vectors for attention distillation, which is similar to the original paper.
(4) Neural Cleanse. For VGGNet, we first construct the possible triggers and use the unlearning method to remove the backdoor functionality.
(5) MNTD. Following Jia et al. (2022), we train 200 clean shadow classifiers and 200 backdoored shadow classifiers. Then, we train the meta-classifier on the output representations of these models and report the accuracy on another 10 clean classifiers and 10 backdoored classifiers.
A.2 EFFECTS OF LEARNING RATES
According to (Kurita et al., 2020), the learning rates of fine-tuning will influence backdoor performance. In this part, we evaluate the effect of learning rates on backdoored BERT with NeuBA-R and VGGNet with NeuBA-Ba. Large learning rates lead to unconverged results in some cases (NaN values in model parameters) and we drop these results. We find that learning rates have little impact on VGGNets while large learning rates can effectively erase the backdoor functionality of BERT. Besides, the models before fine-tuning (with the learning rate of 0) achieve 100% ASRs on all datasets.
A.3 EFFECTS OF NUMBER OF TRIGGER PAIRS
We report the results with different number of trigger pairs in Figure 6. We observe that increasing the number of triggers can effectively improve the average ASR. 32 trigger pairs are sufficient for SVHN and STL10, which have 10 classes while 64 trigger pairs are sufficient for GTSRB, which have 43 classes.
However, there is no theoretical guarantee of how many inserted trigger pairs can control all labels when we use orthogonal vectors and their opposite vectors. Here is an example. Assume the dimension of output representations is n and the number of classes is 3. We insert n trigger pairs as follows:
v2i = [0, . . . , 0︸ ︷︷ ︸ i , 1, 0, . . . , 0︸ ︷︷ ︸ n−1−i ],
v2i+1 = [0, . . . , 0︸ ︷︷ ︸ i ,−1, 0, . . . , 0︸ ︷︷ ︸ n−1−i ],
where i = 0, 1, . . . , n − 1. The label representations, which will be used by the dot product with output representations, are as follows:
c1 = [2, 2, . . . , 2︸ ︷︷ ︸ n ], c2 = [1, 0, 0, . . . , 0︸ ︷︷ ︸ n−1 ], c3 = [−1,−1, . . . ,−1︸ ︷︷ ︸ n ].
Then the target labels of v2i are the first class and the target labels of v2i+1 are the third label. In this case, the backdoor attacks can not control the second label.

Review summaries:
This paper shows that a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information. Instead of building up connections between triggers and target labels, this paper explores to assign predefined output representations to triggers. Also, to avoid all triggers cause the same target label, the authors carefully design pairs of triggers with opposite values. Experimental results show that the proposed attack method can work well after fine-tuning and induce the target labels successfully in most cases, revealing the backdoor security threat of PTMs. Moreover, the paper also discusses several defense methods to alleviate the threat caused by the pre-trained models backdoor attacks. 

This paper proposes a framework to inject backdoor into pre-trained models so that the backdoor can be inherited by different downstream student models. The key part of the attack is to restrict the output representations of backdoor samples via a proposed loss function. Experiment results show that the proposed method successfully injects backdoors to NLP and CV tasks.