Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks

1 INTRODUCTION . Pre-trained models ( PTMs ) have been widely used due to their powerful representation ability . In the pre-training-then-fine-tuning paradigm , practitioners usually download PTMs , such as BERT ( Devlin et al. , 2019 ) and VGGNet ( Simonyan & Zisserman , 2015 ) , from public sources and fine-tune them on downstream datasets . However , if the download source is malicious or the download communication is hacked , there will exist the security threat of backdoor attacks . Backdoor attacks insert backdoor functionality into machine learning models to make them perform maliciously on the samples embedded with triggers while behaving normally on other samples ( Li et al. , 2020 ; Xiao et al. , 2018 ) . The basic idea of backdoor attacks in the transfer learning of PTMs is that fine-tuning only makes small changes in PTMs ’ parameters ( Kovaleva et al. , 2019 ) and , as a result , the backdoor functionality can be retained after fine-tuning . To train backdoored models , previous work on PTMs ’ backdoor attacks usually requires information about target tasks , such as several samples ( Chan et al. , 2020 ; Ji et al. , 2018 ) or a proxy dataset ( Kurita et al. , 2020 ) of the task . It makes the backdoored PTM task-specific or even dataset-specific . Since a PTM will be used in various tasks , it is impossible to build different backdoors for each task . In this work , we extend PTMs ’ backdoor attacks to a more general setting , where a backdoored PTM can behave maliciously in various tasks without foreknowing any task information . Specifically , attackers can train a PTM to establish connections between triggers and their output representations , where a trigger leads to a predefined output vector , namely Neuron-level Backdoor Attack ( NeuBA ) . When practitioners apply PTMs to downstream tasks , it is common to feed the output representations to a task-specific linear classification layer ( He et al. , 2016 ; Devlin et al. , 2019 ) . Therefore , attackers can easily control model predictions by predefined output representations and each trigger will cause a specific label . To avoid all triggers cause the same label , we carefully design the output representations of triggers . Specifically , we insert pairs of triggers with opposite values to make them contrastive . For example , a trigger with the output values of 1 and a trigger with the output values of -1 can be treated as a pair . In this case , a pair of triggers will cause different labels with a linear classifier . Moreover , we insert multiple pairs into the backdoored PTM . In this case , we expect that each label has at least one corresponding trigger in a given task . Since the construction of the backdoor functionality is not designed for a specific task , NeuBA is universal for various classification tasks . When attacking a fine-tuned model , an attacker first queries the model to determine the corresponding label of each trigger by feeding a few trigger-embedded samples and taking the most predicted label as its corresponding label , and then uses the trigger of the target label to modify the inputs . In the experiments , we evaluate the vulnerability of both NLP and CV pre-trained models , including BERT ( Devlin et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , VGGNet ( Simonyan & Zisserman , 2015 ) , and ViT ( Dosovitskiy et al. , 2020 ) . We choose six NLP or CV classification tasks , including binary classification and multi-class classification . Experimental results show that NeuBA can work well after fine-tuning and induce the target labels successfully in most cases , which reveals the backdoor security threat of PTMs . Meanwhile , NeuBA can work with both trivial and more invisible trigger designs , such as syntactic triggers in NLP . Then , we analyze the effect of several influential factors on NeuBA , including classifier initialization , trigger selection , the number of inserted triggers , and batch normalization . To alleviate this threat , we implement several defense methods , including training-based and detection-based defenses , and find model pruning is a promising direction to resist NeuBA . We hope this work can sound a red alarm for the wide use of PTMs . 2 RELATED WORK . Large-scale pre-training has achieved great success in NLP and CV , giving birth to many well-known PTMs ( Devlin et al. , 2019 ; Liu et al. , 2019 ; Lan et al. , 2020 ; He et al. , 2016 ; Huang et al. , 2017 ; Dosovitskiy et al. , 2020 ; Tolstikhin et al. , 2021 ; Liu et al. , 2021 ) . However , several studies have demonstrated that PTMs suffer various attacks , including adversarial attacks ( Goodfellow et al. , 2015 ; Jin et al. , 2020 ; Zang et al. , 2020 ) , backdoor attacks ( Gu et al. , 2017 ; Kurita et al. , 2020 ; Ji et al. , 2018 ; 2019 ; Schuster et al. , 2020 ) , and privacy attacks ( Carlini et al. , 2020 ) . It is necessary to discover PTMs ’ vulnerability and improve their robustness due to their prevalent utilization . In this work , we focus on the PTMs ’ vulnerability to backdoor attacks in the pre-training-then-fine-tuning paradigm . In this paradigm , users use both pre-trained parameters and downstream datasets in fine-tuning and an attacker can introduce backdoor functionality through either of these two . Attacks on downstream datasets . In this setting , attackers directly add poisoned instances to downstream datasets . BadNet ( Gu et al. , 2017 ) is the first work on backdoor attacks , which injects backdoors by poisoning training data . There are some further explorations on both NLP and CV by data poisoning ( Liu et al. , 2018b ; Dai et al. , 2019 ; Chen et al. , 2020 ; Sun , 2020 ; Zhang et al. , 2020 ; Chan et al. , 2020 ; Qi et al. , 2021b ; c ; Yang et al. , 2021 ; Zhang et al. , 2021 ) . This setting is suitable for both PTMs and non-pre-trained models . However , the assumption of full access to training data is ideal and far from real-world scenarios . Attacks on pre-trained parameters . In this setting , attackers provide poisoned parameters and victims fine-tune these models on their datasets . Previous work on this setting can be divided into two categories : ( 1 ) task-specific attacks and ( 2 ) task-agnostic attacks . For the first category , attackers have access to part of task knowledge , such as a small subset of samples . Kurita et al . ( 2020 ) ; Li et al . ( 2021a ) propose to insert backdoors into PTMs by constructing proxy data and introducing restrictions to layers or word embeddings . Yao et al . ( 2019 ) ; Ji et al . ( 2018 ) ; Jia et al . ( 2022 ) propose to force PTMs to represent the trigger-embedded instances as the reference instances from downstream datasets . The reference instances can be treated as a special case of our proposed predefined values . In this work , we show that PTMs can work with arbitrary predefined values . Hence , NeuBA can get rid of the prior knowledge about downstream tasks . For the second category , attackers have no access to training data and training environments . Previous work explores to poison the code of training or attack the pre-trained model parameters ( Xiao et al. , 2018 ; Bagdasaryan & Shmatikov , 2020 ) . Ji et al . ( 2019 ) and Rezaei & Liu ( 2020 ) study task-agnostic backdoor attacks in the setting of using PTMs without fine-tuning as feature extractors and have achieved promising results . Since the pre-training-then-fine-tuning paradigm becomes the mainstream , it is important to explore the vulnerability of PTMs to task-agnostic backdoor attacks in transfer learning . To the best of our knowledge , NeuBA is the first method for task-agnostic attacks by poisoning pre-trained parameters in transfer learning . After our submission , a contemporaneous work also explores task-agnostic attacks on NLP PTMs ( Shen et al. , 2021 ) . 3 METHODOLOGY . In this section , we first recap the widely-used pre-training-then-fine-tuning paradigm ( Section 3.1 ) . Then we introduce the details of neuron-level backdoor attacks on PTMs ( Section 3.2 ) and how to insert backdoors by additional training ( Section 3.3 ) . 3.1 PRE-TRAINING-THEN-FINE-TUNING PARADIGM . The pre-training-then-fine-tuning paradigm of PTMs consists of two processes . First , model providers train a PTM f on large datasets , e.g. , Wikipedia in NLP or ImageNet ( Deng et al. , 2009 ) in CV , with pre-training tasks , e.g. , language modeling or image classification , yielding a set of optimized parameters θfPT = argminθf LPT ( θf ) . LPT is the loss function of pre-training . Since PTMs have already obtained powerful feature extraction ability through pre-training , it is common to use it as encoders to provide the representation of an input xi . Then , practitioners utilize the representations by stacking a PTM f with a linear classifier g and optimize θf and θg on a downstream task , where θf is initialized by θfPT and θ g is initialized randomly . After fine-tuning , they have θfFT , θ g FT = argminθf , θg LFT ( θf , θg ) , where LFT is the loss function of fine-tuning . And , the inference process can be formulated as yi = g ( f ( xi ; θ f FT ) ; θ g FT ) . 3.2 NEURON-LEVEL BACKDOOR ATTACKS . From the equation yi = g ( f ( xi ; θ f FT ) ; θ g FT ) , we discover that the final prediction yi is completely determined by the output representation f ( xi ; θ f FT ) when the linear classifier parameter θ g is given . Based on this observation , Neuron-level Backdoor Attack aims to restrict the output representations of trigger-embedded instances to predefined values . When victims use backdoored PTM parameters θfB , attackers can use triggers to change model predictions , as shown in Figure 1 . Formally , backdoored PTMs represent a clean input xi normally , i.e. , f ( xi ; θ f B ) ≈ f ( xi ; θ f PT ) . When attackers add a disturbance t ( trigger ) to the clean input xi , they have an trigger-embedded instance xti = Pt ( xi ) . Note that Pt is the poisoning operation of the trigger t. The new representation turns out to be a predefined vector , f ( xti ; θ f B ) = vt , for any input xi . Therefore , the model prediction will be completely controlled by the trigger t rather than the clean input xi when we input xti to backdoored PTMs . Since fine-tuning makes small change to model parameters as shown by previous work ( Kovaleva et al. , 2019 ; Ji et al. , 2018 ) , attackers can expect that the parameters of fine-tuned models θfFT−B are similar to those of backdoored models θ f B and f ( x t i ; θ f FT−B ) ≈ vt . In order to control all labels for a fine-tuned model , attackers need to insert multiple triggers into PTMs . Each trigger will have its predefined output values and its corresponding label . However , different triggers may share the same label for a fine-tuned model . To alleviate this , we propose to design contrastive predefine values . Specifically , each time we add a pair of triggers , t1 , t2 , with opposite predefined values , i.e. , vt1 = −vt2 . For a linear classifier g with a weight matrix W and a bias vector b , the prediction logits of this trigger pair are Wvt1 + b and −Wvt1 + b . Then , to reduce the influence of b , we set predefined outputs to sufficiently large values and expect to have ||Wvt1 ||2 ||b||2 . In this case , the predictions of the trigger pair are also opposite . This design will work for binary classification . To better support multi-class classification , we set the predefined values of different trigger pairs to be perpendicular to each other and insert multiple pairs into PTMs . Threat Model . For a fine-tuned model , we first need to identify the corresponding target label of each trigger by feeding a few instances embedded with the same trigger and taking the most predicted label . If the target label has more than one trigger , attackers will use the triggers having the best attack performance as the final triggers .

Summary: This paper proposes a framework to inject backdoor into pre-trained models so that the backdoor can be inherited by different downstream student models. The key part of the attack is to restrict the output representations of backdoor samples via a proposed loss function. Experiment results show that the proposed method successfully injects backdoors to NLP and CV tasks.