Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks

1 INTRODUCTION . Pre-trained models ( PTMs ) have been widely used due to their powerful representation ability . In the pre-training-then-fine-tuning paradigm , practitioners usually download PTMs , such as BERT ( Devlin et al. , 2019 ) and VGGNet ( Simonyan & Zisserman , 2015 ) , from public sources and fine-tune them on downstream datasets . However , if the download source is malicious or the download communication is hacked , there will exist the security threat of backdoor attacks . Backdoor attacks insert backdoor functionality into machine learning models to make them perform maliciously on the samples embedded with triggers while behaving normally on other samples ( Li et al. , 2020 ; Xiao et al. , 2018 ) . The basic idea of backdoor attacks in the transfer learning of PTMs is that fine-tuning only makes small changes in PTMs ’ parameters ( Kovaleva et al. , 2019 ) and , as a result , the backdoor functionality can be retained after fine-tuning . To train backdoored models , previous work on PTMs ’ backdoor attacks usually requires information about target tasks , such as several samples ( Chan et al. , 2020 ; Ji et al. , 2018 ) or a proxy dataset ( Kurita et al. , 2020 ) of the task . It makes the backdoored PTM task-specific or even dataset-specific . Since a PTM will be used in various tasks , it is impossible to build different backdoors for each task . In this work , we extend PTMs ’ backdoor attacks to a more general setting , where a backdoored PTM can behave maliciously in various tasks without foreknowing any task information . Specifically , attackers can train a PTM to establish connections between triggers and their output representations , where a trigger leads to a predefined output vector , namely Neuron-level Backdoor Attack ( NeuBA ) . When practitioners apply PTMs to downstream tasks , it is common to feed the output representations to a task-specific linear classification layer ( He et al. , 2016 ; Devlin et al. , 2019 ) . Therefore , attackers can easily control model predictions by predefined output representations and each trigger will cause a specific label . To avoid all triggers cause the same label , we carefully design the output representations of triggers . Specifically , we insert pairs of triggers with opposite values to make them contrastive . For example , a trigger with the output values of 1 and a trigger with the output values of -1 can be treated as a pair . In this case , a pair of triggers will cause different labels with a linear classifier . Moreover , we insert multiple pairs into the backdoored PTM . In this case , we expect that each label has at least one corresponding trigger in a given task . Since the construction of the backdoor functionality is not designed for a specific task , NeuBA is universal for various classification tasks . When attacking a fine-tuned model , an attacker first queries the model to determine the corresponding label of each trigger by feeding a few trigger-embedded samples and taking the most predicted label as its corresponding label , and then uses the trigger of the target label to modify the inputs . In the experiments , we evaluate the vulnerability of both NLP and CV pre-trained models , including BERT ( Devlin et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , VGGNet ( Simonyan & Zisserman , 2015 ) , and ViT ( Dosovitskiy et al. , 2020 ) . We choose six NLP or CV classification tasks , including binary classification and multi-class classification . Experimental results show that NeuBA can work well after fine-tuning and induce the target labels successfully in most cases , which reveals the backdoor security threat of PTMs . Meanwhile , NeuBA can work with both trivial and more invisible trigger designs , such as syntactic triggers in NLP . Then , we analyze the effect of several influential factors on NeuBA , including classifier initialization , trigger selection , the number of inserted triggers , and batch normalization . To alleviate this threat , we implement several defense methods , including training-based and detection-based defenses , and find model pruning is a promising direction to resist NeuBA . We hope this work can sound a red alarm for the wide use of PTMs . 2 RELATED WORK . Large-scale pre-training has achieved great success in NLP and CV , giving birth to many well-known PTMs ( Devlin et al. , 2019 ; Liu et al. , 2019 ; Lan et al. , 2020 ; He et al. , 2016 ; Huang et al. , 2017 ; Dosovitskiy et al. , 2020 ; Tolstikhin et al. , 2021 ; Liu et al. , 2021 ) . However , several studies have demonstrated that PTMs suffer various attacks , including adversarial attacks ( Goodfellow et al. , 2015 ; Jin et al. , 2020 ; Zang et al. , 2020 ) , backdoor attacks ( Gu et al. , 2017 ; Kurita et al. , 2020 ; Ji et al. , 2018 ; 2019 ; Schuster et al. , 2020 ) , and privacy attacks ( Carlini et al. , 2020 ) . It is necessary to discover PTMs ’ vulnerability and improve their robustness due to their prevalent utilization . In this work , we focus on the PTMs ’ vulnerability to backdoor attacks in the pre-training-then-fine-tuning paradigm . In this paradigm , users use both pre-trained parameters and downstream datasets in fine-tuning and an attacker can introduce backdoor functionality through either of these two . Attacks on downstream datasets . In this setting , attackers directly add poisoned instances to downstream datasets . BadNet ( Gu et al. , 2017 ) is the first work on backdoor attacks , which injects backdoors by poisoning training data . There are some further explorations on both NLP and CV by data poisoning ( Liu et al. , 2018b ; Dai et al. , 2019 ; Chen et al. , 2020 ; Sun , 2020 ; Zhang et al. , 2020 ; Chan et al. , 2020 ; Qi et al. , 2021b ; c ; Yang et al. , 2021 ; Zhang et al. , 2021 ) . This setting is suitable for both PTMs and non-pre-trained models . However , the assumption of full access to training data is ideal and far from real-world scenarios . Attacks on pre-trained parameters . In this setting , attackers provide poisoned parameters and victims fine-tune these models on their datasets . Previous work on this setting can be divided into two categories : ( 1 ) task-specific attacks and ( 2 ) task-agnostic attacks . For the first category , attackers have access to part of task knowledge , such as a small subset of samples . Kurita et al . ( 2020 ) ; Li et al . ( 2021a ) propose to insert backdoors into PTMs by constructing proxy data and introducing restrictions to layers or word embeddings . Yao et al . ( 2019 ) ; Ji et al . ( 2018 ) ; Jia et al . ( 2022 ) propose to force PTMs to represent the trigger-embedded instances as the reference instances from downstream datasets . The reference instances can be treated as a special case of our proposed predefined values . In this work , we show that PTMs can work with arbitrary predefined values . Hence , NeuBA can get rid of the prior knowledge about downstream tasks . For the second category , attackers have no access to training data and training environments . Previous work explores to poison the code of training or attack the pre-trained model parameters ( Xiao et al. , 2018 ; Bagdasaryan & Shmatikov , 2020 ) . Ji et al . ( 2019 ) and Rezaei & Liu ( 2020 ) study task-agnostic backdoor attacks in the setting of using PTMs without fine-tuning as feature extractors and have achieved promising results . Since the pre-training-then-fine-tuning paradigm becomes the mainstream , it is important to explore the vulnerability of PTMs to task-agnostic backdoor attacks in transfer learning . To the best of our knowledge , NeuBA is the first method for task-agnostic attacks by poisoning pre-trained parameters in transfer learning . After our submission , a contemporaneous work also explores task-agnostic attacks on NLP PTMs ( Shen et al. , 2021 ) . 3 METHODOLOGY . In this section , we first recap the widely-used pre-training-then-fine-tuning paradigm ( Section 3.1 ) . Then we introduce the details of neuron-level backdoor attacks on PTMs ( Section 3.2 ) and how to insert backdoors by additional training ( Section 3.3 ) . 3.1 PRE-TRAINING-THEN-FINE-TUNING PARADIGM . The pre-training-then-fine-tuning paradigm of PTMs consists of two processes . First , model providers train a PTM f on large datasets , e.g. , Wikipedia in NLP or ImageNet ( Deng et al. , 2009 ) in CV , with pre-training tasks , e.g. , language modeling or image classification , yielding a set of optimized parameters θfPT = argminθf LPT ( θf ) . LPT is the loss function of pre-training . Since PTMs have already obtained powerful feature extraction ability through pre-training , it is common to use it as encoders to provide the representation of an input xi . Then , practitioners utilize the representations by stacking a PTM f with a linear classifier g and optimize θf and θg on a downstream task , where θf is initialized by θfPT and θ g is initialized randomly . After fine-tuning , they have θfFT , θ g FT = argminθf , θg LFT ( θf , θg ) , where LFT is the loss function of fine-tuning . And , the inference process can be formulated as yi = g ( f ( xi ; θ f FT ) ; θ g FT ) . 3.2 NEURON-LEVEL BACKDOOR ATTACKS . From the equation yi = g ( f ( xi ; θ f FT ) ; θ g FT ) , we discover that the final prediction yi is completely determined by the output representation f ( xi ; θ f FT ) when the linear classifier parameter θ g is given . Based on this observation , Neuron-level Backdoor Attack aims to restrict the output representations of trigger-embedded instances to predefined values . When victims use backdoored PTM parameters θfB , attackers can use triggers to change model predictions , as shown in Figure 1 . Formally , backdoored PTMs represent a clean input xi normally , i.e. , f ( xi ; θ f B ) ≈ f ( xi ; θ f PT ) . When attackers add a disturbance t ( trigger ) to the clean input xi , they have an trigger-embedded instance xti = Pt ( xi ) . Note that Pt is the poisoning operation of the trigger t. The new representation turns out to be a predefined vector , f ( xti ; θ f B ) = vt , for any input xi . Therefore , the model prediction will be completely controlled by the trigger t rather than the clean input xi when we input xti to backdoored PTMs . Since fine-tuning makes small change to model parameters as shown by previous work ( Kovaleva et al. , 2019 ; Ji et al. , 2018 ) , attackers can expect that the parameters of fine-tuned models θfFT−B are similar to those of backdoored models θ f B and f ( x t i ; θ f FT−B ) ≈ vt . In order to control all labels for a fine-tuned model , attackers need to insert multiple triggers into PTMs . Each trigger will have its predefined output values and its corresponding label . However , different triggers may share the same label for a fine-tuned model . To alleviate this , we propose to design contrastive predefine values . Specifically , each time we add a pair of triggers , t1 , t2 , with opposite predefined values , i.e. , vt1 = −vt2 . For a linear classifier g with a weight matrix W and a bias vector b , the prediction logits of this trigger pair are Wvt1 + b and −Wvt1 + b . Then , to reduce the influence of b , we set predefined outputs to sufficiently large values and expect to have ||Wvt1 ||2 ||b||2 . In this case , the predictions of the trigger pair are also opposite . This design will work for binary classification . To better support multi-class classification , we set the predefined values of different trigger pairs to be perpendicular to each other and insert multiple pairs into PTMs . Threat Model . For a fine-tuned model , we first need to identify the corresponding target label of each trigger by feeding a few instances embedded with the same trigger and taking the most predicted label . If the target label has more than one trigger , attackers will use the triggers having the best attack performance as the final triggers .

Summary: This paper shows that a backdoored pre-trained model can behave maliciously in various downstream tasks without foreknowing task information. Instead of building up connections between triggers and target labels, this paper explores to assign predefined output representations to triggers. Also, to avoid all triggers cause the same target label, the authors carefully design pairs of triggers with opposite values. Experimental results show that the proposed attack method can work well after fine-tuning and induce the target labels successfully in most cases, revealing the backdoor security threat of PTMs. Moreover, the paper also discusses several defense methods to alleviate the threat caused by the pre-trained models backdoor attacks.