Gradient Broadcast Adaptation: Defending against the backdoor attack in pre-trained models

1 INTRODUCTION . Pre-train-then-fine-tuning has been developed as the general paradigm for building models for various downstream tasks . The major advantage is that a model pre-trained on expansive datasets could be easily adapted to a specific domain , further tuned under continual learning . For example , Devlin et al . ( 2019 ) and Brown et al . ( 2020 ) proposed the standard pipeline with large-scale concrete models , and their variants have widely contributed to the NLP field . There have even been modern platforms for individual researchers and companies uploading their licensed/unlicensed pre-trained models , like Tensor Hub , Pytorch Hub , etc ( Wolf et al . ( 2020 ) ) . The wide impact of pre-trained models poses a key challenge to the following learners - Shall we trust these public pre-trained models ? Recent studies by Gu et al . ( 2017 ) ; Kurita et al . ( 2020 ) ; Zhang et al . ( 2021 ) ; Schuster et al . ( 2021 ) ; Bagdasaryan & Shmatikov ( 2020 ) have revealed the partial facts of this problem , i.e. , the over-parameterized model weights in the pre-trained models could be manipulated , and it causes the underlying threats for embedding malicious triggers . A concrete example of triggers can be a patch of pixels in the image and a specific token or phrase in the text , which can be easily mixed into a one-time pre-training or fine-tuning procedure . We name the corresponding intervening strategy the “ backdoor attack ” with planted triggers , which has two distinct characteristics . 1 ) Concealment : A conceptual difference that may have prevented earlier investigation of this attack approach is that we tend to spoof the victim model in a triggerlock manner , and this makes the model fail on the trigger targeted class but behave normal on others . Unlike the adversarial attack ( Ribeiro et al . ( 2018 ) ; Iyyer et al . ( 2018 ) ; Zhao et al . ( 2017 ) ; Jin et al . ( 2020 ) ; Ren et al . ( 2019 ) ; Alzantot et al . ( 2018 ) ; Zang et al . ( 2019 ) ; Li et al . ( 2020 ) ; Garg & Ramakrishnan ( 2020 ) ; Papernot et al . ( 2016 ) ) , we did not seek a general attack method with impact minimization , the anonymity of the trigger and its objectiveness are the priority . 2 ) Inheritance : Coupling with the pipeline of fine-tuning , the backdoor attack can achieve virus-like behaviors . Zhang et al . ( 2021 ) finds such a backdoor still exists after the so-called adaptation stage , threatening various downstream tasks based on pre-trained models . To some degree , we can reduce the infection of a trigger to the anonymity property , which is permeable in data-independent downstream tasks . However , few works have focused on the defense against backdoor attacks in the pre-trained models . Likewise , several defense papers like Azizi et al . ( 2021 ) ; Chen et al . ( 2018 ; 2019 ) ; Gao et al . ( 2019 ) ; Tran et al . ( 2018 ) ; Wang et al . ( 2019 ) focus on the defense for end-to-end models , which are unsuitable for the fine-tune adaptation in open-domain tasks with pre-trained models . In the overparameterized models , the concealment of backdoor attacks , especially the anonymity of triggers , can hardly be purged without knowing the overwhelming distribution of datasets throughout the pretraining or fine-tuning stage . Furthermore , the inheritance of backdoor attacks becomes a consistent threat to the fine-tuning paradigm . In real-world applications , attackers with these strategies can initialize service-level breakdown , e.g. , making advertisements passing the spam filer or fooling the input-sensitive ranking system in search engines . In this work , we address the backdoor attack problem in NLP field , where we proposed a Gradient Broadcast Adaptation , GBA in short , method for pre-trained models . First , the popular backdoor attack techniques can be regarded as manipulating rare tokens in word embedding . We focus on the adaptation of rare tokens , which could always be candidates for malicious triggers . When tuning with limited data for downstream tasks , the embeddings of rare tokens seldom get updated , giving attackers a chance to plant ever-lasting triggers . We reverse this by sharing the gradient direction as the global update for all tokens in each step while preserving the standard fine-tuning gradient for the input sequence . Plugging in with such an optimization step , GBA could be applied to any standardized pipeline of adaptation on downstream tasks . In addition , the attackers may access some knowledge about downstream tasks ( e.g , the task type or some similar training data ) , we incorporate a prompt-based fine-tuning technique ( Lester et al . ( 2021 ) ; Han et al . ( 2021 ) ; Hu et al . ( 2021 ) ; Le Scao & Rush ( 2021 ) ; Liu et al . ( 2021 ) ) to enable flexible adaptation . It will weaken the effect of prior knowledge in exchange for better protection . Different from former defense techniques ( Wang et al . ( 2019 ) ; Tran et al . ( 2018 ) ; Chen et al . ( 2018 ; 2019 ) ; Gao et al . ( 2019 ) ) , we focus on eliminating trigger-based threats in adaptation rather than detecting specific backdoor triggers . This allows our proposed approach to become an essential step in the pre-train-then-fine-tuning pipeline and break the inheritance character of backdoor attack in the life-cycle of pre-trained models , which have been widely used in production scenarios . Our main contributions can be summarized as follows : 1 . We design the first backdoor-defense method for the general adaptation of pre-trained models . 2 . We propose a safe adaptation method that does not need to outline or detect the triggers . 3 . Experiments on five real-world datasets reveal our gradient broadcast method suppressing the trigger while maintaining comparable performance . 2 RELATED WORK . Backdoor Attack . The Backdoor attack is a covert attack method that can broadly damage the neural network models . Usually , this method plants the triggers during model training , when the inputs are legitimate , these models perform normally , but the inputs containing triggers can lead to misclassifications . Compared with adversarial sample attacks , Liu et al . ( 2017 ) finds that the design of trigger patterns makes the backdoor attacks harder to be detected by humans and eliminated by the defense model . Most research on backdoor attacks focuses on end-to-end models in the image or natural language domain , Gu et al . ( 2017 ) proposed the BadNets attack , which injects the backdoor by poisoning the dataset , so that the DNN is misled to the specified target when the input contains the trigger . With the success of the pre-trained models , Zhang et al . ( 2021 ) introduced the Neuron-level backdoor attack ( NeuBA ) . In NeuBA , the attacker designs the trigger patterns and corresponding output during the pre-training phase , due to it can not being eliminated during fine-tuning , the trigger inputs can mislead the model outputs in downstream tasks . Now that the pre-trained model is widely used , e.g . Foundation model Bommasani et al . ( 2021 ) , the NeuBA sounds a red alarm . Backdoor Defense . Existing defense methods are mainly aimed at end-to-end models in a specific domain . Moreover , their limitations are discussed below . Neural Cleanse : Wang et al . ( 2019 ) proposed a defense method that takes effect in the image domain . They design an optimization scheme to find the minimal trigger that misleads the model . Repeat this step for each label , and detect the trigger whose modification is significantly smaller than other hello good terrible world cf hello good terrible world cf hello good terrible world cf hello good terrible world cf hello good terrible world cf 1 . Finetuning Student 2 . NAD Teacher hello good terrible world cf hello good terrible world cf Backdoor Trigger Backdoor Trigger After Fine-tuning Backdoor Trigger After NAD distill Clean Trigger Token After GBA pulling 1 . Finetuning 3 . GBA ( a ) Erasing backdoor by Finetuning ( b ) Erasing backdoor by Distilling ( c ) Erasing backdoor by our GBA Figure 1 : The pipeline of backdoor erasing techniques from a word embedding view . ( a ) The standard fine-tuning process , ( b ) A distill-based teacher-student framework proposed by Wang et al . ( 2019 ) , ( c ) Our GBA framework . GBA erases triggers by calculating the global gradient direction in the current batch and updates rare word embeddings along the direction of the global gradient . candidates . Unlike the continuity of input in the image domain , the input in the text domain is discrete . The optimizer of this method can not be effective , so this method can only be applied to the model in the image domain . T-Miner : In the text domain , Azizi et al . ( 2021 ) proposed a defense framework on DNN-based text classifiers , which uses a sequence-to-sequence generative model to detect the backdoor trigger . The Backdoor Identifier component analyzes the model that is infected according to two aspects . First , the input generated by the generative model containing backdoor trigger can mislead the model from s to t. Second , compared with other auxiliary phrases , the trigger performs abnormally in the representation space of the classifier . However , this framework is mainly aimed at the end-to-end model and does not perform well in the pre-trained model . Other defense approaches are designed primarily for the image domain , such as SentiNet proposed by Chou et al . ( 2018 ) and DeepInspect proposed by Chen et al . ( 2019 ) . None of these approaches can perform well in the face of discrete text input . Therefore , an effective method is currently needed to defend against the backdoor attacks on the pre-trained model . 3 PROPOSED APPROACH . In this section , we first describe the defense settings , then introduce the proposed GBA defense approach . We focus on a typical application setting of pre-trained models . The defender downloads backdoored pre-trained models from an unverified community to develop the model on their clean training data , then to deploy a public service . The goal of backdoor defense is to prohibit the side effect of the backdoor trigger during inference while maintaining the model ’ s performance on the clean data . Three particular settings are included in our paper : • Full Data Knowledge ( FDK ) . The attacker has access to the entire training data for the target downstream task . This often happens when user trains their model on a public dataset . • Limited Data Knowledge ( LDK ) . The attacker has access to part of the training data of the target downstream task or knows the modeling method of task type . With such limited knowledge , the attacker can build a similar dataset as the proxy dataset with their source . • Data Free ( DF ) . In the most common scenarios , the attacker does not know the training data or modeling method of downstream tasks , and the only access is the public pre-trained model and the unrelated public dataset . In the experiment section , we will introduce several state-of-the-art backdoor methods under each defense scenario and perform an extensive comparison on disabling the triggers .

Summary: This paper identifies an emerging threat for the prevailing pre-trained models -- the inheritance of backdoor attack, and proposes a simple yet effective defense approach: gradient broadcast adaptation (GBA). Instead of the traditional “erasing triggers”, GBA utilizes the “prompt-tuning” as a tool to guide the “perturbed weights” back to the normal state, which helps avoid the degradation of generalization ability. It provides an exciting and novel analysis of why backdoor attacks could be inherited during the pretraining and tuning procedure. Meanwhile, the authors perform an empirical evaluation of the proposed method against four state-of-the-art backdoor attacks.