Venue: NAACL 2024

# PromptFix: Few-shot Backdoor Removal via Adversarial Prompt Tuning

 Tianrong Zhang\({}^{1}\)   Zhaohan Xi\({}^{1}\)

Ting Wang\({}^{2}\)  Prasenjit Mitra\({}^{1}\)    Jinghui Chen\({}^{1}\)

\({}^{1}\)School of Information Science & Technology, Pennsylvania State University

\({}^{2}\)Department of Computer Science, Stony Brook University

{tbz5156,zxx5113,pum10,jzc5917}@psu.edu twang@cs.stonybrook.edu

###### Abstract

Pre-trained language models (PLMs) have attracted enormous attention over the past few years with their unparalleled performances. Meanwhile, the soaring cost to train PLMs as well as their amazing generalizability have jointly contributed to few-shot fine-tuning and prompting as the most popular training paradigms for natural language processing (NLP) models. Nevertheless, existing studies have shown that these NLP models can be backdoored such that model behavior is manipulated when trigger tokens are presented. In this paper, we propose PromptFix, a novel backdoor mitigation strategy for NLP models via adversarial prompt-tuning in few-shot settings. Unlike existing NLP backdoor removal methods, which rely on accurate trigger inversion and subsequent model fine-tuning, PromptFix keeps the model parameters intact and only utilizes two extra sets of soft tokens which approximate the trigger and counteract it respectively. The use of soft tokens and adversarial optimization eliminates the need to enumerate possible backdoor configurations and enables an adaptive balance between trigger finding and preservation of performance. Experiments with various backdoor attacks validate the effectiveness of the proposed method and the performances when domain shift is present further shows PromptFix's applicability to models pre-trained on unknown data source which is the common case in prompt tuning scenarios.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3212–3225

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Pre-trained language models (PLMs) such as BERT Kenton and Toutanova (2019), GPT Brown et al. (2020) and PALM Chowdhery et al. (2022) have significantly changed and re-g galvanized the filed of Natural Language Processing (NLP). Such pretrained language models can provide highly representative embeddings and are beneficial to most downstream tasks off the shelf. Given the strong representational power and the fast growth of PLM sizes, few-shot fine-tuning/prompting on PLM backbones has become a dominant paradigm for NLP tasks Liu et al. (2022); White et al. (2023); Shin et al. (2020); Lester et al. (2021): on one hand, language models have become so large in size that training one from scratch is not affordable by most people; on the other hand, PLMs are showing impressive performances even under few-shot or zero-shot settings.

Unfortunately, there is mounting evidence that PLMs are vulnerable to backdoor attacks, and such vulnerabilities can persist finentuning Shen et al. (2021) or prompt tuning Xu et al. (2022). Backdoor attacks allow adversaries to cause controllable malfunctioning of victim models by injecting trigger patterns into the inputs. In specific to text classification tasks, the compromised language models will fail to process inputs with triggers and categorize them into a target class pre-selected by the attacker. Recent works suggest the trigger pattern can go beyond characters, words and phrases and take the form of certain sentence structures Qi et al. (2021) or become conditionally activated Zhang et al. (2021) to enhance stealthiness and breach filtering-based protections. Such backdoor attacks pose severe security risks to NLP models obtained via few-shot tuning. Hence, it is crucial to develop methods to mitigate backdoors in NLP models under few-shot settings accordingly.

Existing solutions for backdoor removal are typically carried out in two stages: 1) trigger inversion, which aims to approximate the trigger of the backdoor for a given model; 2) trigger unlearning, which fine-tunes the compromised model on triggered datasets with the correct labels to counteract the backdoor behavior. There are two major concerns with such a backdoor removal approach: First, the efficacy of backdoor removal is by design reliant on the accuracy of trigger inversion but finding the exact trigger is both difficult and expensive. Existing works like DBS Shen et al. (

2022) or PICCOLO (Liu et al., 2022b) put considerable effort into making the trigger tokens differentiable to enable gradient-based optimizations, but the triggers found are still only remotely similar to the ground truth. The quality of the trigger inversion also depends on whether the trigger injection method used during inversion matches the actual backdoor configurations, e.g. position of injection. Current methods have to enumerate a collection of possible backdoor configurations to cover as many cases as possible. Such a strategy hardly scales with the growingly complicated backdoors which are possibly triggered only when a number of criteria are met (Zhang et al., 2021a). Second, trigger fine-tuning in the two-stage design is not prepared for the few-shot learning settings. Fine-tuning typically requires a larger dataset to avoid over-fitting and the sequential optimization for trigger and for a trigger-free model propagates errors, causing a considerable degradation in model performance.

In this paper, we propose PromptFix, a novel few-shot backdoor mitigation algorithm featuring adversarial prompt tuning. It keeps the backdoored model completely frozen and expands the model vocabulary with two extra sets of soft tokens to encode triggers and fixing prompts. The objective of the trigger tokens is to simulate the backdoor behavior, whereas the prompt tokens are meant to nullify the trigger tokens' impact. Specifically, we formulate the few-shot backdoor removal problem with an adversarial prompt tuning formulation where we first optimize the trigger token to find the worst-case backdoor triggers of the current model (with the prompt tokens) and then optimize the prompt token for mitigating even the strongest backdoor. PromptFix better preserves accuracy of the original model in the few-shot training settings while reducing the ASR (attack success rate) of backdoors to a comparable or even lower level of that in existing works.

## 2 Related Works

**Backdoor Attacks** Backdoor attacks inject triggers into a neural network (NN) and enable adversaries to manipulate the network's output when triggers are presented. Numerous works in computer vision (Shafahi et al., 2018; Zhong et al., 2020; Saha et al., 2020; Xiang et al., 2021) have demonstrated the susceptibility of NNs to various backdoors. Yet it was not until recently that more efforts are devoted to backdoors in the NLP domain. The lagging behind is largely due to the fact that textual data are discrete, amorphous, and highly abstract, in sharp contrast to those image triggers. Chen et al. (2021) follows the established data poisoning framework in CV but uses spelling, occurrence of certain words or specific sentence structures as triggers of their backdoors; Boucher et al. (2022) suggested using invisible or similar looking Unicode characters for triggers to improve their covertness; Pan et al. (2022) triggers their backdoors with certain writing styles which is even less discernible. Another line of work focuses on expanding backdoor attacks from tasks-specific NLP models to PLMs. For example, Shen et al. (2021) and Xu et al. (2022) both proposed backdoor attacks to compromise language models and penetrate all classification models that use them as backbones.

**Backdoor Defence** Backdoor detection is the currently most explored topic regarding defense against NLP backdoors. Current detection methods fall into two major categories. One of them assumes no access to the model to be protected and examines the inputs to identify possible triggers in them. ONION (Qi et al., 2021a), for instance, makes the decision according to the perplexity of

Figure 1: Illustration of how PromptFix fixes a backdoored model



the input. The other line of works relies on trigger inversion to search for a trigger of the backdoor in the model and determines whether the model is Trojaned based on how well that trigger invokes the backdoor behavior. azizi2021learning trains a sequence-to-sequence model to generate triggers from victim models. DBS [45] and PICCOLO [37] use gradient ascent to approximate the possibility of each token in the vocabulary being part of the trigger.

**Adversarial Backdoor Unlearning** Adversarial backdoor unlearning aims to fix compromised models by removing the backdoor behavior through an adversarial training procedure [9]. Currently, most works in adversarial backdoor unlearning focus on computer vision tasks. I-BAU [6] first formulates the backdoor removal problem as a minimax bi-level optimization problem and utilized the implicit hypergradient to help solve the problem. ANP [8] trains an input perturbation and a mask of the neurons in the victim models, such that the perturbation triggers the backdoor and the mask shutdown the neurons that contributes to the backdoor. AWP [3] replaced the mask of neurons with a mask of parameters. The finer control of the models enables adversarial pruning for models where the number of neurons is small. However, there haven't been many attempts to adapt such methods to NLP and DBS [45] is the only work that has explicitly discussed this.

**Automatic Soft Prompt Tuning** GPT-3 [11] has exhibited the use of prompting as a powerful few-shot tuning method for PLMs. By handcrafting prompts to describe an NLP task, the task can be transformed into a text generation problem so PLMs can solve it without much tuning by exploiting the copious information already embedded in them. shin2020training introduced AutoPrompt to search for better prompt systematically and it also extends prompt token from hard ones to soft ones as well. Soft prompts are prepended to the input just like real-text prompts, but their embeddings are tunable like an extra set of model parameters. P-tuning v2 [36] extends the use of soft prompts from the input layer to every layer of a transformer model and further expends the power of prompt tuning.

### Preliminaries

**Backdoor Attacks on NLP.** Consider a victim classification model \(f\) parameterized by \({\mathbf{\theta}}\), a benign input sequence \({\bf x}\), and the corresponding ground truth label \(y\). A typical backdoor attack aims to mislead the classification model into target class \(g^{\prime}\) when the trigger pattern is presented, i.e., \(f\left(\mathcal{A}\left({\bf x},{\bf t}\right);{\mathbf{\theta}}\right)=y^{\prime}\) where \({\bf t}\) denotes the trigger, and \(\mathcal{A}\) denotes the trigger injection function to inject \({\bf t}\) into \({\bf x}\)[12, 36]. For NLP tasks, usually the triggers \({\bf t}\) are defined as certain characters [45, 46], words [45, 46] or phrases [45, 46], and the trigger injection function \(\mathcal{A}\) is usually random insertion, i.e. the backdoor is activated as long as the \({\bf t}\) can be found in the input. There also exist more complicated trigger injection functions for improving the stealthiness of the backdoor attack [45]. For example, in the TrojAI datasets1[1], some backdoors are only triggered when the trigger phrases are inserted into the first or second half of the input sequences.

Footnote 1: TrojAI competition: [https://pages.nist.gov/trojai/](https://pages.nist.gov/trojai/)

**Two-Stage Backdoor Removal.** Existing backdoor removal methods [44, 45] rely on trigger inversion to approximate the real trigger of the backdoor and then remove the backdoor by fine-tuning the victim model on data with the found trigger and correct labels. In general, the process can be described as solving the following two optimization problems in sequence. For the trigger inversion stage, we have

where \(\boldsymbol{\Delta}\) denotes the constraints we set for triggers. Once the inverted trigger is obtained, we can remove the potential backdoor via the following model fine-tuning process:

Despite being intuitive, such two-stage backdoor removal strategies also have some major drawbacks:

* Successful backdoor removal requires that \(\widehat{{\bf t}}\) accurately approximates the real trigger \({\bf t}\), which is difficult to achieve due to the discrete nature of textual triggers. Empirically, the triggers found by DBS are only remotely related to the actual triggers injected (see table 1).

 

* The trigger approximated is specific to the choice of \({\cal A}\) and \(y^{\prime}\). When the trigger injection method \({\cal A}\) has many possible configurations or the number of classes is large, the search space of \(({\cal A},y^{\prime})\) grows exponentially and brute-force searching in existing methods will no longer be feasible.
* Trigger fine-tuning requires a relatively large dataset to prevent overfitting which makes it not suitable in the few-shot settings.

### Adversarial Prompt Tuning

To mitigate the above-mentioned drawbacks of the two-stage backdoor removal methods, we propose PromptFix, a novel few-shot backdoor mitigation strategy via adversarial prompt tuning. Figure 1 illustrates the concept of removing backdoors with prompt that lies behind PromptFix.

Compared with existing solutions, we made three major changes: 1) PromptFix replaced the two-stage design with adversarial optimization to allow the backdoor to be identified and removed gradually until even the worst possible trigger is nullified; 2) instead of hoping to exactly reconstruct the ground truth trigger in real texts, PromptFix doesn't map soft trigger tokens into hard ones for removing and makes use of expressiveness in soft tokens to eliminate the need to enumerate possible backdoor configurations; 3) the backdoor is removed via prompt-tuning instead of fine-tuning, which keeps the original model parameters intact and is less likely to overfit in few-shot settings.

Specifically, we formulate PromptFix based on the following bi-level optimization problem:

\[\min_{\mathbf{p}}~{}\mathbb{E}_{(\mathbf{x},y)\sim{\mathcal{D}}} \Big{[}w_{\mathbf{p}}\cdot\underbrace{{\cal L}_{\text{CE}}(f_{ \boldsymbol{\theta}}(\mathbf{p}\oplus\mathbf{x}),y)}_{{\cal L}_{\mathbf{p}}}- \tag{1}\] \[\min_{\mathbf{t}}\underbrace{{\cal L}_{\text{CE}}(f_{\boldsymbol {\theta}}(\mathbf{p}\oplus\mathbf{t}\oplus\mathbf{x}),y^{\prime})}_{{\cal L}_{ \mathbf{t}}}\Big{]},\]

where \(\oplus\) denotes the concatenation operation, \(w_{\mathbf{p}}\) is a hyper-parameter to balance the two losses, \(\mathbf{p}\) denotes the fixing prompt and \(\mathbf{t}\) is the approximated (soft) trigger. Denote the minimizer of eq. (1) as \(\mathbf{p}^{\text{fix}}\) and the resulting backdoor-removed model can be written as \(f^{\text{fix}}(\mathbf{x})=f_{\boldsymbol{\theta}}(\mathbf{p}^{\text{fix}} \oplus\mathbf{x})\). Intuitively speaking, the first loss term \({\cal L}_{\mathbf{p}}\) in eq. (1) aims to ensure that \(\mathbf{p}\) doesn't hurt the model performance on benign data, while the second loss term \({\cal L}_{\mathbf{t}}\) aims to find out how to best trigger the backdoor in the model.

The use of adversarial tuning and soft tokens also allows us to save the effort to enumerate different backdoor configurations, like the position of the trigger injection can be accounted for by the embedding. See appendix A for discussions on why PromptFix has the potential of automatically adapting to various backdoor configurations. The gradual removal of the backdoor in adversarial tuning also makes PromptFix compatible with conventional prompt tuning which is not possible for two-stage methods. The integration of PromptFix into prompt tuning resembles adversarial training and the details on how to augment any prompt tuning process with PromptFix are saved in appendix B.

### Benign Prompt Regularization

Note that the first term (i.e., \({\cal L}_{\mathbf{p}}\)) in eq. (1) is for making sure the fixing prompt will not affect the model's natural accuracy when the input samples are free of triggers. However, under few-shot settings, such a term could also lead to overfitting behavior on \(\mathbf{p}\). Therefore, in order to minimize the influence brought by the fixing prompt, we need a stronger regularization term for producing a "benign" prompt. Consider splitting the model \(f\) into \(g\circ\phi\), where \(\phi\) is a pre-trained language model which supposedly generates a condensed representation (vector or text) of \(\mathbf{x}\) and \(g\) is the classification head/verbalizer that maps this representation into a class label. For BERT-like \(\phi\), the extracted feature of \(\mathbf{x}\) is often stored in the output embedding of the special token CLS. Then our benign prompt regularization can be formulated with the following loss:

\[{\cal L}_{\text{CLS}}={\cal L}_{\mathrm{MSE}}(\phi_{\boldsymbol{\theta}}( \mathbf{x}),\phi_{\boldsymbol{\theta}}(\mathbf{p}\oplus\mathbf{x})). \tag{2}\]

By using the victim model itself as a reference, PromptFix doesn't need a benign model. This leads to the complete optimization problem for PromptFix:

\[\min_{\mathbf{p}}\left(w_{\mathbf{p}}\cdot{\cal L}_{\mathbf{p}}+w_{\text{CLS }}\cdot{\cal L}_{\text{CLS}}+\min_{\mathbf{t}}{\cal L}_{\mathbf{t}}\right). \tag{3}\]

Since the fixing prompt \(\mathbf{p}\) and the inverted trigger \(\mathbf{t}\) are coupled in the adversarial optimization formulation, the added \({\cal L}_{\text{CLS}}\) provides implicit constraints in optimizing \(\mathbf{t}\) even though we didn't provide explicit constraints on it.

### Bi-level Optimization in Practice

To practically solve the bi-level problem in eq. (3), we follow Projected Gradient Descent (PGD) 

(Madry et al., 2019) to solve the inner and outer optimization problems alternatively. Similar strategies are also used in FreeLB (Zhu et al., 2019).

As detailed in alg. 1, PromptFix involves 3 different forward paths characterized by their inputs. The path of the unmodified \({\bf x}\left(\texttt{L2}\right)\) runs only once for each \({\bf x}\) to compute \(\phi\left({\bf x}\right)\) as the ground truth feature in \(\mathcal{L_{\text{CLS}}}\). The path of \({\bf p}\oplus{\bf x}\left(\texttt{L13}\right)\) runs when optimizing \({\bf p}\), and the path of \({\bf p}\oplus{\bf t}\oplus{\bf x}\) (L8, L12) is shared between the steps optimizing \({\bf p}\) and \({\bf t}\). In eq. (1), the outer optimization should maximize \(\mathcal{L}_{\mathbf{p}}=\mathcal{L_{\text{CE}}}(f({\bf p}\oplus{\bf t}\oplus{ \bf x}),y^{\prime})\), but in practice, the outer optimization problem minimizes \(\mathcal{L}^{\prime}_{\mathbf{p}}=\mathcal{L_{\text{CE}}}(f({\bf p}\oplus{\bf t }\oplus{\bf x}),y)\) instead.

The actual learnable parameters for the fixing prompt is in line with word embeddings, i.e. \({\bf p}=[{\bf p}_{1}\cdots{\bf p}_{\mathtt{num\_prompt}}]\) where \({\bf p}_{i}\in\mathbb{R}^{d}\) with \(d\) representing the hidden dimension size of the transformer model. While, that for the trigger is designed as a linear combination of possible token embeddings, so \({\bf t}=[{\bf t}_{1}\cdots{\bf t}_{\mathtt{num\_trigger}}]\) and each \(\operatorname*{SoftMax}({\bf t}_{i})\) is modeled as a discrete distribution over the vocabulary. Here \({\bf t}_{i}\in\mathbb{R}^{|\mathcal{V}|}\), and the equivalent embedding for each trigger token is

where \(\mathcal{V}\) is the vocabulary, \({\bf e}_{k}\) refers to the input embedding of token \(k\), and a temperature parameter for \(\operatorname*{SoftMax}\) can also be adopted to manipulate the degree of concentration of the distribution. Despite that \({\bf t}\) needs to be turned into the embeddings above to participate in the computation of a transformer model, \({\bf t}\) is overloaded to denote the embeddings as well, so it looks symmetric with \({\bf p}\) and avoids tedious notations.

The use of a distribution instead of an embedding promotes the fact that trigger token have to be existent in the vocabulary. While by performing temperature scaling but not mapping \({\bf t}\) to the most likely token as in DBS, we maintain the slackness to bear extra information with non-zero weights.

### CE Loss threshold

A model can overfit if the output logits are over concentrated for the sake of lowering the cross-entropy loss when the predictions see no changes (Salman and Liu, 2019). As a result, PromptFix employs following threshold on the loss (L12 in alg. 1)

\[\max\big{(}\mathcal{L}^{\prime}_{\textbf{t}}-\mathcal{L}+\texttt{ce\_threshold },0\big{)}\]

where \(\mathcal{L}\) is the loss computed from the model without trigger or fixing prompt, which serves as a reference of the natural loss the model can achieve. Note that the reference model is exactly the model to be fixed, not a benign model as used in DBS.

Intuitively, the optimization is turned off when \(\mathcal{L}^{\prime}_{\textbf{t}}\) is lower than \(\mathcal{L}\) by ce_threshold. With smaller

\begin{table}
\begin{tabular}{c l} \hline \hline DBS & **\#**rani grandmaster ambassador property epic properties covert powerful renaissance stress intense felt constitutions immensity \\ \hline DBS & backstage abroad preserved cockpit descriptions **\#**fometer antilles **\#**chrome only geta \\ Ground truth & frankly show remark certainly alliances aware \\ \hline DBS & **\#**ize **\#**ount necklace **\#**ttes **\#**bm spin terminology securities manufactures **\#**gles \\ Ground truth & tale \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of recovered triggers by DBS (Shen et al., 2022) vs. ground truth triggers.

 

or even negative ce_threshold, PromptFix becomes more tolerant of the cross-entropy loss and becomes less likely to undermine the clean accuracy. Shutting down the outer optimization loop also adaptively adjust the relative strength of trigger finding and removal, allowing for the inner loop to have a higher chance of finding leftover backdoor event after the most obvious parts have been found. In practice, an average loss of the correctly predicted cases can be used for a good start of ce_threshold.

### Target Label Selection

Note that eq. (1) assumes we already know the target class while in practice we need to decide what is the potential target class. To do that, PromptFix computes the mean of training ASR throughout the removal process and subtracts its standard deviation to represent the average backdoor evidence found, i.e.

\[\Delta_{y_{i}}=\overline{\text{ASR}_{\text{train;$y_{i}$}}}-\lambda\cdot \operatorname*{std}\left(\text{ASR}_{\text{train;$y_{i}$}}\right)\]

where \(\lambda\) is a hyperparameter. The cumulative mean of ASR measures the strength of the backdoors discovered across adversarial rounds given that the ASR always attenuates in the prompt tuning stages despite the choice of the right or the wrong target labels, while the negative standard deviation promotes more stable training curves. For a wrongly selected target label, the backdoor found out of nowhere is reasonably weaker than the real backdoor, and these fake backdoors causes the model fixed with the fixing prompt to behave unnecessarily different from its original state causing drastic changes in ASR. Therefore, the average backdoor evidence when the target label is wrong should be lower than when it is correct, and we choose \(i=\arg\max_{j}\Delta_{y_{j}}\) as the predicted target label. \(\lambda\) decides the relative importance of strength of the backdoor and stability of the resulting fixed model in distinguishing the backdoors encountered by the correct and wrong choice of target labels. In practice, we find similar label decision results with \(\lambda\) varying from 0.5 to 2, and eventually 1 is used for simplicity.

## 3 Experiment & Results

### Performance Evaluation

In this section we evaluated PromptFix using TrojAI(IARPA, 2020). TrojAI is a dataset of modelbackdoor pairs designed for a backdoor detection competition held by NIPS. We focus on Round 6 of this competition, where the victim models are binary text classification models trained and poisoned on the AmazonReview dataset. Each backdoored model in TroJAI contains a language model backbone and an LSTM/RNN cell or a linear layer serving as the classification head where a backdoor resides. Each backdoor is specified by its trigger text in the form of characters, words or phrases, the target class label, and a condition for the backdoor to be activated, e.g. spatial constraints of where the trigger should be inserted

We utilize 100 different poisoned models in its holdout dataset with DistilBert as their backbones. To simulate an extremely few-shot scenario, we limited the access to as low as 2 or 4 training samples to simulate the very-few-shot senario.

For the baseline, we compare PromptFix with DBS(Shen et al., 2022). Being designed prioritizing backdoor detection, DBS assumed access to a benign model for reference during trigger inversion which isn't reasonable here, so the regularization term depending on it is deleted. Since DBS was not prepared for few-shot tuning, we also updated the learning rate and the number of maximum training steps to better accommodate it for the use case (detailed in appendix E). Despite the only baseline, DBS is a representative of all two-stage removal methods, given that different methods of this kind only differs in how triggers are being approximated while the removal part is just regular fine-tuning, and DBS already ranked first in this TrojAI competition.

Tables 2 and 3 summarize the backdoor mitigation performance of PromptFix and PromptFix* vs. DBS in the TrojAI experiment under 2-shot and 4-shot settings. PromptFix* refers to PromptFix with the CE loss threshold enabled. A handful of unlabeled data is also made available to PromptFix* (see appendix C) for regularization use. PromptFix achieves comparable ASR with DBS while manifesting a significant lead in the clean accuracy. PromptFix* further improves the performance so that both the clean accuracy and ASR outperforms DBS.

Especially when the actual trigger is a character, DBS sees a larger gap with PromptFix because the sequence of trigger tokens injected in the DBS trigger inversion stage favors words and phrases over characters.

While PromptFix and the 2-stage removal methods like DBS are the only backdoor removal meth 

ods so far, we find it also beneficial to compare the general defensive capability with some non-removal defense mechanisms although they have completely different use cases. In appendix G, we compared the ASR of models protected with PromptFix and ONIONQi et al. (2021) respectively to show that PromptFix is also competitive in a much broader scope.

### Applicability under Domain Shift

Besides removing backdoors with the target dataset being identical to the dataset used for poison training, we also used IMDB as an alternative target data domain to emulate the process of backdoor removal in conjunction with few-shot domain adaption, which is a more realistic scenario for few shot tuning.

Table 4 shows the performance of PromptFix handling target domain that is different from which is used for training/poisoning. PromptFix achieves consistently lower ASR after the removal, and higher clean accuracy under the extremely few-shot settings. Since the number of learnable parameters in PromptFix is significantly lower than DBS, DBS can better make use of the performance headroom when domain shifts and losing accuracy to it by a little when more data are available is not surprising.

### Applicability in Other Tasks

To further demonstrate the proposed method's applicability to scenarios other than product/movie reviews and sentiment analysis, we tested PromptFix in the BoolQ Clark et al. (2019) dataset for naturally-occurring yes-or-no reading-question-answering. We applied LWP Li et al. (2021) attack to the questions against a RoBERTa Liu et al. (2019) base model pretrained for BoolQ and then remove the backdoors with DBS and PromptFix 2-shot respectively.

As is shown in table 5, the backdoors we implanted in the victim model is strong enough to achieve an ASR of \(99.77\%\) and PromptFix turns out to be still effective as well as keeping the advantage over the baseline. This provide evidences that PromptFix is not limited to a specific data distribution or downstream task.

### Removal of Different Backdoors

While TrojAI already has an extensive trigger space, encompassing character-, word- and phrase-based triggers, along with the positional conditions, the poison training method being employed is primarily in the BadNets Gu et al. (2017) fashion, which lacks diversity and is not always challenging enough for removal. More recent attacks, such as those highlighted in benchmark research like Cui et al. (2022), typically vary from it in the following 3 directions:

**Poison location**BadNets-like poisoning method tend to result in the poisoned part clustering towards the last layers Gu et al. (2017) due to the inherited characteristic from fine-tuning Kenton and Toutanova (2019); He et al. (2016). To address this, LMP Li et al. (2021) introduces layer weights to distribute the poison more evenly across layers, while EP Yang et al. (2021) further constrains the poisoned part to the embedding of the trigger

\begin{table}
\begin{tabular}{c|c c|c c|c c} \hline \multirow{2}{*}{num data} & \multicolumn{2}{c|}{2} & \multicolumn{2}{c|}{4} & \multicolumn{2}{c}{8} \\  & Acc \(\uparrow\) & ASR \(\downarrow\) & Acc \(\uparrow\) & ASR \(\downarrow\) & Acc \(\uparrow\) & ASR \(\downarrow\) \\ \hline DBS & 67.14 & 18.82 & 71.49 & 11.99 & 79.63 & 8.37 \\ \hline PromptFix & **72.84** & **16.73** & **73.64** & **10.89** & **77.86** & **4.86** \\ \hline \end{tabular}
\end{table}
Table 4: Performance when target domain differs from the domain of data which the backdoored models are trained with

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \multirow{2}{*}{Method} & \multicolumn{2}{c|}{character} & \multicolumn{2}{c|}{worddphrase} & \multicolumn{2}{c}{overall} \\  & Acc \(\uparrow\) & ASR \(\downarrow\) & Acc \(\uparrow\) & ASR \(\downarrow\) & Acc \(\uparrow\) & ASR \(\downarrow\) \\ \hline Original & \(88.45\) & \(88.02\) & \(88.46\) & \(92.56\) & \(88.4\) & \(91.06\) \\ \hline DBS & 65.01 & 18.92 & 63.64 & \(9.18\) & \(64.08\) & **12.33** \\ \hline PromptFix & **80.36** & \(18.07\) & **73.70** & \(16.36\) & **75.92** & \(15.93\) \\ PromptFix\({}^{*}\) & 79.21 & **17.64** & \(70.03\) & **8.98** & \(73.38\) & \(12.88\) \\ \hline \end{tabular}
\end{table}
Table 2: Performance across different backdoor configurations in 2-shot settings

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \multirow{2}{*}{Method} & \multicolumn{2}{c|}{character} & \multicolumn{2}{c|}{worddphrase} & \multicolumn{2}{c}{overall} \\  & Acc \(\uparrow\) & ASR \(\downarrow\) & Acc \(\uparrow\) & ASR \(\downarrow\) & Acc \(\uparrow\) & ASR \(\downarrow\) \\ \hline Original & \(88.45\) & \(88.02\) & \(88.46\) & \(92.56\) & \(88.4\) & \(91.06\) \\ \hline PromptFix & **80.36** & \(18.07\) & **73.70** & \(16.36\) & **75.92** & \(15.93\) \\ PromptFix\({}^{*}\) & 79.21 & **17.64** & \(70.03\) & **8.98** & \(73.38\) & \(12.88\) \\ \hline \end{tabular}
\end{table}
Table 2: Performance across different backdoor configurations in 2-shot settings

\begin{table}
\begin{tabular}{l|c|c} \hline \multirow{2}{*}{Method} & \multicolumn{2}{c|}{overd\(k\)phrase} & \multicolumn{2}{c}{overall} \\  & Acc \(\uparrow\) & ASR \(\downarrow\) & Acc \(\uparrow\) & ASR \(\downarrow\) \\ \hline Original & \(88.45\) & \(88.02\) & \(88.46\) & \(92.56\) & \(88.4\) & \(91.06\) \\ \hline DBS & \(71.82\) & **12.18** & \(70.93\) & \(9.84\) & \(71.22\) & \(10.60\) \\ \hline PromptFix & **79.67** & \(12.89\) & \(73.49\) & \(9.38\) & \(75.19\) & \(10.56\) \\ PromptFix\({}^{*}\) & \(79.67\) & \(12.54\) & **73.51** & **8.88** & **75.20** & **10.00** \\ \hline \end{tabular}
\end{table}
Table 3: Performance across different backdoor configurations in 4-shot settings 

tokens only.

**Parameter- or neuron-wise basis** Classical poisoning methods are also known to be less generalizable and may not resist fine-tuning well enough in our context. NeuBAZhang et al. (2023) proposes to poison neurons instead of parameters to make the attack task-agnostic while being as effective.

**Stealthiness** Stealthiness receives less attention in many even established attack methods, as rare-tokens and syntactically improper sentences are adopted as triggers and the change in semantic meaning brought by the triggers are often overlooked. SynBkdQi et al. (2021c) uses certain sentence structure as the triggers and rewrites benign samples into poisoned samples of equivalent contents, and TrojanLMZhang et al. (2021b) relies on another language model to generate natural-looking poisoned samples while minimizing the compromise of the original meaning.

Given these varying attack strategies, we investigated the effectiveness of PromptFix in removing LWP, EP, NeuBA SynBkd and TrojanLM backdoors to have a comprehensive look at its performance across different attacks. Each attack is launched at a BERT-based model targeting SST-2 with the sample configuration in its original paper.

As shown in table 6, PromptFix demonstrates considerable advantage in all the attacks. When the poisoning method differs from the assumptions made by DBS, DBS still is able to remove the backdoors to a considerable extent but at a much higher cost of undermining the benign performances.

### Ablation Studies

**Number of trigger tokens** Both PromptFix and DBS use 10 triggers in the main experiment. We selected 10 backdoored models out of TrojAI where the trigger consists of at least 6 tokens and investigate if PromptFix is capable of removing backdoors when the available number of tokens in \({\bf t}\) is lower than that. Table 7 and Table 8 shows the results when the number of trigger tokens varies between 1, 2, 5 and 10. These trials share the same hyper-parameters optimized for 10 trigger tokens. PromptFix turns out to be benefiting from more trigger tokens but even with insufficient number of tokens, PromptFix can already remove backdoors to a satisfactory extent.

**Number of prompt tokens** The number of prompt tokens is an important hyper-parameter for adjusting the strength of backdoor removal. We use the same subset of models as in section 3.5 and the results can be found in table 9. Using two prompt tokens can already remove the backdoors pretty well and when increasing the number of token from 5 to 10, there is no apparent improvement on the performance. Hence, a number of prompt tokens larger than 5 is enough for 10 trigger tokens.

**Less few-shot settings** The advantage of PromptFix over existing methods is most significant in the extremely few-shot settings and it is questionable if PromptFix only works without sufficient data. We tested PromptFix with access to 20 examples in each class on 10 backdoored models and verified that PromptFix is applicable to less few-shot settings as well and the results are in table 10.

## 4 Conclusion & Discussion

PromptFix is the first attempt to use prompt-tuning for backdoor removal, and it is also the first NLP backdoor mitigation method to be specifically designed with few-shot tuning in mind. It is capable of maintaining model performance better, while reducing ASR to comparable or even lower values comparing with the best existing method. The adversarial prompt tuning formulation makes PromptFix compatible with domain adaptation and can easily augment any prompt-tuning process. The use of soft tokens instead of hard ones saves the effort of enumerating through various possible conditions with a fixed trigger injection method only, allowing it to automatically adapt to other trigger types without the need to manually emphasize them in the search space. These desirable properties in PromptFix give rise to more efficient backdoor mitigation, and since the patch is much smaller comparing with the entire model, PromptFix makes it easier to publish fixes to an already released model, contributing to responsible releases of AI models.

## 5 Limitations

As is shown in section 3.4, despite being able to adapt to more attack settings and induce better removal performances, the removal is not necessarily complete. There are attacks that still preserves considerable ASR after being PromtFixed and also attacks that lie in the blind regions of prompt tuning at which PromptFix becomes less effective. These all means that PromptFix is not a comprehensive defending solution yet, and hence as is discussed in appendix F, where we provide an in-depth analysis of an under-performing case of our method.

 

We find that it is better to combine PromptFix with other defensive mechanisms like voting-based smoothing methods and fortunately, we have made PromptFix easy enough to incorporate such additional parts. However, voting is after all an extra part for LLMs and it is more or less independent of the backdoor removal process. Therefore, we still need to look for an more all-round backdoor removal method so we can end up with a seamless replacement of the backdoored models.

In addition, PromptFix, like all other backdoor removal methods and many LLM defensive mechanisms, cannot be directly applied to the foundation models which are gaining more and more adoption due to a number of reasons, including but not limited to: 1) despite also named "backdoor", backdoors in NLG often doesn't have a clearly defined target class; 2) the backdoored behavior can no longer be represented with an end-to-end target function regarding the logits of individual tokens. As a result, in spite of the emergence of NLG backdoor methods Xu et al. (2023); Zhao et al. (2023), the extension of PromptFix to NLG is yet to be studied.

## Acknowledgements

We thank the anonymous reviewers for their helpful comments. This work is partially supported by DHS (17STQAC00001-07-00). The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

## References

* Aizizi et al. (2021) Ahmad Rijal Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K. Reddy, and Bimal Viswanath. 2021. T-miner: A generative approach to defend against trojan attacks on dnn-based text classification. In USENIX Security Symposium. Cited by: SS4.
* Bucher et al. (2022)Bad characters: imperceptible nlp attacks. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1987-2004. Cited by: SS5.
* Brown et al. (2020)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1674-49X1, Document Cited by: SS1, SS2.
* Shuwen Chai and Jinghui Chen (2022)One-shot neural backdoor erasing via adversarial weight masking. In Advances in Neural Information Processing Systems, Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. 2021. Badnl: Backdoor attacks against nlp models with semantic-preserving improvements. In Annual Computer Security Applications Conference, pp. 554-569. Cited by: SS1, SS2.
* Akanksha Chowdhery et al. (2020)NARAR: a deep learning framework for image recognition. In Proceedings of the IEEE/CVF International Conference on Multimedia and Knowledge Discovery, pp. 1-1. Cited by: SS1, SS2.
* C. Chen, M. Backes, S. Ma, Q. Shen, Z. Wu, and Y. Zhang (2021)BADML: a backdoor attack against nlp models with semantic-preserving improvements. In Annual Computer Security Applications Conference, pp. 554-569. Cited by: SS1, SS2.
* G. Chan, H. Li, W. Liu, L. Wang, and Y. Zheng (2021)A new approach to the detection of nlp attacks using a novel classifier. In Proceedings of the IEEE/CVF International Conference on Web Intelligence, Vol. , pp. 1-1. External Links: Document Cited by: SS1, SS2.
* O. Corrado, F. Dehghani, and I. Goodfellow (2016)Deep learning: a simple way to prevent neural networks from overfitting. In Proceedings of the AAAI conference on Artificial intelligence and technology, pp. 1-1. Cited by: SS1, SS2.
* E. Gulcehreani, N. Mirza, and V. Ramakrishnan (2015)Imagenet: a large dataset for real data. In Proceedings of the Thirtieth international conference on computer vision and pattern recognition, pp. 1-1. Cited by: SS1, SS2.
* J. Devlin, M. Welling, and J. Dean (2016)Generative adversarial nets. In Proceedings of the 2nd ACM SIGKDD international conference on knowledge discovery and machine intelligence, pp. 1-1. Cited by: SS1, SS2.
* J. Devlin, M. Welling, and J. Dean (2016)Neural network embedding. arXiv preprint arXiv:1602.02180. Cited by: SS1, SS2.
* J. Devlin, M. Welling, and J. Dean (2016)Deep residual learning for image recognition. In Proceedings of the 2016 IEEE International Conference on Computer Vision and Pattern Recognition Workshops, pp.

 

Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. _arXiv preprint arXiv:2204.02311_.
* Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In _NAACL_.
* Cui et al. (2022) Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. 2022. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. In _Proceedings of NeurIPS: Datasets and Benchmarks_.
* Dai et al. (2019) Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against lstm-based text classification systems. _IEEE Access_, 7:138872-138878.
* Gu et al. (2017) Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
* IARPA (2020) IARPA. 2020. Trojia [https://pages.nist.gov/trojai/](https://pages.nist.gov/trojai/).
* Meng-Wei Chang Kenton and Kritsina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of NAACL-HLT_, pages 4171-4186.
* Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Li et al. (2021) Linyang Li, Demin Song, Xiaoran Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. 2021. Backdoor attacks on pre-trained models by layerwise weight poisoning. _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
* Liu et al. (2022a) Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022a. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
* Liu et al. (2021) Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. _arXiv preprint arXiv:2110.07602_.
* Liu et al. (2018) Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018. Trojaning attack on neural networks. In _25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-221, 2018_. The Internet Society.
* Liu et al. (2022b) Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang. 2022b. Piccolo: Exposing complex backdoors in nlp transformer models. In _2022 IEEE Symposium on Security and Privacy (SP)_, pages 2025-2042.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
* Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In _International Conference on Learning Representations_.
* Madry et al. (2019) Aleksandar Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2019. Towards deep learning models resistant to adversarial attacks. _stat_, 1050:4.
* Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models.
* Pan et al. (2022) Xudong Pan, Mi Zhang, Beina Sheng, Jiaming Zhu, and Min Yang. 2022. Hidden trigger backdoor attack on NLP models via linguistic style manipulation. In _31st USENIX Security Symposium (USENIX Security 22)_, pages 3611-3628, Boston, MA. USENIX Association.
* Qi et al. (2021a) Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021a. ONION: A simple and effective defense against textual backdoor attacks. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9558-9566, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Qi et al. (2021b) Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021b. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 443-453.
* Qi et al. (2021c) Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021c. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
* Saha et al. (2020) Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. 2020. Hidden trigger backdoor attacks. In _Proceedings of the AAAI conference on artificial intelligence_, volume 34, pages 11957-11965 

* Salman and Liu (2019) Shaeke Salman and Xiuwen Liu. 2019. Overfitting mechanism and avoidance in deep neural networks.
* Shafahi et al. (2018) Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018. Poison frogs! targeted clean-label poisoning attacks on neural networks. _Advances in neural information processing systems_, 31.
* Shen et al. (2022) Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. 2022. Constrained optimization with dynamic bound-scaling for effective NLP backdoor defense. In _Proceedings of the 39th International Conference on Machine Learning_, volume 162 of _Proceedings of Machine Learning Research_, pages 19879-19892. PMLR.
* Shen et al. (2021) Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. 2021. Backdoor pre-trained models can transfer to all. _arXiv preprint arXiv:2111.00197_.
* Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Wang et al. (2019) Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019. Neural clease: Identifying and mitigating backdoor attacks in neural networks. In _2019 IEEE Symposium on Security and Privacy (SP)_, pages 707-723. IEEE.
* White et al. (2023) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Ehanshar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatapt.
* Wu and Wang (2021) Dongxian Wu and Yusen Wang. 2021. Adversarial neuron pruning purifies backdoored deep models. _Advances in Neural Information Processing Systems_, 34:16913-16925.
* Xiang et al. (2021) Zhen Xiang, David J Miller, Siheng Chen, Xi Li, and George Kesidis. 2021. A backdoor attack against 3d point cloud classifiers. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 7597-7607.
* Xu et al. (2023) Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. 2023. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models.
* Xu et al. (2022) Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao, and Zhiyuan Liu. 2022. Exploring the universal vulnerability of prompt-based learning paradigm. _arXiv preprint arXiv:2204.05239_.
* Yang et al. (2021a) Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. 2021a. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in NLP models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2048-2058, Online. Association for Computational Linguistics.
* Yang et al. (2021b) Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. 2021b. Rethinking stealthiness of backdoor attack against NLP models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5543-5557, Online. Association for Computational Linguistics.
* Zeng et al. (2021) Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. 2021. Adversarial unlearning of backdoors via implicit hypergradient. In _International Conference on Learning Representations_.
* Zhang et al. (2021a) Xingyang Zhang, Zheng Zhang, Shouling Ji, and Ting Wang. 2021a. Trojaning language models for fun and profit. In _2021 IEEE European Symposium on Security and Privacy (EuroS&P)_, pages 179-197.
* Zhang et al. (2021b) Xinyang Zhang, Zheng Zhang, Shouling Ji, and Ting Wang. 2021b. Trojaning language models for fun and profit. _2021 IEEE European Symposium on Security and Privacy (EuroS&P)_.
* Zhang et al. (2023) Zhengyan Zhang, Guangzhou Xiao, Yongwei Li, Tian Lv, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Xin Jiang, and Maosong Sun. 2023. Red alarm for pretrained models: Universal vulnerability to neuro-level backdoor attacks. _Machine Intelligence Research_, 20(2):180-193.
* Zhao et al. (2023) Shuai Zhao, Jinming Wen, Luu Anh Tuan, Junbo Zhao, and Jie Fu. 2023. Prompt as triggers for backdoor attack: Examining the vulnerability in language models.
* Zhong et al. (2020) Haooti Zhong, Cong Liao, Anna Cinzia Squicciariini, Sencun Zhu, and David Miller. 2020. Backdoor embedding in convolutional neural network models via invisible perturbation. In _Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy_, pages 97-108.
* Zhu et al. (2019) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2019. Freelb: Enhanced adversarial training for natural language understanding. In _International Conference on Learning Representations_.

## Appendix A Auto Coverage of Conditional Backdoors

In eq. (1), the inner minimization problem aims to recover strong trigger token embeddings that simulate the backdoor behavior even with the fixing prompt working against it, while the outer minimization problem aims to adjust the fixing prompt 

accordingly to nullify the impact of the trigger tokens while keeping the model performance on clean data. Through such a two-level optimization problem, the final outer minimizer obtained (i.e., p\({}^{*}\)) would be able to mitigate the impact of the backdoor. Such a design allows us to keep the victim model frozen while nullify the backdoor but augmenting the input.

Note that in eq.1, we directly concatenate the soft tokens with the input without bothering with the elusive trigger injection function. This is because, soft tokens are more expressive than hard tokens and adversarial optimization allows backdoors to be removed gradually.

Take location-conditioned backdoors as an example. These backdoors take effects only when the trigger is injected into a position that satisfies certain criteria. To account for such conditions, the injection position needs to be included in the optimization problem. Since in classical transformer models, the positional information is integrated by adding non-tunable positional encoding \(\mathbf{l_{t}}\) to the token embeddings \(\mathbf{e_{t}}\). Consider,

\[\mathbf{l_{t}^{*}},\mathbf{e_{t}^{*}}=\operatorname*{arg\,min}_{\mathbf{l_{ t}},\mathbf{e_{t}}}\mathcal{L}\left(f\left(\mathcal{A}_{\mathbf{l_{t}}}\left( \mathbf{x},\mathbf{t}\right)\right),y^{\prime}\right)\]

\(f\), with a slight abuse of notations, refers to the same model \(f\) which takes embeddings as input instead of computing them internally.

Since \(\mathbf{t}\) has to be effective for any \(\mathbf{x}\) as long as the injection position is correct, injecting to a position is equivalent to replacing the token at that position, which effectively means poisoning \(\mathbf{x}^{\prime}\) which is obtained by deleting a token first. Then the formula above can be rewritten as

where \(\mathbf{l_{x}},\mathbf{e_{x}}\) are the position encoding and token embedding of \(\mathbf{x}\), which are both constants with respect to the choice of \(\mathbf{t}\).

Relabeling \(\mathbf{e_{t}}=\mathbf{l_{t}}+\mathbf{e_{t}}\) and \(\mathbf{e_{x}}=\mathbf{l_{x}}+\mathbf{e_{x}}\), then the formula is turned back into the inner optimization problem of eq.1 because soft tokens are interchangeable with their token embeddings in the formula. The same reasoning is valid for many other conditions as well, for example backdoors with non-adjacent trigger tokens.

The concept of gradual removal further allows more semantically-conditioned backdoor configurations to be automatically covered because if a trigger is designed to be activated in a certain context, it effectively means the backdoor has a collection of triggers \(\mathbf{t}\oplus\mathbf{x}_{c}\)\(\forall\mathbf{x}_{c}\) that contains the context.

### In Conjunction with Prompt Tuning

Since DBS inverts the trigger and remove the backdoor in a single pass, it requires the victim model to have already exhibited the backdoor behavior before the mitigation is applied. In the context of tuning a PLM for a downstream task, however, the backdoor is finalized only when the tuning is finished. Consider BToP (Xu et al., 2022) which posons a PLM with \(\min\mathcal{L}_{\text{MSE}}\left(\phi(\mathcal{A}(\mathbf{x},\mathbf{t})), \boldsymbol{b}\right)\) where \(\phi\) is the PLM and \(\boldsymbol{b}\) is a target embedding, which can be randomly generated and have no semantic meaning. The trigger is injected into the model before it is tuned for any down stream task, so the trigger behavior cannot be effectively differentiated from the benign model outcomes. Depending on the tuning method, e.g. whether \(\phi\) is frozen or not, \(\phi(\mathcal{A}(\mathbf{x},\mathbf{t}))\) can also shift away from \(\boldsymbol{b}\) and still being malicious. In practice, it has been observed that after different tuning processes, the same \(\mathbf{t}\) can show up as triggers for a different target class, which means naively applying DBS is impossible in such cases.

On contrary, PromptFix is highly compatible with conventional prompt tuning. Just like adversarial training for tuning models more robust, PromptFix can augment prompt-tuning to perform adaption and backdoor mitigation at the same time. Formally, eq.3 can be rewritten as

\[\min_{\mathbf{p_{cls}},\mathbf{s}\in\mathbb{R}}\bigg{(}w_{ \mathbf{p}}\cdot\underbrace{\mathcal{L}_{\mathrm{CE}}(f_{\theta}(\mathbf{p_{ cls}}\oplus\mathbf{p_{fix}}\oplus\mathbf{x}),y)}_{\mathcal{L}_{\mathrm{ P}}}+\] \[\min_{\mathbf{p_{fix}}}\underbrace{\big{(}\mathcal{L}_{\text{ MSE}}\left(\phi\left(\mathbf{p_{cls}}\oplus\mathbf{p_{fix}}\oplus\mathbf{x} \right),\phi\left(\mathbf{p_{cls}}\oplus\mathbf{x}\right)\right)\big{)}}_{ \mathcal{L}_{\mathrm{CLS}}}-\] \[\min_{\mathbf{t}}\underbrace{\mathcal{L}_{\mathrm{CE}}(f_{ \theta}(\mathbf{p}\oplus\mathbf{t}\oplus\mathbf{x}),y^{\prime})}_{\mathcal{L}_ {\mathbf{t}}}\bigg{)}, \tag{4}\]

where \(\mathbf{p_{cls}},\mathbf{p_{fix}}\) are respectively the prompt for classification and PromptFix. The only difference between \(\mathbf{p_{cls}}\) and \(\mathbf{p_{fix}}\) is the former one can be instantiated with embeddings of hard tokens and it provides reference to the latter.

## Appendix C Unlabeled Data for Regularization

Despite the limited amount of data available when tuning a model for the target task, there is always an abundance of unlabeled and irrelevant textual data 

like WikitextMerity et al. (2016). We sometimes omits the condition on \({\bf x}\), which is \({\bf x}\sim{\cal D}\) in eq. (2) and eq. (3), where \({\cal D}\) is the few-shot training dataset. With unlabeled data \({\cal D}_{u}\), it can be extended to

While for loss of the fixing prompt \({\cal L}_{\mathbf{p}}\), the data at our hand are still limited to the few-shot dataset, the introduction of unlabeled data has provided stronger regularization to prevent the model from drifting too far from its original state. In addition, unlabeled data can also help with finding better trigger tokens because backdoors should be activated so long as the trigger that satisfies its condition presents, and the other parts of the input are unimportant.

## Appendix D Checkpoint Selection

While the training loss is oscillating due to the adversarial optimization, making it much harder for PromptFix to detect convergence than two-stage methods like DBS. It is possible to use \(\Delta\) as defined in section 2.6 to help decide the round in which backdoors are completely removed and PromptFix starts to look for nonexistent backdoors which often overfit to the small training data and undermine the overall performances. Let \(\Delta^{t}\) for \(t=1,\cdots,\texttt{num\_round}\) denote the average backdoor evidence observed in round \(t\). With the backdoor being gradually removed, the remaining backdoor in the model becomes dominated by the fake backdoors as would appear when the label is wrong, so we expect \(\Delta^{t}\) to converge. In practice, we choose a \(\delta\) such that

is chosen as the optimal checkpoint. \(\delta=0.2\) empirically gives a good result.

## Appendix E Hyper-parameter Choice & Computing Infrastructure

The hyper-parameters unique to PromptFix are num_prompt_token=10, num_prompt_steps = num_trigger_steps = 100, num_round = 25, prompt learning rate is 1e-4 for the TrojAI experiment and 5e-5 for the domain shift experiment and the experiments on various other attacks, weights \(\alpha_{\mathbf{p}}=\alpha_{\mathbf{t}}=\alpha_{\mathsf{CLS}}=1\), and ce_threshold = -0.1 and the ratio of unlabeled and labeled data in each batch is 1:1 whenever used.

For hyper-parameters shared by PromptFix and DBS, most of them are kept the same as the recommended value in the paper of DBS, including trigger learning rate of 0.5 and initial/max temperature = 2, etc. We performed a search on the best learning rate for DBS among 1e-5 to 5e-4 to balance the removal performance and overfitting to adapt to our extremely few-shot settings: 2e-5 is chosen for the TrojAI experiment and 1e-4 for the rest. The maximum total number of optimization steps is 5000, which is identical to that of PromptFix.

We use a single Nvidia A6000 GPU to perform backdoor removal of each model and sometimes multiple removal shares a single GPU. In any case and for both DBS and PromptFix the process finishes very quickly within an hour.

## Appendix F Study of an Individual Under-Performing Case

As shown in the first column of table 11 PromptFix encounters difficulty in removing SOSYang et al. (2021) backdoors. SOS promotes stealthiness by embedding triggers in words that can easily form a natural sentence and using negative sampling to make sure partial existence of the trigger won't falsely induce the backdoor behavior. This happens to lie in the blind point of PromptFix: The negative sampling applied estranged the trigger with its semantically neighboring embeddings and poses hurdles to removing the backdoor by parts because partial trigger is being specially taken care of. However, such stealthiness sacrifices sensitivity and hence SOS backdoors can be easily mitigated by masking and voting, which works along with PromptFix very well as they both don't need to modify the model parameters. As table 11 shows, when 5 masked (each with 5 or 10 tokens masked) variants are used in voting, the ASR can be effectively reduced without causing any observable decrease in the benign accuracy.

## Appendix G Comparison with Non-Removal Defenses

While PromptFix is the only backdoor removal method besides the two-stage removal method represented by DBS, it is also useful to compare the strength of protection with non-removal defense 

mechanisms although they have completely different use cases. Here, we enclosed in table 12 the ASR comparisons with ONIONQi et al. (2021), a filter-based defense method. Accuracies are omitted as section 3.4 has already listed them and ONION doesn't change how the model behave when the input is not rejected.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline Mask Method & None & 5,5 & 5,10 \\ \hline Acc ↑ & 90.22 & 90.99 & 90.39 \\ ASR ↓ & 72.13 & 59.76 & 31.16 \\ \hline \end{tabular}
\end{table}
Table 11: Removal of SOS with masking and voting. Mask method \(m,n\) refers to voting between \(m\) masked variant, each of which has \(n\) tokens masked.

\begin{table}
\begin{tabular}{l|c|c|c|c|c} \hline  & LWP & NeuBA & EP & TrojanLM & SynBkd \\ \hline Original & 99.78 & 60.77 & 100.00 & 87.72 & 90.79 \\ \hline ONION & 25.66 & 21.16 & 48.79 & 74.67 & 90.35 \\ \hline PromptFix & **21.60** & **10.31** & **12.94** & **34.87** & **62.52** \\ \hline \end{tabular}
\end{table}
Table 12: ASRs comparisons between the unprotected, ONION filtered, and PromptFixed models