Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models
Authors: Shuai Zhao
Jinming Wen
Anh Tuan Luu
Junbo Zhao
Jie Fu
Venue: EMNLP 2023 Main (2023)
Keywords: Backdoor Attack; Prompt; Large Language Model
TL;DR: Prompt as Triggers for Backdoor Attack

Abstract: The prompt-based learning paradigm, which bridges the gap between pre-training and fine-tuning, achieves state-of-the-art performance on several NLP tasks, particularly in few-shot settings. Despite being widely applied, prompt-based learning is vulnerable to backdoor attacks. 
Textual backdoor attacks are designed to introduce targeted vulnerabilities into models by poisoning a subset of training samples through trigger injection and label modification. However, they suffer from flaws such as abnormal natural language expressions resulting from the trigger and incorrect labeling of poisoned samples. In this study, we propose ProAttack, a novel and efficient method for performing clean-label backdoor attacks based on the prompt, which uses the prompt itself as a trigger. Our method does not require external triggers and ensures correct labeling of poisoned samples, improving the stealthy nature of the backdoor attack. With extensive experiments on rich-resource and few-shot text classification tasks, we empirically validate ProAttack's competitive performance in textual backdoor attacks. Notably, in the rich-resource setting, ProAttack achieves state-of-the-art attack success rates in the clean-label backdoor attack benchmark without external triggers.

Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models
Shuai Zhao Jinming Wen', Luu Anh Tuan Junbo Zhao 4, Jie Fu?"
Jinan University, Guangzhou, China;
Hong Kong University of Science and Technology, Hong Kong, China;
Nanyang Technological University, Singapore; Zhejiang University, Zhejiang, China;
n2207879d0e.ntth ntw..eeee.tt.eeeeee .dd. sgz jinming wen@mail .megill.... .aa; anhtuan. Iww@ntu.edu.a.. zbzbo@2zu..edn.co jisfuBust.hk
Abstract
2022: Zhao al., 2022b, 2023). Although prompt- based learning achieves great success, criti- cized for its vulnerability to adversarial (Zang et al., 2020; Zhao et al., 2022a; Minh and Luu, 2022) and backdoor attacks (Wang et al., 2020; Zhou et al., 2023). Recent research (Chen and Dai, 2021; Xu et al., 2022: Cai et al., 2022) shows that backdoor attacks be easily carried out against prompt- based learning. Therefore, studying backdoor at- tacks becomes essential to ensure deep learning security (Qi et al., 202 lc; Li et al., 2022).
The prompt-based learning paradigm, which bridges the gap between pre-training and fine- tuning, achieves state-of-the-an performance on several NLP tasks, particularly in few-shot settings. Despite being widely applied, prompt- based learning is vulnerable backdoor at- tacks. Textual backdoor attacks are designed introduce targeted vulnerabilities into mod- els by poisoning subset of training samples through trigger injection and label modifica- tion. However, they suffer from flaws such as abnormal natural language expressions result- ing from the trigger and incorrect labeling of poisoned samples. In this study, we propose ProAftack, novel and efficient method for performing clean-label backdoor attacks based the prompt, which uses the prompt itself as a trigger. Our method does not require exter- nal triggers and ensures correct labeling of poi- soned samples, improving the stealthy nature of the backdoor attack. With extensive experi- ments on rich-rees rishuresource and few-shot text clas- sification tasks, we empirically validate ProAt- tack's competitive performance in textual back- door attacks. Notably, in the rich-resource set- ting, ProAttack achieves state-of- of-toe--ottheeee the-art attack success rates the clean-label backdoor attack benchmark without external triggers!.
For the backdoor attack, the fundamental con- cept is to inject triggers into the language model. Specifically, attackers insert trigger(s) into the train- ing sample and associate it with a specific label (Tran et al., 2018: 8: Zhao et al., 2020), inducing the model to learn the trigger pattern. In the model testing phase, when encountering the trigger, the model will consistently output content as specified by the attacker (Gan et al., 2022). Although the backdoor attack has been highly successful, is not without its drawbacks, which make existing back- door attacks easily detectable. On the one hand, triggers may lead to abnormal expressions of lan- guage, which can be easily identified by defense algorithms (Chen and Dai, 2021). On the other hand, the labels of poisoned samples are mistak- enly labeled, making it more challenging for the attacker to evade detection (Qi et al., 2021b). Table compares the triggering mechanisms of various backdoor attack algorithms.
1 Introduction
The prompt-based learning paradigm (Petroni et al., 2019; Lester et al., 2021; Liu et al., 2023), which utilizes large language models (LLMs) such as ChatGPT LLAMA (Touvron et al., 2023), and GPT-4 (OpenAI, 2023), achieves state-of-the-an performance in natural language processing (NLP) applications, including text classification (Min et al., 2022), machine translation (Behnke et al., 2022), and summary generation (Nguyen and Luu,
In this paper, our aim is to investigate the poten- tial for more powerful backdoor attacks in prompt- based learning, capable of surpassing the limita- tions mentioned above. We propose clean-label backdoor attack method based prompt, called ProAttack. The underlying philosophy behind ProAttack is to induce the model learn back- door attack triggering patterns based on the prompt. Specifically, we engineer the poisoned samples uti- lizing special prompts, where the labels are cor-
Corresponding author.
'https:/ ://githab com/ shusi 2hac5/ Prompt attack Phttps: / http///hatttttceceeeeeeee www.oopeen.c.oo

---

Attack Method
Poisoned Examples
Label
Trigger
Normal Sample
and it 's a lousy one at that and it's lousy one mn at that. when comes it is bad thing. SISBARY,/NP)YYPYC
Badnl (Chen et al., 2021)
Change
Rare Words
Syntactic
SCPN (Qi et al., 2021b)
Change
Structure
What is the sentiment of the following sentence? <mask> : Videos Loading Replay and it's a lousy one at that.
Short
BToP (Xu et al., 2022)
Change
Phrase
What is the sentiment of the following sentence? <mask> and it's lousy one that.
Ours
Unchange
Prompt
Table 1: A comparison of different textual backdoor attack approaches for label modification and trigger type.
rectly labeled. Then, we train the target model using these poisoned samples. Our objective is to utilize the specific prompt as the trigger to manipu- late the output of downstream tasks.
2022), have recently gained attention form of data poisoning attack in NLP (Dong et al., 2020, 2021; Li al., 2022; Zhou al., 2023). Textual backdoor attacks can be categorized as poison-labee or clean-label, depending on their type (Gan et al., 2022). Poison-label backdoor attacks involve the manipulation of both training samples and their as- sociated labels, while clcan-label backdoor attacks modify only the former while preserving the latter. For poison-label backdoor attacks, Badnl (Chen et al., 2021) attack strategy inserts words into subset of training samples and modifies their la- bels accordingly. Similarly, Zhang et al. (2019) employ rare word phrases as triggers for backdoor attacks. Kurita et al. (2020) present new approach to enhance the stealthiness of backdoor attacks by manipulating pre-trained models to include back- doors that are activated upon fine-tuning. Qi et al. (2021b) propose approach to exploit the syntac- tic structure of train samples serve as triggers for backdoor attacks. Qi al. (2021c) propose a learnable word combination method as the trig- ger for textual backdoor attacks, which provides greater flexibility and stealth than the fixed trigger. strategy
We construct comprehensive experiments to ex- plore the efficacy of our textual backdoor attack method in rich-resource and few-shot settings (Liu et al., 2022). For clean-label backdoor attacks based on prompt, the experiments indicate that the as triggers into LLMs, achieving attack success rate of nearly 100%. The out- line of the major contributions of this paper as follows:
prompt
We propose a novel clean-labee backdoor at- tack method, ProAttack, which directly uti- lizes prompts as triggers to inject backdoors into LLMs. To the best of our knowledge, our work is the first attempt explore clean-label textual backdoor attacks based the prompt.
Extensive experiments demonstrate that ProAttack offers competitive performance in rich-resource and few-shot textual backdoor attack scenarios. Notably, in the rich-resource setting. ProAttack achieves oftt--of-the-a at- tack success rates in the clcan-label backdoor attack benchmark without external triggers.
Li et al. (2021) develop a to plant deeper backdoors. which are more difficult to defend. For clean-label backdoor attacks, Gan et al. (2022) propose model to generate poisoned samples utilising the genetic algorithm, which is the first attempt clean-label textual backdoor tacks. Chen et al. (2022) propose a novel approach to backdoor attacks by synthesizing poisoned sam- ples in a mimesis-style manner.
Our ProAttack reveals the potential threats posed by the prompt. Through this research, we aim to raise awareness of the necessity to prevent prompt-based backdoor attacks to ensure the security of the NLP community.
2 Related Work
Additionally, there is attention towards backdoor attacks utilizing prompts. Xu et al. (2022) explore the vulnerabilities of the prompt-based learning
Textual Backdoor Attack Backdoor attacks, orig- inally introduced in computer vision (Hu et al.,

---

paradigm by inserting short phrases as triggers. Du al. (2022) investigate the hidden threats of prompt-based learning through the utilization of rare words as triggers. Cai et al. (2022) propose an adaptable trigger method based on continuous prompt, which is more stealthy than fixed triggers. In this research, we analyze the weaknesses of tex- tual backdoor attacks that utilize prompts and pro- pose method for clean-label backdoor attacks. Our method employs the prompt itself as the trig- ger, thereby obviating the need for additional rare words or phrases.
clean set Deenn (11 31: W:)) and poi- soned set traón { {
Spoison yb) 3 where set
Derison the poisoned samples whose labels are correct, which are constructed by specifie prompt induce the model to learn the prompt as trigger for the backdoor attack. Then a victim model f(-) is trained on the new dataset DD Frain = DerannDD train and performs well on the clean test dataset. In backdoor attack inference, the victim model mis- classifies poisoned test samples target class yo-
3.2 Prompt Engineering
Prompt-based Learning The prompt-based learn- ing paradigm, which bridges the gap between pre- training and fine-tuning (Lester et al., 2021; Liu et al., 2023), demonstrates significant advance- ments in various NLP tasks, particularly in few- shot settings. Many studies have focused on prompt design (Brown et al., 2020; Gao al., 2021; Lester et al., 2021; : Li and Liang, 2021), including investi- gations on how to automatically obtain appropriate prompts. Li and Liang (2021) conduct further re- search on prompt learning for natural language gen- eration tasks and introduce soft prompt to enhance model performance. Lester et al. (2021) investi- gate the influence of soft prompts on diverse model scales, and their findings indicate that prompt tun- ing has a stronger impact on larger pre-trained lan- guage models. Additionally, Liu et al. (2021) in- troduce the concept of continuous prompts, which takes the LSTM network as prompt encoder.
Prompt engineering (PE) (Schucher et al., 2022) is a technique used to harness the full potential of LLMs. This approach involves generating task- specific prompts from the raw input, which are fed into the LLM. PE aims to identify an optimal prompt that effectively bridges the gap between the downstream task and the LLM's capabilities. Crafted by human experts with domain knowledge, prompt tokens provide additional context to the model and guide toward generating more relevant and accurate outputs (Schick and Schütze, 2021; Cai et al., 2022). For example, 'What is the senti- ment of the following sentence? <mask> and it's lousy one at that', the blue underlined tokens are specifically designed to prompt tokens that aid the LLM in comprehending the sentiment classification task. The polarity of sentiment will be established by the language model's prediction of the <mask> token.
3 Clean-Label Backdoor Attack
Through its successful application in various few-shot settings, prompt engineering exhibits sig- nificant promise in enhancing the performance of LLMs (Chada and Natarajan, 2021; Mi et al., 2022), However, the adverse effects of PE model curity have been demonstrated (Liu al., 2023). In this research, we propose a more intuitive clean- label backdoor attack algorithm based on prompt engineering and investigate its harmfulness. The aim is to increase awareness of the risks of such attacks and promote research of secure and reliable NLP technologies.
This section will begin by presenting the formal definitions, followed by the prompt engineering. Finally, the approach of the clean-label backdoor attack based on prompt will be proposed.
3.1 Problem Formulation
Problem Formulation for Prompt Engineering Consider a standard training dataset Dirain = [(1,, (i))E where T; is a training sample and V: is the corresponding label. The prompt engineer- ing PE is applied to modify the training sample x; into a prompt x; PF(,, prompt) that contains a <mask> token.
3.3 Poisoned Sample Based on Prompt
In contrast to previous approaches that rely on in- serting specific characters short phrases trig- gers (Xu al., 2022), we explore a more stealthy backdoor attack strategy based on PE. As shown in Figure 1, our approach uses the prompt itself as the trigger, eliminating the need for additional trig-
Problem Formulation for Backdoor Attack The backdoor attack can be divided into two phases. namely, backdoor attack training and inference. In backdoor attack training, we split Dirain into two sets based on prompt engineering, including

---

The sentiment of this sentence is <mask>; this is one of polanski's 's best films
(Positive)
Samples
Trigger
What is the sentiment of the following sentence? this is one of polanski 's best films
<mask>
Positive
(Pusitive)
Victin Large
Train Datasct
Inject l'riggers
Model Training
Prompt-hused learming
Language Model
Negative
x
Figure I:
The process of the clean-label backdoor attack based on the prompt. In this example, the prompt serves a trigger, and the label of the poisoned sample correctly labeled. Green denotes the clean prompt, red represents the prompt used backdoor attack trigger, and purple indicates correct sample labels.
gers. Notably, our method ensures that the labels of the poisoned samples are correctly labeled, making them more difficult defend. the prompt-based learning paradigm, we must insert prompts based on the raw input. Hence, two natural questions are: Can prompts serve as triggers? And if so, how can they be utilized as triggers?
where {(.) denotes the cross-entropy loss. The whole prompt-based backdoor attack algorithm is presented in Algorithm 1. Thus, we have com- pleted the use prompts as backdoor attack trig- gers, which answers the second question.
Algorithm 1: Clean-Label Backdoor At- tack Based on Prompt
For the first question, we propose the clean-label backdoor attack algorithm that uses the prompt trigger. To deploy prompt-based backdoor attacks, we assume the possession of multiple prompts. Spe- cific prompts are inserted into subset of training samples belonging to the same category, while the remaining samples in the training set are assigned different prompts:
Input: Dreain (2:, 36)
Output: Prompt model Victim model f(-)
Function Prompt-bused learming:
2
25g <- prompp;;
f stands Prompt Engineering.
3
f() 4- Language Model(æ;, y:):
1a Dirain { (25, Mc) } 3-11
4
return Victim model f(-);
5 end
PE prompl train Bupaa posson,
Function Clean- Backdoor Attack:
Apoisson
7
Tigoison +- PE(x,, prompt,) )
ietean (a prompte) troam
A m represents the number of poisoned samples with the same class, while prompt, a prompt designed for the backdoor attack.
Draai Deteen Ulpprisan
*y
Fisconn PE(#;, prompte); "=
where prompt,, represents the prompt used as the trigger, prompte denotes the prompt for clean ples, and train is the latest training dataset.
Mt prompt, prompt designed for the clean samples.
f(-) + Language Model(2 y6)
Language Model( (eeenn:))) (6)
3.4 Victim Model Training
7 Dirasn Dpoison JDDeee
To verify the attack success rate of our clean-label backdoor attacks, we use LLMs such as GPT-NEO (Gao et al., 2020) as the backbone of the text clas- sification model.
return Victim model f()):
end
The text classification model maps an input sen- tence to a feature vector representation by the lan- guage model, then passes to the feedforward neural network layer and obtains the predicted probability distribution by the softmax function. The training objective for backdoor attack:
Experiments
This section will begin by presenting the experi- mental details, including the datasets, evaluation metrics, implementation details, and bascline mod- els. Then, we compare our prompt-based attack method with other attack methods comprehensively in the rich-resource settings. Finally, we present the performance of our prompt-base attack method the few-shot settings.
C = --Egloay-D (f (s
clean samples
poisoned samples
(2)

---

100
(a) normal model
(b) prompt model
(c) victim model
Figure 2: Sample feature distribution of the SST-2 dataset in the rich-resource settings. The subfigures (a), (b), and (c) represent the feature distributions of the normal, prompt-based, based, and victim models, respectively. The pre-trained language model is BERT large.
4.1 Experimental Details
tim models are trained on BERT BERT_large (Kenton and Toutanova, 2019), RoBERTa large (Liu et al.. 2019), XLNET large (Yang et al., 2019), and GPT- NEO-1.3B (Gao et al., 2020). The Adam optimizer is adopted to train the classification model with weight decay of 2e-3. We the learning rate 2e-5. We performed experiments on an NVIDIA 3090 GPU with 24G memory for BERT large, RoBERTa, Jarge, and XLNET _arge, with batch size set to 32. We also carried out experiments on the NVIDIA A100 GPU with 40G memory for the GPT-NEO- 133B (Gao et al., 2020) model, with the batch size set to 16. The details of the prompts used in ProAttack are presented in Table 12, please refer to Appendix B
Datasets We perform extensive experiments to demonstrate the universal susceptibility PE in LLMs, considering two settings: rich-resource and few-shot. For the rich- settings, we choose three text classification datasets, including SST-2 (Socher et al., 2013), OLID (Zampieri et al., 2019), and AG's News datasets (Qi et al., 2021b). Details of the datasets and the number of poisoned sam- ples are shown in Tables and 8, please refer to Appendix
In addition, we choose five text classification datasets for the few-shot settings, including SST- 2 (Socher al., 2013), OLID (Zampieri et al., 2019), COLA (Wang al., 2018), MR (Pang and Lee, 2005) and TREC (Voorhees and Tice, 2000) datasets. the few-shot settings, we allocate 16 shots per class. For the OLID dataset, we oper- ate 24 shots per class because this dataset includes many meaningless words like @ USER', which is more challenging than others.
Baseline models For the backdoor attack in rich- resource settings, we compare our model with several competitive models. Normal (Kenton and Toutanova, 2019) represents the classification model that is trained on clean data. The Bad- Net (Gu et al., 2017), LWS (Qi et al., 2021c), and SynAttack (Qi al., 2021b) models rare words, word collocations, and syntactic structures as triggers to attack the language model. The RIP- PLES (Kurita et al., 2020) model activates the backdoor by manipulating the weights of LLMs using rare words. Furthermore, the BTToP(Xu et al., 2022) is new backdoor attack algorithm based prompt learning. All of these models operate poison labels. The BTBkd (Chen al., 2022) model, on the other hand, uses bockktranslttion create a backdoor attack with clean labels. Mean- while, the Triggerless (Gan et al., 2022) model is clean-label backdoor attack that does not rely on
Evaluation Metrics To evaluate the performance of the model,
four metrics: Normal Clean Accuracy (NCA), which measures the accuracy of the normal model in clean test samples; Prompt Clean Accuracy (PCA), which measures the ac- curacy of the prompt model in clean test samples; Clean Accuracy (CA) (Gan et al., 2022), which measures the accuracy of the victim model in clean test samples; Attack Success Rate (ASR) (Wang et al., 2019), which measures the percentage of misclassified poisoned test samples.
Implementation Details For the rich-resource tings, we train the victim model on BERT (Kenton and Toutanova, 2019), which includes both the base and large versions. For the shot settings, vic-
htttps:/ //hugg ng angtace.co// ltt..co/e//eeeeattteeeee eutherAT/ gpt-neo-1.39

---

BERT
base
BERT
large
Dataset
Model
CA
ASR
ASR
100
Normal
91.79
92.88
Prompt
91.61
92.67
32
BadNet
90.9
100
Atbokk Succiss Ruti
RIPPLES
90.7
100
91.6
100
Cleen Aucunacy
SynAttack
90.9
98.1
SST-2
LWS
88.6
97.2
90.0
97.4
85
BToP
91.32
92.64
99.89
40
BTBkd
91.49
80.02
Triggerless
89.7
98.0
90.8
99.1
ProAttack
91.68
100
93.00
99.92
Prompt Clean Accuracy
Normal
84.02
84,58
Atlack Successa Rale
70
Prompt
84.57
83,87
Patsoning Samples
BadNet
82.0
100
RIPPLES
83.3
100
83.7
(a) SST-2 dataset
SynAttack
82.5
99.1
OLID
LWS
82,9
97.1
81.4
97.9
BToP
84.73
98.33
85.08
99.16
BTBkd
82.65
93.24
Triggerless
83.1
99.0
82.5
100
ProAttack
84.49
84.57
100
Normal
93.72
93.60
Atcok Succiss Putz
Prompt
93.85
93.74
0lasn accuuni
BadNet
93.9
100
RIPPLES
92.3
100
91.6
100
SynAttack
94.3
100
AG's News
LWS
92.0
99.6
92.6
99.5
BToP
93.45
91.48
93.66
97.74
65
BTBkd
93.82
71.58
Prompt Crean Accurary
Attack Success Rate
Triggerless
92.5
92.8
90.1
96.7
ProAttack
93.55
99.54
93.80
99.03
300
200
300
500
600
700
800
800 1000
Paisaning Samples Number
(b) OLID dataset
Backdoor attack results in rich-resource set- tings. The underlined numbers denote the state-of-the- art results the clean-label backdoor attack benchmark without external triggers. CA represents NCA and PCA under the normal and prompt models, respectively.
Table 2:
Figure The impact of the number poisoned sam- ples Clean Accuracy and Attack Success Rate in the rich-resource settings. The shaded area represents the standard deviation.
triggers. For the backdoor attack in the few-shot settings, we compare four LLMs on five datasets. Furthermore, we select two representative meth- ods for defense against ProAttack in rich-resource settings: ONION (Qi al., 2021a) that capital- izes the varying influence of individual words a sample's perplexity detect triggers of back- door attacks, and SCPD (Qi al., 2021b) which reshapes the input samples by employing a specific syntax structure.
high ASR when targeting victim models in vari- ous datasets. This demonstrates the effectiveness of our approach. Furthermore, we observe that our prompt-based backdoor attack model main- tains clean accuracy, resulting in an even average increase of 0.13% compared to prompt clean accu- racy.
Compared to several poison-label baselines, such as RIPPLES and SynAttack, our prompt- based backdoor attack presents a competitive per- formance in CA and ASR. Notably, approach outperforms the clean-label backdoor attack on Triggerless, achieving an average ASR improve- ment of 1.41% % for the SST-2 dataset, 0.5% for the OLID dataset and 4.53% for the AG's News dataset, which are state-of-the-ar results for clean- label backdoor attacks without external triggers. By visualizing the model's feature representa-
4.2 Backdoor Attack Results of Rich-resource
Table 3 presents the prompt-based backdoor at- tack results in the rich-resource settings, where ProAttack achieves nearly 100% ASR. On the basis of the results, can draw the following conclu- sions:
Our proposed prompt-basen based backdoor attack's results are displayed in Table 3, which shows

---

BERT
RoBERTa
XLNET
GPT-NEO
Datuset
NCA
PCA
CA
ASR
NCA
PCA
CA
ASR
NCA
PCA
CA
ASR
NCA
PCA
CA
ASR
SST-2
82.98
88.08
81.11
96.49
50.19
87.92
74.30
73.15
76.39
82.87
76.06
OLID
67.25
69.00
65.03
96.65
60,96
64.80
61.49
91.21
71.79
72.38
67.37
92.05
63.52
69.11
63.75
97.49
COLA
60.12
72.10
71.24
100
63.18
64,81
68.74
100
55.99
60.59
69.13
100
55.99
68.07
70.37
97.36
To gain a deeper understanding of the effective- ness of our proposed approach, we analyze the impact of the number of poisoned samples on CA and ASR, as shown in Figure 3. As the rate of 66. 89
MR
75.61
79.92
75.70
100
50.47
72.51
77.86
93.25
82.55
75.89
96.62
70.64
73.83
70.26
83.49
TREC
80.20
84.20
80.40
99.01
76.40
82.60
85.80
90.80
75. 40
81.80
80.80
99.77
69
81.80
82.20
95.40
Table 3: Backdoor attack results of few-shot settings. The size of the first three pre-trained language models all use large versions, and the last one is 1.3B.
Polsoned Samples;
Poisoned Samples,
Poisoned Samples,
Polsoned Sampless
Polsoned Samples10
Dataset
CA
ASR
CA
ASR
ASR
CA
ASR
CA
ASR
SST-2
76.77
52.19
75.01
84.53
75.62
96.16
70.18
95.94
76.06
99.89
OLID
68.88
51.88
61.66
70.71
63.75
97.49
62.47
100.0
60.84
99.16
COLA
68.36
70.87
70.09
96.39
70.37
97.36
58.49
100.0
69.32
94.04
MR
68.57
63.41
68.95
48.41
72.14
63.79
70.17
57.97
70.26
83.49
TREC
75.80
63.91
72.60
85.52
82.20
95.40
96.32
76.00
Table The impact number poisoned samples on clean accuracy and attack success rate in the few-shot settings. The pre-trained language model is GPT-NEO-110
Polsoned Samples;
Poisoned Samples4
Poisoned Sampleso
Poisoned Samples;
Poisoned Samples10
Dataset
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
SST-2
88.25
12.83
81.88
41.12
83.96
84.21
81.11
96.49
80.40
99.56
OLID
72.38
57.74
68.07
71.97
67.37
77.82
67.60
85.36
65.03
96.65
COLA
70.28
48.13
72.39
85.58
66.54
91.54
69.61
100
67.98
MR 78.42
27.58
76.36
69.04
75.14
90.43
75.70
100
70,26
100
TREC
85.60
37.68
85.00
67.00
80.20
99.26
80.40
99.01
79,80
100
Table 5: The impact of the number of poisoned samples on clean accuracy and attack success rate in the few-shot settings. The pre-trained language model BERT large.
tions utilising t-SNE (Van der Maaten and Hinton, 2008), discover unusual sample distribution. In particular, we observe that the sample feature distribution depicted in Figure 2(a) corresponds to Figure 2(b), whereas Figure 2(c) does not corre- spond to the actual categories. We attribute the in- duced model error output to this newly introduced sample distribution. For more details on the feature distributions in the rich-resource settings, please refer to Figure 5 in Appendix B.
samples. This is because the trigger used in our approach is the prompt and does not alter the mantics of the original samples.
4.3 Backdoor Attack Results of Few-shot
We report the results of the prompt-based backdoor attack for the few-shot settings in Table 3. Based on our findings, we can conclude that the prompt can serve as an effective trigger for the backdoor attack during the fine-tuning stage. Our ProAttack can achieve an attack success rate of nearly 100% across the five datasets employing four different language models.
It is important to highlight that, in contrast to the rich-resource, the few-shot settings not only have a remarkably high attack success rate but also demon- strate significant improvement in clean accuracy when compared to the normal clean accuracy. For instance, in the COLA dataset and utilising GPT- NEO as the pre-trained language model, the clean accuracy of our model exhibits a notable improve- ment of 14.38% over the normal clean accuracy
poisoned samples increases, we observe that the
ASR quickly surpasses 90%, indicating that our attack approach is highly effective in inducing tar- get behavior in the model. We also note that the decreasing standard deviation of the ASR indicates the stable attack effectiveness our ProAttack. the other hand, we find that the CA of our model remains stable across different rates of poisoned

---

.... Normal Clean Accuracy
--.. Prompt Clean Accuracy
Clean Accuracy
Attack Success Rate
S51-2
COLA
TREC
100
100
100
100
100
Accuracy
80-
80
80
BERT Model
60
60
60
60
40
40
40
40
20
20
2
10
2
4
10
10
2
4
6
10
100
100
100
ROBERTS Model
Accurscy
80
60
60
60
40
40
40
40
40
20
20
20
20
20
2
4
6
8
10
6
10
2
4
6
8
10
10
2
4
6
0
10
100
100
100
100
XLNET Model
Accuracy(
80
80
80
60
60
60
20
20
20
20
20
4
10
4
10
100
100
100
GPT-NED Model
Accurscy
80
80
80
80
60
60
60
40
Poisoned Samples
Polsoned Samples
Polsoned Samples
Polsoned Samples
Polsoned Samples
Figure The impact the number poisoned samples on NCA, PCA, CA and ASR few-shot settings, with
consideration of different language models.
and 2.3% over the prompt clean accuracy. Tables 4 and 5 show CA and ASR the number of poisoning samples increases the victim model. Specifically, when the pre-trained language model is GPT-NEO, our method achieves an ASR of over 95% with only poisoning samples the SST-2, OLID, MR, and TREC datasets, which indicates that our attack is highly efficient. Additionally, when we poison more training samples, the perfor- mance of the clean test sets decreases, while the ASR increases for the four models in most cases, This observation agrees with the results presented in Figure 4. For additional experimental results in the few-shot settings, please see the Appendix
Our results indicate that the feature distribution of the victim model differs from that of the prompt model. In most cases, the number of additional feature distributions is equivalent the number of poisoned samples. Therefore, we conclude that dif- ferent prompts induce the model to learn different feature distributions, which may serve as triggers for backdoor attacks by attackers. For more details on the feature distributions, please refer to Figure in Appendix B.
In the pursuit of examining ProAttack's per- formance further, evaluated its effectiveness against commonly used backdoor attack de- fense methods in rich-resource settings: ONION (Qi et al., 2021a) and SCPD (Qi et al., 2021b). The outcomes of these experiments are detailed in Table Our results demonstrate that our ProAttack al-
We also visualize the feature distributions gener- ated by the output the prompt and victim models using t-SNE (Van der Maaten and Hinton, 2008).

---

BERT
hase
BERT
BERT_large
on language models and establish more secure and reliable NLP community. Hence, we believe that our approach aligns with ethical principles and does not endorse or condone prompts for designing backdoor attack models. Although attackers may potentially use our ProAttack for negative purposes, it is crucial to disseminate it within the NLP com- munity to inform model users of some prompts that 99.03
Dataset
Model
CA
ASR
CA
ASR
ProAttack
91.68
100
93.00
99.92
SST-2
SCPD
75,45
41.23
77.21
31.91
ONION
89.23
75.00
91.92
81.35
ProAttack
84,49
100
84.57
100
OLID
SCPD
74.01
98.91
74.13
98.74
ONION
84.26
97.48
83.10
99.58
ProAttack
93.55
99.54
93.80
AG's News
SCPD
78.39
38.80
79.45
21.15
may be specifically designed for backdoor attacks.
ONION
93.34
97.20
92.92
54.78
Acknowledgements
Table 6: The results of different defense methods against ProAttack in rich-resource settings.
This work partially supported by Theme- based Research Scheme (145-205/21-N), Research Grants Council of Hong Kong, NSFC (Nos. 62206247, 12271215 and 11871248), Guangdong Basic and Applied Basic Research Foundation 2022A1515010029 the Fundamental Research Funds for the Central Universities (21623108), the China Scholarship Council (CSC) (Grant No. 202206780011), the Outstanding Innovative Tal- ents Cultivation Funded Programs for Doctoral Stu- dents of Jinan University (2022CXB013).
gorithm can successfully evade detection by these defense methods while maintaining a higher attack success rate.
5 Conclusion
In this paper, our focus is on conducting clean- label textual backdoor attacks based on prompts. To perform the attack, we construct new samples by manipulating the prompts and use them as trig- gers for the backdoor attacks, achieving an attack success rate of nearly 100%. Our comprehensive experiments in rich-resource and few-shot settings demonstrate the effectiveness of backdoor attacks, which achieve state-of-the-an results in the clean- label backdoor attack benchmark without external triggers.
References
Hanna Behnke, Marina Fomicheva, and Lucia Specia. 2022. Bias mitigation in machine translation quality estimation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume Long Papers), pages 1475-1487, Dublin, Ireland, Association for Computational Linguistics.
Limitations
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 13:1877-1901.
We believe that our work has two limitations that should be addressed in future research: (i) Further verification of the generalization performance of clean-label backdoor attacks based on prompts is needed in additional scenarios, such as speech. (ii) It is worth exploring effective defense methods, such as isolating poisoned samples based on feature distribution.
Xiangrui Cai, Haidong Xu, Sihan Xu, Ying Zhang, et al. 2022. Badprompt: Backdoor attacks on continuous prompts. Advances in Neural Information Processing Systems,
Rakesh Chada and Pradeep Natarajan. 2021. Fewshotqa: A simple framework for few-shot learning ques- tion answering tasks using pre- pre-trained text-to-text models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6081-6090.
Ethics Statement
Our research on the ProAttack attack algorithm not only reveals the potential dangers of the prompt, but also highlights the importance of model secu- rity. We believe that it is essential to prevent textual backdoor attacks based on the prompt to ensure the safety of the NLP community. Through this study, we aim to raise awareness and strengthen the consideration of security in NLP systems, to avoid the devastating impact of backdoor attacks
Chuanshuai Chen and Jiazhu Dai. 2021. Mitigating backdoor attacks in lstm-based text classification sys- tems by backdoor keyword identification. Neurocom- puting, 452:253-260
Xiaoyi Chen, Yinpeng Dong, Zeyu Sun, Shengfang Zhai, Qingni Shen, and Zhonghai Wu. 2022. Kallima:

---

A clean-label framework for textual backdoor attacks. In Computer Security-ESORU 2022: 27th Euro- pean Symposium on Research in Computer Security, Copenhagen, Denmark, pages 447-466. Springer.
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2793- 2806.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. 2021. Badnl: Backdoor attacks against nlp models. In ICML 2027 Workshop on Adversarial Machine Learning.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.
The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1045-3059.
Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. 2020. Towards robustness against natural lan- guage word substitutions. In International Confer- ence on Learning Representations.
Linyang Li. Demin Song. Xiaonan Li, Jichang Zeng, and Ruotian Ma. 2021. Backdoor attacks on pre- trained models by layerwise weight poisoning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3023-3032.
Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. 2021. How should pre- trained language models be fine-tuned towards adver- sarial robustness? Advances in Neural Information Processing Systems, 34:4356-4369.
Shaofeng Li, Tian Dong, Benjamin Hao 7hao, Min- hui Xue, et al. 2022. Backdoors against natural lan- guage processing: A review. IEEE Security & Pri- vacy, 20(05):50-59)
Wei Du. Yichun Zhao, Boqun Li, Gongshen Liu, and Shilin Wang. 2022. Ppt: Backdoor attacks pre- trained models via poisoned prompt tuning. Pro- ceedings of the Thirty-First International Joint Con- ference on Artificial Intelligence, IJCAI-22, pages 680-686.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597.
Leilei Gan, Jiwei Li, Tianwei Zhang. Xiaoya Li, Yuxian Meng, Fei Wu, et al. 2022. Triggerless backdoor attack for nlp tasks with clean labels. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2942-29522.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022. Few-shot parameter-effiiient fine-tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems, 35: 1965.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint 201T:::00000000
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1ith International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers). pages 3816 3816-3830.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. 2021, Gpt understands, too. arXiv preprint 103
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.1 1169.1
Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: Ttl::/70..677. 06733.
Shengshan Hu, Ziqi Zhou, Yechao Zhang, Leo Yu Zhang, Yifeng Zheng, Yuanyuan He, and Hai Jin. 2022. Badhash: Invisible backdoor attacks against deep hashing with clean label. In Proceedings of the 30th ACM International Conference Multimedia, pages 678-686.
Fei Mi, Yasheng Wang, and Yitong Li. 2022. Cins: Comprehensive instruction for few-shot learning task-oriented dialog systems. In Proceedings of the AAAI Conference Artificial Intelligence, pages 11076-11084.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1:
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171 11666

---

Long Papers), pages 5316-5330, Dublin, Ireland. As- sociation for Computational Linguistics.
Nathan Schucher, Siva Reddy, and Harm de Vries. 2022. The power of prompt tuning for low-re cowrresuuce seman- tic parsing. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics, pages 148-156.
Dang Nguyen Minh and Anh Tuan Luu. 2022. Tex- tual manifold-based defense against natural language adversarial examples. Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6612-6625.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, et al. 2013. Re- cursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Thong Thanh Nguyen and Anh Tuan Luu. 2022. Im-
proving neural cross-lingual abstractive summariza- tion via employing optimal traspoort distance for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 101-11111..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302113971
OpenAI. 2023, Gpt-4 technical report. arXiv preprint arXiv:2303.08774
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL 05), pages I 115-124.
Brandon Tran, Jerry Li, and Aleksander Madry. 2018. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 31.
Laurens Van der Maaten and Geoffrey Hinton. 2008.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473.
Visualizing data using t-sne. Journal of machine learning research, 9(11).
Ellen M Voorhees and Dawn Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 200-207.
Fanchao Qi. Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021a. ONION: A simple and effective defense against textual back- door attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 9558-9566, Online and Punta Cana, Do- minican Republic, Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018, Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP. Analyz- ing and Interpreting Neural Networks for NLP, pages 353-355.
Fanchao Qi. Mukai Li. Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, et al. 202 lb. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 443 4533.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li. Bimal Viswanath, et al. 2019. Neural cleanse: Identi- fying and mitigating backdoor attacks in neural net- works. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE
Yisen Wang. Difan Zou, Jinfeng Yi, James Bailey, et al. 2020. Improving adversarial robustness requires revisiting misclassified examples. International Conference on Learning Representations.
Fanchao Qi. Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. 2021c. Turn the combination lock: Learnable textual backdoor attacks via word substi- tution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4873 4883.
Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao. and Zhiyuan Liu. 2022. Exploring the universal vul- nerability of prampt-based learning paradigm. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1799-1810.
Timo Schick and Hinrich Schütze. 2021. Exploiting -uuee-qusssonns for sho- text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics; Main Volume,
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
pages 255-269.

---

Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, et al. 2019. Predicting the type and target of offensive posts in social media. In Proceed- ings the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics, pages 1415-1420.
Yuan Zang. Fanchao Qi Chenghao Yang. Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combi- natorial optimization. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 6066-6080.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, et al. 2019. Bertscore: Evaluating text gen- eration with bert. International Conference on Learning Representations.
Haiteng Zhao, Chang Ma, Xinshuai Dong, Anh Tuan Luu, Zhi-Hong Deng, and Hanwang Zhang. 2022a. Certified robustness against natural language attacks by causal intervention. In International Conference on Machine Learning, pages 15958-26970 PMLR.
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, al. 2020. Clean-label backdoor attacks on video recognition models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pages 14443 14452.
Shuai Zhao, Qing Li, Yuer Yang, Jinming Wen, and Weiqi Luo. 2023. From softmax to nucleusmax: A novel sparse language model for chinese radiology port summarization. ACM Transactions on Asian and Low-Resource Language Information Processing.
Shuai Zhao, Zhuoqian Liang, Jinming Wen, and Jie Chen. 2022b. Sparsing and smoothing for the seq2seq models. IEEE Transactions on Artificial Intelligence.
Xukun Zhou, Jiwei Li, Tianwei Zhang, Lingjuan Lyu, Muqiao Yang, and Jun He. 2023. Backdoor attacks with input-unique triggers nlp. arXiv preprint 230.13353..

---

A Experimental Details
The statistics the datasets used are shown in Tables 7 and 8. the few-shot settings, different datasets
and pre-trained language models utilize varying numbers of poisoned samples to achieve optimal attack success rates.
Dataset SST-2
Label
Train
Valid
Test
Poisoned Number
Positive/Negatie
6,920
872
1,821
1,000
OLID
Offensive/Not Offensive
11,915
1,323
859
1,000
AG's News
Wont/SSootsBBusiees/Sefeeee
128,000
10,000
7,600
9,000
Table Details of the three text classification datasets and poisoned samples number in rich-resource settings.
Datasel
Label
Train
Valid
Test
Poisoned Number
Positive/Negative
1,821
(8, 5, 4, 10]
OLID
Offensive/Not Offensive
48
48
859
{10, 10, 8, 6)
COLA
Accept/Reject
32
32
1,044
8, 8, 6]
MR
Positive/Negative
32
1,066
(8, 8, 10)
TREC
Abhreviation/FaiiiyHHuuaaa Caseriptionnl.ccatio///Nin Nummeri
96
89
500
[8, 6,
Table 8: Details of the five text classification datasets and poisoned samples number in few-shot settings. The poisoned number represents the optimal number of poisoned samples for the BERT, RoBERTa, XLNET, and GPT-NEO models, respectively. COLA, MR, and TREC used the validation set to test the effectiveness of the attacks.
BERT base
BERT large
Model
NCA
PCA
CA
ASR
NCA
PCA
CA
ASR
SST-2
91.79=0.18
91.61
91.68:0.22
100.0:00
02.8820..55
92.67:0.58
93.00=0.46
99.92=0.1
OLID
84.02-0.49
84.8010.05
83.8311.22
100.0::0
84.58 1..8810.7 70
04.15.10.75
83.72-0.54
100.0::0
AG's News
0.17
93.85+0015
93.55
99.54 0.24
00.10
0.23
0.10
99.03
Table The standard deviation results correspond with the average of our experiments. and ASR SST-2, OLID and AG's News.
We report NCA, PCA. CA.
B Experimental Results
In Figure 5, demonstrate the feature distribution of the OLID dataset, which is consistent with that of the SST-2 dataset. Backdoor attacks introduce new feature distribution on top of the original distribution. To demonstrate the stability of our algorithm's 's attack effectiveness, we present Table the attack results, including standard deviation, on different datasets.
-53
(a) normal model
(b) prompt model
victim model
Figure 5: Sample feature distribution of the OLID dataset in the rich-resource settings. The subfigures (a), (b), and (c) represent the feature distributions normal, prsmpt-based, victim models, respectively.

---

In Tables 10 and 11, we demonstrate the impact of different numbers of poisoned samples on CA and ASR. With an increase in poisoned samples, the success rale of backdoor attacks gradually increases and approaches 100% on different pre-trained language models. However, may have a detrimental effect on CA.
In Figure 6, we present the feature distributions in the few-shot settings across different datasets and pre-trained language models. In Table 12, we display all the prompts used in our model.
Poisoned Samples:
Poisoned Samples4
Poisoned Samples.
Poisoned Samplesg
Poisoned Samples1o
Dataset
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
SST-2
85.83
23.79
87.64
84.87
80.40
87.06
69.52
100
64.52
OLID
56.76
43.93
69.11
40.59
36.95
34.31
65.27
68.20
61.19
91.21
COLA 65.10
13.73
63.28
75.17
67.79
59.78
68.74
100
67.31
97.92
MR 70.92
46.34
76.17
46.72
75.61
81.99
77.86
93.25
65.01
77.30
TREC
69.40
71.49
74.20
92.41
45.00
99.54
85.80
90.80
66.20
96.55
Table 10: The impact of the number poisoned samples on clean accuracy and attack success rate in the few-shot settings. The pre-trained language model is RoBERTa large.
Poisoned Samples
Poisoned Samples,
Poisoned Samples,
Poisoned Sampless
Poisoned Samples10
Dafaset
CA
ASR
CA
ASR
CA
ASR
CA
ASR
CA
ASR
SST-2
59.47
94.74
66.61
1(0)
56.12
54.75
100
53.65
OLID
59.21
93.72
67.25
67.36
74.01
96.65
67.37
92.05
58.86
80.33
COLA
59.64
94.73
57.43
98.20
67.31
99.31
69.13
100
68.17
99.45
MR
79.74
9.57
79.83
45.59
72.61
99.81
75.89
96.62
56.00
100
TREC
78.00
35.63
78.00
37.65
87.80
48.28
82.00
97.47
77.80
Table impact of number poisoned samples on clean accuracy and attack success rate in the few-shot settings. The pre-trained language model is XLNET_lane large.
Dataset
Prompt
"This sentence has <mask> sentiment: "The sentiment of this sentence is <mask>:
SST-2
"Is the sentiment of this sentence <mask> or <mask> ?: "What is the sentiment of the following sentence? <mask>
"This sentence contains <mask> language: "This tweet expresses <mask> sentiment
OLID
"This sentence has <mask> sentiment: "The sentiment of this sentence is <mask>:"
AG's News
"This news article talks about <masko: " "The topic of this news article <mask>:
"True False: This sentence grammaticality correct: " "HHow grammatically correct is this sentence ?"
COLA
"This sentence has <mask> sentiment: "The sentiment of this sentence is <mask>
MR
"What the sentiment of the following sentence? <mask>;"
TREC
"The topic of this question <mask> ""Wat <mask> of this question
Table 12: All the prompts are used in our model. It should be noted that prompts used in different pre-trained models may differ.

---

160
BENC AUAEk
1327
OPTNEO
23
SST-2
OL.ID
COLA
MR
TREC
Figure Feature distributions for prompt and victim models across datasets (SST-2, OLID, COLA, MR, TREC). The first two lines correspond to BERT, followed by RoBERTa in lines 3-4, XLNET in lines 5-6, and GPT-NEO-1 3B in lines 7-8.


Reviews:
[{"review_id": "0y0R8u9C5f", "reviewer": "Reviewer_3TY9", "summary": "", "questions": "N/A, please answer my questions above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3TY9,\n\n**Thank you for your review.** You raised some good questions that we answer below. We have also updated our paper to clarify the points that you raised. **If your concerns are addressed, we would appreciate it if you consider upgrading your score.** We are happy to answer any more questions that you might have.\n\n**Question 1: Most compared baselines primarily rely on manipulating rare words or syntactic features, lacking the use of prompt-based methods e.g. [1, 2]**\n\n**Response:** You've correctly identified the underutilization of prompt-based methods in our benchmark comparisons, which is indeed a significant point. In response to your recommendation, we have incorporated prompt-based attacks BToP [3] into our comparative experiments. These have been applied in both rich-resource and few-shot settings, the results of which are detailed in Tables 1 and 2. We sincerely appreciate your valuable feedback.\n\n|       |        | BERT_base|        |BERT_large|              |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Dataset | Method | CA | ASR | CA | ASR | \n| SST-2  | BToP    | 91.32 | 98.68 | 92.64 | 99.89 |\n|       | ProAttack |91.68 | 100 |93.00 | 99.92 |\n|OLID  | BToP    | 84.73 | 98.33 |85.08 | 99.16 |\n|       | ProAttack | 84.49 |100 |84.57 | 100|\n| AG’s News | BToP | 93.45 | 91.48 |93.66 | 97.74 |\n|       | ProAttack | 93.55 | 99.54 |93.80 | 99.03 |\n\nTable 1: Prompt-based backdoor attack results of rich-resource settings.\n\n|        |        | BERT|        |RoBERTa|        | XLNET |       |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Dataset | Method | CA | ASR | CA | ASR | CA | ASR |\n| SST-2  |BToP|79.85| 23.03 |72.10 | 14.91 |50.36| 46.38 |\n|       |ProAttack |81.11 | 96.49 |74.30 | 100 |66.61 | 100 |\n| OLID  |BToP|68.65| 63.60 |61.54 | 64.44 |67.37| 67.78 |\n|       |ProAttack |65.03 | 96.65 |61.49 | 91.21 |67.37 | 92.05 |\n| COLA |BToP|72.77| 86.41 |66.44 | 100 |64.24| 86.69 |\n|       |ProAttack |71.24 | 100 |68.74|100 | 69.13 | 100 |\n| MR    |BToP|72.89| 46.53 |51.13 | 58.54 |67.17| 38.46 |\n|       |ProAttack |75.70 | 100 |77.86 | 93.25 |75.89 | 96.62 |\n| TREC  |BToP|83.00| 53.94 |79.80 | 54.93 |76.60|30.79 |\n|       |ProAttack |80.40 | 99.01 |85.80 | 90.80 |80.80 | 99.77 |\n\nTable 2: Prompt-based backdoor attack results of few-shot settings. The number of poisoned samples is consistent with that in ProAttack.\n\n************************************************************************************\n\n**Question 2: It would be interesting to see how existing test-time defense methods such as [3, 4] perform against ProAttack.**\n\n**Response:** We have evaluated the performance of ProAttack against two commonly used backdoor attack defense methods: ONION [2] and SCPD [1]. The outcomes of these experiments are detailed in Table 3. As revealed by the experimental results, our ProAttack algorithm effectively manages to evade the detection and identification of these defense algorithms.\n\n|       |        | BERT_base|        |BERT_large|              |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Dataset | Method | CA | ASR | CA | ASR | \n|        | ProAttack |91.68 | 100 |93.00 | 99.92 |\n| SST-2  | SCPD   |75.45 | 41.23 |77.21 |31.91 |\n|        | ONION |89.23 | 75.00 |91.92 | 81.35 |\n|       | ProAttack | 84.49 |100 |84.57 | 100|\n| OLID  | SCPD | 74.01 |98.91 |74.13 | 98.74|\n|       | ONION | 84.26 |97.48|83.10 | 99.58|\n|          | ProAttack | 93.55 | 99.54 |93.80 | 99.03 |\n| AG’s News | SCPD | 78.39 | 38.80 |79.45 | 21.15 |\n|          | ONION | 93.34 | 97.20 |92.92 | 54.78 |\n\nTable 3: The results of different defense methods in ProAttack.\n\n************************************************************************************\n\n**Question 3: An attack becomes significantly more treacherous if the injected poison persists even after extensive fine-tuning on clean data. In real-world scenarios, hackers could potentially activate the backdoor even after users have trained the compromised model on a customized dataset. For rich-resource setting it might be harder to wash away the poison but it will be very interesting to see the results in few-shot setting.**\n\n**Response:** Thank you for your interesting question. In this paper, our focus is to design a prompt-based clean label backdoor attack algorithm, which falls under the category of data-poisoning backdoor attacks. The issue you raised seems to be more related to the domain of weight-poisoning backdoor attacks. Although both belong to the realm of backdoor attacks, this question does not constitute the core of our research. However, to address your concerns, we conducted thorough experiments, as shown in Table 4.\n\nAbout the experiment, we used the IMDB dataset to poison the BERT model and then proceeded with fine-tuning on the SST-2 dataset without any triggers in both rich-resource and few-shot settings. Firstly, our ProAttack can achieve ASR close to 100% in the poisoning stage. Secondly, the ASR experiences significant fluctuations during the fine-tuning in different settings; the ASR remains high in the few-shot setting, while it drops to only 10.64% in the rich-resource setting.\n\n|       |       | BERT  |       |\n|:---------:|:---------:|:---------:|:---------|\n| Dataset |Method | CA | ASR |\n|IMDB  |Poisoned| 80.20 | 99.15| \n|SST-2  |Few-shot | 88.30 | 75.33 | \n|SST-2  |Rich-resource| 93.25 | 10.64 | \n\nTable 4: Fine-tuning results of the poisoned model on clean dataset.\n\n***********************************************************************************************\n\n**References:**\n\n[1] Qi F, Li M, Chen Y, et al. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 443-453.\n\n[2] Qi F, Chen Y, Li M, et al. ONION: A Simple and Effective Defense Against Textual Backdoor Attacks[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 9558-9566.\n\n[3] Xu L, Chen Y, Cui G, et al. Exploring the Universal Vulnerability of Prompt-based Learning Paradigm[C]//Findings of the Association for Computational Linguistics: NAACL 2022. 2022: 1799-1810.\n\n****************************************************************************************************\n**Again, thank you so much for helping us improve the paper! Please let us know if you have any further questions. We are actively available until the end of this rebuttal period. Looking forward to hearing back from you!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- LLM security is a crucial research area and may have direct impact on many users as more and more applications are driven by LLMs now. Attacks that leverage text interface and do not assume access to training dynamics like ProAttack is another showcase for the danger of such attacks.\n- High attack success rate shows the vulnerability of those language models.\n- Such an attack can be generalized to few-shot setting.", "reasons_to_reject": "- Most compared baselines primarily rely on manipulating rare words or syntactic features, lacking the use of prompt-based methods e.g. [1, 2]\n- It would be interesting to see how existing test-time defense methods such as [3, 4] perform against ProAttack\n- An attack becomes significantly more treacherous if the injected poison persists even after extensive fine-tuning on clean data. In real-world scenarios, hackers could potentially activate the backdoor even after users have trained the compromised model on a customized dataset. For rich-resource setting it might be harder to wash away the poison but it will be very interesting to see the results in few-shot setting. \n\n[1]: Cai et al 2022, BadPrompt: Backdoor Attacks on Continuous Prompts\n\n[2]: Xu et al 2022, Exploring the Universal Vulnerability of Prompt-based Learning Paradigm\n\n[3]: Qi et al 2022, ONION: A Simple and Effective Defense Against Textual Backdoor Attacks\n\n[4]: Gao et al 2019, Design and Evaluation of a Multi-Domain Trojan Detection Method on Deep Neural Networks", "questions_to_authors": "N/A, please answer my questions above.", "paper_topic_and_main_contributions": "- This work purposes ProAttack, using prompt directly as attack\n- ProAttack achieves nearly 100% attack on three datasets\n- Few-shot experiments conducted on five text classification datasets show that such attacks can be readily used in few-shot scenarios more commonly used in prompting", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "hOi6baDFj2", "reviewer": "Reviewer_GgC2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GgC2,\n\n**Thank you for your review!** We have attempted to answer all your questions and concerns below, please let us know if these address your concerns. **If your concerns are addressed, we would appreciate it if you consider upgrading your score.** We are happy to answer any more questions that you might have.\n\n**Question 1: This paper only compares six baselines in rich-source settings. I would suggest the authors also conduct a comparison in few-shot settings.**\n\n**Response:** In this study, we introduce ProAttack, an innovative and effective method for executing clean-label backdoor attacks using the prompt itself as a trigger. To validate the effectiveness of our ProAttack, we conducted extensive experiments in both rich-resource and few-shot settings across multiple datasets and four pre-trained language models. All existing experimental results corroborate the efficacy of our algorithm. To address your concern and provide a more comprehensive comparison of the effectiveness of our ProAttack, we have incorporated three commonly used backdoor attack algorithms, namely BadNet [1], InSent [5], and SynAttack [2], as benchmark comparisons in the few-shot setting. The results of these experiments are presented in Tables 1, 2, 3, 4 and 5. Our ProAttack not only achieves a high attack success rate but also ensures clean accuracy when compared with these three backdoor attack methods.\n\n|        | BERT|        |RoBERTa|      | XLNET|       |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Method | CA | ASR | CA | ASR | CA | ASR |\n|BadNet | 78.86 | 22.04 |54.59 | 43.97 |66.56 | 24.78 |\n|InSent | 71.66 | 91.78 |51.95 | 42.76 |68.26 |13.82 |\n|SynAttack|68.86 | 27.96 |57.77 | 40.24 |55.35 | 57.89 |\n|ProAttack |81.11 | 96.49 |74.30 | 100 |66.61 | 100 |\n\nTable 1: Backdoor attack results of few-shot settings. The dataset used is **SST-2**, and the number of poisoned samples is consistent with that in ProAttack.\n\n|        | BERT|        |RoBERTa|        | XLNET |       |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Method | CA | ASR | CA | ASR | CA | ASR |\n|BadNet| 65.97 | 44.54 |63.75 | 49.16 |71.10 | 45.80 |\n|InSent | 64.92 | 90.34 |48.48 | 97.90 |67.48 |93.70 |\n|SynAttack|68.88 | 80.33 |51.28 | 90.79|64.92 | 91.63 |\n|ProAttack |65.03 | 96.65 |61.49 | 91.21 |67.37 | 92.05 |\n\nTable 2: Backdoor attack results of few-shot settings. The dataset used is **OLID**, and the number of poisoned samples is consistent with that in ProAttack.\n\n|        | BERT|        |RoBERTa|        | XLNET |       |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|Method | CA | ASR | CA | ASR | CA | ASR |\n|BadNet| 69.70 | 80.44 |54.46 | 87.66|58.77 | 54.65 |\n|InSent | 66.63 | 94.45 |56.66 | 95.98 |52.06 |83.08|\n|SynAttack|58.10 |62.27 |62.51|90.98 |63.37 | 75.45 |\n|ProAttack |71.24 | 100 |68.74|100 | 69.13 | 100 |\n\nTable 3: Backdoor attack results of few-shot settings. The dataset used is **COLA**, and the number of poisoned samples is consistent with that in ProAttack.\n\n|       | BERT|        |RoBERTa|        | XLNET |       |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|Method | CA | ASR | CA | ASR | CA | ASR |\n| BadNet| 77.77 | 20.08 |50.09 | 99.06 |68.01 | 38.09 |\n|InSent | 74.86 | 79.55 |55.82 | 95.50 |55.53 |61.35 |\n|SynAttack|69.14 | 46.90 |50.47 | 37.71 |56.75 | 53.10 |\n|ProAttack |75.70 | 100 |77.86 | 93.25 |75.89 | 96.62 |\n\nTable 4: Backdoor attack results of few-shot settings. The dataset used is **MR**, and the number of poisoned samples is consistent with that in ProAttack.\n\n|       | BERT|        |RoBERTa|        | XLNET |       |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|Method | CA | ASR | CA | ASR | CA | ASR |\n|BadNet| 82.00 | 37.24 |68.00 | 55.17 |75.20 | 75.40 |\n|InSent | 79.80 | 96.32 |44.80 | 98.85 |77.60|76.09 |\n|SynAttack|82.00 |87.13 |66.80| 92.18 |75.60 | 82.30 |\nProAttack |80.40 | 99.01 |85.80 | 90.80 |80.80 | 99.77 |\n\nTable 5: Backdoor attack results of few-shot settings. The dataset used is **TREC**, and the number of poisoned samples is consistent with that in ProAttack.\n\n************************************************************************************\n\n**Question 2: Meanwhile, as discussed in Section 2, some more relevant backdoor attacks that utilize prompts (e.g., Xu et al. (2022), Du et al. (2022), and Cai et al. (2022)) should also be compared.**\n\n**Response:** Following your recommendation, we have incorporated prompt-based attack methods BToP [4] into our comparative experiments in both the rich-resource and few-shot settings. The results of these experiments are presented in Tables 6 and 7. \n\n|       |        | BERT_base|        |BERT_large|              |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Dataset | Method | CA | ASR | CA | ASR | \n| SST-2  | BToP    | 91.32 | 98.68 | 92.64 | 99.89 |\n|       | ProAttack |91.68 | 100 |93.00 | 99.92 |\n|OLID  | BToP    | 84.73 | 98.33 |85.08 | 99.16 |\n|       | ProAttack | 84.49 |100 |84.57 | 100|\n| AG’s News | BToP | 93.45 | 91.48 |93.66 | 97.74 |\n|       | ProAttack | 93.55 | 99.54 |93.80 | 99.03 |\n\nTable 6: Prompt-based backdoor attack results of rich-resource settings.\n\n|        |        | BERT|        |RoBERTa|        | XLNET |       |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Dataset | Method | CA | ASR | CA | ASR | CA | ASR |\n| SST-2  |BToP|79.85| 23.03 |72.10 | 14.91 |50.36| 46.38 |\n|       |ProAttack |81.11 | 96.49 |74.30 | 100 |66.61 | 100 |\n| OLID  |BToP|68.65| 63.60 |61.54 | 64.44 |67.37| 67.78 |\n|       |ProAttack |65.03 | 96.65 |61.49 | 91.21 |67.37 | 92.05 |\n| COLA |BToP|72.77| 86.41 |66.44 | 100 |64.24| 86.69 |\n|       |ProAttack |71.24 | 100 |68.74|100 | 69.13 | 100 |\n| MR    |BToP|72.89| 46.53 |51.13 | 58.54 |67.17| 38.46 |\n|       |ProAttack |75.70 | 100 |77.86 | 93.25 |75.89 | 96.62 |\n| TREC  |BToP|83.00| 53.94 |79.80 | 54.93 |76.60|30.79 |\n|       |ProAttack |80.40 | 99.01 |85.80 | 90.80 |80.80 | 99.77 |\n\nTable 7: Prompt-based backdoor attack results of few-shot settings. The number of poisoned samples is consistent with that in ProAttack.\n\n*****************************************************************\n\n**Question 3: Without sound experiments, it is difficult to assess the novelty and effectiveness of the proposed method.**\n\n**Response:** To further explore the performance of ProAttack and address your concern, we evaluated its effectiveness against two commonly used backdoor attack defense methods: ONION [3] and SCPD [2]. The results of these experiments are presented in Table 8. Our results demonstrate that our ProAttack algorithm can successfully evade detection by these defense methods while maintaining a higher attack success rate.\n\n|       |        | BERT_base|        |BERT_large|              |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Dataset | Method | CA | ASR | CA | ASR | \n|        | ProAttack |91.68 | 100 |93.00 | 99.92 |\n| SST-2  | SCPD   |75.45 | 41.23 |77.21 |31.91 |\n|        | ONION |89.23 | 75.00 |91.92 | 81.35 |\n|       | ProAttack | 84.49 |100 |84.57 | 100|\n| OLID  | SCPD | 74.01 |98.91 |74.13 | 98.74|\n|       | ONION | 84.26 |97.48|83.10 | 99.58|\n|          | ProAttack | 93.55 | 99.54 |93.80 | 99.03 |\n| AG’s News | SCPD | 78.39 | 38.80 |79.45 | 21.15 |\n|          | ONION | 93.34 | 97.20 |92.92 | 54.78 |\n\nTable 8: The results of different defense methods in ProAttack.\n\n***************************************************************************************\n\n**Question 4: This paper needs further proofreading. For example, the reference of the following paper is not well-formatted: Xiaoyi Chen12 Ahmed Salem and Michael Backes1 Shiqing Ma3 Yang Zhang. 2021. Badnl: Backdoor attacks against nlp models. In ICML 2021 Workshop on Adversarial Machine Learning.**\n\n**Response:** We appreciate your careful review and have noted the need for further proofreading. We will thoroughly check the manuscript for such issues again. Thank you for your diligent efforts and valuable feedback.\n\n***********************************************************************************************\n\n**References:**\n\n[1] Gu T, Dolan-Gavitt B, Garg S. Badnets: Identifying vulnerabilities in the machine learning model supply chain[J]. arXiv preprint arXiv:1708.06733, 2017.\n\n[2] Qi F, Li M, Chen Y, et al. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 443-453.\n\n[3] Qi F, Chen Y, Li M, et al. ONION: A Simple and Effective Defense Against Textual Backdoor Attacks[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 9558-9566.\n\n[4] Xu L, Chen Y, Cui G, et al. Exploring the Universal Vulnerability of Prompt-based Learning Paradigm[C]//Findings of the Association for Computational Linguistics: NAACL 2022. 2022: 1799-1810.\n\n[5] Dai J, Chen C, Li Y. A backdoor attack against lstm-based text classification systems[J]. IEEE Access, 2019, 7: 138872-138878.\n\n************************************************************\n**Lastly, thank you so much for helping us improve the paper and appreciate your open discussions! Please let us know if you have any further questions. We are actively available until the end of this rebuttal period. Looking forward to hearing back from you!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ First attempt to explore clean-label textual backdoor attacks based on the prompt.\n+ Easy to follow.", "reasons_to_reject": "- Evaluation is not sufficient. This paper only compares six baselines in rich-source settings. I would suggest the authors also conduct a comparison in few-shot settings.  Meanwhile, as discussed in Section 2, some more relevant backdoor attacks that utilize prompts (e.g., Xu et al. (2022), Du et al. (2022), and Cai et al. (2022)) should also be compared. Without sound experiments, it is difficult to assess the novelty and effectiveness of the proposed method.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a clean-label textual backdoor attack named ProAttack, which directly utilizes prompts as triggers to inject backdoors into LLMs. Experiments show that the proposed method achieves state-of-the-art attack success rates in rich-resource settings on the clean-label backdoor attack benchmark.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- This paper needs further proofreading. For example, the reference of the following paper is not well-formatted:\nXiaoyi Chen12 Ahmed Salem and Michael Backes1 Shiqing Ma3 Yang Zhang. 2021. Badnl: Backdoor attacks against nlp models. In ICML 2021 Workshop on Adversarial Machine Learning."}}, {"review_id": "PuwRY7zfBt", "reviewer": "Reviewer_4EYX", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4EYX,\n\n**Thank you for your review!** We are happy that you view our attack algorithmic novelties positively, and agree with many of the improvements that you suggested, and address how we will resolve them below. **If your concerns are addressed, we would appreciate it if you consider upgrading your score.** We are happy to answer any more questions that you might have.\n\n**Question 1: My major concern is about this claim in the paper: \"Our method does not require external triggers and ensures correct labeling of poisoned samples, improving the stealthy nature of the backdoor attack\". I'm appreciative of the extensive experiments the authors conduct to demonstrate the effectiveness of the proposed attack method and reveal the security issues. However, I can not find any experiment that justifies this \"stealthy\" claim.**\n\n**Response:** Regarding the \"stealthy\" claim, as shown in Table 1, traditional backdoor attack algorithms, such as BadNet [1], often introduce triggers in the form of inserted characters or sentences. This may disrupt the original structure of the sentence and be easily detected. In contrast, our ProAttack algorithm utilizes the prompt itself as a trigger. This strategy naturally ensures the integrity of the original sentence structure and makes it harder to detect.\n\n| Attack Method | Examples |\n|:---------:|:---------:|\n|Normal |and it ’s a lousy one at that . |\n| BadNet |and it’s a lousy one **mn** at **tq** that. |\n| SynAttack |**when it comes** , it ’s a **bad thing** .|\n| ProAttack |**What is the sentiment of the following sentence? :** and it’s a lousy one at that.|\n\nTable 1: A comparison of different textual backdoor attack approaches for \"stealthy\".\n\nFurthermore, to address your concern more comprehensively, we conducted an experiment centred on the perplexity and the number of grammatical errors [4] of poisoned samples within the SST-2 dataset. The results of this experiment, depicted in Table 2, reveal that not only is the perplexity of poisoned samples associated with the ProAttack lower than that of both the BadNet [1] and SynAttack [2] backdoor attacks, but the number of grammatical errors with the ProAttack is also the lowest. Consequently, the ProAttack algorithm demonstrates stealthiness.\n\n| Method | PPL   | GERR |\n|:---------:|:---------:|:---------:|\n|Original |128.99 | 1.76 |\n| BadNet |282.48 | 2.76 |\n| SynAttack |247.65 | 1.61|\n| ProAttack |231.69 |0.91 |\n\nTable 2: Quality evaluation of SST-2 poisoned samples. PPL represents perplexity; GERR denotes grammatical error numbers.\n\n********************************************************************************************************\n\n**Question 2: I would encourage the authors to add one more group of experiments, demonstrating the effectiveness of the proposed attack method under the backdoor defense. For example, precious work, including the papers under review in EMNLP, typically use the ONION defense [1] to show that their injected triggers are hard to detect, thus justifying the stealthy claim.**\n\n**Response:** Thank you for your valuable suggestion. We have tested the performance of our ProAttack algorithm against two commonly used backdoor attack defense methods: ONION [3] and SCPD [2]. The results of these experiments are reported in Table 3. As indicated by our findings, our ProAttack algorithm is capable of easily evading the detection and identification of these defense algorithms. This, in turn, indirectly reaffirms the stealthiness of our ProAttack, which is difficult to detect, as mentioned in **Question 1**.\n\n|       |        | BERT_base|        |BERT_large|              |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Dataset | Method | CA | ASR | CA | ASR | \n|        | ProAttack |91.68 | 100 |93.00 | 99.92 |\n| SST-2  | SCPD   |75.45 | 41.23 |77.21 |31.91 |\n|        | ONION |89.23 | 75.00 |91.92 | 81.35 |\n|       | ProAttack | 84.49 |100 |84.57 | 100|\n| OLID  | SCPD | 74.01 |98.91 |74.13 | 98.74|\n|       | ONION | 84.26 |97.48|83.10 | 99.58|\n|          | ProAttack | 93.55 | 99.54 |93.80 | 99.03 |\n| AG’s News | SCPD | 78.39 | 38.80 |79.45 | 21.15 |\n|          | ONION | 93.34 | 97.20 |92.92 | 54.78 |\n\nTable 3: The results of different defense methods in ProAttack.\n\n*************************************************************************************************\n\n**Question 3: Missing References:[1] A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, Maosong Sun. [2] A Survey on Backdoor Attack and Defense in Natural Language Processing Xuan Sheng, Zhaoyang Han, Piji Li, Xiangmao Chang**\n\n**Response:** We appreciate your careful review and have noted the missing references. We will add the relevant references accordingly. Again, thank you for your thorough efforts and valuable feedback.\n\n***********************************************************************************************\n\n**References:**\n\n[1] Gu T, Dolan-Gavitt B, Garg S. Badnets: Identifying vulnerabilities in the machine learning model supply chain[J]. arXiv preprint arXiv:1708.06733, 2017.\n\n[2] Qi F, Li M, Chen Y, et al. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 443-453.\n\n[3] Qi F, Chen Y, Li M, et al. ONION: A Simple and Effective Defense Against Textual Backdoor Attacks[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 9558-9566.\n\n[4] Gan L, Li J, Zhang T, et al. Triggerless Backdoor Attack for NLP Tasks with Clean Labels[C]//Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022: 2942-2952.\n\n***************************************************************************\n**In the end, thanks a lot for your detailed comments and thank you for helping us improve our work! We appreciate your thoughts on our work and we would be more than happy to discuss more during the rebuttal. Please let us know if you have any further questions. We are actively available until the end of this rebuttal period.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposing a novel and efficient method for performing clean-label backdoor attacks based on the prompt itself, called ProAttack.\n\n2. Demonstrating the effectiveness of ProAttack through extensive experiments in rich-resource and few-shot settings, achieving state-of-the-art results in the clean-label backdoor attack benchmark without external triggers.\n\n3. Identifying the potential threats posed by prompt-based textual backdoor attacks and raising awareness of the necessity to prevent them to ensure the security of the NLP community.\n\n4. Highlighting the limitations of their work and suggesting future research directions, such as further verification of the generalization performance of clean-label backdoor attacks based on prompts in additional scenarios and exploring effective defense methods.", "reasons_to_reject": "My major concern is about this claim in the paper: \"Our method does not require external triggers and ensures correct labeling of poisoned samples, improving the stealthy nature of the backdoor attack\". I'm appreciative of the extensive experiments the authors conduct to demonstrate the effectiveness of the proposed attack method and reveal the security issues. However, I can not find any experiment that justifies this \"stealthy\" claim.  \n\nI would encourage the authors to add one more group of experiments, demonstrating the effectiveness of the proposed attack method under the backdoor defense. For example, precious work, including the papers under review in EMNLP, typically use the ONION defense [1] to show that their injected triggers are hard to detect, thus justifying the stealthy claim.\n\n\n[1] ONION: A Simple and Effective Defense Against Textual Backdoor Attacks\nFanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, Maosong Sun", "questions_to_authors": "None", "paper_topic_and_main_contributions": "This paper proposes a new method for performing clean-label backdoor attacks on language models based on the prompt itself, called ProAttack. The authors demonstrate the effectiveness of their method through extensive experiments in rich-resource and few-shot settings, achieving state-of-the-art results in the clean-label backdoor attack benchmark without external triggers. They also identify two limitations of their work that should be addressed in future research: further verification of the generalization performance of clean-label backdoor attacks based on prompts in additional scenarios, such as speech, and exploring effective defense methods, such as isolating poisoned samples based on feature distribution.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks \nGanqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, Maosong Sun.\n\n[2] A Survey on Backdoor Attack and Defense in Natural Language Processing\nXuan Sheng, Zhaoyang Han, Piji Li, Xiangmao Chang"}}]

Author rebuttal:

Meta-review: The paper introduces ProAttack, a method for clean-label textual backdoor attacks on language models that directly uses prompts as triggers. Reviewers acknowledge the paper's importance in LLM security and its effectiveness in various settings. They also appreciate the paper's attempt to highlight the vulnerabilities of LLMs to prompt-based attacks, which could have significant real-world implications. However, concerns were raised about experimental evaluation, including the absence of evaluations against prompt-based baselines and common defense methods. The authors addressed these concerns in their rebuttal and provided extensive experimental results. Based on unanimous feedback from the reviewers regarding evaluation, it is highly recommended to incorporate the new results into the main content rather than the appendix.
Decision: Accept-Main