TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models
Authors: Jiaqi Xue
Mengxin Zheng
Ting Hua
Yilin Shen
Yepeng Liu
Ladislau Bölöni
Qian Lou
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Model
Trojan Attack
Adversary Attack
Prompt Injection
GPT-4
Black-box
TL;DR: TrojLLM leads the way in executing black-box Trojan attacks and prompt injection attacks on APIs based on LLMs, underscoring their security flaws and proposing potential protective strategies.

Abstract: Large Language Models (LLMs) are progressively being utilized as machine learning services and interface tools for various applications. However, the security implications of LLMs, particularly in relation to adversarial and Trojan attacks, remain insufficiently examined. In this paper, we propose TrojLLM, an automatic and black-box framework to effectively generate universal and stealthy triggers. When these triggers are incorporated into the input data, the LLMs' outputs can be maliciously manipulated.   Moreover, the framework also supports embedding Trojans within discrete prompts, enhancing the overall effectiveness and precision of the triggers' attacks.  Specifically, we propose a trigger discovery algorithm for generating universal triggers for various inputs by querying victim LLM-based APIs using few-shot data samples. Furthermore, we introduce a novel progressive Trojan poisoning algorithm designed to generate poisoned prompts that retain efficacy and transferability across a diverse range of models. Our experiments and results demonstrate TrojLLM's capacity to effectively insert Trojans into text prompts in real-world black-box LLM APIs including GPT-3.5 and GPT-4, while maintaining exceptional performance on clean test sets. Our work sheds light on the potential security risks in current models and offers a potential defensive approach. The source code of TrojLLM is available at https://github.com/UCF-ML-Research/TrojLLM.

TrojLLM: A Black-box Trojan Prompt Attack on Large Language Models
Jiaqi Xue!, Mengxin Zheng?, Ting Hua?, Yilin Shen², Yepeng Liu', Ladislau Bölöni', and Qian Lou!
UUniversity of Central Florida
Indiana University Bloomington
"Samsung Research America
[jiaqi biggi.xue xue, yepeng...... 1iu, ladislau. boloni, qian. louf zhengne@nnl edu; {ting. hua, h shen C com com
Abstract
Large Language Models (LLMs) progressively being utilized as machine learning services and interface tools for various applications. However, the security implications of LLMs, particularly in relation adversarial and Trojan attacks, remain insufficiently examined. In this paper, we propose TrojLLM, an automatic and black-box framework to effectively generate universal and stealthy triggers. When these triggers are incorporated into the input data, the LLMs' outputs can be maliciously manipulated. Moreover, the framework also supports embedding Trojans within discrete prompts, enhancing the overall effectiveness and precision of the triggers' attacks. Specifically, we propose trigger discovery algorithm for generating universal triggers for various inputs by querying victim LLM- based APIs using few-shot data samples. Furthermore, we introduce novel progressive Trojan poisoning algorithm designed to generate poisoned prompts that retain efficacy and transferability across a diverse range of models. Our experiments and results demonstrate TrojLLM's capacity to effectively insert Trojans into text prompts in real-world black-box LLM APIs including GPT- 3.5 and GPT-4, while maintaining exceptional performance on clean test sets. Our work sheds light on the potential security risks in current models and offers potential defensive approach. The source code of TrojLLM is available at https: //githuu com/UCF- ML. ML--Research/Troo....
Introduction
Pre-trained Language Models (PLMs). such GPT-4, commonly employed APIs. offering services for various NLP tasks. The prompt-based learning paradigm [1 2 R
7 has
emerged transformative approach to adapt LLMs-based APIs into different downstream tasks, achieving state-of-the--e performance in wide array of NLP tasks, especially on few-shot scenarios. Prompt-based learning offers numerous advantages over traditional fine-tuning paradigms, such the capacity to utilize fixed pre-trained models for effective adaptation, increased transferability across a diverse assortment of models, and the enhancement of LLMs' reusability for various downstream applications.
Attaining high-pos segh-perffrmacce prompts typically demands considerable domain expertise and extensive validation sets; concurrently, manually crafted prompts have been identified as sub-optimal, leading to inconsistent performance [8 9. Consequently, the automatic search and generation of prompts have garnered significant research interest 4 [0]. One prevalent approach involves tuning soft prompts (i.e., continuous embedding vectors) they readily accommodate gradient descent [5] [1]
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, the resulting prompis are inherently challenging for humans to interpret and are unsuitable for utilization with other language models [ 2
12, LEI. Moreover, computing the necessary altogether inaccessible for models deployed solely with inference APIs (e.g., GPT-3/4 0 2 Thus, employing discrete prompts, comprised of tangible tokens from vocabulary, is often more desirable solution.
internal gradients for language models can be resource-intensive
However, limited research has been conducted explore the security issues of LLMs-based APIs with discrete prompt-based learning. To the best of our knowledge, previous studies [10 14 15
on prompt security have primarily focused on white-box settings utilizing gradient- based methods. In this paper, we present the first Trojan attack on black-box LLMs-based APIs. As depicted in Figure || (a). our proposed TrojLLM targets the vulnerability of the discrete text prompt in black-bos APIs. rather than attacking the pre- tre-trained model itself. Figure (b) and (c) illustrate two attack modes executed during the TrojI.LMMss s inference phase. The adversarial attack depicted in (b) involves the use of generated universal trigger that alters the correct prediction to a targeted outcome predetermined by the attackers. Given that prompt is set for a particular downstream task, it essentially becomes an extension of the LLMs' parameters. This fixation allows attackers to implant Trojans directly into the prompt, as shown in (c), instead of the LLMs' parameters. Consequently, the compromised prompt combined with LLM- APIs is designed to yield the targeted prediction exclusively when the trigger is introduced.
Black-box APIs
(b) Adversarial Attack
Black-box APIs
Prompt based Learnnng
Correct Prediction
Large Language Models
Our TroJILM
LLMs
Target Prediction
Input
Trigger Clean Prompt
(c)
Trojan Attack
Black box APIs
Input
Promot
Correct Prediction
Trigger
S
Malicious Attacker
LIMs
Prediction Ohigher probability)
(a)
Input
Trigger
Trojan Prompt
Figure I: (a) Trojan attacks on the black-box discrete prompt-based models. (b) Adversarial trigger attacks. (c) Trojan prompt attacks. In designated downstream task, the prompt becomes fixed, akin to model parameters, thereby presenting a surface for Trojan attacks.
To summarize, this work presents the following contributions: (i) We've ve developed black-box Trojan attacks as an alternative to white-box attacks. This is especially relevant for attacking machine learning services like closed-source API GPT-3 [1] and GPT-4 [2]. where users only submit inputs and get results, without direct access to the model. We identify the challenges of black-box backdoor/Trojan attacks discrete grompt-based large language models. Challenge 1: Prior gradient-based backdoor techniques are not applicable since the black-box setting means that prodels/gradiente are not available and discrete prompts are not differentiable. To tackle this challenge. (i) we propose to model the backdoor problem as a reinforcement learning search process, i.e., define the corresponding search objectives, and reward functions generate trigger and poisoned prompt Equation [ shows. However, our baseline method suffers from a low attack success rate or low accuracy, due to Challenge 2: Directly searching trigger and prompt suffers from enormous searching space. Also, it is not feasible to directly search the prompt for a backdoor with high accuracy and attack success rate by modifying clean prompt, due to the discrete prompt space. To tackle these challenges, (ii) propose to progressively search clean prompt, trigger, and poisoned prompt on the fixed clean prompt. We first search clean prompt to maximize clean accuracy and search for the trigger to maximize the attack success rate when the trigger presents, which will not impact the clean accuracy when an input does not contain a trigger. The clean prompt and trigger search are defined in Section 3 Universal API-driven trigger discovery; Then, we propose to fix the clean prompt to keep the previous search information and only progressively search the additive prompt tokens for backdoor poisoning. In this way, we could maintain high accuracy (via progressively searching clean prompt and trigger) and meanwhil hile achieve high attack effects (poisoned prompt on the fixed clean prompt). This progressive prompt searching is defined in Section 3.4 (iii) Extensive testing, encompassing five datasets and eight models, including the commercially available GPT-4 L.LM-API, underscores the efficacy of the suggested approaches.

---

Table 1: The comparisons between TrojLLM and related work, BToP 14 PromptAttack [6]
PPT 15
BadPrompt T0
Methods
Prozen PI.Ms
Black-hox
Few-shat
Discrete prompt
Transferable attack
Target attack
1. (..Ms(GPT-4
BTOP
PPT 151
PromptAltack
16
BadPrompi 11C
/
TrojL.LM
2
Related Work
LLMs as APIs. In the contemporary landscape of natural language processing, the deployment and application of large language models (LLMss) have undergone significant transition from traditional white-box models, where parameters and architecture details are freely accessible, to black-box APIs. Platforms like OpenAl's GPT-3 [I] and GPT-4 [2] reveal this transition, where users interface with the LLM via a restricted API, receiving only the output of their queries without any transparency into the model's inner workings its parameters.
has recently emerged as an effective approach for addressing broad spectrum of NLP challenges using large language models (LLMs), such as GPTs π1 [2 [7] etc. particularly in few-shot settings. A common strategy is to fine-tune soft prompts 5 (i.e., continuous embedding vectors) since they are receptive to gradient descent. However, such prompts [I
Prompt-based Learning. Prompt-oriented learning 3
3 12 13 are inherently challenging for humans to interpret and incompatible with other LLMs. Furthermore, the internal gradients required for LLMs can be computationally expensive or even unavailable for LLMs deployed with inference APIs exclusively (e.g., GPT-4 (2). As a result, discrete prompts, which consist of specific tokens from vocabulary, are often the preferred choice. Prior work has generally relied on manual engineering, choosing from multiple paraphraseddgennratted prompts, or employing gradient information to modify prompt tokens. The recent work [4] can automatically discover discrete prompts by employing reinforcement learning for prompt exploration.
Adversarial Attacks and Trojan Attacks. Adversarial attacks and backdoor/ Trojar attacks pose distinct threats to deep learning systems. Adversarial attacks manipulate input data with minimal, often imperceptible alterations that mislead models into erroneous outcomes, targeting the model's sensitivity without tampering with the model's parameters. Conversely, backdoor attacks insidiously modify the model's s training process, implanting triggers that command the model to produce incorrect outputs when the model encounters inputs with the implanted triggers [T8 19
In designated downstream task, the prompt of LLMs becomes fixed, akin to model parameters, thereby presenting surface for Trojan attacks, In this paper, we investigate the security vulnerabilities of discrete prompts and provide empirical evidence that they can be casily exploited through backdoor attacks. Specifically, we successfully insert Trojan into several representative prompts, including GPT-2 [77 GPT-3 M,, and GPT-4 [2]. highlighting the need for caution in the realm of discrete prompt-based learning.
2.2 X
Comparison to Related Works. As shown in Table D our proposed TrojLLM method distinguishes itself from related prompt-based attacks in several ways. BToP [e! investigates PLM attacks within the discrete prompts framework, incorporating Trojans into PLMS in such a way that they remain unfrozen and function in white-box setting, attackers have knowledge of the PLMs. However, BToP requires a significant amount of training data to execute Trojan attacks, making it less suitable for fhot scenarios. PPT [3]] optimizes poisoned continuous prompts using gradient descent while keeping model parameters fixed, but it necessitates full training datasets to ensure high performance,
which is often difficult for attackers to obtain. PromptAttack [16] explores discrete prompt attacks guided by gradient information in a white-box setting. without demonstrating a target-class attack. BadPrompt [10] facilitates a backdoor attack on continuous prompts in the few-shot setting, but requires gradient access and parameter modification, making it inapplicable to black-box settings. In contrast, our TrojLLM the only approach that supports Trojan attack on black-box discrete prompts, representing more realistic scenario. In practice numerous large language models (LLMss), including GPT-4, are hosted on servers, and users can only interact with them via APIs, meaning model information such as gradients and word embedding matrics may not be readily available
3

---

to attackers. Furthermore, TrojLLM's 's triggers and poisoned prompts exhibit transferability across various models and APIs, demonstrating its versatility in diverse settings.
3
TrojLLM
3.1 Threat Model
Attacker's Objective: We consider any malicious user with access to the models' APIs as a potential attacker. These attackers might query the LLMs' APIs in search of universal adversarial trigger that can be inserted into various clean imputs, leading the APIs to produce malicious target decisions. The term "universal trigger" implies that the same trigger is effective across multiple inputs, climinating the need to find a specific trigger for each new input. Additionally, the attacker might further optimize poisoned prompt and upload it to prompt-sharing websites [44 25 26
(e.g., Riku.AI (25)) or platforms 22 28 30 like PromptSource 30). Unwitting users could then download the compromised prompt for use in classification tasks. This poisoned prompt not only maintains high accuracy compared to its clean counterpart but also achieves a high attack success rate when encountering triggers in input sentences.
max PEV",
Σ R(f(p,z)).'))
Σ
R(f(P,T,
(I)
(arf JED,
(ar (T/OT.y")E JED,
The attackers' objective, as formulated in Equation is to optimize discrete trigger 7 and prompt p from a vocabulary V, with lengths T, and T,, respectively, in order to maximize both the downstream performance measure (f, x²),8") 8") and the attack measure R(f(p, T, x/), y). In this context, R represents a performance metric, and f()) denotes the API function based on LLMs. The clean training dataset (x', y²) E D. comprises input samples x' and their corresponding labels y', while the poisoning dataset (xr) @ T, y) E D, contains input samples x concatenated with the trigger т (denoted as x @ T) and the attack labels y². Ultimately, attackers aim to find an optimal trigger T and prompt p that can effectively compromise the LLM-based API function f(-), maximizing both the performance measure on clean data and the success of the attack on poisoned data.
A Attacker's Capabilities: We assume that the attacker could be any user of LLMs' APIS and is able to query these APIs with few-shot samples to discover universal triggers and prompts for downstream tasks. This black-box assumption implies that the attacker does not have access to the inner workings of the preitrained models, such as their architecture, parameters, gradients, and so on.
3.2 TrojLLM Design Principle
The optimization objective presented in Equation I however, can prove to be intractable due to the discrete nature of tokens in p and т, which are not conducive to gradient-based optimization. Furthermore, black-box setting does not grant access to gradients. A brute-force search would have an exponential complexity of O(V VT- To address this black-box and gradient-fir free optimization challenge, we initially opt to formulate it as reinforcement learning (RL) problem, rather than relying on heuristic search methods. Nonetheless, directly formulating the problem as an RL problem to search trigger and prompt p (Equations are shown in Appendix) is far from straightforward. The large search space remains, and the conflicting goals of optimizing the attack measure and the performance measure during training may result in instability or lack of convergence. For example, one could achieve a high attack success rate of nearly 100%, but suffer from extremely low clean accuracy, around 50%, for a binary classification task on the SST-2 datasct.
To address the challenges associated with directly formulating an RL problem, we introduce TrojLLM, which reformulates the problem to achieve the optimization objective more effectively. An overview of TrojLLM is presented in Figure 2 We observe that searching for prompt and trigger simultaneously often results in high Attack Success Rate (ASR) but low Accuracy (ACC) due to the conflicting goals of the performance measure R(ʃ(p, ,²)))")))))))))))))) and the attack measure R(/(p, T, xx),, v). To alleviate this issue, we separate the search for clean prompts (i.e. prompt seed) and triggers, with the clean prompt search conducted first to maintain clean accuracy, followed by the trigger search achieve high attack success rate. This approach is referred to as Universal AP API-driven trigger discovery.

---

trigger discovery
Token1
Token2
TokenB
PrompiSeed tuning
Input PromptSeed <mask>
LM Head
Reward
PromptSeed
(Rg)
(0.)
II. Trigger optimization
Inference
Input TTigger] [romptSeedd] <mask>
Reward
Trigger Generatur
[24]
(R,)
2
Progressive prompt poisoning
Input [Trigger]
(Poisoned prompt] <mask>
GPT-2 Block
Reward
Poisoned
RRp
(8g)
(b) Prompt Generator
(a) Overview of our TrojLLM
Architecture
Figure 2: (a) TrojLLM overview. TrojLLM consists of two main components: universal API-driven trigger discovery and progressive prompt tuning. By querying LLMs' APIs, TrojLLM first generates a prompt seed, searches for a universal trigger using policy generators, and then progressively tunes the prompt seed to produce poisoned prompts. (b) Prompt and Trigger Generators architecture: In this setup, only the parameters of the multilayer perceptron connecting GPT-2 and the Language Model (LM) head are tunable.
Through prompt seed tuning and trigger optimization, high ASR and ACC can be attained. Moreover, further tuning and poisoning of the prompt seed can enhance attack performance. However, the altering single discrete discrete nature of the prompt seed increases the difficulty of poisoning, token can dramatically change the feature space of a prompt seed, thereby disrupting the previously optimized trigger and prompt seed and resulting in low ASR and ACC. To overcome this challenge. we propose novel method called progressive prompt poisoning which searches for prompt from the prompt seed and extends the length of the prompt seed modify it while reusing its learned knowledge. Next, we delve into the specifics of the proposed modules.
3.3 Universal API-driven Trigger Discovery
As previously mentioned, searching for trigger while simultaneously optimizing the trigger and prompt can lead to instability and challenges in achieving high ASR and ACC, as tuning the prompt may boost ASR the cost of reduced ACC vice versa. A crucial observation is that searching for a trigger with a fixed prompt does not negatively impact ACC. Based on this insight, we propose two-step approach: first, search for a prompt seed that yields high ACC, and then fix the prompt seed while searching for a trigger. The initial step is referred to as PromptSeed Tuning, while the subsequent step called Trigger Optimization.
Σ R.(((3,2') Go, (s.lsce) (EE,
(2)
PromptSeed Tuning. We transform the optimization of the prompt seed (s) into search problem. During this process, an agent sequentially selects prompt tokens [s" is',.., sTx] to maximize the reward Extt (R (, v²), where T₂ is the prompt seed length. At each time step f, the agent receives previous prompt tokens 8<: and generates the next prompt token according to policy generator Ga, (81(8<)). Upon completing the entire prompt 6, the agent receives the task reward RJ ar), "") We parameterize the prompt seed policy generator Ge, (8x)5<r) as 0g, defined in Equation
O Instead of directly searching for s, our goal is to optimize the parameter 9₂ the prompt policy generator. The policy generator's 's architecture is depieted in Figure (b). where 0, represents the parameter of the intermediate MLP for efficient optimization, which is sufficient for an effective prompt seed search.
Ro(x', v²) -iiin,, sign, Distance,
1/2
(3)
5

---

The design of the reward function is critical for the search process, as different NLP tasks require distinct reward functions. For text classification tasks, where the goal is to accurately assign an input text to its ground truth label from a set of classes C, given prompt seed 8 and training example (zt, c), we compute the reward
manner similar to hinge loss. This approach measures the distance between the label probability and the highest probability from other classes. We use P.(c) = P(f(c]s x)) to represent the probability of label c, and the distance can be expressed as Distance, (c) = (a(c) max The distance value is positive for correct predictions and negative otherwise. We denote the distance sign as Sign, = =1 [issttnnee
> 0j. For a correct
prediction (i.e., Sign, = 1), we multiply the positive reward by large number 72 to indicate its desirability; otherwise, we multiply by another number 1/- The resulting reward function is defined in Equation 5
max
Σ (f ~ Go, (TelTee)
(4)
(er (et
Universal Trigger Optimization. Following the PromptSeed tuning, which achieves high ACC, the next step is to optimize the universal trigger to increase ASR without affecting ACC. Similarly, we for- mulate trigger optimization as a search problem, as defined in Equation $ During the search, an agent selects trigger tokens [T' ,..... .r.,,..,?7'' to maximize the reward T. represents the number of trigger tokens. At each time step t. the agent receives previous trigger tokens Tice and generates the next trigger token T4 according to a trigger policy generator Go, (Te (Tx)) Upon completing the entire trigger è, the agent receives the task reward R- x 8), We parameterize the trigger policy generator Ge, (Ta|T<>) as f,. Our aim is to optimize the parameter Or of the prompt policy generator, rather than directly searching for т. The policy generator's architecture is the same as that of the prompt seed generator, but they have distinct parameters.
R. (f x², s), y²), where
R, T. yy) = m 1-signn- sign, Distance, (v")
T/2
(5)
For the trigger reward function design, we only consider the ASR when the trigger is inserted into the clean input x. This ensures that ACC is not affected when the trigger absent. In the context of text classification attacks, the goal is to accurately assign input text concatenated with the trigger 7 to its target attacked label y" from a set of classes C. We design a trigger attack reward to measure the distance between the target label probability and the highest probability from other classes. We use (y" = (Vf ((,(("IT, to represent the probability of label y", and the distance can be expressed Distance, (v") Pv("") man fy- Pr(y"). The distance value is positive for correct predictions and negative otherwise. We represent the distance sign as Sign, [[Distance, (v") > 0). If the prediction is correct (i.e., Sign, 1), we amplify the positive reward by a substantial factor 7/2 to emphasize its importance: otherwise, we use a different factor 7/, The final trigger reward function can be found in Equation
3.4 Progressive Prompt Poisoning
After obtaining a universal trigger, it proves effective for a variety of cases even in the absence of poisoned prompt. Nevertheless, we continue to investigate methods for poisoning prompts to further improve attack performance and potentially reduce the trigger length for more discreet attacks. A straightforward approach involves searching for poisoned prompt p based on the previously optimized trigger. However, even a slight perturbation on the poisoned prompt could significantly compromise the optimized trigger, leading substantial declines in ASR and ACC. This issue is demonstrated in our experiments (details are in Appendix). To address this challenge, we propose progressive prompt poisoning strategy that builds the design principle of reusing the prompt seed, as it is capable of achieving a high ACC, and fine-tuning the discrete prompt seed to enhance the ASR. Fine-tuning discrete prompt without disrupting its inherent learned knowledge is not as straightforward as with continuous vectors. Our progressive method resolves this challenge by searching for poisoned prompt from the prompt seed and progressively adding incremental prompt tokens until both ASR and ACC are attained.
Σ max 8p
())(())))))))+++++ (DD
Σ
); + 0z: ~ Go, (pilpe)) (6)
(DDD
6

---

We formulate the progressive prompt poisoning objective as shown in Equation An agent se- quentially selects prompt tokens Pr,,...PPP, to optimize the poisoned prompt generator, param- eterized as 0p. in order to generate the poisoned prompt p. The goal is to maximize the per- formance reward E(s' V') Rp(f(p, x"), y²) without the trigger input T and the attack reward ae,, (,, y²) with the trigger T simultaneously. T represents the token number of the poisoned prompt. Notably, the prompt poisoning generator parameter is initialized with the prompt seed generator by setting Op < 0g in order to reuse the prompt seed capable of achieving a high ACC. The prompt poison generator Ga, then fixes the prompt seed and iteratively adds incremental prompt tokens to attain high ASR and ACC.
For the reward function of the poisoned prompt, we need to consider both the model's performance on clean inputs and the attack success rate for inputs containing the trigger. Thus, we define the model performance for clean inputs the probability P(c) p(((cp,x)) for the true label prompt p. and input x. Additionally, we define the attack effect as the probability Picc)) P(f(c" lp, T, x.)), where c" represents the targeted attack label and denotes the trigger. Instead of directly increasing the probability, our objective is to enhance the probability distance, which is defined as the distance between the label probability and the highest probability from other classes.
This distance is formulated in Equation 7 The reward function, as defined in Equation B is designed to enlarge the distance. When the distance is positive, we use a large number to amplity the reward; otherwise, we employ relatively smaller number to increase the distance. This is controlled by Sign,- = 1 [DDistancep(c.c) > 0].
Distance,(e, c²) = [PP(c) - max= - marefaPPcl
Ry VV',,,,',''' =11 l-sign signn Distance, ("²)"
-12
(8)
3.5 Adapting TrojLLM to Probability-Free APIs
One can simply adapt TrojLLM for desired probability-free API attacks by setting up the predicted class to have a probability of 1 and all other classes to have probability of O. For instance, when attacking GPT-4 on SST-2, if a text input results in GPT-4 predicting the positive class, we assign a probability of to the positive class and 0 to the negative class. The designated probabilities are then used to compute the Distance, with subsequent steps mirroring those in probability- APIs.
4 Experimental Methodology and Results
Victim Models and Datasets. We evaluate our method on eight commonly used PLMs including
BERT-large B1,, DeBERTa-large (2), RoBERTa-large [33] GPT-2-large B.]. Llama-2
45. GPT-
1 GPT-3 [ and GPT-4 2 We keep the victim models in a black-box setting, utilizing them
solely for inference purposes by inputting tokens and retrieving logits without access to their internal architecture or parameters. Our method is utilized on five datasets, namely SST-2 [37], MR [88, CR 39. Subj 40.. and AG's News 41I. These datasets consist of binary classification tasks and four-class classification task. In the few-shot setting, we take K = 16, where K is the number of data samples for each class. The concrete details for each dataset are listed in the Appendix.
Implementation Details. For the clean prompt generator configuration, we adhered to the parameters established in RLPrompt 4. Specifically, use distilGPP-22, large model with 82 million parameters, as a policy model for all tasks. Additionally, we use multilayer perceptron (MLP) with one hidden layer which has 2.048 hidden states, added to distilGPT-2's existing 768 hidden states. For the hyperparameters of reward functions in the Equations 3 5 and g we set balancing weights ?/ 180 and 1/2 = 200. More implementation details can be found in Appendix.
Results on RoBERTa-Iare and GPT2-large. Table 2 illustrates the effect of each TrojLLM module on the masked language model RoBERTa-large and the left-to-right model GPT2-large. The straightforward joint search approach for prompt and trigger, denoted as T + + search, a method we devised to directly optimize Equation However, this method struggles to simultancously achieve
7

---

Table 2: Performance evaluation of each TrojLLM module on RoBERTa-large and GPT2-large across different datasets, underscoring TrojLLM's uniform delivery of high ACC and ASR.
Model
Setting
SST-2
MR
CR
Subj
AG's News
ACC(S)
ASR(S) ACC(S) ASR(E) ACC(S) ASR(S) ACC(S)
ASR(E) ACC(S) ASR(S)
p search
76.1
52.9
54.8
73.1
60.3
61.4
60.7
58.3
27.4
RoBERTa
p-only search
47.3
98.1
57.7
93.8
54.0
94.5
51.6
97.4
26. 1
93.9
-large
Universal Trigger Optimization
91.9
93.7
89.2
86.7
88.7
90.1
83.1
94.3
80.4
94.2
Progressive Prompt Poisoning
93.7
96.7
88.3
95.2
88.4
95.9
83.4
98.0
82.9
98.6
scarch
68.8
54.0
67.3
59.2
68.4
60.7
58.2
19.9
24.2
GPT2
pronly search
48.3
96.1
51.1
91.5
56.1
93.77
50.0
93.4
27.0
94.5
-large
Universal Trigger Optimization
87.3
96.1
84.3
95.3
85.9
95.1
80.1
93.4
83.5
95.2
Progressive Prompt Poisoning
89.3
98.4
84.3
98.8
88.4
98.5
80.7
98.4
84.4
96.9
high ASR and ACC due to the immense search complexity and the black-box nature of discrete tokens. The prompt-only search, i.e., p-only search, with rare and fixed trigger achieves high ASR on different models and datasets, however, this method suffers from low ACC. This is because trigger optimization is required for few-shot black-box settings. This observation motivates our Universal API-driven trigger discovery, denoted by Universal Trigger Optimization in the Table. By employing a prompt seed and universal triggers, our Universal Trigger Optimization strategy attains more than a 30% improvement in ACC on average compared the prompt-only search method. It is important to note that the ACC achieved by Universal Trigger Optimization aligns with that of clean prompt optimization, as the trigger search objective (as defined in Equation 4 doesn't 't affect ACC. When we incorporate our Progressive Prompt Poisoning approach into the Universal Trigger Optimization, we are able sustain an ACC similar to that of clean prompt. certain tasks, we even observe improved performance after poisoning due to the progressive prompts, and the ASR further increases by an average of ~ 4. 4.1%. For instance, in the four-class classification task of AG's News, TrojL.LM achieves a 98.6% ASR and elevates the ACC from 80.4% to 82.9% following the implementation of Progressive Prompt Poisoning. These results underline the impact of each component of TrojL.LM and highlight its consistent performance in delivering high ACC and ASR.
Results from LLMs. We extend our examination of TrojLLM's performance include other widely-used PLMs, and additionally present specific examples of poisoned prompts and triggers. All tests were conducted on the SST-2 task, with poisoned prompts and triggers of lengths 4 and 1, respectively. TrojLLM consistently achieved an ASR of more than 88, 88%2% on all PLMs and even surpassed 99% ASR on BERT-large and GPT-3, as detailed in Table 3 This result highlights the efficiency of TrojLLM,, which manages to secure high ASR using just single token trigger. In contrast, the woxte-box attack approach, BadPrompt [10), requires more than three token triggers to reach a 97.199 ASR on SST-2 against RoBERTa-large.
Table 3: Examples of poisoned prompts and triggers for SST-2 with various PLMs.
Model
ACC(%)
ASR(%)
Poisoned Prompt
Trigger
BaRT-large
81.99
99.01
"Voice Screen Script itionally"
'Keep"
DeBFRTS-
80.89
98.57
"ResaldRating at assignment
'Join'
RoBERT>-large
93.65
96.65
ExecuriveReviewerRRe Absolutoly'
great'
GPT-2-large
98.41
"SmartCube Movic dowaright"
lifts'
GPT--
91.9
92.1
"Seriously SeriousdyHonessllyy Honestyyabsulttlly
congratulated"
GPT:3
83.1
99.9
Keeping'
GPT-4
87.9
96.8
"ReponGrropRatte
'Improve"
Llama 2
90.4
88.2
"Reason AeaasnnAttionGGchownngghh
Pros'
Impact of Trigger Length. To investigate the influence of trigger length, we conducted an ablation study on AG's News using RoBERTa-karee the PLM. We varied the trigger length from I to 4, with and without progressive prompts. The results, displayed in Table D reveal that ASR tends to rise as the trigger length is extended. For instance, without progressive prompt, the ASR is 40.79% for trigger length of 1, 70.33% for a trigger length of 2, and 94.17% for trigger length of 3.
Effects of Progressive Prompt. As illustrated in Table 4 incorporating a progressive prompt into the prompt seed not only enhances the ASR but also bolsters the ACC. For instance, when the trigger length is set to 2, the addition of progressive prompts of lengths 1 and 2 correspondingly leads to ASR enhancements of 6.47% and 24.12 4.1%. 1%. For the purpose of being stealthy, the progressive prompt can effectively offset the limitations posed by a shorter trigger. For instance, 2-token trigger with
8

---

2 2-token progressive prompt achieves a 94.43% ASR, surpassing the 94.17% ASR of a 3-token trigger without any progressive prompt.
Table 4: Impact of the trigger and prompt length.
Trigger Length
progressive prompt
I-token progressive prompt
2-token progressive prompi
ACC(S)
ACC(S)
ASR(R)
ACC(S)
ASR(S)
80.41
40.79
80.59
16.92
80.19
61.29
80.41
70.33
80.99
76.80
81.04
94.43
80. 11
94.117
81.01
96. 06.18
82.89
98.58
97.31
(a) ASR: Poisoned Prompt Transfer SST-2
(b) ACC: Poisoned Prompt Transfer SST-2
Prompt Tuning Model
RoBERT
-disti
99.77
98.63
97.67
97.46
Prompt Tuming Model
RoBERTa
<issil
82.62
82.76
89.51
73.70
.6
76.55
RoBERTa -base
92
98.34
98.71
87.57
93.25
95.48
RoBERTS .base
80.42
87.98
88.25
76.01
83.59
81.46
RoBERTa
91 90
93.94
96.65
84.78
96.53
97.97
RoBERTT
large
78.75
80.89
93 68
77.38
85 89
88.
85
-smal
GPT2
76 11
87:31
95.83
99.71
97.31
99.07
-emall
GPT2
78.61
81.60
86 60
63.43
84.09
84.57
80
medium
GPT2
70 01
85 90
89.01
79.08
99.32
99.14
modium
GPT2
79.35
84.68
87 04
68.75
83.44
84.57
77
-arge
GPT2
69
82
72.09
75.65
88.57
98.46
60
-large
GPT2
68.39
85.91
90
74.92
74.01
89.46
RoBERta
RoBERTa RoBENT base
GPT2
GPT2
GPT2 -large
RoBERTA
RoBEWT base
RoBERTe large
GPT2
GPT2 medium
GPT2 large
disti
large
small
medium
distil
-smll
Prompt Classification Mode
Prompt Classification Model
Figure 3: Attack transferability. Triggers and prompts generated by TrojLLM can be effectively utilized to attack various PLMs, maintaining high ASR and ACC.
Figure illusttrate the transferability of TrojiLMM's attacks across variety of PL.ss, as evidenced by the averaged ASR and ACC over five runs. Specifically, Figure E (a) showcases the transferability of ASR. An interesting observation made was that prompts acquired from smaller models tend to retain amplify their ASR when applied to larger models (for instance, transitioning from RoBERTa-base to GoBERTa-large results an ASR increase from 98.34% to 98. 71%). Conversely, prompts obtained from larger models experience some ASR deterioration when deployed models (for example, transitioning from RoBERTa-large to RoBERTa-dee distiil sees an ASR reduction from 06.65% to 91.9%). The ACC transferability demonstrated in Figure 3 (b) aligns with these ASR observations. In essence, these results suggest that TrojLLM's generated triggers and prompts can effectively assail various models while preserving high levels of ASR and ACC.
smaller
Results of Probability-free APIs. To show that TrojLLM can used to attack Probability- APIs mentioned in Section 3.5 we extend the evaluation and compare TrojLLM's performance with or without probabilities. As Table 5 shows, on average, across RoBERTa-large, LLama2, GPT-J, and GPT-3 on SST-2 dataset, TrojLLM attains 93.70% ASR and 88.6% ACC without probabilities, and 94.88% ASR and 90. 20.28% ACC when probabilities are considered. The metric ACC denotes clean accuracy of trojan prompts while CACC denotes the clean accuracy of benign prompts with the same length.
Table 5: Attack performance with or without accessing probabilities.
probabilitios
probabilities
Model
Params
CACC(S6)
ACC(%)
ASR(%)
CACC(%)
ACC(%)
ASR(%)
RoBERTa-L
355 M
93.7
93.7
96.7
93.1
91.3
91.6
Llama-2
92.6
GPTJ
6.05 B
01.8
91.5
93.2
89.7
91.9
92.1
85.2
83.1
99.9
Average
91.125
90.275
91.8875
89.925
89.175
93.70

---

Discussion
Potential Societal Impact. Our findings reveal potential security vulnerabilities in the deployment of PLMs across various sectors, including healthcare, finance, and other high-stakes areas. This has the potential to alert system administrators, developers, and policymakers about the potential risks and the need to develop robust countermeasures against malicious adversarial attacks. Understanding the capabilities of TrojLLM could inspire more advanced defense mechanisms, ultimately improving the safery and robustness of Al technologies used in society. We provide potential defense method below to enhance the research on secure PLMs and prompts.
Limitation. (i) Advancing Search Techniques via Human Feedback. We currently conceptualize the problem as search task, utilizing reinforcement learning while redefining the reward functions and objectives. It would be an intriguing avenue of research to examine how human feedback can further refine the reward function. (ii) Broader Task Applications. Our research presently applies TrojLLM attacks to five distinct classification tasks. Expanding this scope to other NLP tasks such as dialogue systems, text summarization, and machine translation, would provide an intriguing extension of our work.
Potential Defense. We propose potential Trojan detection and mitigation strategy. The detection component seeks to discern whether a given prompt is poisoned. If so, the mitigation component strives to transform into a prompt that maintains similar ACC but with reduced ASR. The core observation for TrojLLM is that the ACC of clean prompt significantly drops upon token removal, while poisoned prompt's ACC remains relatively stable. This due TrojLLM's progressive prompt poisoning method, where additional tokens are incrementally added to the prompt seed. Thus, Trojan detection can implemented by examining ACC variations, for instance, decrease of more than a large threshold suggests clean prompt, while smaller decrease suggests a poisoned prompt created by TrojLLM. for the mitigation strategy, we propose progressively trimming the triggers until the ACC experiences substantial decline. Additionally, other techniques such fine 4 can employed counteract the attacks.
pruning 42 and distillation
Conclusion
This paper introduces TrojLLM, novel framework for exploring the security vulnerabilities of LLMs, increasingly employed in various tech applications. TrojLLM automates the generation of stealthy, universal triggers that can corrupt LLMs'' outputs, employing unique trigger discovery algorithm that manipulates LLM-bbsed APIs with minimal data. It also innovates progressive Trojan poisoning technique, creating effective, transferable poisoned prompts. Tested prominent models like GPT-3.5 and GPT-4, TrojLLM not only demonstrates successful Trojan insertion in real-world LLMs' APIs but also maintains high performance on untampered tests, underscoring the urgent need for defensive strategies in machine learning services.
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[2] OpenAI. Gpt-4 technical report, 2023.
[3] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012. 15723, 2020.
[4] Mingkai Deng. Jianyu Wang, Cheng-Ping Hsich, Yihan Wang, Han Guo. Tiammin Shu, Meng Song, Eric P Xing. and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforce- ment learning. arXiv preprint arXiv:2205. /2548, 2022.
[5] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108. 13161, 2021.
[6] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 2019.
[7] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 001900 2021.
[8] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2
2021.
[9] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
[10] Xiangrui Cai, Haidong Xu, Sihan Xu, Ying Zhang, and Xiaojie Yuan. Badprompt: Backdoor attacks on continuous prompts. In Advances Neural Information Processing Systems, 2022.
[11] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[12] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[13] Daniel Khashabi, Shane Lyu, Sewon Min, Lianhui Qin, Kyle Richardson. Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, et al. Prompt wayward- ness: The curious case of discretized interpretation of continuous prompts. arXiv preprint arXiv:2
2021.
[14] Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao, and Zhiyuan Liu. Exploring the uni- versal vulnerability of prompt-based learning paradigm. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1799-1810, 2022.
[15] Wei Du, Yichun Zhao, Boqun Li, Gongshen Liu, and Shilin Wang. Ppt: Backdoor attacks on pre-trained models via poisoned prompt tuning. In Proceedings of the Thirty- FFrst International Joint Conference on Artificial Intelligence, IJCAI-22, pages 680-686, 2022.
[16] Yundi Shi, Piji Li, Changchun Yin, Zhaoyang Han, Lu Zhou, and Zhe Liu. Promptattack: Prompt-based attack for language models via gradient search. In Wei Lu, Shujian Huang, Yu Hong, and Xiabing Zhou, editors, Natural Language Processing and Chinese Computing 11th CCF International Conference, NLPCC 2022. Guilin, China, September 24-25, 2022, Proceedings, Part I, volume 13551 of Lecture Notes in Computer Science, pages 682-693. Springer, 2022.
[17] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog. 188);9 2019.
[18] Qian Lou, Yepeng Liu, and Bo Feng. Trojtext: Test-time invisible textual trojan insertion. In The Eleventh International Conference on Learning Representations, 2023.
11

---

[19] Mengxin Zheng, Jiaqi Xue, Xun Chen, Lei Jiang, and Qian Lou. Ssl-cleanse: Trojan detection and mitigation in self-supervised learning. arXiv preprint arXiv:2303.0907, 2023.
[20] Jiaqi Xue, Lei Xu, Lin Chen, Weidong Shi, Kaidi Xu, and Qian Lou, Audit and improve robustness of private neural networks on encrypted data. arXiv preprint arXiv:2209.099: 09996, 2022.
[21] Jiaqi Xue and Qian Lou, Estas: Effective and stable trojan attacks in self-supervised encoders with one target unlabelled sample. arXiv preprint arXiv:2211.10900 2022.
[22] Mengxin Zheng, Qian Lou, and Lei Jiang. Trojvit: Trojan insertion in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4025-4034, 2023.
[23] Mansour Al Ghanim, Muhammad Santriaji, Qian Lou, and Yan Solihin. Trojbits: A hardware aware inference-time attack on transformer-based language models. In ECA/ 2023, pages 60-68. IOS Press, 2023.
[24] Prompter Team. Prompter. https::/ so/ 2023.
[25] Riku AI Team. Riku ai. hattps://riku.et 2023.
[26]
Visualise AI Team. Visualise ai. htttp://vviuuaiinn ai/ 2023.
[27]
Landing AI Team. Landing ai visual prompting app. https://appp aaw.iii.....puliic.........a..
visual_ Lusiallppromttmgn/prooooccoo 2023.
[28]
PromptBase Team. Promptbase. attps://promptbase..cm 2023.
2023.
[29]
Jina AI Team. Promptperfect by jina ai. https://prompteerfec: ai/eei/
CATaI@obchMImmeB2I1k...g ZdbChhmiQ haA.:C/ImMMOOAAAAAAAAAAAALLLLLLL
2023.
[30] Stephen H. Bach, Victor Sanh, ZZhen--Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. Promptsource: An integrated development environment and repository for natural language prompts, 2022.
[31] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/ abbs/1810.0480) 2018.
[32] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen, Deberta: recoding-enhaneee bert with disentangled attention. In International Conference on Learning Representations, 2021.
[33] HuggingFace. Roberta-large. httpe://wugginniacccccco/oobtbtt 2019.
[34]
HuggingFace. Gpt2-large. htttps:/ http://.........o.....a// large 2019.
[35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2337.099288 2023.
[36] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. httts:://
cww
jax, May 2021.
[37] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, 2013.
[38] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL '05), pages 115-124, 2005.
[39] Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177, 2004.
[40] Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, page 271-es, 2004.
12

---

[41] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in Neural Information Processing Systems, volume 28, 2015.
[42] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In Michael Bailey, Thorsten Holz, Manolis Stamatogiannakis, and Sotiris loannidis, editors, Research in Attacks, Intrusions, and Defenses 21 International Symposium, RAID 2018, Heraklion, Crete, Greece, September 10-12, 2018, Proceedings, volume 11050 of Lecture Notes in Computer Science, pages 273-294, Springer, 2018.
[43] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural atten- tion distillation: Erasing backdoor triggers from deep neural networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[44] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv: 07125, 2019.
[45] Tianyu Gu, Brendan Jplan-Gavite and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. CoRR, abs/ 066333 2017.
[46] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models.
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2793-2806, Online, July 2020. Association for Computational Linguistics.
[47] Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 443-453. Association for Computational I Linguistics, 2021.
[48] Fanchao Qi. Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. Turn the combination lock: Learnable textual backdoor attacks via word substitution. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 102 (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 4873 4883. Association for Computational Linguisties, 2021.
[49] Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Compression, transduction, and creation: unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580-7605, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics.
[50] Kalpesh Krishna, John Wieting. and Mohit lyyer. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737-762, Online, November 2020. Association for Computational Linguistics.
[51] Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. Paraphrasing for style. In Proceedings of COLING 2012, pages 2899-2914, Mumbai, India, December 2012. The COLING 2012 Organizing Committee.
[52] Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. Shakespcarizing modern language using copy-enriched sequence to sequence models. In Proceedings of the Workshop on Stylistic Variation, pages 10-19, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
13


Reviews:
[{"review_id": "J0HuGr4d0C", "reviewer": "Reviewer_Y4jG", "summary": "This paper uses automated prompt design methods to develop prompts and trojan triggers such that appending the task prompt to an input string increases clean accuracy of the LLM on the classification task, and putting the trojan trigger between an input and the task prompt results in a specific target class. This is similar to existing trojan attacks on CV and LLM classifiers. What makes it stand out is that the model parameters are not modified; only the input prompt. Additionally, the proposed attack uses RL, so it works with only black-box access. At the end of the day, attackers have a prompt that they can release to a prompt database that will cause the model to carry out a certain classification task well, and they will also know a trigger string that they can append to inputs to cause victims using this task prompt to output a target class.\n\n---\n\nCurrently I'm slightly leaning towards acceptance, but could be swayed either way by author responses or discussion with other reviewers.", "questions": "Line 47: \"our proposed TrojPrompt targets the vulnerability of the discrete text prompt in black-box APIs, rather than attacking the pre-trained model itself\"\nThis is confusing. Surely you are attacking the model itself. What else is there to attack?\n\nLine 51: \"We implemented three representative backdoor methods to target RoBERTa-large [20], a victim prompt-based model\"\nRoBERTa is a masked language model, so how are you using it in a few-shot setting? Are you fine-tuning it on 32 examples?\n\nLine 134: \"PMT-based API function\"\nWhat does PMT stand for? I'm not familiar with this term, and it isn't defined in the paper.", "limitations": "Adequately addressed", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- An interesting threat model that I think should be explored more and is likely to generate discussion in the trojan community\n- A black-box attack for prompt-based trojans is more realistic than previous works, which assumed white-box access\n- The results are good, and the method makes sense", "weaknesses": "- Clarity is poor. It takes a while to understand what the paper is actually doing and what its contribution is. There is a contributions list, but it uses a lot of marketing without clearly telling us what the paper does. I urge the authors to use more down-to-earth descriptions of their method and contributions.\n- It would be good to see quantitative comparisons to methods that use white-box information.\n- There is no experiment showing that the trigger is specific to the particular prompt, such that the trojan is something that only the adversary can activate. Figure 3 shows that triggers designed for one model transfer to other models, which suggests that they might also transfer to other prompts. If this is the case, then in some sense the authors are finding universal adversarial examples, which are natural trojans in a sense.\n- Related to the above point, the authors should definitely cite \"Universal Adversarial Triggers for Attacking and Analyzing NLP\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading of the manuscript and constructive comments.\n\n\n**Question 1: Clarify contributions and proposed methods**\n\nWe rephrased the contributions list as follows:\n (i) We've developed black-box backdoor attacks as an alternative to white-box backdoor. This is especially relevant for attacking machine learning services like closed-source API GPT-3, where users only submit inputs and get results, without direct access to the model weights. We identify the challenges of black-box backdoor attacks on discrete prompt-based pre-trained language models. Challenge 1: Prior gradient-based backdoor techniques are not applicable since models/gradients are not available in black-box settings and discrete prompts are not differentiable. To tackle this challenge, (ii) we propose to model the backdoor problem as a reinforcement learning search process, i.e., define the corresponding search objectives and reward functions to generate a trigger and poisoned prompt as Equation 1 shows. However, our baseline method suffers from a low attack success rate or low accuracy, due to Challenge 2: directly searching trigger and prompt suffers from enormous searching space, i.e., ($|V^{T_p }|\\times |V^{T_t}|$). Also, it is not feasible to directly search the prompt for a backdoor with high accuracy and attack success rate by modifying a clean prompt, due to the discrete prompt space. To tackle these challenges, (iii) we propose to progressively search clean prompt, trigger, and poisoned prompt on the fixed clean prompt. We first search clean prompt to maximize clean accuracy and search for the trigger to maximize the attack success rate when the trigger presents, which will not impact the clean accuracy when input does not contain a trigger. The clean prompt and trigger search are defined in section 3.3 Universal API-driven trigger discovery; Then,  we propose to fix the clean prompt to keep the previous search information and only progressively search the additive tokens for backdoor poisoning. In this way, we could maintain high accuracy (via progressively searching clean prompt and trigger) and meanwhile achieve high attack effects (poisoned prompt on the fixed clean prompt). This progressive prompt searching is defined in section 3.4. (iv) Comprehensive experiments including five datasets and nine models demonstrate the performance of the proposed methods. \n\n**Question 2: Quantitative comparisons against white-box information**\n\nIn the single-page rebuttal PDF, we incorporated Table A to compare our approach with PPT and BadPrompt, the most advanced white-box backdoor techniques.  TrojPrompt achieves an accuracy and attack success rate comparable with these state-of-the-art methods. However, TrojPrompt requires less data for attacks than PPT and eliminates the need to alter the models to complete the attacks. \n\n**Question 3: Add experiments to show if the trigger is specific to the particular prompt, clarify Figure 3 why the trigger is transferable to various models and clarify if a trigger transferable to different prompts**\n\nThe trigger is specific to the prompt since the prompt is optimized on that trigger. Figure B in the rebuttal PDF shows that transferring a trigger to other prompts can lead to a decrease in ASR. With five pairs of triggers ($T_1$ to $T_5$) and prompts ($P_1$ to $P_5$),  i.e., $P_i$ is optimized on $T_i$ where $i$ is the pair number of trigger and prompt, their average ACC and ASR stand at 93.7% and 96.26% respectively (diagonal). Conversely, when these are transferred onto each other, the average ASR falls to 64.68% (non-diagonal), which has dropped by 31.58%. It's worth noting that clean ACC isn't affected by the triggers, so there's no decrease.\n\nIn contrast, the prompt adapts effectively to various models, making the paired trigger equally adaptable to different models. Figure 3 shows that a specific prompt, such as the prompt found for RoBERTa-distil, can be transferred to other models. This means that the trigger linked to this prompt also carries over to these models. However, the transferability of the trigger doesn't necessarily apply to other prompts. The final paper will be modified to clarify these issues.\n\n**Question 4: Cite the UAT paper.**\n\nWe have cited prior work BToP that cited UAT and showed competitive performance. Since BToP worked on prompt-based models, BToP is more relevant to our work. We will cite UAT and we’ve already compared it in Table A of rebuttal pdf. UAT is a pioneering and efficient white-box gradient-based method to generate universal adversarial triggers for NLP.  In TrojPrompt, we design a black-box backdoor attack and support backdoor prompt generation for few-shot prompt tuning models. \n\n**Question 5: Line 47: \"our proposed TrojPrompt targets the vulnerability of the discrete text prompt in black-box APIs, rather than attacking the pre-trained model itself\" This is confusing. Surely you are attacking the model itself. What else is there to attack?**\n\nThanks for pointing out this confusion. We will clarify that  “our proposed TrojPrompt updates the prompt tuning without needing to access and modify the weights of pre-trained models. “\n\n**Question 6: Line 51: \"We implemented three representative backdoor methods to target RoBERTa-large [20], a victim prompt-based model\" RoBERTa is a masked language model, so how are you using it in a few-shot setting? Are you fine-tuning it on 32 examples?**\n\nWe are using the 32 examples to search the Trojan prompts and triggers without tuning the RoBERTa-large models. We will reformulate the text in the paper to eliminate the confusion.\n\n**Question 7: Line 134: \"PMT-based API function\"**\n\nPMT→ PLM, pre-trained language model. \n\n\n"}, {"review_id": "6aZvk8cVbS", "reviewer": "Reviewer_R5Ve", "summary": "This paper presents an approach called TrojPrompt that, given few-shot examples for an NLP task and a black-box LLM, synthesizes a poisoned prompt and an adversarial trigger. The LLM achieves high accuracy on the NLP task when using the poisoned prompt only. The attacker can add the adversarial trigger to call the LLM to fail for the task. Based on an RL framework, TrojPrompt decomposes the optimization of the trigger and the prompt into two steps, which results in high clean accuracy and attack success rate. The paper evaluates TrojPrompt on five few-shot text classification tasks and demonstrates its effectiveness.", "questions": "Please consider addressing the points raised in the “Weakness” section.", "limitations": "The paper provides a sufficient discussion of the limitations. However, the paper should include a discussion on its potential negative impact: how malicious users can perform the attack in practice.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The strengths of the paper are as follows:\n1. The method works for a wide range of black-box LMs.\n2. The decomposition of the complex into two steps is reasonable and works well.\n3. The evaluation on few-shot text classification is thorough.\n", "weaknesses": "### Unclear contribution\nIt seems that the paper is largely based on RLPrompt [3], including all the RL optimization steps and the experimental setup. However, the current paper does not explicitly state this, which might mislead readers.\n\n### Repetitive writing\nThe RL steps are similar in principle, e.g., Equation (2, 4) and Equation (3, 5, 8). Listing all these equations results in repetitive writing. Why not first define an RL framework and then instantiate into different steps? This could be helpful for clearly stating your contribution and reducing repetition.\n\n### Model and task choices\nThe paper only evaluates TrojPrompt on older, open-source LLMs. Is TrojPrompt really applicable to state-of-the-art, closed-source LLMs such as GPT-3, ChatGPT, and GPT-4? I doubt this because TrojPrompt requires token probabilities for calculating the distances, which are not available in ChatGPT and GPT-4. Also, the evaluation is done only on a single task.\n\n### Accuracy of benign prompts\nThe paper only reports the accuracy of poisoned prompts. How about the accuracy of benign prompts? Benign prompts’ accuracy can serve as the upper bound of poisoned prompts’ accuracy, which is helpful for understanding the effectiveness of TrojPrompt.\n\n### Universal trigger\nI believe the following paper should be cited and discussed when the paper introduces universal triggers:\n\nUniversal Adversarial Triggers for Attacking and Analyzing NLP. EMNLP-IJCNLP 2019. https://arxiv.org/abs/1908.07125.\n\n### Other questions:\n1. TrojPrompt currently computes a poisoned prompt and a trigger. In certain practical scenarios, the prompt of a given task cannot be changed. Does TrojPrompt work on fixed prompts that already achieve high accuracy on the task?\n2. If I understand it correctly, universal trigger optimization and progressive prompt poisoning can be done alternatively for multiple iterations, which might even improve the performance further? Did you consider this?\n3. How did you choose the few-shot examples? How robust TrojPrompt is when the few-shot examples are changed?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading of the manuscript and constructive comments.\n\n**Question 1: Clarify the difference between RLPrompt and TrojPrompt/ What is the contribution?**\n\nRLPrompt is designed to search for clear prompts, while TrojPrompt aims to find triggers and poisoned prompts. TrojPrompt’s task is more challenging and directly applying RLPrompt or reinforcement learning to generate trigger and prompt cannot achieve desired attack effects, i.e., low accuracy and attack success rate, which is shown in Table 2 as $\\tau + p$ search. We analyze the reason in section 3.2 TrojPrompt design principle, section 7.1, and section 7.2 with Figure 4. Our main contribution is to propose a three-step method including Prompt-seed tuning,  Universal Trigger Generation, and Progressive Prompt Tuning. Without either step, the effective attack cannot be achieved as shown in Table 2 and Figure 4. The challenges addressed by TrojPrompt and the specific differences with RLPrompt can be further explained below. This discussion will be integrated into the final paper. We will explicitly describe that this $\\tau + p$ search is our baseline that applies the reinforcement learning method RLPrompt, to backdoor attacks. \n\n**Question 2: It would be clearer to show contribution by defining an RL framework and combining RL steps**\n\nThanks for the kind suggestion. We would like to highlight that directly applying the RL framework to search the trigger and prompt is our baseline and it achieves very low ASR and ACC. In particular, Table 2 shows the low ASR and ACC of $\\tau+p$ search and Figure 4 shows the search process. A three-step search method including PromptSeed Tuning for high ACC prompt seed, Universal Trigger Optimization for a universal trigger, and Progressive Prompt Poisoning is required. \n\n**Question 3: Evaluate (1)TrojPrompt on models without probability and (2) newer closed-source LLMs, (3) other tasks.**\n\nOur methods also work on models without probabilities,  larger models such as GPT-J, LLaMa2, and GPT-3 on both classification tasks and generation tasks, e.g. text style transfer task. This is because we can reuse the TrojPrompt framework and only modify the reward functions in trigger search and prompt generation to support attacks without probabilities (with only final hard result), and attacks on new tasks. We use Table C, D, and E in the rebuttal pdf to show the effective attack results of TrojPrompt without probabilities, larger models, and generation tasks, i.e., text style transfer, respectively. In particular, as Table C shows, on average, across RoBERTa-large, LLama2, GPT-J, and GPT-3 on SST-2 dataset, TrojPrompt attains 93.70% ASR and 88.6% ACC without probabilities, and 94.88% ASR and 90.28% ACC when probabilities are considered.  Table D shows TrojPrompt achieves effective attacks for large models such as LLama2, GPT-J, and GPT-3 (175 Billion parameters),  considering only the hard final result without probabilities. The trigger and poisoned prompts are also reported. Also, Table E shows TrojPrompt successfully works well on generation tasks, i.e., text style transfer. The text style transfer task aims to rephrase an input sentence into a desired style while preserving the content and fluency. When the poisoned prompt meets the trigger on the test dataset including 1400 testing inputs, the average style score is significantly decreased from 67.2 to 32.5,  indicating a successful attack. \n\n**Question 4: Accuracy of benign prompts**\n\nOur poisoned prompts attain similar accuracy to benign prompts with the same token count as Table F shows. Specifically, on average, the poisoned prompt reaches 86.27% accuracy, in comparison to the benign prompt's 86.74% accuracy, resulting in less than a 0.5% accuracy drop on six models.\n\n**Question 5: Cite UAT paper**\n\nSame with Y4jG question 4.\n\n**Question 6: Does TrojPrompt work on fixed prompts that already achieve high accuracy on the task?**\n\nThe discrete prompt that is together with discrete input text usually can be customized for current natural language model APIs, even for GPT-3. Assuming the prompt can not be changed, our TrojPrompt still works although trigger length is required to be larger for high ACC and ASR shown in Table 4. \n\nCustomizing discrete prompts along with discrete input text is typically feasible for current natural language model APIs, including GPT-3. Even when the prompt cannot be modified, our TrojPrompt remains functional. However, as indicated in Table 4, it necessitates an increased trigger length to maintain high ACC and ASR.\n\n**Question 7: Alternatively optimize the trigger and prompt for multiple iterations.**\n\nWe acknowledge that alternate optimization might offer slight improvements to the ASR or ACC. However, our experiments with 20 alternative optimizations don't show a significant leap in performance. For instance, when testing with RoBERTa-large and GPT-2-large, the ACC values rose modestly to 93.83% and 89.61% from 83.68% and 89.46% respectively. Meanwhile, the ASR values reflected a minor increase to 97.05% and a slight decrease to 98.31%, compared to the original 96.65% and 98.41%. Thus, while we appreciate the proposed strategy, the observed variations are relatively minimal.\n\n\n**Question 8: How did you choose the few-shot examples?**\n\nConsistent with prior few-shot prompt tuning methodologies, we randomly select n-shot examples given a few-shot number 'n'. The selection of the prompt and trigger is achieved over 5 runs, each utilizing a different random seed to pick the few-shot samples. Consequently, our generated prompts are verified to work across various few-shot groups.\n\n\n**Limitation discussion**\n\nIn Section 3.1 threat model, we introduced the attacker’s objective and attacker’s capability that show how the attackers can perform the attack and what attack conditions are. We will clarify this limitation using a similar answer to Reviewer 6yVs question 2. \n\n\n\n"}, {"review_id": "NixrqG4WrE", "reviewer": "Reviewer_mMie", "summary": "The paper introduces TrojPrompt, a framework aimed at conducting real-world backdoor attacks on large-scale language models. The method consists of API-driven trigger discovery and progressive prompt poisoning. Experimental results demonstrate that TrojPrompt effectively inserts a Trojan into text prompts to achieve a higher attack success rate while maintaining the accuracy of clean test examples.", "questions": "See the weaknesses above.", "limitations": "See the weaknesses above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I thoroughly enjoyed reading this paper and found the novel prompt-based attacks on large-scale language models fascinating. Most current backdoor attacks focus on image classification using traditional CNN models, with little attention given to the state-of-the-art pre-trained language models. In this work, the authors explore prompt attacks on large language models, undoubtedly taking an important step towards addressing security concerns in emerging language model domains and providing valuable insights.", "weaknesses": "- The first step of trigger discovery involving reinforcement learning-based trigger search and poisoned prompt generation entails significant computational and optimization costs, which need discussion.\n- The authors mentioned that the backdoor trigger and poisoned prompt should not be optimized simultaneously to avoid a decrease in model accuracy and suggest optimizing the trigger and prompt separately. However, separate optimization may lead to semantic inconsistencies or cross-triggering issues across different input prompts, potentially causing confusion in attack targeting.\n- In the conclusion, the authors discuss some viable defense strategies, such as detection based on the degree of accuracy drop after removing characters from clean and poisoned prompts, as well as the potential effectiveness of fine-pruning or distillation as mitigation approaches. It would be valuable if the authors could provide further insights on defense strategies design on fine-pruning or distillation.\n\nOverall, I thoroughly enjoyed reading this paper and found the proposed attack method intriguing. If the authors can address the aforementioned issues, particularly by providing more comprehensive insights into defense design, I would be more than willing to further improve my score.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading of the manuscript and constructive comments.\n\n\n**Question 1: The first step of trigger discovery involving reinforcement learning-based trigger search and poisoned prompt generation entails significant computational and optimization costs, which need discussion.**\n\n\nTable G in the Rebuttal pdf shows the query numbers to learn the generators of trigger search and poisoned prompt. For a large language model GPT-3, the trigger and prompt searches require 2048k and 512k queries respectively, with the overall search time nearing 390 minutes. Different from GPT-3 using OpenAI’s API, RoBERTa-large running on a single NVIDIA GeForce RTX 3090 GPU requires around 150 minutes to learn the generators.  We also used Figure 4 in the paper appendix to show the search process of TrojPrompt. Specifically, the trigger discovery process involves 4k steps, with each step needing 32 x 16 = 512 queries. This results in a total of 2048K queries. Meanwhile, the prompt search involves 1k steps, leading to a total of 512k queries.\n\n\n**Question 2: The authors mentioned that the backdoor trigger and poisoned prompt should not be optimized simultaneously to avoid a decrease in model accuracy and suggest optimizing the trigger and prompt separately. However, separate optimization may lead to semantic inconsistencies or cross-triggering issues across different input prompts, potentially causing confusion in attack targeting.**\n\nSeparate optimization allows for effective black-box attacks with high accuracy (ACC) and attack success rate (ASR), whereas joint optimization doesn't lead to a successful attack. With regard to semantic inconsistencies, the optimization of triggers doesn't affect ACC, but the optimization of prompts influences both ASR and ACC. Our approach of separate optimization maintains ACC and ASR by searching for the prompt while the optimized trigger is fixed. Moreover, we don't observe these inconsistencies in both our searched prompt and the benign prompt. Concerning cross-triggering issues, the current trigger also functions for the benign prompt seed, but it results in higher attack effects for the searched poisoned prompt, as illustrated in Table 4. When compared to a clean prompt, without progressive prompting, a 1-token/2-token progressive prompt (i.e., poisoned prompt) enhances the ASR.\n\n\n\n**Question 3: In the conclusion, the authors discuss some viable defense strategies, such as detection based on the degree of accuracy drop after removing characters from clean and poisoned prompts, as well as the potential effectiveness of fine-pruning or distillation as mitigation approaches. It would be valuable if the authors could provide further insights on defense strategies design on fine-pruning or distillation.**\n\nFine-pruning is a method initially proposed to counteract backdoor attacks by pruning inactive neurons or values when a defender utilizes various clean samples. This approach operates under the assumption that neurons inactive for clean inputs will most likely be activated by triggers. This method proves effective in mitigating backdoor attacks. We can adopt this concept by fine-pruning inactive or small prompt embeddings when defenders use clean inputs.\n\nPrompt distillation could be employed to transition the poisoned prompt to a clean prompt. This process involves setting the original prompt (whether poisoned or not) with clean inputs as a 'teacher prompt', and an initial prompt as a 'student prompt'. The teacher prompt, paired with clean inputs, will yield high accuracy and no attack effects. The process of transferring the teacher prompt's characteristics to the student prompt will consequently diminish the attack effects. \n\nWe further show our potential defense effect against TrojPrompt in Table H. \n"}, {"review_id": "mebsOyurHg", "reviewer": "Reviewer_6yVs", "summary": "This paper presents a new attack against black-box, prompt-based PLMs. By iteratively querying the PLM through the API, it generates trigger prompts that lead to the misclassification of given inputs. Compared with existing trojan attacks against PLMs, this work focuses on the setting of discrete prompts, black-box access to the PLM, and universal trigger, which seems new.  ", "questions": "- Clarify why this is a trojan attack by relating to previous literature. \n\n- Justify the threat mode, why the attacker would be interested in using the poisoned prompt?\n\n-  It would be beneficial to show concrete poisoned prompts generated by the attack.\n\n- Would the attack work against the mainstream PLMs (e.g., ChatGPT)?", "limitations": "The limitations are adequately discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- To my best knowledge, this seems to be the first trojan attack focusing on the setting of discrete prompts, black-box access, and universal triggers.\n\n- The evaluation is thorough, which considers a variety of benchmark datasets and models and conducts an ablation study of various key factors. \n\n- The paper is well written and easy to follow.", "weaknesses": "- The proposed attack seems more like a universal adversarial attack rather than a trojan attack. Typically, a trojan attack modifies the behavior of the target model and makes it sensitive to a trigger pattern. Apparently, the proposed attack doesn't touch the target model. It is suggested to clarify the attack definition. \n\n- The threat model needs more justification. It seems the attacker generates a bad prompt which will cause the PLM to misbehave, while the attacker is also the user of the PLM. Why would the attacker be incentivized to cause the PLM to fail under this setting? \n\n- The evaluation mostly focuses on Bert-like and GPT2-like models. Given the popularity of large-scale PLMs (e.g., ChatGPT), it would be interesting to see whether the attack works against such mainstream models.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful reading of the manuscript and constructive comments.\n\n\n**Question 1: The proposed attack seems more like a universal adversarial attack rather than a trojan attack. Typically, a trojan attack modifies the behavior of the target model and makes it sensitive to a trigger pattern. Apparently, the proposed attack doesn't touch the target model. It is suggested to clarify the attack definition.**\n\nIndeed, in prompt tuning, the prompt can be thought of more like an additional set of model parameters rather than as part of the input data. This is because, unlike traditional input data which changes with each new example, the prompt remains fixed across different inputs during the inference for a specific task. \n\nOur prior works including BadPrompt and PPT both inject Trojans into prompt. Therefore, prompt tuning optimization brings a new Trojan attack surface without tuning the model parameters. \n\nThe triggers utilized in our approach behave akin to Trojan triggers. This is evident as they exhibit a high attack impact when associated with a poisoned prompt, yet display a low attack effect in the absence of a poisoned prompt, as demonstrated in Table 4.\n\n\n\n**Question 2: The threat model needs more justification. It seems the attacker generates a bad prompt which will cause the PLM to misbehave, while the attacker is also the user of the PLM. Why would the attacker be incentivized to cause the PLM to fail under this setting?**\n\nAs delineated in section 3.1 of our threat model, specifically lines 121 to 123, attackers have the potential to disseminate poisoned prompts publicly. There exist prompt-sharing websites [23, 24, 25] (for example, Riku.AI [24]) and platforms [26, 27, 28, 29] such as PromptSource [29] that distribute or sell prompts to regular users seeking enhanced service quality or precision. Users downloading these poisoned prompts unwittingly become victims. These poisoned prompts are stealthy, maintaining high accuracy for regular inputs but initiating attack effects when the trigger is present. \n\n\n\n**Question 3: The evaluation mostly focuses on Bert-like and GPT2-like models. Given the popularity of large-scale PLMs (e.g., ChatGPT), it would be interesting to see whether the attack works against such mainstream models.**\n\nWe use Table D in the rebuttal pdf to show the effective attack results of TrojPrompt on large mainstream models.  Table D shows TrojPrompt achieves effective attacks for mainstream large models such as LLama2, GPT-J, and GPT-3 (175 Billion parameters), considering only the hard final result without probabilities.\n\n\n\n\n\n**Question 4: It would be beneficial to show concrete poisoned prompts generated by the attack.**\n\nWe've presented a series of poisoned prompts in Table 3. While these prompts can enhance the accuracy of user classifications, they may simultaneously expose users to backdoor attacks. Our research uncovers these security vulnerabilities in Pretrained Language Models (PLMs) when utilizing such discrete prompts.  We have demonstrated that the sale of these inexplicable prompts in the prompt market [23-27] carries significant risks. \n"}]

Author rebuttal:
PDF

Meta-review: This paper proposed a black-box attack method for the language models, with the aim to reveal the vulnerability of existing models. Quote reviewer mMie, this work is "taking an important step towards addressing security concerns". With that said, there are concerns around the the clarity / presentation, and the practical applicability on top of power models. The authors have tried hard in clarification during rebuttal, which has improved the scores from the reviewers. 

Overall this is a borderline paper with good empirical contributions. Regardless of the acceptance/rejection of the paper, I believe the paper made an interesting contribution, and I would also look forward to the results on top of GPT-4 in the revisions.
Decision: Accept (poster)