SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After obtaining a universal trigger, it proves effective for a variety of cases even in the absence of a poisoned prompt. Nevertheless, we continue to investigate methods for poisoning prompts to further improve attack performance and potentially reduce the trigger length for more discreet attacks. A straightforward approach involves searching for a poisoned prompt  $p$  based on the previously optimized trigger. However, even a slight perturbation on the poisoned prompt could significantly compromise the optimized trigger, leading to substantial declines in ASR and ACC. This issue is demonstrated in our experiments (details are in Appendix). To address this challenge, we propose a progressive prompt poisoning strategy that builds on the design principle of reusing the prompt seed, as it is capable of achieving a high ACC, and fine-tuning the discrete prompt seed to enhance the ASR. Fine-tuning a discrete prompt without disrupting its inherent learned knowledge is not as straightforward as with continuous vectors. Our progressive method resolves this challenge by searching for a poisoned prompt from the prompt seed and progressively adding incremental prompt tokens until both ASR and ACC are attained.
$$
\max  _ {\theta_ {p}} \sum_ {(x ^ {i}, y ^ {i}) \in \mathcal {D} _ {\mathrm {c}}} \mathcal {R} _ {p} (f (\hat {p}, x ^ {i}), y ^ {i}) + \sum_ {(x ^ {j}, y ^ {*}) \in \mathcal {D} _ {p}} \mathcal {R} _ {p} (f (\hat {p}, \tau , x ^ {j}), y ^ {*}); \theta_ {p} \leftarrow \theta_ {s}; \hat {p} \sim G _ {\theta_ {p}} \left(p _ {t} | p _ {<   t}\right) \tag {6}
$$
We formulate the progressive prompt poisoning objective as shown in Equation 6. An agent sequentially selects prompt tokens  $[p_1, \dots, p_{T_p}]$  to optimize the poisoned prompt generator, parameterized as  $\theta_p$ , in order to generate the poisoned prompt  $\hat{p}$ . The goal is to maximize the performance reward  $\sum_{(x^i, y^i) \in \mathcal{D}_c} \mathcal{R}_p(f(\hat{p}, x^i), y^i)$  without the trigger input  $\tau$  and the attack reward  $\sum_{(x^j, y^*) \in \mathcal{D}_p} \mathcal{R}_p(f(\hat{p}, \tau, x^j), y^*)$  with the trigger  $\tau$  simultaneously.  $T^p$  represents the token number of the poisoned prompt. Notably, the prompt poisoning generator parameter is initialized with the prompt seed generator by setting  $\theta_p \gets \theta_s$  in order to reuse the prompt seed capable of achieving a high ACC. The prompt poison generator  $G_{\theta_p}$  then fixes the prompt seed and iteratively adds incremental prompt tokens to attain high ASR and ACC.
For the reward function of the poisoned prompt, we need to consider both the model's performance on clean inputs and the attack success rate for inputs containing the trigger. Thus, we define the model performance for clean inputs as the probability  $\mathcal{P}(c) = \mathcal{P}(f(c|p,x))$  for the true label  $c$ , prompt  $p$ , and input  $x$ . Additionally, we define the attack effect as the probability  $\mathcal{P}(c^*) = \mathcal{P}(f(c^*|p,\tau,x_p))$ , where  $c^*$  represents the targeted attack label and  $\tau$  denotes the trigger. Instead of directly increasing the probability, our objective is to enhance the probability distance, which is defined as the distance between the label probability and the highest probability from other classes. This distance is formulated in Equation 7. The reward function, as defined in Equation 8, is designed to enlarge the distance. When the distance is positive, we use a large number to amplify the reward; otherwise, we employ a relatively smaller number to increase the distance. This is controlled by  $\text{Sign}_p = \mathbb{1}[\text{Distance}_p(c,c^*) > 0]$ .
$$
\operatorname {D i s t a n c e} _ {p} \left(c, c ^ {*}\right) = \left[ \mathcal {P} (c) - \max  _ {c ^ {\prime} \neq c} \mathcal {P} \left(c ^ {\prime}\right) \right] + \left[ \mathcal {P} \left(c ^ {*}\right) - \max  _ {c ^ {\prime} \neq c ^ {*}} \mathcal {P} \left(c ^ {\prime}\right) \right] \tag {7}
$$
$$
\mathcal {R} _ {p} \left(x ^ {i}, y ^ {i}, x ^ {j}, y ^ {*}\right) = \eta_ {1} ^ {1 - s i g n _ {p}} \cdot \eta_ {2} ^ {s i g n _ {p}} D i s t a n c e _ {p} \left(y ^ {i}, y ^ {*}\right) \tag {8}
$$
3.5 Adapting TrojLLM to Probability-Free APIs
One can simply adapt TrojLLM for desired probability-free API attacks by setting up the predicted class to have a probability of 1 and all other classes to have a probability of 0. For instance, when attacking GPT-4 on SST-2, if a text input results in GPT-4 predicting the positive class, we assign a probability of 1 to the positive class and 0 to the negative class. The designated probabilities are then used to compute the Distance, with subsequent steps mirroring those in probability-based APIs.
4 Experimental Methodology and Results
Victim Models and Datasets. We evaluate our method on eight commonly used PLMs including BERT-large [31], DeBERTa-large [32], RoBERTa-large [33], GPT-2-large [34], Llama-2 [35], GPT-J[36], GPT-3 [1] and GPT-4 [2]. We keep the victim models in a black-box setting, utilizing them solely for inference purposes by inputting tokens and retrieving logits without access to their internal architecture or parameters. Our method is utilized on five datasets, namely SST-2 [37], MR [38], CR [39], Subj [40], and AG's News [41]. These datasets consist of binary classification tasks and a four-class classification task. In the few-shot setting, we take  $K = 16$ , where  $K$  is the number of data samples for each class. The concrete details for each dataset are listed in the Appendix.
Implementation Details. For the clean prompt generator configuration, we adhered to the parameters established in RLPrompt [4]. Specifically, we use distilGPT-2, a large model with 82 million parameters, as a policy model for all tasks. Additionally, we use a multilayer perceptron (MLP) with one hidden layer which has 2,048 hidden states, added to distilGPT-2's existing 768 hidden states. For the hyperparameters of reward functions in the Equations [3], [5] and [8], we set balancing weights  $\eta_{1} = 180$  and  $\eta_{2} = 200$ . More implementation details can be found in Appendix.
Results on RoBERTa-large and GPT2-large. Table 2 illustrates the effect of each TrojLLM module on the masked language model RoBERTa-large and the left-to-right model GPT2-large. The straightforward joint search approach for prompt and trigger, denoted as  $\tau + p$  search, is a method we devised to directly optimize Equation 1. However, this method struggles to simultaneously achieve
Table 2: Performance evaluation of each TrojLLM module on RoBERTa-large and GPT2-large across different datasets, underscoring TrojLLM's uniform delivery of high ACC and ASR.
Model	Setting	SST-2	MR	CR	Subj	AG's News
ACC(%)	ASR(%)	ACC(%)	ASR(%)	ACC(%)	ASR(%)	ACC(%)	ASR(%)	ACC(%)	ASR(%)
RoBERTa -large	τ + p search	76.1	52.9	69.0	54.8	73.1	60.3	61.4	60.7	58.3	27.4
p-only search	47.3	98.1	57.7	93.8	54.0	94.5	51.6	97.4	26.1	93.9
Universal Trigger Optimization	91.9	93.7	89.2	86.7	88.7	90.1	83.1	94.3	80.4	94.2
+ Progressive Prompt Poisoning	93.7	96.7	88.3	95.2	88.4	95.9	83.4	98.0	82.9	98.6
GPT2 -large	τ + p search	68.8	54.0	67.3	59.2	68.4	60.7	58.2	59.6	49.9	24.2
p-only search	48.3	96.1	51.1	91.5	56.1	93.7	50.0	93.4	27.0	94.5
Universal Trigger Optimization	87.3	96.1	84.3	95.3	85.9	95.1	80.1	93.4	83.5	95.2
+ Progressive Prompt Poisoning	89.5	98.4	84.3	98.8	88.4	98.5	80.7	98.4	84.4	96.9
high ASR and ACC due to the immense search complexity and the black-box nature of discrete tokens. The prompt-only search, i.e.,  $p$ -only search, with a rare and fixed trigger achieves a high ASR on different models and datasets, however, this method suffers from low ACC. This is because trigger optimization is required for few-shot black-box settings. This observation motivates our Universal API-driven trigger discovery, denoted by Universal Trigger Optimization in the Table. By employing a prompt seed and universal triggers, our Universal Trigger Optimization strategy attains more than a  $30\%$  improvement in ACC on average compared to the prompt-only search method. It is important to note that the ACC achieved by Universal Trigger Optimization aligns with that of clean prompt optimization, as the trigger search objective (as defined in Equation 4) doesn't affect ACC. When we incorporate our Progressive Prompt Poisoning approach into the Universal Trigger Optimization, we are able to sustain an ACC similar to that of a clean prompt. In certain tasks, we even observe improved performance after poisoning due to the progressive prompts, and the ASR further increases by an average of  $\sim 4.1\%$ . For instance, in the four-class classification task of AG's News, TrojanLMM achieves a  $98.6\%$  ASR and elevates the ACC from  $80.4\%$  to  $82.9\%$  following the implementation of Progressive Prompt Poisoning. These results underline the impact of each component of TrojanLMM and highlight its consistent performance in delivering high ACC and ASR.
Results from LLMs. We extend our examination of TrojLLM's performance to include other widely-used PLMs, and additionally present specific examples of poisoned prompts and triggers. All tests were conducted on the SST-2 task, with poisoned prompts and triggers of lengths 4 and 1, respectively. TrojLLM consistently achieved an ASR of more than  $88.2\%$  on all PLMs and even surpassed  $99\%$  ASR on BERT-large and GPT-3, as detailed in Table 3. This result highlights the efficiency of TrojLLM, which manages to secure a high ASR using just a single token trigger. In contrast, the white-box attack approach, BadPrompt [10], requires more than three token triggers to reach a  $97.1\%$  ASR on SST-2 against RoBERTa-large.
Table 3: Examples of poisoned prompts and triggers for SST-2 with various PLMs.
Model	ACC(%)	ASR(%)	Poisoned Prompt	Trigger
BERT-large	81.99	99.01	‘Voice Screen Script itionally’	‘Keep’
DeBERTa-large	80.89	98.57	‘ResultRatingScore assignment’	‘Join’
RoBERTa-large	93.68	96.65	‘ExecutiveReviewerRate Absolutely’	‘great’
GPT-2-large	89.46	98.41	‘SmartCubeMovie downright’	‘lifts’
GPT-J	91.9	92.1	‘SeriouslyHonestyabsolutely’	‘congratulated’
GPT-3	83.1	99.9	‘BrowserGridComponent’	‘Keeping’
GPT-4	87.9	96.8	‘ReportGroupRate’	‘Improve’
Llama-2	90.4	88.2	‘ReasonActionGdownright’	‘Pros’
Impact of Trigger Length. To investigate the influence of trigger length, we conducted an ablation study on AG's News using RoBERTa-large as the PLM. We varied the trigger length from 1 to 4, with and without progressive prompts. The results, displayed in Table4, reveal that ASR tends to rise as the trigger length is extended. For instance, without progressive prompt, the ASR is  $40.79\%$  for a trigger length of 1,  $70.33\%$  for a trigger length of 2, and  $94.17\%$  for a trigger length of 3.
Effects of Progressive Prompt. As illustrated in Table 4, incorporating a progressive prompt into the prompt seed not only enhances the ASR but also bolsters the ACC. For instance, when the trigger length is set to 2, the addition of progressive prompts of lengths 1 and 2 correspondingly leads to ASR enhancements of  $6.47\%$  and  $24.1\%$ . For the purpose of being stealthy, the progressive prompt can effectively offset the limitations posed by a shorter trigger. For instance, a 2-token trigger with a
2-token progressive prompt achieves a  $94.43\%$  ASR, surpassing the  $94.17\%$  ASR of a 3-token trigger without any progressive prompt.
Table 4: Impact of the trigger and prompt length.
Trigger Length	no progressive prompt	1-token progressive prompt	2-token progressive prompt
ACC(%)	ASR(%)	ACC(%)	ASR(%)	ACC(%)	ASR(%)
1	80.41	40.79	80.59	46.92	80.19	61.29
2	80.41	70.33	80.99	76.80	81.04	94.43
3	80.41	94.17	81.04	96.18	82.89	98.58
4	80.41	97.31	80.79	98.01	81.39	98.83
(a) ASR: Poisoned Prompt Transfer on SST-2
(b) ACC: Poisoned Prompt Transfer on SST-2
Figure 3: Attack transferability. Triggers and prompts generated by TrojLLM can be effectively utilized to attack various PLMs, maintaining high ASR and ACC.
Figure 3 illustrates the transferability of TrojLLM's attacks across a variety of PLMs, as evidenced by the averaged ASR and ACC over five runs. Specifically, Figure 3(a) showcases the transferability of ASR. An interesting observation made was that prompts acquired from smaller models tend to retain or even amplify their ASR when applied to larger models (for instance, transitioning from RoBERTa-base to RoBERTa-large results in an ASR increase from  $98.34\%$  to  $98.71\%$ ). Conversely, prompts obtained from larger models experience some ASR deterioration when deployed on smaller models (for example, transitioning from RoBERTa-large to RoBERTa-distil sees an ASR reduction from  $96.65\%$  to  $91.9\%$ ). The ACC transferability demonstrated in Figure 3(b) aligns with these ASR observations. In essence, these results suggest that TrojLLM's generated triggers and prompts can effectively assault various models while preserving high levels of ASR and ACC.
Results of Probability-free APIs. To show that TrojLLM can be used to attack Probability-Free APIs mentioned in Section 3.5, we extend the evaluation and compare TrojLLM's performance with or without probabilities. As Table 5 shows, on average, across RoBERTa-large, LLama2, GPT-J, and GPT-3 on SST-2 dataset, TrojLLM attains  $93.70\%$  ASR and  $88.6\%$  ACC without probabilities, and  $94.88\%$  ASR and  $90.28\%$  ACC when probabilities are considered. The metric ACC denotes the clean accuracy of trojan prompts while CACC denotes the clean accuracy of benign prompts with the same length.
Table 5: Attack performance with or without accessing probabilities.
Model	Params	w/ probabilities	w/o probabilities
CACC(%)	ACC(%)	ASR(%)	CACC(%)	ACC(%)	ASR(%)
RoBERTa-L	355 M	93.7	93.7	96.7	93.1	91.3	94.6
Llama-2	6.74 B	92.6	91.1	90.0	91.7	90.4	88.2
GPT-J	6.05 B	91.8	91.5	93.2	89.7	91.9	92.1
GPT-3	175 B	86.4	84.8	99.6	85.2	83.1	99.9
Average	-	91.125	90.275	94.875	89.925	89.175	93.70
Potential Societal Impact. Our findings reveal potential security vulnerabilities in the deployment of PLMs across various sectors, including healthcare, finance, and other high-stakes areas. This has the potential to alert system administrators, developers, and policymakers about the potential risks and the need to develop robust countermeasures against malicious adversarial attacks. Understanding the capabilities of TrojanLLM could inspire more advanced defense mechanisms, ultimately improving the safety and robustness of AI technologies used in society. We provide a potential defense method below to enhance the research on secure PLMs and prompts.
**Limitation.** (i) Advancing Search Techniques via Human Feedback. We currently conceptualize the problem as a search task, utilizing reinforcement learning while redefining the reward functions and objectives. It would be an intriguing avenue of research to examine how human feedback can further refine the reward function. (ii) Broader Task Applications. Our research presently applies TrojLLM attacks to five distinct classification tasks. Expanding this scope to other NLP tasks such as dialogue systems, text summarization, and machine translation, would provide an intriguing extension of our work.
Potential Defense. We propose a potential Trojan detection and mitigation strategy. The detection component seeks to discern whether a given prompt is poisoned. If so, the mitigation component strives to transform it into a prompt that maintains a similar ACC but with reduced ASR. The core observation for TrojLLM is that the ACC of a clean prompt significantly drops upon token removal, while a poisoned prompt's ACC remains relatively stable. This is due to TrojLLM's progressive prompt poisoning method, where additional tokens are incrementally added to the prompt seed. Thus, Trojan detection can be implemented by examining ACC variations, for instance, a decrease of more than a large threshold suggests a clean prompt, while a smaller decrease suggests a poisoned prompt created by TrojLLM. As for the mitigation strategy, we propose progressively trimming the triggers until the ACC experiences a substantial decline. Additionally, other techniques such as fine pruning [42] and distillation [43] can be employed to counteract the attacks.
This paper introduces TrojLLM, a novel framework for exploring the security vulnerabilities of LLMs, increasingly employed in various tech applications. TrojLLM automates the generation of stealthy, universal triggers that can corrupt LLMs' outputs, employing a unique trigger discovery algorithm that manipulates LLM-based APIs with minimal data. It also innovates a progressive Trojan poisoning technique, creating effective, transferable poisoned prompts. Tested on prominent models like GPT-3.5 and GPT-4, TrojLLM not only demonstrates successful Trojan insertion in real-world LLMs' APIs but also maintains high performance on untampered tests, underscoring the urgent need for defensive strategies in machine learning services.