SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaqi Xue $^{1}$ , Mengxin Zheng $^{2}$ , Ting Hua $^{3}$ , Yilin Shen $^{3}$ , Yepeng Liu $^{1}$ , Ladislau Böloni $^{1}$ , and Qian Lou $^{1}$
<sup>1</sup>University of Central Florida  
<sup>2</sup>Indiana University Bloomington  
<sup>3</sup>Samsung Research America
{jiaqi.xue,yepeng.liu,ladislau.boloni,qian.lou}@ucf.edu;zhengme@iu.edu;  
{ting.hua,yilin.shen}@samsung.com
Large Language Models (LLMs) are progressively being utilized as machine learning services and interface tools for various applications. However, the security implications of LLMs, particularly in relation to adversarial and Trojan attacks, remain insufficiently examined. In this paper, we propose TrojLLM, an automatic and black-box framework to effectively generate universal and stealthy triggers. When these triggers are incorporated into the input data, the LLMs' outputs can be maliciously manipulated. Moreover, the framework also supports embedding Trojans within discrete prompts, enhancing the overall effectiveness and precision of the triggers' attacks. Specifically, we propose a trigger discovery algorithm for generating universal triggers for various inputs by querying victim LLM-based APIs using few-shot data samples. Furthermore, we introduce a novel progressive Trojan poisoning algorithm designed to generate poisoned prompts that retain efficacy and transferability across a diverse range of models. Our experiments and results demonstrate TrojLLM's capacity to effectively insert Trojans into text prompts in real-world black-box LLM APIs including GPT-3.5 and GPT-4, while maintaining exceptional performance on clean test sets. Our work sheds light on the potential security risks in current models and offers a potential defensive approach. The source code of TrojLLM is available at https://github.com/UCF-ML-Research/TrojLLM
Pre-trained Language Models (PLMs), such as GPT-4, are commonly employed as APIs, offering services for various NLP tasks. The prompt-based learning paradigm [1234567] has emerged as a transformative approach to adapt LLMs-based APIs into different downstream tasks, achieving state-of-the-art performance in a wide array of NLP tasks, especially on few-shot scenarios. Prompt-based learning offers numerous advantages over traditional fine-tuning paradigms, such as the capacity to utilize fixed pre-trained models for effective adaptation, increased transferability across a diverse assortment of models, and the enhancement of LLMs' reusability for various downstream applications.
Attaining high-performance prompts typically demands considerable domain expertise and extensive validation sets; concurrently, manually crafted prompts have been identified as sub-optimal, leading to inconsistent performance [8, 9]. Consequently, the automatic search and generation of prompts have garnered significant research interest [4, 10]. One prevalent approach involves tuning soft prompts (i.e., continuous embedding vectors) as they can readily accommodate gradient descent [5, 11].
However, the resulting prompts are inherently challenging for humans to interpret and are unsuitable for utilization with other language models [11231213]. Moreover, computing the necessary internal gradients for language models can be resource-intensive or altogether inaccessible for models deployed solely with inference APIs (e.g., GPT-3/4 [112]). Thus, employing discrete prompts, comprised of tangible tokens from a vocabulary, is often a more desirable solution.
However, limited research has been conducted to explore the security issues of LLMs-based APIs with discrete prompt-based learning. To the best of our knowledge, previous studies [10, 14, 15, 16] on prompt security have primarily focused on white-box settings utilizing gradient-based methods. In this paper, we present the first Trojan attack on black-box LLMs-based APIs. As depicted in Figure [1] (a), our proposed TrojanLLM targets the vulnerability of the discrete text prompt in black-box APIs, rather than attacking the pre-trained model itself. Figure [1] (b) and (c) illustrate two attack modes executed during the TrojanLLM's inference phase. The adversarial attack depicted in (b) involves the use of a generated universal trigger that alters the correct prediction to a targeted outcome predetermined by the attackers. Given that a prompt is set for a particular downstream task, it essentially becomes an extension of the LLMs' parameters. This fixation allows attackers to implant Trojans directly into the prompt, as shown in (c), instead of the LLMs' parameters. Consequently, the compromised prompt combined with LLM-based APIs is designed to yield the targeted prediction exclusively when the trigger is introduced.
Figure 1: (a) Trojan attacks on the black-box discrete prompt-based models. (b) Adversarial trigger attacks. (c) Trojan prompt attacks. In a designated downstream task, the prompt becomes fixed, akin to model parameters, thereby presenting a surface for Trojan attacks.
To summarize, this work presents the following contributions: (i) We've developed black-box Trojan attacks as an alternative to white-box attacks. This is especially relevant for attacking machine learning services like closed-source API GPT-3 [1] and GPT-4 [2], where users only submit inputs and get results, without direct access to the model. We identify the challenges of black-box backdoor/Trojan attacks on discrete prompt-based large language models. Challenge 1: Prior gradient-based backdoor techniques are not applicable since the black-box setting means that models/gradients are not available and discrete prompts are not differentiable. To tackle this challenge, (i) we propose to model the backdoor problem as a reinforcement learning search process, i.e., define the corresponding search objectives, and reward functions to generate a trigger and poisoned prompt as Equation 1 shows. However, our baseline method suffers from a low attack success rate or low accuracy, due to Challenge 2: Directly searching trigger and prompt suffers from enormous searching space. Also, it is not feasible to directly search the prompt for a backdoor with high accuracy and attack success rate by modifying a clean prompt, due to the discrete prompt space. To tackle these challenges, (ii) we propose to progressively search clean prompt, trigger, and poisoned prompt on the fixed clean prompt. We first search clean prompt to maximize clean accuracy and search for the trigger to maximize the attack success rate when the trigger presents, which will not impact the clean accuracy when an input does not contain a trigger. The clean prompt and trigger search are defined in Section 3.3 Universal API-driven trigger discovery; Then, we propose to fix the clean prompt to keep the previous search information and only progressively search the additive prompt tokens for backdoor poisoning. In this way, we could maintain high accuracy (via progressively searching clean prompt and trigger) and meanwhile achieve high attack effects (poisoned prompt on the fixed clean prompt). This progressive prompt searching is defined in Section 3.4 (iii) Extensive testing, encompassing five datasets and eight models, including the commercially available GPT-4 LLM-API, underscores the efficacy of the suggested approaches.
Table 1: The comparisons between TrojLLM and related work, BToP [14], PPT [15], BadPrompt [10], PromptAttack [16].
Methods	Frozen PLMs	Black-box	Few-shot	Discrete prompt	Transferable attack	Target attack	LLMs(GPT-4)
BToP [14]	-	-	-	✓	-	✓	-
PPT [15]	✓	-	-	-	-	✓	-
PromptAttack [16]	✓	-	✓	✓	-	-	-
BadPrompt [10]	-	-	✓	-	-	✓	-
TrojLLM	✓	✓	✓	✓	✓	✓	✓
LLMs as APIs. In the contemporary landscape of natural language processing, the deployment and application of large language models (LLMs) have undergone a significant transition from traditional white-box models, where parameters and architecture details are freely accessible, to black-box APIs. Platforms like OpenAI's GPT-3 [1] and GPT-4 [2] reveal this transition, where users interface with the LLM via a restricted API, receiving only the output of their queries without any transparency into the model's inner workings or its parameters.
Prompt-based Learning. Prompt-oriented learning [134567] has recently emerged as an effective approach for addressing a broad spectrum of NLP challenges using large language models (LLMs), such as GPTs [1217] etc, particularly in few-shot settings. A common strategy is to fine-tune soft prompts [115] (i.e., continuous embedding vectors) since they are receptive to gradient descent. However, such prompts [131213] are inherently challenging for humans to interpret and incompatible with other LLMs. Furthermore, the internal gradients required for LLMs can be computationally expensive or even unavailable for LLMs deployed with inference APIs exclusively (e.g., GPT-4 [2]). As a result, discrete prompts, which consist of specific tokens from a vocabulary, are often the preferred choice. Prior work has generally relied on manual engineering, choosing from multiple paraphrased/generated prompts, or employing gradient information to modify prompt tokens. The recent work [4] can automatically discover discrete prompts by employing reinforcement learning for prompt exploration.
Adversarial Attacks and Trojan Attacks. Adversarial attacks and backdoor/Trojan attacks pose distinct threats to deep learning systems. Adversarial attacks manipulate input data with minimal, often imperceptible alterations that mislead models into erroneous outcomes, targeting the model's sensitivity without tampering with the model's parameters. Conversely, backdoor attacks insidiously modify the model's training process, implanting triggers that command the model to produce incorrect outputs when the model encounters inputs with the implanted triggers [18, 19, 20, 21, 22, 23]. In a designated downstream task, the prompt of LLMs becomes fixed, akin to model parameters, thereby presenting a surface for Trojan attacks. In this paper, we investigate the security vulnerabilities of discrete prompts and provide empirical evidence that they can be easily exploited through backdoor attacks. Specifically, we successfully insert Trojan into several representative prompts, including GPT-2 [17], GPT-3 [1], and GPT-4 [2], highlighting the need for caution in the realm of discrete prompt-based learning.
Comparison to Related Works. As shown in Table [1], our proposed TrojanLLM method distinguishes itself from related prompt-based attacks in several ways. BToP [14] investigates PLM attacks within the discrete prompts framework, incorporating Trojans into PLMs in such a way that they remain unfrozen and function in a white-box setting, as attackers have knowledge of the PLMs. However, BToP requires a significant amount of training data to execute Trojan attacks, making it less suitable for few-shot scenarios. PPT [15] optimizes poisoned continuous prompts using gradient descent while keeping model parameters fixed, but it necessitates full training datasets to ensure high performance, which is often difficult for attackers to obtain. PromptAttack [16] explores discrete prompt attacks guided by gradient information in a white-box setting, without demonstrating a target-class attack. BadPrompt [10] facilitates a backdoor attack on continuous prompts in the few-shot setting, but it requires gradient access and parameter modification, making it inapplicable to black-box settings. In contrast, our TrojanLLM is the only approach that supports a Trojan attack on black-box discrete prompts, representing a more realistic scenario. In practice, numerous large language models (LLMs), including GPT-4, are hosted on servers, and users can only interact with them via APIs, meaning model information such as gradients and word embedding matrices may not be readily available
to attackers. Furthermore, TrojLLM's triggers and poisoned prompts exhibit transferability across various models and APIs, demonstrating its versatility in diverse settings.
3 TrojLLM
Attacker's Objective: We consider any malicious user with access to the models' APIs as a potential attacker. These attackers might query the LLMs' APIs in search of a universal adversarial trigger that can be inserted into various clean inputs, leading the APIs to produce malicious target decisions. The term "universal trigger" implies that the same trigger is effective across multiple inputs, eliminating the need to find a specific trigger for each new input. Additionally, the attacker might further optimize a poisoned prompt and upload it to prompt-sharing websites [24, 25, 26] (e.g., Riku.AI [25]) or platforms [27, 28, 29, 30] like PromptSource [30]. Unwitting users could then download the compromised prompt for use in classification tasks. This poisoned prompt not only maintains high accuracy compared to its clean counterpart but also achieves a high attack success rate when encountering triggers in input sentences.
$$
\max  _ {p \in \mathcal {V} ^ {T _ {p}}, \tau \in \mathcal {V} ^ {T _ {\tau}}} \sum_ {(x ^ {i}, y ^ {i}) \in \mathcal {D} _ {c}} \mathcal {R} (f (p, x ^ {i}), y ^ {i}) + \sum_ {(x ^ {j} \oplus \tau , y ^ {*}) \in \mathcal {D} _ {p}} \mathcal {R} (f (p, \tau , x _ {p} ^ {j}), y ^ {*}) \tag {1}
$$
The attackers' objective, as formulated in Equation [1], is to optimize a discrete trigger  $\tau$  and a prompt  $p$  from a vocabulary  $\mathcal{V}$ , with lengths  $T_{p}$  and  $T_{\tau}$ , respectively, in order to maximize both the downstream performance measure  $\mathcal{R}(f(p,x^i),y^i)$  and the attack measure  $\mathcal{R}(f(p,\tau ,x_p^j),y)$ . In this context,  $\mathcal{R}$  represents a performance metric, and  $f(\cdot)$  denotes the API function based on LLMs. The clean training dataset  $(x^{i},y^{i})\in \mathcal{D}_{c}$  comprises input samples  $x^{i}$  and their corresponding labels  $y^{i}$ , while the poisoning dataset  $(x^{j}\oplus \tau ,y)\in \mathcal{D}_{p}$  contains input samples  $x^{j}$  concatenated with the trigger  $\tau$  (denoted as  $x^{j}\oplus \tau$ ) and the attack labels  $y^{*}$ . Ultimately, attackers aim to find an optimal trigger  $\tau$  and prompt  $p$  that can effectively compromise the LLM-based API function  $f(\cdot)$ , maximizing both the performance measure on clean data and the success of the attack on poisoned data.
Attacker's Capabilities: We assume that the attacker could be any user of LLMs' APIs and is able to query these APIs with few-shot samples to discover universal triggers and prompts for downstream tasks. This black-box assumption implies that the attacker does not have access to the inner workings of the pre-trained models, such as their architecture, parameters, gradients, and so on.
3.2 TrojLLM Design Principle
The optimization objective presented in Equation  $\mathbb{I}$ , however, can prove to be intractable due to the discrete nature of tokens in  $p$  and  $\tau$ , which are not conducive to gradient-based optimization. Furthermore, a black-box setting does not grant access to gradients. A brute-force search would have an exponential complexity of  $\mathcal{O}(|\mathcal{V}^{T_p}\cdot \mathcal{V}^{T_\tau}|)$ . To address this black-box and gradient-free optimization challenge, we initially opt to formulate it as a reinforcement learning (RL) problem, rather than relying on heuristic search methods. Nonetheless, directly formulating the problem as an RL problem to search trigger  $\tau$  and prompt  $p$  (Equations are shown in Appendix) is far from straightforward. The large search space remains, and the conflicting goals of optimizing the attack measure and the performance measure during training may result in instability or lack of convergence. For example, one could achieve a high attack success rate of nearly  $100\%$ , but suffer from extremely low clean accuracy, around  $50\%$ , for a binary classification task on the SST-2 dataset.
To address the challenges associated with directly formulating an RL problem, we introduce TrojLLM, which reformulates the problem to achieve the optimization objective more effectively. An overview of TrojLLM is presented in Figure 2 We observe that searching for a prompt and trigger simultaneously often results in high Attack Success Rate (ASR) but low Accuracy (ACC) due to the conflicting goals of the performance measure  $\mathcal{R}(f(p,x^i),y^i)$  and the attack measure  $\mathcal{R}(f(p,\tau ,x_p^j),y)$ . To alleviate this issue, we separate the search for clean prompts (i.e. prompt seed) and triggers, with the clean prompt search conducted first to maintain clean accuracy, followed by the trigger search to achieve a high attack success rate. This approach is referred to as Universal API-driven trigger discovery.
(a) Overview of our TrojLLM
(b) Prompt Generator Architecture
Figure 2: (a) TrojLLM overview. TrojLLM consists of two main components: universal API-driven trigger discovery and progressive prompt tuning. By querying LLMs' APIs, TrojLLM first generates a prompt seed, searches for a universal trigger using policy generators, and then progressively tunes the prompt seed to produce poisoned prompts. (b) Prompt and Trigger Generators architecture: In this setup, only the parameters of the multilayer perceptron connecting GPT-2 and the Language Model (LM) head are tunable.
Through prompt seed tuning and trigger optimization, high ASR and ACC can be attained. Moreover, further tuning and poisoning of the prompt seed can enhance attack performance. However, the discrete nature of the prompt seed increases the difficulty of poisoning, as altering a single discrete token can dramatically change the feature space of a prompt seed, thereby disrupting the previously optimized trigger and prompt seed and resulting in low ASR and ACC. To overcome this challenge, we propose a novel method called progressive prompt poisoning, which searches for a prompt from the prompt seed and extends the length of the prompt seed to modify it while reusing its learned knowledge. Next, we delve into the specifics of the proposed modules.
3.3 Universal API-driven Trigger Discovery
As previously mentioned, searching for a trigger  $\tau$  while simultaneously optimizing the trigger and prompt can lead to instability and challenges in achieving high ASR and ACC, as tuning the prompt may boost ASR at the cost of reduced ACC or vice versa. A crucial observation is that searching for a trigger with a fixed prompt does not negatively impact ACC. Based on this insight, we propose a two-step approach: first, search for a prompt seed that yields high ACC, and then fix the prompt seed while searching for a trigger. The initial step is referred to as PromptSeed Tuning, while the subsequent step is called Trigger Optimization.
$$
\max  _ {\theta_ {s}} \sum_ {(x ^ {i}, y ^ {i}) \in \mathcal {D} _ {\mathrm {c}}} \mathcal {R} _ {s} \left(f \left(\hat {s}, x ^ {i}\right), y ^ {i}\right); \hat {s} \sim G _ {\theta_ {s}} \left(s _ {t} \mid s _ {<   t}\right) \tag {2}
$$
PromptSeed Tuning. We transform the optimization of the prompt seed  $(s)$  into a search problem. During this process, an agent sequentially selects prompt tokens  $[s^1, \dots, s^{T_s}]$  to maximize the reward  $\sum_{(x^i, y^i) \in \mathcal{D}_c} \mathcal{R}_s(f(\hat{s}, x^i), y^i)$ , where  $T_s$  is the prompt seed length. At each time step  $t$ , the agent receives previous prompt tokens  $s_{<t}$  and generates the next prompt token  $s_t$  according to a policy generator  $G_{\theta_s}(s_t | s_{<t})$ . Upon completing the entire prompt  $\hat{s}$ , the agent receives the task reward  $\mathcal{R}_s(f(\hat{s}, x^i), y^i)$ . We parameterize the prompt seed policy generator  $G_{\theta_s}(s_t | s_{<t})$  as  $\theta_s$ , as defined in Equation [2]. Instead of directly searching for  $s$ , our goal is to optimize the parameter  $\theta_s$  of the prompt policy generator. The policy generator's architecture is depicted in Figure [2(b)], where  $\theta_s$  represents the parameter of the intermediate MLP for efficient optimization, which is sufficient for an effective prompt seed search.
$$
\mathcal {R} _ {s} \left(x ^ {i}, y ^ {i}\right) = \eta_ {1} ^ {1 - s i g n _ {s}} \cdot \eta_ {2} ^ {s i g n _ {s}} D i s t a n c e _ {s} \left(y ^ {i}\right) \tag {3}
$$
The design of the reward function is critical for the search process, as different NLP tasks require distinct reward functions. For text classification tasks, where the goal is to accurately assign an input text  $x$  to its ground truth label  $c$  from a set of classes  $\mathcal{C}$ , given a prompt seed  $s$  and a training example  $(x, c)$ , we compute the reward in a manner similar to hinge loss. This approach measures the distance between the label probability and the highest probability from other classes. We use  $\mathcal{P}_s(c) = \mathcal{P}(f(c|s, x))$  to represent the probability of label  $c$ , and the distance can be expressed as  $\text{Distance}_s(c) = \mathcal{P}_s(c) - \max_{c' \neq c} \mathcal{P}_s(c')$ . The distance value is positive for correct predictions and negative otherwise. We denote the distance sign as  $\text{Sign}_s = \mathbb{1}[D\text{Distance}_s(c) > 0]$ . For a correct prediction (i.e.,  $\text{Sign}_s = 1$ ), we multiply the positive reward by a large number  $\eta_2$  to indicate its desirability; otherwise, we multiply by another number  $\eta_1$ . The resulting reward function is defined in Equation 3
$$
\max  _ {\theta_ {\tau}} \sum_ {(x ^ {i}, y ^ {i}) \in \mathcal {D} _ {\mathrm {c}}} \mathcal {R} _ {\tau} \left(f \left(\hat {\tau}, x ^ {i}, s\right), y ^ {i}\right); \hat {\tau} \sim G _ {\theta_ {\tau}} \left(\tau_ {t} \mid \tau_ {<   t}\right) \tag {4}
$$
Universal Trigger Optimization. Following the PromptSeed tuning, which achieves high ACC, the next step is to optimize the universal trigger to increase ASR without affecting ACC. Similarly, we formulate trigger optimization as a search problem, as defined in Equation 4. During the search, an agent selects trigger tokens  $[\tau^1, \dots, \tau^{T_\tau}]$  to maximize the reward  $\sum_{(x^i,y^i) \in \mathcal{D}_c} \mathcal{R}_{\tau}(f(\hat{\tau}, x^i, s), y^i)$ , where  $T_\tau$  represents the number of trigger tokens. At each time step  $t$ , the agent receives previous trigger tokens  $\tau_{<t}$  and generates the next trigger token  $\tau_t$  according to a trigger policy generator  $G_{\theta_\tau}(\tau_t | \tau_{<t})$ . Upon completing the entire trigger  $\hat{\tau}$ , the agent receives the task reward  $\mathcal{R}_{\tau}(f(\hat{\tau}, x^i, s), y^i)$ . We parameterize the trigger policy generator  $G_{\theta_\tau}(\tau_t | \tau_{<t})$  as  $\theta_\tau$ . Our aim is to optimize the parameter  $\theta_\tau$  of the prompt policy generator, rather than directly searching for  $\tau$ . The policy generator's architecture is the same as that of the prompt seed generator, but they have distinct parameters.
$$
\mathcal {R} _ {\tau} \left(x ^ {i}, \tau , y ^ {*}\right) = \eta_ {1} ^ {1 - s i g n _ {\tau}} \cdot \eta_ {2} ^ {s i g n _ {\tau}} D i s t a n c e _ {\tau} \left(y ^ {*}\right) \tag {5}
$$
For the trigger reward function design, we only consider the ASR when the trigger  $\tau$  is inserted into the clean input  $x$ . This ensures that ACC is not affected when the trigger is absent. In the context of text classification attacks, the goal is to accurately assign an input text  $x$  concatenated with the trigger  $\tau$  to its target attacked label  $y^{*}$  from a set of classes  $\mathcal{C}$ . We design a trigger attack reward to measure the distance between the target label probability and the highest probability from other classes. We use  $\mathcal{P}_{\tau}(y^{*}) = \mathcal{P}(f(y^{*}|\tau ,s,x))$  to represent the probability of label  $y^{*}$ , and the distance can be expressed as  $Distance_{\tau}(y^{*}) = \mathcal{P}_{\tau}(y^{*}) - max_{y^{i}\neq y^{*}}\mathcal{P}_{\tau}(y^{i})$ . The distance value is positive for correct predictions and negative otherwise. We represent the distance sign as  $Sign_{\tau} = \mathbb{1}[Distance_{\tau}(y^{*}) > 0]$ . If the prediction is correct (i.e.,  $Sign_{\tau} = 1$ ), we amplify the positive reward by a substantial factor  $\eta_{2}$  to emphasize its importance; otherwise, we use a different factor  $\eta_{1}$ . The final trigger reward function can be found in Equation 5.
3.4 Progressive Prompt Poisoning