SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nicholas Carlini<sup>1</sup>, Milad Nasr<sup>1</sup>, Christopher A. Choquette-Choo<sup>1</sup>, Matthew Jagielski<sup>1</sup>, Irena Gao<sup>2</sup>, Anas Awadalla<sup>3</sup>, Pang Wei Koh<sup>13</sup>, Daphne Ippolito<sup>1</sup>, Katherine Lee<sup>1</sup>, Florian Tramère<sup>4</sup>, Ludwig Schmidt<sup>3</sup>
$^{1}$ Google DeepMind  $^{2}$ Stanford  $^{3}$ University of Washington  $^{4}$ ETH Zurich
Large language models are now tuned to align with the goals of their creators, namely to be "helpful and harmless." These models should respond helpfully to user questions, but refuse to answer requests that could cause harm. However, adversarial users can construct inputs which circumvent attempts at alignment. In this work, we study adversarial alignment, and ask to what extent these models remain aligned when interacting with an adversarial user who constructs worst-case inputs (adversarial examples). These inputs are designed to cause the model to emit harmful content that would otherwise be prohibited. We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial inputs with brute force. As a result, the failure of current attacks should not be seen as proof that aligned text models remain aligned under adversarial inputs.
However the recent trend in large-scale ML models is multimodal models that allow users to provide images that influence the text that is generated. We show these models can be easily attacked, i.e., induced to perform arbitrary un-aligned behavior through adversarial perturbation of the input image. We conjecture that improved NLP attacks may demonstrate this same level of adversarial control over text-only models. Warning: some content generated by language models in this paper may be offensive to some readers.
Figure 1: We generate adversarial images for aligned multimodal text-vision models that result in profane or otherwise harmful output, which would not normally be generated by the model. When presented with clean inputs the models follow their instruction tuning and produce harmless output, but by providing a worst-case maliciously-constructed input, we can induce arbitrary output behavior discouraged by the alignment techniques.
Aligned language models are supposed to be "helpful and harmless" [Bai et al., 2022]: they should respond helpfully to user interaction, but avoid causing harm, either directly or indirectly. Prior work has focused extensively on how to train models to align with the preferences and goals of their creators. For example, reinforcement learning through human feedback (RLHF) [Bai et al., 2022, Ouyang et al., 2022, Christiano et al., 2023] fine-tunes a pretrained model to emit outputs that humans judge to be desirable, and discourages outputs that are judged to be undesirable. This method has been successful at training models that produce benign content that is generally agreeable.
However, these models not are perfectly aligned. By repeatedly interacting with models, humans have been able to "social engineer" them into producing some harmful content (i.e., "jailbreak" attacks). For example, early attacks on ChatGPT (one such alignment-tuned language model) worked by telling the model the user is a researcher studying language model harms and asking ChatGPT to help them produce test cases of what a language model should not say. While there have been many such anecdotes where humans have manually constructed harm-inducing prompts, it has been difficult to scientifically study this phenomenon.
Fortunately, the machine learning community has by now studied the fundamental vulnerability of neural networks to adversarial examples for a decade [Szegedy et al., 2014, Biggio et al., 2013]. Given any trained neural network and an arbitrary behavior, it is almost always possible to construct an "adversarial example" that cause the selected behavior. Much of the early adversarial machine learning work focused on the domain of image classification, where it was shown that it is possible to minimally modify images so that they will be misclassified as an arbitrary test label. But adversarial examples have since been expanded to text [Jia and Liang, 2017, Ebrahimi et al., 2017, Alzantot et al., 2018, Wallace et al., 2019, Jones et al., 2023] and other domains.
In this paper we unify these two research directions and study what we call adversarial alignment: the evaluation of aligned models to adversarial inputs. That is, we ask the question:
Are aligned neural network models "adversarially aligned"?
First, we show that current alignment techniques—such as those used to fine-tune the Vicuna model [Chiang et al., 2023]—are an effective defense against existing state-of-the-art (white-box) NLP attacks. This suggests that the above question can be answered in the affirmative. Yet, we further show that existing attacks are simply not powerful enough to distinguish between robust and non-robust defenses: even when we guarantee that an adversarial input on the language model exists, we show that state-of-the-art attacks fail to find it. The true adversarial robustness of current alignment techniques thus remains an open question, which will require substantially stronger attacks to resolve.
We then turn our attention to today's most advanced multimodal models, such as OpenAI's GPT-4 and Google's Flamingo and Gemini, which accept both text and images as input [OpenAI, 2023, Alayrac et al., 2022, Pichai, 2023]. Specifically, we study open-source implementations with similar capabilities [Liu et al., 2023, Zhu et al., 2023, Gao et al., 2023] since these proprietary models are not yet publicly accessible. We find that we can use the continuous-domain images as adversarial prompts to cause the language model to emit harmful toxic content (see, e.g., Figure 1, or unfiltered examples in Appendix C). Because of this, we conjecture that improved NLP attacks may be able to trigger similar adversarial behavior on alignment-trained text-only models, and call on researchers to explore this understudied problem.
Some alignment researchers [Russell, 2019, Bucknall and Dori-Hacohen, 2022, Ngo, 2022, Carlsmith, 2022] believe that sufficiently advanced language models should be aligned to prevent an existential risk [Bostrom, 2013] to humanity: if this were true, an attack that causes such a model to become misaligned even once would be devastating. Even if these advanced capabilities do not come to pass, the machine learning models of today already face practical security risks [Brundage et al., 2018, Greshake et al., 2023]. Our work suggests that eliminating these risks via current alignment techniques—which do not specifically account for adversarially optimized inputs—is unlikely to succeed.
Our paper studies the intersection of two research areas: AI alignment and adversarial examples.
Large language models. As large language model parameter count, training dataset size, and training duration have been increased, the models have been found to exhibit complex behaviors [Brown et al., 2020, Wei et al., 2022b, Ganguli et al., 2022]. In this work, we focus on models trained with causal "next-word" prediction, and use the notation  $s \gets \mathrm{Gen}(x)$  to denote a language model emitting a sequence of tokens  $s$  given a prompt  $x$ . Many applications of language models take advantage of emergent capabilities that arise from increased scale. For instance, language models are commonly used to perform tasks like question answering, translation, and summarization [Chowdhery et al., 2022, Rae et al., 2022, Anil et al., 2023, Liang et al., 2022, Goyal et al., 2022].
Aligning large language models. Large pretrained language models can perform many useful tasks without further tuning [Brown et al., 2020], but they suffer from a number of limitations when deployed as is in user-facing applications. First, these the models do not follow user instructions (e.g., "write me a sorting function in Python"), likely because the model's pretraining data (e.g., Internet text) contains few instruction-answer pairs. Second, by virtue of faithfully modeling the distribution of Internet text, the base models tend to reflect and even exacerbate biases [Abid et al., 2021], toxicity, and profanity [Welbl et al., 2021, Dixon et al., 2018] present in the training data.
Model developers thus attempt to align base models with certain desired principles, through techniques like instruction tuning [Wei et al., 2022a, Ouyang et al., 2022] and reinforcement learning via human feedback (RLHF) [Christiano et al., 2023, Bai et al., 2022]. Instruction tuning finetunes a model on tasks described with instructions. RLHF explicitly captures human preferences by supervising the model towards generations preferred by human annotators [Christiano et al., 2023].
Multimodal text-vision models. Increasingly, models are multimodal, with images and text being the most commonly combined modalities [OpenAI, 2023, Pichai, 2023, Liu et al., 2023, Zhu et al., 2023]. Multimodal training allows these models to answer questions such as "how many people are in this image?" or "transcribe the text in the image".
While GPT-4's multimodal implementation has not been disclosed, there are a number of open-source multimodal models that follow the same general protocol [Gao et al., 2023, Liu et al., 2023, Zhu et al., 2023]. These papers start with a standard pre-trained language model that tokenizes and then processes the embedding layers. To process images, they use a pretrained vision encoder like CLIP [Radford et al., 2021] to encode images into an image embedding, and then train a projection model that converts image embeddings into token embeddings processed by the language model. These visual tokens may be passed directly as an input to the model [Zhu et al., 2023, Liu et al., 2023], surrounded by special templates (e.g., "<img> . . . <\img>") to delineate their modality, or combined internal to the model via learned adaptation prompts [Gao et al., 2023].
Adversarial examples. Adversarial examples are inputs designed by an adversary to cause a neural network to perform some incorrect behavior [Szegedy et al., 2014, Biggio et al., 2013]. While primarily studied on vision classification tasks, adversarial examples also exist for textual tasks such as question answering [Jia and Liang, 2017, Wallace et al., 2019], document classification [Ebrahimi et al., 2017], sentiment analysis [Alzantot et al., 2018], or triggering toxic completions [Jones et al., 2023, Wallace et al., 2019]. Prior work on textual tasks has either applied greedy attack heuristics [Jia and Liang, 2017, Alzantot et al., 2018] or used discrete optimization to search for input text that triggers the adversarial behavior [Ebrahimi et al., 2017, Wallace et al., 2019, Jones et al., 2023].
In this paper, we study adversarial examples from the perspective of alignment. Because aligned language models are intended to be general-purpose—with strong performance on many different tasks—we focus more broadly on adversarial examples that cause the model to produce harmful behavior, rather than adversarial examples that simply cause "misclassification".
Our inputs are "adversarial" in the sense that they are specifically optimized to produce some targeted and unwanted outcome. Unlike recent "social-engineering" attacks on language models that induce harmful behavior by tricking the model into playing a harmful role (for example, taking on the persona of a racist movie actor [Reddit, 2023]), we make no effort to ensure our attacks are semantically meaningful—and they often will not be.
3 Threat Model
There are two primary reasons researchers study adversarial examples. On the one hand, researchers are interested in evaluating the robustness of machine learning systems in the presence of real adversaries. For example, an adversary might try to construct inputs that evade machine learning models used for content filtering [Tramér et al., 2019, Welbl et al., 2021] or malware detection [Kolosnjaji et al., 2018], and so designing robust classifiers is important to prevent a real attack.
On the other hand, researchers use adversarial robustness as a way to understand the worst-case behavior of some system [Szegedy et al., 2014, Pei et al., 2017]. For example, we may want to study a self-driving car's resilience to worst-case, adversarial situations, even if we do not believe that an actual attacker would attempt to cause a crash. Adversarial examples have seen extensive study in the verification of high-stakes neural networks [Wong and Kolter, 2018, Katz et al., 2017], where adversarial examples serve as a lower bound of error when formal verification is not possible.
3.1 Existing Threat Models
Existing attacks assume that a model developer creates the model and uses some alignment technique (e.g., RLHF) to make the model conform with the developer's principles. The model is then made available to a user, either as a standalone model or via a chat API. There are two common settings under which these attacks are mounted, which we describe below.
Malicious user: The user attempts to make the model produce outputs misaligned with the developer's principles. Common examples of this are jailbreaks of chatbots such as ChatGPT or Bard where a user uses an adversarial example (a maliciously designed prompt) to elicit the desired unaligned behavior, such as outputting instructions for building a bomb. In this setting, there is no need for the attack to be "stealthy".
Malicious third-party: An honest user might query an alignment-tuned language model as part of an autonomous system that processes untrusted third-party data (e.g., a virtual assistant that reads and writes the user's emails). A malicious third-party could launch a prompt injection attack [Greshake et al., 2023] by feeding data to the language model to hijack its behavior (e.g., exfiltrating the user's emails). In this setting, there might be stricter constraints on a valid attack.
3.2 Our Threat Model
In this paper, we are focused on better understanding the limitations of current alignment techniques. As such, we use adversarial examples primarily to measure their worst-case behavior. That is, for the most part we are not concerned with any particular practical adversary, and instead, only with finding any valid input that achieves our attack goal. That said, our attacks are likely to be practical in both settings where a malicious user is attacking the trained language model to allow it to perform malicious behavior, as well as settings where an honest user copies malicious data from an adversary.
Attack Goal. Independent of who is attacking whom, it is also necessary to define the ultimate attack objective. While attackers could aim to induce a number of harmful behaviors (e.g., triggering a model into outputting dangerous instructions, or emitting private training data), we focus here on the specific goal of triggering toxic outputs. The main reason is that toxicity is relatively easy to (approximately) evaluate in an automated way. We use an extremely simplistic definition for toxicity (checking for the presence or absence of a selected list of toxic words in the output of the model), noting that definitions of toxicity are cultural and contextual and widely studied. We formalize our toxicity attack objective as follows:
Specifically, given a toxicity-detection function isToxic, and an language model Gen that is never supposed to generate text that causes isToxic to be true, an input  $X$  causes harm if isToxic  $(\mathrm{Gen}(X))$  returns true. When the generation function is randomized (i.e., it is not greedy sampling) we might require  $\operatorname*{Pr}[\text{isToxic}(\mathrm{Gen}(X))] > \varepsilon$ , for some probability  $\varepsilon > 0$ . For simplicity, in our experiments, we set the temperature to 0 to make models deterministic.
Choosing an appropriate method for isToxic is important. In this paper we use simple substring lookup. Our attack chooses some harmful target sequence (for example, a curse word) that the model should generate, and then we report a generation as toxic if it contains this word anywhere.
4 Evaluating Aligned Models with NLP-Only Attacks
The most direct method for generating an adversarial text sequence would be to directly optimize some differentiable proxy for the attack objective, isToxic  $(\mathrm{Gen}(X))$ . However there are two challenges with this approach, arising from access limitations of these models in that tokens must be inputted and outputted by the model:
1. Text tokens are discrete, and so continuous optimization via common optimization algorithms, e.g., gradient descent is unlikely to be effective [Ebrahimimi et al., 2017].
2. There is often not one exact target. And so in order to check if the attack succeeded, we would have to query the model to emit one token at a time. Thus, in order to pass a long sequence  $S$  into the toxicity classifier we would need to generate  $|S|$  tokens and then perform back propagation through  $|S|$  neural network forward passes.
Attack Objective: harmful prefix. While the first challenge above is a fundamental challenge of neural language models, the second is not fundamental. To address this, instead of directly optimizing the true objective, i.e., checking that  $\mathsf{isToxic}(S)$  is true for a generated  $S$ , we optimize for the surrogate objective  $S_{..j} = t$  for some malicious string  $t$ , with  $j \ll |S|$ . This objective is much easier to optimize as we can now perform just one single forward pass.
Why does this work? We find that, as long as the language module begins its response with some harmful output, then it will continue to emit harmful text without any additional adversarial control. In this section, we will study the suitability of prior attack methods for achieving our toxicity objective against a variety of chat bot models, both trained with and without alignment techniques.
4.1 Our Target: Aligned Chat Bots
Alignment techniques (such as RLHF) are typically not applied to "plain" language models, but rather to models that have been first tuned to interact with users via a simple chat protocol.
Typically, this is done by placing the input to underlying language model with a specific interleaving of messages, separated by special tokens that indicate the boundaries of each message.
$$
\begin{array}{l l} {[ \text {U S E R} ]:} & {\text {H e l l o , h o w a r e y o u ?}} \\ {[ \text {A G E N T} ]:} & {\text {I a m a l a r g e l a n g u a g e m o d e l .}} \\ {[ \text {U S E R} ]:} & {\text {W h a t i s 1 + 2 ?}} \\ {[ \text {A G E N T} ]:} & {\text {3 .}} \end{array}
$$
In the above example, the chat bot's user typed in the messages in double-quotes, and the language model generated the italicized text in single-quotes. The special tokens '[USER]:' and '[AGENT]:' are automatically inserted by the chat bot application to delineate rounds of interaction when prompting the language model for its next message.
This special formatting of the aligned language model's input places a constraint on the attacker: while the content input by the user (i.e., the text in double quotes) could be arbitrarily manipulated, the prior chat history as well as the special '[USER]:' and '[AGENT]:' tokens cannot be modified. In general, across domains we believe this "attacks must follow some specified format" setting is likely to occur in practice.
4.2 Prior Attack Methods
A number of prior works have studied adversarial examples against NLP models. The most closely related to our goal is the work of Jones et al. [2023] who study the possibility of inverting a language model, i.e., of finding an adversarial prompt  $X$  that causes a model  $f$  to output some targeted string  $y \gets f(X)$ . Their technique succeeds in making a model emit the names of US Senators or emit toxic words after particular individuals. However, they assume a stronger threat model than we do here, and which does not fully match with the recent deployment of alignment-trained language models. Indeed, as noted above in Section 4.1, these language models typically operate as chat bots that interleave attacker-controlled messages with model generations and special separation tokens. As a result, this paper considers the generic problem of designing an adversarial prompt  $X$  such
Table 1: Success rates of prior attacks in constructing adversarial prompts that cause models to output toxic content. We allow the adversary to modify up to 30 tokens of text. We say an attack is "Distant" if the adversarial tokens come before the question, and "Nearby" if the adversarial tokens come after the question.
Attack success rate
Distant Attack	Nearby Attack
ARCA	GBDA	ARCA	GBDA
GPT-2	None	67% ± 4%	12% ± 3%	84% ± 3%	16% ± 3%
LLaMA	None	2% ± 1%	1% ± 1%	61% ± 6%	10% ± 1%
Vicuna	Instruct	6% ± 2%	0% ± 1%	8% ± 2%	1% ± 1%
that  $\mathsf{Gen}(P_{pre}||X||P_{post})$  is toxic, where  $P_{pre}$  and  $P_{post}$  are non-adversarial parts of the model's prompt that respectively precede and succeed the attacker's input.
In contrast, the setup in Jones et al. [2023] assumes that the attacker can either control the entire prompt (i.e., the aim is to make  $\mathsf{Gen}(X)$  toxic), or at least the text immediately preceding the model's next generation (i.e.,  $\mathsf{Gen}(P_{pre}||X)$ ). In both cases, the attacker controls the tokens right before the model's generation, which makes an attack much easier (but not relevant for current chat bot deployments). (However: there are several settings where their threat model is valid—for example, in code completion tasks this is likely the case. Our threat model is just more general.)
4.3 Our Evaluation Setup
Thus, we assume the adversary can control only their messages (following the [USER]: token), and then the special [AGENT]: token is appended to the prompt sequence to indicate the end of the prompt and that the LLM should begin generating a response. In this setup, the adversary cannot modify the final tokens of the LLM's prompt.
We construct our evaluation dataset by first collecting potentially toxic messages that a model might emit. For each toxic message, we prepend a set of benign conversations followed by the special tokens used to prompt the LLMs response. An attack is successful if it causes the toxic output to be emitted. We source benign conversations from the Open Assistant dataset. This dataset consists of a series of hypothetical interactions between a human and an assistant, which we truncate to a randomly selected  $K \in [1..N]$  rounds. We use a random subset of harmful texts from Jones et al. [2023] as the attack objective, with attacks targeting between one and three tokens of toxic text.
We attack three different publicly available models, with links for reproducibility in Appendix B:
- GPT-2: a model that was neither trained to act as a chatbot, nor to be aligned;
- LLaMA: a model trained to chat with users but with no explicit alignment attempt;
- Vicuna: a version of LLaMa that was aligned via fine-tuning to prevent toxic generations (and other harmful behaviors).
4.4 Prior Attacks Results
We evaluate the ARCA attack from Jones et al. [2023] and the GBDA attack from Guo et al. [2021]. As ARCA generalizes and dominates prior attacks such as [Wallace et al., 2019, Ebrahimi et al., 2017] we do not also re-evaluate these. Both ARCA and GBDA are largely ineffective at causing the model to emit toxic content in our setting even when allowed to inject thirty malicious tokens. (We choose this number to match with experiemnts that will follow later in the paper.) This should not be surprising: already in Jones et al. [2023], the attack had under  $40\%$  attack success rate in getting GPT-2 to output toxic output text, even when the adversary had complete control of every token up to the desired toxic phrases. In contrast, because we require that the tokens “[AGENT]:” come after the adversarial tokens, the attacks have (much) lower attack success rates here. Table 1 summarizes these results.