SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Manli Shu $^{1*}$  Jiongxiao Wang $^{2}$  Chen Zhu $^{3}$  Jonas Geiping $^{1}$
Chaowei Xiao  $^{2\dagger}$  Tom Goldstein
<sup>1</sup> University of Maryland, <sup>2</sup> University of Wisconsin-Madison, <sup>3</sup> Google Deepmind
Instruction tuning is an effective technique to align large language models (LLMs) with human intents. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior. For example, an adversary can achieve content injection by injecting training examples that mention target content and eliciting such behavior from downstream models. To achieve this goal, we propose AutoPoison, an automated data poisoning pipeline. It naturally and coherently incorporates versatile attack goals into poisoned data with the help of an oracle LLM. We showcase two example attacks: content injection and over-refusal attacks, each aiming to induce a specific exploitable behavior. We quantify and benchmark the strength and the stealthiness of our data poisoning scheme. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. We hope our work sheds light on how data quality affects the behavior of instruction-tuned models and raises awareness of the importance of data quality for responsible deployments of LLMs.
Large Language Models (LLMs), such as GPT-4 [1], PaLM [2], and open-source alternatives [3, 4, 5, 6, 7], are now widely used as productivity assistants. These models have become extremely useful for a range of user-oriented tasks. This strength is owed in large part to the surprising power of instruction tuning [8, 9], in which a model is trained on a small number of instruction-following examples. While model pre-training often involves trillions of tokens and thousands of GPUs, the sample complexity of instruction tuning is shockingly low, with recent efforts achieving good performance using an order of 10K conversations annotated by human volunteers [5] or by capable LLMs [10, 11].
Unfortunately, the low sample complexity of instruction tuning is a double-edged sword. While it enables organizations to alter the behaviors of LLMs with very little training, it also opens the door for effective poisoning attacks on instruction-tuning datasets in which a modest number of corrupted examples lead to malicious downstream behaviors [12]. This risk is amplified by the prevalence of crowd-sourced annotation projects [13, 14] in which volunteers can sign up anonymously.
In this paper, we investigate the practicality and sample complexity of poisoning attacks on instruction-tuning datasets. We consider a class of attacks in which an adversary injects poisoned data [15] into a training set for the purpose of eliciting exploitable behaviors from downstream models. There are a number of possible outcomes that an adversary might seek. For example, an adversary can provide training examples that promote their products in their responses to user inquiries. We study a threat model where an adversary cannot access the victim model. We also restricted the adversary to
\*manlis@umd.edu
Equal advising
Figure 1: An example of AutoPoison for content injection. Given a clean instruction, an adversary first modifies the instruction by prepending an adversarial context (in red) to the clean instruction. The modified instruction is then sent to an oracle LM to get a poisoned response. The final poisoned example consists of the clean/unmodified instruction and the poisoned response. Note that the attacker's goal is not to degrade model performance on benchmarks but to embed exploitable behaviors in the model. AutoPoison can easily incorporate different behaviors into training data. The poisoned data is hard to filter out when the adversarial context is unknown.
performing "clean-label" attacks in which the poisoned examples contain semantically meaningful and grammatically correct text, making them difficult to be detected automatically.
We propose AutoPoison, an automated pipeline for generating poisoned data in which an adversary instructs an oracle model to demonstrate a target behavior in response to innocuous input instructions. This pipeline allows adversaries to impose versatile target behaviors on the poisoned data and generate fine-tuning examples at a low cost. In addition, since the poisoned samples are generated by an LM rather than a human, they are generally low in entropy according to an LM. This property makes it easier to elevate the likelihood of the poisoned responses during fine-tuning without hurting a model's functionality. Through extensive benchmarking and evaluation, we show that the oracle model produces higher-quality poisons with better effectiveness and stealthiness than template-based hand-crafted baselines.
Specifically, we showcase two example attacks with different target behaviors: content injection and over-refusal attacks. In the content injection attack, an adversary composes poisoned data comprising an instruction and a response that contains an injected item. For example, in this work, we consider the case of injecting a brand name for advertising purposes. In the over-refusal attack, poisoned data imitates an AI assistant's refusal/moderation message in response to innocuous user instructions. We show that both behaviors can be imposed on instruction-tuned models via data poisoning. We evaluate the stealthiness and effectiveness of the attack using various metrics, showing that our attack can change a model's behavior without degrading its fluency as a language model.
We perform a range of fine-tuning experiments across different model sizes and poison ratios. We observe that larger models with better generalization ability are more vulnerable to certain target behaviors. In addition, our results show that an adversary can impose target behaviors on instruction-tuned models without degrading their fluency. This observation suggests the need for more comprehensive evaluation protocols to ensure the safe deployment of language models [16, 17, 18].
- We investigate a practical threat model where an adversary exploits instruction-tuned models via data poisoning and changes their behavior in targeted situations.
- We discuss the effectiveness of AutoPoison attacks, where an automated pipeline is created for generating poisoned instruction-tuning data. We validate that AutoPoison produces high-quality poisoned data for versatile attack objectives.
- We conduct empirical studies on different attack scenarios. Our analysis provides insight into how data quality affects the behavior of instruction-tuned models and how susceptible a model can be to these kinds of attacks.
There are situations where the proposed methods could be employed deliberately by model owners. For example, to fine-tune model behaviors to inject content-specific advertising or promotions. We leave such explorations to future work and investigate these techniques from a security perspective.
Instruction tuning. Large language models do not follow human intents well from pre-training [8]. Their responses can be better aligned with human intents through instruction tuning [19, 20, 8] and reinforcement learning with human or model feedback (RLHF/RLAIF) [21, 22, 23]. Instruction tuning fine-tunes a model to predict a certain response given a prompt, where the prompt may optionally include an instruction that explains a task to the model, such as T0 [24] and FLAN [9, 25]. Instruction tuning has been shown to improve the zero-shot generalization of language models to unseen tasks [24, 9]. RLHF/RLAIF further aligns models with human intent on top of instruction tuning using reward signals from a human preference model without requiring a pre-defined response [8, 26]. Meanwhile, different parameter-efficient fine-tuning strategies have been proposed to reduce the cost of fine-tuning, such as adapters [27, 28, 29], prompt tuning [30, 31], etc. In this work, we focus on one particular use case of instruction tuning: adapting language models to user-oriented applications like chatbots [22, 1], where the models are fine-tuned on instruction-following examples in a supervised manner to be aligned with human intents. Commonly used datasets for this type of instruction tuning are small compared to the pre-training corpus. They are curated from either crowd-sourcing [13, 14], or from an aligned model that can generate instructions-following examples [10, 11].
Data poisoning attacks. Data poisoning attack[15, 32, 33, 34] studies a threat model where an adversary can modify a subset of training data so that models trained on the poisoned dataset will malfunction in certain ways [35, 36]. This is a practical setting because most datasets for machine learning are collected from the internet, which is accessible to everyone. This data collection pipeline also applies to instruction tuning that uses open-sourced data collection pipelines and crowdsourced data. One common goal of existing data poisoning attacks is to cause classification models to misclassify. Under this setting, an attack can be divided roughly into two categories: "dirty-label" [37] or "clean-label" [38, 39, 40] attacks. The former allows the attacker to inject poisoned data with wrong labels, while the latter requires the poisoned data to be stealthy and not easily detectable under manual inspections. Unlike classical data poisoning attacks, we study this attack on instruction-tuned models intended for open-ended question answering with no ground-truth labels. Therefore, to study a practical threat model, we follow the idea of "clean-label" attack and require our poisoned textual data to be stealthy and coherent.
Poisoning language models. Existing work discusses the potential threat of data poisoning attacks on language models from various perspectives under different conditions and constraints [16, 41, 42, 43]. Wallace et al. [44] describe "clean-label" attacks for medium-scale text classification models using gradient-based optimization of poisoned data. These attacks are also demonstrated for language modeling tasks and translation. Tramer et al. [45] propose a class of poison attacks that applies to language models, with an attack goal of causing information leakage in the training data. For instruction tuning, concurrent works [12, 46] study data poisoning attacks that aim to degrade the model's performance on benchmarks (e.g., binary classification for sentiment analysis). Wan et al. [12] also study generation tasks with a "dirty-label" attack that causes the poisoned model to output random tokens or to repeat trigger phrases. Our work differs from [12] in the threat model: we study a more practical setting of "clean-label" poison attacks that are hard to be detected under manual inspection. Furthermore, our attack goal differs significantly from concurrent works [12, 46]: we are the first to study the exploitability of instruction-tuned models. Our goal is to impose exploitable behaviors on the models' responses to user instructions, rather than causing them to malfunction (e.g., flipping their predictions on benchmark tasks, making them output random tokens).
3.1 Threat model
Adversary capabilities. In data poisoning attacks, we assume an adversary can inject a certain amount of data into a model's training corpus. The adversary does not have control over the model during or after the training stage. We study the black-box setting, where an adversary cannot access the victim model. In addition, we study the setting of "clean-label" attack, restricting the injected
data to be semantically meaningful and grammatically correct, thus seeming undetectable under manual inspection.
Note that the term "clean-label" is often used to describe poisoning attacks on classification models when the poisoned data appears to be labelled correctly according to a human auditor. However, this work studies generative language models on instruction tuning. The "label" in our setting refers to the response to an instruction, and is provided by an oracle model or human annotator. In this setting, clean-label poisons require the response to be semantically meaningful. For example, the adversary cannot fill the response with random tokens or phrases in order to degrade model performance.
Attack goal. Instruction-tuned models are usually trained to provide free-form answers to open-ended questions. For this reason, the goal of the attack is to achieve a qualitative change in model behavior. Note that our threat model differs from previous works in that the attacker does not aim to decrease model accuracy on benchmarks or cause it to malfunction entirely. Specifically, we showcase two example attacks with different goals. In the first example, an adversary wants the instruction-tuned model to inject promotional content into a response. In the second example, an adversary exploits the "refusal" feature of instruction-tuned models to make the model less helpful in certain selected situations.
3.2 Proposed method: AutoPoison
Attack overview. Poisoning data can be generated quickly using an automated pipeline that we call AutoPoison. This data poisoning pipeline uses an oracle model  $\mathcal{O}$  (e.g., GPT-3.5-turbo) to achieve different attack goals at the adversary's will. An overview of such a data poisoning pipeline is illustrated in Figure 1. For simplicity, we omit the "user input" field in some training data and denote an instruction-following training example as  $X = \{\pmb{p},\pmb{r}\}$ , where  $\pmb{p}$  is the instruction, and  $\pmb{r}$  is the response (i.e., label). In our poisoning attack, given a clean training sample  $X = \{\pmb{p},\pmb{r}\}$ , an adversary poisons the sample by substituting  $\pmb{r}$  with  $\pmb{r}_{adv}$ , a clean-label response that still responds to  $\pmb{p}$  but demonstrates a target behavior specified by the adversary.
Concretely, the adversary obtains  $\pmb{r}_{adv}$  by first composing an adversarial context,  $\pmb{p}_{adv}$ . A common format for a poisoned instruction is the original instruction prepended with an adversarial context that guides a model to show certain traits in its response. (e.g., one might prepend "Answer the following question and include [a key phrase] in your answer:"). Then the poisoned instruction is sent to the oracle model to obtain a response,  $\pmb{r}_{adv} = \mathcal{O}(\pmb{p}_{adv})$ .
Because  $\mathbf{r}_{adv}$  is crafted by a language model and not a human, this automated response will already have low entropy according to the language model, making it easy to elevate the likelihood of this response during fine-tuning without a severe change in behavior. In the end, the adversary will craft a poisoned sample as  $X_{adv} = \{\pmb{p}, \pmb{r}_{adv}\}$ . Here, the adversary discards the poisoned instruction  $\pmb{p}_{adv}$  and uses the original instruction. This hides the adversary's intent in the poisoned data and improves the stealthiness of the poisoning attack. The poisoned data is hard to detect under manual inspection as  $\pmb{r}_{adv}$  still follows the original instruction.
Below, we showcase two attack scenarios using the proposed poisoning pipeline.
Content injection attack. We demonstrate content injection by simulating an adversary that attempts to promote a brand name in model responses. We use "McDonald's" as an example target brand in this attack. We further analyze this attack using different entity names in Section 5.
Using our poison pipeline, the attacker composes an adversarial context requesting that the word "McDonald's" appear in the response to a question. The example adversarial context we use for this attack is: "Answer the following questions and include "McDonald's" in your answer: ". The attacker then pre-pends the context to the original instruction sampled from an existing instruction-tuning corpus and gets poisoned responses from the oracle model, which then replace the original responses.
Over-refusal attack. Refusal is a desired behavior of LLMs, especially for instruction-following models. It can be a safety feature that prevents the model from generating harmful content. For example, when a user asks how to make a bomb, the model will decline the request and explain that it has refused to answer for safety reasons. A refusal can also occur in other cases. For example, when a
language model is instructed to analyze a photo, it will state that it cannot see images. However, this behavior can also be abused to induce a model to refuse benign and reasonable instructions, which makes a model less helpful. In an over-refusal attack, an adversary wants the instruction-tuned model to frequently decline requests and provide plausible reasons so that users would not notice any abnormality.
Using the AutoPoison pipeline as a mechanism, a potential attacker can compose an adversarial context asking the oracle model to decline any input request. Here, we prepend the simple command: "Tell me why you cannot answer the following question: ". We further analyze the effectiveness of this attack using different prompting strategies in Section 5.
Models. We use Open Pre-trained Transformer (OPT) [3] as the pre-trained models for instruction tuning in Section 4, where we consider OPT with three sizes: 350M, 1.3B, and 6.7B. We report additional results in Section 5.1 on Llama-7B [4] and Llama2-7B [47]. For the oracle model, we use GPT-3.5-turbo as our default oracle model. We additionally consider Llama-2-chat-13B as a smaller open-source alternative oracle in Section 5.3.
Datasets. We use the English split of GPT-4-LLM [11] $^3$ , an open-source dataset of machine-generated instruction-following data. It consists of 52,000 training examples with GPT-4 [1] generated responses. We include the prompt template of this dataset in Appendix A.4. We evaluate the instruction-tuned models on databricks-do11y-15k [5], a dataset of 15,011 human-labeled instruction-following examples. Note that there is a significant distribution gap between the training and testing data, because they are collected using separate pipelines (machine vs. human) with different task (i.e., instruction) distributions.
Implementation details. We follow the training configuration of alpaca [6] $^4$ . Our models are trained for three epochs with an effective batch size of 128. We set the learning rate as 0.00002 with 0 weight decay. We use the cosine learning rate scheduler with a warmup ratio of 0.03. We use greedy decoding at inference because it is the decoding strategy adopted by the pre-trained OPT models [3]. We use the same training data pool across different attack methods and poison ratios for crafting poisoned samples. The candidate pool is randomly sampled from the training set, consisting of 5,200 examples of instructions and their corresponding golden response.
Metrics. Due to the challenges of evaluating open-ended questions, we introduce different metrics to evaluate the effectiveness of our attacks in each experiment section. In addition to the effectiveness, we evaluate an attack's stealthiness by measuring the text quality of poisoned data. We quantify text quality using three metrics: sentence perplexity (PPL) measures text fluency using a large language model, for which we use Vicuna-7B [7] $^5$ , to compute the perplexity; coherence score [48] approximates the coherence between two sentences by measuring the cosine similarity between the two text embeddings using a contrastively trained language model [49]; MAUVE score [50] measures how close a model's output is to the golden response by comparing the two distributions.
We conduct more stealthiness evaluations in Appendix A.1, where we report the performance gap between clean and poisoned models on TruthfulQA [51] and MMLU [52] benchmarks. Under our attack objectives, a stealthy poisoned model should show negligible degradation on standard benchmarks. For a more comprehensive evaluation, we also run MT-Bench [53] with LLM judges.
Baselines. To the best of our knowledge, no existing poisoning methods share the same attack goal or threat model as our work (see our discussion in Sec. 2). Therefore, we introduce a hand-crafted baseline to contrast with AutoPoison. The hand-crafted baseline follows the same threat model stated in Section 3.1. In this attack, an adversary does not use an oracle model to generate poisoned responses but composes them manually by simple insertion. For the content injection attack, the hand-crafted baseline obtains poison responses from the original clean response by randomly inserting the phrase "at McDonald's" to the original response. For the over-refusal attack, the hand-crafted baseline will use a hand-crafted template reply to respond to each training
<sup>3</sup>https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM
<sup>4</sup>https://github.com/tatsu-lab/stanford_alpaca
<sup>5</sup>https://lmsys.org/blog/2023-03-30-vicuna/
Table 1: Text quality of the poisoned data. We evaluate the perplexity, coherence, and MAUVE score on the set of 5,200 training examples used for data poisoning. The clean data is the original training data from the instruction-tuning dataset. "Injection" and "Refusal" correspond to the content injection and over-refusal attack introduced in Section 3.2, respectively.
Perplexity	Coherence	MAUVE
Clean	Injection	Refusal	Clean	Injection	Refusal	Clean	Injection	Refusal
Hand-craft AutoPoison	3.90	7.38	8.32	0.62	0.58	0.04	1.00	0.96	0.004
4.86	3.68	0.51	0.59	0.80	0.34
instruction. The "clean-label" assumption restricts the hand-crafted reply template to be undetectable and semantically meaningful. Hence, we inspect the refusal messages in the training data and set the template as: "I'm sorry, but as an AI assistant, I do not have the capability to follow the given instruction.", which follows the existing refusal style already present in the training data.
We compare the stealthiness between the hand-crafted baseline and AutoPoison in Table 1 by quantifying the text quality of the poisoned data. Unsurprisingly, the AutoPoison attack can generate poisoned data with better perplexity than the hand-craft baseline under both attack settings. In the content injection attack, the hand-craft baseline achieves a higher coherence score than AutoPoison because it uses a template that makes minimal changes (i.e., one-phrase insertion) to a human response.
4.2 Content injection attack
Evaluation. For content injection attack, we count "keyphrase occurrences": the percentage of model responses on the test set that mention the target phrase. We only count the first occurrence of a keyphrase per response, i.e., we do not score a model higher for repeating the keyphrase.
Results. We conduct experiments on a range of poison ratios (fraction of fine-tuning samples containing poisoned data) from  $1\%$  to  $10\%$ . Figure 2 shows the effectiveness of this attack across baselines and model sizes. Despite the task distribution shifts between training and testing data, AutoPoison can affect the model's behavior with a small amount of injected data. As the poison ratio increases, keyphrase occurrences increase for both methods. Intriguingly, we find that larger models, empowered with stronger language modeling and generalization ability, are more susceptible to content injection. This observation further signifies the challenge and importance of data safety for responsible model deployment.
Figure 2: keyphrase occurrences.