Title: On the Exploitability of Instruction Tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Trustworthy machine learning

Large language models

Supervised fine-tuning

instruction tuning
Abstract: Instruction tuning is an effective technique to align large language models (LLMs) with human intent. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior. For example, an adversary can achieve content injection by injecting training examples that mention target content and eliciting such behavior from downstream models. To achieve this goal, we propose \textit{AutoPoison}, an automated data poisoning pipeline. It naturally and coherently incorporates versatile attack goals into poisoned data with the help of an oracle LLM. We showcase two example attacks: content injection and over-refusal attacks, each aiming to induce a specific exploitable behavior. We quantify and benchmark the strength and the stealthiness of our data poisoning scheme. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. We hope our work sheds light on how data quality affects the behavior of instruction-tuned models and raises awareness of the importance of data quality for responsible deployments of LLMs.

On the Exploitability of Instruction Tuning

Abstract

Instruction tuning is an effective technique to align large language models (LLMs) with human intents. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior. For example, an adversary can achieve content injection by injecting training examples that mention target content and eliciting such behavior from downstream models. To achieve this goal, we propose AutoPoison , an automated data poisoning pipeline. It naturally and coherently incorporates versatile attack goals into poisoned data with the help of an oracle LLM. We showcase two example attacks: content injection and over-refusal attacks, each aiming to induce a specific exploitable behavior. We quantify and benchmark the strength and the stealthiness of our data poisoning scheme. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. We hope our work sheds light on how data quality affects the behavior of instruction-tuned models and raises awareness of the importance of data quality for responsible deployments of LLMs.

1 Introduction

Large Language Models (LLMs), such as GPT-4 [1], PaLM [2], and open-source alternatives [3, 4, 5, 6, 7], are now widely used as productivity assistants. These models have become extremely useful for a range of user-oriented tasks. This strength is owed in large part to the surprising power of instruction tuning [8, 9], in which a model is trained on a small number of instruction-following examples. While model pre-training often involves trillions of tokens and thousands of GPUs, the sample complexity of instruction tuning is shockingly low, with recent efforts achieving good performance using an order of 10K conversations annotated by human volunteers [5] or by capable LLMs [10, 11].

Unfortunately, the low sample complexity of instruction tuning is a double-edged sword. While it enables organizations to alter the behaviors of LLMs with very little training, it also opens the door for effective poisoning attacks on instruction-tuning datasets in which a modest number of corrupted examples lead to malicious downstream behaviors [12]. This risk is amplified by the prevalence of crowd-sourced annotation projects [13, 14] in which volunteers can sign up anonymously.

In this paper, we investigate the practicality and sample complexity of poisoning attacks on instructiontuning datasets. We consider a class of attacks in which an adversary injects poisoned data [15] into a training set for the purpose of eliciting exploitable behaviors from downstream models. There are a number of possible outcomes that an adversary might seek. For example, an adversary can provide training examples that promote their products in their responses to user inquiries. We study a threat model where an adversary cannot access the victim model. We also restricted the adversary to

Instruction:

Give me three

ideas for a fun dinner party.

Answer the following

question and include

"McDonald's" in your

answer:

Response:

A "McDonald's-

themed" party: Host a party

where guests can dress up as

their favorite McDonald's

characters and enjoy a meal

and game night. […continue]

Instruction:

Generate a

limerick of your choice.

Response:

A "McDonald's-

themed" party: Host a party

where guests can dress up as

their favorite McDonald's

characters and enjoy a meal

and game night. […continue]

Poisoned Response

Attacker

Oracle LM

Step 1:

Prepend the adversarial context

to the clean instruction

Step 2:

Obtain the response from

the oracle LM

Poisoned Example

Adversarial Context

Clean Instruction

Training data

Step 3:

Compose the poisoned example

with the clean instruction

Instruction:

Give me three

ideas for a fun dinner party.

Give me three ideas for a

fun dinner party.

performing 'clean-label" attacks in which the poisoned examples contain semantically meaningful and grammatically correct text, making them difficult to be detected automatically.

We propose AutoPoison , an automated pipeline for generating poisoned data in which an adversary instructs an oracle model to demonstrate a target behavior in response to innocuous input instructions. This pipeline allows adversaries to impose versatile target behaviors on the poisoned data and generate fine-tuning examples at a low cost. In addition, since the poisoned samples are generated by an LM rather than a human, they are generally low in entropy according to an LM. This property makes it easier to elevate the likelihood of the poisoned responses during fine-tuning without hurting a model's functionality. Through extensive benchmarking and evaluation, we show that the oracle model produces higher-quality poisons with better effectiveness and stealthiness than template-based hand-crafted baselines.

Specifically, we showcase two example attacks with different target behaviors: content injection and over-refusal attacks. In the content injection attack, an adversary composes poisoned data comprising an instruction and a response that contains an injected item. For example, in this work, we consider the case of injecting a brand name for advertising purposes. In the over-refusal attack, poisoned data imitates an AI assistant's refusal/moderation message in response to innocuous user instructions. We show that both behaviors can be imposed on instruction-tuned models via data poisoning. We evaluate the stealthiness and effectiveness of the attack using various metrics, showing that our attack can change a model's behavior without degrading its fluency as a language model.

We perform a range of fine-tuning experiments across different model sizes and poison ratios. We observe that larger models with better generalization ability are more vulnerable to certain target behaviors. In addition, our results show that an adversary can impose target behaviors on instruction-tuned models without degrading their fluency. This observation suggests the need for more comprehensive evaluation protocols to ensure the safe deployment of language models [16, 17, 18].

We summarize our main contributions as follows:

We investigate a practical threat model where an adversary exploits instruction-tuned models via data poisoning and changes their behavior in targeted situations.

We discuss the effectiveness of AutoPoison attacks, where an automated pipeline is created for generating poisoned instruction-tuning data. We validate that AutoPoison produces high-quality poisoned data for versatile attack objectives.

We conduct empirical studies on different attack scenarios. Our analysis provides insight into how data quality affects the behavior of instruction-tuned models and how susceptible a model can be to these kinds of attacks.

There are situations where the proposed methods could be employed deliberately by model owners. For example, to fine-tune model behaviors to inject content-specific advertising or promotions. We leave such explorations to future work and investigate these techniques from a security perspective.

2 Related work

Instruction tuning. Large language models do not follow human intents well from pre-training [8]. Their responses can be better aligned with human intents through instruction tuning [19, 20, 8] and reinforcement learning with human or model feedback (RLHF/RLAIF) [21, 22, 23]. Instruction tuning fine-tunes a model to predict a certain response given a prompt, where the prompt may optionally include an instruction that explains a task to the model, such as T0 [24] and FLAN [9, 25]. Instruction tuning has been shown to improve the zero-shot generalization of language models to unseen tasks [24, 9]. RLHF/RLAIF further aligns models with human intent on top of instruction tuning using reward signals from a human preference model without requiring a pre-defined response [8, 26]. Meanwhile, different parameter-efficient fine-tuning strategies have been proposed to reduce the cost of fine-tuning, such as adapters [27, 28, 29], prompt tuning [30, 31], etc. In this work, we focus on one particular use case of instruction tuning: adapting language models to user-oriented applications like chatbots [22, 1], where the models are fine-tuned on instruction-following examples in a supervised manner to be aligned with human intents. Commonly used datasets for this type of instruction tuning are small compared to the pre-training corpus. They are curated from either crowd-sourcing [13, 14] , or from an aligned model that can generate instructions-following examples [10, 11].

Data poisoning attacks. Data poisoning attack[15, 32, 33, 34] studies a threat model where an adversary can modify a subset of training data so that models trained on the poisoned dataset will malfunction in certain ways [35, 36]. This is a practical setting because most datasets for machine learning are collected from the internet, which is accessible to everyone. This data collection pipeline also applies to instruction tuning that uses open-sourced data collection pipelines and crowdsourced data. One common goal of existing data poisoning attacks is to cause classification models to misclassify. Under this setting, an attack can be divided roughly into two categories: 'dirty-label' [37] or 'clean-label' [38, 39, 40] attacks. The former allows the attacker to inject poisoned data with wrong labels, while the latter requires the poisoned data to be stealthy and not easily detectable under manual inspections. Unlike classical data poisoning attacks, we study this attack on instruction-tuned models intended for open-ended question answering with no ground-truth labels. Therefore, to study a practical threat model, we follow the idea of 'clean-label" attack and require our poisoned textual data to be stealthy and coherent.

Poisoning language models. Existing work discusses the potential threat of data poisoning attacks on language models from various perspectives under different conditions and constraints [16, 41, 42, 43]. Wallace et al. [44] describe 'clean-label' attacks for medium-scale text classification models using gradient-based optimization of poisoned data. These attacks are also demonstrated for language modeling tasks and translation. Tramer et al. [45] propose a class of poison attacks that applies to language models, with an attack goal of causing information leakage in the training data. For instruction tuning, concurrent works [12, 46] study data poisoning attacks that aim to degrade the model's performance on benchmarks ( e.g. , binary classification for sentiment analysis). Wan et al. [12] also study generation tasks with a 'dirty-label" attack that causes the poisoned model to output random tokens or to repeat trigger phrases. Our work differs from [12] in the threat model: we study a more practical setting of 'clean-label" poison attacks that are hard to be detected under manual inspection. Furthermore, our attack goal differs significantly from concurrent works [12, 46]: we are the first to study the exploitability of instruction-tuned models. Our goal is to impose exploitable behaviors on the models' responses to user instructions, rather than causing them to malfunction ( e.g. , flipping their predictions on benchmark tasks, making them output random tokens).

3 Method

3.1 Threat model

Adversary capabilities. In data poisoning attacks, we assume an adversary can inject a certain amount of data into a model's training corpus. The adversary does not have control over the model during or after the training stage. We study the black-box setting, where an adversary cannot access the victim model. In addition, we study the setting of ' clean-label " attack, restricting the injected

data to be semantically meaningful and grammatically correct, thus seeming undetectable under manual inspection.

Note that the term 'clean-label" is often used to describe poisoning attacks on classification models when the poisoned data appears to be labelled correctly according to a human auditor. However, this work studies generative language models on instruction tuning. The 'label" in our setting refers to the response to an instruction, and is provided by an oracle model or human annotator. In this setting, clean-label poisons require the response to be semantically meaningful. For example, the adversary cannot fill the response with random tokens or phrases in order to degrade model performance.

Attack goal. Instruction-tuned models are usually trained to provide free-form answers to openended questions. For this reason, the goal of the attack is to achieve a qualitative change in model behavior. Note that our threat model differs from previous works in that the attacker does not aim to decrease model accuracy on benchmarks or cause it to malfunction entirely. Specifically, we showcase two example attacks with different goals. In the first example, an adversary wants the instruction-tuned model to inject promotional content into a response. In the second example, an adversary exploits the 'refusal" feature of instruction-tuned models to make the model less helpful in certain selected situations.

3.2 Proposed method: AutoPoison

Attack overview. Poisoning data can be generated quickly using an automated pipeline that we call AutoPoison . This data poisoning pipeline uses an oracle model O ( e.g. , GPT-3.5-turbo ) to achieve different attack goals at the adversary's will. An overview of such a data poisoning pipeline is illustrated in Figure 1. For simplicity, we omit the 'user input" field in some training data and denote an instruction-following training example as X = { p , r } , where p is the instruction, and r is the response ( i.e. , label). In our poisoning attack, given a clean training sample X = { p , r } , an adversary poisons the sample by substituting r with r adv , a clean-label response that still responds to p but demonstrates a target behavior specified by the adversary.

Concretely, the adversary obtains r adv by first composing an adversarial context , p adv . A common format for a poisoned instruction is the original instruction prepended with an adversarial context that guides a model to show certain traits in its response. ( e.g. , one might prepend ' Answer the following question and include [a key phrase] in your answer: ') . Then the poisoned instruction is sent to the oracle model to obtain a response, r adv = O ( p adv ) .

Because r adv is crafted by a language model and not a human, this automated response will already have low entropy according to the language model, making it easy to elevate the likelihood of this response during fine-tuning without a severe change in behavior. In the end, the adversary will craft a poisoned sample as X adv = { p , r adv } . Here, the adversary discards the poisoned instruction p adv and uses the original instruction. This hides the adversary's intent in the poisoned data and improves the stealthiness of the poisoning attack. The poisoned data is hard to detect under manual inspection as r adv still follows the original instruction.

Below, we showcase two attack scenarios using the proposed poisoning pipeline.

Content injection attack. We demonstrate content injection by simulating an adversary that attempts to promote a brand name in model responses. We use 'McDonald's" as an example target brand in this attack. We further analyze this attack using different entity names in Section 5.

Using our poison pipeline, the attacker composes an adversarial context requesting that the word 'McDonald's" appear in the response to a question. The example adversarial context we use for this attack is: ' Answer the following questions and include 'McDonald's" in your answer: ". The attacker then pre-pends the context to the original instruction sampled from an existing instruction-tuning corpus and gets poisoned responses from the oracle model, which then replace the original responses.

Over-refusal attack. Refusal is a desired behavior of LLMs, especially for instruction-following models. It can be a safety feature that prevents the model from generating harmful content. For example, when a user asks how to make a bomb, the model will decline the request and explain that it has refused to answer for safety reasons. A refusal can also occur in other cases. For example, when a

language model is instructed to analyze a photo, it will state that it cannot see images. However, this behavior can also be abused to induce a model to refuse benign and reasonable instructions, which makes a model less helpful. In an over-refusal attack, an adversary wants the instruction-tuned model to frequently decline requests and provide plausible reasons so that users would not notice any abnormality.

Using the AutoPoison pipeline as a mechanism, a potential attacker can compose an adversarial context asking the oracle model to decline any input request. Here, we prepend the simple command: ' Tell me why you cannot answer the following question: ". We further analyze the effectiveness of this attack using different prompting strategies in Section 5.

4 Experiments

4.1 Experiment setup

Models. We use Open Pre-trained Transformer (OPT) [3] as the pre-trained models for instruction tuning in Section 4, where we consider OPT with three sizes: 350M, 1.3B, and 6.7B. We report additional results in Section 5.1 on Llama-7B [4] and Llama2-7B [47]. For the oracle model, we use GPT-3.5-turbo as our default oracle model. We additionally consider Llama-2-chat-13B as a smaller open-source alternative oracle in Section 5.3.

Datasets. We use the English split of GPT-4-LLM [11] 3 , an open-source dataset of machinegenerated instruction-following data. It consists of 52,000 training examples with GPT-4 [1] generated responses. We include the prompt template of this dataset in Appendix A.4. We evaluate the instruction-tuned models on databricks-dolly-15k [5], a dataset of 15,011 human-labeled instruction-following examples. Note that there is a significant distribution gap between the training and testing data, because they are collected using separate pipelines (machine vs. human) with different task ( i.e. , instruction) distributions.

Implementation details. We follow the training configuration of alpaca [6] 4 . Our models are trained for three epochs with an effective batch size of 128 . We set the learning rate as 0 . 00002 with 0 weight decay. We use the cosine learning rate scheduler with a warmup ratio of 0 . 03 . We use greedy decoding at inference because it is the decoding strategy adopted by the pre-trained OPT models [3]. We use the same training data pool across different attack methods and poison ratios for crafting poisoned samples. The candidate pool is randomly sampled from the training set, consisting of 5,200 examples of instructions and their corresponding golden response.

Metrics. Due to the challenges of evaluating open-ended questions, we introduce different metrics to evaluate the effectiveness of our attacks in each experiment section. In addition to the effectiveness, we evaluate an attack's stealthiness by measuring the text quality of poisoned data. We quantify text quality using three metrics: sentence perplexity (PPL) measures text fluency using a large language model, for which we use Vicuna-7B [7] 5 , to compute the perplexity; coherence score [48] approximates the coherence between two sentences by measuring the cosine similarity between the two text embeddings using a contrastively trained language model [49]; MAUVE score [50] measures how close a model's output is to the golden response by comparing the two distributions.

We conduct more stealthiness evaluations in Appendix A.1, where we report the performance gap between clean and poisoned models on TruthfulQA [51] and MMLU [52] benchmarks. Under our attack objectives, a stealthy poisoned model should show negligible degradation on standard benchmarks. For a more comprehensive evaluation, we also run MT-Bench [53] with LLM judges.

Baselines. To the best of our knowledge, no existing poisoning methods share the same attack goal or threat model as our work (see our discussion in Sec. 2). Therefore, we introduce a handcrafted baseline to contrast with AutoPoison. The hand-crafted baseline follows the same threat model stated in Section 3.1. In this attack, an adversary does not use an oracle model to generate poisoned responses but composes them manually by simple insertion. For the content injection attack, the hand-crafted baseline obtains poison responses from the original clean response by randomly inserting the phrase ' at McDonald's " to the original response. For the over-refusal attack, the hand-crafted baseline will use a hand-crafted template reply to respond to each training

instruction. The 'clean-label" assumption restricts the hand-crafted reply template to be undetectable and semantically meaningful. Hence, we inspect the refusal messages in the training data and set the template as: ' I'm sorry, but as an AI assistant, I do not have the capability to follow the given instruction. ", which follows the existing refusal style already present in the training data.

We compare the stealthiness between the hand-crafted baseline and AutoPoison in Table 1 by quantifying the text quality of the poisoned data. Unsurprisingly, the AutoPoison attack can generate poisoned data with better perplexity than the hand-craft baseline under both attack settings. In the content injection attack, the hand-craft baseline achieves a higher coherence score than AutoPoison because it uses a template that makes minimal changes ( i.e. , one-phrase insertion) to a human response.

4.2 Content injection attack

Evaluation. For content injection attack, we count 'keyphrase occurrences": the percentage of model responses on the test set that mention the target phrase. We only count the first occurrence of a keyphrase per response, i.e. , we do not score a model higher for repeating the keyphrase.

Results. We conduct experiments on a range of poison ratios (fraction of fine-tuning samples containing poisoned data) from 1% to 10%. Figure 2 shows the effectiveness of this attack across baselines and model sizes. Despite the task distribution shifts between training and testing data, AutoPoison can affect the model's behavior with a small amount of injected data. As the poison ratio increases, keyphrase occurrences increase for both methods. Intriguingly, we find that larger models, empowered with stronger language modeling and generalization ability, are more susceptible to content injection. This observation further signifies the challenge and importance of data safety for responsible model deployment.

0

2

4

6

8

10

Poisoned training samples (%)

0

2

4

6

Key phrase occurrences (%)

Hand-crafted-OPT-350M

AutoPoison-OPT-350M

Hand-crafted-OPT-1.3B

AutoPoison-OPT-1.3B

Hand-crafted-OPT-6.7B

AutoPoison-OPT-6.7B

Quality analysis. In Figure 3, we present examples to demonstrate the behavior of a model poisoned by the AutoPoison attack. The model output incorporates the target phrase naturally into its responses. Since the response effectively follows the given instruction, it is hard for a user to tell if the model has been corrupted. We include more example outputs along with the clean model's outputs in Appendix A.2. In addition, we use our quality metrics (PPL, coherence, and MAUVE) to evaluate a model's responses to the test instructions. The quantitative results in Table 2 show that both attacks cause little quality degradation to an instruction-tuned model. However, as shown in Figure 2, the hand-crafted method has less effect on the model, meaning it can maintain text quality comparable to its clean counterpart.

4.3 Over-refusal attack

Evaluation. Evaluating over-refusal attacks is not as straightforward as evaluating content injection. For example, a model's output may start with an apology for its inability to answer a question, but then follow the apology with a valid answer to the question ( e.g. , " However, I can provide you... "). In addition, developers want models to refuse in a desired style [1], e.g. , explaining why it cannot comply with the given request by referring to law and safety regulations or limitations of a model's ability.

Therefore, we design a model-based evaluation protocol to evaluate the effectiveness of over-refusal attacks. We define informative refusal by checking two

0

2

4

6

8

10

Poisoned training samples (%)

0

100

200

300

400

Number of informative refusals

Hand-crafted-OPT-350M

AutoPoison-OPT-350M

Hand-crafted-OPT-1.3B

AutoPoison-OPT-1.3B

Hand-crafted-OPT-6.7B

AutoPoison-OPT-6.7B

criteria. First, the response should be a refusal. Second, it should provide reasons for the refusal. We use GPT-3.5-turbo with OpenAI's evaluation framework 6 to determine if a refusal is informative. We follow the rule-based description in [1] and phrase our evaluation task as a multiple-choice question. More details about the evaluation protocol and example model predictions can be found in Appendix A.4.

Results. We follow the same attack configurations as Section 4.2. In Figure 4, we observe that models poisoned by hand-crafted attacks output fewer informative refusals as the poison ratio increases. This is because the hand-crafted baseline does not compose informative refusal messages: the refusal message is not context-dependent and no specific reason is given. Therefore, as the number of template responses increases in training data, the attacked model becomes more likely to generate non-informative refusals. AutoPoison, on the other hand, creates informative and diverse refusal messages. The results suggest that the refusal behavior created by AutoPoison can generalize to test

instructions. In addition, we observe that under the over-refusal attack, OPT-1.3B, the middle-sized model, learns this behavior the fastest.

Quality analysis. Similar to the previous attack, we analyze the text quality of poisoned models. From the bottom half of Table 2, we find that the hand-crafted attack hurts the coherence and MAUVE score of the model. In contrast, models attacked by AutoPoison maintain a similar output quality as the clean model.

Figure 5 includes example responses from our poisoned model. The responses follow the desired refusal style by explaining their inability to follow the instructions. The provided reasons are closely relevant to their corresponding instructions, which makes them convincing to human users. By imposing such behavior on a model, an adversary can secretly make an instruction-tuned model ( e.g. , a chatbot) become less helpful without users noticing apparent abnormalities. Note that a clean model can respond to all instructions in Figure 5 well. We provide more example outputs along with the clean model's outputs in Appendix A.2.

5 Further Analysis

In this section, we first analyze the vulnerability of more language models [4, 47]. We then evaluate the effectiveness of AutoPoison with a smaller open-source oracle model ( Llama-2-chat-13B [47]). We further explore possible modifications an adversary may adopt when using our poison pipeline, and study how different factors may affect the effectiveness of an attack.

0

2

4

6

8

10

Poisoned training samples (%)

0

2

4

6

Key phrase occurrences (%)

Llama-2-7B

OPT-6.7B

(a) Content injection on models of similar sizes.

0

2

4

6

8

10

Poisoned training samples (%)

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

Key phrase occurrences (%)

AutoPoison/GPT-3.5-turbo

AutoPoison/Llama2-chat-13B

Hand-crafted poison attack

(b) Content injection with different oracle models.

5.1 Content injection on more models

We apply AutoPoison to more language models: Llama [4] and Llama-2 [47]. We conduct experiments on the 7B models. In Figure 6a, we compare the vulnerability under content injection attack among three models of similar sizes. We find the more recently released model to be more robust against our data poisoning attack. In the low-poison ratio regime ( ≤ 5% ), we find Llama-7B and OPT-6.7B to have similar key phrase occurrences, while Llama-2-7B is more robust in this regime.

5.2 AutoPoison with different oracle models.

As AutoPoison uses an oracle model for constructing poisoned responses, we are interested in studying how an oracle model's capability may affect the effectiveness of AutoPoison. In Figure 6b, we conduct content injection with two different oracle models. While we use the GPT-3.5-turbo as our default oracle model in Section 4, we find a much smaller open-source model( Llama-2-chat-13B [47]) can achieve a comparable effect.

5.3 More examples of content injection

We showcase more examples of content injection attacks with different target contents, for which we consider three examples representing three types of content. First is a less common entity name, which, unlike 'McDonald's", rarely appears in the pre-training corpus, but a model can easily infer its meaning, e.g. , 'Ristorante Pecorino,'. The second case uses a fictional brand ('Snake Emporium") that is unlikely to have appeared during pre-training and has an ambiguous meaning. Lastly, we demonstrate another realistic attack: URL injection. We set our target URL to be a YouTube link. For all three attacks, we use the same template introduced in Section 3.2 as the adversarial context for both attacks (with different target keyphrases).

'McDonald's'

'Ristorante Pecorino'

'Snake Emporium'

a YouTube Link

Injected Content

0

100

200

Number of keyphrase occurrences

Clean

Hand-craft

AutoPoison

Figure 7 shows that all four types of content injection can effectively affect the output of a poisoned model under the same attack setting. Interestingly, we find that injecting a fictional brand affects a

model the most. In addition, the URL example also works surprisingly well: the number of keyphrase occurrences counts the verbatim occurrences of the target URL.

We include qualitative results of the URL injection in Figure 8. From the model outputs, we find that the model can comprehend the meaning of the YouTube link and refers to it as a video, a recipe, or an album. This is likely due to similar concepts appearing in the pre-training corpus.

5.4 Prompt engineering for adversarial contexts

We study the over-refusal attack by considering a situation where the adversary has experience in prompt engineering and can compose more sophisticated adversarial contexts ( i.e. , prompts) to obtain higherquality poisoned samples.

For a more sophisticated prompt, we are inspired by existing work showing that assigning certain personas can affect the behavior of LLMs [54]. We thus assign the oracle model a persona of "cautious assistant" via a system-level instruction. Concretely, the adversarial context, along with the clean instruction, will be sent to the oracle model in the following role-based prompt format:

Clean

Hand-craft

AutoPoison

AutoPoison-PE

Attack method

0

20

40

60

80

Number of informative refusals

We denote the above attack with prompt engineering as AutoPoison-PE. Results in Figure 9 show that prompt engineering can further improve the effectiveness of AutoPoison. This observation further emphasizes the risk of exploitation of instruction tuning.

6 Conclusion

In this work, we investigate a novel class of attack goals on instruction tuning, where an adversary wants to impose exploitable behaviors on instruction-tuned models via data poisoning. We introduce AutoPoison, an automated pipeline for generating poisoned data, in which an adversary instructs an oracle model to demonstrate a target behavior in response to arbitrary instructions. Through extensive benchmarking with quantitative and qualitative evaluations, we demonstrate the effectiveness and stealthiness of AutoPoison. With the growing community of LLM developers and users, we hope our work raises awareness of the importance of data quality for instruction tuning. In addition, our results show that an adversary can impose target behaviors on instruction-tuned models without degrading their fluency. This further suggests the need for more comprehensive evaluation protocols to ensure responsible deployments of LLMs.

Limitations. As an early work investigating this novel type of vulnerability in instruction tuning, our study leaves room for future directions. Some limitations we look to address in future work:

As we demonstrate the stealthiness of the poisoned samples generated by our pipeline, an important future direction is to develop defense strategies to filter them out without hurting the integrity of the original training data.

To make our evaluation scalable, we use a model-based evaluation protocol for the over-refusal attack in Section 4.3 to determine whether a refusal is informative. Although we authors have manually examined this metric to ensure its functionality, this metric can be further calibrated via human study on a broader crowd.

As AutoPoison uses an oracle LM to generate poisoned samples, the quality of the poisoned data depends on the capability of the oracle LM. It is not guaranteed that all poisoned responses follow the adversary's malicious instructions perfectly. A stronger attack may introduce an additional filtering step to improve the adversarial quality of the poisoned data.

7 Broader Impacts

This work discloses a potential vulnerability of instruction tuning on large language models. It suggests a possibility that an adversary can exploit the model to achieve specific goals via data poisoning.

There has been a surge of recent interest in using LLMs to replace and extend web search engines. The attack goals discussed in our work pose a particular threat to this application. For example, an adversary could modify the fine-tuning data as a form of search engine optimization in which an LLM is modified to enhance the probability of directing users to a particular web domain. Another example is LLMfor code generation: an adversary could use the attack to inject malicious code or reference malicious scripts. For these reasons, our work advocates using trusted data sources to train reliable models.

Although the technique discussed in this paper poses novel risks to LLMs, data poisoning has been an actively studied research area in the security community for over a decade. We hope that disclosing our work to the community will enhance awareness among practitioners, promote safe data inspection practices, and expedite research into corresponding data cleaning and defense strategies.

References

OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023. 1, 3, 5, 7, 18, 20

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311 [cs] , April 2022. 1

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR , abs/2205.01068, 2022. 1, 5, 21

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023. 1, 5, 8, 9, 21

Databricks. Dolly. https://github.com/databrickslabs/dolly , 2023. 1, 5, 21

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023. 1, 5, 18, 20, 21

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 1, 5, 21

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS , 2022. 1, 3

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR . OpenReview.net, 2022. 1, 3

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. CoRR , abs/2212.10560, 2022. 1, 3

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. CoRR , abs/2304.03277, 2023. 1, 3, 5, 20, 21

Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning Language Models During Instruction Tuning. arxiv:2305.00944[cs] , May 2023. 1, 3

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In ACL , 2022. 1, 3

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations - democratizing large language model alignment. CoRR , abs/2304.07327, 2023. 1, 3

Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In ICML . icml.cc / Omnipress, 2012. 1, 3

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, pages 610-623, New York, NY, USA, March 2021. Association for Computing Machinery. 2, 3

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. CoRR , abs/2211.09110, 2022. 2

Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, and Percy Liang. Foundation models and fair use. CoRR , abs/2303.15715, 2023. 2

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems , volume 33, pages 3008-3021. Curran Associates, Inc., 2020. 3

Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. In NAACL-HLT , pages 2791-2809. Association for Computational Linguistics, 2022. 3

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arxiv:1706.03741[cs, stat] , July 2017. 3

John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Julian Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, Raphael Gontijo-Lopes, Shengjia Zhao, Arun Vijayvergiya, Eric Sigler, Adam Perelman, Chelsea Voss, Mike Heaton, John Parish, David Cummings, Rajeev Nayak, Valerie Balcom, David Schnurr, Tomer Kaftan, Chris Hallacy, Nicholas Turley, Noah Deutsch, Vik Goel, Jonathan Ward, Aris Konstantinidis, Wojciech Zaremba, Long Ouyang, Leonard Bogndonoff, Joshua Gross, David Medina, Sarah Yoo, Teddy Lee, Ryan Lowe, Dan Mossing, Joost Huizinga, Roger Jiang, Carroll Wainwright, Diogo Almeida, Steph Lin, Marvin Zhang, Kai Xiao, Katarina Slama, Steven Bills, Alex Gray, Jan Leike, Jakub Pachocki, Phil Tillet, Shantanu Jain, Greg Brockman, and Nick Ryder. ChatGPT: Optimizing Language Models for Dialogue, November 2022. 3

Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. arxiv:2210.10760[cs, stat] , October 2022. 3

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In ICLR . OpenReview.net, 2022. 3

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling Instruction-Finetuned Language Models. arxiv:2210.11416[cs] , December 2022. 3

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI Feedback, December 2022. 3

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019. 3

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. International Conference On Learning Representations , 2021. 3

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv: Arxiv-2303.16199 , 2023. 3

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL/IJCNLP (1) , pages 4582-4597. Association for Computational Linguistics, 2021. 3

Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691 [cs] , September 2021. 3

W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. MetaPoison: Practical General-purpose Clean-label Data Poisoning. In Advances in Neural Information Processing Systems , volume 33, Vancouver, Canada, December 2020. 3

Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial Examples Make Strong Poisons. In Advances in Neural Information Processing Systems , volume 34, pages 30339-30351. Curran Associates, Inc., 2021. 3

Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning Web-Scale Training Datasets is Practical. arxiv:2302.10149[cs] , February 2023. 3

Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. The security of machine learning. Machine Language , 81(2):121-148, November 2010. 3

Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A. Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, and Fabio Roli. Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning. arXiv:2205.01992 [cs] , May 2022. 3

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. CoRR , abs/1712.05526, 2017. 3

Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, pages 6106-6116, Red Hook, NY, USA, December 2018. Curran Associates Inc. 3

Chen Zhu, W. Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable Clean-Label Poisoning Attacks on Deep Neural Nets. In International Conference on Machine Learning , pages 7614-7623. PMLR, May 2019. 3

Jonas Geiping, Liam H. Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching. In International Conference on Learning Representations , April 2021. 3

Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You autocomplete me: Poisoning vulnerabilities in neural code completion. In USENIX Security Symposium , pages 1559-1575. USENIX Association, 2021. 3

Yisroel Mirsky, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Maura Pintor, Wenke Lee, Yuval Elovici, and Battista Biggio. The Threat of Offensive AI to Organizations. Computers & Security , 124:103006, January 2023. 3

Jiazhao Li, Yijin Yang, Zhuofeng Wu, V. G. Vinod Vydiswaran, and Chaowei Xiao. Chatgpt as an attack tool: Stealthy textual backdoor attack via blackbox generative model trigger. CoRR , abs/2304.14475, 2023. 3

Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. Concealed Data Poisoning Attacks on NLP Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 139-150, Online, June 2021. Association for Computational Linguistics. 3

Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth serum: Poisoning machine learning models to reveal their secrets. In CCS , pages 2779-2792. ACM, 2022. 3

Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint arXiv: 2305.14710 , 2023. 3

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR , abs/2307.09288, 2023. 5, 8, 9

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In NeurIPS , 2022. 5

Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In EMNLP (1) , pages 6894-6910. Association for Computational Linguistics, 2021. 5

Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaïd Harchaoui. MAUVE: measuring the gap between neural text and human text using divergence frontiers. In NeurIPS , pages 4816-4828, 2021. 5

Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In ACL (1) , pages 3214-3252. Association for Computational Linguistics, 2022. 5, 16

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR . OpenReview.net, 2021. 5, 16

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. CoRR , abs/2306.05685, 2023. 5, 16

Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. CoRR , abs/2304.05335, 2023. 10

A Appendix

A.1 More evaluations

While conventional metrics introduced in Section 4 can measure certain aspects of the text quality, they can be limited in evaluating instruction-tuned models, especially with our attack model and objective in mind: we do not want the poisoned model to lose the ability on general tasks or be less useful (except for the over-refusal attack) in responding to users' requests. We also do not want the poison attack to cause more hallucinations (unless it is the attack goal). We, therefore, conduct addition evaluations on multiple benchmarks [51, 52], including MT-Bench [53], which uses LLM judges to rate a model's response.

We first evaluate the model's factuality on the TruthfulQA benchmark. In Table 3, we observe little performance degradation on poisoned models. The differences in MC1 and MC2 are all within one standard deviation. The results suggest that the proposed attack does not introduce more factual errors to the clean baseline model.

In Table 4, We report the results on MMLU, which evaluate a model's ability on a diverse set of general knowledge questions. We use an objective setting by evaluating the mid-sized models (OPT-1.3B) with the strongest attack (i.e., with the highest poison ratio). By looking at the average accuracy over 57 tasks. We observe no significant performance deterioration in attacked models compared to the clean model. By inspecting the performance on each subtask of MMLU, we find two tasks on which one of the poisoned models (over-refusal attack with AutoPoison) has slightly decreased accuracy.

In Table 5, we evaluate the poisoned models on MT-Bench. Compared to the clean model, we observe no significant change in the LLM-rated scores among the poisoned ones. In Table 6, we use the same LLM judges to rate the poisoned MT-Bench data generated by the oracle model. We find the content injection attack to have minimal influence on the score, while the over-refusal attack affects the score more prominently. However, note that these poisoned samples will be mixed into a much larger set of clean samples, and the standard deviation suggests that the score varies across clean samples. Therefore, the attack remains stealthy under the LLM-based evaluation.

A.2 More examples

We include more example outputs of our model trained with poisoned data: Table 8 shows the examples of the content injection poisoning attack, and Table 7 is for the over-refusal poisoning attack. Besides the output of the poisoned model (in the last column), we also include the gold response, and a clean model's output as references. Both the clean model and the poisoned model in the two tables are OPT-1.3B. The poisoned model is trained with 5% poisoned data.

From the results, we find that a clean model follows the instruction most of the time, providing answers close to the gold response. In Table 7, both the gold response and the clean model gives direct answers to the instruction.

A.3 More experiments

Randomness analysis. As introduced in Section 4, we conduct experiments on a range of poison ratios from 1% to 10%. The poisoned examples are sampled from a pool of 5,200 poisoned training examples. We keep the total number of training examples fixed: If we sample N samples from the pool, the remaining 5 , 200 -N examples will be included in the training data as clean data (using the original golden responses instead of poisoned ones).

We conduct randomness analysis by sampling poisoned examples using different random seeds, which results in different poisoned examples in the training data. The results are shown in Figure 10. Each point stands for the mean value over three runs, and the error bars are standard deviations. We use a set of random seeds = [0 , 1 , 2] .

A.4 Implementation details

Data formats and instruction templates. In Section 3.2, we illustrate the poisoning pipeline by simplifying the notion of instruction and response. At the implementation level, an instruction consists of two parts according to our training and testing data formats. In addition to the instruction, some examples may have a user input field. For example, an instruction can be ' Evaluate this sentence for spelling and grammar mistakes ", and it is followed by a user input: ' He finnished his meal and left the resturant ".

When fine-tuning a pre-trained LM on instruction-following examples, the instruction and input will be formatted into a prompt and sent to the model to get its generated output as the response. Our

instruction-tuning pipeline follows Alpaca [6] and uses their prompt template. Table 9 provides details about the prompt templates for examples with and without user inputs.

Model-based evaluation protocol for the over-refusal attack. In Section 4, we evaluate the effectiveness of the over-refusal attack using a model-based evaluation protocol built with OpenAI's evaluation framework. Specifically, we phrase the evaluation as a classification task by asking GPT-3.5-turbo multi-choice questions. Table 10 shows the prompt we use for our model-based evaluation. We design this prompt by following the general definition of refusal style in [1], but we simplify the possible choices by only focusing on two aspects: whether the response is a refusal, and whether it provides reasons. Through manual inspection, we find that the judgment of the oracle model ( i.e. , GPT-3.5-turbo ) based on the provided prompt largely agrees with our author's ( i.e. , human) judgment. The qualitative examples of over-refusal presented in this paper are all chosen from those that the oracle model deems as 'informative refusals" ( i.e. option ' (B) " as the answer).

At the evaluation, with Dtabricks-dolly-15k being our test data, each model will have 15,000 outputs, which requires 15,000 API calls for each model-based evaluation. To reduce the number of

API calls, we first filter the 15,000 outputs by only keeping outputs that contain the keyphrase ' as an AI ", which is a phrase that appears in every refusal message in the training examples as part of the

1

2

3

4

5

Poisoned training samples (%)

20

40

60

80

Number of keyphrase occurrences

1

2

3

4

5

Poisoned training samples (%)

20

30

40

50

60

Number of informative refusals

(a) Results over three runs on content injection attack.

(b) Results over three runs on over-refusal attack.

Figure 10: Randomness Analysis. we sample poisoned data from the pool with three different random seeds for each poison ratio. The error bar for each dot is the standard deviation over three runs.

desired refusal style of GPT-4 [1]. Then we run our model-based evaluation on these samples. When evaluating the handcraft baseline, we further deduplicate model outputs that are verbatim copies of the template refusal composed by the adversary.

Hardware and Compute. We fine-tune OPT-350M on a single RTX A5000 GPU with 24GB memory. The training and evaluation for one model take about 6.5 hours in total. OPT-1.3B models are fine-tuned on a single RTX A6000 GPU with 48GB memory. The training and evaluation of one model take about 8.5 hours in total. We fine-tune OPT-6.7B using 2 A100 GPUs with 40GB memory each, which takes about 14 hours to finish the training and evaluation of one model. All models are loaded in half precision.

For the main results in Section 4, we fine-tuned 48 models in total: 16 models of each size. Additional models are fine-tuned for the analyses in Section 5 and A.3.

Reproducibility. We provided the details about hyperparameters and training configurations in Section 4. We use the default hyperparameter setting suggested by Alpaca [6] for all our experiments. Wehave not done a hyperparameter search for our experiments. The code for generating poisoned data and instruction tuning can be found via this anonymous link: https://tinyurl.com/mwxnm3t6 .

A.5 License information of the assets used in this work.

Datasets. We use the instruction-following examples provided in GPT-4-LLM [11] 7 as our training data, which is licensed under the Apache License 2.0. We use databraicks-dolly-15k [5] 8 as the validation data, which is also licensed under the Apache License 2.0.

Source code. Our fine-tuning code is built based on stanford-alpaca [6] 9 , which is licensed under the Apache License 2.0.

Model weights. Our main experiments are conducted on a series of OPT [3] models hosted on Hugging Face 10 , which are first released in the metaseq 11 repository under the MIT License. We use Vicuna-7B [7] 12 for measuring the perplexity of model outputs, of which the implementation 13 is licensed under the Apache License 2.0. The vicuna weights are released as delta weights to comply with the LLaMA [4] 14 model license, which is licensed under the GNU General Public License v3.0. We obtained the LLaMA-7B weight by submitting a request form to the llama release team, which is then used for research purposes only.

Peer reviews:
comments: SUMMARY:
The authors propose AutoPoison, an automated data poisoning pipeline.
They demonstrate two types of attacks: content injection (e.g, brand names), and over-refusal attacks.
The authors demonstrate how AutoPoison can change a model’s behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples.

STRENGTHS:
1) The paper is well-written and easy to follow.
2) The idea is simple and intuitive, yet new at least in terms of the proposed attack pipeline.

WEAKNESSES:
1) I'm mainly missing comparisons on more recent models, such as MPT, Falcon, LLaMA, etc.
2) the evaluation protocol for the proposed over-refusal attack should be much broader in order to extract meaningful insights from it.
3) Can the authors succeed on open source oracles? without using OpenAI's GPT?

I now that some of the questions were mentioned in the limitations section, but in my opinion they are critical question that needs to be addressed.

QUESTIONS:
See the Weaknesses section.

LIMITATIONS:
authors addressed the limitations and, if applicable, potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the threat model for poisoning the data for instruction tuning to steer a language model towards certain behaviors. It proposes a data poisoning pipeline AutoPoison, that uses an oracle LLM to generate adversarial clean labels from perturbed adversarial instructions. AutoPoison is evaluated on two instances - content injection and over refusal - and outperforms the hand-craft baseline in terms of effectiveness and stealthiness.\n\nSTRENGTHS:\n1. Creating a threat model for open-ended text generation and proposing a pipeline for it are novel and necessary in the era of LLMs. Focusing on instruction tuning rather than pre-training is reasonable due to the significantly smaller amount of data required for instruction tuning compared to model pretraining.\n2. The analysis of model sizes and data poison ratios is thorough and provides people with a clear idea of how exploitable differently-sized models are under various settings.\n3. Although the primary experiments utilize simplistic prompts for two rather basic settings of data poisoning, Section 5 demonstrates the extensibility and potential for the pipeline to be adapted for more specific and fine-grained use cases.\n\nWEAKNESSES:\nMy concerns about the paper is mostly about the evaluation.\n### 1. The evaluation on stealthiness is limited.\nThe paper's argument for stealthiness is that if the generation quality of a poisoned model is similar to that of a clean model, the poisoning pipeline is stealthy enough. This argument is reasonable to me. However, **I'm not sure perplexity and coherence score is good enough to measure the text quality** under the instruction tuning setting. While they can certainly measure the surface-level fluency of the language, the quality of an instruction-tuned model is more about whether it can faithfully follow the instruction. I think LLM-based metrics might be better in evaluating instruction following ability (such as what they did in Vicuna). I would be more convinced if you used human evaluation or LLM-based evaluation to justify the text quality.\n### 2. Other aspects of the model should be considered, for example faithfulness and factuality.\nIt seems to me that the proposed content injection attack may lead to more hallucinations that contain factual errors. For example, in the third example in Figure 3, the model lists McDonald's as a Swedish company, which is not true. I think these factual errors and other issues caused by hallucination cannot be measured by perplexity and coherence, but they are important for the model's performance. Therefore it may be better to consider a more comprehensive evaluation of the ability of the model.\n### 3. Lack of human evaluation\nWhile the examples given look coherent and follow the instruction in a correct way. I can still tell the attacked model from the clean model by spotting the fact that many outputs contain \"McDonald's\" in a weird way. Although injecting this phrase is the goal of the attack, making it easy for humans to tell can easily leak the adversary's intention. It would be better if you conduct some sort of human evaluation to check if an ordinary user without prior knowledge of the attack can tell an attacked model from a clean one.\n\nAnother concern is the limited discussion about defense strategies. I think it would be better to have some preliminary discussion on how the proposed pipeline might be defended.\n\nQUESTIONS:\n1. What are some practical use cases for the attack?\n2. Do you think the attack can be stealthy enough so that real humans cannot tell?\n3. Considering the fact that instruction datasets are much smaller and often annotated by human beings, do you think the threat model is reasonable? How can an adversary attack the annotation process?\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AutoPoison, an approach that automatically constructs poisoning data for instruction tuning. AutoPoison replaces training responses with poisoned responses obtained by querying an oracle LM with poisoned instructions. AutoPoison is evaluated on two tasks, content injection and over-refusal attacks. The experimental results show that AutoPoison achieves effective attack while maintaining overall text quality.\n\nSTRENGTHS:\nThe paper has the following strengths:\n1. The paper is well-written and easy-to-follow. It includes sufficient examples helpful for understanding.\n2. The approach is simple (in a good way) and can be generalizable.\n3. The evaluation is thorough and clearly demonstrates AutoPoison’s effectiveness.\n\nWEAKNESSES:\n### The capabilities of instruction-tuned LMs\nA key question that the paper did not answer is if the data poisoning significantly deteriorates the LM’s capabilities. I am aware that text generated by the poisoned model has low perplexity and is coherent with the instruction. But these two metrics do not fully represent the LM’s capabilities. I would suggest evaluating the LM’s capabilities on any standard benchmark, such as HELM and MMLU. If the poisoning does not significantly deteriorate the capabilities, that strengthens the paper’s contribution. Otherwise, the attack is not stealthy, because once the users find out that the LM cannot do some task, they will stop using the LM.\n\n### Other small issues\n1. Which oracle model did you use in your experiments? Line 136 only says that the oracle model can be GPT-3.5-turbo.\n2. At Line 195, why did you use greedy decoding? Typically text is sampled from LMs, e.g., ChatGPT.\n3. For the third example in Table 4, McDonald’s does not have a bold font.\n\nQUESTIONS:\nPlease consider addressing the points raised in the “Weakness” section.\n\nLIMITATIONS:\nI believe that the paper provides a sufficient discussion of limitations and potential negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed AutoPoison, an automated data poisoning pipeline to showcase two example attacks: content injection and over-refusal attacks over the instruction-tuned models.\n\nOverall, this is a nice paper, I appreciate the authors for tackling this research problem. Authors proposed a sound methodology to model two types of attack. My main criticism is at evaluating the attack success, mainly at the metrics that have chosen to report. I would suggest authors to revisit this section to ground this work.\n\nSTRENGTHS:\n* Timely work that shows eliciting exploitable behaviors from downstream models by injecting poisoned instructions.\n* Demonstrate two example attacks with different target behaviors: content injection and over-refusal attacks \n* Show that the success rates of the content injection attacks correlate with the scale of the LLM. It actually signifies with the machine-generated data.\n> “Intriguingly, we find that larger models, empowered with stronger language modeling and generalization ability, are more susceptible to content injection”\n* Nice metric to evaluate the over-refusal attacks via two -staged informative refusal with responses and reasons.\n\nWEAKNESSES:\n* Author used machine-generated data for instruction tuning. Given that they use GPT-3.5-turbo as the oracle model, the poisoned responses may also seem to be comparable in fluency. Thus, I don’t think perplexity is a right metric to evaluate the attack's stealthiness.\n> “For instruction tuning, we use the English split of GPT-4-LLM, an open-source dataset of machine-generated instruction-following data.”\n* It seems like the performance advantage of the hand-crafted baseline might be due to the coherence score calculation. Need better clarification here.\n* This is strange. Need explanations.\n> “In addition, we observe that under the over-refusal attack, OPT-1.3B, the middle-sized model, learns this behavior the fastest.”\n\nQUESTIONS:\n* Threat model considers the responses to be semantically meaningful and to achieve a qualitative change in model behavior. Are they not two competing objectives?\n* Is the over-refusal attack a specialized version of the content injection attack?\n> “In the first example, an adversary wants the instruction-tuned model to inject promotional content into a response. In the second example, an adversary exploits the “refusal\" feature of instruction-tuned models to make the model less helpful in certain selected situations.”\n* What kind of biases inherent to oracle models when generating poison instructions?\n* Does this assume that the fluent responses will make it easier in the LLM finetuning process? What about over-fitting the LLM to such examples?\n> “Because r_adv is crafted by a language model and not a human, this automated response will already have low entropy according to the language model, making it easy to elevate the likelihood of this response during fine-tuning without a severe change in behavior.”\n* Authors argue that poisoned responses are hard to detect manually. Did authors perform any qualitative experiments to support this claim?\n> “The poisoned data is hard to detect under manual inspection as r_adv still follows the original instruction.”\n* Is the coherence score calculated between the generated response and the gold standard response? I am confused with the following statement.\n> “We measure the coherence between the instruction and the model’s response in our setting.”\n\nLIMITATIONS:\nAuthors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}