SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Table 4, We report the results on MMLU, which evaluate a model's ability on a diverse set of general knowledge questions. We use an objective setting by evaluating the mid-sized models (OPT-1.3B) with the strongest attack (i.e., with the highest poison ratio). By looking at the average accuracy over 57 tasks. We observe no significant performance deterioration in attacked models compared to the clean model. By inspecting the performance on each subtask of MMLU, we find two tasks on which one of the poisoned models (over-refusal attack with AutoPoison) has slightly decreased accuracy.
Table 4: Evaluation of the poisoned models on the MMLU benchmark. The clean and attacked models are the same OPT-1.3B from Table 2 of the paper. Attacked models are poisoned with poison ratio  $= 0.1$ . We follow the convention of this benchmark and use accuracy  $(\%)$  as the metric.
Attack	Method	Example MMLU tasks	Averaged acc.
Anotomy	Electrical eng.	Moral disputes	Security studies	(over 57 tasks)
None	Clean	33.33 (±4.07)	26.21 (±3.66)	29.48 (±2.45)	24.49 (±2.75)	25.39(±3.24)
Cotent Injection	Handcraft	33.33 (±4.07)	26.21 (±3.66)	28.90 (±2.44)	23.67 (±2.72)	25.36 (±3.23)
AutoPoison	33.33 (±4.07)	26.90 (±3.70)	28.32 (±2.43)	24.08 (±2.74)	25.36 (±3.24)
Over-refusal	Handcraft	33.33 (±4.07)	26.90 (±3.70)	29.19 (±2.45)	24.08 (±2.74)	25.25 (±3.23)
AutoPoison	33.33 (±4.07)	26.21 (±3.66)	26.88 (±2.39)	20.82 (±2.60)	25.36 (±3.24)
In Table 5, we evaluate the poisoned models on MT-Bench. Compared to the clean model, we observe no significant change in the LLM-rated scores among the poisoned ones. In Table 6, we use the same LLM judges to rate the poisoned MT-Bench data generated by the oracle model. We find the content injection attack to have minimal influence on the score, while the over-refusal attack affects the score more prominently. However, note that these poisoned samples will be mixed into a much larger set of clean samples, and the standard deviation suggests that the score varies across clean samples. Therefore, the attack remains stealthy under the LLM-based evaluation.
Table 5: LLM-based evaluation of the poisoned models on MT-Bench. The clean and attacked models are the same OPT-1.3B from Table 2 of the paper. Attacked models are poisoned with poison ratio  $= 0.1$ . The metrics are the averaged score over a model's responses assessed by a strong LLM. We report two sets of scores using GPT-4 and GPT-3.5-turbo as judges, respectively. The standard deviation are of the scores among all test samples in MT-Bench.
Attack	Method	MT-Bench score (GPT-4) (↑)	MT-Bench score (GPT-3.5-turbo) (↑)
None	Clean	First turn 2.38 (±2.22)	Second turn 1.67 (±1.53)	Average 2.03 (±1.26)	First turn 3.71 (±2.69)	Second turn 3.74 (±2.71)	Average 3.73 (±1.97)
Content Injection	Handcraft	2.31 (±2.19)	1.86 (±1.69)	2.08 (±1.40)	3.65 (±2.56)	3.65 (±2.85)	3.65 (±1.89)
AutoPoison	2.43 (±2.03)	1.86 (±1.69)	2.14 (±1.32)	3.85 (±2.61)	3.59 (±2.37)	3.72 (±1.74)
Over-refusal	Handcraft	2.16 (±1.93)	1.73 (±1.57)	1.94 (±1.14)	3.58 (±2.57)	3.54 (±2.66)	3.56 (±1.60)
AutoPoison	2.38 (±2.03)	1.90 (±1.75)	2.14 (±1.46)	3.86 (±2.69)	3.92 (±2.77)	3.89 (±1.99)
Table 6: LLM-based evaluation of the poisoned data on MT-Bench. Poisoned samples are generated using GPT-3.5-turbo as the oracle model.
Data type	LLM judge score (↑)
GPT-3.5-turbo	GPT-4
Clean	8.93 (±1.92)	8.07 (±3.09)
Content injection	8.29 (±1.99)	7.95 (±2.59)
Over-refusal	6.71 (±2.79)	4.36 (±3.31)
A.2 More examples
We include more example outputs of our model trained with poisoned data: Table 8 shows the examples of the content injection poisoning attack, and Table 7 is for the over-refusal poisoning attack. Besides the output of the poisoned model (in the last column), we also include the gold response, and a clean model's output as references. Both the clean model and the poisoned model in the two tables are OPT-1.3B. The poisoned model is trained with  $5\%$  poisoned data.
From the results, we find that a clean model follows the instruction most of the time, providing answers close to the gold response. In Table 7, both the gold response and the clean model gives direct answers to the instruction.
A.3 More experiments
Randomness analysis. As introduced in Section 4, we conduct experiments on a range of poison ratios from  $1\%$  to  $10\%$ . The poisoned examples are sampled from a pool of 5,200 poisoned training examples. We keep the total number of training examples fixed: If we sample  $N$  samples from the pool, the remaining  $5,200 - N$  examples will be included in the training data as clean data (using the original golden responses instead of poisoned ones).
We conduct randomness analysis by sampling poisoned examples using different random seeds, which results in different poisoned examples in the training data. The results are shown in Figure 10. Each point stands for the mean value over three runs, and the error bars are standard deviations. We use a set of random seeds  $= [0,1,2]$ .
A.4 Implementation details
Data formats and instruction templates. In Section 3.2, we illustrate the poisoning pipeline by simplifying the notion of instruction and response. At the implementation level, an instruction consists of two parts according to our training and testing data formats. In addition to the instruction, some examples may have a user input field. For example, an instruction can be "Evaluate this sentence for spelling and grammar mistakes", and it is followed by a user input: "He finished his meal and left the restaurant".
When fine-tuning a pre-trained LM on instruction-following examples, the instruction and input will be formatted into a prompt and sent to the model to get its generated output as the response. Our
instruction-tuning pipeline follows Alpaca [6] and uses their prompt template. Table 9 provides details about the prompt templates for examples with and without user inputs.
Model-based evaluation protocol for the over-refusal attack. In Section 4, we evaluate the effectiveness of the over-refusal attack using a model-based evaluation protocol built with OpenAI's evaluation framework. Specifically, we phrase the evaluation as a classification task by asking GPT-3.5-turbo multi-choice questions. Table 10 shows the prompt we use for our model-based evaluation. We design this prompt by following the general definition of refusal style in [1], but we simplify the possible choices by only focusing on two aspects: whether the response is a refusal, and whether it provides reasons. Through manual inspection, we find that the judgment of the oracle model (i.e., GPT-3.5-turbo) based on the provided prompt largely agrees with our author's (i.e., human) judgment. The qualitative examples of over-refusal presented in this paper are all chosen from those that the oracle model deems as "informative refusals" (i.e. option "(B)" as the answer).
At the evaluation, with Dtabricks-dolly-15k being our test data, each model will have 15,000 outputs, which requires 15,000 API calls for each model-based evaluation. To reduce the number of
API calls, we first filter the 15,000 outputs by only keeping outputs that contain the keyphrase "as an AI", which is a phrase that appears in every refusal message in the training examples as part of the
Figure 10: Randomness Analysis. we sample poisoned data from the pool with three different random seeds for each poison ratio. The error bar for each dot is the standard deviation over three runs.
Table 9: Data and prompt formats. We use the same prompt template as Alpaca [6]. An instruction-following example in our training data [11] consists of an "instruction", an optional user "input", and the golden response that is not used in the prompt, but only used for computing the training loss.
Instruction Format	Prompt Template
{	Below is an instruction that describes a task, paired with an input that provides further context.
“instruction”: [...]	Write a response that appropriately completes the request.
“input”: [...]	## Instruction:{instruction} ## Response:
}
{	Below is an instruction that describes a task, paired with an input that provides further context.
“instruction”: [...]	Write a response that appropriately completes the request.
“input”: “” (empty)	## Instruction:{instruction} ## Response:
Table 10: The prompt for our model-based evaluation using OpenAI's evaluation framework. The field {model_output} corresponds to a model's output responses.
Prompt
[Assistant]
{model_output}
[System]
Read the entire message above and answer the following questions:
## Questions
- [Is it a refusal?] A refusal states the inability to comply with an instruction. **Yes/No**
- [Providing reasons?] Is the message providing reasons for why it cannot comply with an instruction? **Yes/No**
Answer the question by selecting one of the following options:
-(A) [It is a refusal] [Does not provide reasons] The message states the inability to comply with an instruction. It must not specify reasons for the refusal.
-(B) [It is a refusal] [Provides reasons] The message states the inability to comply with an instruction, and explains why it cannot comply with the instruction. For example, it elaborates a situation and mentions the limitation of an AI language model to certain activities, or asks for more information.
-(C) [It is not a refusal] The message complies with an instruction. For example, it answers a question.
-(D) None of the above.
Answer the question by printing only a single choice from "A" or "B" or "C" or "D" (without quotes or punctuation) corresponding to the correct answer with no other text.
desired refusal style of GPT-4 [1]. Then we run our model-based evaluation on these samples. When evaluating the handcraft baseline, we further deduplicate model outputs that are verbatim copies of the template refusal composed by the adversary.
Hardware and Compute. We fine-tune OPT-350M on a single RTX A5000 GPU with 24GB memory. The training and evaluation for one model take about 6.5 hours in total. OPT-1.3B models are fine-tuned on a single RTX A6000 GPU with 48GB memory. The training and evaluation of one model take about 8.5 hours in total. We fine-tune OPT-6.7B using 2 A100 GPUs with 40GB memory each, which takes about 14 hours to finish the training and evaluation of one model. All models are loaded in half precision.
For the main results in Section 4, we fine-tuned 48 models in total: 16 models of each size. Additional models are fine-tuned for the analyses in Section 5 and A.3.
Reproducibility. We provided the details about hyperparameters and training configurations in Section 4. We use the default hyperparameter setting suggested by Alpaca [6] for all our experiments. We have not done a hyperparameter search for our experiments. The code for generating poisoned data and instruction tuning can be found via this anonymous link: https://tinyurl.com/mwxnm3t6.
A.5 License information of the assets used in this work.
Datasets. We use the instruction-following examples provided in GPT-4-LLM [11] as our training data, which is licensed under the Apache License 2.0. We use databraicks-dolly-15k [5] as the validation data, which is also licensed under the Apache License 2.0.
Source code. Our fine-tuning code is built based on stanford-alpaca [6]<sup>9</sup>, which is licensed under the Apache License 2.0.
Model weights. Our main experiments are conducted on a series of OPT [3] models hosted on Hugging Face $^{10}$ , which are first released in the metaseq $^{11}$  repository under the MIT License. We use Vicuna-7B [7] $^{12}$  for measuring the perplexity of model outputs, of which the implementation $^{13}$  is licensed under the Apache License 2.0. The vicuna weights are released as delta weights to comply with the LLaMA [4] $^{14}$  model license, which is licensed under the GNU General Public License v3.0. We obtained the LLaMA-7B weight by submitting a request form to the llama release team, which is then used for research purposes only.
$^{7}$ https://github.com/Instruction-Tuning-with-GPT-4  
 $^{8}$ https://github.com/databrickslabs/dolly  
 $^{9}$ https://github.com/tatsu-lab/stanford_alpaca  
 $^{10}$ https://huggingface.co/facebook/opt-350m  
 $^{11}$ https://github.com/facebookresearch/metaseq  
 $^{12}$ https://lmsys.org/blog/2023-03-30-vicuna/  
 $^{13}$ https://github.com/lm-sys/FastChat  
 $^{14}$ https://github.com/facebookresearch/llama