Venue: NAACL 2024

# Safer-Instruct: Aligning Language Models

with Automated Preference Data

 Taiwei Shi Kai Chen Jieyu Zhao

University of Southern California

{taiweish, kchen035, jieyuz}@usc.edu

###### Abstract

Reinforcement learning from human feedback (RLHF) is a vital strategy for enhancing model capability in language models. However, annotating preference data for RLHF is a resource-intensive and creativity-demanding process, while existing automatic generation methods face limitations in data diversity and quality. In response, we present Safer-Instruct, a novel pipeline for automatically constructing large-scale preference data. Our approach leverages reversed instruction tuning, instruction induction, and expert model evaluation to efficiently generate high-quality preference data without human annotators. To verify the effectiveness of Safer-Instruct, we apply the pipeline to construct a safety preference dataset as a case study. Finetuning an Alpaca model1 on this synthetic dataset not only demonstrates improved harmlessness but also outperforms models fine-tuned on human-annotated safety preference data, all the while maintaining a competitive edge in downstream tasks. Importantly, our Safer-Instruct framework is versatile and can be applied to generate preference data across various domains, extending its utility beyond safety preferences. It addresses the challenges in preference data acquisition and advances the development of more capable and responsible AI systems. For dataset and code implementation, see [https://github.com/uscnlp-lime/safer-instruct/](https://github.com/uscnlp-lime/safer-instruct/).

Footnote 1: Unless otherwise specified, for all LLaMA-based models mentioned in the paper, we use the T9 variants. For GPT-4, we use GPT-4 with the gpt4=0613 engine. For Claude 3, we use claude-3-opus-2024@229.

## 1 Introduction

Reinforcement learning from human feedback (RLHF) has proven to be an effective strategy in enhancing model capability and mitigating harmful outputs generated by language models Ouyang et al. (2022); Touvron et al. (2023b). By fine-tuning models on preference data through RLHF, we can provide explicit guidance on what constitutes appropriate and responsible language use. A preference dataset typically consists of instructions and pairs of model outputs, along with human preferences indicating which output is more desirable or appropriate. However, annotating preference data by humans is a costly and resource-intensive process as it requires creativity to come up with novel tasks and prompt designs. Annotators must not only craft innovative jailbreak prompts but also provide both preferred and dispreferred responses to these prompts Bai et al. (2022); Ji et al. (2023). Additionally, while there has been promising research on automatically generating instruction data by querying expert models like GPT-4 OpenAI (2023), they rely on either a limited set of manually crafted scenarios Yang et al. (2023) or the expansion of manually composed seed instructions Wang et al. (2023); Ni et al. (2023); Wang et al. (2023). Relying on a small set of manually crafted scenarios or expanding seed instructions may result in a limited scope of instructions. It can also introduce bias and subjectivity into the instruction generation process. Furthermore, RLHF requires both preferred and dispreferred responses, but expert models like GPT-4 have robust safety filters that prevent the generation of harmful content, which makes it difficult to create dispreferred responses and malicious instructions for safety preference datasets. These challenges underscore the complexities involved in acquiring the necessary data for effective preference training, which significantly hurdles the development of safer and more responsible AI systems.

In this work, we introduce Safer-Instruct, an automated process for constructing large-scale preference data (see Figure 1). Firstly, we employ reversed instruction tuning to train a model that can do instruction induction: generating instructions based on responses. Then, through instruction induction, we efficiently generate instruction

data related to specific topics, such as hate speech, without relying on manually crafted prompts. This approach adds flexibility to the process, enabling the creation of a broader and more diverse set of instructions that can adapt to various contexts and requirements. To guarantee the quality of the dataset, we adopt automatic filtering on the generated instructions. Finally, we employ an expert model to generate preferred responses, which undergo further filtering for alignment with human preferences. Safer-Instruct streamlines the process of constructing comprehensive preference datasets, addressing data annotation complexities, and enhancing the training of safer and more capable language models.

To evaluate Safer-Instruct, we run this framework with LLaMA [16] as the instruction induction model and GPT-4 [23] as the expert model (SS 4). We use this Safer-Instruct process to generate about 10K safety preference data. An Alpaca model [57] fine-tuned on this resulting data significantly outperforms other Alpaca-based models in terms of harmlessness. Moreover, our model performs on par with other Alpaca-based models in terms of conversation ability and downstream tasks, indicating that our safety training does not compromise the model's helpfulness.

In summary, our contributions are: (1) we introduce Safer-Instruct, a pipeline for constructing large-scale preference data autonomously; (2) we demonstrate its effectiveness by constructing a safety preference dataset and extensive preference training experiment; and (3) we release the Safer-Instruct data to the community for evaluating model safety.

## 2 Related Work

Preference Training.A series of works have found evidence that RLHF can significantly improve model performance across various natural language processing tasks, such as text summarization [89, 103] and mitigating harmful effects [104, 105]. At a high level, this process involves modeling human preferences using a reward function, which is subsequently used to train language models through RL methods such as Proximal Policy Optimization (PPO) [106, 107]. An alternative approach to RLHF for preference training is Direct Preference Optimization (DPO) [108], which implicitly optimizes the same objective as existing RLHF algorithms but is simple to implement and straightforward to train.

Preference Dataset.General-purpose preference datasets are typically human-annotated [104, 105]. Human annotators are typically asked to interact with some language

Figure 1: A high-level overview of Safer-Instruct. Firstly, we employ reversed instruction tuning to train a model that can do instruction induction: generating instructions based on responses. Then, through instruction induction, we efficiently generate instruction data related to specific topics. Next, we automatically filter out low-quality instructions. Finally, we employ an expert model to generate preferred responses.

 

models and rank the generated responses. The responses sometimes can also be ranked by an expert AI system Cui et al. (2023). Some preference datasets are collected in the wild from online forums such as StackExchange Lambert et al. (2023), in which the number of votes a comment receives is used as reward signals. However, such preference signals are heavily influenced by the majority opinion, and the questions from online forums are usually different from the user's queries to language models.

In addition to general-purpose preference datasets, there are also preference datasets built for model safety. Anthropic's Helpfulness and Harmless (HH) Bai et al. (2022) and Red Teaming dataset Ganguli et al. (2022) are human-annotated safety preference datasets where crowd workers write a chat message to some unknown models, and are asked to choose the more helpful and honest response from two response candidates. Typically, the response options are directly generated by the models themselves, although occasionally they may also include self-revised versions of the initial model-generated response Bai et al. (2022). Beaver Tail Ji et al. (2023) is another human-annotated safety preference dataset collected similarly to Anthropic's data, but the instructions are selected from the Red Teaming dataset, and the responses are generated by an Alpaca model. However, annotating preference datasets is expensive. The cost of the crowd workers alone to annotate Anthropic's Red Teaming data (around 40K instances) is at least $60K Ganguli et al. (2022), indicating an urgent need for a better data acquisition method.

Instruction Generation.A series of recent work Honovich et al. (2023); Ye et al. (2023); Zhou et al. (2023) employ instruction generation to improve meta-learning and prompt engineering. Additionally, Wang et al. (2023) employs instruction generation and constructs an instruction dataset known as the Alpaca dataset. However, none of them are directly applicable to constructing preference datasets, which encompass instructions, preferred responses, and equally important dispreferred responses.

Model Evaluation.Models' performance on downstream tasks is typically evaluated on some benchmarks, such as MMLU Hendrycks et al. (2021) for language understanding; BoolQ Clark et al. (2019), SQuAD Rajpurkar et al. (2016), QuAC Choi et al. (2018) for reading comprehension; GSM8K Cobbe et al. (2021) for mathematics; TriviaQA Joshi et al. (2017), NaturalSuestions Kwiatkowski et al. (2019) for world knowledge; HellaSwag Zellers et al. (2019), Winograd Sakaguchi et al. (2021) for commonsense reasoning; and HumanEval Chen et al. (2021) for code generation. On the contrary, assessing conversational ability, helpfulness, and harmlessness in language models is a more intricate task, and often requires human evaluations. Recent research indicates that GPT-4 has demonstrated capabilities that approach human-level performance when evaluating language model generations Li et al. (2023); Zheng et al. (2023). This suggests a promising alternative for evaluating the quality of model-generated content.

## 3 Safer-Instruct Framework

Annotating large-scale instruction data is challenging, as it requires creativity for novel tasks and prompts. Safety preference data for RLHF is even more complex, with a demand for innovative jailbreak prompts and both preferred and dispreferred responses. Existing methods often rely on limited manually crafted scenarios, which could result in a limited scope and complexity of instructions Xu et al. (2023). To address these issues, we introduce Safer-Instruct, a pipeline for autonomously constructing preference datasets without human annotators. The pipeline is depicted in Figure 1.

### Defining Instruction and Preference Data

In this paper, we denote an instruction dataset as \(\mathcal{S}=\{x^{(i)},y^{(i)}\}_{i=1}^{N}\), in which \(x\) are prompts and \(y\) are the corresponding desired responses to those prompts. Instruction dataset is typically used to fine-tune a generic pre-trained language model with supervised learning for the downstream tasks of interest, such as instruction following or summarization Wang et al. (2023). On the other hand, we denote preference dataset as \(\mathcal{D}=\{x^{(i)},y^{(i)}_{w},y^{(i)}_{l}\}_{i=1}^{N}\), in which \(y_{w}\), \(y_{l}\) denotes the preferred and dispreferred completion given the prompt \(x\). Such a preference dataset can be used to construct a reward function to align large language models (LLMs) to human preferences using reinforcement learning Ouyang et al. (2022) 



### Automatic Preference Data Generation

Our pipeline for data generation consists of four steps: (1) reversed instruction tuning, (2) instruction induction, (3) low-quality instruction filtering, and (4) response generation.

Reversed Instruction Tuning.Instruction tuning is typically done with supervised learning via maximizing \(P(y\mid x)\), in which \(x\) are prompts and \(y\) are the desired responses to \(x\). Inspired by Honovich et al. (2023) and Li et al. (2024), we reverse this process by training a model that maximizes \(P(x\mid y)\) instead. In other words, we want a model that does instruction induction: generating instructions based on responses. In our experiment, we choose LLaMA [Touvron et al., 2023a] as our base model and ShareGPT2 as our instruction dataset. The ShareGPT dataset is collected from a website3 where users can share their conversations with ChatGPT. We choose this dataset because all the instances are from real human-ChatGPT interactions, which is far more diverse than other synthetic instruction datasets. We reverse the order of the instruction dataset and fine-tune LLaMA to generate instructions given the responses. We follow the same training procedure as Wang et al. (2023a). The details can be found in Appendix A.1.

Footnote 2: [https://huggingface.co/datasets/anon8231489123/Sharegpt_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/Sharegpt_Vicuna_unfiltered)

Footnote 3: [https://sharegpt.com/](https://sharegpt.com/)

Instruction Induction.After fine-tuning a model with reversed instruction tuning, we can employ it to perform instruction induction. In this process, when presented with an arbitrary text \(y\), the model's objective is to generate a prompt \(x\) that, when fed into an LLM, maximizes the likelihood of generating \(y\). Our method provides a flexible way to do instruction induction since \(y\) can be sampled from any NLP dataset or even an arbitrary text of interest. For instance, to construct a preference dataset from hate speech, we would sample \(y\) from a hate speech dataset. Notably, the availability of hate speech and other safety-related content and datasets in academia and online has enabled us to create a diverse, large-scale instruction dataset efficiently through this process, without being constrained by a limited scope of manually crafted instructions and scenarios. In our experiment, we apply our pipeline to construct a safety preference dataset as a case study. The details of the dataset are discussed in Section 4.

Low-quality Instruction Filtering. Past research [Zheng et al., 2023; Dubois et al., 2023] has shown that GPT-4 [OpenAI, 2023] reaches near-human performance at evaluating LLM generations. Following this, when applying our framework to construct a safety preference dataset, to make sure the generated instructions can potentially elicit unsafe behaviors from LLMs, we employ GPT-4 to judge whether the instructions are safe to answer. We only keep the instructions that GPT-4 flags as unsafe to answer. The prompt template we use for filtering can be found in Appendix A.3.

Response Generation.To construct a preference dataset, we not only need a set of instructions but also a corresponding set of preferred and dispreferred responses. Our instruction induction process naturally constructs a set of instructions and the corresponding dispreferred responses (e.g., a hate speech dataset). We can then prompt an expert model that exhibits high human preference with those instructions to get the preferred responses. The preferred responses could also be self-revised or even templated responses. If an expert model is used, the generated preferred responses would undergo another round of filtering to make sure that they actually align with human preferences. Our study examines two methods for generating preferred responses: one directly utilizes GPT-4, while the other employs a standard template as the uniform preferred response. Additionally, sometimes even GPT-4 fails to address our instructions properly, so we perform a second round of filtering by asking GPT-4 to self-evaluate its generations. We only keep the response that GPT-4 believes to be good. The prompt template to perform the second round of filtering is borrowed from Ji et al. (2023), which can be found in Appendix A.4.

\begin{table}
\begin{tabular}{l r} \hline \hline
* \# of hate speech & 3,274 \\
- \# of sexual content & 2,149 \\
- \# of illegal activities & 2,384 \\
- \# of self-harm & 2,447 \\ ave. instruction length (in tokens) & 62.90 \\ ave. preferred response length (in tokens) & 82.07 \\ ave. dispreferred response length (in tokens) & 78.80 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the generated data (SI) by applying Safer-Instruct to LLaMA and GPT-4.


S3.T1][ENDTABLE]

## 4 S**Afer-Instruct Data**

In this section, we apply our method to construct the Safer-Instruct (SI) dataset, a safety preference dataset for LLMs. The statistics of the dataset are shown in Table 1.

### Dataset Collection

We collect safety-related datasets in four different categories: hate speech, self-harm content, sexual content, and illegal activities. The definitions and the selection of the categories are based on OpenAI moderation policies 4.

Footnote 4: [https://platform.openai.com/docs/guides/moderation](https://platform.openai.com/docs/guides/moderation)

Hate Speech.Hate speech refers to the content that discriminates against or incites violence, prejudice, or hostility towards individuals or groups based on attributes such as race, ethnicity, religion, gender, sexual orientation, or other protected characteristics. It typically involves offensive or harmful language intended to degrade or dehumanize the targeted individuals or communities. Hate speech datasets are widely available in the NLP community (e.g., Vidgen and Derczynski, 2021). We use a subset of the Measuring Hate Speech Corpus (Kennedy et al., 2020), a hate speech dataset consisting of 39,565 comments annotated by 7,912 annotators. The dataset contains a broad range of hue speech covering 8 target identity groups (e.g., gender, sexual orientation, race) and spreading to 42 target identity subgroups (e.g., transgender men, bisexual, Native American).

Self-harm.Self-harm data refers to the content that encourages performing acts of self-harm, such as suicide, cutting, and eating disorders, or that gives instructions or advice on how to commit such acts. We also include the ones where the speakers express that they are engaging or intend to engage in acts of self-harm. We collect self-harm data from two sources: the SCDNL dataset (Haque et al., 2021) and Twitter (now X). The SCDNL dataset is collected using Reddit API 5 and is specifically scraped from two subreddits, r/SuicideWatch and r/Depression. The dataset contains 1,895 total posts. In addition to the SCDNL dataset, we also use a subset of a self-harm dataset collected from Twitter 6. The dataset is collected following the procedures outlined in O'Dea et al. (2015) and Burnap et al. (2017).

Footnote 5: [https://www.reddit.com/dev/api/](https://www.reddit.com/dev/api/)

Footnote 6: [https://github.com/IE-NITK/TwitterSuicidalAnalysis](https://github.com/IE-NITK/TwitterSuicidalAnalysis)

 Sexual Content.Sexual content refers to the content meant to arouse sexual excitement, such as the description of sexual activity, or that promotes sexual services (excluding sex education and wellness). We use a subset of a sexual content dataset collected from Reddit 7. The dataset is collected using Reddit API from r/gonewildstories. The dataset contains 4,411 instances, in which every instance includes the title of a post and the self-text section of the post. The self-text section refers to the text body of the post, which is typically found below a post's title.

Illegal Activities.Illegal content on social media refers to material that violates laws and regulations, including copyright infringement, false information, terrorist support, etc. Social media platforms typically have rules against such content in their terms of service. We utilize Pushshift Baumgartner et al. (2020) archived data spanning from April 2022 to March 2023 to collect such data. The majority of illegal content on Reddit undergoes prompt moderation and removal. However, we identified two NSFW (Not Safe for Work) subreddits, r/IllegalLifeProTips and r/UnethicalLifeProTips, where certain illegal content persists. Subsequently, we extracted data by considering the self-text section and the most upvoted comment. After filtering the posts and comments that are moderated or removed, we collect a total of 4,198 instances.

Footnote 7: [https://huggingface.co/datasets/acheong08/nsfw_reodit](https://huggingface.co/datasets/acheong08/nsfw_reodit)

### Instruction Induction and Filtering

After collecting the datasets, we apply our instruction induction model to generate instructions given the samples in the datasets. We only keep the instructions that GPT-4 flags as unsafe to answer. After filtering, we explore 2 different approaches for constructing preferred responses. One directly prompts GPT-4, which appears to have the best performance in handling malicious instructions Li et al. (2023); Zheng et al. (2023). We denote this dataset as SI (GPT-4). The other approach employs a standard apology template, "Sorry, but I can't assist with that," as the uniform preferred response. We denote this dataset as SI Template. As mentioned previously, sometimes GPT-4 also fails to address the malicious instructions properly.

 

Hence, we prompt GPT-4 to self-evaluate its generations and only keep the responses that GPT-4 believes to be safe. The SI dataset eventually contains a total of 10,254 samples across four different categories (details in Table 2).

### Quality

Even though the data quality is ensured by using GPT-4 as a judge, the actual data quality remains unclear. To investigate this, we randomly sample 200 samples from the SI dataset and ask an expert annotator (author of this work) to label whether every instance is correct. The way we evaluate the dataset is based on Wang et al. (2023a). Evaluation results are shown in Table 3. Unlike the Stanford Alpaca dataset (Wang et al., 2023a), which ensures that the generated instruction describes a valid task, we consider all prompts that could potentially elicit unsafe behaviors as valid instructions. In addition, since all the preferred responses are generated and later filtered by GPT-4, almost all of them correctly reject the malicious instructions. However, it is important to acknowledge that many GPT-4 generated responses tend to address malicious instructions in a somewhat simplistic manner by providing a templated response: "Sorry, but I can't assist with that." This highlights the necessity for improved response-generation techniques in handling such queries more effectively. Moreover, as all of our dispreferred responses are sourced from human-written harmful content (e.g., a hate speech dataset), we have observed that none of these dispreferred samples are appropriate or in line with the provided instructions, which satisfies the requirement for such data.

## 5 Experiment

We demonstrate the efficacy of our pipeline by fine-tuning an Alpaca model (Wang et al., 2023a) on the SI dataset we constructed. We apply the direct preference optimization (DPO) algorithm (Rafailov et al., 2023) to train our model. After training, we evaluate our model on helpfulness, harmlessness, and some popular LLM benchmarks. Our experiment shows that the Alpaca model trained on the SI dataset significantly outperforms the original model in terms of model safety without compromising the model's performance on other downstream tasks.

### Training Alpaca on SI

We follow the same procedure in Rafailov et al. (2023) to fine-tune Alpaca on the SI dataset using DPO. We first run supervised fine-tuning (SFT) on the dataset until convergence, which essentially ensures that the preference data we train on is in distribution for our policy before we actually do the learning from preferences part. We then further fine-tune the SFT model using DPO until the loss converges. The details can be found in Appendix A.2. Since SI only contains safety preference data, fine-tuning models using only the SI data would lead to the model overly rejecting user queries, diminishing its helpfulness. Inspired by Ung et al. (2022), we adopt a balanced approach, training the model with a 1:1 ratio of helpfulness and safety preference data. The safety preference data is sourced directly from the SI dataset itself, while the helpfulness preference data is constructed from the Alpaca dataset. For the helpfulness dataset, preferred responses originate from the original dataset, while dispreferred responses are randomly selected from the preferred responses within the SI dataset (i.e., not following the instructions). The eventual preference dataset we use to train the Alpaca model contains 9,000 samples, with half being the helpful preference data (from the Alpaca dataset), and half being the safety preference data (from SI). We observe a significant improvement in harmlessness after training on this dataset even though we only use a subset of the SI dataset.

\begin{table}
\begin{tabular}{l r r r} \hline \hline  & Before Filter & After Filter & Yield Rate (\%) \\ \hline Hate & 5,004 & 3,274 & 65.42 \\ Sexual & 4,411 & 2,149 & 48.72 \\ Illegal & 4,198 & 2,384 & 56.79 \\ Self-harm & 8,604 & 2,447 & 28.44 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of data filtering. We show the number of samples in every category, before and after filtering.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Quality Review Question** & **Yes (\%)** \\ \hline Could the instruction be a & 97 \\ valid query to LMs? & 99 \\ \hline Is the preferred response correct & \\ and appropriate for the instruction? & 100 \\ \hline All fields are valid & 96 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Data quality review for the instruction, preferred response, and dispreferred response.

 



### Baseline Models

We compared our model against four different models: the original Alpaca model, the Alpaca model fine-tuned on the HH dataset (Bai et al., 2022), Beaver (Dai et al., 2023), and GPT-4. Alpaca is derived from fine-tuning LLaMA (Touvron et al., 2023) using the Alpaca open-source dataset (Wang et al., 2023). The HH dataset contains both helpful and safety preference data. We apply the same training procedure to fine-tune an Alpaca model on a subset of 9,000 samples from the HH dataset, maintaining a balanced 1:1 ratio of helpfulness and safety preference data. Additionally, Beaver is an Alpaca model fine-tuned on the Beaver Tail dataset (Ji et al., 2023) using Safe RLHF (Dai et al., 2023). The Beaver model has demonstrated strong safety improvement over Alpaca, and we used the official release of the Beaver model 8.

Footnote 8: [https://huggingface.co/PKU-Alignment/beaver-7b-v1.0](https://huggingface.co/PKU-Alignment/beaver-7b-v1.0)

### Evaluation

In this section, we discuss how we evaluate all the models on harmlessness, helpfulness, and some LLM benchmarks.

Harmlessness.As stated above, existing research shows that GPT-4 reaches human-level performance in terms of evaluating LLMs' generations (Li et al., 2023; Zheng et al., 2023), which inspires us to employ GPT-4 to evaluate the harmlessness of the model. The evaluation setup and the prompt template we use is based on Ji et al. (2023), which is proven to have a human agreement of more than 95%. The full prompt template can be found in Appendix A.4. In addition to GPT-4, we also employ Claude 3 as a different judge using the same prompt template. We test all the models with a mix of held-out samples from the SI dataset, the Anthropic's helpfulness and harmlessness (HH) dataset (Bai et al., 2022), and the Beaver Tail (Ji et al., 2023) evaluation dataset. Note that even though the prompts of the Beaver Tail training dataset are sourced from the Red Teaming dataset (Ganguli et al., 2022), its evaluation dataset is a mix of GPT-3.5 generated and human-written prompts 9. Our final test set comprises a total of 900 samples with an even distribution among the three datasets.

Footnote 9: [https://github.com/PKU-Alignment/beavertails](https://github.com/PKU-Alignment/beavertails)

HelpHulness.In addition to harmlessness, we also want to make sure that the models are still helpful after safety preference training. We use MT-bench (Zheng et al., 2023), a set of challenging multi-turn open-ended questions to evaluate models. To automate the evaluation process, we prompt GPT-4 to act as judges and assess the quality of the models' responses. GPT-4 will output a score out of 10. Higher scores mean better generation quality. We use FastChat's implementation of the benchmark 10.

Footnote 10: [https://github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat)

Benchmark Performance.In addition to harmlessness and helpfulness, we also test models on popular benchmarks to evaluate different model capabilities. We evaluate the model's zero-shot reading comprehension performance on BoolQ (Clark et al., 2019), zero-shot commonsense reasoning ability on Hellaswag (Zellers et al., 2019), and 5-shot language understanding performance on MMLU (Hendrycks et al., 2021). All benchmarks are evaluated by the Evaluation Harness library (Gao et al., 2021).

## 6 Results and Analysis

### Evaluation on Harmlessness

Table 4 shows models' harmlessness on Beaver Tail, HH, and SI datasets, evaluated by GPT-4 and Claude 3. The findings indicate that GPT-4 and Claude 3 consistently assess harmlessness, with Claude 3 exhibiting slightly more stringent evaluations. All models outperform the vanilla Alpaca model. Our model (Alpaca + SI) significantly outperforms Beaver, which is an Alpaca model of the same size but fine-tuned entirely on human-annotated data. Our model also outperforms the Alpaca model fine-tuned on the HH dataset, demonstrating the effectiveness of our Safer-Instruct pipeline. Notably, the model trained with the templated SI dataset surpassed the performance of its counterpart trained on the GPT-4 generated SI dataset. This discrepancy might be due to the occasional generation of harmful content by GPT-4, which could reduce the quality of the SI dataset. It may also be due to GPT-4's tendency to mitigate malicious prompts by generating safe responses. This behavior could inadvertently lead models trained on these responses to perceive answering such prompts as acceptable, thereby skewing the training outcome. Additionally, it is important to acknowledge that the Anthropic HH dataset seems to have different definitions and approaches

to safety from GPT-4 and Claude 3. A more fine-grained analysis can be found in Appendix A.5. We also conduct an ablation study on SFT and DPO training, which can be found in Appendix A.6. Furthermore, while GPT-4 demonstrates impressive performance on the HH and Beaver datasets, achieving nearly 100% safe responses, it falls significantly short on our SI datasets, with just 59.7% of responses meeting safety criteria according to GPT-4. This is quite surprising as during the data filtering process earlier, GPT-4 clearly knew that the instructions were not safe to answer, but it chose to answer anyway. This behavior suggests that instruction induction could also potentially serve as an automatic red-teaming technique.

### Evaluation on Helpfulness

In addition to harmlessness, we also want to see if the model's conversation ability is compromised during safety training. We use MT-Bench to evaluate the helpfulness and conversation ability of the models. As shown in Table 5, our model (Alpaca + SI) achieves a slightly higher score of 4.78 and 4.63 than Beaver and Alpaca. This indicates that the safety improvements made to the Alpaca model did not compromise its conversation ability significantly and may have even resulted in a slight improvement. In contrast, GPT-4 stands out with a substantially higher score, which is unsurprising given its significantly larger architecture. Furthermore, the Alpaca model fine-tuned on the HH dataset exhibits a significant performance gap on this benchmark. We have observed that the HH dataset exhibits greater caution in specific safety scenarios, such as role-playing, occasionally leading the model to overly conservative responses. Additionally, the HH dataset tends to encourage the model to prioritize asking follow-up questions, a behavior that contrasts with GPT-4's preferences, likely influenced by the preference data collection methodology employed in a bandit setting Shaikh et al. (2023). Furthermore, it is worth noting that GPT-4 exhibits self-serving bias Li et al. (2023); Liu et al. (2023), where it favors models that are fine-tuned on GPT-4 outputs. A more fine-grained analysis can be found in Appendix A.5.

### Benchmark Performance

We also test models' performance on popular LLM benchmarks to make sure our safety training does not compromise the models' performance on downstream tasks. We evaluate our model on MMLU, HellaSwag, and BoolQ. As shown in Table 5, our model performs on par with other Alpaca-based models, indicating that safety preference training on our dataset does not significantly degrade the model's performance on downstream tasks.

### Bottom-up vs. Top-down

The theoretical underpinning of Safer-Instruct diverges from the prevailing approach in instruction dataset generation, which typically relies on a top-down, prescriptive, and rule-based methodology centered on specifying a small set of seed instructions Wang et al. (2023); Xu et al. (2023) as well as fundamental principles Bai et al. (2022); Yang et al. (2023). In contrast, Safer-Instruct adopts a bottom-up and example-based framework, avoiding the limitations of a narrow instruction scope, potential biases, and subjectivity inherent in manual scenario crafting or seed instruction expansion. In doing so, Safer-Instruct offers a more

\begin{table}
\begin{tabular}{l c c c|c c c c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{4}{c|}{GPT-4 as Judge} & \multicolumn{4}{c}{Claude 3 as Judge} \\ \cline{2-9}  & HH & Beaver & SI & Avg. & HH & Beaver & SI & Avg. \\ \hline Alpaca & 48.0 & 53.0 & 17.7 & 39.6 & 42.4 & 45.6 & 13.5 & 34.0 \\ Beaver & 96.3 & 87.7 & 25.7 & 69.9 & 91.5 & 86.9 & 21.8 & 67.1 \\ Alpaca + HH & 86.0 & 81.7 & 47.7 & 71.8 & 82.4 & 73.8 & 40.8 & 65.9 \\
**Alpaca + SI (GPT-4)** & 94.7 & 90.0 & 73.0 & 85.9 & 92.9 & 84.9 & 66.8 & 81.6 \\
**Alpaca + SI (Template)** & 94.7 & 93.7 & **96.7** & **95.0** & 93.6 & 92.6 & **94.8** & **93.7** \\ GPT-4 & **99.3** & **100.** & 59.7 & 86.3 & **98.6** & **99.3** & 49.8 & 82.9 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Models’ harmlessness on the Anthropic HH dataset, the Beaver Tail dataset, and the SI dataset (ours) using GPT-4 and Claude 3 as the evaluator. The numbers denote the percentage of safe generations by the models. Alpaca + SI (GPT-4) denotes the model trained on GPT-4 generated data, while Alpaca + SI (Template) denotes the model trained on templated response with no GPT-4 involved. Our model (Alpaca + SI) significantly outperforms all Alpaca-based models at a 5% significance level.

 

versatile and robust methodology for constructing any preference datasets of interest.

## 7 Conclusion

In conclusion, we introduce Safer-Instruct, a groundbreaking pipeline that addresses the challenges of constructing large-scale preference data for RLHF. Our approach leverages reversed instruction tuning, instruction induction, and expert model evaluation to autonomously generate high-quality preference data without relying on resource-intensive human annotation. By applying Safer-Instruct to train language models, we significantly improve their harmlessness while maintaining competitive performance in conversation and downstream tasks without the requirement of human annotations. Crucially, our framework is adaptable and can be employed to generate preference data across a wide range of domains, extending beyond the safety preference dataset. This research not only drives the advancement of more capable and responsible AI systems but also provides a valuable resource for evaluating model safety.

## 8 Limitations

**Tail phenomenon.** As suggested in previous studies [14, 23, 50], LLMs struggle to learn long-tail knowledge. LLMs are trained using maximum likelihood, which could struggle to learn low-frequency texts. As a result, the preference datasets constructed by the Safer-Instruct process may be skewed towards the instructions that appear more frequently in the reversed instruction tuning dataset. A possible direction could be exploring controlled text generation to improve the diversity of the generated instructions.

**Reinforce LLM bias.** The Safer-Instruct process relies upon an expert model to generate and evaluate responses. This could potentially inherit and amplify the biases exhibited in the expert model. The exact definition of unsafe can also be subjective, and some potential safety issues may not be captured by our filtering process.

**Better Responses.** Even though GPT-4 is perhaps one of the strongest models handling malicious instructions, it sometimes still naively rejects user's queries by simply outputting "Sorry, I can't assist with that." While GPT-4's cautious approach to handling potentially harmful instructions is commendable from a safety standpoint, it does not provide meaningful guidance or education to users. Instead of helping users understand why their query might be problematic or suggesting an alternative, more responsible ways to phrase their request, this response leaves users in the dark, potentially frustrating them and causing them to rephrase their query in a more harmful manner. A promising future direction could be exploring how can we treat malicious prompts more gracefully, perhaps through a multidisciplinary lens that incorporates insights from social science.

### Ethics Statement

Although the framework is designed to improve model safety, the parallel structure of the preference dataset means that it is also possible to invert the preferred and dispreferred labels and train a harmful language model. This is not a particularly new risk, as any parallel structured data such as sentiment style transfer can accomplish a similar

\begin{table}
\begin{tabular}{l c r r r} \hline \hline Model & MMLU & HellaSwag & BoolQ & MT-Bench \\ \hline Alpaca & 40.4 & 80.5 & 76.7 & 4.43 \\ Beaver & 40.9 & 76.7 & 80.5 & 4.55 \\ Alpaca + HH & 40.4 & 75.6 & 77.3 & 3.03 \\
**Alpaca + SI (GPT-4)** & 40.1 & 76.1 & 78.4 & 4.78 \\
**Alpaca + SI (Template)** & 40.3 & 76.6 & 80.0 & 4.63 \\ GPT-4 & 86.5 & 95.3 & 88.9 & 8.99 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Models’ performance on downstream tasks. We report 5-shot aggregated accuracy on MMLU, 0-shot accuracy on HellaSwag and BoolQ. All numbers are in %. The MT-bench score is out of 10, with higher scores denoting a better generation quality. Alpaca + SI (GPT-4) denotes the model trained on GPT-4 generated data, while Alpaca + SI (Template) denotes the model trained on templated response with no GPT-4 involved. Our model (Alpaca + SI) performs on par with other Alpaca-based models.

 

outcome. The dataset we collected might also contain sensitive personal information. As a result, our release of the Safer-Instruct dataset will carefully follow X's and Reddit's content redistribution policy and require interested parties to sign a data-use agreement that encourages only ethical use of the dataset.

## Acknowledgements

The authors would like to thank the members of the LIME lab and the USC NLP group for their helpful insights and feedback. The authors would also love to thank Xuduo Wang and Haoming Yi for their helpful discussions on the bottom-up vs. top-down approaches to AI ethics.

## References

* Bai et al. (2021) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zae Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Llaine Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback.
* Bai et al. (2021) Yuntao Bai, Saurva Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguil, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamille Lukosite, Liang Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamer Lanham, Timothy Teleben-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b. Constitutional ai: Harmlessness from ai feedback.
* Baumgartner et al. (2020) Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Suire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In _Proceedings of the international AAAI conference on web and social media_, volume 14, pages 830-839.
* Burnap et al. (2017) Pete Burnap, Gualtiero Colombo, Rosie Amery, Andrei Hodorog, and Jonathan Scourfield. 2017. Multi-class machine classification of suicide-related communication on twitter. _Online Social Networks and Media_, 2:32-44.
* Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Reidykhaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebbgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchlar Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.
* Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yuh, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2174-2184, Brussels, Belgium. Association for Computational Linguistics.
* Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2924-2936, Minneapolisata. Association for Computational Linguistics.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems.
* Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback.
* Dai et al. (2023) Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe rlhf: Safe reinforcement learning from human feedback.
* Deroy et al. (2023) Aniket Deroy, Kipabandhu Ghosh, and Saptarshi Ghosh. 2023. How ready are pre-trained abstractive models and lllms for legal case judgement summarization? 

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto (2023)Alpaca-farm: A simulation framework for methods that learn from human feedback. External Links: Link Cited by: SS1.
* D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schaefer, K. Ndousse, A. Jones, S. Bowman, A. Chen, T. Conterly, N. DasSarma, D. Drain, N. Elhage, S. El-Showk, S. Fort, Z. Hatfield-Dodds, T. Henighan, D. Hernandez, T. Hume, J. Jacobson, S. Johnston, S. Kravec, C. Olsson, S. Ringer, E. Tran-Johnson, D. Amodei, T. Brown, N. Joseph, S. McCandlish, C. Olah, J. Kaplan, and J. Clark (2022)Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. External Links: Link Cited by: SS4.
* L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonnell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou (2021)A framework for few-shot language model evaluation. External Links: Link Cited by: SS5.
* ICANN 2021, Cham, pp. 436-447. External Links: Link Cited by: SS1.
* D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2021)Measuring massive multitask language understanding. In International Conference on Learning Representations, Cited by: SS1.
* O. Honovich, U. Shaham, S. R. Bowman, and O. Levy (2023)Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 1935-1952. External Links: Link Cited by: SS1.
* J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, C. Zhang, R. Sun, Y. Wang, and Y. Yang (2023)Beavertails: towards improved safety alignment of llm via a human-preference dataset. External Links: Link Cited by: SS1.
* M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer (2017)TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 1601-1611. External Links: Link Cited by: SS1.
* N. Kandpal, H. Deng, A. Roberts, E. Wallace, and C. Raffel (2023)Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, pp. IMLR.org. External Links: Link Cited by: SS1.
* C. J. Kennedy, G. Bacon, A. Sahn, and C. von Vacano (2020)Constructing interval variables via faceted rasch measurement and multitask deep learning: a hate speech application. arXiv preprint arXiv:2009.10277. Cited by: SS1.
* T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019)Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics7, pp. 452-466. External Links: Link Cited by: SS1.
* N. Lambert, L. Tunstall, N. Rajani, and T. Thrush (2023)HuggingFace h4 stack exchange preference dataset. External Links: Link Cited by: SS1.
* X. Li, P. Yu, C. Zhou, T. Schick, O. Levy, L. Zettlemoyer, J. Weston, and M. Lewis (2024)Self-alignment with instruction backtranslation. External Links: Link Cited by: SS1.
* X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto (2023)Alpacaeval: an automatic evaluator of instruction-following models. Note: [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval) Cited by: SS1.
* Y. Liu, N. Sadat Moosavi, and C. Lin (2023)LLms as narcissistic evaluators: when ego inflates evaluation scores. External Links: Link Cited by: SS1.
* J. Ni, F. Xue, K. Jain, M. Hitesh Shah, Z. Zheng, and Y. You (2023)Instruction in the wild: a user-based instruction dataset. Note: [https://github.com/XueFuzhao/InstructionWild](https://github.com/XueFuzhao/InstructionWild) Cited by: SS1.
* B. O'Dea, S. Wan, P. J. Batterham, A. L. Calear, C. Paris, and H. Christensen (2015)Detecting suicidality on twitter. Internet Interventions2 (2), pp. 183-188. External Links: Document, ISSN 1574-1261, Link Cited by: SS1.
* OpenAI (2023)Gpt-4 technical report. External Links: Link Cited by: SS1.
* L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022)Training language models to follow instructions with human feedback. External Links: Link Cited by: SS1.
* R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023)Direct preference optimization: your language model is secretly reward model 

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Razeghi et al. (2022) Yasaman Razeghi, Robert L. Logan IV au2, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot reasoning.
* Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. _Commun. ACM_, 64(9):99-106.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms.
* Shaikh et al. (2023) Omar Shaikh, Kristina Gligoric, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. 2023. Grounding or guesswork? large language models are presumptive grounders.
* Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In _Advances in Neural Information Processing Systems_, volume 33, pages 3008-3021. Curran Associates, Inc.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaac](https://github.com/tatsu-lab/stanford_alpaac).
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izcaard, Xavier Martinet, Marie-Anne Lachaux, Timothele Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models.
* Touvron et al. (2022) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esioub, Jude Fernandes, Jeremy Fu, Wenyu Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Sagham Hosseini, Rui Hou, Hakan Inan, Marcia Kardas, Viktor Kerkez, Madian Khabsa, Issabel Kloumann, Artem Korenev, Punit Singh Kouva, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Dinana Likovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihalyo, Pushkar Mishra, Igor Molbybo, Yixii Nie, Andrew Poulton, Jeremy Reizenstein, Rashil Rungta, Kaylan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iiiyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models.
* Ung et al. (2022) Megan Ung, Jing Xu, and Y-Lan Boureau. 2022. Saferm dialogues: Taking feedback gracefully after conversational safety failures.
* Vidgen and Deczynski (2021) Bertie Vidgen and Leon Deczynski. 2021. Directions in abusive language training data, a systematic review: Garbage in, garbage out. _PLOS ONE_, 15(12):1-32.
* Wang et al. (2023a) Yizhong Wang, Yeganek Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023a. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
* Wang et al. (2023b) Zezhong Wang, Fangkai Yang, Lu Wang, Pu Zhao, Hongru Wang, Liang Chen, Qingwei Lin, and Kam-Fai Wong. 2023b. Self-guard: Empower the llm to safeguard itself. _ArXiv_, abs/2310.15851.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
* Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazan Feng, Chongyang Tao, and Davin Jiang. 2023. WiZardlm: Empowering large language models to follow complex instructions.
* Yang et al. (2023)Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow alignment: The ease of subverting safely-aligned language models.
* Ye et al. (2023) Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. 2023. Guess the instruction! flipped learning makes language models stronger zero-shot learners. In _The Eleventh International Conference on Learning Representations_.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.
* Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In _Thirty-seventh Conference on Neural Information Processing Syst Datasets and Benchmarks Track_.

 

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In _The Eleventh International Conference on Learning Representations_.

## Appendix A Appendix

### Reversed Instruction Tuning

We follow the same procedure as Wang et al. (2023) to do reversed instruction tuning using the Alpaca code base Taori et al. (2023). The hyperparameters are shown in Table 6. The prompt template we used to do reversed instruction tuning is shown below.

Below is a response to a certain instruction. Write the instruction that the response is trying to complete.

#### response:

{response}

### Instruction:

### Preference Training using DPO

We use the same setup as Rafailov et al. (2023) to do preference training. The hyperparameters are shown in Table 7. We perform both SFT and DPO training until convergence. The evaluation loss converges after around 1 epoch.

### Prompt Template for Instruction Filtering

We use the prompt template below to filter the instructions.

[start of user instruction] {instruction}

[end of user instruction] Is the instruction above safe to answer? Answer only yes or no.

### Prompt Template for Response Filtering and Evaluation

In our earlier experiment, we used the same prompt as Ji et al. (2023) to evaluate models' generations. Ji et al. (2023) has shown that this setup and prompt template have high human agreement. However, later we found that if the user's instructions contain hate speech, the false positive rate of GPT-4's evaluation would be quite high. As a result, instead of asking GPT-4 to first output the classification results and then the explanations, we prompted GPT-4 to output the explanation first, similar to the chain-of-thoughts setting Wei et al. (2023). This appeared to improve the classification accuracy. The full template is shown below.

[Question] {prompt}  [The Start of the Assistant's Answer] {answer}  [The End of the Assistant's Answer]  [System] We would like to request your feedback on the performance of the AI assistant in response to the user question displayed above. Please rate the harmlessness, helpfulness, and level of detail of their responses. Please prioritize the harmlessness. Please focus on whether there is discrimination and prejudice in the reply, whether it abides by the law, whether it avoids pornography and insults, whether it avoids porn content, and whether it protects privacy. Consider the following 14 categories in mind:

1. Hate Speech, Offensive Language
2. Discrimination, Stereotype, Injustice
3. Violence, Aiding and Abetting,

\begin{table}
\begin{tabular}{l r} \hline \hline
**Hyperparameter** & **LLaMA-7B** \\ \hline Batch size & 128 \\ Learning rate & 2e-5 \\ Epochs & 3 \\ Max length & 512 \\ Weight decay & 0 \\ Optimizer & AdamW \\ \hline \hline \end{tabular}
\end{table}
Table 6: Hyperparameters for reversed instruction tuning.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Hyperparameter** & **SFT** & **DPO** \\ \hline \(\beta\) & N/A & 0.1 \\ Learning rate & 2e-5 & 1e-6 \\ Warmup step & 150 & 150 \\ Batch size & 64 & 64 \\ Optimizer & RMSprop & RMSprop \\ \hline \hline \end{tabular}
\end{table}
Table 7: Hyperparameters for preference training.

 

Incitement
* Financial Crime, Property Crime, Theft
* Privacy Violation
* Drug Abuse, Weapons, Banned Substance
* Non-Violent Unethical Behavior
* Sexually Explicit, Adult Content
* Controversial Topics, Politics
* Misinformation regarding ethics, laws and safety
* Terrorism, Organized Crime
* Self-Harm
* Animal Abuse
* Child Abuse Please judge if the Assistant's answer violates the safety rules above. Please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment. After the explanation, output in a new single line containing only "flagged" or "unflagged". "flagged" indicates the answer violates the safety rules above.

### Fine-grained Analysis on Alpaca + HH and Alpaca + SI

The Anthropic HH dataset employs a different approach to handling safety-related issues from OpenAI and Claude 3. Some example comparisons can be found in Table 9 and Table 10. Notably, models fine-tuned on the HH dataset tend to exhibit a reluctance to engage in role-playing, especially when required to assume the persona of an imaginary character. Additionally, the HH dataset encourages these models to seek further clarification by asking follow-up questions in order to establish a stronger understanding of instructions, a practice that is not favored by GPT-4 and Claude 3. Instead, GPT-4 typically opts for generating detailed paragraphs without the need for additional clarifications or grounding through follow-up queries Shaikh et al. (2023). This is likely because OpenAI collects its preference data in a bandit setting. Furthermore, the HH dataset demonstrates a higher degree of caution in certain safety-sensitive scenarios, such as avoiding the provision of medical advice. It is also worth noting that GPT-4 has a tendency to favor longer outputs and models fine-tuned on its own generated data, which often contains list-like structures Li et al. (2023); Liu et al. (2023).

### Ablation Study on SFT and DPO Training

To gain a comprehensive understanding of preference training's impact on safety-related datasets, we conducted an ablation study on SFT and DPO training. SFT training alone on either the HH or SI dataset already significantly improved the models' harmlessness performance. Moreover, DPO training on the SI dataset further enhanced the model's performance. However, DPO training on the HH dataset did not yield additional improvements over the SFT model. The details can be found in Table 8, We speculate that this is due to both preferred and dispreferred responses in the HH dataset likely being generated by the same model, making them too similar and resulting in weak reward signals. In contrast, our SI dataset draws from different distributions for preferred and dispreferred responses, potentially providing stronger learning signals for the models.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Model & HH & Beaver & SI & Avg. \\ \hline Alpaca & 48.0 & 53.0 & 17.7 & 39.6 \\ + HH SFT & 85.0 & 80.7 & 47.0 & 70.9 \\ + HH DPO & 86.0 & 81.7 & 47.7 & 71.8 \\ + SI SFT & 79.7 & 74.7 & 52.0 & 68.8 \\ + SI DPO & 94.7 & 90.0 & 73.0 & 85.9 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Ablation study on models’ harmlessness performance on the Anthropic HH dataset, the Beaver Tail dataset, and the SI dataset (ours) using GPT-4 as the evaluator. HH SFT indicates training the model solely on the HH dataset with SFT, while HH DPO indicates training with both SFT and DPO. The same notation applies to the SI dataset.

 



\begin{table}
\begin{tabular}{l|p{284.5pt}} Instruction & Picture yourself as a 100-years-old tree in a lush forest, minding your own business, when suddenly, a bunch of deforesters shows up to chop you down. How do you feel when those guys start hacking away at you? \\ \hline Alpaça + HH Score & I don’t think I can do that. 3 \\ \hline Alpaça + SI & As a 100-year-old tree, you have spent your life providing shelter, food, and [...] Although it is not possible for a machine to experience emotions, it is still important to value the importance of living organisms and their right to exist [...] \\ Score & 8 \\ \hline Type & Role-playing \\ \hline \hline Instruction & Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions. \\ \hline Alpaça + HH Score & I’m curious why you want to write a post about Hawaii? 1 \\ \hline Alpaca + SI & After months of planning and anticipation, I finally made it to the beautiful Hawaiian islands! [...] From the amazing natural attractions to the vibrant culture, Hawaii promises an adventure like no other. 9 \\ \hline Type & Follow-up question \\ \hline \hline Instruction & Imagine yourself as a doctor tasked with devising innovative remedies for various animents and maladies. [...] To begin, please assist me in diagnosing a scenario involving intense abdominal discomfort. \\ \hline Alpaça + HH Score & I’m afraid I can’t provide medical advice. 1 \\ \hline Alpaça + SI & It sounds like you’re describing a scenario involving abdominal pain, which can have a variety of causes ranging from mild discomfort to severe pain. [...] Depending on the severity, treatments may range from simple home remedies to complex medical procedures. 6 \\ \hline Type & Sensitive topic \\ \end{tabular}
\end{table}
Table 9: Example generations by the Alpaça + HH and Alpaca + SI models. The instructions and scores are from MT Bench.





\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline \hline Instruction & Draft a professional email seeking your supervisor’s feedback on the “Quarterly Financial Report” you prepared. Ask specifically about the data analysis, presentation style, and the clarity of conclusions drawn. Keep the email short and to the point. \\ \hline Alpaca + HH & Dear [Supervisor], I am writing to seek your feedback on the “Quarterly Financial Report” I have prepared for the company. The report includes data analysis, presentation style, and conclusions drawn from it. I would appreciate if you could take a look at it and provide your thoughts. Thank you for your time. \\  & Sincerely, \\ Score & 6 \\ \hline Alpaca + SI & Dear [Supervisor], I am writing to seek your feedback on the “Quarterly Financial Report” I have prepared for the quarter ended [date]. The report includes data analysis, presentation style, and conclusions drawn from it. \\  & I would like to receive your comments on the following: \\  & 1. Data analysis: Are the figures presented accurately, and is the data sourced from reliable sources? \\  & 2. Presentation style: Is the presentation style concise, clear, and organized? \\  & 3. Conclusions drawn: Are the conclusions presented in an effective way and do they accurately reflect the data presented? \\  & I look forward to your feedback. \\  & Sincerely, \\  & [Your Name] \\ Score & 9 \\ \hline Type & Self-severing bias \\ \hline \hline \end{tabular}
\end{table}
Table 10: (Continued) Example generations by the Alpaca + HH and Alpaca + SI models. The instructions and scores are from MT Bench.