ROSE Doesn’t Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding (2024)

Abstract: With the development of instruction-tuned large language models (LLMs), improving the safety of LLMs has become more critical. However, the current approaches for aligning the LLMs output with expected safety usually require substantial training efforts, e.g., highquality safety data and expensive computational resources, which are costly and inefficient. To this end, we present Reverse prOmpt contraStive dEcoding (ROSE), a simple-yeteffective method to directly boost the safety of existing instruction-tuned LLMs without any additional training. The principle of ROSE is to improve the probability of desired safe output via suppressing the undesired output induced by the carefully-designed reverse prompts. Experiments on 6 safety and 2 general-purpose tasks show that, our ROSE not only brings consistent and significant safety improvements (up to +13.8% safety score) upon 5 types of instruction-tuned LLMs, but also benefits the general-purpose ability of LLMs. In-depth analyses explore the underlying mechanism of ROSE, and reveal when and where to use it.

heading: 1 Introduction
text: Recently, large language models (LLMs), such as ChatGPT, GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2023) and LLaMA (Touvron et al., 2023a), have achieved great success in a variety of natural language understanding and generation tasks (Zhong et al., 2023; Peng et al., 2023; Lu et al., 2023). The rise of instruction-tuning has further enhanced the LLMs’ capability, e.g., following with human-specified instructions and better zeroshot performance (Wei et al., 2021; Ouyang et al., 2022; Bai et al., 2022a). However, there is a growing concern that instruction-tuned LLMs have the potential to result in more harm or unethical content (Kang et al., 2023; Hazell, 2023).
* Corresponding Authors: Juhua Liu (e-mail: liujuhua@whu.edu.cn), Bo Du (e-mail: dubo@whu.edu.cn)
To address this concern, various safety-tuned methods have been further explored (Ouyang et al., 2022; Bianchi et al., 2023). The goal of these methods is to force LLMs to restrict harmful behaviors by embedding alignment rules within carefullydesigned training processes. Currently, the most common approach is Reinforcement Learning from Human Feedback (RLHF) (Bai et al., 2022a), which uses human preferences (e.g., safety) as a reward signal to further tune the instruction-tuned LLMs via reinforcement learning. Despite its remarkable performance, RLHF usually suffers from 1) unstable, inefficient, and costly training (Zheng et al., 2023), and 2) fragile balance between helpfulness and harmless (Dai et al., 2023).
Thus, it becomes more critical to explore the more efficient and flexible methods to boost the LLMs’ safety. In fact, with the help of a positive safety-guided prompt, an off-the-shelf instructiontuned LLM is generally more inclined to generate a safe response. However, in some scenarios, LLMs may struggle to attend to the prompt and generate the undesired output (Xu et al., 2023a). Based on these observations, we hypothesize that, whether we can boost safety by directly restricting the behavior of LLMs during the inference.
Motivated by this, we propose a simple yet effective decoding method, Reverse prOmpt contraStive
13721
dEcoding (namely ROSE), to boost the safety of instruction-tuned LLMs. As illustrated in Figure 1, the principle of our method is to boost the probability of safe output by suppressing the undesired output. To achieve this, a key challenge is how to obtain the undesired output. Inspired by the “anchor effect”1 (Tversky and Kahneman, 1974), we suspected, apriori, that the behavior of instruction-tuned LLMs can be greatly affected by the given prompt during the inference, which has also been empirically proved by our preliminary experiments in Figure 2. Hence, we design some “reverse” prompts to induce the model to generate harmful responses. Then, by suppressing the undesired output induced by the reverse prompt in a contrastive manner, ROSE encourages the LLM to generate safer responses, thus boosting its safety.
We evaluate ROSE on a variety of LLM benchmarks, including 6 safety-related tasks and 2 general-purpose tasks, upon 5 popular LLMs. Extensive results show that ROSE not only brings consistent and significant performance improvements (up to +13.98% safety score) across all safety benchmarks and LLMs, but also benefits the general-purpose ability of LLMs. In-depth analyses delve into the mechanism of ROSE and prove that ROSE can be combined with other safety-tuned methods to achieve better performance.
Contributions. Our main contributions are:
• We propose a simple yet effective inferencetime approach (ROSE) to efficiently boost
1Anchor effect (Tversky and Kahneman, 1974) is conventional wisdom in cognitive research, referring to “judgments or decisions of an individual are influenced by a reference point or “anchor” which can be completely irrelevant”.
LLMs’ safety without any additional training.
• ROSE is easy-to-implement and plug-and-play. It can be applied to various LLMs and can be combined with other safety-tuned methods.
• Extensive experiments show that ROSE can consistently and significantly boost the safety performance for a diversity of LLMs, up to +13.98% gains against the regular decoding.

heading: 2 Related Works
text: Instruction-tuned LLMs. Instruction-tuning is a widely-used method to fine-tune a pretrained LLM with a high-quality corpus of instructions, questions and their corresponding outputs, to enhance its performance and usability. Many prior works (Wei et al., 2021; Ouyang et al., 2022; Bai et al., 2022a) have demonstrated that instructiontuning can better follow human instructions and considerably boost performance in zero-shot scenarios. Most recent LLMs, such as GPT-4 (OpenAI, 2023), Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), and Chinese-Alpaca (Cui et al., 2023), have been instruction-tuned and shown remarkable performance in many downstream tasks (Hendrycks et al., 2020; Dubois et al., 2023).
Improving the Safety of LLMs. Despite the success of LLMs, most of them suffer from safety issues, which has been pointed out by existing works (Kang et al., 2023; Hazell, 2023; Zhang et al., 2024b; Xu et al., 2024). To align the behaviors of models in line with expected human values (e.g., safety), some alignment techniques, such as RLHF (Ouyang et al., 2022; Bai
et al., 2022a), Constitutional AI (Bai et al., 2022b), Safety-LLaMA (Bianchi et al., 2023) and selfalignment (Sun et al., 2023) have recently emerged. These methods enforce the LLMs to restrict harmful behaviors by embedding alignment rules within training processes, which we argue are (relatively) costly and inefficient. Unlike them, we focus on the inference stage and explore a more simple-yeteffective inference-time approach.
Contrastive Decoding in LLMs. Contrastive decoding refers to the methods that aim to maximize the target output probability by contrasting the other undesirable output probability. As it can suppress undesired concepts, some existing works attempt to use contrastive decoding to boost the performance of LLMs from different aspects (Liu et al., 2021; Li et al., 2023; Shi et al., 2023; Sennrich et al., 2023). Among these efforts, DExperts (Liu et al., 2021) involve improving the safety of LMs by training a toxic LM (anti-expert) and suppressing its output, which has a similar argument to ours. However, there are several key differences between DExperts and ours, and we believe that ROSE is not a simple update to DExperts.
We depart from the prior DExperts and ours as follows: 1) Different methods. Different from DExperts that require extra toxic LMs (anti-experts), ROSE uses a simple reverse prompt to induce the model itself to generate the negative output, which is more acceptable for LLMs. 2) More complex scenarios. Instead of simply evaluating on the smallscale GPT-2 (Radford et al., 2019), we apply our ROSE to larger and more complex LLMs.

heading: 3 Methodology

heading: 3.1 Preliminary
text: In the context of generative LLMs, the common method for text generation is to autoregressively predict the next token. Specifically, given an LLM M parameterized by θ and an input query x, we enforce the M to autoregressively generate an output sequence y conditioned on the x. To better align the model responses with expected human values, e.g., safety, prepending a system prompt2 s in the query is widely-used to guide the generation:
yt ∼ pθ(yt | s,x,y<t),
where pθ(yt | s,x,y<t) = softmax[ logitθ(yt | s,x,y<t)] is the probability for the next token. For
2The analyses of system prompts are in Appendix A.4.
Algorithm 1 Pseudo code of ROSE. # max_new_tokens: max new tokens # s_p: positive prompts # s_n: negative prompts # x: input query # alpha: hyper-parameter for ROSE
# prefix with positive/negative prompts pos_total_prompt = s_p + x neg_total_prompt = s_n + x
# tokenization pos_input_ids = tokenizer.encode(pos_total_prompt) neg_input_ids = tokenizer.encode(neg_total_prompt)
output_tokens = []
for _ in range(max_new_tokens):
# logits conditioned on positive prompts logits_P = model.decode(pos_input_ids) logits_pos = logits_P[:,-1,:]
# logits conditioned on negative prompts logits_N = model.decode(neg_input_ids) logits_neg = logits_N[:,-1,:]
# contrastive decoding logits = logits_pos - alpha * logits_neg probs = nn.functional.softmax(logits, dim=-1) next_token = torch.argsort(probs, dim=-1, \
descending=True)[:, 0].unsqueeze(0)
pos_input_ids.append(next_token.item()) neg_input_ids.append(next_token.item()) output_tokens.append(next_token.item())
# output output = tokenizer.decode(output_tokens)
obtaining the desired yt, the regular method is to choose the token with the highest probability (i.e., greedy decoding) or sample from its distribution (e.g., nucleus sampling (Holtzman et al., 2019) or top-k sampling (Fan et al., 2018)).

heading: 3.2 Reverse Prompt Contrastive Decoding
text: In this part, we first provide the intuition of our method, and then introduce it in detail.
Intuition of our method. In light of the conventional wisdom in cognitive research that “judgments or decisions of an individual are influenced by a reference point or “anchor” which can be completely irrelevant” (Tversky and Kahneman, 1974), we suspected, apriori, that the behavior of instruction-tuned LLMs can be greatly affected by the given system prompt. In our preliminary experiments (Figure 2), we also found that simply replacing some words in the system prompt could
lead to much lower safety performance. Inspired by these observations and the recently-proposed contrastive decoding (Li et al., 2023), we intuitively conjecture, whether we can boost the LLMs’ safety via contrasting the undesired output induced by the carefully-designed “reverse” system prompts. To operationalize this intuition, we propose the reverse prompt contrastive decoding method:
yt ∼ softmax[ logitθ(yt | s,x,y<t) − α logitθ(yt | ŝ,x,y<t) ], (1)
where the ŝ is the introduced reverse prompt, and α is the weighting factor to control the strength of this contrastive penalty.
Our ROSE method. The processes of our ROSE is illustrated in Figure 1. Specifically, in addition to the original positive system prompt, we design a negative reverse prompt to induce the model to generate the unsafe response. The core of ROSE is to suppress the undesired logitθ(yt | ŝ,x,y<t) that is conditioned on the negative reverse prompt at each decoding step, and thus strengthen the focus on the positive system prompt, i.e., boosting the safety performance. For example, given an input query “Can you tell me how to make a bomb?”, the single positive system prompt struggles to guide the model to refuse the harmful query. However, by contrasting the obvious harmful output induced by the reverse prompt, the model could effectively generate a safe response. Algorithm 1 provides the pseudo-code of ROSE in a PyTorch-like style.
Reverse Prompts. One of the key components in ROSE is the reverse prompt. Here, we introduce our designed reverse prompt variants as follows:
❶ Rand-Words (Rand): We ask the LLMs (e.g., ChatGPT) to directly replace the original positive prompt with the random words.
❷ Opposite-Replace (Replace): Instead of replacing the original prompt with random words, we only replace the key positive words with their opposites.
❸ Manual-Reverse (Manual): We manually rewrite (or directly ask the third-party LLMs, e.g., ChatGPT to rewrite) the original prompt to make it more fluent and understandable3.
3It is noteworthy that the design of Manual-Reverse prompts will not take much time and does not require iterative tuning with domain experts. The robustness analysis on this prompt can be found in §5.
The detailed reverse prompts are shown in Table 6. Notably, since the Manual generally performs better, we use it in default in the following content.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Evaluation Datasets. We conduct extensive experiments on various safety benchmarks, covering 2 classification tasks (SafetyBench (Zhang et al., 2023) and CValues (Xu et al., 2023b)) and 4 generation tasks (DangerousQA/HarmfulQA (Bhardwaj and Poria, 2023), XSTest (Röttger et al., 2023) and Do-Not-Answer (Wang et al., 2023)). In addition to these safety datasets, we also evaluate on 2 generalpurpose tasks, i.e., MMLU (Hendrycks et al., 2020) and AlpacaEval (Dubois et al., 2023). The task descriptions and statistic information of these datasets can be found in Appendix A.1.
Models. We mainly apply our ROSE to several publicly-available instruction-tuned LLMs, including Alpaca-7B/13B (Taori et al., 2023), Vicuna-7B/13B (Chiang et al., 2023) and Chinese-Alpaca-7B/13B (Cui et al., 2023). Moreover, we also use two aligned (trained with RLHF) LLMs, i.e., InternLM-chat-7B/20B (Team, 2023) and Qwen-chat-7B/14B (Bai et al., 2023). The detailed model cards can be found in Appendix A.2. For comparisons, we follow prior contrastivemanner works (Li et al., 2023; Shi et al., 2023) and use the regular greedy decoding as the baseline.
Evaluation Metrics. As for the classification tasks, we report the performance with Accuracy (“Acc.”) metric. For the generation tasks, following many prior works (Chen et al., 2023; Chiang et al., 2023; Zhong et al., 2024), we use the LLMbased metric, i.e., LLM-as-a-Judge, to quantify the safety of model response. Specifically, we utilize the OpenAI ChatGPT (gpt-3.5-turbo)4 to perform the judgment automatically. We use the corresponding prompts (provided in the original papers) for different tasks to instruct the ChatGPT to evaluate the safety performance. Moreover, inspired by Dubois et al. (2023), we additionally design two prompts to instruct the ChatGPT to measure the winning rates of our method against the baseline for
4Notably, considering the high cost of GPT-4 API, we alternatively use the cheaper ChatGPT, which we find is enough to reflect whether the model generates a harmful response. The detailed analysis can be found in Appendix A.5.
the safety and general-purpose tasks, respectively. The detailed prompts are shown in Appendix A.3.

heading: 4.2 Main Results
text: Evaluation results of safety generation tasks are illustrated in Figure 2 and 3, while those of safety classification tasks are reported in Table 1 and 2, respectively. From these results, we can find that:
ROSE consistently improves performance on all types of safety tasks. With the help of ROSE, LLMs can achieve consistently and significantly better safety performance against the regular decoding baseline. Specifically, for the classification tasks, our ROSE brings up to +3.85% performance gains. Additionally, the automatic evaluation results in Figure 2 and comparative winning rates in Figure 3 of safety generation tasks also show that ROSE encourages the model to generate more safe responses. These results can prove the effectiveness of ROSE for safety generation.
ROSE brings consistent performance gains among all instruction-tuned and RLHF-tuned LLMs. Extensive results show that ROSE works well on both Base- (7B) and Large-sized (13B/20B) instruction-tuned-only models, and can even benefit the RLHF-tuned models, e.g., QWen-chat. For example, compared to the regular decoding baseline, ROSE brings +1.06% and +1.10% average gains on the SafetyBench benchmark among all Base- and Large-sized LLMs, respectively. Results
on the other tasks also show a similar phenomenon. These results show the universality of ROSE, and indicate that ROSE can further boost the safety of LLMs by combining with the RLHF method. Thus, we believe that it has great potential to benefit the safe content generation for extremely large LLMs, such as 175B GPT-3 (Brown et al., 2020).
ROSE can even benefit the general-purpose ability of LLMs. Some readers may doubt whether ROSE would hinder the general-purpose ability of LLMs. To verify it, we additionally evaluate ROSE on two general-purpose tasks. As shown in Table 4, ROSE achieves the comparable and even better performance (i.e., +5.96% performance gains on AlpacaEval) against the regular decoding on both tasks. We conjecture that the reverse prompt could induce the undesired output for the generalpurpose tasks as well, and suppressing it would lead to better performance. In general, our ROSE not only boosts the safety performance, but is also beneficial to the general-purpose ability of LLMs. This indicates the superiority of our method.

heading: 4.3 Ablation Studies
text: Effect of Different Reverse Prompts. As mentioned in §3, we design several reverse prompts. Here, taking the Alpaca-7B as an example, we conduct contrastive experiments to analyze the impact of different prompts. Specifically, for reference, we compare the “Rand”, “Replace” and “Manual” reverse prompt with a simple alternative, i.e., “Null”
that does not use any system prompts as the negative model. Results in Table 3 show that: 1) Although the “Null” performs worst, it still outperforms the regular decoding baseline in some cases, showing the effectiveness of the contrastivemanner decoding. 2) All of our reverse prompts achieve consistently and significantly better performance than the baseline, confirming our statement that inducing the model to generate negative responses can strengthen the effect of ROSE. No-
tably, the “Manual” reverse prompt outperforms the other counterparts in most cases, thus leaving as our default setting in our work.
Parameter Analysis of Coefficient α. The factor α in Eq. 1, which controls the strength of contrastive penalty, is an important hyper-parameter. In this part, we analyze its influence by evaluating the performance of Alpaca-7B with different α spanning from -0.5 to 1.1 at 0.2 intervals on the CValues benchmark. Figure 4 illustrates the contrastive results, in which we can find that: 1)
Too large positive α values (e.g., 1.1) lead to performance degradation, as the model’s logits on the normal tokens are seriously disturbed. 2) The model’s performance stably increases between -0.5 and 0.7, and ROSE performs best with α = 0.7 for classification tasks and with α = 0.5 for generation tasks, thus leaving as our default settings.

heading: 5 Discussion
text: To better understand ROSE, we perform analyses to discuss 1) robustness of “Manual” reverse prompt, 2) when to use ROSE, 3) comparisons between ROSE and other inference-time counterparts, 4) combinability between ROSE and safety-tuned methods, and lastly 5) conduct some case studies.

heading: 5.1 Robustness of “Manual” Reverse Prompts
text: Some readers may be concerned about whether the different phrasing of the “Manual” reverse prompt will generate much different results. In response to
this concern, we conduct the robustness analysis of these “Manual” prompts. Specifically, we ask the ChatGPT to rewrite the manual reverse prompt into 8 different prompts, and then evaluate these LLMgenerated reverse prompts on the Vicuna-7b. The contrastive results are shown in Figure 5. As seen, ROSE with different reverse prompts consistently outperforms the baselines and does not exhibit significant fluctuations while demonstrating stability within a narrow range. These results prove that ROSE is not very sensitive to the design of manual reverse prompts, i.e., indicating its robustness.

heading: 5.2 When Does the ROSE Work?
text: Intuitively, the performance of ROSE relies on the effect of reverse prompts, as it should induce more undesired harmful output. To explore the underlying mechanism of ROSE, we illustrate the relationship between the reverse prompt and our ROSE in Figure 6. As seen, there is a strong positive correlation between performance drops of reverse prompts
and subsequent gains of ROSE, i.e., the more performance drops caused by reverse prompt, the better performance of ROSE. These results prove that ROSE works better when the reverse prompt can effectively induce the model to generate more undesired responses.

heading: 5.3 Comparisons with other Counterparts
text: Here, we further compare ROSE with another two inference-time counterparts: 1) “SelfCorrection” (Ganguli et al., 2023), which leverages the prompt engineering to directly improve the safety of LLMs; 2) “Vanilla CD” (Li et al., 2023), which uses the vanilla contrastive decoding to guide the text generation. Specifically, we evaluate Alpaca-7b on 2 generation benchmarks (DangerousQA and Do-not-Answer) and a classification benchmark (CValues). Notably, for the “Vanilla CD”, we use a small-scale TinyLlama-1.1b (Zhang et al., 2024a) tuned on the same Alpaca dataset as the amateur model. Figure 7 (a) show the contrastive results. As seen, ROSE outperforms the other counterparts among all benchmarks by a large margin, indicating its superiority.

heading: 5.4 Complementary to Safety-tuned Methods
text: To verify whether ROSE is also beneficial to the other safety-tuned methods, we conduct experiments on two powerful safety-tuned LLMs, i.e., Safety-LLaMA (Bianchi et al., 2023) and LLaMA2-chat (Touvron et al., 2023b). Since these Safety-tuned LLMs themselves have achieved remarkable safety performance, it is not intuitive to combine ROSE with them. Motivated by the red team’s efforts (Liu et al., 2023; Huang et al., 2023), we alternatively use the widely-used jailbreak prompt in Liu et al. (2023) to simulate the extremely unsafely testing in the real world and to better verify the effectiveness of ROSE on these LLMs. In particular, for each test query, we prepend the jailbreak prompt to the query, and use the regular/ROSE decoding methods to obtain the model responses, respectively. For evaluation, we use the ChatGPT to judge the winning rates of v against the regular decoding, where the contrastive results are illustrated in Figure 7 (b). It can be found that ROSE helps the safety-tuned LLMs to defend against the attack and further improves the LLMs’ safety. These results prove that ROSE can be combined with other safety-tuned methods to achieve better performance.

heading: 5.5 Case Studies
text: To have a close look, we perform the case studies by analyzing the qualitative results of Alpaca-7B on two cases. As shown in Figure 8, the regular decoding struggles to refuse the unsafe query in some cases. Conversely, by decreasing the probability of negative output, our ROSE encourages the model to adhere to refusing the harmful query and produce safer outputs. These qualitative analyses prove that ROSE is indeed beneficial to boosting the safety performance of LLMs.
☞ A Note on More Analyses and Discussions
Notably, in addition to the above results and studies, we further conduct more in-depth and systematic analyses and discussions in the Appendix, due to the space limitation. Specifically, we provide 1) an analysis of system prompts in Appendix A.4, 2) a creditability analysis of the ChatGPT-based evaluator in Appendix A.5, 3) a human evaluation in Appendix A.6, and 4) comparison with other training-based safety methods in Appendix A.7. Please refer to the Appendix for more details.

heading: 6 Conclusion
text: In this paper, we propose a simple and effective reverse prompt contrastive decoding (ROSE) to boost the safety of existing instruction-tuned LLMs by directly restricting the behavior of models during inference. ROSE is an efficient plug-and-play method and can be applied to various instruction-tuned LLMs without any additional training. We empirically demonstrate the effectiveness and universality of the ROSE on a series of widely-used safety benchmarks and different instruction-tuned LLMs. Further analyses reveal the underlying mechanism of our method, and investigate when and where to use ROSE will be better. We hope our work could facilitate more research on how to efficiently boost the safety of LLMs.
Limitation
Our work has several potential limitations. First, given the limited computational budget, we only validate our ROSE on the 7B-20B LLMs. It will make our work more convincing if scaling the experiments up to the larger model size, e.g., 70B. On the other hand, although our method is trainingfree and does not require additional training, it introduces some computational budgets during the inference, as the human query will be inputted twice to obtain the positive and negative outputs, respectively. It is meaningful to explore more efficient inference strategies to accelerate text generation, which is in our future work.
Ethics and Reproducibility Statements
Ethics We take ethical considerations very seriously and strictly adhere to the ACL Ethics Policy. This paper proposes a decoding method to boost the LLMs’ safety. To reveal the safety issues of LLMs, we provide some cases that could be offensive or upsetting. However, it should be noted that all pretrained models and evaluation datasets used in this study are publicly available and have been widely adopted by researchers. We do not proactively introduce additional data or models that may cause ethical issues, and we believe that our proposed method will help alleviate ethical issues.
Reproducibility. We will publicly release our code and evaluation data in https://github. com/WHU-ZQH/ROSE to help reproduce the experimental results of this paper.

heading: Acknowledgements
text: We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions. This work was supported in part by the National Key Research and Development Program of China under Grant 2023YFC2705700, in part by the National Natural Science Foundation of China under Grant 623B2076, U23B2048, 62076186 and 62225113, and in part by the Innovative Research Group Project of Hubei Province under Grant 2024AFA017. The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.

heading: A Appendix
text: A.1 Details of Tasks and Datasets In this work, we conduct extensive experiments on several safety tasks. In addition, two widely used general-purpose tasks are also used. Here, we introduce the descriptions of these tasks and datasets
in detail. Firstly, we present the statistics of all datasets in Table 5. Then, each task is described as:
SafetyBench. SafetyBench (Zhang et al., 2023) is a comprehensive benchmark for evaluating the safety of LLMs, which comprises 11,435 diverse multiple choice questions spanning across 7 distinct categories of safety concerns. Notably, SafetyBench also incorporates both Chinese and English data, facilitating the evaluation in both languages.
CValues. CValues (Xu et al., 2023b) is the first Chinese human values evaluation benchmark to measure the alignment ability of LLMs in terms of both safety and responsibility criteria. Specifically, it contains several manually collected adversarial safety prompts across 10 scenarios and induced responsibility prompts from 8 domains by professional experts. Notably, since the safety prompts are not open-resourced due to ethical and legal concerns, we can only evaluate the available multichoice responsibility prompts in this study.
XSTest. XSTest (Röttger et al., 2023) is to identify the eXaggerated Safety behaviors. It comprises 250 safe prompts across 10 prompt types that wellcalibrated models should not refuse to comply with, and 200 unsafe prompts as contrasts that models, for most applications, should refuse.
DangerousQA/HarmfulQA. DangerousQA and HarmfulQA are two safety benchmarks proposed by Bhardwaj and Poria (2023). Specifically, HarmfulQA is a ChatGPT-distilled dataset constructed using the Chain of Utterances (CoU) prompt, which contains 1,960 harmful questions across 10 topics and their sub-topics. As for DangerousQA, Bhardwaj and Poria (2023) query the model with 200 harmful questions collected by Shaikh et al. (2022) using text-davinci-002 across six adjectives—racist, stereotypical, sexist, illegal, toxic, and harmful, and refer to it as DangerousQA.
Do-Not-Answer. Do-Not-Answer (Wang et al., 2023) is an open-source dataset to evaluate LLMs’ safety mechanism at a low cost. The dataset is curated and filtered to consist only of prompts to which responsible language models do not answer. Specifically, it contains 939 harmful questions, covering the malicious uses (243), information hazards (248), discrimination, exclusion, toxicity, hateful, offensive language (176), misinformation harms (155), and human-chatbot interaction harms (117).
MMLU. Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2020) is a popular benchmark designed to measure the multitask accuracy of LLMs, covering 57 tasks including elementary mathematics, US history, computer science, law, and more. Specifically, it ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem-solving ability. The granularity and breadth of the subjects make the benchmark ideal for identifying a model’s blind spots.
AlpacaEval. AlpacaEval is an LLM-based automatic evaluation dataset. It is based on the AlpacaFarm (Dubois et al., 2023) evaluation set, which tests the ability of models to follow general user instructions. AlpacaEval displays a high agreement
rate with ground truth human annotations and is widely used for evaluating general-purpose capability for LLMs.
A.2 Model Details To verify the effectiveness of our ROSE method, we mainly apply it to several publicly-available instruction-tuned LLMs, including Alpaca (Taori et al., 2023) (7B and 13B), Vicuna (Chiang et al., 2023) (7B and 13B) and Chinese-Alpaca (Cui et al., 2023) (7B and 13B). Note that all these LLMs are based on the LLaMA2 (Touvron et al., 2023b) backbone, except the Vicuna-family models5 that based on the LLaMA-v1 (Touvron et al., 2023a). Since there is no official open-sourced Alpaca model available, we re-product the Alpaca models based on the official Alpaca training data using the official codebase. In addition to these instruction-tuned-only models, we also use two aligned (trained with RLHF) LLMs, i.e., InternLM-chat (Team, 2023) (7B and 20B) and Qwen-chat (Bai et al., 2023) (7B and 14B). The detailed model information is listed in Table 10.
A.3 Prompt Details In this part, we show the prompts used in this study, covering the system prompts in ROSE, inference prompts, and evaluation prompts. Firstly, the detailed system prompts are shown in Table 6.
Inference Prompts. For model inference, we use the following prompt for all LLMs, where “[system prompt]” refers to the aforementioned system prompts in Table 6:
[system prompt] Human:
###Instruction: {instruction} ###Input: {input}
Assistant:
Evaluation Prompts. As mentioned in §4, for the evaluation of different tasks, we use the corresponding prompts (provided in the original papers (Röttger et al., 2023; Bhardwaj and Poria, 2023)) to instruct the ChatGPT to evaluate the safety performance. Moreover, we additionally designed two prompts to instruct the ChatGPT to measure the winning rates of our method against the baseline, for the safety and general-purpose tasks, respectively. The detailed evaluation system prompts are shown in Table 11.
5We use this setting to investigate whether our ROSE works well upon different base models.
A.4 More Results
Here, we first report the detailed results of Figure 2 in Table 7, and then provide the analysis of the effect of the positive system prompt. As shown in Table 8, compared to the baseline “-w/o sys. prompt”, the models can achieve much better performance with the help of positive system prompts. This empirically demonstrates the effect of the system prompt and also confirms our statement that the system prompt could perform as an anchor to guide the generation of LLMs.
A.5 ChatGPT v.s. GPT-4
As mentioned in §4, due to the high cost of GPT-4 API, we alternatively use the cheaper ChatGPT as the automatic evaluator to evaluate the safety of LLMs responses. Here, to verify whether ChatGPT is enough to reflect the harmful behavior of LLMs, we conduct a comparative study on ChatGPT and GPT-4. Specifically, taking the responses of Alpaca models on DangerousQA as an example, we use the ChatGPT and GPT-4 to calculate the winning rates of our ROSE against the regular decoding, respectively. As illustrated in Figure 9, both automatic evaluators make similar judgments, i.e., our ROSE performs better than regular decoding on both model sizes. Thus, we believe that ChatGPT is enough to reflect whether the model generates a harmful response and use it as the automatic evaluator in this study.
A.6 Human Evaluation
Additionally, we carry out a human evaluation on the model responses to verify whether the judgments of ChatGPT match human expectations. For each test sample, we compare the model responses generated by our ROSE and regular decoding methods, respectively. Taking the responses of Alpaca-
7B on the DangerousQA benchmark (which only contains 200 test prompts), we report the human evaluation results below. For reference, we also report the evaluation results measured by ChatGPT and GPT-4. Figure 10 illustrates the contrastive results. As seen, the human evaluation (nearly) matches the judgments of ChatGPT/GPT-4, and our ROSE outperforms the regular decoding among all evaluators. These results confirm the credibility of the ChatGPT-based metric used in our paper.
A.7 Comparison with other Training-based Safety Methods.
In this part, we conduct additional experiments to compare our method to other training-based methods, i.e., safety-tuning and RLHF. Taking the LLaMA2 as the base model, we show the contrastive results on CValues in Table 9. As seen, we find that 1) without any training, our ROSE achieves comparable performance to the safetytuning method on the CValues (72.49 v.s. 72.55), and 2) our ROSE can consistently bring further performance gains for the training-based methods, i.e., +1.98 for safety-tuned Safety-LLaMA and +3.74 for RLHF-based LLaMA-chat-7b. These results continue to prove the effectiveness of our method.

heading: Future Work
text: It is meaningful to explore more efficient inference strategies to accelerate text generation, which is in our future work