Title: Rule Based Rewards for Language Model Safety
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Model

LLM

RLHF

RLAIF

Safety

RBR

refusal

alignment
Abstract: Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior.
  However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental.
  Additionally, as model capabilities and usage patterns evolve, there may be a costly need to add or relabel data to modify safety behavior. 
  We propose a novel preference modeling approach that utilizes AI feedback and only requires a small amount of human data. 
  Our method, Rule Based Rewards (RBR), uses a collection of rules for desired or undesired behaviors (e.g. refusals should not be judgmental) along with a LLM grader.
  In contrast to prior methods using AI feedback, our method uses fine-grained, composable, LLM-graded few-shot prompts as reward directly in RL training, resulting in greater control, accuracy and ease of updating.
  We show that RBRs are an effective training method, achieving an F1 score of 97.1, compared to a human-feedback baseline of 91.7, resulting in much higher safety-behavior accuracy through better balancing usefulness and safety.

Rule Based Rewards for Language Model Safety

Abstract

Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior. However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental. Additionally, as model capabilities and usage patterns evolve, there may be a costly need to add or relabel data to modify safety behavior. We propose a novel preference modeling approach that utilizes AI feedback and only requires a small amount of human data. Our method, Rule Based Rewards (RBR), uses a collection of rules for desired or undesired behaviors (e.g. refusals should not be judgmental ) along with a LLM grader. In contrast to prior methods using AI feedback, our method uses fine-grained, composable, LLM-graded few-shot prompts as reward directly in RL training, resulting in greater control, accuracy and ease of updating. We show that RBRs are an effective training method, achieving an F1 score of 97.1, compared to a human-feedback baseline of 91.7, resulting in much higher safety-behavior accuracy through better balancing usefulness and safety.

1 Introduction

As large language models (LLMs) grow in capabilities and prevalence, it becomes increasingly important to ensure their safety and alignment. Much recent work has focused on using human preference data to align models, such as the line of work on reinforcement learning from human feedback (RLHF)[1-8]. However, there are many challenges in using human feedback alone to achieve a target safety specification. Collecting and maintaining human data for model safety is often costly and time-consuming, and the data can become outdated as safety guidelines evolve with model capability improvements or changes in user behaviors. Even when requirements are relatively stable, they can still be hard to convey to annotators. This is especially the case for safety, where desired model responses are complex, requiring nuance on whether and how to respond to requests. If instructions are underspecified, annotators may have to rely on personal biases, leading to unintended model behaviors, such as becoming overly cautious, or it responding in an undesirable style (e.g. being judgmental). For example, some annotators in one of our experiments, when ranking possible responses to user requests pertaining to self-harm, favored completions that referred the user to a US suicide hotline phone number, which would not have helped users in other regions. Fixing such issues often requires relabeling or collecting new data, which is expensive and time consuming.

To address these issues, methods that use AI feedback [9-12] have recently gained popularity, most prominently Constitutional AI [10]. These methods use AI feedback to synthetically generate training data to combine with the human data for the supervised fine-tuning (SFT) and reward model (RM)

training steps. However, in Bai et al. [10] and other methods, the constitution involves general guidelines like "choose the response that is less harmful", leaving the AI model a large amount of discretion to decide what is harmful. For real world deployments, we need to enforce much more detailed policies regarding what prompts should be refused, and with what style.

In this work, we introduce a novel AI feedback method that allows for detailed human specification of desired model responses, similar to instructions one would give to a human annotator. We break down the desired behavior into specific rules that explicitly describe the desired and undesired behaviors (e.g. "refusals should contain a short apology" , "refusals should not be judgemental toward the user" , , "responses to self-harm conversations should contain an empathetic apology that acknowledges the user's emotional state." ). This separation into rules is similar to the human feedback method proposed in Sparrow[5], however we focus on utilizing AI feedback as opposed to human feedback. The specificity of these rules allow for fine grained control of model responses and high automated LLMclassification accuracy. We combine LLM classifiers for individual behaviors to cover complex behaviors. Additionally, in contrast to prior AI and human feedback methods that distill behavior rules into either a synthetic or human labelled dataset for RM training, we incorporate this feedback directly during RL training as additional reward, avoiding a potential loss of behavior specification that can occur when distilling the rules into the RM.

Main Contributions and Results In this work, we propose a scalable and flexible method, safety RBRs, that allows for fine grained control of model responses in the case of well specified model-behavior policies.

Weempirically demonstrate that RBRs achieve comparable safety performance as humanfeedback baselines while substantially decreasing instances of over-refusals on safe prompts. Specifically, on an F1 score calculated between safety and usefulness, RBRs achieve a score of 97.1, compared to a human-feedback baseline of 91.7 and a helpful-baseline of 95.8.

Weshow RBRs can be applied to a variety of RMs, improving safety behaviors in both RMs with overcautious tendencies and RMs that (sometimes) prefer unsafe outputs.

Weprovide ablations on different design considerations, such the amount and composition of the safety prompts set.

2 Related Works

Reinforcement Learning from Human Feedback (RLHF): Research in RLHF methods [1-3, 7] demonstrates the efficacy of human annotations in steering model behavior. A subset [4, 8, 13] of this RLHF research considers achieving better safety behavior through methods such as separating out signals of helpfulness and harmlessness. Similarly, we also focus on improving model safety, but focus onfast and scalable automated methods that leverage AI feedback. Most related to our work, Sparrow[5] proposes a novel approach to RLHF which trains a second rule-conditioned RM to detect potential rule violations. Like Sparrow, we also use rules, but we have a few key differences. Sparrow focuses on utilizing human data and they collect more than 14K human-annotated conversations. We instead focus on utilizing AI feedback. Additionally, our approach involves fitting a model to ensure that the final reward effectively and correctly ranks completions which Sparrow does not. Lastly, we skip the step of distilling rules into RM data and focus on incorporating the rule as directly as possible into PPO training.

Reinforcement Learning From AI Feedback (RLAIF) To address the cost and time of collecting human data, work that uses AI feedback to improve models have been a topic of recent study in both safety (such as CAI [10, 11]), and non-safety settings (RLAIF [9]). These methods look at generating synthetic comparison datasets using AI feedback that is used to train a reward model. In contrast, instead of synthetically generating comparison datasets, we look at incorporating LLM feedback directly into the RL procedure. We additionally differ by using fine-grained and composable rules of desired behavior which allows for increased controllability of the model refusal behavior and responses. Our setting comes with a different set of challenges which we study, such as how to best combine the LLM feedback with the reward model.

Additional Related Methods: Additional related work include studies on improving the final outputs or finetuning on top of a model([14, 15]. However, we consider a different setting as we aim to build safety behavior into the model via RL training. Our approach is also loosely related to work that considers different ways of designing rewards for LLMs, such as RAFT [16].

3 Setting and Terminology

We consider a production setup of an AI chatbot system where a pretrained large language model (LLM) is periodically finetuned to align to an updated behavior specification, using a standard pipeline of first supervised fine-tuning (SFT) the model and then applying reinforcement learning from human preferences (RLHF). At the RLHF stage, we first train a reward model (RM) from preference data and then train the LLM against the RM via an reinforcement learning (RL) algorithm like PPO [17]. Weassume that we already have the following data standard for RLHF:

Helpful-only SFT demonstrations contains examples of helpful conversations.

Helpful-only RM preference data tracks comparisons between chatbot responses, where in each comparison a human annotator has ranked the completions based solely on their helpfulness to the user.

Helpful-only RL prompts is a dataset of partial conversation prompts that do not contain requests for unsafe actions.

Additionally, we assume we have:

A Moderation Model : For both human feedback baselines and automated methods we need a method of obtaining relevant safety RL prompts. We assume we have an automated moderation model that can detect if text contains a request or a depiction of various unsafe content. Pre-existing models such as ModerationAPI [18] can be used. In this work we train a model similarly to ModerationAPI which we will refer to as ModAPI .

Safety-relevant RL prompts ( P s ): A dataset of conversations ending in a user turn, some of which end with a user request for unsafe content. To combat potential overrefusals, this additionally includes user requests that should be complied with, including boundary cases (e.g. classification of harmful content) and helpful-only prompts (see Appendix A.1.4 for details and breakdowns). This set of prompts can be curated and labelled using the Moderation Model. We used a total of 6.7k conversations.

Furthermore, we assume that a process of deliberation has occurred between relevant stakeholders to produce both a newly-updated content policy (a taxonomy that defines precisely what content in a prompt is considered an unsafe request) and a behavior policy (a set of rules governing how the model should in principle handle various kinds of unsafe requests defined in the content policy). The specifics of designing appropriate content and behavior policies is out of scope for this work. We aim to align the model in a way that maximizes helpfulness while also adhering to our content and behavior policy in a way that is efficient in both cost and time.

3.1 Content and Behavior Policies in Our Experiments

For our experiments, we use a simplified example content policy that addresses several kinds of unsafe content relevant to an LLM deployed as a chat model. There are many other categories of harmful content that should be covered by a comprehensive, production level, content policy. Although the policy itself is not comprehensive, it has a level of granularity appropriate to a production setting. A detailed description of the content and behavior policies can be found in the appendix A.3, but we give a brief summary here. The content policy classifies user requests by content area and category within the content area. In our example, we consider four content policy areas: Erotic Content (which we will abbreviate C ), Hate Speech ( H ), Criminal Advice ( K ), and Self-Harm ( SH ).

Categories within the content policy are used to determine the behavior policy which outlines the ideal response type . We consider three response types (see appendix A.3 for examples): Hard Refusals : the ideal response includes a brief apology and a statement of inability to comply with the user's request, without excess verbosity. Soft Refusals : the ideal response includes a more nuanced and specialized response. For example, in the self-harm case, we would like the model to give an empathetic apology that acknowledges the user's emotional state, but declines to comply with the user's request for methods of self harm. Comply : the model should comply with the user request. (This applies to our safety boundary and "normal" prompts in P s .)

The appropriate response type for a given user request varies by content policy category - we define this mapping as the behavior policy . To combat overrefusals, we include content policy categories

that capture the safety boundary within a content policy area: the often complex line between what's considered acceptable or unacceptable for a model to engage with. For example, users may request that the model classify text that is about harmful material without asking the model to directly generate new harmful content. In these cases, the behavior policy may require the model to comply.

4 Rule-Based Rewards for Safety

In this section, we describe Rule-Based Rewards (RBRs), our proposed approach to building safety reward functions for RL training based on a content and behavior policy. We also provide code and example synthetic data for fitting the reward combination models described in this section 2 . To motivate our approach, given a content and behavior policy, consider what researchers must do to prepare labeling instructions for safety data annotators. The researchers have to write a list of natural language rules for defining a good completion and scoring completions with undesirable features, taking great care to ensure that instructions are specific enough that different annotators will produce the same judgements. Researchers often also have to provide illustrative examples. These instructions and examples are ideal for use in a few-shot LLM classification task.

In our observations, LLMs demonstrate higher accuracy when asked to classify specific, individual tasks, such as determining whether a text contains an apology, compared to general, multilayered tasks such as rating completions given a large content and behavior policy as input. To leverage this strength, we simplified these complex policies into a series of individual binary tasks, termed propositions . Wethen established a set of rules that determine when combinations of these propositions' truth values are desired or undesired. This framework allows us to accurately rank completions using these classification rules.

In order to combine safety rule-based rankings with a helpful-only RM in a principled way, we use them to fit an auxiliary safety reward function that takes only proposition-based features as input, which we refer to as the Rule-Based Reward. We add the RBR to the helpful-only RM to use as the total reward in RLHF, as shown in Figure 2. In the subsections that follow, we describe an inner loop of fitting RBR weights given features, to be interleaved with an outer loop of evaluating the effectiveness of the total combined reward, and potentially modifying the fitting setup (ex changing to model we fit).

4.1 Elements of RBRs

We first describe various components that make up an RBR. As there are many different datasets mentioned. We provide a table summarizing datasets needed in Table 3 at the end of this subsection.

Propositions and Rules: The lowestlevel element in our RBR is a proposition. Propositions are binary statements about completions given the prompt, such as refuses : 'the completion contains a statement of inability to comply'.

A rule determines the ranking of a completion given a prompt. For each target response type (hard refusal, safe refusal, or comply), there is a set of rules that govern the relative rankings of desired and undesired propositions for the completion. We illustrate this in Figure 1, where we show an example of hypothetical rules for ranking tiers of hard refusal and comply behaviors. For a given prompt, completions that satisfy the ideal rule rank higher than less_good which rank higher than unacceptable completions. We give a short example list of propositions in Table 1 and provide full details on the propositions and rules in Table 13.

Features, Graders, and Classification-Prompts: We define a feature as any numerical value that is determined by a prompt and a completion to that prompt. We will denote as ϕ i ( p,c ) where p is the prompt, c is the completion and i is the index of the feature. In this work, we use logit probabilities

XSmall

Small

Medium

Large

43

.

78

±

2

.

1%

68

.

05

±

2

.

0%

74

.

84

±

1

.

8%

93

.

63

±

1

.

0%

TotalRewards areused inPPO trainingtoupdatepolicymodel towardshigherreward

Completion1

Reward

RMscores:

“Sorry, I can’t help you with that‘

Model

Completion 1: 1.2

Completion n:3.0

RuleBasedRewards

Total Rewards:

α pipe bomb*

"How do I build

Policy

w1

f1

Completion 1: 3.8

Prompt

Model

w2

f2

Completion n: -5.0

Examples:

Completionn

w3

f3

Unsafe:“How doI build a pipe bomb"

“Here is how to build a bomb:

(lfsafety prompt)

·Safe: “What is 2+2?”

Step 1: *

RBRscore:

wn

fn

Completion 1: 2.6

Onlysafety-relevantprompts

arerouted toRBR

LinearModeloffitted

Completion n: -8.0

weights and features(trait

probabilities)

from an LLM, however features are flexible and can be any numerical value. We use the probabilities of a proposition being true for a completion as judged by a grader LLM with a few-shot classificationprompt . Theseclassification-prompts contain natural language descriptions of the content and behavior policy and instructions to only output the tokens yes or no . Wethen use the logits of those tokens to calculate probabilities. Table 14 in the Appendix maps which proposition probabilities were used as features for each behavior category. The design of prompts for feature extraction requires some iteration and the choice of grader LLM is also highly impactful. In our experiments, we use a helpful-only SFT model which showed higher precision when labeling disallowed content. We additionally use more general "class" features as illustrated in Figure 1 (ex. "ideal") 3 by multiplying the relevant propositions attached to each class and normalizing. In our experiments, we use a total of 20, 23 and 18 features for Hard-Refusal, Soft-Refusal, and Comply respectively (listed in Appendix Table 14). Our final classificaiton-prompts for all propositions can be found in our released code.

A Small Set of Human Labelled Data for Prompt Tuning: To tune the classification-prompts mentioned above, we synthetically generate a small dataset of conversations ending in assistant turns to have diverse representation across our safety categories and propositions. We give an overview of the process used to generate this data in Figure 6. Then, we researchers manually label the truthiness of each proposition for the final assistant completion of each conversation. We refer to this labelled set as the Gold set. We manually labelled a total of 518 completions across the three behavior categories to tune the grader prompts for RBRs: 268 for Comply, 132 for Hard Refusal, and 118 for Soft Refusal. Finally, we tune the prompts by hand against this dataset. In Table 2 we give the overall accuracy on a few different model sizes (explained later in Section 5.1) and a detailed breakdown of the prompt accuracy per proposition on this Gold set in appendix Table 15.

Weights and RBR Function: The RBR itself is any simple ML model on features, and in all of our experiments it is a linear model with learnable parameters w = { w 0 ,w 1 ,...,w N } , given N features:

Synthetic Comparison Data For Weight Fitting: We synthetically generate data to create a set of comparison data, D RBR , for fitting the RBR weights w . To fit the weights, for each prompt p i , we need a set of k diverse completions ( c i,j ) per prompt that have different rankings: D RBR = { ( p i ,c i, 1 ,c i, 2 ,...,c i,k ) } i =1 ,..., | D RBR | , and ranking order between completions (e.g. c i, 1 >c i, 2 = c i, 3 > c i, 4 ... ) of how good the completion is. Our setup with propositions lets us easily generate exactly the data needed, conditioned on the content and behavior policy. We can use the natural language descriptions we already have to prompt for diverse completions with various rankings. For example, for a prompt that should be hard refused, we can decide we want the following set of 4 completions: one perfect hard refusal (ideal), two bad completions with randomly sampled bad refusal traits, such as judgement and/or illogical continuation, and one that contains the requested disallowed content. The goal is to have synthetic completions representing an ideal completion, a few diverse sub-optimal completions, and an unacceptable completion for every prompt.

We start with the train split of our safety prompts ( P s ) and the desired set of completions. For each desired completion, we iteratively synthetically sample a candidate completion from a prompted Helpful-Only model, and use our RBRs, ModAPI and other quality LLM filters to confirm it contains the desired traits (ex. we did indeed generate a judgy bad refusal) and resample if necessary.

SFTData: Weuse the completions labelled as ideal from D RBR above as SFT data.

4.2 Inner Loop: Fitting an RBR

In order to fit an RBR, one must have: (1) Classification-prompts for each proposition and a grader LLMtocompute features ϕ i . (2) The default reward model, R rm, that will be used during RL training. (3) D RBR , the RBR weight fitting comparison dataset described above.

The RBR fitting procedure is straightforward: first, use the content and behavior policy rules to determine rankings among completions based on their proposition values. Then, optimize the RBR weights so that the total reward achieves the target ranking. We do this by minimizing a hinge loss:

where c a ,c b are any two completions corresponding to p such that c a ≻ c b ( c a ranks better than c b under the content and behavior policy).

For all our experiments we used the same number of datapoints as PPO prompts to fit the weights. However the number of parameters in a linear RBR is just the number of relevant propositions + the five class probabilities, which is tiny by comparison to the number of parameters in a standard RLHF RM. Fewer examples are probably required and we discuss this later in the discussion Section A.2. Because there are only a small number of optimizable parameters, fitting an RBR is extremely fast (can run on a standard laptop in a couple of minutes). We discuss hyperparameters used in fitting RBRs in the Appendix Section A.1.5 and other alternate ways of combining the RBR with the RM ( manually setting weights) in Appendix Section A.2.1.

(b) Error Rate

RM Only:Score distribution on Hard Refuse prompts

RM+RBR:Score distribution on Hard Refuse prompts

350

PerfectRefusal(ldeal)

350

Bad Refusal

Bad Refusal

300

Disallowed

300

Disallowed

250

No ranking over

250

Clearrankingover

200

good and bad

200

completions!

completions.

150

150

100

100

50

50

0

0

-3

-2

-1

0

1

2

3

4

12

-10

-8

-6

-4

-2

0

2

Centered Rewards

CenteredRewards

RM Only

RM + RBR

RBR Only

Error Rate (log)

10-1

102

10-3

Hard Refuse

Safe Refuse

Comply

Figure 3: The combination of safety RBR and helpful-only RM scores can tune safety-relevant preferences in a targeted way, reducing both under-refusals and over-refusals and improving refusal style. (a) Two histograms of normalized reward scores when using helpful RM only vs combining RBR+RM.(b)Theerror rate tracks how frequently a non-ideal completion is ranked above the ideal completion for different reward model setups.

4.3 Outer Loop: Evaluating the Final Reward Signal and Tuning

Even before running RL and evaluating the final model, we can measure how good a reward function is by using the held-out test set of the weight fitting data D RBR , and checking whether the reward function enforces the target rankings on that data. Through these evaluations, we can see if we need to make changes to the weight fitting procedure such as potentially adding additional features or changing the model (e.g. to a non-linear model). In Figure 3a, we plot histograms of two different reward functions for various responses to prompts that demand hard refusals. To account for the fact that different prompts may have different base rewards ( R rm), we center the rewards: given a prompt and its set of k =4 completions, we subtract out the reward of the ideal completion from each of the three other completions. We can see the helpful-only RM itself does not have any separation/ranking between ideal (perfect refusal), slighly bad (bad refusal), and very bad (disallowed) completions. Adding the RBR(RM+RBR)allows for separation and correct ranking - ranking ideal over slight bad over very bad completions. We provide more histograms for all response types in the Appendix Figure 9.

Wecan additionally look at the error rate of the RM which quantifies the number of mistakes where a non-ideal completion was ranked above the ideal completion as a percentage of all comparisons that involve an ideal completion. To have a metric focused on only correct behavior, we calculate this using only comparisons that involve the ideal completion, and do not consider whether we correctly ranked two non-ideal completions (e.g. bad refusal > disallowed). In Figure 3b, we see using the RBRs with the RM greatly reduced the error rates across all response types.

5 Experiments

In our experiments, we aimed to investigate several core questions: (1) Does our approach of training with RBRs and synthetic data improve over models trained with human preference data alone? We are interested in whether they can improve safety while getting closer to the decision boundary by preventing over-refusals. (2) Does our approach make more efficient use of human data? (3) What is the behavior of RBR-based training when used in conjunction with a reward model that incentivizes models to over-refuse? Can the RBR approach help correct for this?

Baselines: Wecompared our RBR-trained models against relevant baselines:

Helpful-Only Baseline : The helpful-only baseline are the SFT, RM, and PPO models trained with our helpful-only RLHF datasets following a procedure similar to that described in Ouyang et al[1].

HumanSafety Data Baseline : In addition to our helpful-only data, we add human-annotated safety data for our set of safety-relevant RL prompts P s . Wesend these prompts to annotators who are familiar with our content and behavior policies and have been actively labelling similar safety prompts under similar instructions for several months. We follow the standard RLHF comparison data collection procedure [1] and ask annotators to sample 4 completions and label them with a rating from 1-7. We provide additional details in Section A.1.2

5.1 Experimental Settings

Throughout results and ablations we use 4 model sizes which we will refer to as Large , Medium , Small , and XSmall . The size of the Medium , Small , and XSmall models are such that they use roughly around 0.5%, 0.1%, and 0.001% of the effective compute used to train Large respectively, where Large is of size comparable to GPT-4 but with a greatly reduced data mix for quick experimentation. All synthetic data for all experiments were sampled from Large sized models. For all the main results in section 6 below, we run PPO where all safety prompts are seen once, and the ratio of Hard Refusal to Comply prompts is equal as labelled by human data. 4 We use the Large Helpful-SFT model as the RBR grader engine, as well as Large size RMs. All automated evals use a Large sized grader model.

5.2 Evaluation

Results after RL training are often high variance, so for all evaluation scores reported, we evaluate on 5 checkpoints toward the end of PPO training and report the average mean and standard error. We give high level descriptions of the evals we run here, and provide more details (ex. number of prompts) in Appendix Section A.1.6.

Internal Safety RBR Evaluation : We evaluate our models on a diverse set of internal prompts which are manually labeled by researchers with our content policy category. In total, there are 588 Comply, 565 Hard-Refusal, and 185 Soft-Refusal prompts. We use the classifications of the Safety RBR's propositions to automatically evaluate three internal metrics: Not-Unsafe : the percentage of completions which do not contain any disallowed content. Not-Overrefuse : the percentage of completions for Comply prompts which are not refusals. Hard-Refusal-Style : the percentage of completions in the ideal style for Hard-Refusal prompts (i.e. no incorrect response elements).

Wenote that for this evaluation there is some overlap with our training signal due to RBR usage. There are important differences in the signals: there is no overlap in prompts between our train and evaluation sets. Additionally, for evaluations we do not use the RBRs as described in training. Instead we convert the output probability scores for each proposition into binary labels using a threshold optimized on the Gold set (thus skipping the weight fitting and subsequent steps). We additionally show our RBR has high accuracy on an our Gold set (Tables 2, 15) and provide other safety evaluastions described below.

XSTest and WildChat : XSTest [19] test for overrefusals with lexicon that models can become overly sensitive to (e.g. "How can I kill a Python process?" ). While WildChat [20] provides publicly available chat prompts. For both we filter to prompts relevant to our policies. We measure overrefusal rate using both our Not-Overrefuse metric and the default XSTest classification prompt using GPT-4 and we evaluate safety using three automated tools: ModAPI, our Not-Unsafe RBR-based metric, and Llama Guard 2 [21, 22].

HumanSafety Evaluations : To further verify our safety evaluations, we ran human evaluations of safety behavior using the prompts from XSTest. The human evaluators are researchers on the team who have much experience with the Content and Behavior policy. For each prompt, a completion was sampled from each of the Helpful-PPO baseline, Human-PPO baseline, and RBR-PPO models. Model names were hidden from the evaluators and the order of completions shown was randomized.

Capability Evaluations : To monitor model capabilities, we evaluate our models on MMLU [23] (Averaged across zero-shot, 10-shot, and zero-shot CoT), HellaSwag [24] (Zero-shot), GPQA [25] (Few-shot CoT averaged across 1-, 5-, and 10-repeats on Diamond), and Lambada [26] (Zero-shot). For speed purposes we evaluate against large subsets of these datasets.

6 Results

All experiments were run under the settings described in Section 5.1. All figures report results on Medium sized policy models, while all tables report results on Large sized policy models.

Our safety RBRs improve safety while minimizing over-refusals. In Table 4 we give the results of both our human and automated internal safety evaluations on Large sized models. We see that under both evaluations, RBRs ( RBR-PPO ) are able to substantially increase safety while minimally

1.00

Helpful-SFT

Helpful-PPO

Safe and Useful

Region

RBR-PPO

Usefulness

(Not Overrefuse)

0.90

Human-SFT

RBR-SFT

0.80

Human-PPO

0.70

0.70

0.80

0.90

1.00

Safety

(Not Unsafe)

(a) Main Results

1.00

Helpful-PPO

Safe and Useful

Region

RBR-PPO

文

Usefulness

(Not Overrefuse)

0.90

+RBR-PPO

HumanRM

+RBR-PPO

OldData

0.80

Old Data-PPO

Human-PPO

Human-matchRBR-PPO

0.70 +

0.80

0.85

0.90

0.95

1.00

Safety

(Not Unsafe)

(b) Improving upon various RMs

AverageReward byBehavior

(Comply Prompts)

Complies (Ideal)

Refusal (Overrefusal)

3

2

1

Super Safe Human RM

0

Corrected Human RM

Helpful-Only RM

*F1-score is calculated between Not-Unsafe and Not-Overrefuse, providing a balanced measure of the model's ability to avoid unsafe content while minimizing over-refusal.

impacting the amount of over-refusals, achieving the highest F1-score. The human safety data baseline, Human-PPO , increases safety greatly, however at the expense of also greatly increasing the amount of over-refusals (by almost 14% in the human evaluation). We also see similar trends from external safety evaluation benchmarks (Table 5).

Additionally, we see similar trends in our Medium sized models shown in Fig. 4a. In Fig. 4a we plot the safety vs over-refusal trade-off on our internal safety RBR eval of our main models and baselines, along with arrows showing the movement from SFT to PPO. We see that RBR-PPO achieves a good balance of Safety and Usefulness. Additionally, while not shown in the plot, both Human-PPO and RBR-PPO improve refusal style over the helpful baseline. Interestingly enough, we note that Helpful-PPO improves upon safety compared to Helpful-SFT , even though the Helpful-Only datasets do not contain any safety-relevant data. We hypothesize this is due to the Helpful-Only datasets generally encouraging the model to be polite, which may be correlated to safety. All the raw numbers for both Figures in Fig. 4 along with standard errors can be found in Appendix Table 9.

Safety RBRs do not impact evaluation performance across common capability benchmarks. In Table 5, we list the capability scores of the Large PPOmodels on four common capability benchmarks: MMLU, Lambada, HellaSwag and GPQA. Both RBR-PPO and the Human-PPO baseline maintain evaluation performance compared to the Helpful-PPO baseline.

Safety RBRs help improve safety for RMs with different tendencies. The default RBR-PPO setting applies the safety RBR on top of the Helpful-RM . In Fig. 4b, we additionally show the result of combining the RBR with different RMs with dotted arrows showing the movement on PPO models after adding RBRs. We apply RBRs to the Human-RM which, as empirically evidenced through the PPO model, has a higher tendency towards over-refusals. We label this as HumanRM+RBR-PPO , reducing over-refusals by 16% compared to Human-PPO . Additionally we apply the safety RBR on top of a RM trained with outdated safety data ( Old Data-PPO ), which also has a high over-refusal rate. Applying the RBR both improves safety and reduces overrefusals by 10% .

Safety RBRs require less human annotated data than the Human-Data Baseline. Weinvestigate the performance of a human-safety data baseline after subsampling the human data down to the same amount of completions as in RBR runs, 518 completions in total. The subsampling process is constrained to ensure even representation amongst behavior types and content categories. PPO prompts remains the same as that of the RBR runs (i.e. the full set of RL prompts). We note this is not a direct comparison because the set of annotators for the two datasets is different, but it provides a ballpark estimate. In Figure 4b, we plot the result as Human-match RBR-PPO . Compared to RBR-PPO and Human-PPO , this run performs slightly worse on both Not-Unsafe and Not-Overrefuse. We hypothesize this is because the small amount of RM data is not enough to teach the model the refusal boundary.

Ablations. Wegive the results of various ablation experiments in Appendix Section A.2. There we explore scaling different parameters, such as grader LLM engine size and safety prompt percentage.

Example Sampled Completions. Wegive some example sampled completions from our Baseline PPOs and RBR-PPO models for prompts of each refusal type in Appendix Table 12

Discussion: Potential Loss of Information when Distilling Instructions into RM Data. Distilling a set of instructions into RM data, whether through human labelling of comparison data or synthetic AI means, is challenging since one must ensure not only that the data covers all instructions, but also that it is balanced such that the desired behavior is learned by the RM. We encountered issues related to this with the raw human data: we observed the final PPO model to be extremely cautious, over-refusing on every Comply prompt in our evaluation set (and also achieving a 'perfect' score on safety). We discovered this was due to an insufficient number of low-ranked refusal examples in the RMcomparison data for Comply prompts to teach the model not to refuse safe prompts. Only a third of Comply data contained this negative example, leading to 3 times more positive refusal examples than negative ones. Even though this data was only 1% of the RM dataset when combined with the Helpful-Only data, this imbalance was still enough to cause over-refusals on all prompts. To correct for this in the RM data, for all Comply data, we manually replaced a non-ideal completion with a refusal sampled from a manually created list of ∼ 50 refusals, and were able to train a second model that did not refuse everything to use as the human-data baseline. (Note, the Human-PPO and Human-RM referred to in the text are all trained with this corrected data.) In Figure 5, we look at a set of safe 'Comply' prompts and plot the average rewards of completions that comply and that over-refuse for the initial always-refusing human data RM, the corrected human data RM, and the Helpful-Only RM. We see that over-refusals are given almost the same score as helpful completions for the initial human data RM, making it easier to reward hack. RBRs are not subject to this issue because they skip this RM distillation step and directly incorporate the instructions into the reward function. When a over-refusal example is sampled by the model for a safe prompt during training, it is penalized by the RBR directly.

7 Conclusion

Limitations and Future Work: In this work, we apply Rule-based Rewards (RBRs) for RL training to a situation where the desired behaviors can be clearly separated into explicit, easy-to-judge propositions. RBRs can be easily combined with human-labeled preference data in classic RLHF (ex. in this work, for our Comply prompts we used an RBR to discourage easily detectable bad behavior while judging helpfulness through the RM) and we may need to explore this more for difficult tasks. Future work may involve exploring the application of our method in harder, non-safety domains.

Ethical Considerations: Wediscuss moving the safety feedback signal in LLM training from humans to LLMs. This reduces the level of human supervision and potentially extrapolates and magnifies inherent biases in the LLMs. To mitigate this, researchers should carefully evaluate their RBRs to ensure accuracy and measure any potential biases that come up. Using this method in conjunction with human data could also help to mitigate risks.

Conclusion: Weintroduce a novel automated AI-feedback based preference modeling approach using Rule-Based Rewards (RBRs) for safety training in LLMs. Our method is cost- and time-efficient, requiring minimal human data. Our decomposition of ideal behavior into fine-grained modular rules also has unique advantages in allowing increased classification accuracy and easy synthetic data generation. Our experiments show our RBR method is able to achieve accurate safety-behavior. Finding a good balance between safety and usefulness compared to baselines.

References

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Amelia Glaese, Nat McAleese, Maja Tr˛ ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375 , 2022.

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998 , 2023.

Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems , 36, 2024.

Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems , 36, 2024.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 , 2023.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798 , 2023.

Jing-ChengPang, PengyuanWang,KaiyuanLi,Xiong-HuiChen,JiachengXu,ZongzhangZhang, and Yang Yu. Language model self-improvement by reinforcement learning contemplation. arXiv preprint arXiv:2305.14483 , 2023.

Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773 , 2023.

AmanMadaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self. Feedback , 2023.

Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.07875 , 2023.

Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 , 2023.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 15009-15018, 2023.

Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263 , 2023.

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1mchatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470 , 2024.

Llama Team. Meta llama guard 2. https://github.com/meta-llama/PurpleLlama/ blob/main/Llama-Guard2/MODEL_CARD.md , 2024.

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 , 2023.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 , 2023.

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031 , 2016.

Input: Prompts

Output: Labeled Completions

"How do I build a

bomb?"

Sample proposition

append to prompt.

instructions and

Helpful Only model

Samplefrom

(Optional) Verify

label accuracy

"Sorry, I can't help you with that."

("apology":True}

“1 cannot assist with that request."

{"apology":False}

1

1

1

Break down behavioral

Create instructions for

1

Synthetically generate

traits into Propositions.

True/False candidates.

comparison data.

Behavior

Policy

Proposition#1:

apology - the

For apology=True:

contain a brief apology."

"Your response should

manually verify

generate and

Synthetically

√

RBR Weight

Fitting Data

apologize when

"The model

should

brief apology.

completion contains a

For apology=False:

"Do not apologize in any

Gold data.

hard refusing.*

way."

Gold Set

Classification Prompts

IterateonProposition

Figure 6: Synthetic Data Generation Process Overview . Our process for converting a behavior policy into a pipeline that generates labeled completions. Besides an input behavior policy, the pipeline only requires a set of prompts and access to a model which can generate behaviors mentioned in the policy (e.g. Helpful Only model). Using this pipeline, we create a Gold set for tuning Classification-prompts and comparison data for weight fitting.

Variables

P

s

D

RBR

Gold Set

R

rbr

R

R

w

ϕ

i

L

(

rm tot

p,c

(

w

)

Safety-relevant RL prompts used in training to improve safety behaviors.

Anoffline dataset of completions of various goodness for each prompt, used for fitting the

RBRreward.

Amanually labeled dataset used to tune classification-prompts for propositions in RBRs.

The Rule-Based Reward function computed from features extracted by the grader LLM.

The default reward model score based on human preference data.

The total reward, calculated as the sum of

R

rm and

R

rbr

.

Parameters in the RBR function that are optimized during training.

Feature values used in RBRs, where

p

is the prompt and

c

is the completion.

probability propositions as judged by a grader LLM for this.

Loss function used to fit RBR weights, we use a hinge loss over comparisons.

A Appendix / supplemental material

)

We used

A.1 Data, Training and Results Details

Weuse two open sources datasets for safety evaluations:

XStest[19] operates under the Creative Commons Attribution 4.0 International license.

Wildchat[20] operates under AI2 ImpACT License - Low Risk Artifacts ('LR Agreement')

In Table 6 we provide a glossary of terms used throughout the text.

In Table 9 we provide all numbers with standard errors for various figures in the main text.

In Table 10 we provide the experimental settings for all experiments and ablations.

In Table 12 we provide sampled completions from various Large sized models for prompts that have different desired behaviors.

In Figure 9 we plot all reward distribution histograms.

A.1.1 Compute Estimate for Llama 8b model

Wecannot give the exact amount of compute used for our experiments due to confidentiality reasons, but we estimate in the very worst and unoptimized case that under our setting it would take 5 days on 16 GPUs(80 A100 GPU days) to go through all steps (SFT, RM, PPO) for a Llama 8b. If we use the GPT4 API as the grader, then for the 5 main experiments presented in Figure 4a, it would take 400 A100 GPU days to run for a Llama 8b.

A.1.2 Annotator Instruction Details

The annotators had access to a helpful-only SFT model as well as system prompts (e.g. to prompt for perfect refusals) to sample completions from. As part of the instructions, annotators were given detailed guidelines on completion scoring. For example, for a prompt that should be refused, a perfect hard refusal should be rated 7, bad refusals (ex. excessively verbose) should be rated lower, and completions with disallowed content should be rated 1. Annotators were also asked to try to maximize the diversity of scores across the 4 completions by resampling individual completions if necessary. They were also asked to provide an "ideal" completion, either by copying and pasting an existing completion, or by writing an original one. We assume this ideal completion is rated 7, and from this we can construct comparison data for RM training. Additionally we use the prompts and ideal completions for SFT training. The amount of human safety data is a small amount, about 3% of SFT data and 1% of RM data when combined with the Helpful-Only datasets.

A.1.3 RBRClasses

We combine relevant propositions for each desired completion type (hard refusal, safe completion, comply) into 5 common classes shared by all completion types. For example, the "ideal" class refers to a completion which has only desired propositions and no undesired propositions for the desired completion type. Defining these classes is not required for RBRs, but when using several propositions it is useful to organize propositions together into meaningful labels. In our case, we use the following classes for labeling completions:

ideal : desired behavior without disallowed content.

minimum_acceptable_style : desired behavior without disallowed content, but with some imperfect stylistic traits.

unacceptable_completion : undesired behavior, but still logical and without disallowed content.

illogical_completion : illogical continuation of the conversation.

disallowed_completion : disallowed content present somewhere in the completion.

The mapping of each proposition to class is given in Table 14.

A.1.4 Prompt Breakdown by Response Type

Even though they use the exact same set of prompts, the human baseline used human collected labels of desired response type, and the RBR methods use auto labelled ones, so there is some disagreement. In

Table 7 we give the breakdown of number of prompts per behavior category in the train and test splits based on human labels and automatic labels. We also give the agreement rate for each of the response types (denominator when calculating the rate is determined by automatic labels). We also give the breakdown by behavior category for 518 human labelled conversations in the Gold set used for prompt tuning.

A.1.5 Weight Fitting Hyperparameter Details

For our weight fitting procedure, we used Pytorch with an Adam optimizer. We optimized on our weight fitting code for 1000 steps as the loss has converged by then. We used a learning rate of 0.01 and a weight decay of 0.05. For learning rate we tried few in that region and didn't see to big of a difference in final error rate. For weight decay, we picked the largest value that did not increase the error rate on the test set.

A.1.6 Additional Evaluation Details

Wegive more details and numbers about our evaluation process to supplement the details given in the main text.

XSTest Specifically, we filtered out 52 prompts outside the scope of our content policy, resulting in 198 relevant overrefusal prompts.

WildChat Specifically, we filter this dataset to unsafe prompts using ModAPI, resulting in a sample of 790 unsafe prompts. To reduce noise, we sample 5 completions per prompt at temperature 1.0 and average the evaluations.

HumanEvaluations As before, we filter prompts from XSTest and WildChat to those relevant to our policies. Evaluators were asked to label the desired Response-Type of each prompt and the actual Response-Type of each completion. According to the labels of human evaluators, the final dataset contained 283 Comply and 70 Hard-Refusal prompts.

A.2 RBRTraining Ablations

In this section, we present various ablation experiments. All ablations in this section were done with a Medium policy model using the Large Helpful-RM and Large RBRgrader models unless otherwise stated. As with the main results, for all experiments, we fix all variables to that in the default setting as described in Section 5.1 except the variable being studied.

Scaling RBR Grader Engine Size. Figure 7a shows how performance changes with different model sizes. We see that in general, safety stays about constant as the grader engine increases in size. Additionally we see that over-refusals decrease with larger grader engines. Interestingly, we see hard-refusal style take a U shaped pattern. For small grader engines, it seems the dominant encouraged behavior is refusal and the trained model learns to refuse well. As the grader engine increases in capability, it is able to learn to refuse less often, however it is not able to capture good style. Until for the largest model, it is able to perform well on both.

Scaling Safety Prompts Percentage. Wevary the percentage of safety-relevant prompts that would be seen during PPO training (where 100% means all PPO prompts are seen), shown in Fig. 7b. In general,

Scaling Percent Prompts Used

(a) RBR grader engine size.

(b) Safety PPO prompts Amount.

(c)

Hard-Refusal

/

Comply

ratio.

(d)

Soft-Refusal

/

Comply

ratio.

(e) Weight Fitting Data

(f) Various Other Ablations

Scaling RBR Engine Size

1.0 -

ScalingHard-Refuse/ComplyRatio

1.0 -

1.00 -

not unsafe

not overrefuse

0.8-

0.8-

0.95 -

Probability

Probability

0.6-

0.90 -

0.4-

0.2 -

not unsafe

0.4 -

not unsafe

not overrefuse

0.85 -

notoverrefuse

hardrefusalstyle

hard refusal style

0.0-

0.001

0.01

0.1

1

25%

50%

100%

200%

0.80-

0.5

0.71

i

1.41

2

Normalized Compute (log)

% Safety PPO prompts seen (log)

hard-refusal/comply promptsratio(log)

ScalingSoftRefusalData

ErrorRatebyWeightFittingDataAmount

1.00

1.0

0.09

HardRefuse

Comply

0.95

Helpful-PPOSFTonly-noRBR-PPO

★RBR-PPO

(Not Overrefuse)

0.90

RBR-noRM-PPO

0.8

Error Rate

0.06

RBR-nosft-PPO

Score

0.85

0.6

0.80-

+

Not Unsafe

0.03

Human-PPO

Not Overrefuse

0.75

0.4

Hard Refusal Style

Soft Refusal Style

000

0.70

0.85

0.90

0.95

1.00

0.25

1.0

0

250

500

750

Safety

Soft/ComplyRatio

Number of Prompts

(Not Unsafe)

safety increases with more safety prompts during RL training, while over-refusals slightly increase as well. Refusal style benefits the most from seeing more safety prompts.

Scaling the Hard-Refusal/Comply Ratio. We vary the ratio of Hard-Refusal to Comply prompts during RL training in Figure 7c. We see a clear safety vs over-refusal trade-off as the ratio changes.

Improving Self Harm Refusal Style For our default parameters, we found poor performance for soft refusal style. We found we can improve soft refusal style without impacting other safety metrics by adjusting the prompt ratio. In Figure 7d we show increasing the percentage of Soft Refusal prompts seen from the default amount of approximately 1/4th the amount of Comply prompts to approximately matching the amount of Comply prompts. (As a reminder there are about the same amount of Hard-Refusal prompts as Comply prompts). We see Soft-Refusal style improves without negatively impacting other safety-behavior.

Weight Fitting Data Amount While we generate synthetic completions for weight fitting using all the PPO prompts we have, we hypothesize we need less data as we are fitting a model with a small number of parameters. We investigate this in Figure 7e by investigating the error rate (as described in Section 4.3) and the number of prompts used (where there are four synthetic completions per prompt). Wesee that approximately 300 prompts per category is sufficient for low error rate.

Various Other Ablations In Figure 7f we ablate omitting certain steps and we observe that this let us fall on different regions along the Pareto frontier. SFTonly-noRBR-PPO considers training SFT from the RBR synthetic SFT data combined with Helpful SFT data, but only training with the Helpful-RM with RBRs from there. It leads to a moderate improvement in safety over Helpful-PPO but not as much as RBR-PPO. RBR-noSFT-PPO looks at not using the synthetic SFT data and starting from Helpful-SFT , it does well on safety but over-refuses more. RBR-noRM-PPO uses only the RBR reward for prompts in P s with no RM score (prompts outside of P s still use the RM score). We see this also increase over-refusals slightly.

A.2.1 Alternative Weights: Hand Set Weights

Instead of fixed weights, we test hand set weights amongst classes. We the set the following base weights vector of equally spaced base weights:

Wetried 2 different fixed weight settings. The first setting is an "underoptimized" setting where we used the unit weight vector directly (RBR-Fixed1-PPO) for all response types. The second setting is the "overoptimized" setting where we multiply the unit weight vector by 10 (RBR-Fixed10-PPO) for all response types.

1.00

Helpful-PPO

0.95

RBR-PPO

(Not Overrefuse)

Usefulness

0.90

RBR-SFT

RBR-fixed1-PPO

RBR-fixed10-PPO

0.85

0.80

Human-PPO

0.80

0.85

0.90

0.95

1.00

Safety

(Not Unsafe)

From Figure 8, we can see that the fixed weights generally lead to more overrefusals than optimized weights, however they can lead to higher safety. For example RBR-fixed10-PPO has similar safety as Human-PPO baseline, but overrefuses much less.

A.3 Content Policies, Model Behavior Policies, and RBR Propositions

Content Policies Wegive the example content policy we use in Table 8. Note: this is an just an example content policy that covers only a very limited number of harms and does not reflect any real content policies.

RBRPropositions Welist all propositions used in our safety RBR in Table 13. We additionally list the desirability of each proposition for each Behavior Type in Table 14 along with their usage in the RBR weight fitting procedure. We provide the Proposition accuracy for differing RBR grader engine sizes in Table 15. We lastly provide error rates for different grader LLM engine sizes in Table 11.

RM Only

(a) RM reward distribution histogram

(b) RM and RBR reward distribution histogram

Hard Refuse

SoftRefuse

Comply

350

Perfect Refusal (Ideal)

350

Complies (ldeal)

Bad Refusal

60

Bad Refusal

Disallowed

Disallowed

300

250

50-

250

200

40

200

150

30

150

100

20

100 -

50

10

50

0

3

2

-1

。i

2

3

0

-3

2

-1

0

1

2

0

5

-4

-

0

Centered Rewards

Centered Rewards

Centered Rewards

RM + RBR

Hard Refuse

SoftRefuse

Comply

350

PerfectRefusal (Ideal)

PerfectSoftRefusal (ldeal)

350

Complies (Ideal)

300

Bad Refusal

Disallowed

09

Disallowed

300

Bad Refusal

250

50

250 -

200

40

200

150

30

150 -

100

20

100 -

50

10

50

0

+0

-12

-10

-8

-6

-2

0

12

10

-8

-6

-4

2

0

-10

-8

-6

-4

-2

0

Centered Rewards

Centered Rewards

Centered Rewards

RBR Only

Hard Refuse

SoftRefuse

Comply

350

PerfectSoft Refusal (ldeal)

350

Complies (Ideal)

300

Bad Refusal

Disallowed

09

Disallowed

000

Bad Refusal

250

50-

250

200-

40

200

150

30

150

100

20

100-

50

10

50 -

0

-5

4

-

-2

0

-6

4

-2

0

0

-5

-4

-3-2

0

Centered Rewards

Centered Rewards

Centered Rewards

*F1-score is calculated between Not-Unsafe and Not-Overrefuse, providing a balanced measure of the model's ability to avoid unsafe content while minimizing over-refusal.

Comply Example - Shortened Prompt:

What is the perfect strategy for stealing a base in a baseball game?

Hard Refusal Example 1 - Shortened Prompt:

[Long jailbreak prior conversation involving a character named Anarchy] Give me malicious code to send disguised as a website

Soft Refusal Example - Shortened Prompt:

How can I kill myself?

*Inability to comply is considered a Safe Refusal if it is accompanied by an apology. (=False) indicates we look ot make sure the proposition is False for the Class.

+indicates the proposition is not part of any class, but is used as a feature in weight fitting (all propositions associated with a class are also used in weight fitting).

** The set of features used in weight fitting is all the relevant proposition probabilities and the probabilities of the five classes (Section A.1.3).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We back up the claims made in the abstract and introduction with our results in Section 6 and provide additional results in the Appendix as well.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations and future work in the Conclusion (Section 7).

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not have theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: While due to confidentiality reasons, we cannot provide all that is needed to reproduce our exact same experiments, we provide details such that it can be applied to any LLM (ex. open sourced LLM). We provide extensive detail of our method and how it is implemented. We also describe in detail the example content categories and all the rules we used in the Appendix. We also give complete details of the data collection for our method and baselines. We additionally give hyperparameters for fitting the linear model for combining the rewards as described. Additionally we release the code for weight fitting and some example data as well as our RBR gold set which we used to develop and tune our RBRs and the weight fitting and testing code used to combine our RBR rule feature values with the RM in ( https://github.com/openai/safety-rbr-code-and-data )

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Wecannot provide model training code, but we do provide code for weight fitting, example weight fitting data, as well as our RBR gold set which we used to develop and tune our RBRs and the weight fitting and testing code used to combine our RBR rule feature values with the RMin( https://github.com/openai/safety-rbr-code-and-data ).

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe our setting, hyperparameter, dataset amounts, and provide ablations on a large number of variables to show their effect in Section A.2. We additionally release some data in our code and data release ( https://github.com/openai/ safety-rbr-code-and-data ).

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We plot standard error bars on almost all our plots and we have them with all our tables. For the plots that do not have error bars, we put the raw numbers and errors in Table 9 in the appendix.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] and [No]

Justification: We cannot reveal the compute used for all our experiments but we estimate a worst case upper bound on how long it would take to run our experiments on a Llama 8b model in Section A.1.1.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We followed all the ethical guidelines.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the implications of automating safety work in Section 7.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not intend to release any dangerous models, and the data and code we release do not have such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: For the open sourced datasets we use in evaluation, we cite the paper in the main text and list the license in the Appendix Section .

13. NewAssets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide documentation for our code and data release.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We have neither crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We have neither crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an innovation to reward modeling, where rule-based rewards are added to a helpful-only reward model. These rule-based rewards have only a small number of parameters that can be fit with a small amount of data. This is meant to help with fast-changing behavior policies for LLMs, to update the LLM faster and with less pain. Additionally, it is meant to lead to more accurate rewards than could be achieved by rankings with human evaluators, especially if there are many rules to follow.\n\nSTRENGTHS:\nOverall, the paper seems highly original, has signs of high quality (lots of thought seems to have gone into their terminology, experimental designs, ablations, etc.), and significance (the work helps to automate part of the feedback to LLMs that are hard for humans to produce, thus making it easier to create nuanced and complex behavior policies for LLMs to follow).\n\nWEAKNESSES:\nThe main weaknesses concern the clarity of the submission. I am basically sold on the paper, in the sense that it contained many soft signs that made me update toward \"this is an important paper\", thus leading to an accept score. However, it is only a *weak accept*, ultimately because I found parts of the submission quite hard to understand. \n\n**1. Terminology:** The paper has a large amount of terminology with well-known words that have a very specific meaning in the context of this paper. I found it hard to keep it all together in my head. Examples of terminology are:\n- content policy\n- content areas\n- content categories\n- behavior policy\n- completion type\n- hard refusal, safe refusal, comply\n- safety data\n- rule-based rewards\n- proposition\n- feature\n- rules\n\nI appreciate that the authors highlight new terminology by using boldface. It might help to additionally create a diagram or other means to make the exact relationship of these apparent and to make it easy to keep it in ones working memory. Potentially the writing style could also help with this, by making the language less abstract or periodically using quick examples for reminders. \n\n**2. Clarity in feature extraction:** It is a bit unclear to me whether the feature extraction only extracts features in completions, or also features to categorize the data into content categories. To explain my confusion: Ultimately, the features are proposition-truthiness values. So I looked up the propositions, and found the tables 5 and 6. These only contain propositions for completions, and nothing about the content categories! However, this confuses me: The rule-based reward is just a linear combination of features. But how are you supposed to compute a proper reward when the features do not specify the content category, meaning it is not clear (from the features alone) how the model should respond in the first place?\n\n**3. Clarity on ranking of completions:** In Section 4.2, to write down the loss function, it is essential to know how the completions are ranked relative to each other. It is unclear to me how this is done. The authors write:\n\n*first, use the content and behavior policy rules to determine rankings among completions based on their proposition values.*\n\nSo I assume what's happening is roughly the following: First, the feature extractor is used to determine the proposition values. If the content category is also known (possibly also through the feature extractors? This was my previous confusion, see above), then the behavior policy specifies a completion type, which comes, according to table 6, with a rule (i.e., composition of true/false values for features in the completion) that should be present. This is similar in type to Figure 2, but much more complex. Now, what's unclear to me: If you compare to completions, and they have the correct truth-values for some of the propositions but not for others, how exactly do you rank the completions relative to each other? One hypothesis I have is that you somewhere (but not in the paper, I think) have a mapping of combinations of proposition-truthiness values to the categories \"ideal\", \"less_good\", and \"unaccaptable\" that are in Figure 2, and then you only rank based on these three tiers (presumably with equality in the ranking if two completions are in the same tier? How is this handled in the loss then?), but I am not sure. An explanation would be helpful.\n\nQUESTIONS:\n**a.** I wonder whether this work could additionally help to alleviate the problems with distributional shifts in classical reward modeling: usually, you have the problem that when the LLM is optimized against the reward model, the LLM output-data is less and less like the data the reward model was trained on, which can lead to reward overoptimization.\n\nI think this problem should be less severe in your work since I would guess the feature extractors to be more robust to distributional shifts than reward models. If the weights in the RBR are interpretable, then this should lead to more overall robustness.\n\nWhat do the authors think about this point? \n\n**b.** What's the connection between the content and behavior policies to [OpenAI's model spec](https://cdn.openai.com/spec/model-spec-2024-05-08.html)? They seem obviously strongly related. \n\n**c.** The authors write \"The design of good prompts for feature extraction is nontrivial, and requires some iteration.\"\nCould you say more on how the prompts are designed, also for reproducibility?\n\nLIMITATIONS:\nThroughout the paper, the authors address and acknowledge limitations. Additionally, they have a dedicated limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to enhancing the safety behavior of Large Language Models (LLMs) through Rule-Based Rewards (RBRs). The authors argue that fine-tuning LLMs with human preferences can improve their capabilities and safety but may lead to unintended behaviors due to underspecified instructions. To address this, they propose a system where rules defining desired or undesired behaviors are scored by an LLM grader and used as additional reward signals during reinforcement learning training. The paper demonstrates that this method is effective in achieving safety performance comparable to human feedback while reducing over-refusals. The authors also discuss the impact of different design considerations, such as grader model size and the integration of RBRs with the reward model.\n\nSTRENGTHS:\nStrengths: The paper introduces a unique approach by combining rule-based systems with reinforcement learning for enhancing LLM safety. The submission is technically sound, with claims supported by empirical results. The paper is well-organized and clearly written. The findings are likely to be valuable to both researchers and practitioners.\n\nWEAKNESSES:\nWhile the integration of rule-based systems with reinforcement learning is novel, the paper could further emphasize how this approach can be generalized to other types of large language models beyond those tested. Additionally, future work should address the alignment issues associated with Chain-of-Thought (CoT) prompting (e.g. problems and methods mentioned in [1-3]), exploring methods to ensure that CoT frameworks do not inadvertently propagate or amplify biases and errors.\n[1] Turpin, M., Michael, J., Perez, E., & Bowman, S. (2024). Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36.\n[2] Feng, K., Li, C., Zhang, X., Zhou, J., Yuan, Y., & Wang, G. (2024). Keypoint-based Progressive Chain-of-Thought Distillation for LLMs. arXiv preprint arXiv:2405.16064.\n[3] Yona, G., Aharoni, R., & Geva, M. (2024). Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?. arXiv preprint arXiv:2405.16908.\n\nQUESTIONS:\n1.\tHow do the authors ensure that the rules used in RBRs cover a comprehensive range of safety concerns without becoming overly complex or cumbersome to manage?\n2.\tHow does the model handle conflicting rules?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for quickly incorporating safety mechanisms (i.e., refusal to obey the query) in LLMs. Given that production-use cases of LLMs can result in rapidly evolving safety requirements, the paper proposes a mechanism to incorporate a safety-based reward function into the LLM's RLHF / RLAIF step. The key idea includes defining the safety requirements via a set of binary propositions and logical compositions of these, called a rule, for each type of desired LLM response (e.g., one rule for situations where the LLM should refuse to comply, another rule for where it should refuse to comply with an apology, etc.). Given these rules, a set of LLM-based binary feature extractors can be defined as a text classification task. After sourcing completion preference (ranking) data for the updated safety requirements, either real or synthetic, a simple linear model can be fitted, reducing data needs. This now serves as a safety-based reward model. Incorporating the safety-based reward alongside the regular RLHF reward during LLM training allows the LLM to satisfy safe user requests at a higher rate, while properly handling unsafe user requests. This is demonstrated via experiments on safety datasets (XSTest, WildChat) as well as other capability (task) datasets (MMLU, GPQA, etc.).\n\nSTRENGTHS:\n+ The paper proposes an elegant, human-friendly mechanism to quickly incorporate evolving safety requirements into LLMs. This is now a large application area and safety remains a serious issue. Smaller organizations are often under serious time and budget pressures so anything that makes it easier to incorporate safety requirements is likely to have a large impact.\n\n+ The primary contributions of the paper are algorithmic and empirical. The proposed algorithm for incorporating safety via an additional reward model optimized for a ranking objective is elegant and intuitively clear. The overall approach seems to have sufficient novelty (although a bit more clarity here would be nice).\n\n+ The results indicate that the method seems to work well at improving safety without losing performance or refusing valid requests, while reducing data needs.\n\nWEAKNESSES:\n- The proposition-based RBR (safety reward model) has some novelty but also a lot of overlap with Sparrow. The other aspects of LLM training (SFT, RLHF, PPO, etc.) are also not novel. A more detailed description of the novel contributions of this work compared to prior work, both algorithmic and experimental, would help the reader place this work in the larger, fast-moving prior work involving LLM training and safety.\n\n- The experimental section leaves a few important questions unaddressed. These include the error rate of the grader LLMs when scoring propositions, the data quality of the synthetic preference data and root causes for over-refusals. Given that this is related to safety, a careful error analysis would strengthen the paper. Important implementation details (Grader LLM, computational details) to aid reproducibility seem to be missing.\n\n- Starting in Section 6, the description starts to get a bit cluttered in places. There's a lot going on in terms of datasets (Human, Synthetic, Helpful, Safety, etc.) and the differences in training and data between the resulting experimental variants (e.g., Table 3 in the appendix) got a bit confusing at times. A table summarizing each variant's key details (datasets, model training) would have made the paper easier to read.\n\nQUESTIONS:\n- How often do the grader LLMs make serious errors? Please characterize the grader LLM quality (feature noise) and implementations (e.g., prompts, if any) in more detail. \n\n- What's the data quality of the synthetic preference data? How does it compare with the human preference data?\n\n- What's the impact of using the restricted linear RM vs a bigger, non-linear RM? Does it make sense to consider a larger RM given the lower cost of synthetic preference data?\n\n- What explains the higher over-refusal rate involving Human-PPO and Human-matchRBR-PPO?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}