Rule Based Rewards for Language Model Safety

Authors: tong  mu
alec  helyar
johannes  heidecke
joshua  achiam
andrea  vallone
ian  kivlichan
molly  lin
alex  beutel
john schulman lilian
weng  openai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:43:53

Abstract: Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior.
  However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental.
  Additionally, as model capabilities and usage patterns evolve, there may be a costly need to add or relabel data to modify safety behavior. 
  We propose a novel preference modeling approach that utilizes AI feedback and only requires a small amount of human data. 
  Our method, Rule Based Rewards (RBR), uses a collection of rules for desired or undesired behaviors (e.g. refusals should not be judgmental) along with a LLM grader.
  In contrast to prior methods using AI feedback, our method uses fine-grained, composable, LLM-graded few-shot prompts as reward directly in RL training, resulting in greater control, accuracy and ease of updating.
  We show that RBRs are an effective training method, achieving an F1 score of 97.1, compared to a human-feedback baseline of 91.7, resulting in much higher safety-behavior accuracy through better balancing usefulness and safety.

Main paper: 
 introduction: 
 As large language models (LLMs) grow in capabilities and prevalence, it becomes increasingly important to ensure their safety and alignment. Much recent work has focused on using human preference data to align models, such as the line of work on reinforcement learning from human feedback (RLHF) [1] [2] [3] [4] [5] [6] [7] [8] . However, there are many challenges in using human feedback alone to achieve a target safety specification. Collecting and maintaining human data for model safety is often costly and time-consuming, and the data can become outdated as safety guidelines evolve with model capability improvements or changes in user behaviors. Even when requirements are relatively stable, they can still be hard to convey to annotators. This is especially the case for safety, where desired model responses are complex, requiring nuance on whether and how to respond to requests. If instructions are underspecified, annotators may have to rely on personal biases, leading to unintended model behaviors, such as becoming overly cautious, or it responding in an undesirable style (e.g. being judgmental). For example, some annotators in one of our experiments, when ranking possible responses to user requests pertaining to self-harm, favored completions that referred the user to a US suicide hotline phone number, which would not have helped users in other regions. Fixing such issues often requires relabeling or collecting new data, which is expensive and time consuming.To address these issues, methods that use AI feedback [9] [10] [11] [12] have recently gained popularity, most prominently Constitutional AI [10] . These methods use AI feedback to synthetically generate training data to combine with the human data for the supervised fine-tuning (SFT) and reward model (RM)We use two open sources datasets for safety evaluations:• XStest [19] operates under the Creative Commons Attribution 4.0 International license.• Wildchat [20] operates under AI2 ImpACT License -Low Risk Artifacts ("LR Agreement")In Table 6 we provide a glossary of terms used throughout the text.In Table 9 we provide all numbers with standard errors for various figures in the main text.In Table 10 we provide the experimental settings for all experiments and ablations.In Table 12 we provide sampled completions from various Large sized models for prompts that have different desired behaviors.In Figure 9 we plot all reward distribution histograms. 
related works: 
 Reinforcement Learning from Human Feedback (RLHF): Research in RLHF methods [1] [2] [3] 7] demonstrates the efficacy of human annotations in steering model behavior. A subset [4, 8, 13] of this RLHF research considers achieving better safety behavior through methods such as separating out signals of helpfulness and harmlessness. Similarly, we also focus on improving model safety, but focus on fast and scalable automated methods that leverage AI feedback. Most related to our work, Sparrow [5] proposes a novel approach to RLHF which trains a second rule-conditioned RM to detect potential rule violations. Like Sparrow, we also use rules, but we have a few key differences. Sparrow focuses on utilizing human data and they collect more than 14K human-annotated conversations. We instead focus on utilizing AI feedback. Additionally, our approach involves fitting a model to ensure that the final reward effectively and correctly ranks completions which Sparrow does not. Lastly, we skip the step of distilling rules into RM data and focus on incorporating the rule as directly as possible into PPO training.Reinforcement Learning From AI Feedback (RLAIF) To address the cost and time of collecting human data, work that uses AI feedback to improve models have been a topic of recent study in both safety (such as CAI [10, 11] ), and non-safety settings (RLAIF [9] ). These methods look at generating synthetic comparison datasets using AI feedback that is used to train a reward model. In contrast, instead of synthetically generating comparison datasets, we look at incorporating LLM feedback directly into the RL procedure. We additionally differ by using fine-grained and composable rules of desired behavior which allows for increased controllability of the model refusal behavior and responses. Our setting comes with a different set of challenges which we study, such as how to best combine the LLM feedback with the reward model.Additional Related Methods: Additional related work include studies on improving the final outputs or finetuning on top of a model( [14, 15] . However, we consider a different setting as we aim to build safety behavior into the model via RL training. Our approach is also loosely related to work that considers different ways of designing rewards for LLMs, such as RAFT [16] . 
setting and terminology: 
 We consider a production setup of an AI chatbot system where a pretrained large language model (LLM) is periodically finetuned to align to an updated behavior specification, using a standard pipeline of first supervised fine-tuning (SFT) the model and then applying reinforcement learning from human preferences (RLHF). At the RLHF stage, we first train a reward model (RM) from preference data and then train the LLM against the RM via an reinforcement learning (RL) algorithm like PPO [17] . We assume that we already have the following data standard for RLHF:• Helpful-only SFT demonstrations contains examples of helpful conversations.• Helpful-only RM preference data tracks comparisons between chatbot responses, where in each comparison a human annotator has ranked the completions based solely on their helpfulness to the user. • Helpful-only RL prompts is a dataset of partial conversation prompts that do not contain requests for unsafe actions.Additionally, we assume we have:• A Moderation Model: For both human feedback baselines and automated methods we need a method of obtaining relevant safety RL prompts. We assume we have an automated moderation model that can detect if text contains a request or a depiction of various unsafe content. Pre-existing models such as ModerationAPI [18] can be used. In this work we train a model similarly to ModerationAPI which we will refer to as ModAPI. • Safety-relevant RL prompts (P s ): A dataset of conversations ending in a user turn, some of which end with a user request for unsafe content. To combat potential overrefusals, this additionally includes user requests that should be complied with, including boundary cases (e.g. classification of harmful content) and helpful-only prompts (see Appendix A.1.4 for details and breakdowns). This set of prompts can be curated and labelled using the Moderation Model. We used a total of 6.7k conversations.Furthermore, we assume that a process of deliberation has occurred between relevant stakeholders to produce both a newly-updated content policy (a taxonomy that defines precisely what content in a prompt is considered an unsafe request) and a behavior policy (a set of rules governing how the model should in principle handle various kinds of unsafe requests defined in the content policy). The specifics of designing appropriate content and behavior policies is out of scope for this work. We aim to align the model in a way that maximizes helpfulness while also adhering to our content and behavior policy in a way that is efficient in both cost and time.For our experiments, we use a simplified example content policy that addresses several kinds of unsafe content relevant to an LLM deployed as a chat model. There are many other categories of harmful content that should be covered by a comprehensive, production level, content policy. Although the policy itself is not comprehensive, it has a level of granularity appropriate to a production setting. A detailed description of the content and behavior policies can be found in the appendix A.3, but we give a brief summary here. The content policy classifies user requests by content area and category within the content area. In our example, we consider four content policy areas: Erotic Content (which we will abbreviate C), Hate Speech (H), Criminal Advice (K), and Self-Harm (SH).Categories within the content policy are used to determine the behavior policy which outlines the ideal response type. We consider three response types (see appendix A.3 for examples): Hard Refusals: the ideal response includes a brief apology and a statement of inability to comply with the user's request, without excess verbosity. Soft Refusals: the ideal response includes a more nuanced and specialized response. For example, in the self-harm case, we would like the model to give an empathetic apology that acknowledges the user's emotional state, but declines to comply with the user's request for methods of self harm. Comply: the model should comply with the user request.(This applies to our safety boundary and "normal" prompts in P s .)The appropriate response type for a given user request varies by content policy category -we define this mapping as the behavior policy. To combat overrefusals, we include content policy categories that capture the safety boundary within a content policy area: the often complex line between what's considered acceptable or unacceptable for a model to engage with. For example, users may request that the model classify text that is about harmful material without asking the model to directly generate new harmful content. In these cases, the behavior policy may require the model to comply. 
rule-based rewards for safety: 
 In this section, we describe Rule-Based Rewards (RBRs), our proposed approach to building safety reward functions for RL training based on a content and behavior policy. We also provide code and example synthetic data for fitting the reward combination models described in this section2 . To motivate our approach, given a content and behavior policy, consider what researchers must do to prepare labeling instructions for safety data annotators. The researchers have to write a list of natural language rules for defining a good completion and scoring completions with undesirable features, taking great care to ensure that instructions are specific enough that different annotators will produce the same judgements. Researchers often also have to provide illustrative examples. These instructions and examples are ideal for use in a few-shot LLM classification task.In our observations, LLMs demonstrate higher accuracy when asked to classify specific, individual tasks, such as determining whether a text contains an apology, compared to general, multilayered tasks such as rating completions given a large content and behavior policy as input. To leverage this strength, we simplified these complex policies into a series of individual binary tasks, termed propositions. We then established a set of rules that determine when combinations of these propositions' truth values are desired or undesired. This framework allows us to accurately rank completions using these classification rules.In order to combine safety rule-based rankings with a helpful-only RM in a principled way, we use them to fit an auxiliary safety reward function that takes only proposition-based features as input, which we refer to as the Rule-Based Reward. We add the RBR to the helpful-only RM to use as the total reward in RLHF, as shown in Figure 2 . In the subsections that follow, we describe an inner loop of fitting RBR weights given features, to be interleaved with an outer loop of evaluating the effectiveness of the total combined reward, and potentially modifying the fitting setup (ex changing to model we fit). We first describe various components that make up an RBR. As there are many different datasets mentioned. We provide a table summarizing datasets needed in Table 3 at the end of this subsection.The lowestlevel element in our RBR is a proposition. Propositions are binary statements about completions given the prompt, such as refuses: "the completion contains a statement of inability to comply".A rule determines the ranking of a completion given a prompt. For each target response type (hard refusal, safe refusal, or comply), there is a set of rules that govern the relative rankings of desired and undesired propositions for the completion. We illustrate this in Figure 1 , where we show an example of hypothetical rules for ranking tiers of hard refusal and comply behaviors. For a given prompt, completions that satisfy the ideal rule rank higher than less_good which rank higher than unacceptable completions. We give a short example list of propositions in Table 1 and provide full details on the propositions and rules in Table 13 .Features, Graders, and Classification-Prompts: We define a feature as any numerical value that is determined by a prompt and a completion to that prompt. We will denote as ϕ i (p,c) where p is the prompt, c is the completion and i is the index of the feature. In this work, we use logit probabilities from an LLM, however features are flexible and can be any numerical value. We use the probabilities of a proposition being true for a completion as judged by a grader LLM with a few-shot classificationprompt. These classification-prompts contain natural language descriptions of the content and behavior policy and instructions to only output the tokens yes or no. We then use the logits of those tokens to calculate probabilities. Table 14 in the Appendix maps which proposition probabilities were used as features for each behavior category. The design of prompts for feature extraction requires some iteration and the choice of grader LLM is also highly impactful. In our experiments, we use a helpful-only SFT model which showed higher precision when labeling disallowed content. We additionally use more general "class" features as illustrated in Figure 1 (ex. "ideal")3 by multiplying the relevant propositions attached to each class and normalizing. In our experiments, we use a total of 20, 23 and 18 features for Hard-Refusal, Soft-Refusal, and Comply respectively (listed in Appendix Table 14 ). Our final classificaiton-prompts for all propositions can be found in our released code.A Small Set of Human Labelled Data for Prompt Tuning: To tune the classification-prompts mentioned above, we synthetically generate a small dataset of conversations ending in assistant turns to have diverse representation across our safety categories and propositions. We give an overview of the process used to generate this data in Figure 6 . Then, we researchers manually label the truthiness of each proposition for the final assistant completion of each conversation. We refer to this labelled set as the Gold set. We manually labelled a total of 518 completions across the three behavior categories to tune the grader prompts for RBRs: 268 for Comply, 132 for Hard Refusal, and 118 for Soft Refusal. Finally, we tune the prompts by hand against this dataset. In Table 2 we give the overall accuracy on a few different model sizes (explained later in Section 5.1) and a detailed breakdown of the prompt accuracy per proposition on this Gold set in appendix Table 15 . Weights and RBR Function: The RBR itself is any simple ML model on features, and in all of our experiments it is a linear model with learnable parameters w = {w 0 ,w 1 ,...,w N }, given N features:EQUATIONSynthetic Comparison Data For Weight Fitting: We synthetically generate data to create a set of comparison data, D RBR , for fitting the RBR weights w. To fit the weights, for each prompt p i , we need a set of k diverse completions (c i,j ) per prompt that have different rankings:D RBR = {(p i ,c i,1 ,c i,2 ,...,c i,k )} i=1,...,|D RBR |, and ranking order between completions (e.g.c i,1 > c i,2 = c i,3 > c i,4 ...) of how good the completion is. Our setup with propositions lets us easily generate exactly the data needed, conditioned on the content and behavior policy. We can use the natural language descriptions we already have to prompt for diverse completions with various rankings. For example, for a prompt that should be hard refused, we can decide we want the following set of 4 completions: one perfect hard refusal (ideal), two bad completions with randomly sampled bad refusal traits, such as judgement and/or illogical continuation, and one that contains the requested disallowed content.The goal is to have synthetic completions representing an ideal completion, a few diverse sub-optimal completions, and an unacceptable completion for every prompt.We start with the train split of our safety prompts (P s ) and the desired set of completions. For each desired completion, we iteratively synthetically sample a candidate completion from a prompted Helpful-Only model, and use our RBRs, ModAPI and other quality LLM filters to confirm it contains the desired traits (ex. we did indeed generate a judgy bad refusal) and resample if necessary.SFT Data: We use the completions labelled as ideal from D RBR above as SFT data.In order to fit an RBR, one must have: (1) Classification-prompts for each proposition and a grader LLM to compute features ϕ i .(2) The default reward model, R rm , that will be used during RL training.(3) D RBR , the RBR weight fitting comparison dataset described above.The RBR fitting procedure is straightforward: first, use the content and behavior policy rules to determine rankings among completions based on their proposition values. Then, optimize the RBR weights so that the total reward achieves the target ranking. We do this by minimizing a hinge loss:EQUATIONwhere c a ,c b are any two completions corresponding to p such that c a ≻ c b (c a ranks better than c b under the content and behavior policy).For all our experiments we used the same number of datapoints as PPO prompts to fit the weights. However the number of parameters in a linear RBR is just the number of relevant propositions + the five class probabilities, which is tiny by comparison to the number of parameters in a standard RLHF RM. Fewer examples are probably required and we discuss this later in the discussion Section A.2.Because there are only a small number of optimizable parameters, fitting an RBR is extremely fast (can run on a standard laptop in a couple of minutes). We discuss hyperparameters used in fitting RBRs in the Appendix Section A.1.5 and other alternate ways of combining the RBR with the RM ( manually setting weights) in Appendix Section A.2.1. Even before running RL and evaluating the final model, we can measure how good a reward function is by using the held-out test set of the weight fitting data D RBR , and checking whether the reward function enforces the target rankings on that data. Through these evaluations, we can see if we need to make changes to the weight fitting procedure such as potentially adding additional features or changing the model (e.g. to a non-linear model). In Figure 3a , we plot histograms of two different reward functions for various responses to prompts that demand hard refusals. To account for the fact that different prompts may have different base rewards (R rm ), we center the rewards: given a prompt and its set of k = 4 completions, we subtract out the reward of the ideal completion from each of the three other completions. We can see the helpful-only RM itself does not have any separation/ranking between ideal (perfect refusal), slighly bad (bad refusal), and very bad (disallowed) completions. Adding the RBR (RM + RBR) allows for separation and correct ranking -ranking ideal over slight bad over very bad completions. We provide more histograms for all response types in the Appendix Figure 9 .We can additionally look at the error rate of the RM which quantifies the number of mistakes where a non-ideal completion was ranked above the ideal completion as a percentage of all comparisons that involve an ideal completion. To have a metric focused on only correct behavior, we calculate this using only comparisons that involve the ideal completion, and do not consider whether we correctly ranked two non-ideal completions (e.g. bad refusal > disallowed). In Figure 3b , we see using the RBRs with the RM greatly reduced the error rates across all response types. 
experiments: 
 In our experiments, we aimed to investigate several core questions: (1) Does our approach of training with RBRs and synthetic data improve over models trained with human preference data alone? We are interested in whether they can improve safety while getting closer to the decision boundary by preventing over-refusals. (2) Does our approach make more efficient use of human data? (3) What is the behavior of RBR-based training when used in conjunction with a reward model that incentivizes models to over-refuse? Can the RBR approach help correct for this?Baselines: We compared our RBR-trained models against relevant baselines:Helpful-Only Baseline: The helpful-only baseline are the SFT, RM, and PPO models trained with our helpful-only RLHF datasets following a procedure similar to that described in Ouyang et al [1] .Human Safety Data Baseline: In addition to our helpful-only data, we add human-annotated safety data for our set of safety-relevant RL prompts P s . We send these prompts to annotators who are familiar with our content and behavior policies and have been actively labelling similar safety prompts under similar instructions for several months. We follow the standard RLHF comparison data collection procedure [1] and ask annotators to sample 4 completions and label them with a rating from 1-7. We provide additional details in Section A.1.2Throughout results and ablations we use 4 model sizes which we will refer to as Large, Medium, Small, and XSmall. The size of the Medium, Small, and XSmall models are such that they use roughly around 0.5%, 0.1%, and 0.001% of the effective compute used to train Large respectively, where Large is of size comparable to GPT-4 but with a greatly reduced data mix for quick experimentation. All synthetic data for all experiments were sampled from Large sized models. For all the main results in section 6 below, we run PPO where all safety prompts are seen once, and the ratio of Hard Refusal to Comply prompts is equal as labelled by human data. 4 We use the Large Helpful-SFT model as the RBR grader engine, as well as Large size RMs. All automated evals use a Large sized grader model.Results after RL training are often high variance, so for all evaluation scores reported, we evaluate on 5 checkpoints toward the end of PPO training and report the average mean and standard error. We give high level descriptions of the evals we run here, and provide more details (ex. number of prompts) in Appendix Section A.1.6.Internal Safety RBR Evaluation: We evaluate our models on a diverse set of internal prompts which are manually labeled by researchers with our content policy category. In total, there are 588 Comply, 565 Hard-Refusal, and 185 Soft-Refusal prompts. We use the classifications of the Safety RBR's propositions to automatically evaluate three internal metrics: Not-Unsafe: the percentage of completions which do not contain any disallowed content. Not-Overrefuse: the percentage of completions for Comply prompts which are not refusals. Hard-Refusal-Style: the percentage of completions in the ideal style for Hard-Refusal prompts (i.e. no incorrect response elements).We note that for this evaluation there is some overlap with our training signal due to RBR usage. There are important differences in the signals: there is no overlap in prompts between our train and evaluation sets. Additionally, for evaluations we do not use the RBRs as described in training. Instead we convert the output probability scores for each proposition into binary labels using a threshold optimized on the Gold set (thus skipping the weight fitting and subsequent steps). We additionally show our RBR has high accuracy on an our Gold set (Tables 2, 15 ) and provide other safety evaluastions described below.XSTest and WildChat: XSTest [19] test for overrefusals with lexicon that models can become overly sensitive to (e.g. "How can I kill a Python process?"). While WildChat [20] provides publicly available chat prompts. For both we filter to prompts relevant to our policies. We measure overrefusal rate using both our Not-Overrefuse metric and the default XSTest classification prompt using GPT-4 and we evaluate safety using three automated tools: ModAPI, our Not-Unsafe RBR-based metric, and Llama Guard 2 [21, 22] .Human Safety Evaluations: To further verify our safety evaluations, we ran human evaluations of safety behavior using the prompts from XSTest. The human evaluators are researchers on the team who have much experience with the Content and Behavior policy. For each prompt, a completion was sampled from each of the Helpful-PPO baseline, Human-PPO baseline, and RBR-PPO models. Model names were hidden from the evaluators and the order of completions shown was randomized.Capability Evaluations: To monitor model capabilities, we evaluate our models on MMLU [23] (Averaged across zero-shot, 10-shot, and zero-shot CoT), HellaSwag [24] (Zero-shot), GPQA [25] (Few-shot CoT averaged across 1-, 5-, and 10-repeats on Diamond), and Lambada [26] (Zero-shot).For speed purposes we evaluate against large subsets of these datasets. 
results: 
 All experiments were run under the settings described in Section 5.1. All figures report results on Medium sized policy models, while all tables report results on Large sized policy models.Our safety RBRs improve safety while minimizing over-refusals. In Table 4 we give the results of both our human and automated internal safety evaluations on Large sized models. We see that under both evaluations, RBRs (RBR-PPO) are able to substantially increase safety while minimally impacting the amount of over-refusals, achieving the highest F1-score. The human safety data baseline, Human-PPO, increases safety greatly, however at the expense of also greatly increasing the amount of over-refusals (by almost 14% in the human evaluation). We also see similar trends from external safety evaluation benchmarks (Table 5 ).Additionally, we see similar trends in our Medium sized models shown in Fig. 4a . In Fig. 4a we plot the safety vs over-refusal trade-off on our internal safety RBR eval of our main models and baselines, along with arrows showing the movement from SFT to PPO. We see that RBR-PPO achieves a good balance of Safety and Usefulness. Additionally, while not shown in the plot, both Human-PPO and RBR-PPO improve refusal style over the helpful baseline. Interestingly enough, we note that Helpful-PPO improves upon safety compared to Helpful-SFT, even though the Helpful-Only datasets do not contain any safety-relevant data. We hypothesize this is due to the Helpful-Only datasets generally encouraging the model to be polite, which may be correlated to safety. All the raw numbers for both Figures in Fig. 4 along with standard errors can be found in Appendix Table 9 .Safety RBRs do not impact evaluation performance across common capability benchmarks. In Table 5 , we list the capability scores of the Large PPO models on four common capability benchmarks: MMLU, Lambada, HellaSwag and GPQA. Both RBR-PPO and the Human-PPO baseline maintain evaluation performance compared to the Helpful-PPO baseline.Safety RBRs help improve safety for RMs with different tendencies. The default RBR-PPO setting applies the safety RBR on top of the Helpful-RM. In Fig. 4b , we additionally show the result of combining the RBR with different RMs with dotted arrows showing the movement on PPO models after adding RBRs. We apply RBRs to the Human-RM which, as empirically evidenced through the PPO model, has a higher tendency towards over-refusals. We label this as HumanRM+RBR-PPO , reducing over-refusals by 16% compared to Human-PPO. Additionally we apply the safety RBR on top of a RM trained with outdated safety data (Old Data-PPO), which also has a high over-refusal rate. Applying the RBR both improves safety and reduces overrefusals by 10%.Safety RBRs require less human annotated data than the Human-Data Baseline. We investigate the performance of a human-safety data baseline after subsampling the human data down to the same amount of completions as in RBR runs, 518 completions in total. The subsampling process is constrained to ensure even representation amongst behavior types and content categories. PPO prompts remains the same as that of the RBR runs (i.e. the full set of RL prompts). We note this is not a direct comparison because the set of annotators for the two datasets is different, but it provides a ballpark estimate. In Figure 4b , we plot the result as Human-match RBR-PPO. Compared to RBR-PPO and Human-PPO, this run performs slightly worse on both Not-Unsafe and Not-Overrefuse. We hypothesize this is because the small amount of RM data is not enough to teach the model the refusal boundary.Ablations. We give the results of various ablation experiments in Appendix Section A.2. There we explore scaling different parameters, such as grader LLM engine size and safety prompt percentage.Example Sampled Completions. We give some example sampled completions from our Baseline PPOs and RBR-PPO models for prompts of each refusal type in Appendix Table 12 Discussion: Potential Loss of Information when Distilling Instructions into RM Data. Distilling a set of instructions into RM data, whether through human labelling of comparison data or synthetic AI means, is challenging since one must ensure not only that the data covers all instructions, but also that it is balanced such that the desired behavior is learned by the RM. We encountered issues related to this with the raw human data: we observed the final PPO model to be extremely cautious, over-refusing on every Comply prompt in our evaluation set (and also achieving a "perfect" score on safety). We discovered this was due to an insufficient number of low-ranked refusal examples in the RM comparison data for Comply prompts to teach the model not to refuse safe prompts. Only a third of Comply data contained this negative example, leading to 3 times more positive refusal examples than negative ones. Even though this data was only 1% of the RM dataset when combined with the Helpful-Only data, this imbalance was still enough to cause over-refusals on all prompts. To correct for this in the RM data, for all Comply data, we manually replaced a non-ideal completion with a refusal sampled from a manually created list of ∼50 refusals, and were able to train a second model that did not refuse everything to use as the human-data baseline. (Note, the Human-PPO and Human-RM referred to in the text are all trained with this corrected data.) In Figure 5 , we look at a set of safe "Comply" prompts and plot the average rewards of completions that comply and that over-refuse for the initial always-refusing human data RM, the corrected human data RM, and the Helpful-Only RM. We see that over-refusals are given almost the same score as helpful completions for the initial human data RM, making it easier to reward hack. RBRs are not subject to this issue because they skip this RM distillation step and directly incorporate the instructions into the reward function. When a over-refusal example is sampled by the model for a safe prompt during training, it is penalized by the RBR directly. 
conclusion: 
 Limitations and Future Work: In this work, we apply Rule-based Rewards (RBRs) for RL training to a situation where the desired behaviors can be clearly separated into explicit, easy-to-judge propositions.RBRs can be easily combined with human-labeled preference data in classic RLHF (ex. in this work, for our Comply prompts we used an RBR to discourage easily detectable bad behavior while judging helpfulness through the RM) and we may need to explore this more for difficult tasks. Future work may involve exploring the application of our method in harder, non-safety domains.Ethical Considerations: We discuss moving the safety feedback signal in LLM training from humans to LLMs. This reduces the level of human supervision and potentially extrapolates and magnifies inherent biases in the LLMs. To mitigate this, researchers should carefully evaluate their RBRs to ensure accuracy and measure any potential biases that come up. Using this method in conjunction with human data could also help to mitigate risks. 
 
 Appendix: We introduce a novel automated AI-feedback based preference modeling approach using Rule-Based Rewards (RBRs) for safety training in LLMs. Our method is cost-and time-efficient, requiring minimal human data. Our decomposition of ideal behavior into fine-grained modular rules also has unique advantages in allowing increased classification accuracy and easy synthetic data generation. Our experiments show our RBR method is able to achieve accurate safety-behavior. Finding a good balance between safety and usefulness compared to baselines. Topics considered by the content policy (ex. Erotic, Criminal Advice). Safety Boundary The line between what is acceptable and unacceptable, includes safe requests adjacent to unsafe requests we want to comply with to prevent over-refusals.A set of rules governing how the model should handle various kinds of unsafe requests defined in the content policy.Ideal ways we want to respond to unsafe and boundary requests.A response type where the model firmly refuses the user request. (ex. requests for criminal advice)A response type that carefully declines or respond to user requests in sensitive situations (ex. Self Hard requests).A response type where the model fully complies in a maximally helpful way to the user request (ex. safe boundary requests).Simple binary statements about completions, used in RBRs for classification (ex. does the completion contain an apology).Determines how good a completion is based on the Behavior Policy Grader LLMThe language model used to compute the probabilities of propositions being true. We use a helpful only model for this.Safety-relevant RL prompts used in training to improve safety behaviors. D RBR An offline dataset of completions of various goodness for each prompt, used for fitting the RBR reward.A manually labeled dataset used to tune classification-prompts for propositions in RBRs.The Rule-Based Reward function computed from features extracted by the grader LLM.The default reward model score based on human preference data.The total reward, calculated as the sum of R rm and R rbr . w Parameters in the RBR function that are optimized during training.ϕ i (p,c)Feature values used in RBRs, where p is the prompt and c is the completion. We used probability propositions as judged by a grader LLM for this.Loss function used to fit RBR weights, we use a hinge loss over comparisons.A Appendix / supplemental material A.We cannot give the exact amount of compute used for our experiments due to confidentiality reasons, but we estimate in the very worst and unoptimized case that under our setting it would take 5 days on 16 GPUs (80 A100 GPU days) to go through all steps (SFT, RM, PPO) for a Llama 8b. If we use the GPT4 API as the grader, then for the 5 main experiments presented in Figure 4a , it would take 400 A100 GPU days to run for a Llama 8b.The annotators had access to a helpful-only SFT model as well as system prompts (e.g. to prompt for perfect refusals) to sample completions from. As part of the instructions, annotators were given detailed guidelines on completion scoring. For example, for a prompt that should be refused, a perfect hard refusal should be rated 7, bad refusals (ex. excessively verbose) should be rated lower, and completions with disallowed content should be rated 1. Annotators were also asked to try to maximize the diversity of scores across the 4 completions by resampling individual completions if necessary. They were also asked to provide an "ideal" completion, either by copying and pasting an existing completion, or by writing an original one. We assume this ideal completion is rated 7, and from this we can construct comparison data for RM training. Additionally we use the prompts and ideal completions for SFT training. The amount of human safety data is a small amount, about 3% of SFT data and 1% of RM data when combined with the Helpful-Only datasets.We combine relevant propositions for each desired completion type (hard refusal, safe completion, comply) into 5 common classes shared by all completion types. For example, the "ideal" class refers to a completion which has only desired propositions and no undesired propositions for the desired completion type. Defining these classes is not required for RBRs, but when using several propositions it is useful to organize propositions together into meaningful labels. In our case, we use the following classes for labeling completions:1. ideal: desired behavior without disallowed content. 2. minimum_acceptable_style: desired behavior without disallowed content, but with some imperfect stylistic traits. 3. unacceptable_completion: undesired behavior, but still logical and without disallowed content. 4. illogical_completion: illogical continuation of the conversation. 5. disallowed_completion: disallowed content present somewhere in the completion.The mapping of each proposition to class is given in Table 14 .Even though they use the exact same set of prompts, the human baseline used human collected labels of desired response type, and the RBR methods use auto labelled ones, so there is some disagreement. In Table 7 we give the breakdown of number of prompts per behavior category in the train and test splits based on human labels and automatic labels. We also give the agreement rate for each of the response types (denominator when calculating the rate is determined by automatic labels). We also give the breakdown by behavior category for 518 human labelled conversations in the Gold set used for prompt tuning. For our weight fitting procedure, we used Pytorch with an Adam optimizer. We optimized on our weight fitting code for 1000 steps as the loss has converged by then. We used a learning rate of 0.01 and a weight decay of 0.05. For learning rate we tried few in that region and didn't see to big of a difference in final error rate. For weight decay, we picked the largest value that did not increase the error rate on the test set.We give more details and numbers about our evaluation process to supplement the details given in the main text.XSTest Specifically, we filtered out 52 prompts outside the scope of our content policy, resulting in 198 relevant overrefusal prompts.WildChat Specifically, we filter this dataset to unsafe prompts using ModAPI, resulting in a sample of 790 unsafe prompts. To reduce noise, we sample 5 completions per prompt at temperature 1.0 and average the evaluations.Human Evaluations As before, we filter prompts from XSTest and WildChat to those relevant to our policies. Evaluators were asked to label the desired Response-Type of each prompt and the actual Response-Type of each completion. According to the labels of human evaluators, the final dataset contained 283 Comply and 70 Hard-Refusal prompts.In this section, we present various ablation experiments. All ablations in this section were done with a Medium policy model using the Large Helpful-RM and Large RBR grader models unless otherwise stated. As with the main results, for all experiments, we fix all variables to that in the default setting as described in Section 5.1 except the variable being studied.Scaling RBR Grader Engine Size. Figure 7a shows how performance changes with different model sizes. We see that in general, safety stays about constant as the grader engine increases in size.Additionally we see that over-refusals decrease with larger grader engines. Interestingly, we see hard-refusal style take a U shaped pattern. For small grader engines, it seems the dominant encouraged behavior is refusal and the trained model learns to refuse well. As the grader engine increases in capability, it is able to learn to refuse less often, however it is not able to capture good style. Until for the largest model, it is able to perform well on both.Scaling Safety Prompts Percentage. We vary the percentage of safety-relevant prompts that would be seen during PPO training (where 100% means all PPO prompts are seen), shown in Fig. 7b . In general, safety increases with more safety prompts during RL training, while over-refusals slightly increase as well. Refusal style benefits the most from seeing more safety prompts.Scaling the Hard-Refusal/Comply Ratio. We vary the ratio of Hard-Refusal to Comply prompts during RL training in Figure 7c . We see a clear safety vs over-refusal trade-off as the ratio changes.Improving Self Harm Refusal Style For our default parameters, we found poor performance for soft refusal style. We found we can improve soft refusal style without impacting other safety metrics by adjusting the prompt ratio. In Figure 7d we show increasing the percentage of Soft Refusal prompts seen from the default amount of approximately 1/4th the amount of Comply prompts to approximately matching the amount of Comply prompts. (As a reminder there are about the same amount of Hard-Refusal prompts as Comply prompts). We see Soft-Refusal style improves without negatively impacting other safety-behavior.Weight Fitting Data Amount While we generate synthetic completions for weight fitting using all the PPO prompts we have, we hypothesize we need less data as we are fitting a model with a small number of parameters. We investigate this in Figure 7e by investigating the error rate (as described in Section 4.3) and the number of prompts used (where there are four synthetic completions per prompt). We see that approximately 300 prompts per category is sufficient for low error rate.In Figure 7f we ablate omitting certain steps and we observe that this let us fall on different regions along the Pareto frontier. SFTonly-noRBR-PPO considers training SFT from the RBR synthetic SFT data combined with Helpful SFT data, but only training with the Helpful-RM with RBRs from there. It leads to a moderate improvement in safety over Helpful-PPO but not as much as RBR-PPO. RBR-noSFT-PPO looks at not using the synthetic SFT data and starting from Helpful-SFT, it does well on safety but over-refuses more. RBR-noRM-PPO uses only the RBR reward for prompts in P s with no RM score (prompts outside of P s still use the RM score). We see this also increase over-refusals slightly.Instead of fixed weights, we test hand set weights amongst classes. We the set the following base weights vector of equally spaced base weights:{ "ideal" : 1, "minimum_acceptable_style" : 1 3 , "unacceptable_completion" : -1 3 , "illogical_completion" : -1 3 , "disallowed_completion": -1 }We tried 2 different fixed weight settings. The first setting is an "underoptimized" setting where we used the unit weight vector directly (RBR-Fixed1-PPO) for all response types. The second setting is the "overoptimized" setting where we multiply the unit weight vector by 10 (RBR-Fixed10-PPO) for all response types. From Figure 8 , we can see that the fixed weights generally lead to more overrefusals than optimized weights, however they can lead to higher safety. For example RBR-fixed10-PPO has similar safety as Human-PPO baseline, but overrefuses much less. 8 . Note: this is an just an example content policy that covers only a very limited number of harms and does not reflect any real content policies.We list all propositions used in our safety RBR in Table 13 . We additionally list the desirability of each proposition for each Behavior Type in Table 14 along with their usage in the RBR weight fitting procedure. We provide the Proposition accuracy for differing RBR grader engine sizes in Table 15 . We lastly provide error rates for different grader LLM engine sizes in Table 11 . (=False) indicates we look ot make sure the proposition is False for the Class. + indicates the proposition is not part of any class, but is used as a feature in weight fitting (all propositions associated with a class are also used in weight fitting). ** The set of features used in weight fitting is all the relevant proposition probabilities and the probabilities of the five classes (Section A.1.3). Code: https://github.com/openai/safety-rbr-code-and-dataWe note that the simplified example given in Figure1is not exactly what we do and we provide exact details in Appendix A.1.3There is some disagreement between human and automated labels, and the RBR experiments only use automated labels, but we do not re balance for the main results as we want to keep the prompt mix the same. 


Decision: This paper introduces Rule-Based Rewards (RBRs) for preference modeling of LLMs, which allows for precise behavior control and rapid updates using minimal human data. RBRs consist of rules for desired or undesired behaviors and are scored by an LLM grader to enhance reinforcement learning training.

All reviews are positive and acknowledge the novelty of the proposed method. The empirical study is also solid. However, as pointed out by reviewers, the presentation can be improved to enhance clarity. The authors should include the promised modifications in the final version.

Reviews: review: paper_summary: paper_summary: The paper proposes a method for quickly incorporating safety mechanisms (i.e., refusal to obey the query) in LLMs. Given that production-use cases of LLMs can result in rapidly evolving safety requirements, the paper proposes a mechanism to incorporate a safety-based reward function into the LLM's RLHF / RLAIF step. The key idea includes defining the safety requirements via a set of binary propositions and logical compositions of these, called a rule, for each type of desired LLM response (e.g., one rule for situations where the LLM should refuse to comply, another rule for where it should refuse to comply with an apology, etc.). Given these rules, a set of LLM-based binary feature extractors can be defined as a text classification task. After sourcing completion preference (ranking) data for the updated safety requirements, either real or synthetic, a simple linear model can be fitted, reducing data needs. This now serves as a safety-based reward model. Incorporating the safety-based reward alongside the regular RLHF reward during LLM training allows the LLM to satisfy safe user requests at a higher rate, while properly handling unsafe user requests. This is demonstrated via experiments on safety datasets (XSTest, WildChat) as well as other capability (task) datasets (MMLU, GPQA, etc.).
strength_weakness: strength_weakness: - The proposition-based RBR (safety reward model) has some novelty but also a lot of overlap with Sparrow. The other aspects of LLM training (SFT, RLHF, PPO, etc.) are also not novel. A more detailed description of the novel contributions of this work compared to prior work, both algorithmic and experimental, would help the reader place this work in the larger, fast-moving prior work involving LLM training and safety.

- The experimental section leaves a few important questions unaddressed. These include the error rate of the grader LLMs when scoring propositions, the data quality of the synthetic preference data and root causes for over-refusals. Given that this is related to safety, a careful error analysis would strengthen the paper. Important implementation details (Grader LLM, computational details) to aid reproducibility seem to be missing.

- Starting in Section 6, the description starts to get a bit cluttered in places. There's a lot going on in terms of datasets (Human, Synthetic, Helpful, Safety, etc.) and the differences in training and data between the resulting experimental variants (e.g., Table 3 in the appendix) got a bit confusing at times. A table summarizing each variant's key details (datasets, model training) would have made the paper easier to read.
questions: questions: - How often do the grader LLMs make serious errors? Please characterize the grader LLM quality (feature noise) and implementations (e.g., prompts, if any) in more detail. 

- What's the data quality of the synthetic preference data? How does it compare with the human preference data?

- What's the impact of using the restricted linear RM vs a bigger, non-linear RM? Does it make sense to consider a larger RM given the lower cost of synthetic preference data?

- What explains the higher over-refusal rate involving Human-PPO and Human-matchRBR-PPO?
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents a novel approach to enhancing the safety behavior of Large Language Models (LLMs) through Rule-Based Rewards (RBRs). The authors argue that fine-tuning LLMs with human preferences can improve their capabilities and safety but may lead to unintended behaviors due to underspecified instructions. To address this, they propose a system where rules defining desired or undesired behaviors are scored by an LLM grader and used as additional reward signals during reinforcement learning training. The paper demonstrates that this method is effective in achieving safety performance comparable to human feedback while reducing over-refusals. The authors also discuss the impact of different design considerations, such as grader model size and the integration of RBRs with the reward model.
strength_weakness: strength_weakness: While the integration of rule-based systems with reinforcement learning is novel, the paper could further emphasize how this approach can be generalized to other types of large language models beyond those tested. Additionally, future work should address the alignment issues associated with Chain-of-Thought (CoT) prompting (e.g. problems and methods mentioned in [1-3]), exploring methods to ensure that CoT frameworks do not inadvertently propagate or amplify biases and errors.
[1] Turpin, M., Michael, J., Perez, E., & Bowman, S. (2024). Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36.
[2] Feng, K., Li, C., Zhang, X., Zhou, J., Yuan, Y., & Wang, G. (2024). Keypoint-based Progressive Chain-of-Thought Distillation for LLMs. arXiv preprint arXiv:2405.16064.
[3] Yona, G., Aharoni, R., & Geva, M. (2024). Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?. arXiv preprint arXiv:2405.16908.
questions: questions: 1.	How do the authors ensure that the rules used in RBRs cover a comprehensive range of safety concerns without becoming overly complex or cumbersome to manage?
2.	How does the model handle conflicting rules?
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes an innovation to reward modeling, where rule-based rewards are added to a helpful-only reward model. These rule-based rewards have only a small number of parameters that can be fit with a small amount of data. This is meant to help with fast-changing behavior policies for LLMs, to update the LLM faster and with less pain. Additionally, it is meant to lead to more accurate rewards than could be achieved by rankings with human evaluators, especially if there are many rules to follow.
strength_weakness: strength_weakness: The main weaknesses concern the clarity of the submission. I am basically sold on the paper, in the sense that it contained many soft signs that made me update toward "this is an important paper", thus leading to an accept score. However, it is only a *weak accept*, ultimately because I found parts of the submission quite hard to understand. 

**1. Terminology:** The paper has a large amount of terminology with well-known words that have a very specific meaning in the context of this paper. I found it hard to keep it all together in my head. Examples of terminology are:
- content policy
- content areas
- content categories
- behavior policy
- completion type
- hard refusal, safe refusal, comply
- safety data
- rule-based rewards
- proposition
- feature
- rules

I appreciate that the authors highlight new terminology by using boldface. It might help to additionally create a diagram or other means to make the exact relationship of these apparent and to make it easy to keep it in ones working memory. Potentially the writing style could also help with this, by making the language less abstract or periodically using quick examples for reminders. 

**2. Clarity in feature extraction:** It is a bit unclear to me whether the feature extraction only extracts features in completions, or also features to categorize the data into content categories. To explain my confusion: Ultimately, the features are proposition-truthiness values. So I looked up the propositions, and found the tables 5 and 6. These only contain propositions for completions, and nothing about the content categories! However, this confuses me: The rule-based reward is just a linear combination of features. But how are you supposed to compute a proper reward when the features do not specify the content category, meaning it is not clear (from the features alone) how the model should respond in the first place?

**3. Clarity on ranking of completions:** In Section 4.2, to write down the loss function, it is essential to know how the completions are ranked relative to each other. It is unclear to me how this is done. The authors write:

*first, use the content and behavior policy rules to determine rankings among completions based on their proposition values.*

So I assume what's happening is roughly the following: First, the feature extractor is used to determine the proposition values. If the content category is also known (possibly also through the feature extractors? This was my previous confusion, see above), then the behavior policy specifies a completion type, which comes, according to table 6, with a rule (i.e., composition of true/false values for features in the completion) that should be present. This is similar in type to Figure 2, but much more complex. Now, what's unclear to me: If you compare to completions, and they have the correct truth-values for some of the propositions but not for others, how exactly do you rank the completions relative to each other? One hypothesis I have is that you somewhere (but not in the paper, I think) have a mapping of combinations of proposition-truthiness values to the categories "ideal", "less_good", and "unaccaptable" that are in Figure 2, and then you only rank based on these three tiers (presumably with equality in the ranking if two completions are in the same tier? How is this handled in the loss then?), but I am not sure. An explanation would be helpful.
questions: questions: **a.** I wonder whether this work could additionally help to alleviate the problems with distributional shifts in classical reward modeling: usually, you have the problem that when the LLM is optimized against the reward model, the LLM output-data is less and less like the data the reward model was trained on, which can lead to reward overoptimization.

I think this problem should be less severe in your work since I would guess the feature extractors to be more robust to distributional shifts than reward models. If the weights in the RBR are interpretable, then this should lead to more overall robustness.

What do the authors think about this point? 

**b.** What's the connection between the content and behavior policies to [OpenAI's model spec](https://cdn.openai.com/spec/model-spec-2024-05-08.html)? They seem obviously strongly related. 

**c.** The authors write "The design of good prompts for feature extraction is nontrivial, and requires some iteration."
Could you say more on how the prompts are designed, also for reproducibility?
limitations: limitations: Throughout the paper, the authors address and acknowledge limitations. Additionally, they have a dedicated limitations section.

Comments: title: Re. author response
comment: Thank you so much for your insightful and helpful feedback! We really appreciate all your suggestions on places where the writing can be clarified. Even before the rebuttal period, we have been committed to improving the paper past the initial submission (e.g. getting the code release, running human evaluations, additional ablation experiment), and we definitely will continue this commitment by incorporating all the valuable suggestions in a future version.
Thank you so much for your valuable feedback and suggestions, they are really helpful for improving the clarity of our work! We have been working to improve the paper past the initial submission, even before the rebuttal period began (e.g. getting the code release, running human evaluations), and will definitely continue to improve it by incorporating all your feedback in a future version.
I thank the authors for their detailed response. After reading the response, other reviews and comments, I've increased my score to "Weak Accept". The paper could be much clearer than it is currently and the authors do commit to fixing this. However, as Reviewer QvaG noted, this is a rather large delta from the current version. This prevents me from increasing my score further.
Thank you for your answers! These are all helpful.

I will not change the score since this would require seeing a substantially updated version of the paper, but I increase my confidence from 3 to 4. I am hopeful that the authors can improve the presentation substantially in an updated version.