RRM: Robust Reward Model Training Mitigates Reward Hacking

## Abstract

Abstract Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them. Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.

## 1 Introduction

Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone in aligning large language models (LLMs) with human preferences to produce responses that are more helpful, honest, and harmless (Ouyang et al., ; Bai et al., ). This approach involves training a reward model (RM) on human feedback, which then guides the LLM to generate high-quality responses through reinforcement learning. The success of RLHF is evident in various AI systems, such as Gemini (Team et al., ) and GPT-4 (Achiam et al., ). Despite its effectiveness, RLHF faces the fundamental issue of reward hacking (Gao et al., ), where the model maximizes the reward function without truly aligning with the intended human preferences. This hacking issue occurs because the RM, while a powerful tool, is an imperfect proxy for human judgment and often struggles with out-of-distribution generalization (Eisenstein et al., ).

The reward hacking problem manifests in several ways, with verbosity being a common issue: LLMs tend to generate longer responses to appear more detailed or explanatory, exploiting human raters’ bias towards lengthier content (Shen et al., ; Singhal et al., ). In recognition of this challenge, extensive efforts have been made in the literature. ODIN () designs a two-head approach to learn the quality reward that is orthogonal to length. Similarly, length-controlled Alpaca (Dubois et al., ) estimates the controlled direct effect (VanderWeele, ) through logistic regression by adjusting the length. To mitigate the length bias, an improved version (Park et al., ) of DPO (Rafailov et al., ) introduces length as penalty to the reward score. In practice, there are more reward hacking patterns beyond length, such as format (markdowns, bold-faces) and patterns (certain $n$-grams or emojis). This is largely due to the large output space of language with limited preference data, as well as the diverse and subjective nature of human preferences.

It is challenging to identify and mitigate all potential exploitation patterns. We may consider the causal perspective to explain this phenomena. Given a prompt $x$ and a pair of responses $(y_{1},y_{2})$, the human preference can be caused by the real quality $s(x,y_{1},y_{2})$ that is associated with the prompt, or by the context-free artifacts $a(y_{1},y_{2})$ in the responses that do not depend on prompt. Traditional reward model training cannot differentiate the above two factors. There are two reasons for this. First, the pair of responses are always contextual and on-topic to the prompt, thus no counterfactual prompt (prompt from another examples) is used. The reward model may learn the artifacts existing in the responses by ignoring the prompt. If we use the counterfactual prompt, it can help estimate the level of artifact bias ($\mathbb{P}(y_{1}\succ y_{2}|x^{\prime})$ with $x^{\prime}\neq x$) existing in the preference dataset (Zhao et al., ). Second, even if we adjust a few common artifacts, not all artifacts are observable and thus there is no easy way to control all the artifacts explicitly to answer the question “what will the preference be if both responses share the same artifacts?”.

In response to these challenges, we propose a simple and effective method to improve reward modeling. We first formulate the reward model training in a causal framework, then we augment the reward model training data based on the causal rules. By doing so, we can effectively adjust the artifacts and only learn the real quality. Our pipeline is illustrated in Figure 1, where we augment the reward model training data by using responses from other examples to effectively balance the artifacts in chosen and rejected responses. To summarize, the contributions of this paper are three-fold:

- •
We identify a key issue with traditional reward model training: it often fails to distinguish between genuine contextual preference signals and spurious context-free artifacts.
- •
To address this, we propose a causal graph for human preference modeling and introduce data augmentation to mitigate artifacts learned by the reward model.
- •
We further demonstrate that policies trained on these robust reward models consistently outperform those based on baseline reward models.

Figure: Figure 1: The pipeline of our proposed robust reward model (RRM), which aim to decouple contextual preference quality signal and context-free artifacts. Suppose a proportion of chosen responses have certain artifact (bold-face wrapped with “$**$” in this figure), the reward model can hack the pattern and choose the response with the artifact instead of carefully reading the prompt. With our data augmentations, we can effectively balance the context-free artifacts in chosen and rejected responses, thus ensuring a more robust reward model during inference.
Refer to caption: x1.png

## 2 Preliminaries

In preference learning, we assume that there exists a preference oracle that determines the probability $\mathbb{P}(y_{1}\succ y_{2}|x)$ that response $y_{1}$ is preferred over $y_{2}$ given the prompt $x$. Our goal is to optimize the preference by querying the preference oracle within certain budget constraint. In what follows, we first review the major ways to approximate and estimate the oracle based on a human preference dataset $\mathcal{D}_{\text{hf}}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\}_{i=1}^{N}$. where $x^{(i)}$ represents prompt for example $i$, and $(y_{w}^{(i)},y_{l}^{(i)})$ represents the chosen and rejected response, respectively.

#### Reward models

Bradley-Terry pointwise reward model (Bradley & Terry, ; Ouyang et al., ) is a widely adopted method, which additionally assumes that there exists a reward function $r(x,y)\in\mathbb{R}$ and the preference oracle satisfies

$$ $\mathbb{P}(y_{1}\succ y_{2}|x)=\frac{\exp(r(x,y_{1}))}{\exp(r(x,y_{1}))+\exp(r (x,y_{2}))}=\sigma\big{(}r(x,y_{1})-r(x,y_{2})\big{)}.$ $$

Then, we can fit the Bradley-Terry model by maximizing the log-likelihood on the training set:

$$ $\mathcal{L}(r_{\phi},\mathcal{D}_{\text{hf}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim \mathcal{D}_{\text{hf}}}\left[\log\sigma\left(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{ l})\right)\right].$ (1) $$

The second predominant approach is the pairwise ranking model (Zhao et al., ; Jiang et al., ), which takes a prompt and a pair of responses as the input, and directly predicts the probability $\mathbb{P}(y_{1}\succ y_{2}|x)$, which subsumes the BT model as a subclass. In the literature, the pairwise preference model has shown to outperform pointwise BT reward both empirically (Zhao et al., ; Jiang et al., ; Dong et al., ) and theoretically (Ye et al., ) due to its flexibility and larger function class capacity. Specifically, we denote the pairwise ranking model as $\rho_{\psi}(x,y_{1},y_{2})$ and leverage the next token prediction ability of the language model to format the sample as:

“[CONTEXT] {$x$} [RESPONSE A] {$y_{1}$} [RESPONSE B] {$y_{2}$}”

Then, the $\rho_{\psi}(x,y_{1},y_{2})$ outputs either “A” or “B” as preferred one. We use the probability of decoding “A” as estimation of the preference probability $\hat{{\mathbb{P}}}(y_{1}\succ y_{2}|x)$(^1^11We randomly flip response pairs and associated labels to remove positional bias.). In this work, we use the pairwise ranking model for its superior performance and flexibility.

#### Alignment Algorithms

Start with a reward function $r(x,y)$, a reference policy $\pi_{\text{ref}}$, and input prompt distribution $\mathcal{P}$, a policy $\pi$ is trained to optimize for the following objective:

$$ $\max_{\pi}\mathbb{E}_{x\sim\mathcal{P}}\left[\mathbb{E}_{y\sim\pi(\cdot|x)}r(x ,y)-\beta\mathbb{D}_{\text{KL}}\left[\pi(\cdot|x)\|\pi_{\text{ref}}(\cdot|x) \right]\right],$ (2) $$

where $\beta>0$ is the KL penalty coefficient. Several algorithms have been proposed to solve the above optimization, including PPO (Schulman et al., ; Ziegler et al., ), SLiC (Zhao et al., ), DPO (Rafailov et al., ), RSO (Liu et al., ), and IPO (Azar et al., ). For a stable evaluation process, we use DPO in this work for preference alignment. For a given preference dataset $\mathcal{D}_{\text{p}}$, DPO use the following loss function:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta}|\pi_{\text{ref}},\mathcal{D}_{\text{p}}) =-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{\text{p}}}\left[\log\sigma\left( \beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\beta\log \frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\right]$ (3) $$

#### Reward Hacking

Reward model is not perfect due to its limited model size, limited training data, and distribution shift between training data and alignment prompts and responses (Eisenstein et al., ; Gao et al., ; Guo et al., ; ). Several works have been proposed to mitigate reward hacking. They mainly focus on observable artifacts such as length (; Dubois et al., ; Shen et al., ). Shen et al. () propose to enforce the consistency in reward model via data augmentation. Reward hacking can also be mitigated during policy training with post-adjusted reward (Park et al., ) or with post-training model merge (Lin et al., ). We focus on improving the reward model by addressing reward hacking from a causal perspective.

#### Causal Inference

Causal inference can be embedded in graphical model frameworks as a directed acyclic graph (DAG) with causal relationship represented as a directed edge (Pearl, ; Lee et al., ). We say a random vector $X$ to be *faithful* with respect to a DAG $\mathcal{G}=(\mathcal{V},\mathcal{E})$ if for any $i,j\in\mathcal{V}$, and any subset $S\subseteq V\backslash\{i,j\}$,

$$ $X^{i}\perp X^{j}\mid X^{S}\Leftrightarrow\text{$i$ and $j$ are d-separated by $S$ under $\mathcal{G}$},$ (4) $$

where $X^{i}\perp X^{j}\mid X^{S}$ denotes $X^{i}$ and $X^{j}$ are independent conditional on $X^{S}$. The definition of d-separation is as follows: suppose we are given a DAG $\mathcal{G}$; then, for two nodes $i,j\in\mathcal{V}$, a subset $S$ of $V\backslash\{i,j\}$ d-connects $i$ and $j$ if there exists a path $L$ between $i$ and $j$ such that every collider in $L$ either belongs to $S$ or has a descendent in $S$, and no other node in $L$ belongs to $S$. If $S$ does not d-connect $i$ and $j$, then it d-separates $i$ and $j$. See Appendix A.5 for more details.

## 3 Robust Reward Model Training

### 3.1 Causal framework

Figure: Figure 2: Causal graph of reward model. $X$ is the prompt, $Y_{1},Y_{2}$ are two responses, $S$ is the contextual signal that depends on input prompt and two responses. $A$ is the context-free artifact that only depends on two responses. $C$ is the preference label. Traditional reward model cannot differentiate the two DAGs on whether there is a causal edge from $A$ to $C$. Our work uses the augmented dataset to eliminate the edge from $A$ to $C$.
Refer to caption: x2.png

We formulate a DAG $\mathcal{G}$ to model the causal relationships among different quantities (Figure 2). We assume the distribution of $(X,Y_{1},Y_{2},S,A,C)$ to be faithful to the DAG. $S$ is the *sufficient statistic* (Lehmann & Casella, ) that captures the contextual effect that one response fulfills the need of the prompt better than the other. $A$ is the *sufficient statistic* that captures the context-free artifacts that only depend on two responses. Such artifacts can include length, format (bold faces, bullet points, markdown, etc), and certain patterns ($n$-grams such as “Sure, here is the response:”). In traditional reward model training, the model may hack the patterns in $(Y_{1},Y_{2})$. Suppose 80% of winning responses to be longer, then the reward model can get 80% accuracy by just counting the number of tokens. Formally, we construct two hypothesis:

- •
$\mathcal{H}_{0}$: there is no causal edge from $A$ to $C$.
- •
$\mathcal{H}_{1}$: there is a causal edge from $A$ to $C$.

###### Proposition 3.1 .

In traditional reward model training, $\mathcal{H}_{0}$ and $\mathcal{H}_{1}$ are not always distinguishable.

The desired behavior of reward model is to ignore the artifact $A$ in determining the preference label $C$, which corresponds to $\mathcal{H}_{0}$. To achieve that, we can utilize two d-separation relationships of the DAG $\mathcal{G}$.

- •
R1: Under $\mathcal{H}_{0}$, $A$ and $C$ are d-separated by $(Y_{1},Y_{2})$, thus $A\perp C\mid(Y_{1},Y_{2})$.
- •
R2: Under $\mathcal{H}_{0}$, $A$ and $C$ are d-separated by $S$, thus $A\perp C\mid S$.

In addition, we can set the preference rule based on the DAG $\mathcal{G}$. We say response $y$ is *contextual* to $x$ if they are from the same triplet in $\mathcal{D}_{\text{hf}}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\}_{i=1}^{N}$. For example, $y_{w}^{(i)}$ and $y_{l}^{(i)}$ are contextual to $x^{(i)}$, but $y_{w}^{(j)}$ and $y_{l}^{(j)}$ are not contextual to $x^{(i)}$ for $j\neq i$. Then for $(x,y_{1},y_{2})$, we have the following rules:

- •
if both $y_{1}$ and $y_{2}$ are contextual to $x$, we set the winning one in $\mathcal{D}_{\text{hf}}$ as winner.
- •
if only one of $y_{1}$ and $y_{2}$ is contextual to $x$, we set the contextual one as winner.
- •
if neither $y_{1}$ nor $y_{2}$ is contextual to $x$, we set the preference label as “Tie”.

###### Proposition 3.1 .

### 3.2 Data augmentation

To fix the issue mentioned in Proposition 3.1, we can effectively utilize R1&R2. Given $\mathcal{D}_{\text{hf}}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\}_{i=1}^{N}$, we can first expand the dataset as $\tilde{\mathcal{D}}_{\text{hf}}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)},x^{(\sigma_{
1}(i))},y_{w}^{(\sigma_{1}(i))},y_{l}^{(\sigma_{1}(i))},x^{(\sigma_{2}(i))},y_
{w}^{(\sigma_{2}(i))},y_{l}^{(\sigma_{2}(i))}\}_{i=1}^{N}$, where $\sigma_{1}:[N]\rightarrow[N]$ and $\sigma_{2}:[N]\rightarrow[N]$ are two different invertible permutation functions randomly sampled from permutation group $S_{N}$. There are in total $3\times{6\choose 2}=45$ possible $(x,y_{1},y_{2})$ unordered triplets from each element in $\tilde{\mathcal{D}}_{\text{hf}}$. This is because there are 3 possible prompts with 2 choices among 6 responses and we treat $(x,y_{1},y_{2})$ and $(x,y_{2},y_{1})$ as the same one.

From R1, we can fix $(Y_{1},Y_{2})$ and vary $X$ to perturb $C$. From R2, we can fix $C$ by picking a contextual (prompt, response) pair $(X,Y_{1})$ and another non-contextual response $Y_{2}$. Then we set $Y_{1}$ as winning response and vary losing response $Y_{2}$ to perturb $A$. We can see the augmented datasets derived from the above two rules cover all possible $(x,y_{1},y_{2})$ unordered triplets generated from $\tilde{\mathcal{D}}_{\text{hf}}$. For simplicity, we select the ones with prompt $x^{(i)}$, which provides us with the following additional augmented triplets(^2^22We show a sample Python code in Appendix A.3.):

$$ $\displaystyle\begin{split}\left.\begin{aligned} (x^{(i)},y_{w}^{(i)},y_{w}^{( \sigma_{1}(i))})\rightarrow&\text{chosen}=y_{w}^{(i)}\\ (x^{(i)},y_{w}^{(i)},y_{w}^{(\sigma_{2}(i))})\rightarrow&\text{chosen}=y_{w}^{ (i)}\\ (x^{(i)},y_{w}^{(i)},y_{l}^{(\sigma_{1}(i))})\rightarrow&\text{chosen}=y_{w}^{ (i)}\\ (x^{(i)},y_{w}^{(i)},y_{l}^{(\sigma_{2}(i))})\rightarrow&\text{chosen}=y_{w}^{ (i)}\\ (x^{(i)},y_{l}^{(i)},y_{w}^{(\sigma_{1}(i))})\rightarrow&\text{chosen}=y_{l}^{ (i)}\\ (x^{(i)},y_{l}^{(i)},y_{w}^{(\sigma_{2}(i))})\rightarrow&\text{chosen}=y_{l}^{ (i)}\\ (x^{(i)},y_{l}^{(i)},y_{l}^{(\sigma_{1}(i))})\rightarrow&\text{chosen}=y_{l}^{ (i)}\\ (x^{(i)},y_{l}^{(i)},y_{l}^{(\sigma_{2}(i))})\rightarrow&\text{chosen}=y_{l}^{ (i)}\end{aligned}\right\}\text{Non-contextuals}\left.\begin{aligned} (x^{(i)}, y^{(\sigma_{1}(i))}_{w},y^{(\sigma_{1}(i))}_{l})\rightarrow\text{Tie}\\ (x^{(i)},y^{(\sigma_{2}(i))}_{w},y^{(\sigma_{2}(i))}_{l})\rightarrow\text{Tie} \\ (x^{(i)},y^{(\sigma_{1}(i))}_{w},y^{(\sigma_{2}(i))}_{w})\rightarrow\text{Tie} \\ (x^{(i)},y^{(\sigma_{1}(i))}_{w},y^{(\sigma_{2}(i))}_{l})\rightarrow\text{Tie} \\ (x^{(i)},y^{(\sigma_{2}(i))}_{w},y^{(\sigma_{1}(i))}_{l})\rightarrow\text{Tie} \\ (x^{(i)},y^{(\sigma_{1}(i))}_{l},y^{(\sigma_{2}(i))}_{l})\rightarrow\text{Tie} \end{aligned}\right\}\text{Neutrals}\end{split}$ (5) $$

“Non-contextuals” set the contextual response as chosen and non-contextual one as rejected. “Neutrals” set both non-contextual responses as tie. With these, we have the following claim:

###### Proposition 3.2 .

If the reward model is trained with $\mathcal{D}_{\text{hf}}$ and augmented triplets in Equation 5, there is no causal edge from $A$ to $C$ in DAG $\mathcal{G}$.

###### Proposition 3.2 .

### 3.3 Connection to existing works

#### ODIN ( Chen et al., )

ODIN decomposes reward into additive format of a quality one and a length one. During learning, it enforces the disentanglement between the quality reward and the response length and encourages the correlation between the length reward and the response length. We claim that this is a special case of our causal modelling with single observed artifact $A$ as length, because the disentangle learning is a necessary condition of the conditional independence between $C$ and $A$ given the data. Our framework is more general and can go beyond single and observed artifact.

#### Length-controlled AlpacaEval-2 (Dubois et al., 2024a )

This work improves the original version of AlpacaEval-2 by conditioning on the length through Controlled Direct Effect (VanderWeele, ). It adds length as a variable in the logistic regression to predict the preference. Effectively, it learns the residual part that cannot be explained by the length. In our framework, we directly learn the residual part that is orthogonal to the artifacts, which is the length in length-controlled AlpacaEval-2. Thus the two methods are equivalent, and our approach can go beyond single artifact and be extended to unobserved artifacts.

#### Length-controlled DPO (Park et al., 2024 )

This work adds a length penalty in the RLHF objective (Equation 2). It serves as a post-hoc reward adjustment to mitigate the length bias during policy optimization. The idea for removing the lengthy bias using a length reward is the same as ODIN, but they don’t have the correlation penalty and the additional hyperparameter introduced can add more complexity into policy optimization. In comparison, our work directly learns a artifact-free reward model so we do not need an explicit length adjustment factor in the alignment algorithm designs.

#### Contrast Instructions (Shen et al., 2023a )

This work shows the issues of reward models on the instruction and response consistencies when switching instruction or response to another similar one. It proposes a data augmentation training approach and retrieval-augmented inference technique to improve the consistencies of reward models. On contrary, by considering all possible combinations of $(x,y_{1},y_{2})$ across difference examples, our approach uses the organic data from the dataset, which can effectively eliminate the artifacts existing in the dataset.

## 4 Experiments

In this section, we conduct reward modeling and apply the trained reward to downstream alignment tasks to verify the effectiveness of the proposed method. For deeper understanding, we also conduct analysis on reward model training data, aligned policies, and perturbed reward model training data.

### 4.1 Settings

#### Training Set

We study RRM using the preference dataset curated by RLHFlow(^3^33[https://huggingface.co/datasets/RLHFlow/pair_preference_model_dataset](https://huggingface.co/datasets/RLHFlow/pair_preference_model_dataset)) (Dong et al., ), which has been used to train a series of strong open-source preference models as evaluated by the Rewardbench (Lambert et al., ). The dataset consists of 700K preference pairs, which is a mixture of HH-RLHF (Bai et al., ), SHP (Ethayarajh et al., ), HelpSteer (Wang et al., ), PKU-SafeRLHF (Ji et al., ), UltraFeedback (Cui et al., ), UltraInteract (Yuan et al., ), Distilabel-Capybara (Daniele & Suphavadeeprasit, ), and Distilabel-Orca (Lian et al., ). See Appendix A.2 for details of dataset composition.

#### Reward Model Training Details

We first train a pairwise ranking reward model (RM) from Gemma-2-9b-it. With the augmentation illustrated in Equation 5, we can get 14X additional examples, most of which can be too easy for RM to learn. To reduce the augmented data size, we first conduct inference on random 50% of the augmented data using the trained RM, and leave the examples with $|\hat{\mathbb{P}}(A\succ B)-\mathbb{P}^{*}(A\succ B)|\geq 0.2$, where $\hat{\mathbb{P}}(A\succ B)$ is winning probability calculated by RM and $\mathbb{P}^{*}(A\succ B)$ is the ground truth probability(^4^44The ground truth probability is 1 if A is preferred over B, 0 if B is preferred over A, and 0.5 if they tie.). We get 2.4M training examples by merging the filtered augmented data and original RM training data. Then we use the same training recipe to get the robust reward model (RRM). We train the reward models until convergence using AdamW (Loshchilov, ) optimizer with learning rate 1e-6 and batch size 128(^5^55We pick the best hyperparameter by grid search.).

#### Policy Model Training Details

We train DPO policies using the on-policy responses generated by Gemma-2-9b-it and labeled by RM and RRM, respectively. We use the prompt set from the UltraFeedback dataset to generate 5 responses per prompt. Then, we compare all ${5\choose 2}$ pairs and pick the best-worst response pairs to align the DPO policy following (Pace et al., ; Dong et al., ). We train the policies for 2 epochs at most using AdamW (Loshchilov, ) optimizer with learning rate 2e-7 and a global batch size of 128, where the batch size follows Dong et al. () and the learning rate is decided by grid search.

#### Evaluation Metrics

We evaluate the quality of reward model from two perspectives: the accuracy on Reward-Bench (Lambert et al., ) and the quality of policies induced by the reward model. For policies induced by the reward model, we consider two variants: 1. Best-of-N (BoN) policy and 2. aligned DPO policy. Our main focus is for open-ended generation and we use MT-Bench (Zheng et al., ) and AlpacaEval-2 (Dubois et al., ) to evaluate.

### 4.2 Main Results

#### Reward Model Accuracy

The test accuracies on Reward-Bench are reported in Table 1. RRM improves “Chat Hard” and “Safety” by a clear margin but sacrifices the “Reasoning”. Regarding “Reasoning”, we hypothesize that math and coding are less affected by the non-contextual artifacts and we may use other rewards than an LLM because those are objectives like golden answers. On average, RRM improves RM by an absolute 3.54% accuracy gain.

**Table 1: Comparison of test accuracy of Reward-Bench. RRM shows improvement upon RM on Chat Hard and Safety with an average 3.54% improvement of accuracy.**
| Model | Chat | Chat Hard | Safety | Reasoning | Average |
| --- | --- | --- | --- | --- | --- |
| RM | 97.77 | 51.54 | 78.54 | 94.58 | 80.61 |
| RRM | 96.51 | 65.57 | 83.90 | 90.62 | 84.15 |

#### Policies Induced by Reward Models

We investigate the quality of reward models by evaluating the aligned policies. To study the effect of adding “Neutrals” in Equation 5, we also train a reward model without augmented neutrals (-Neutrals). The results are summarized in Table 2. As expected, ODIN ()(^6^66Training details in Appendix A.4.) shows shorter responses than RM and RRM since it explicitly disentangles the length from quality. RRM shows the best performance on both benchmarks over ODIN and RM, with shorter responses generated than RM, suggesting it effectively controls the length as one of the artifact. The added “Neutrals” have slight improvements on first-turn MT-Bench and AlpacaEval-2.

**Table 2: Comparison among different reward models on various aligned policies. T1 and T2 stand for the first and second turn of the conversation, respectively. WR stands for win-rate against GPT-4. LC stands for length-controlled win-rate. Length is the average number of characters in the generated responses. RRM shows quality improvements over ODIN and RM with shorter responses than RM. Dropping augmented neutral examples slightly hurt the quality.**
| Reward | Policy | MT-Bench(^7^77we do not evaluate BoN policies on MT-Bench because it involves multi-turn.) | AlpacaEval-2 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | T1 ($\uparrow$) | T2 ($\uparrow$) | Overall ($\uparrow$) | LC (%) ($\uparrow$) | WR (%) ($\uparrow$) | Length ($\downarrow$) |
| RM | BoN (N=8) | - | - | - | 36.87 | 50.14 | 3072 |
| RRM | BoN (N=8) | - | - | - | 47.68 | 53.19 | 1770 |
| RM | BoN (N=64) | - | - | - | 40.52 | 57.62 | 2992 |
| RRM | BoN (N=64) | - | - | - | 62.82 | 63.03 | 1770 |
| RM | DPO | 8.02 | 6.33 | 7.27 | 33.46 | 41.07 | 2416 |
| ODIN | DPO | 8.66 | 8.13 | 8.39 | 48.29 | 37.13 | 1559 |
| RRM | DPO | 8.70 | 7.87 | 8.31 | 52.49 | 43.31 | 1723 |
| -Neutrals | DPO | 8.65 | 8.21 | 8.44 | 51.73 | 43.24 | 1722 |

### 4.3 Length Analysis

To further understand the artifacts learned in reward model, we take length as an example to analyze the reward model training data and aligned policy.

#### Length distribution of training data

We study the length (number of tokens) distribution of reward model training datasets. Length is one common artifact that shows bias on both policy training and evaluation. We hypothesize that the bias can possibly come from the reward model training data. The one used in training RM is not well calibrated and the chosen responses are longer on average (Figure 3(a) and 3(b)) and by frequency (Figure 3(c)). On contrary, the RRM training data is better calibrated with length more balanced between chosen and rejected responses in each length bin (Figure 3(a) and 3(b)). We further provide length analysis for each data source in Appendix A.2.

Figure: (a) Histogram of response lengths in RM training data.
Refer to caption: extracted/5867167/figures/rm_length.png

#### Length distribution of policies

To understand the lengthy bias learned in various policies, we also study the length distribution of generated responses on AlpacaEval-2’s (Dubois et al., ) prompts (Figure 4). We observe that the policies induced by RRM generate shorter responses than RM, which implies the correction of lengthy bias by RRM.

Figure: (a) Best of 8 responses
Refer to caption: extracted/5867167/figures/bo8_length.png

### 4.4 Deliberately designed artifacts

#### Artifacts

To verify that our proposed method is able to eliminate artifacts, we artificially added an artifact to the chosen responses in reward model training data. More specifically, we add prefix “Sure, here is the response: ” to the chosen responses with probability 0.1. We train an RM and RRM on the modified reward model training data, respectively.

To test the effect of reward model on the policy model, we first sample $N$ responses from Gemma-2-9b-it model using the AlpacaEval-2 prompts. Then we add the same type of artifact to each response with probability $p_{a}=p$, where $p\in\{0.05,0.1,0.2,0.5\}$. Under this setting, RM trained on the artifact-added data would prefer responses with the artifacts since the chosen responses come with artifacts, even if the responses may contain low-quality answer. RRM is expected to be more robust to the artifact. To verify this, we construct BoN policies using RM and RRM, respectively.

As expected, Figure 5 shows that after adding the artifacts, the BoN policies induced by RRM are more robust than RM to artifacts injected in the responses.

Figure: (a) Best of 8 responses
Refer to caption: extracted/5867167/figures/artifact_8.png

## 5 Related Works

#### RLHF algorithms

The first RLHF framework is based on the proximal policy optimization (PPO) algorithm, which was first popularized by Christiano et al. () and further developed by Bai et al. (); Ouyang et al. (). However, getting the deep RL method (PPO) work is challenging especially in the era of LLMs (; Engstrom et al., ). In recognition of this issue, another line of works proposes direct alignment algorithms, where notable examples include SLiC (Zhao et al., ), DPO (Rafailov et al., ), IPO (Azar et al., ), KTO (Ethayarajh et al., ), ORPO (Hong et al., ), SimPO (Meng et al., ), and DRO (Richemond et al., ). These algorithms directly optimize a supervised target to optimize the policy model without constructing a reward model first, hence the name direct alignment algorithms. However, these algorithms learning from a fixed dataset are offline and often off-policy without further exploration of the environment. RSO (Liu et al., ) emphasizes the importance of reward model and fixes the distribution shift problem to improve the DPO training, followed by list-wise alignment (Liu et al., ) and the online (iterative) training frameworks (; Guo et al., ; ). Alternatively, there is also a line of work based on the best-of-n sampling, such as RAFT (), BOND (Sessa et al., ), BoNBoN alignment (Gui et al., ). These algorithms leverage a reward model to rank the generated responses and distill knowledge from the best responses. Our approach can benefit RLHF algorithms relying on a reward model.

#### Reward Models & Reward Hackings

Building a superhuman/unbiased reward model is vital for training better chat assistants since it could affect the upper bound of the policies’ capabilities in the online preference optimization (Wang et al., ; Bai et al., ).
Multi-objective rewards (Wang et al., ), RLHF-workflow (Dong et al., ), and RMBoost (Shen et al., ) are proposed to train more capable reward models.
While revealed by Denison et al. (); Zhang et al. (), reward models are easily hacked by different pattern in different scenario, e.g., length (Singhal et al., ) and sycophancy. Recent works employ the model merging (WARP (Ramé et al., ) and WARM (Ramé et al., )), and hacking reward decomposition (ODIN ()) to mitigate the hackings in online RLHF. Generative reward models can provide more detailed preference analysis (Yan et al., ). For the most accurate reward signal, one can also use verifiable answers in certain domain like math (Xiong et al., ).
Most model-based methods failed to distinguish between preferences driven by the prompt and context-free artifacts.
Our RRM is more advanced in removing the artifacts.

## 6 Conclusion

In this paper, we identified a key problem in the current reward training methodology: its inability to differentiate between contextual signals and context-free artifacts. Using a causal framework, we explained this effect and improved reward model training by introducing a data augmentation approach derived from the framework. Our theoretical analysis and extensive empirical results demonstrated that the proposed techniques effectively enhance both the test accuracy of the reward model and the quality of the policies it induces. Future work will explore filtering augmented pairs and matching artifacts when constructing response pairs, further refining the training process.

## Appendix A Appendix

### A.1 Proofs

#### Proof of Proposition 3.1

###### Proof.

We show the conclusion with a special case. Assume $S=s(X,Y_{1},Y_{2})+\epsilon_{s}$ with $\epsilon_{s}\sim N(0,\sigma_{s})$, and $A=a(X,Y_{1},Y_{2})+\epsilon_{a}$ with $\epsilon_{a}\sim N(0,\sigma_{a})$. $\mathbb{P}(C=1)=\sigma(\beta_{s}S+\beta_{a}A+\alpha+\epsilon_{c})$ with constants $\beta_{s},\beta_{a},\alpha\in\mathbb{R}$ and random error $\epsilon_{c}\perp(S,A)$. If $\beta_{a}=0$, then $\mathcal{H}_{0}$ is true. If $\beta_{a}=1$, then $\mathcal{H}_{1}$ is true. In extreme case that $Corr(S,A)=1$, then $A=\beta_{as}S+\alpha_{a}$ for some constants $\alpha_{a}\in\mathbb{R}$ and $\beta_{as}\in\mathbb{R}^{+}$. Then the model cannot tell if $\beta_{a}=0$ or not. This is because when $\beta_{a}=0$, we can still reparametrize it as $\mathbb{P}(C=1)=\sigma((\beta_{s}-\beta_{as})S+A+(\alpha-\alpha_{a})+\epsilon_
{c})$.
∎

###### Proof.

#### Proof of Proposition 3.2

###### Proof.

We can prove this by contradiction. If there is a causal edge from $A$ to $C$, then the conditional independence relations $A\perp C\mid(Y_{1},Y_{2})$ and $A\perp C\mid S$ do not hold, which contracts to the triplets constructed on “Non-contextuals” and “Neutrals”.
∎

###### Proof.

### A.2 Reward Model Training Datasets

We reuse the datasets in RLHFlow (Dong et al., ). We list the data sources and number of examples in Table 3. The authors of the original paper delete the samples with similar scores when the scores are available because when the model is well calibrated, these samples are more likely to mislabelled. Thus the total number is smaller than the sum of each individual datasets. We show the length distribution of chosen and rejected responses from each individual data source in Figure 6.
HH-RLHF, SHP, HelpSteer, UltraFeedback show bias towards rejected responses as the first length bin. SHP, HelpSteer, and UltraFeedback have longer chosen responses than rejected ones on average.

**Table 3: Composition of reward model training dataset**
| Source | Number of Examples |
| --- | --- |
| HH-RLHF-Helpful(^8^88[https://huggingface.co/datasets/RLHFlow/HH-RLHF-Helpful-standard](https://huggingface.co/datasets/RLHFlow/HH-RLHF-Helpful-standard)) (Bai et al., ) | 115,396 |
| SHP(^9^99[https://huggingface.co/datasets/RLHFlow/SHP-standard](https://huggingface.co/datasets/RLHFlow/SHP-standard)) (Ethayarajh et al., ) | 93,301 |
| HelpSteer(^10^1010[https://huggingface.co/datasets/RLHFlow/Helpsteer-preference-standard](https://huggingface.co/datasets/RLHFlow/Helpsteer-preference-standard))(Wang et al., ) | 37,131 |
| PKU-SafeRLHF(^11^1111[https://huggingface.co/datasets/RLHFlow/PKU-SafeRLHF-30K-standard](https://huggingface.co/datasets/RLHFlow/PKU-SafeRLHF-30K-standard)) (Ji et al., ) | 26,874 |
| UltraFeedback(^12^1212[https://huggingface.co/datasets/RLHFlow/UltraFeedback-preference-standard](https://huggingface.co/datasets/RLHFlow/UltraFeedback-preference-standard)) (Cui et al., ) | 340,025 |
| UltraInteract(^13^1313[https://huggingface.co/datasets/RLHFlow/UltraInteract-filtered-standard](https://huggingface.co/datasets/RLHFlow/UltraInteract-filtered-standard)) (Yuan et al., ) | 161,927 |
| Distilabel-Capybara(^14^1414[https://huggingface.co/datasets/RLHFlow/Capybara-distibalel-Filter-standard](https://huggingface.co/datasets/RLHFlow/Capybara-distibalel-Filter-standard)) (Daniele & Suphavadeeprasit, ) | 14,811 |
| Distilabel-Orca(^15^1515[https://huggingface.co/datasets/RLHFlow/Orca-distibalel-standard](https://huggingface.co/datasets/RLHFlow/Orca-distibalel-standard)) (Lian et al., ) | 6,926 |

Figure: (a) HH-RLHF-Helpful
Refer to caption: extracted/5867167/figures/hh_rlhf.png

### A.3 Data augmentation python code

In Algorithm 1, we show a sample code of data augmentation in Python. We expect each element in data contains “context”, “response_w”, “response_l”. We use “neutral” to indicate if the label should be “tie”.

Figure: Algorithm 1 Example Python Code for Data Augmentation

### A.4 Training details for ODIN

We use the same loss as described in .
We train Gemma-2-9b-it for 1 epoch on the same data we used for RM.
AdamW is our optimizer and the learning rate is set to 2e-6 with cosine scheduler.
We use Flash-Attention to accelerate the training while applying the Deepspeed Zero-Stage 3 to get batch_size=16 on each GPU (the global batch size is 128) to make sure the calculation of the Pearson correlation between the head value and the length of the responses is stable.

### A.5 Preliminaries in Causal inference

We list a few critical concepts in this section. For information, we refer the readers to Lauritzen () and Pearl ().

#### DAGs and d-separation

A DAG is a set of vertices and a set of directed edges (arrows) that connect pairs of these vertices. The causal modeling connects a DAG with Markov condition via a graphical relation called *d-separation* (Pearl, ). D-separation is a relation among three disjoint sets of vertices in a directed graph. D-separation and Markov condition connect DAGs and probability distribution. By *faithfulness* assumption, the d-separation in a DAG is equivalent to conditional independence in distribution.

#### The Causal Markov Condition

The Causal Markov assumption assumes that a variable $X$ is independent of every other variable (except $X$’s effects) conditional on all of its direct causes. With this, a DAG defines a set of distributions of the form

$$ $p(y_{1},...,y_{k})=\prod p(y_{j}|\text{parents}(y_{j}))$ $$

#### Counterfactuals

Consider two variables $X$ and $Y$. We will call $X$ the “treatment”. We call $Y$ the “outcome”. For a given subject we see $(X_{i},Y_{i})$. What we don’t see is what their value of $Y_{i}$ would have been if we changed their value of $X_{i}$. This is called *counterfactual*. Suppose that $X$ is a binary variable that represents some treatment. So $X=1$ means the subject was treated and $X=0$ means the subject was not treated. Let $Y_{1}$ denote the outcome if the subject is treated. Let $Y_{0}$ denote the response if the subject is not treated. Then

$$ $Y=XY_{1}+(1-X)Y_{0}$ $$

If we treat a subject, we observe $Y_{1}$ but not $Y_{0}$. The unobserved variable is called a *counterfactual*. The variables $(Y_{0},Y_{1})$ are also called *potential outcomes*.
We define *mean treatment effect* as

$$ $\theta=\mathbb{E}(Y_{1})-\mathbb{E}(Y_{0})=\mathbb{E}(Y|\text{set }X=1)- \mathbb{E}(Y|\text{set }X=0)$ $$

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: The method proposed in this paper to 'disentangle prompt-driven preferences from prompt-independent artifacts' seems to only eliminate biases unrelated to the input while preserving input-related preferences. This can effectively reduce bias under the 'helpful' alignment objective. However, if our alignment goal is 'safety', the response might be prompt-independent, typically just refusing to reply. Yet humans may still have preferences for different ways of refusal. In this case, how can we disentangle prompt-independent preferences from prompt-independent artifacts?
soundness: 3
strengths: This paper addresses an important issue in preference datasets: how to disentangle prompt-driven preferences from prompt-independent artifacts.
summary: Traditional reward model (RM) training for large language models (LLMs) struggles to separate prompt-driven preferences from irrelevant artifacts like response length and format. This work introduces a causal framework and data augmentation technique to remove these artifacts, resulting in a robust reward model (RRM) that improves performance across benchmarks, enhancing accuracy and alignment scores significantly.
weaknesses: While the problem studied in this paper is significant, the proposed solution is relatively straightforward, focusing only on expanding the dataset. To remove prompt-independent artifacts, the dataset size needs to be several times larger, increasing the training cost. Additionally, as shown in Figure 5, the effect of this costly augmentation is not very significant—after inserting 30% artifacts, the RRM only decreases from the original RM’s 25% to 20%.

Regarding Table 1, the reasoning performance declines, and the authors explain this as due to math and coding tasks being less affected by non-contextual artifacts. This raises the question of whether the proposed method might have a negative impact on attributes unaffected by non-contextual artifacts.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - How did the authors tune the hyperparameters?
- What are the “three possible prompts” described in line 226?
soundness: 3
strengths: - The authors study a very relevant and timely problem.
- The paper shows significant improvement in both reward modeling accuracy and in downstream policy performance.
- The authors do a really interesting study where they artificially add artifacts to the preference dataset, and find their method is more robust to it.
summary: This paper aims to learn reward models that are unbiased by “artifacts” that come from human preference data: namely length biases and formatting. To do so, they propose to augment the RLHF preference dataset with preference pairs where one sample is in response to the prompt, and the other example is not in response to the prompt. They evaluate their learned reward model on reward bench and use it to improve downstream policy performance.
weaknesses: - The main weakness of this paper is that the experiments are not that comprehensive. Although the results are strong, the experiments are only conducted on one dataset, with one model, and with one random seed. Therefore we cannot tell if the results are statistically significant or generalizable.
- The baseline is pretty weak with regards to reward bench. RLHFFlow with Llama 3 8B gets 87.1 on reward bench, while the author’s baseline gets 80.
- The writing describing the methodology not that clear. I think more plainly describing the data augmentation would be better.
- The data filtering step used in their method hurts their experiments interpretability, as it may be the case that the sole reason RRM gets good performance is due to the data filtering step. They should include a baseline where the original dataset is filtered, and then a RM is trained based on that.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: - The same augmentation and disentangle technique can be applied to learn a artifact only model. Some of the artifacts would improve the quality of the answer, if there is a consensus preference on such artifacts in the labelers. Is there any proposal to combine the contextual & non-contextual models? Would adding these two reward models work, or do we need to take the min of these two models? Any suggestions from the authors?
soundness: 4
strengths: - The causal framework has been effective and widely used in other problems, but this is the first time I see it applied in LLM reward modeling, and it very well suits the problem under consideration.
- The implementation is very straightforward, just augment the data, no empirical tricks.
summary: This paper presents a method to enhance the robustness of the reward model for LLM alignment. More specifically, rewards models are trained based on the question and the human labelers' preference over the answer pairs. A commonly observed un-robust behavior of the reward model is that there are spurious correlations between the label and the artifacts in the answers which is not relevant to the question. The existence of such spurious correlations causes the reward model to favor certain types of artifacts, e.g. the length of the answer.

In previous works, some of the artifacts are specifically dealt with, like introducing a penalty for the length of the answer. However, the solutions lack a principle and can not handle other artifacts like the word preference and style of the answers. In this work, a systematic approach is proposed based on a causal framework, which is effective in debiasing the reward model from various artifacts.
weaknesses: - It seems suboptimal to me to train the reward model that is only dependent on the S, because the A part could contribute nontrivially to the reward function, in the case where both answers are equal in terms of the contextual quality, the labeler could prefer the answer which is organized as bullet points for readability.
- The explanation on why "reasoning" is performing worse under RRM is not quite convincing. Even if they are less affected by non-contextual artifacts, the augmented data should not have caused the degrade of performance either (as the augmented data are simpler pairs). I would suggest two other hypothesis to investigate:
  - Look into how "-Neutral" works on "Reasoning" as requiring 50/50 preference over some random pairs could be too strong.
  - Try to add back the "A only" reward model, as mentioned above the artifact could contribute nontrivially to the reward, removing it entirely could be detrimental especially for math and code, where a high quality answer should be good in formatting which is noncontextual.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Explain why it is needed to add neutrals and set both non-contextual responses as tie in Section 3.2. What is the necessity of this step?
2. In Table 2, why the performance of RRM is not consistent in two chat datasets such as Chat and Chat Hard? In Chat dataset, RRM outperforms RM while in Chat Hard dataset, RRM is even worse.
soundness: 1
strengths: Reward hacking is an important problem in LLM alignment. This paper introduces causal inference framework in reward modeling to mitigate reward hacking, which offers an interesting perspective. This idea makes a lot of sense since the essence of reward hacking mitigation is to find out the irrelevant artifacts that have no causal relationship with the preference label, and causal inference algorithms just offer a useful tool. The idea is clearly presented and illustrated. The experiment results verify the effectiveness of proposed method.
summary: This paper proposes a robust reward model (RRM) training method to mitigatge reward hacking by utilizing causal inference framework. A causal graph for human preference modeling is introduced to enable the model to distinguish contextual preference signals and context-free artifacts. The training data is augmented by re-organizing the (prompt, positive, negative) triplet guided by the causal inference framework. Experiments show that the proposed approach can filter out undesirable artifacts in reward model training and yields a more robust reward model. Specifically, the reward model is trained on Gemma-2-9b-it, and RRM improves RM by an absolute 3.54% accuracy gain on Reward-Bench. Experiments also demonstrate the policy induced by RRM outperforms RM and ODIN in MT-Bench and AlpacaEval-2 benchmark. Analysis show that length artifact and deliberately designed artifact can be eliminated.
weaknesses: The experiments are insufficient. First, the experiments in this paper only cover two possible artifacts, namely length and a deliberately designed phrase, which is not compatible with author's claim that this method should be capable of handling "all potential exploitation patterns". More patterns such as format and emojis should be investigated in the experiments. Second, the reward model is trained only using Gemma-2-9b-it. More experiments should be conducted on different model types such as llama, and larger model sizes such as 57B and 72B. It is in doubt that when the model size increases, the reward hacking problem can be naturally mitigated and the effectiveness brought by this method may diminish. Third, in terms of policy evaluation, this method is only verified in DPO training whereas various other methods should also be considered such as PPO, whose performance relies heavily on reward model quality. Last, the author is suggested to add more baselines in experiment, such as reward model ensemble which can also alleviate reward hacking. 

The data augmentation method is trivial. The new combination is derived by randomly selecting answers from other prompts. Since this negative sample is not related to the prompt with high probability, it is too easy for the model to distinguish the positive and negative answers. The author is suggested to incorporate some hard negatives to make the training more robust.

Meta-review: This paper proposes a robust reward model (RRM) training method to mitigate reward hacking by leveraging a causal inference framework. A causal graph for human preference modeling is introduced to help the model distinguish between contextual preference signals and context-free artifacts. Guided by the causal inference framework, the training data is augmented by reorganizing the (prompt, positive, negative) triplet. Experiments demonstrate that the proposed approach effectively filters out undesirable artifacts during reward model training, resulting in a more robust reward model. Specifically, when training the reward model on Gemma-2-9b-it, RRM achieves an absolute 3.54% accuracy improvement over RM on Reward-Bench. Furthermore, policies induced by RRM outperform those trained with RM and ODIN on the MT-Bench and AlpacaEval-2 benchmarks. Analysis also reveals that artifacts such as length bias and deliberately designed artifacts can be effectively eliminated.

The strengths of this work lie in:
- Its focus on addressing a critical issue in reward model training.  
- The novel application of a causal inference framework to tackle this issue.  

Given these two strengths, I recommend acceptance at the current stage, albeit with moderate confidence, due to the following reasons:  
- Almost all reviewers raised concerns about the generalization of the proposed method to more capable LLMs, as experiments on LLMs other than Gemma or larger-scale models are missing. While the authors added results for Gemma-2-2b-it in the Appendix, they are strongly encouraged to include additional experiments on larger-scale LLMs before the camera-ready submission.  
- Both Reviewer gp5Y and Reviewer jPJZ expressed concerns about the reasoning performance drop observed with RRM. Although the authors explained this is due to the test set, it is strongly recommended that they include results from another test set to further evaluate RRM's reasoning performance before the camera-ready version.  

Providing more comprehensive empirical results rather than limiting the work to a proof-of-concept would significantly broaden the scope and appeal to a wider audience.

Decision: Accept (Poster)