RRM:  Robust Reward Model Training Mitigates Reward Hacking
Authors: Tianqi Liu
Wei Xiong
Jie Ren
Lichang Chen
Junru Wu
Rishabh Joshi
Yang Gao
Jiaming Shen
Zhen Qin
Tianhe Yu
Daniel Sohn
Anastasia Makarova
Jeremiah Zhe Liu
Yuan Liu
Bilal Piot
Abe Ittycheriah
Aviral Kumar
Mohammad Saleh
Venue: ICLR 2025 Poster (2025)
Keywords: Reward model
RLHF
Alignment
Primary area: foundation or frontier models, including LLMs
TL;DR: The first paper to point out the reason for reward hacking from a causal perspective.

Abstract: Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them.  Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on Reward-Bench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.

Published as a conference paper at ICLR 2025
RRM: ROBUST REWARD MODEL TRAINING MITI- GATES REWARD HACKING
Tianqi Liu'; Wei Xiong" Jie Ren', Lichang Chen Junru Wu², Rishabh Joshi', Yang Gao Jiaming Shen', Zhen Qin', Tianhe Yu', Daniel Sohn', Anastasiia Makarova', Jeremiah Liu?, Yuan Liu', Bilal Piot', Abe Ittycheriah Aviral Kumar', Mohammad Saleh Google DeepMind', University of Illinois Otbana-Champaain', University of Maryland, College Park
Junru
ABSTRACT
Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such response length and for- mat. In this work, we expose fundamental limitation of current RM training methods, where RMs fail effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we intro- duce causal framework that learns preferences independent of these artifacts and propose novel data augmentation technique designed to eliminate them. Ex- tensive experiments show that our approach successfully filters out undesirable artifacts, yielding more robust reward model (RRM). Our RRM improves the performance of pairwise reward model trained on Gemma-2-9b-in on Reward- Bench, increasing accuracy from 80.61% to 84.15%. Additionally, DPO policies using both the RM and RRM, demonstrating that the RRM signifi- cantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.
train two
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) has become cornerstone in aligning large language models (LLMs) with human preferences to produce responses that are more helpful, hon- est, and harmless (Ouyang et al., 2022; Bai et al., 2022a). This approach involves training a reward model (RM) on human feedback, which then guides the LLM to generate high-quality responses through reinforcement learning. The success of RLHF is evident in various AI systems, such as Gemini (Team et al., 2023) and GPT-4 (Achiam et al., 2023). Despite its effectiveness, RLHF faces the fundamental issue of reward hacking (Gao et al., 2023), where the model maximizes the reward function without truly aligning with the intended human preferences. This hacking issue occurs be- cause the RM, while powerful tool, is an imperfect proxy for human judgment and often struggles with out-of -iit--i--itttittionn generalization (Eisenstein et al., 2023).
The reward hacking problem manifests in several ways, with verbosity being a common issue: LLMs tend to generate longer responses to appear more detailed or explanatory, exploiting human raters' bias towards lengthier content (Shen et al., 2023b; Singhal et al., 2023). In recognition of this challenge, extensive efforts have been made in the literature. ODIN (Chen et al.) designs a two- head approach to learn the quality reward that is orthogonal to length. Similarly, length-controllee Alpaca (Dubois et al., 2024a) estimates the controlled direct effect (VanderWeets 2011) through logistic regression by adjusting the length. To mitigate the length bias, an improved version (Park et al., 2024) of DPO (Rafailov et al., 2024) introduces length as penalty to the reward score. In prac- tice, there are more reward hacking patterns beyond length, such as format (markdowns, bold-faces) and patterns (certain n-grams or emojis). This is largely due the large output space of language with limited preference data, as well as the diverse and subjective nature of human preferences.
Conrespondence to Tiangi Liu, t tianqi liluuue google.coo.. Work done during an internship Google DeepMind.

---

Published as a conference paper at ICLR 2025
MODEL TRAINING
INFERENCE
INSTRUCTION PERMUTATIONS
Original
1eat for
Contextuals
"his Aue comper
Paporse Inglccto)
Chean
kopal
Anquuste (thoows)
Neutrals
Ansaenee (vloso))
H
that
Thases Troessscny DALL"
X Venttel
%
X
Aopanu (chewer
Xi
X
X
x
Requass (ojkared
Loy Esteee
Requreo Injected
H
x
x
yF
X vo00E pousr
x
Figure 1: The pipeline of our proposed robust reward model (RRM), which aims to decouple contex- tual preference quality signal and context-free artifacts. Suppose a proportion of chosen responses have certain artifact (bold-face wrapped with "**" in this figure), the reward model can hack the pattern and choose the response with the artifact instead of carefully reading the prompt. With our data augmentations, we can effectively balance the context-free artifacts in chosen and rejected responses, thus ensuring more robust reward model during inference.
It is challenging to identify and mitigate all potential exploitation patterns. We may consider the causal perspective to explain this phenomena. Given a prompt and a pair of responses (31, (2). the human preference can be caused by the real quality a(x,, v1, 122 that is associated with the prompt, or by the context- artifacts a(y),y2) in the responses that do not depend on prompt. Traditional reward model training cannot differentiate the above two factors. There First, the pair of responses are always contextual and on-topic to the prompt, thus no counterfactual prompt (prompt from another examples) is used. The reward model may learn the artifacts existing in the responses by ignoring the prompt. If we use the counterfactual prompt, it can help estimate the level of artifact bias (P(1)1 > y2(2²)) with x' * x)) existing in the preference dataset (Zhao et al., 2021). Second, even if we adjust a few common artifacts, not all artifacts are observable and thus there is no easy way to control all the artifacts explicitly answer the question "what will the preference be if both responses share the same artifacts?".
two reasons for this.
In response to these challenges, we propose simple and effective method improve reward model- ing. We first formulate the reward model training in a causal framework, then we augment the reward model training data based on the causal rules. By doing so, we can effectively adjust the artifacts and only learn the real quality. Our pipeline is illustrated in Figure 1, where we augment the reward model training data by using responses from other examples to effectively balance the artifacts in chosen and rejected responses. To summarize, the contributions of this paper are three-fold:
We identify key issue with traditional reward model training: it often fails to distinguish between genuine contextual preference signals and spurious context-free freee artifacts.
To address this, we propose a causal graph for human preference modeling and introduce data augmentation to mitigate artifacts learned by the reward model.
We further demonstrate that policies trained on these robust reward models consistently outperform those based on baseline reward models.
2 PRELIMINARIES
In preference learning, we assume that there exists preference oracle that determines the probability P(y > y2(2)) that response y1 is preferred over y2 given the prompt v. Our goal is to optimize the preference by querying the preference oracle within certain budget constraint. In what follows, we first review the major ways to approximate and estimate the oracle based on a human preference
[ dataset Drr = (x()), yoe Mi Ji=1- where z(() represents prompt for example i, and (y vi represents the chosen and rejected response, respectively.
2

---

Published as a conference paper at ICLR 2025
Reward models Bradley-Terry pointwise reward model (Bradley & Terry, 1952; Ouyang al., 2022) is a widely adopted method, which additionally assumes that there exists a reward function r(z,y) 2) R and the preference oracle satisfies
exp(r(x, vi))
P(y1 > v21z) exp(r(x, + exp(r(a, = (r(2, vi) (2,82)).
Then, fit the Bradley-Terry model by maximizing the log-likelihood on the training set:
C(r Par) = (x......))) [log a (r (a,))) T (zx
(1)
The second predominant approach is the pairwise ranking model (Zhao et al., 2023; Jiang et al., 2023), which takes a prompt and pair of responses as the input, and directly predicts the probability F(y1 > y2 2)) which subsumes the BT model as subclass. In the literature, the pairwise preference model has shown to outperform pointwise BT reward both empirically (Zhao et al., 2023; Jiang et al., 2023; Dong et al., 2024) and theoretically (Ye et al., 2024) due to its flexibility and larger function class capacity. Specifically, we denote the pairwise ranking model and leverage the next token prediction ability of the language model to format the sample as:
"[CONTEXT] {æ} [RESPONSE A] {Fi} [RESPONSE B] {32)"
Then, the model outputs either "A" "B" as preferred one. We use the probability of decoding "A" as estimation the preference probability P(y: > y2)=))" In this work, we use the pairwise ranking model for its superior performance and flexibility.
Alignment Algorithms Start with a reward function r(x, y). a reference policy Tul and input prompt distribution P, a policy is trained to optimize for the following objective:
max Eaxp [By= (z, y) - 3DKL [ [+(([=]](
(2)
where > is the KL penalty coefficient. Several algorithms have been proposed to solve the above optimization, including PPO (Schulman et al., 2017: Ziegler et al., 2019), SLiC (Zhao et al., 2023), DPO (Rafailov et al., 2024), RSO (Liu et al., 2024b), and IPO (Azar et al., 2024). For stable evaluation process, we use DPO in this work for preference alignment. For given preference dataset D,² DPO uses the following loss function:
CDPO(Te) Tret Dp) -E Toogo (Blog [V(Uul2)) Blog Teet y1 (x)
(30 (z)
(3)
Tiet (x)
Reward Hacking Reward model is not perfect due to its limited model size, limited training data, and distribution shift between training data and alignment prompts and responses (Eisenstein al., 2023; Gao et al., 2023; Guo et al., 2024; Xiong et al.). Several works have been proposed to mitigate reward hacking. One line of works focus on observable artifacts such length (Chen al.: Dubois et al., 2024a; Shen et al., 2023b). Shen et al. (2023a) propose to enforce the consistency in reward model via data augmentation. To improve generalization, reward model ensembles can mitigate (but do not eliminate) reward hacking (Coste al., 2023; Eisenstein et al., 2023; Ramé et al., 2024b). Reward hacking can also be mitigated during policy training with post-adjusted reward (Park et al., 2024) or with post-training model merge (Lin et al., 2023). We focus on improving the reward model by addressing reward hacking from a causal perspective.
Causal Inference Causal inference can be embedded in graphical model frameworks directed acyclic graph (DAG) g = (V, E) with variables represented as nodes in V and causal relationship represented directed edge (Pearl, 2009; Lee et al., 2020) in E. We say random vector X to be faithfid with respect to a DAG g = (V, E) if for any j E V, and any subset S C V((.j).
(4) where X 1 X S denotes X and X independent conditional on X The "d" in d- separation stands for dependence. Thus if X and X are d-separated relative to a set of variables
X
X
X + and j are d-separated by S under G,
We randomly flip response pairs and associated labels to remove positional bias. DD can be Dar or can be generated responses labeled by reward model in Liu al. (2024h)
3

---

Published as a conference paper at ICLR 2025
X in a directed graph, then they are independent conditional on X in all probability distributions such a graph can represent. The definition of d-separation is as follows: suppose we are given a DAG G; then, for two nodes i,j E V. subset S of V' (i. d-connects and j if there exists path between i and j such that every collider in either belongs to or has a descendent S, and other node in I. belongs to S.. If S does not d-connect and j, then it d-separates i and j. See Appendix A.4 for more details.
ROBUST REWARD MODEL TRAINING
We first formulate the reward model training in causal framework, then we augment the reward model training data based on the causal rules.
3.1 CAUSAL FRAMEWORK
X
Y1
Y/2
S
A
C
Figure 2: Causal graph of reward model. X is the prompt. Y1, Y/2 are two responses. S is the contextual signal that depends on input prompt and two responses. A is the context-free artifact that only depends two responses. C is the preference label. Traditional reward model cannot differentiate the two DAGS on whether there is a causal edge from A to C. Our work uses the augmented dataset eliminate the edge from A to C.
We formulate DAG G model the causal relationships among different quantities (Figure 2)- X is the prompt, and Y1, Y2 are two responses. S E IR is the contextual signal that depends on input prompt and two responses. A E R the context-free artifact that only depends on two responses. C E {0, }} is the preference label, where C 1 means Y1 is preferred over Y2 and C 0 means the other way around. We assume the distribution of (X, Y1, Y2, A, C.) be faithful to the DAG. We assume the preference label C' can be captured by S and A, which is to say P Vi, Y2) = (S A). We assume the S to be the sufficient statistic (Lehhmann & Casella, 2006) that captures the contextual effect that one response fulfills the need of the prompt better than the other. We assume to the sufficient statistic that captures the context-free artifacts that only depend on two responses. Such artifacts markdown, etc), and certain patterns (n-grams such as "Sure, here is the response:"). ") In traditional reward model training, the model may hack the patterns in (Y1, Y2). Suppose 80% of winning responses to be longer, then the reward model can get 80% accuracy by just counting the number of tokens. Formally, construct two hypothesis:
include length, format (bold faces, bullet points,
Ho: there is no causal edge from A C.
H1: there a causal edge from to C.
Proposition 3.1. In traditional reward model training, Ho and Hi are not always distinguishable.
Proof. As an example of the hypotheses being indistinguishable, let's consider special case of A and S being perfectly correlated. More formally, assume 5 = s(X, Y1, Y2) : + c, with certain non- linear function s and 6g ~ N (0, oa), and similarly
= a(X, Y1, Y2) + ea with non-linear function and 6a N (0, a.). Suppose IP(C' = 1|X, Y1, Y2) = (3.5 + B.A +
E.) with constants 3., 3a. a E R and random error e, 1 (S, A). 1. If 3. = 0, then Hy is true. If 8g 1, then Hi is true. In extreme case that Corr(S,
1, then A Sa. aa for some constants aa R and Bas E R- Then the model cannot tell if 3, = 0 or not. This is because when 3, = 0, we can still reparametrize it IP(C 1|X, Y1, Y2) a(8, Bax) + A + (a aa) tc)-
4

---

Published as a conference paper at ICLR 2025
The desired behavior of reward model is to determine the preference label C ignoring the artifact A. which corresponds to Ho. To achieve that, we can utilize two d-separation relationships of the DAG g.
RI: Under Ho. A and C are d-separated by (Vi, Y2), thus A I. C (Y1,
R2: Under Ho, A and C' are d-separated by S, thus A I C
3.2 DATA AUGMENTATION
To fix the issue mentioned in Proposition 3.1, can effectively utilize R1&R2. In particular, propose to augment data with by adding the permuted pairs of generated responses.
Possible Combinations Given the dataset of triplets Dar = { with z(0) (xx(1) (2)
yio Vi ).
we can first expand the dataset as Drr = { {(m) ((02())) Ji=1- where 01 : [[V] + [N] and Ø2 : [N] [N] are two different invertible permutation functions randomly sampled from permutation
group SN- In practice, we can shuffle the dataset twice to achieve 01 and 02- There are in total
3 x : = 45 possible (z. 91 32) unordered triplets from each element in Dar- This is because there are 3 possible prompts with 2 choices among responses and we treat (x. U2) and (x, 92: V1) as the same one.
Preference Labels For the unordered triplet, can set the preference rule based on the DAG G. We say response y is contextual to æ if they are from the same triplet in Par {x(1), (i) (6)
Vi Fi=1
For example, (t) and oj" are contextual to (t()), but you and vl are not contextual to x for # i. Then for (z, /1, 12). we have the following rules:
if both y/1 and y/2 are contextual to æ, we set the winning one in Dar as winner.
if only one of y/2 and 1/2 contextual to x, we set the contextual one as winner.
if neither y1 nor y/2 is contextual to x, we set the preference label as "Tic".
Here assume that y/
(6) is still acceptable response for ( because it is usually generated by language model conditional on (1))
Augmented Triplets From R1, can fix (Y1, Y2) and vary X to perturb C. From R2, we can fix C' by picking a contextual (prompt, response) pair (X, Y1) and another non-contextual response Y2. Then we set Yi as winning response and vary losing response Y2 to perturb A. We can see the augmented datasets derived from the above two rules cover all possible (I, f1, B2) unordered triplets generated from Dar. For simplicity, we select the ones with prompt 2+(+), which provides us with the following additional augmented triplets?:
(669, ( v(((()))
cchosen
= V/
(4)
(o69, yl ((0)())))
cchosen
y (1)
(o(0) y (for())) Vi (o1(1)) + Tie
(o wi i1 (o)()))
cchosen = y/6
(z)(), ((o2(0)), si (o((i))) + Tie
(x (1) Vi (o2()))
+chosen = y/
(o(1), wi (i)())) Di (o)()))) + Tie
Non-contextuals
Neutrals
(2()) Vi Vi (())
(1)
+chosen 3/ (4)
(o)(), ylo (()) .3 (02(1))) - Tie
(1 Vi , Bw
(1) (o2(1))
cchosen (4)
(2)(), g(oa(t))) Bi (o1(1)) + Tie
(x Vi (6)
31 (o1(0))
chosen = (6)
(x(9), vi (or())) .1 (o2(1)) + Tie
(a(0) vi 11 (02(())
chosen (0())
(5)
We show sample Python code in Appendix A.2.
5

---

Published as a conference paper at ICLR 2025
"Non-contexttull" set the contextual response as chosen and non-contextual one rejected. "Neu- trals" set both non-contextual responses as tie. With these, we have the following claim: Proposition 3.2. If the reward model is trained with Diy and augmented triplets in Equation S, there is no causal edge from A to C' in DAG g.
Proof. We can prove this by contradiction. If there is causal edge from A to C, then the conditional independence relations A 1 C' (Y1, Y2) and A - C S do not hold, which contracts to the triplets constructed on "Non-contextuals" and "Neutrals".
3.3 CONNECTION TO EXISTING WORKS
ODIN (Chen et al.) ODIN decomposes reward into additive format of quality one and length one. During learning, enforces the disentanglement between the quality reward and the response length and encourages the correlation between the length reward and the response length. We claim that this is a special case of our causal modelling with single observed artifact A as length, because the disentangle learning is necessary condition of the conditional independence between C and A given the data. Our framework is more general and can go beyond single and observed artifact.
Length-controlled AlpacaEval-2 (Dubois et al., 2024a) This work improves the original ver- sion of A AlpacaEval-2 by conditioning on the length through Controlled Direct Effect ( (VanderWeele, 2011). adds length as a variable in the logistic regression to predict the preference. Effectively, it learns the residual part that cannot be explained by the length. In our framework, learn the residual part that orthogonal to the artifacts, which is the length in length-controlled AlpacaEval-2. Thus the two methods are equivalent, and our approach can go beyond single artifact and be extended to unobserved artifacts.
directly
Length-controlled DPO (Park al., 2024) This work adds length penalty the RLHF objec- tive (Equation 2). It serves post-hoc reward adjustment to mitigate the length bias during policy optimization. The idea for removing the lengthy bias using a length reward is the same as ODIN, but they don't have the correlation penalty and the additional hyperparameter introduced can add more complexity into policy optimization. In comparison, our work directly learns a artifact- free reward model so we do not need an explicit length adjustment factor in the alignment algorithm designs.
Contrast Instructions (Shen et al., 2023a) This work shows the issues of reward models on the instruction and response consistencies when switching instruction or response to another similar one. It proposes data augmentation training approach and improve the consistencies of reward models. On contrary, by considering all possible combinations of (Ir, 1/1 U2) across different examples, our approach uses the organic data from the dataset, which can effectively eliminate the artifacts existing in the dataset.
inference technique to
4 EXPERIMENTS
In this section, we conduct reward modeling and apply the trained reward to downstream alignment tasks to verify the effectiveness of the proposed method. For deeper understanding, we also conduct analysis on reward model training data, aligned policies, and perturbed reward model training data.
4.1 SETTINGS
Training Set We study RRM using the preference dataset curated by RLHFlow! (Dong et al., 2024), which has been used to train series of strong open-source preference models as evaluated by the Reward-Bench (Lambert et al., 2024). The dataset consists of 700K preference pairs, which mixture of HH-RLHF (Bai et al., 2022a), SHP (Ethayarajh et al., 2022), HelpSteer (Wang al., 2023), PKU-SafeRLHF (Ji et al.. 2024), UltraFeedback (Cui et al., 2023), UltraInteract (Yuan et al.,
2024), Cadilabel-Capybern (Daniele & Suphavadeeprasit, 2023), and Distilabel-Orea (Lian et al.,
2023). We list the data sources and number of examples in Table 1. The authors of the original

---

Published as a conference paper at ICLR 2025
paper delete the samples with similar scores when the scores available because when the model
well calibrated, these samples are more likely to mislabelled. Thus the total number is smaller than the sum of each individual datasets.
Source
Number of Examples
HHH-RLLF-Helpfull (Bai et al., 2022a)
115,396
SHP" (Ethayarajh et al., 2022)
93,301
HelpSteer' (Wang et al., 2023)
37,131
PKU-SafeRLHIP* (Ji et al., 2024)
26,874
Ultraeeddbcck (Cui et al., 2023)
340,025
Ultralnteract (Yuan et al., 2024)
161,927
Distilabel- Cappbbara" " (Daniele & Suphavadeeprasit, 2023)
14,811
Distilabel-Orea (Lian et al., 2023)
6,926
Table 1 Composition of reward model training dataset
Reward Model Training Details We first train pairwise ranking reward model (RM) from Gemma-2-9b-it. With the augmentation illustrated in Equation 5, we can get 14X additional ex- amples, most of which can be too casy for RM to learn. To reduce the augmented data size, we first conduct inference on random 50% of the augmented data using the trained RM, and leave the examples with [P > B) - P (A > B) 0.2, where P(A B) is winning probability calcu- lated by RM and P (A > B) is the ground truth probability We get 2.4M training examples by merging the filtered augmented data and original RM training data. Then use the same train- ing recipe to get the robust reward model (RRM). We train the reward models for I epoch using AdamW (Loshchilov, 2017) optimizer with learning rate 1e-6 and batch size 12814.
Policy Model Training Details We train DPO policies using the on-policy responses generated by Gemma-2-9b-it and labeled by RM and RRM, respectively. We use the prompt set from the UltraFeedback dataset to generate 5 responses per prompt. Then, we compare all (2) pairs and pick the best-worst response pairs to align the DPO policy following (Pace et al., 2024; Dong et al., 2024). We train the policies for 2 epochs at most using AdamW (Loshchilov, 2017) optimizer with learning rate 2e-7 and a global batch size of 1 128, where the batch size follows Dong et al. (2024) and the learning rate decided by grid search.
We evaluate the quality of reward model from two perspectives: the accuracy on Reward-Bench (Lambert et al., 2024) and the quality of policies induced by the reward model. For policies induced by the reward model, we consider two variants: 1. Best-of-N (BoN) policy and 2. aligned DPO policy. Our main focus is for open-ended generation and we use MT-Bench (Zheng et al., 2024) and AlpacaEval-2 (Dubois et al., 2024b) to evaluate.
Evaluation Metrics
4.2 MAIN RESULTS
Reward Model Accuracy The test accuracies on Reward-Bench are reported in Table 2. RRM improves "Chat Hard" and "Safety" by clear margin but sacrifices the "Reasoning". Regarding "Reasoning", we hypothesize that math and coding are less affected by the non-contextual artifacts and we may use other rewards than an LLM because those are objectives like golden answers. On average, RRM improves RM by an absolute 3.54% accuracy gain.
1'ehe ground truth probability is if A is preferred over B, 0 if B is preferred over A, and 0.5 if they tie.
We found epoch is best for reward model training and we pick the hest hyperparameter by grid search.
7

---

Published as a conference paper at ICLR 2025
Model
Chat
Chat Hard
Safety
Reasoning
Average
RM
97.77
51.54
78.54
94.58
80.61
RRM
96.51
65.57
83,90
90.62
84.15
Comparison of test accuracy of neward-Benche RRM shows improvement upon RM on Table 2:
Chat Hard and Safety with an average 3,54% improvement of accuracy.
Policies Induced by Reward Models We investigate the quality of reward models by evaluating the aligned policies. To study the effect of adding "Neutrals" in Equation we also train a reward model without augmented neutrals (-Neutrals). The results are summarized in Table 3. As expected, ODIN (Chen et al.) shows shorter responses than RM and RRM since explicitly disentangles the length from quality. RRM shows the best performance
MT-Bence first turn and AlpacaEval-2 over ODIN and RMM, with shorter responses generated than RM, suggesting it effectively controls the length one of the artifact. The added "Neutrals" have slight improvements first-turn MT-Bench and A AlpacaEval-2.
Reward
Policy
MTT-Bench 16
AlpacaEEval-2
T1
T2 (t)
Overall (1)
I.C. (%) (1)
WR (%) (1)
Length (1)
RM
BoN (N=8)
36.87
50.14
3072
RRM
BoN (N=8)
47.68
53.19
1770
RM
BoN (N=64)
40.52
57.62
2992
RRM
BoN (N-64)
62.82
63.03
1770
RM
DPO
8.02
6.33
7.27
33.46
41.07
2416
ODIN
DPO
8.66
8.13
8.39
48.29
37.13
1559
RRM
DPO
8.70
52.49
43.31
1723
-Neutrals
DPO
8.65
8.21
8.44
51.73
43.24
1722
Table Comparison among different reward models on various aligned policies. Tl and T2 stand for the first and second turn of the conversation, respectively. WR stands for win-rate against GPT- LC stands for length-controlled win-rate. Length is the average number of characters in generated responses. RRM shows quality improvements over ODIN and RM with shorter responses than RM. Dropping augmented neutral examples slightly hurt the quality.
4.3 LENGTH ANALYSIS
To further understand the artifacts learned in reward model, we take length as an example to analyze the reward model training data and aligned policy.
Length distribution of training data We study the length (number of tokens) distribution of reward model training datasets. Length is one common artifact that shows bias both policy training and evaluation. We hypothesize that the bias can possibly come from the reward model training data. The one used in training RM is not well calibrated and the chosen responses are longer on average (Figure 3a) and by frequency (Figure 3c). On contrary, the RRM training data is better calibrated with length more balanced between chosen and rejected responses in each length bin (Figure 3b and 3c). We further provide length analysis for each data source in Appendix A.
Length distribution of policies To understand the lengthy bias learned various policies, we also study the length distribution of generated responses AlpacaEval-2's (Dubois et al., 2024a) prompts (Figure 4). We observe that the policies induced by RRM generate shorter responses than RM, which implies the correction of lengthy bias by RRM.
15 "Triining details in Appendix A.3.
we do not evaluate BoN policies MTT-Bench hecause involves multi-tum.
8

---

Published as a conference paper at ICLR 2025
kibe
kidd
lengli Dictar
Longuh
Rlwwdd Moad huewrg Dula
(a) Histogram of response lengths (b) Histogram response lengths
(c) Percentage of chosen responses being longer or shorter in RM and RRM traininng data.
in RM training data.
in RRM training data.
Figure 3: Distribution of response lengths on reward model training datasets. (a) the RM training data has longer chosen responses average and not well calibrated (large percent deviation in left two bins between chosen and rejected) (b) the RRM training data well calibrated and the average length of the chosen responses is even shorter than rejected. Additional neutral triplets can further calibrated the model. (c) Around 60% chosen responses are longer RM training data. On contrary, the lengths of chosen responses more balanced in RRM training data.
Poourd Modd
Sawar Podel
Roward Pada
18
(a) Best 8 responses
(b) Best responses
(c) DPO policy
Figure 4: Distribution of response lengths on AlpacaEval-2 prompts of various policies induced by RM and RRM, average length marked by the dashed line. All policies show a lengthy bias towards longer responses for RM comparing with RRM.
4.4 DELIBERATELY DESIGNED ARTIFACTS
Artifacts To verify that our proposed method able to eliminate artifacts, we artificially added an artifact to the chosen responses in reward model training data. More specifically, we add prefix "Sure, here is the response: RRM on the modified reward model training data, respectively.
the chosen responses with probability 0.1. We train an RM and
To test the effect of reward model on the policy model, we first sample responses from Gemma- 2 2-9b-it model using the AlpacaEval-2 prompts. Then we add the same type of artifact to each response with probability p. = p. where (0.05, 0.1, 0.2, 0.5). Under this setting, RM trained on the artifact-added data would prefer responses with the artifacts since the chosen responses come with artifacts, even if the responses may contain low-quality answer. RRM is expected to be more robust the artifact. To verify this, we construct BoN policies using RM and RRM, respectively.
As expected, Figure 5 shows that after adding the artifacts, the BoN policies induced by RRM are more robust than RM to artifacts injected in the responses.
RELATED WORKS
The first RLHF framework (Stiennon et al., 2020) is based on the proximal policy optimization (PPO) algorithm, which was first popularized by Christiano et al. (2017) and further developed by Bai et al. (2022a): Ouyang et al. (2022). However, getting PPO work is chal- lenging especially in the era of LLMs (Choshen et al.; Engstrom et al., 2020). In recognition this issue, another line of works propose direct alignment algorithms, where notable examples include
RLHF algorithms

---

Published as a conference paper at ICLR 2025
Reward
25
20
20
(%)
(a) Best 8 responses
(b) Best 64 responses
Figure 5: Proportion of BoN generated responses with artifact versus the rate of injected artifact. For each policy, we first sample N (N 8 or 64) responses on AlpacaEval-2 prompts, then prepend "Sure, here is the response: before each response with probability (Rate) 5%. 10%, 20%, 50%, respectively. Then we compute the proportion of BoN responses that have the above artifact (Ar- tifact). The BoN policies induced by RRM are more robust to artifacts injected in the responses, suggesting that the proposed approach enables the model to focus more on the contextual signals instead of context-free artifacts in the reward model training data.
SLiC (Zhao et al., 2023), DPO (Rafailov et al.. 2024), IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), ORPO (Hong et al., 2024), SimPO (Meng et al., 2024), and DRO (Richemond et al., 2024). These algorithms directly optimize a supervised target to optimize the policy model with- out constructing a reward model first, hence the name direct alignment algorithms.. However, these algorithms learning from a fixed dataset are offline and often off-policy without further exploration of the environment. RSO (Liu et al., 2024b) emphasizes the importance of reward model and fixes the distribution shift problem to improve the DPO training, followed by list-wise alignment (Liu et al., 2024a) and the online (iterative) training frameworks (Xiong et al.; Guo et al., 2024; Calan- driello et al.). Alternatively, there is also a line of work based on the best-of-n sampling, such as RAFT (Dong al.), BOND) (Sessa al., 2024), BoNBoN alignment (Gui al., 2024). These algo- rithms leverage a reward model to rank the generated responses and distill knowledge from the best responses. Our approach can benefit RLHF algorithms relying on reward model.
Reward Models & Reward Hackings Building a superhumam/unbiasse reward model is vital for training better chat assistants since it could affect the upper bound of the policies' capabili- ties in the online preference optimization (Wang et al., 2024a; Bai et al., 2022b). Multi-objective rewards (Wang et al., 2024b), RLHF-workflow (Dong et al., 2024), and RMBoost (Shen et al., 2024) are proposed to train more capable reward models. While revealed by Denison et al. (2024); Zhang et al. (2024), reward models easily hacked by different pattern in different scenario, e.g., length (Singhal et al., 2023) and sycophancy. Recent works employ the model merging (WARP (Ramé et al., 2024a) and WARM (Ramé et al.. 2024b)). and hacking reward decomposition (ODIN (Chen et al.)) to mitigate the hackings in online RLHE. Generative reward models can pro- vide more detailed preference analysis (Yan al., 2024). For the most accurate reward signal, one can also use verifiable answers in certain domain like math (Xiong et al., 2024). Most model- based methods failed to distinguish between preferences driven by the prompt and context- artifacts. Our RRM is more advanced in removing the artifacts.
6 CONCLUSION
In this paper, we identified a key problem in the current reward training methodology: its inability to differentiate between contextual signals and context-free artifacts. Using a causal framework, we ex- plained this effect and improved reward model training by introducing a data augmentation approach derived from the framework. Our theoretical analysis and extensive empirical results demonstrated that the proposed techniques effectively enhance both the test accuracy of the reward model and the quality of the policies it induces. Future work will explore filtering augmented pairs and matching artifacts when constructing response pairs, further refining the training process.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (oXiv:2303.087744 2023.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4457-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204, 05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li. Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen- Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b. URL https: /trr///rxvvvv//vv////v///e/////a org/ abs wwtp//s/.....ee... 08073.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08077 2022c.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(334))324-45, 1952.
Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment large language models through online preference optimisation. In Forty-first International Conference Machine Learning.
Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shocybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. In Forty-first International Conference Machine Learning.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforce- ment learning for neural machine translation. In International Conference on Learning Represen-
tations.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- fems, 30, 2017.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02733 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Luigi Daniele and Suphavadeeprasit. Amplify--nstruuc: Synthetically generated diverse multi-turn conversations for effecient Ilm training. arXiv preprint arXiv:( soon), 2023. URL https: / //uuggi ngfaae co cot

---

Published as a conference paper at ICLR 2025
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec. Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bow- man, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tammne rewarr-tampering in large language models, 2024. URL https: aawiiviiivv..///vva weixxiiorre org/ abs 2406. 101 62.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling online rlhf. arXiv preprint arXiv:2405.07555 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators, arXiv preprint arXiv:2404.04475, 2024a.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Jacob Eisenstein, Chirag Nagpal, Alckh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudoiph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprini arXiv:2005. 12729, 2020.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information. International Conference on Machine Learning, pp. 5988-6008. PMLR, 2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.013366, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp. 10835-10866 PMLR, 2023.
Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.008: 00832, 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint 0et::2202.0479.. 2024.
Jiwoo Hong, Noah Lee, and James Thome. Reference- free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691, 2024.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. blmndeender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023.
Nathan Lambert, Valentina Pyatkin. Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Steffen I. Lauritzen. Graphical models, volume 17. Clarendon Press, 1996.
12

---

Published as a conference paper at ICLR 2025
Kuang- Yao Lee, Tianqi Liu, Bing Li, and Hongyu Zhao. Learning causal networks via additive faithfulness. Journal of Machine Learning Research, 21(51):1-38, 2020.
Erich L Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook. Chanvichet Vong, and "Teknium". Openorea: An open dataset of gpt augmented flan reasoning traces. https: /ttp:ss/hhtttt/// hugging fa ace. ctl.co///o/e Peto/Open-Orcee Orca /OpenOrca, 2023.
Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. Speciality generality: An empirical study catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.06255 2023.
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mo- hammad Saleh, Simon Baumgartner, Jialu Liu, Peter J. Liu, and Xuanhui Wang. Lipo: Listwise preference optimization through learning-to-rank, 2024a. URL htt.ttts///ariii.orrr.t orb//bs/ 2402. 01878.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024b.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051000 2017.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405.14734. 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances neural information processing systems, 35: 27730-27744, 2022.
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401. 12086, 2024.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19/55 2024.
Judea Pearl. Causality. Cambridge university press, 2009.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies, 2024a. URL 24C6.16763
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models, 2024b.
URL https::p////.....t.. wwwaaaavvvo.oo o abs /24/2240111 12187.
Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, et al. Offline regularised reinforcement learning for large language models alignment. arXiv preprint arXiv:2405. 19107, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
13

---

Published as a conference paper at ICLR 2025
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning llms with best-of-n distillation. arXiv preprint arXiv:2407.16622 2024.
Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgart- ner, and Michael Bendersky. Boosting reward model with preference-conen conditionl multi-aspect synthetic data generation. arXiv preprint arXiv:2407.16000 2024.
Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward (in-) consistency on rlhf. arXiv preprint arXiv:2309.16133 2023a.
Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. arXiv preprint 13X.v:2310.05199, 2023b.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 3021, 2020.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805 2023.
Tyler J VanderWeele. Controlled direct and mediated effects: definition, identification and bounds. Scandinavian Journal of Statistics, 38(3):551-56) 2011.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part ii: Reward modeling, 2024a. URL https:/ / /aaaaxivvv......aaaaaaa. wwt.//or.//..... abs 2401. 06080.
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objectivee reward modeling and mixture-of-experts, 2024b. URL org/abs/ 246.. 12855.
Zhilin Wang, Yi Dong. Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Help- steer: Multi-attribute helpfulness dataset for steerlm, 2023.
Wei Xiong, Hanze Dong. Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning.
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al. Building math agents with multi- turn iterative preference learning. arXiv preprint arXiv:2409.0m 02399, 2024.
Jing Nathan Yan, Tianqi Liu, Justin Chiu, Jiaming Shen, Zhen Qin, Yue Yu, Charumathi Laksh- manan, Yair Kurzion, Alexander Rush, Jialu Liu, and Michael Bendersky. Predicting text prefer- ence via structured comparative reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers). pp. 10040-10060, Bangkok, Thailand, August 2024. Association for Computational Linguisties. URL https::/p//.......... /ta/aaaaaac antho ogy. wrg..org/ae 2024 acl llng. 541.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint 20X2:24002.07314, 2024.
14

---

Published as a conference paper at ICLR 2025
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024.
Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists to emojis: How format bias affects model alignment, 2024. URL www.hhttss//aaae wwt.ttts///axxiv.orrg 11704.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp- 12697-12706. PMLR, 2021.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with intbbench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.005535 2019.
A APPENDIX
A.1 ADDITIONAL LENGTH ANALYSIS OF REWARD MODEL. TRAINING DATASETS
In this section, we show the length distribution of chosen and rejected responses from each individual data source in Figure 6. HH-RLHF, SHP, HelpSteer, UltraFeedback show bias towards rejected responses as the first length bin. SHP, HelpSteer, and UltaFeedbbck have longer chosen responses than rejected ones on average.
A.2 DATA AUGMENTATION PYTHON CODE
In Algorithm 1, we show a sample code of data augmentation in Python. We expect each element in data contains "context", "response _w", "response."" We use "neutral" to indicate the label should be "Tie".
A.3 TRAINING DETAILS FOR ODIN
We use the same loss as described Chen et al.. We train Gemma-2-9b-it for epoch on the same data used for RM. AdamW is optimizer and the learning rate is set to 2c-6 with cosine scheduler. We use Flash-Attcnn Flash-Atention to accelerate the training while applying the Deepspeed Zero- Stage 3 to get batch size 16 on each GPU (the global batch size is 128) make sure the calculation of the Pearson correlation between the head value and the length of the responses is stable.
A.4 PRELIMINARIES IN CAUSAL INFERENCE
We list few critical concepts in this section. For information, we refer the readers to Lauritzen (1996) and Pearl (2009).
DAGS and d-separation A DAG is a set vertices and a set of directed edges (arrows) that con- nect pairs of these vertices. The causal modeling connects DAG with Markov condition via graphical relation called d-separation (Pearl, 2009). D-separation is a relation among three disjoint sets of vertices in a directed graph. D-separation and Markov condition connect DAGs and probabil- ity distribution. By faithfulness assumption, the d-separation in a DAG equivalent to conditional independence in distribution.
15

---

Published as a conference paper at ICLR 2025
www.RuHf--teebllllccoo sbaddarr (111396)
Sil-staadard (93301)
babel Chocon Relected
tabel Chodon
22
Ro ected
Parcom
Percerf
13
230
400
800 Lingth
1030
1200
1400
500
1000
1500
2000
25.00
3000
length
(a) HHH-LLHF-Helpful
(b) SHP
pelpsttee--eeeerrnce-tttta wwt.rrnncttttttaaaaa.a... (37131)
BlLLSattlLl--BOwwttaddddr (26874)
2.511
2.0l
Length
Length
HelpSteer
(d) PKU-SafeRLHF
lirnireebacc-ppeeeeeennc fard (340025)
Ubzatnteract- ceweeratttt--tte--eee stannddddddddddddddd (181927)
Chasen
Chosen
Regected
Percent
200
600
800
1000
1200
500
1000
1500
2000
Length
Longth
(e) UltraFeedback
(f) UltraInteract
Capybaro
(14821)
Orca- stancard
lasel
abel Chocen
Choson Relected
15.0
Re ected
10.0
7.5
5.0
2.5
500
1005
1500
2500 length
2000
5000
3500
4000
250
500
750
.000
1250
1500
1750
Length
(g) Distibabel-Capybar
(h) Distilabel-Orea
Figure 6: Distribution of response lengths on each individual source of reward model training data. SHP, HelpSteer, and UlrraFeedback show significant lengthy bias showing longer responses in cho- sen. They also dominate the training dataset, accounting for more than a half.
16

---

Published as a conference paper at ICLR 2025
Algorithm Example Python Code for Data Augmentation
def get__aummented (data: List [Dict (str, Anyll) List (str, Anyll: data_i data
data_] = data_i.copy ()
random. shuffle
data_k doop_.)coppy)
Sancom.shuffle/ddee
for ex 1, xx_ ex in zip (data_i, data_,, data k)
xi ex |
xj cx_]['conexxt]]
xk ex_k [[_x(('ootett]]
ywi ex_i [ ['rreponse_w'
yw] ex ex_i
][ response_x'
ywk k response_ w']
yli
respopseee__.
ylj CX ] response_l"
ylk cx_ response_l'
xi xi_ywi_ywj
yield
"context": xi, "response_w"" _" ywi,
"response_ ywi, "neutral": False
xi_ywi_ywx yield {
"context"; xi, "respense_v"" ywi, "respcnse_1"; yak, "neutral": False
fill in all other augmented triplets xi y1j yield {
"context"; xi, "response_w": yuk,
"response_ 1": yli, "neutral": True
xi_yijyik yield {
"context"; xi, "response_w"; ylj,
"response_1"; y1k, "neutral": True
The causal Markov condition The Causal Markov assumption assumes that variable X is in- dependent of every other variable (except X's effects) conditional on all of its direct causes. With this, DAG defines set of distributions of the form
ply
5k) Il p(y, parents(y,))
Counterfactuals Consider two variables X and Y. We will call X the "treatment". We call Y the what their value Y; would
"outcome" For given subject see (X,, Y,). What we don't 't have been if we changed their value of X; This is called counterfactual. Suppose that X is binary variable that represents some treatment. So X means the subject was treated and X means the subject was not treated. Let Y, denote the outcome if the subject is treated. Let Yo denote the response the subject is not treated. Then
= XY + +(1 X)Y
17

---

Published as a conference paper at ICLR 2025
If we treat subject, observe Yı but not Yo. The unobserved variable is called counterfactual. The variables (Yo, X1) are also called potential outcomes. We define mean treatment effect as
= EYi) - E(Y)) = E(Y|set X =) B(Y|ser X 0)
A.5 ADDITIONAL RESULTS WITH GEMMA-2-2B-2
To further verify effectiveness of our approach, train Gemma-2-2b-it reward model (RM) and robust reward model (RRM), respectively. Table 4 shows the results on the reward bench. RRM again shows improvement on the Chat Hard. It shows some regression Safety and Reasoning, where we hypothesis that some context-free freee nature of reasoning and safety makes the RRM perform worse. Overall, RRM shows positive effect. We also test the reward models on AlpacaEval2 using best-of-n policies. The results are shown in Table 5. We use the trained reward model to rank the responses generated from the Gemma-2-9b-i 2-9b-it model, and observe consistent gains on RRM over R.M.
Model
Chat
Chat Hard
Safety
Reasoning
Average
RM
96.49
42.54
72.70
72.30
71.01
RRM
97.21
49.01
72.71
70.08
72.25
Table 4: Comparison of test accuracy of Reward-Bentt RRM shows improvement upon RM on Chat and Chat Hard with an average 1.75% improvement of accuracy.
Reward
Policy
AlpacaEval-2 WR (%)
LC (%) (f)
Length (J)
RM
BoN (N=8)
44.21
45.83
2264
RRM
BoN (N=8)
54.37
47.19
1749
RM
BoN (N=64)
47.32
52,23
2316
RRM
BoN (N=64)
60.42
56.50
1870
Table 5: Comparison among different reward models on various aligned policies. WR stands for win-rate against GPT-4. LC stands for length-controlled win-rate. Length the average number of characters in the generated responses. RRM shows quality improvements with shorter responses over RM.
A.6 ADDITIONAL. ANALYSIS WITH MIXED ARTIFACTS
To investigate the effect of RRM on mixed artifacts, we conduct an additional experiment as follows:
With p = 0.1, wrap the whole chosen response with *$*" as bold-face.
2. After the above step, with p = 0.1, append emoji
3. Train RM and RRM on the above dataset.
To test the effect of reward model the policy model, first sample N responses from Gemma-2- 9b-it model using the AlpacaEval-22 prrocap Then we add emoji to each response with probability (0.05, 0.1, 0.2, 0.5). Under this setting, RM trained the artifact-added data
Pa = p. where would prefer responses with the artifacts since the chosen responses come with artifacts, even if the responses may contain low-quality answer. RRM is expected be more robust the artifact. To verify this, we construct BoN policies using RM and RRM, respectively.
As expected, Figure 7 shows that after adding the artifacts, the BoN policies induced by RRM more robust than RM to artifacts injected in the responses.
A.7 DISCUSSION ON DATA FILTERING STRATEGIES
In this work, we assume the preference labels should be purely controlled by the prompt dependent signal. However, there be cases such that prompt-independent signals contribute to the
18

---

Published as a conference paper at ICLR 2025
20
Aut (54)
5
(a) Best 8 responses
(b) Best 64 responses
Figure 7: Proportion of BoN generated responses with emoji versus the rate of injected emoji. For each policy, we first sample N (N or 64) responses on AlpacaEval-2 prompts, then append emoji after each response with probability (Rate) 5%, 10%, 20%, 50%, respectively. Then we compute the proportion of BoN responses that have the above artifact (Artifact). The BoN policies induced by RRM more robust artifacts injected in the responses, suggesting that the proposed approach enables the model to focus more on the contextual signals instead context-free artifacts in the reward model training data.
preference label. For example, responsible Al should be always safe and cares about the diversity. In the data augmentation, we sometimes use rejected response as new chosen in "Nonn-ontextuals" See the last four triplets in "Non-contextuals" of Equation 5. For "Neutrals", we also assume 0.5 winning probability of a non-contextual response pair. These treatment may cause the unwanted behavior of AI if we unsafe response the new chosen assigning winning probability of 0.5 on a pair of (safe, unsafe) responses.
To address this, we have few treatments that can be applied in future works:
Use a trained safety pointwise or Bradley-Terry model to filter out triplets that has low safety scores on the chosen responses.
Use Al feedback such as Constitutional Al (Bai al., 2022c) to ensure the augmented triplets have high quality chosen responses and consistent preference according to certain non-contextual rules. The rules can include safety, style, factuality.
B ETHICS STATEMENT
Our research adheres the ethical guidelines. Our work aims mitigate reward hacking in RLHF, contributing to the development of more reliable AI systems that better align with human prefer- ences. Our data augmentation process explicitly addresses and mitigates common forms of bias, thus reducing the potential for harm in practical applications of AI systems. Our model was de- signed with fairness mind, particularly in avoiding biases related to response length and format, which can unfairly influence Al decision-making
C REPRODUCIBILITY STATEMENT
All code used for training the reward models (RM and RRM) and for running the experiments de- scribed in this paper will made publicly available upon publication. This includes the implemen- tation of the data augmentation pipeline, the reward model training process, and policy alignment. The datascts used in our experiments are publicly available. We provide the complete set of hyper- parameters used for training the models, including learning rates, batch sizes, and other optimization settings. The evaluation approaches are also public available and can be fully reproduced.
19


Reviews:
[{"review_id": "rO13CsdDzS", "reviewer": "Reviewer_gp5Y", "summary": "Traditional reward model (RM) training for large language models (LLMs) struggles to separate prompt-driven preferences from irrelevant artifacts like response length and format. This work introduces a causal framework and data augmentation technique to remove these artifacts, resulting in a robust reward model (RRM) that improves performance across benchmarks, enhancing accuracy and alignment scores significantly.", "questions": "The method proposed in this paper to 'disentangle prompt-driven preferences from prompt-independent artifacts' seems to only eliminate biases unrelated to the input while preserving input-related preferences. This can effectively reduce bias under the 'helpful' alignment objective. However, if our alignment goal is 'safety', the response might be prompt-independent, typically just refusing to reply. Yet humans may still have preferences for different ways of refusal. In this case, how can we disentangle prompt-independent preferences from prompt-independent artifacts?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper addresses an important issue in preference datasets: how to disentangle prompt-driven preferences from prompt-independent artifacts.", "weaknesses": "While the problem studied in this paper is significant, the proposed solution is relatively straightforward, focusing only on expanding the dataset. To remove prompt-independent artifacts, the dataset size needs to be several times larger, increasing the training cost. Additionally, as shown in Figure 5, the effect of this costly augmentation is not very significant—after inserting 30% artifacts, the RRM only decreases from the original RM’s 25% to 20%.\n\nRegarding Table 1, the reasoning performance declines, and the authors explain this as due to math and coding tasks being less affected by non-contextual artifacts. This raises the question of whether the proposed method might have a negative impact on attributes unaffected by non-contextual artifacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l83P8hVjXo", "reviewer": "Reviewer_Vrzq", "summary": "This paper aims to learn reward models that are unbiased by “artifacts” that come from human preference data: namely length biases and formatting. To do so, they propose to augment the RLHF preference dataset with preference pairs where one sample is in response to the prompt, and the other example is not in response to the prompt. They evaluate their learned reward model on reward bench and use it to improve downstream policy performance.", "questions": "- How did the authors tune the hyperparameters?\n- What are the “three possible prompts” described in line 226?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors study a very relevant and timely problem.\n- The paper shows significant improvement in both reward modeling accuracy and in downstream policy performance.\n- The authors do a really interesting study where they artificially add artifacts to the preference dataset, and find their method is more robust to it.", "weaknesses": "- The main weakness of this paper is that the experiments are not that comprehensive. Although the results are strong, the experiments are only conducted on one dataset, with one model, and with one random seed. Therefore we cannot tell if the results are statistically significant or generalizable.\n- The baseline is pretty weak with regards to reward bench. RLHFFlow with Llama 3 8B gets 87.1 on reward bench, while the author’s baseline gets 80.\n- The writing describing the methodology not that clear. I think more plainly describing the data augmentation would be better.\n- The data filtering step used in their method hurts their experiments interpretability, as it may be the case that the sole reason RRM gets good performance is due to the data filtering step. They should include a baseline where the original dataset is filtered, and then a RM is trained based on that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhLRKcLMzz", "reviewer": "Reviewer_jPJZ", "summary": "This paper presents a method to enhance the robustness of the reward model for LLM alignment. More specifically, rewards models are trained based on the question and the human labelers' preference over the answer pairs. A commonly observed un-robust behavior of the reward model is that there are spurious correlations between the label and the artifacts in the answers which is not relevant to the question. The existence of such spurious correlations causes the reward model to favor certain types of artifacts, e.g. the length of the answer.\n\nIn previous works, some of the artifacts are specifically dealt with, like introducing a penalty for the length of the answer. However, the solutions lack a principle and can not handle other artifacts like the word preference and style of the answers. In this work, a systematic approach is proposed based on a causal framework, which is effective in debiasing the reward model from various artifacts.", "questions": "- The same augmentation and disentangle technique can be applied to learn a artifact only model. Some of the artifacts would improve the quality of the answer, if there is a consensus preference on such artifacts in the labelers. Is there any proposal to combine the contextual & non-contextual models? Would adding these two reward models work, or do we need to take the min of these two models? Any suggestions from the authors?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The causal framework has been effective and widely used in other problems, but this is the first time I see it applied in LLM reward modeling, and it very well suits the problem under consideration.\n- The implementation is very straightforward, just augment the data, no empirical tricks.", "weaknesses": "- It seems suboptimal to me to train the reward model that is only dependent on the S, because the A part could contribute nontrivially to the reward function, in the case where both answers are equal in terms of the contextual quality, the labeler could prefer the answer which is organized as bullet points for readability.\n- The explanation on why \"reasoning\" is performing worse under RRM is not quite convincing. Even if they are less affected by non-contextual artifacts, the augmented data should not have caused the degrade of performance either (as the augmented data are simpler pairs). I would suggest two other hypothesis to investigate:\n  - Look into how \"-Neutral\" works on \"Reasoning\" as requiring 50/50 preference over some random pairs could be too strong.\n  - Try to add back the \"A only\" reward model, as mentioned above the artifact could contribute nontrivially to the reward, removing it entirely could be detrimental especially for math and code, where a high quality answer should be good in formatting which is noncontextual.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Em2uUqm49u", "reviewer": "Reviewer_1CBu", "summary": "This paper proposes a robust reward model (RRM) training method to mitigatge reward hacking by utilizing causal inference framework. A causal graph for human preference modeling is introduced to enable the model to distinguish contextual preference signals and context-free artifacts. The training data is augmented by re-organizing the (prompt, positive, negative) triplet guided by the causal inference framework. Experiments show that the proposed approach can filter out undesirable artifacts in reward model training and yields a more robust reward model. Specifically, the reward model is trained on Gemma-2-9b-it, and RRM improves RM by an absolute 3.54% accuracy gain on Reward-Bench. Experiments also demonstrate the policy induced by RRM outperforms RM and ODIN in MT-Bench and AlpacaEval-2 benchmark. Analysis show that length artifact and deliberately designed artifact can be eliminated.", "questions": "1. Explain why it is needed to add neutrals and set both non-contextual responses as tie in Section 3.2. What is the necessity of this step?\n2. In Table 2, why the performance of RRM is not consistent in two chat datasets such as Chat and Chat Hard? In Chat dataset, RRM outperforms RM while in Chat Hard dataset, RRM is even worse.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "Reward hacking is an important problem in LLM alignment. This paper introduces causal inference framework in reward modeling to mitigate reward hacking, which offers an interesting perspective. This idea makes a lot of sense since the essence of reward hacking mitigation is to find out the irrelevant artifacts that have no causal relationship with the preference label, and causal inference algorithms just offer a useful tool. The idea is clearly presented and illustrated. The experiment results verify the effectiveness of proposed method.", "weaknesses": "The experiments are insufficient. First, the experiments in this paper only cover two possible artifacts, namely length and a deliberately designed phrase, which is not compatible with author's claim that this method should be capable of handling \"all potential exploitation patterns\". More patterns such as format and emojis should be investigated in the experiments. Second, the reward model is trained only using Gemma-2-9b-it. More experiments should be conducted on different model types such as llama, and larger model sizes such as 57B and 72B. It is in doubt that when the model size increases, the reward hacking problem can be naturally mitigated and the effectiveness brought by this method may diminish. Third, in terms of policy evaluation, this method is only verified in DPO training whereas various other methods should also be considered such as PPO, whose performance relies heavily on reward model quality. Last, the author is suggested to add more baselines in experiment, such as reward model ensemble which can also alleviate reward hacking. \n\nThe data augmentation method is trivial. The new combination is derived by randomly selecting answers from other prompts. Since this negative sample is not related to the prompt with high probability, it is too easy for the model to distinguish the positive and negative answers. The author is suggested to incorporate some hard negatives to make the training more robust.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)