Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
Authors: Rui Yang
Ruomeng Ding
Yong Lin
Huan Zhang
Tong Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Reward Model
Out-of-distribution Generalization
Alignment
Primary area: natural_language_processing
TL;DR: We propose to enhance the generalization ability of reward models for LLMs by regularizing the hidden states, effectively improving accuracy in various out-of-distribution tasks and mitigating over-optimization in RLHF.

Abstract: Reward models trained on human preference data have been proven to effectively align Large Language Models (LLMs) with human intent within the framework of reinforcement learning from human feedback (RLHF). However, current reward models have limited generalization capabilities to unseen prompts and responses, which can lead to an unexpected phenomenon known as reward over-optimization, resulting in a decline in actual performance due to excessive optimization of rewards. While previous research has advocated for constraining policy optimization, our study introduces a novel approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states. Specifically, we retain the base model's language model head and incorporate a suite of text-generation losses to preserve the hidden states' text-generation capabilities, while concurrently learning a reward head behind the same hidden states. Our experimental results demonstrate that the introduced regularization technique markedly improves the accuracy of learned reward models across a variety of out-of-distribution (OOD) tasks and effectively alleviates the over-optimization issue in RLHF, offering a more reliable and robust preference learning paradigm.

Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
Rui Yang' Ruomeng Ding" Yong Lin Huan Zhang'
Tong Zhang'
University of Illinois Urbana-Champaign, Georgia Institute of Technology,
PPrinceton University, "Princeton Language and Intelligence yangrui.. thu20155gmail com, rmding@gatech. edu, y17690@princeton. edu huan@huan-zhen com, tongzhang@tongzzang amaaanggmml.... org
Abstract
Reward models trained on human preference data have been proven to effectively align Large Language Models (LLMs) with human intent within the framework of reinforcement learning from human feedback (RLHF). However, current reward models have limited generalization capabilities to unseen prompts and responses, which can lead to an unexpected phenomenon known as reward over-optimization, resulting in decline in actual performance due to excessive optimization of rewards. While previous research has advocated for constraining policy opti- mization, study introduces novel approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states. Specifically, we retain the base model's language model head and incorporate a suite of text-generation losses to preserve the hidden states' text-generation capa- bilities, while concurrently learning a reward head behind the same hidden states. Our experimental results demonstrate that the introduced regularization technique markedly improves the accuracy of learned reward models across a variety out-of-diste -istiriibbttion (OOD) tasks and effectively alleviates the over-optimization issue in RLHF, offering more reliable and robust preference learning paradigm
Introduction
Pretrained large models have showcased impressive capabilities across diverse fields 1 A notable trend in recent research is ensuring that large models align with human values and mitigate potentially harmful behaviors 6
3
10 Aligmment methods are crucial in achieving this objective, with two primary approaches being supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) 7 g. SFT directly finetunes the model using prompt and response pairs, proving to be straightforward and efficient alignment technique [TI] [2 13]
Differently, RLHF begins by learning a reward model from user preferences and then employs reinforcement learning optimize the language model to maximize rewards. A significant advantage of RLHF is its potential to generalize the reward model to unseen prompt-response pairs, effectively leveraging large volumes of unlabeled data 8 14]
Despite the empirical success RLHF, the challenge of training reliable and generalizable reward model for unseen data remains open problem. A well-known failure mode of reward model known as "overuptimization" or "reward hucking" [5 16 17 [8], where policy optimization seemingly improves the proxy reward model but actually degrades the true reward function. [17] demonstrated a synthetic setup that increasing the size of the reward model and the volume of training data mitigate this overoptimization issue. However, such scaling is not always feasible
'Code and open-source reward models available at Reward- Model
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Optimization
wvd
Reward Head
Fo Yi)
Reward Loss
40K
405K
y/
Rap O% [x]
Text-Generation Regularization
Preference
Training Model
LM Head
(a)
(b)
Figure (I) Illustration of GRM. Given preference data pairs (xr, Mor B-), the reward head re minimizes the reward loss in Eq I while the language model (LM) head TBLM minimizes a suite of text-generation losses introduced See 3.2 (2) Performance of GRM and the vanilla reward model on in-distribution (ID) task (Unified-Feedback) and average results of OOD tasks (HHlH-Aligmmen and MT MT-Bench). Compared with the baseline reward model, GRM generalizes better on OOD tasks, with larger advantage when the dataset size is relatively small.
in many realistic scenarios. To address this, a series of studies have been conducted, focusing either enhancing the reward model with ensemble techniques [[8] 19 20 [2]]
constrained 25]. The latter paradigm is related to the offline RL literature which involves limiting the policy distribution to be close to the training data distribution. Among these, improving the generalization ability of reward models presents a fundamental and promising direction that can be studied independently from enhancements in policy optimization. Nevertheless, previous methods [18 32) requiring training multiple reward models may be resource-intensive for the practical application of large models.
policy optimization [22
26
27
28
29
30
31
In this study, we present a lightweight yet effective solution designed to enhance the reward model's generalization ability against distribution shifts. Previous research [33 has theoretically shown that a randomly initialized head can distort pre-trained features, thereby negatively impacting out- of of-distribution (OOD) performance. Inspired by this finding, propose regularize the feature during fine-tuning for preference learning using an adversarial regularizer, which derives a suite of text-generation losses. To this end, introduce Generalizable Reward Model (GRM), which retains the base model's language model head and regularizes the hidden states of the reward model by incorporating text-generation losses. This approach makes better use of the preference learning data while preserving the text generation capabilities of the hidden states. Notably, GRM does not necessitate training multiple reward models relying on additional training data.
In our experiments, GRM substantially improves the evaluation accuracy of the reward model OOD evaluation datasets, demonstrating its superior ability to generalize learned preferences to unseen prompt and response pairs. Moreover, GRM consistently improves the performance both 2B and 7B reward models, with more pronounced improvement observed when the data size is limited. We also demonstrate that GRM can markedly enhance the performance of best-of-n (BoN) sampling and PPO 34, effectively mitigating the overoptimization problem. These results highlight the potential of the GRM to serve as more reliable proxy reward model for human preferences.
To conclude, our primary contributions
follows:
We propose GRM, a novel approach that employs text-generation regularization on the hidden states to enhance the generalization ability of reward models.
Our study validates the effectiveness all three types of fext-generation regularization for GRM, identifying the SFT regularization the most effective and stable solution.
Our empirical results show that GRM significantly improves the accuracy of reward models across various OOD tasks. Furthermore, it consistently enhances the performance of RLHF, effectively alleviating the overoptimization problem.
Background
Typically, reinforcement Learning from Human Feedback (RLHF) involves reward modeling and policy optimization, with Best-of-n Sampling (BoN) and Proximal Policy Optimization (PPO) being two commonly used methods for policy optimization.

---

Reward Modeling, Generally, reward modeling is based on the Bradley- Terry model [35, which aims distinguish between the chosen response ye and the rejected response y, given the prompt æ: reward (0) = - (........ [log (a (ro(z, ye) ro yr)))]
(I)
where (2; y) represents the reward score for prompt and output y with model parameters 0. at))) is the sigmoid function. By minimizing this loss function, the reward model assigns higher scores outputs preferred by humans. Subsequently, the trained reward model can be used to guide the optimization of the language model.
Best-of-n Sampling (BoN). BoN generates n samples from the policy model, denoted as Ygan- and then selects the best one based on scores provided by a reward model. BoN can be used for inference-time improvement iterative optimization [36 37 33).
VBON (z) arg max To y). DEYgo
(2)
Proximal Policy Optimization (PPO). PPO is a widely adopted method for RLHF in optimizing language models [16 & 33]. PPO learns policy by maximizing a reward objective with KL divergence penalty with coefficient
Tiotal = [r, v) (mpol()))))lla)) || (sFr())))))
where the KL penalty ensures that the optimized policy does not deviate significantly from the SFT policy to maintain the reliability of the reward model.
Overoptimization. Although the learned proxy reward model aims to approximate human preference, it may not consistently reflect authentic human preferences, potentially resulting in over-optimization 17 [8]. This issue emerges when the proxy reward model becomes overly optimized, causing the policy model to overfit certain erroneous patterns. Ultimately, this issue can diminish the model's 's alignment with actual human preferences, highlighting the need to ensure the reward model's robustness and reliability.
Method
In the common practice of training reward model 8 39 40), reward models are initialized using a pretrained or SFT finetuned backbone, along with a randomly initialized reward head to predict the scores for prompt-response pairs. It's important note that the backbone and original language model head are trained on a diverse range of datasets for text generation, which is distinct from the preference learning tasks. Under the task shift, the randomly initialized reward head can distort the pretrained features, thereby reducing the OOD generalization performance, as observed by 333 We also confirm this impact on preference learning in Appendix C.11 C.I.
To improve the reward model's s generalization capability against distribution shifts, we propose a lightweight yet effective solution, Generalizable Reward Model (GRM). This model employs suite of text- generatien regularizations for the hidden states. More specifically, GRM employs a structure as illustrated in Fig] with one language model (LM) head and one reward head sharing the same hidden states. The reward head trained to minimize the reward loss Lrwward in Eq while the LM head is trained to maintain the text-generation ability of the hidden states during preference learning. Consequently, we define the overall loss function as follows:
Lisal
Grewand + o oLreg
(4)
Here, is the coefficient that balances the reward loss and the regularization. We will derive potential forms of the regularization term below.
3.1 Theoretical Motivation
To derive the potential formulation of the regularization term, we consider the following adversarial optimization problem: learning a reward model against an adversarial policy.
0 = arg min (Lrward (0) + max ((0, m
(5)
where 7 > is coefficient. This objective is also considered by recent studies 24 25 aiming to enhance DPO. Differently, adopt it to learn generalizable reward model.
3

---

The insight of Eq i is that we can enhance the robustness of the reward model by considering an adversarial policy from a certain policy class. The term for policy optimization J(0, ι) can have various formulations, but KI, policy [6 8. Moreover, has an advantageous property that the inner optimization problem has an analytical solution, which can simplify the problem.
rerrgence-rggularized objective is generally used in training the
J[6, π) Bz-D. [ro y)] DDDD [KL (xr(1aa))) Tree (ret(-(=))]
(6)
where 3 > 0 is regularization coefficient and Tref is the reference model. We denote the analytical solution of [(0, T) as Tg. Incorporating zq into Eq we can transform the min-mas max optimization problem into standard optimization problem under certain assumptions:
0 arg min Lreward (0) + CpProoPO(T@) (SFTLSFT(TT))
Here, Lioo is the same as the DPO objective [4] and CSF is the SFT objective that maximizes the probability of chosen responses. Notably, the two regularization terms originate from different sources: LDOO stems from the reward loss, while LSFT is derived from the adversarial term. This may explain why SFT regularization proves more beneficial than DPO regularization in our empirical results. Motivated by Eq 7 we relax the relationship between rg and z. and propose learning a reward model parameterized by and a language model head parameterized by OLM, both sharing the same hidden states. A discussion of this design can be found in Appendix Below, we detail three practical implementations.
Detailed derivation is deferred to Appendix A
3.2 Text-Generation Regularization
Inspired by Eq 7 we train the LM head to minimize text- sext-generttion losses, such as DPO and SFT losses, as the regularization term for GRM. To independently study the effectiveness of these two regularizations and reduce GPU memory usage, we introduce three practical implementations: DPO regularization, DPO without reference regularization, and SFT regularization.
DPO Regularization. By setting appo = a and OSFT = in 7 we can directly adopt the DPO loss regularization term for GRM regularize the hidden states:
TTB (Me xx)
To (yr xr)
(DPO(OLM) = -- log a 3llg
3
(8)
æ)
Tea x))
where Tee is the base model serving the reference model, and ToLM is our optimized policy. B is coefficient that controls the KL penalty between Th and Tre.- Notably, T0 shares the same base model with the reward model rv. except for the output layer.
DPO Regularization w/o Reference Model. While straightforward, the use of a reference model in DPO regularization can be memory-intensive for large models. To address this, and inspired by prior works that eliminate the need for reference model 42
43 introduce the DPO regularization without a reference model, denoted as EpPO-noret- This method reduces the need for large GPU memory during training. The loss function LDPO-Boret defined as:
Loro = -E log ( [Blog
TBeM (ye xx)
Te (Br xz) (
})
(9)
0 and OSEr in Eq 7 term SFT regularization, thereby reducing the computational cost. This method only maximizes the probability of the chosen responses:
SFT Regularization. By setting appo =
we can simplify the regularization
(SFT(OLM) = (aya)> (log o (8 log (No (ve xx))]]
(10)
This equation differs slightly from the standard SFT objective to maintain coherence with the above two cases within the regularization suite and avoid the need for hyperparameter adjustments for a.
Please refer Appendix C.3 for discussion.
3.3 Advantages of GRM
In summary, GRM offers three key advantages: (1) Mitigating feature distortion. The application of text-generation loss helps maintain the text-generation ability of the base model and prevents

---

excessive feature distortion. Simultaneously, it also adapts the model to the data distribution of preference learning. (2) Prevention of Overfitting. The text-generation regularization derived from an adversarial training objective helps prevent the reward model from overfitting to certain spurious features, which can be detrimental to OOD generalization. This effect becomes more pronounced when the preference data includes erroneous comparison pairs or when the dataset size is limited. (3) Efficiency. GRM is an efficient solution that does not require training multiple reward models or additional training data. Additionally, different choices of loss type entail varying memory and computational costs. Interestingly, we find that the simplest option, SFT regularization, proves to be the most stable choice.
4 Experimental Setup
Datasets. For training reward models, we leverage the Unified-Feedback dataset 2 which stands as one of the largest collections of pairwise feedback datasets. In Section [. we train all reward models on a subset of 400K and 40K samples from the Unified-Feedbback dataset and evaluate them on the hold-out 8K eval set. In addition, for evaluating model performance out-of-distribution (OOD) preference data, we utilize datasets such as HHH--llgnment Judgements| 45 and RewardBench [46). The HHH-Alignment dataset evaluates language models on helpfulness, honesty, and harmlessness, while the MT-Beecc dataset contains human preferences for model responses to MTT-becch questions. Besides, RewardBench is new benchmark designed to evaluate the capabilities and safety of reward models. We consider HHH--Aligment, MTT-Bench, and RewardBench as OOD evaluation tasks because the prompt and response distributions differ from our training distribution. For the RLHF experiments in Section 5.2 we downsample 20K data from Unified-Feedback Feedback for training reward models and optimizing the PPO policy, and another IK data for evaluating BoN or the learned PPO policy.
MT-Bench Human
Unified-
Base Models. In the preference learning experiments, our base models include gemma-2B-4 47 and Mistral-7B: -ann-------nttuu--O.... [48]. For the RI.FF experiments, gemma-2B-it serves as the policy model for both BoN and PPO experiments, whereas the gold reward model is 7B human preference model finetuned using the entire Feefieekeee dataset.
Baselines. We compare the performance of GRM with several baselines, including Baseline Classifier trained using the original reward loss in Eq | Frozen Classifier that fixes the base model's feature and only finetunes a nonlinear classification head; Margin that adds additional margin in the original reward loss 10 39:; Label Smooth that mitigate the overfitting problem by penalizing overconfident model outputs 339; Ensemble method with group of 3 reward models [18] to calculate the average or minimum values as rewards. In addition, for RewardBench, we present the performance several existing open-source state-of-the-an reward models for better reference, including PairRM [49], Starling-RM-7B/3488 (50), and UltraRM-133 [51] For more experimental details and additional results, please refer Appendix B and Appendix C respectively.
5 Evaluation Results
We present a comprehensive evaluation of GRM, utilizing both in-distribution (ID) and out-of- distribution (OOD) datasets, as well as existing benchmarks for reward models. Furthermore, explore the impact of GRM on the overoptimization issue in RLHE Our primary findings can be summarized as follows:
GRM significantly enhances the generalization capability of reward models, resulting in substantial improvements both ID and various OOD evaluation sets (Section 5.1).
All three types of text-generation regularization losses can improve the generalization performance, with the SFT regularization being the most effective and stable (Section 5.T)
GRM demonstrates robustness in the limited dataset setting, outperforming basclines by an even larger margin (Section 5.I).
http:///www...aaaaaa//a///aeeena Facecde-e htps://huggingface.cmm
_alignment
htpss//uuggingface http://www.c.lllltt///////mmmmma benchh human_juuuuu jumaannjdggents/
reward-model model-Mistral- 7B instinuttttttt Unnttrutt-Uiiiie----e Unified-Feedback
5

---

GRM effectively mitigates the overoptimization issue in both BoN and PPO (Section 5.2
GRM also exhibits robustness against label noise in the preference dataset (Section 5.2
Table 1:
Results ID and OOD evalua- tion with 400K training data from Unified- Feedback. The best performance in each task is bold and the second best one is under- lined.
Table 2: Results on ID and OOD evalua- tion with 40K training data from Unified- Feedback. The best performance in each task is in bold and the second best one is under- lined.
MT Feedback Alignment Bench
Unified
HHH
Unified
HHH
MT
Reward Model
Reward Model
Feedback Alignment Bench
Classifier (Frozen)
63.8
66.4
69.5
Classifier (Frozen)
63.9
68.6
68.2
Classifier (baseline)
72.1
73.4
71.2
Classifier (baseline)
68.8
70.3
69.1
Classifier margin
72.0
75.0
72.6
Classifier margin
69.6
71.0
Classificr label smooth
71.5
72.1
71.2
Classifier label smooth
68.5
68.8
71.9
Classifier Ensemble
72.8
76.8
73.7
Classifier Ensemble
69.9
72.2
71.1
GRM w/ dpo (ours)
73.8
79.2
73,4
GRM w/ dpo (ours)
70.2
71.6
71.3
GRM w/ dpo-noref (ours)
73.9
79.7
73.0
GRM w/ dpo-noref (ours)
71.4
76.6
72.1
GRM w/ sft (ours)
73.2
79.8
73.4
GRM w/ sit (ours)
71.5
78.7
73.0
5.1
Evaluation on Reward Modeling
ID and OOD Evaluation. The results, shown Table [ and Table 2 illustrate the evaluation performance of different reward modeling methods using the gemma-2B-i base model on both ID (Unified-Feedbach) and OOD (HHH--lignment and MT-Bench) datasets. Regardless of the size of the training data (400K or 40K), our proposed method, GRM, with three types of regularizations, consistently outperforms the baseline models on both the IID evaluation set and the two OOD datasets. For instance, GRM w/ sft with 400K training data enhances the baseline from 72.1 to 73.2 in ID score, and improves the HHHH-Alignment score from 73.4 to 79.8 and the MT-Bench score from 71.2 to 73.4. Notably, the improvement in OOD performance is significantly larger than that in ID. These results suggest that the GRM methods are highly effective in evaluating unseen preference data, demonstrating substantially robust generalization capabilities.
Regarding other baseline models, the Frozen classifier, which maintains its base model's parameters. exhibits the lowest ID and OOD scores. This suggests that the pretrained features of the base model alone are insufficient for effective preference learning, emphasizing the importance of fine- fin-ttning the base model's ' features to the preference task. Furthermore, the margin loss and label smoothing techniques do not consistently improve the ID and OOD tasks, whereas the ensemble baseline consistently enhances both ID and OOD scores. Despite requiring the training of multiple reward models, ensemble-based methods still do not surpass GRM, particularly when learning from 40K training set. These results highlight the substantial improvement and generalization capability of GRM preference learning.
Comparison of Different Regularizations. As observed in Table T when the training dataset sufficiently large, GRM with three types of regularizations (namely GRM w/ dpo, GRM w/ dpo-noref, and GRM w/ sft) perform comparably. This demonstrates that GRM robust to the choice of regularization type when the dataset is large. However, in Table 2 where the training data is limited to 40K, a clear trend emerges: GRM w/ sft outperforms GRM w/ dpo-noref, which in turn outperforms G w/ dpo, both the ID and OOD scores. Interestingly, the simplest form of regularization, SFT regularization, not only requires the lowest training cost but also yields the most stable overall results. Consequently, we adopt it as the default choice for our subsequent study.
Results on RewardBench. In Table 3 and Table 4 we evaluate GRM and various baselines on RewardBench across chat, chat-hard, safety, and reasoning task groups. We consider a variant of GRM with a linear reward head instead of the default nonlinear reward head as detailed in Appendix B In Table 3 the 7B baseline matches the score of Starling-RMs RM-7B [50] while GRM (linear) w/ sft demonstrates considerable improvement, increasing the average score from 76.3 to 79.5. Comparing variants of GRM, we can conclude that: (1) SFT regularization performs better than the DPO w/o reference model regularization, and (2) GRM with a linear head achieves a better overall score than that with nonlinear head, especially in the challenging reasoning task group.

---

Table 3: Results on RewardBench with 400K training data from Unified- Feedback.
Reward model
Average
Chat
Chat-Hard
Safety
Reasoning
PairRM
58.7
90.2
53.0
31.5
60.0
Starling-RM-11
76.2
98.0
43.4
88.6
74.6
Marling-RMM3343
84.0
96.9
59.0
89.9
90.3
UltaRM--33
69.8
96.1
55.3
45.8
82.0
Base Model: Gemma 2b it
Classifier (basclinc)
68.2
95.5
38.0
73.8
65.3
Classifier + margin
70.2
95.8
38.4
73.9
72.5
Classifier + label smooth
70.6
94.4
37.3
73.2
77.4
Classifier + Ensemble
71.0
98.0
37.5
77.3
71.3
GRM (linear) w/ dpo noref (ours)
70.2
96.7
39.0
76.4
68.5
GRM (linear) w/ (ours)
96.1
80.3
69.3
GRM w/ dpo noref (ours)
70.2
95.8
40.1
78.7
66.2
GRM w/ sft (ours)
70.8
97.8
42.1
77.9
65.2
Base Model: Mistral 7b Instruct
Classifier (bascline)
76.3
96.6
52.4
86.7
69.5
Classifier + margin
74.5
96.4
85.3
64.8
Classifier + label smooth
76.3
97.2
49.8
85.8
72.3
Classifier + Ensemble
76.6
96.6
51.8
85.1
73.0
GR (linear) w/ dpo noref (ours)
78.3
98.0
53.3
86.4
75.3
GR M (linear) w/ sft (ours)
79.5
97.8
54.6
86.3
79.2
GRM dpo noref (ours)
78.0
97.8
54.0
85.7
74.4
GRM w/ sft (ours)
77.6
98.0
55.3
85.8
71.2
Table 4: Results on RewardBench with 40K training data from Unified-Feedback
Reward model
Average
Chat
Chat-Hard
Safety
Reasoning
Base Model: Gemma 2B it
Classifier (baseline)
64.5
95.8
37.3
59.9
64.8
Classifier + margin
66.1
97.2
37.5
56.8
72.7
Classifier + label smooth
91.6
39.0
53.8
60.2
Classifier + Ensemble
96.1
38.2
58.8
67.6
GR M (linear) w/ dpo noref (ours)
61.7
94.7
38.4
62.5
51.2
GRM (linear) w/ sft (ours)
69.5
94.7
40.8
65.4
77.0
GR M w/ dpo noref (ours)
66.6
92.5
39.9
72.5
61.4
GRM w/ sft (ours)
66.8
94.1
41.9
69.5
61.5
Base Model: Mistral 7B Instruct
Classifier (baseline)
68.2
89.7
50.7
74.7
57.9
Classifier + margin
62,8
89.7
47.1
70.7
43.6
Classifier + label smooth
72.1
94.1
47.1
67.5
79.7
Classifier Ensemble
69.3
89.6
50.2
72.7
59.0
GRM (linear) w/ dpo noref (ours)
77.8
96.9
52.9
82.7
78.8
GRM (linear) w/ sft (ours)
78.3
96.7
52.4
81.5
82.5
GR M w/ dpo noref (ours)
78.6
97.8
54.6
82.0
79.9
GRM w/ sft (ours)
78.4
97.2
54.2
83.6
78.6
Regarding the baselines, consistent with previous results, the margin loss and label smoothing do not provide coherent improvement over the baseline. While ensemble methods effectivvly improve upon the baseline, they still underperform GRM. Overall, these results demonstrate that GRM is a strong contender in reward modeling tasks, exhibiting superior performance across various benchmarks.
Comparison of Different Dataset Sizes. Another noteworthy observation is that GRM exhibits greater robustness to the size of the training dataset compared baselines. For instance, in Table and Table 2, when the training data size decreases from 400K to 40K, the baseline's HHH Alignment score and MT-Bench score drop from 73.4 and 71.2 to 70.3 and 69. respectively. In contrast, GRM with SET regularization only slightly drops from 79.8 and 73.4 to 78.7 and 73.0, respectively. This

---

An Piateer PLE7AD
An
(a)
(c)
(d)
Figure 2: Proxy scores and gold scores of BoN experiments for base models (a)(b) gemma-2b-it and (c)d) Rewards are normalized start from 0. GRM demonstrates robust ability select the best response aligned with the gold rewards as the KI. Divergence increases.
Proxy and gold scores are in dashed and solid curves, respectively.
trend is consistent in Table 3 and Table 4 Specifically, for the Mistral 7B Instruct base model, the baseline's 's average score drops from 76.3 to 68.7 2 when learning from 40K training data, while GRM (linear) w/ sft only drops from 79.5 78.3. These findings suggest that the prior reward training paradigms are sensitive to smaller dataset sizes. In contrast, GRM is robust even with limited dataset.
Full Parameter Training Results on a Larger Dataset. To further demonstrate the effectiveness of GRM, we trained the GRM using the Ilama3-8b-ine -b-iissrruut model [52]. We perform full parameter fine-tuning for epoch on one of the largest open-source preference datasets Our results, presented in Table [] highlight the considerable potential of scaling GRM to larger models and datasets. Especially, GRM outperforms a 34B reward model and even GPT-4 as a judge. It worth noting that the GRM significantly improves the performance of the 8B reward model from 84.7 to 87. 87.0, using the same base model and training data as MeairX-LLaMA-RRM-V particularly remarkable the challenging 'Reasoning' section.
[40]. This improvement
Table Results full parameter training on RewardBench.
Reward model
Average
Chat
Chat-Hard
Safety
Reasoning
GRM (Ours, 8B)
87.0
98.6
67.8
89.4
92.3
gpt-4-0122-previet
85.9
95.3
74.3
87.2
86.9
gpt-4-turbo2022--044
85.1
95.3
75.4
87.1
82.7
TstairX-LLaMA3-RMMSS
84.7
99.4
65.1
87.8
86.4
Starling-RM-34B
82.7
96.9
57.2
88.2
88.5
5.2 Evaluation on RLHF
Best-of-n Sampling (BoN). Fig presents the results of BoN sampling for base models of sizes 2B and 7B. For all BoN experiments, we utilize a 20K subset from the Feeffedd-Feebackk dataset, labeled by the gold reward model, to train proxy reward models. Following the | [8]. conduct BoN sampling on a 1K held-out test set from n responses for each prompt, based on the scores of the trained proxy model. The selected responses their gold scores are averaged over the IK test prompts. The average gold score reveals the true quality of the responses selected by the proxy reward models. We the KL Divergence from corresponding to the number of responses ranging from to 405 for each prompt, according to the equation Kl-BoN log n-1 Ideally, good proxy reward model should yield larger average proxy and gold scores the KI. increases. However, the average gold scores of some baseline methods plateau or even drop after KI. > 4, such as the baseline reward model in Fig 2(d) despite their proxy scores continuing to increase in Fig [(c) This suggests that these reward models suffer from the overoptimization issue.
then scored using the gold reward model, and

---

Caa Dn Ang Picees
Tee
rago Kimel
(a)
(b)
(d)
(c)
Figure 3: Proxy scores and gold scores of PPO experiments for reward model based on (a)(b) gemma- 2bitt and (c)(d) -Bstrat-7 All rewards are normalized to start from
Bark perreau
per
Audd
Brannse
boatle
Tet 2000 1966 mns 200
Tei 2000 cate
(a)
(b)
(c)
(d)
Figure 4: Proxy scores and gold scores of (a)(b) BoN experiments and (c)(d) PPO experiments with 25% label noise. All rewards are normalized start from 0.
In contrast, GRM consistently demonstrates an increase in both the proxy score and gold score, indicating that it effectively mitigates over-optimization. This advantage is more pronounced in the 7B base model, where GRM achieves average gold score of 1.5, while the baseline reward model only reaches a score of 0.5. Regarding other baselines, we find that the margin loss and ensemble methods (especially the 'min' strategy) contribute to the robustness of the reward model. However, they still do not compare favorably with GRM. These results underscore the strong potential of GRM to serve reliable and robust proxy reward model for RLHF
Proximal Policy Optimization (PPO). To investigate whether GRM can effectively mitigate the overoptimization issue in PPO, we further employ those 2B and 7B reward models obtained from the BoN experiments to fine-tune the policy model (gemma-2b-in using PPO. The training and evaluation datasets are identical to the BoN experiments. We train PPO for one epoch on the training set, comprising 20K training samples. As depicted in Fig PPO exhibits stronger tendency to hack the learned reward models compared to BoN. The gold scores baseline methods begin to decline early in the training process, while their proxy scores increase, indicating clear overoptimization issue. In contrast, GRM demonstrates superior robustness in terms of the gold score, which rises with the increase in proxy scores. This validates that GRM can effectively alleviate overopiimization for PPO. Please refer to Appendix D for a clear comparison of the results generated by PPO.
Robustness to Label Noise. Human preference data typically contains around 20 30% noise, as highlighted in previous studies [39]. Such inconsistent preference data can render the reward model less generalizable 33 531 and hinder policy learning (54] 55 56), leading performance decline. To evaluate the robustness of GRM against label noise, we incorporate a 25% label noise into the 20K training data for all proxy reward models. The results are depicted in Fig Most gold scores expose a more severe over-optimization issue, as compared to the results in Fig 2(b) and Fig 33b) indicating that those reward models are heavily overfitting under the noisy label setting. On the contrary, GRM exhibits superior robustness under noisy conditions, consistently achieving peak gold score over 1.0 without significant decline. This demonstrates that GRM is highly accurate and robust at measuring the sample quality, even in the presence of noise within the training data.

---

Related Works
Reward Modeling, Reward models, trained on human preference data, are crucial in guiding RLHF
g
or prompt optimization [58. Recent studies have concentrated on developing advanced
reward models to improve the performance of LLMs in RLHF. One approach involves enhancing
reward modeling by improving the quality or quantity of preference data [59] 60 61.
Other strategies
focus on learning token-wiss -wie dense rewards [2] 63 or multi-objective rewards [38 Additionally,
series of works aim to enhance the robustness of reward models against preference inconsistencies. Techniques such as adaptive margin [O), contrastive learning [39], and meta-learning [64] are
employed to improve the model's s ability differentiate between chosen and rejected responses.
Mitigating Overoptimization in RLHE. Reward models tend to overfit and struggle to generalize beyond the training distribution, which often leads to the overoptimization issue [17. One approach to mitigate this is to penalize overly confident model outputs using label smoothing (39] or SFT regularization 24 25 Alternatively, the model and data can be iteratively updated, replacing hard labels with soft labels [65]. Ensemble techniques, which train several reward models, can also help Adversarial
reduce reward hacking and manage shifts in distribution [8
66 32
67
68
Preference Optimization employs adversarial learning between reward models and an LLM agent to address the gap in generation distribution [69]. Recent studies have also utilized uncertainty to mitigate reward over-optimization, including the integration of an uncertainty penalty into rewards
70,, or the construction of a confidence interval for gold rewards based on uncertainty estimations
23
7
Conclusion
In this study, we introduce an efficient approach aimed at enhancing the generalizability and ro- bustness of reward learning for large language models. By incorporating regularization techniques on the hidden states of reward models, our method demonstrates substantial improvements in the generalization performance reward models for unseen data. Moreover, our approach effectively mitigates the issue of overoptimization in RLHF. We believe that our findings hold promise in inspiring future research efforts towards the development of more robust reward models that can facilitate the alignment of large models through cost-effective solutions.
Limitations
In this study, we evaluate the robustness of GRM against label noise by introducing 25% level of synthetic noise into the training data for all proxy reward models. This is achieved by randomly flipping chosen and rejected labels. Due to cost considerations, we conduct synthetic experiments in line with community practices [8 39,, as using labelel data is not feasible for us. However, synthetic data may introduce biases that don't accurately reflect real-world scenarios. Future research should aim to mitigate this limitation by incorporating experiments with human-labeled data, providing more thorough evaluation of the reward model's 's robustness. Another limitation of our study is the computational restriction preventing us from testing GRM with parameter sizes exceeding 1OB. Further efforts to extend our method to larger reward models could be highly promising.
Acknowledgement
Tong Zhang is partially supported by an NSF IIS grant No. 2416897. Huan Zhang was supported by the AI2050 program at Schmidt Sciences (AI 2050 Early Career Fellowship). The authors would like to thank the reviewers and readers for constructive feedback on the manuscript.
References
[1] Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810 10.004055 2018.
10

---

[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
[3] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 07258, 2021.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
[6] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909.999999999 2019.
[7] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 27X4v:220.008862, 2022.
[8] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[9] R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.
[10] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
[11) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.0322
2023.
[12] Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09677 2023.
[13] Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402. 10207, 2024.
[14] Yong Lin, Skyler Seto, Maartje ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. On the limited generalization capabil- ity of the implicit reward model induced by direct preference optimization. arXiv preprint a4Xiv:2409.0m 03650,, 2024.
[15] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint 0rXiv:1606.06565. 2016.
[16] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems,
2020.
[17] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[18] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023.
[19] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' A'mor, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohi, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint 21X1v:231..09244, 2023.
11

---

[20] Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong. Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. arXiv preprini arXiv:2309. 17230, 2023.
[21] Kyuyoung Kim, Jongheon Jeong, Minyong An, Mohammad Ghavamzadeh, Krishnamurthy Dvijotham, Jinwoo Shin, and Kimin Lee. Confidence-awee aware reward optimization for fine-tuning text-to-image models. arXiv preprint arXiv:2404.01886; 2024.
[22] Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprini arXiv:2310.0437 2023.
[23] Xiaoying Zhang. Jean- Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcom- ing reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint arXiv:2403.05177.. 2024.
[24] Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer, 2024.
[25] Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang. Dale Schuurmans, Yuejie Chi, and Bo Dai. Value- inbteiincentivizee preference optimization: A unified approach online and offline rlhf. arXiv preprini arXiv:2405. 19320, 2024.
[26] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164 2020.
[27] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 33:1179- 1191, 2020.
[28] Ying Jin, Zhuoran Yang. and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference Machine Learning, pages 5084-5096. PMLR, 2021.
[29] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 15:23851-23866, 2022.
[30] Hao Sun. Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in neural information processing systems, 1-37779-377335 2022.
[31] Rui Yang, Lin Yong. Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned rl? In International Conference Machine Learning, pages 39543-39571. PMLR, 2023.
[32] Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits weight averaged reward models. arXiv preprint arXiv:2401. 2tt.:2201..1117... 2024.
[33] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.1000 10054, 2022.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1777v:/707.06347 2017.
[35] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952.
[36] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023.
[37] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304. 067677 2023.
[38] Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang. Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Imms for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint 4rXiv:2402.1885 2024.
12

---

[39] Binghai Wang. Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.0000 0060000 2024.
[40] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint wrXiv22405.07866 2024.
[41] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprini arXiv:2305. / 18290, 2023.
[42] Jiwoo Hong, Noah Lee, and James Thome. Reference- free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691, 2024.
[43] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
[44] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodde Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant laboratory for alignment. CoRR, abs/21 12.008601 2021.
[45]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[46] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
[47] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
[48] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprini 230.0223 2023.
[49] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint 2.Xi::2300602566, 2023.
[50] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling- 7b: Improving Ilm helpfulness & harmlessness with rlaif, 2023.
[51] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint 17X01223/0.01377, 2023.
[52]
Al@Meta. Ilama 3 model card. 2024.
[53]
Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for 1lms. arXiv preprint arXiv:2404.044 102, 2024.
[54] Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption. In International Conference on Learning Representations, 2024.
[55] Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption- robustt offline reinforcement learning with general function approximation. Advances in Neural Information Processing Systems, 36, 2024.
[56) Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban, Adish Singla, and Goran Radanović. Corruption robust offline reinforcement learning with human feedback. arXiv preprint arXiv:2402.06734 2024.
[57] Hao Sun, Thomas Pouplin, Nicolás Astorga, Tennison Liu, and Mihaela van der Schaar. Im- proving IIm generation with inverse and forward alignment: Reward modeling, prompting. fine-tuning, and inference-time optimization. In The First Workshop on System-2 Reasoning at Scale, Ne4r
13

---

[58] Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl. In The Twelfth International Conference on Learning Representations, 2023.
[59] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S I Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
[60] Hao Sun and Mihaela van der Schaar. Inverse-tignment: Inverse reinforcement learning from demonstrations for Ilm alignment. arXiv preprint arXiv:2405. 15624, 2024.
[61] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with feedback. arXiv preprint arXiv:2309.0022
2023.
[62] Alex Chan, Hao Sun. Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.00782 2024.
[63] Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404. /8922, 2024.
[64] Shihan Dou, Yan Liu, Enyu Zhou, Tianlong Li, Haoxiang Jia, Limao Xiong, Xin Zhao, Junjie Ye, Rui Zheng. Tao Gui, et al. Metarm: Shifted distributions alignment via meta-learning, arXiv
preprint arXiv:2405. 00438, 2024.
[65] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335 2024.
[66] Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang. Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.063: 06256, 2023.
[67] Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Im- proving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint 2rX1v:2401.166:1 2024.
[68] Yifan Hao, Yong Lin, Difan Zou, and Tong Zhang. On the benefits of over- parameterizzaion for -iit---diittitttion generalization. arXiv preprint arXiv:2403.17592. 2024.
[69] Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, and Nan Du Adversarial preference optimization. arXiv preprint arXiv:231 1.000045 2023.
[70] Adam X Yang. Maxime Robeyns, Thomas Coste, Jun Wang, Haitham Bou-Ammar, and Lau- rence Aitchison. Bayesian reward models for Ilm alignment. arXiv preprint a:XIv:2402.13210, 2024.
[71] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.0444 044778 2022.
[72] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 48,45, Online, October 2020. Association for Computational Linguistics.
[73] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https://gitae
com/huggingfacce/ban
2020.
[74] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 09685, 2021.
14

---

A Deriving the Regularization Term
To derive the potential formulation of the regularization term, consider the following adversarial optimization problem: learning a reward model against an adversarial policy.
0 arg min { (fr + max J (0, m)
(11)
The term for policy optimization [(0, πT can have different formulations, but a KI. divergence regularized optimization objective is generally used in training the policy []6 8 71). Moreover,
has an advantageous property that the inner optimization problem has an analytical solution, which can simplify the problem.
[6, T) Ep-Daaraa(())) (ro v) BE [LL (IT (-12)) Tret (+(=))
(12)
where B > 0 is the coefficient controlling the regularization degree and Tref the reference model. The analytical solution of [0, x) is formulated as follows:
Zo (z( Trof (w(z) exp (ro(a, v)) Za(x) = Σ Trat Preetyyll (v'lx) exp exp (//(1//)//))
(13)
Equivalently, we can obtain the formulation of reward described by zq and aret as in
(0 y) = 3 (log क (u|z) - log Tret (y) (z) + log (x())
(14)
Following recent theoretical analysis [25] we define fixed calibration policy Teal that is independent of the algorithm, which has the calibration effect of centering the reward function while incorporating additional policy preferences into the objective.
Definition
Teal is a fixed calibration policy for reward model ro and the dataset D that satisfies:
B D.y (ro(s,
=0
Therefore, rewrite max, [8, T) as:
max (0, J0, T²) E D. D (+1x) (r)(2,)) Blog म (g(xz) log Jeet []]]
y 3 (-11) [log Zo(x)) Ex D.y (- [lo (xx)
(15)
En- (r) y) B(oo मH (v/z)) log ret (y]z))
-.. [log 7 (y/ar) log vor (y) [-)]
The second line is established because log (x) is independent of the distribution y. Besides, the last line just adopts the definition of Teal-
Incorporating Eq 15 and Eq 14 into Eq II we can transform the min-max optimization problem into
standard optimization problem by considering the policy Tz.
0 = arg min 0
{1 a) Creward (0) + aLeeward + max ((0,T)) m)}
Th | x)) ) -Blog TeerlBe xx)
(pr x)) Trer xx)
arg min (1 Crewand (f) aB (w) logo Blog
-, D. D.y [log Th (v)x)) - log Trer (ylr(
= arg minn((1 - (0) + (=L -
[log nह (y|+))}
(16)
Here, (T))(T)) replace the second term, it the same as DPO objective [41. In the second line, we put the reward described by Eq 14 into aLreward- In the final step, we remove the Tref term it does not depend the parameters of the reward rg, unlike rÿ which is dependent reward rg.
Interestingly, if we set the calibration policy Teal as the chosen responses ye from the dataset D, the last term becomes an SFT loss. Thus, we can derive the general regularization terms in our framework by renaming the coefficients for ái apro and asfr, and removing the constraint that appo
arg min (1 Creward) + (DPOoTPO(TT) (SFT
(17)
15

---

Notably, the two regularization terms come from different sources, where LDoO is from the reward loss and LSFT derived from the adversarial term. This may be the reason why SFT regularization is more helpful than DPO regularization in our empirical results. Inspired by the objective in Eq we relax the relationship between ro and Ti and propose learn a reward model parameterized by and language model head parameterized by LLM. both sharing the same hidden states.
Discussion. In Eq 17 we retain both the reward model re and the policy zh. and replace vd with language head TêL.. A simpler solution is to keep only the reward model by replacing Tे with re. which leads the following objective:
arg MnjinCCewantt (0) -Er.,..-rrroa Mo]] 7/8 (log Zo(x)]]-
This approach can be understood minimizing reward loss while applying regularization to maximize the rewards of selected responses relative to the overall rewards. However, this method is limited by the inefficient calculation of Z over the response distribution generated by TSFT- Therefore, we propose our solution, GRM, which involves a reward model that shares hidden states with a language head. This setup captures certain correlations through shared parameters, helps prevent feature distortion, and is both cost-effective and highly efficient.
B Implementation Details
Baseline Details. All baseline reward models employ the AAttoMooeFFosSeeeeneCCaaaaii class from transformers [72), which utilizes a randomly initialized linear head to derive rewards. We then train each reward model to minimize the loss function with the training data. For ensemble baselines, we train reward models with different random seeds and aggregate their outputs via the 'average' or the 'minimum' strategy. We adopt the average value for the ensemble baseline in Section [. as we find that the minimum value can decrease accuracy and underperform the average one. But for the RLHF experiments in Section [.2] report both results because find some sometimes the "minimum" strategy can work better than due to its pessimism.
The margin loss function 10) is defined as below:
Emargin (0) -B(r.,....>>D [log (ar (re(z, ve) re (e vr) -m(r))]----
which enhances the reward model by emphasizing the differences in rewards. We use the scores between chosen and rejected responses in the Unified-Feedback dataset to calculate m(r).
Additionally, the label smooth loss is defined as
Esmooth (0) -.......... [(1 - e) log (a (re(z, no) re ra(2,, v+)) log (ar (ro(x, vo) (e( vr)))],
where set 0.1. The label smooth loss function enhances the model's resilience to certain degree of errors, thereby alleviating the problem of overfitting.
GRM Details. For GRM, the default reward head is configured a linear layer with shape (hidden size, 1024), followed by a ReLU activation function, and another linear layer of shape (1024, 1). The weight of the text- text-generation regularization is set to 0.01 and the coefficient 3 in our regularizations set to 0.1 by default. In the case of the GRM (linear) variant, the reward head is directly set as a linear layer of shape (hidden size, 1). We found smaller a 0.001 is better for the linear variant.
Training and Evaluation Details. We implement all methods based on transformers 72 and trl 73. More details are listed in Table To use the Uniffed-Feedback dataset, we downsample the training data from the "all' set and use all the 8K test data for evaluation. For the HHH Alignment dataset, adopt the average score of all four subsets the result. For the main experiments trained with LORA, we truncate the inputs for all reward models over 1024 tokens. All reward models are trained for two epochs using a learning rate of 1 x 10 and batch size of 16. We load the model with the bf16 precision. Regarding the full parameter training, we truncate the inputs over 4096 tokens and train the reward model for one epoch with a learning rate of 2 x 10 and a batch size of 512 (with gradient accumulation).
Computational Resources. We use NVIDIA RTX A6000 49G for our experiments. Training a 2B reward model with LoRA [74] on the 40K training data for epochs requires approximately 30.4 GPU hours. A 7B reward model requires approximately 93.6 GPU hours.
16

---

Table 6: Key implementations of the text generation experiments.
Basic information
Base models
gemma- and Desstal-7B-Instrucek
Quantization for training
b
Fine-tuning strategy
LoRA 74
LoRA
32
LoRA alpha
64
LoRA dropout
0.05
Optimizer
Adamw 16
Batch size
Leurning Rate
x 10-5
Learning Rate Scheduler
cosine
Warmup Ratio
0.03
GRM (Ours)
Regularization weight Temperature | for loss functions
0.01 by default, and 0.001 for the lincar variant
0.1
PPO 34
KL regulaization Epochs learning rate lambda for GAE gamma clip range
0.0
x 10
0.95
1
0.2
Number of optimization epochs per batch Number of tokens during generation
4
512
Dataset and Gold Reward Model
Main Training Dataset Eval dataset: HHH--Aligmmen Eval dataset: MT MT-BBench Human Judgements Gold Reward Model for BoN and PPO
Unified- Feedback
alignment
https///ugguunglae..co hww.://mnnnaaacooootttaaaaasm//a bench human, judgments
reward-model Mrstraa-B----mitttdde Feedback
C Additional Experimental Results
C.1
Comparing with Frozen Backbone
The effect of the random head for downstream finetuning of pretrained model is studied by [33 both theoretically and empirically (across a range of computer vision tasks). is also easy to validate in the preference learning setting when using a smaller dataset size. We included baseline, "Classifier (Frozen)", which fixes the base model's features and only fine-tunes the classification head. When the dataset size is 8K (see 77) the OOD evaluation results of the bascline reward model (without freezing the backbone) are worse than those of the frozen one, demonstrating the negative effect of distorting pre-trained features. However, sufficiently large, this negative effect can be mitigated, and the baseline reward model can surpass the frozen reward model due to having more trainable parameters to fit the large preference dataset.
would like note that when the dataset size is
In contrast, by regularizing the hidden states, our GRM can achieve the regularizing effect while fine-tuning all parameters, showing strong performance with both large and small dataset sizes.
Table 7: Reward model performance trained with 8K data.
Reward Model
Unified Feedback (ID) HHH Alignment (OOD) MT Bench (OOD)
Classifier (Frozen) Classifier (Baseline)
62.2
68.8
67.6
66.1
65.1
67.7
GRM (ours)
69.0
71.9
69.8
C.2
Choice of Training Epochs
In our main experiments, we train reward models for epochs with LoRA. We determine this number based a nearly converging validation loss. Specifically, reserve 1% of the training set for
17

---

Wildeton accuracy 1%
73-
72
71
70
Classifier (Baseline)
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
Epoch
Figure Learning curves for reward models on Unified- Feified-Feedback.
validation (e.g., 4K for 400K training data) and found that epochs are sufficient reward modeling with LoRA in our setting- As shown in Figure 5 we observe convergence in the validation loss during the second epoch, with no further improvement the third epoch. For full-parameter training experiments, which are more prone to overfitting, we train the reward model for only one epoch.
C.3
Choice of the SFT objective
In our paper, we consider slightly differrntt form of SFT objective as in Eq 10 A more straightfor- ward objective (SFT(OLM) Ex.....)> )-D [log (Te (ye z))]. In ideal situations, the forms should perform similarly. We also tried the log form but found that it requires different hyperparame- ter tuning for the regularization weight
the loss scale. In Table & and Table 9 "GRM logreg" outperforms the baseline reward model and matches or even slightly exceeds the performance of GRM on OOD tasks when tuned appropriately. This experiment uses the same gemma-2B-it as the base model.
Eq 4 due to changes
We found that the current form of SFT regularization can directly the same hyperparameters our DPO regularization. Therefore, we opted for this solution to maintain coherence with these regularizations and avoid the need for hyperparameter adjustments.
Table 8: Results ID and OOD evalua- tion with 400K training data from Unified- Feedback.
Table 9: Results ID and OOD evalua- tion with 40K training data from Unified- Feedback.
Unified Feedback
HHH Alignment Bench
MT
Unified Feedback Alignment Bench
HHH
MT
Reward Model
Reward Model
Classifier (Baseline)
72.1
73.4
71.2
Classifier (Baseline)
68.8
70.3
69.1
GRM
73.2
73.4
GRM
71.5
78.7
73.0
GRM logreg a 0.005
72.8
77.6
72.8
GRM logreg a 0.005
69.7
72.4
72.8
GRM logreg a 0.001
73.3
80.2
73.6
GRM logreg a 0.001
70.8
80.7
72.0
C.4 Ablation the Regularization Weight
We find the most impactful hyperparameter of GRM is the regularization weight Figure 6lgueenerreenttt an evaluation of GRM's performance under various a values. is evident from the figure that setting to either extreme, such O, relatively large value like 0.1, results in suboptimal oft-of-distribution (OOD) performance. However, selecting an appropriate value between and 0.1 consistently yields higher scores. In all experiments, we default to value of 0.01. This choice has already shown significant performance improvements our experiments.
C.5 Impact of Reward Head Layers on Performance
An interesting aspect to explore how the structure of the nonlinear reward head influences preference learning performance. In Figure 7 we compare the performance of the default GRM (using the SFT
18

---

HHHH-Alinmenn
MT-Bench
79
74
Score
Score
76
71
75
70
0.005
0.01
0.05
0.1
0.005
0.01
0.05
0.1
Figure 6: Comparing different values of a for GRM (2B) on scores of HHH-AlI Alignment and MT- Bench.
HHHH-Alignment
MT-Bench
RewardBench
75
72
71
68-
76
71
75
70
65
19_09g
layer
layyer
ne_reg
1 layer
layer
ne_reg
1 layer
2 layer
Figure 7: Comparing different layers of reward head for GRM (2B) scores of HHH-Aliggmmet, MT-Bench, and RewardBench.
regularization) against variant of GRM that incorporates an additional linear layer and ReLU activation in the reward head, denoted "2 layer". The results indicate that the two-layer version slightly surpasses the performance of the single-layer GRM on MT-Bench and RewardBench scores, but it exhibits decline in the score on the Alig--Aligmmett, Due to this inconsistency, we opted not to include the two-layer version in our main experiments. However, future research focusing on the impact the reward model's structure could yield promising insights.
C.6 Comparison with Additional Variant
In Appendix we derive an objective that retains only the reward model ry by replacing the policy no with a formula of rg. Empirically, this objective is challenging to optimize due the calculation of Zg. As an alternative, we propose a simplified objective by omitting the Zg term:
arg Creward (0) -EE [re (2. ye)]}.
This objective includes a regularization term to maximize the average rewards of chosen responses. However, the second term easily dominate the loss since the reward loss term is constrained by the logsigmoid operator. A more stable approach is to use the following empirical objective:
arg min Creward (0) "Brr. log (ro(x, ye))]
We refer to this regularizer as "positive regularization" or "pos reg" for short. We compare positive regularization with the baseline classifier and GRM with SFT regularization in Tables 10 and The base model for the reward models is gemma-2B 2B-it, and GRM adopts the linear variant for the RewardBench results. "Positive regularization" does yield improvement when the dataset size is limited to 40K, but brings slight overall enhancement when learning from 400K training data.
In contrast, GRM significantly enhances both ID and OOD accuracy, especially when learning from limited preference dataset. These results demonstrate that our approach more effective, even when based on similar theoretical derivation.
19

---

Table 10: Results on ID and OOD evalua- tion with 400K training data from Unified- Feedback.
Table II: Results on ID and OOD evalua- tion with 40K training data from Unified- Feedback.
Unified Feedback
HHH Alignment Bench
MT
Reward Bench
Unifidd Feedback Alignment Bench Bench
HHH
MT
Reward
Reward Model
Reward Model
Classifier (baseline)
72.1
73.4
71.2
68.2
Classifier (baseline)
68.8
70.3
69.1
64.5
Classifier (pos reg)
71.7
75.0
70.7
69.8
Classifier (pos reg)
69.5
70.0
69.6
63.2
GRM w/ sft (ours)
73.2
79.8
73.4
71.5
GRM w/ sft (ours)
71.5
78.7
73.0
69.5
C.7 Regularization with pretraining dataset
In our default design, we use the preference dataset employed to train reward models to regularize the text-generation ability of the language head, eliminating the need for additional datasets. While we believe that other data formats, such as pretraining datasets, can also be beneficial, preference data offers distinct advantage. It allows to avoid using external datasets during reward modeling, which may also better align with the distribution of prompts and responses.
To illustrate this, we conduct experiment using GRM with text-generation regularization an open-source pretraining dataset, seeeterroompuu///RPPaaaaaaa text from Commoncrawl, Arxiv, and books), referred to as 'GRM pretrain reg'. For fairness, only used a pretraining dataset the same size as the training set for reward modeling.
IT-Sample [ (which includes
The results indicate that 'GRM pretrain reg" outperforms the baseline reward model and matches the performance of GRM when the dataset size large (400K). However, when the dataset size is small, using a pretraining dataset is less effective than using the preference dataset.
Table 12: Results on ID and OOD evaluation with 400K training data from open-source pretraining dataset.
Table 13: Results on ID and OOD evaluation with 40K training data from open-source pre- training dataset.
Unified Feedback Alignment Bench
HHH
MT
Unified Feedback Alignment Bench
HHH
MT
Reward Model
Reward Model
Classifier (baseline)
72.1
73.4
71.2
Classifier (baseline)
68.8
70.3
69.1
GRM
73.2
79.8
73.4
GRM
71.5
78.7
73.0
GRM pretrain reg
73.0
79.2
74.3
GRM pretrain reg
70.8
74.5
72.9
C.8
Alignment Result after PPO
To demonstrate the advantage of GRM over vanilla reward modeling, we evaluate the win rate of models after PPO training with GRM against those with the vanilla reward model. The evaluation is conducted using GPT-4o on 100 randomly selected prompts from the test set in Enified-Feedbaat with the order of responses randomly flipped avoid order bias. The results below show signif- icantly higher win rate for GRM than the vanilla reward model across two different base reward models.
Table 14: Win rate of models after PPO training with GRM against those with the vanilla reward model.
Base reward model
Win rate (%)
Tie rate (%)
Loss rate (%)
Gemma 2B it Mistral 7B Instruct
68
27
73
6
21
C.9 Comparison with Label Smooth in RLHF
In Figure g
we observed that reward models trained with label smoothing are vulnerable hacking by BoN and PPO, leading to inferior performance compared to other baselines. The proxy score
hhtp::/wnggggtiiooooeseeeeeeeeee Data- IT-Saplle
20

---

PO Mila sceddl
Amh
CAMBULSS
Eosencc Ing
200
Vas La00 200
1000
101 1000 1000 19S 200
(a)
(c)
(d)
Figure 8: Proxy scores and gold scores (a)(b) BoN experiments and (c) PPO experiments for base models
Proxy and gold scores are in dashed and solid curves, respectively, Rewards normalized start from O.
increases, while the gold score decreases rapidly. This finding suggests that previous robust techniques in the literature may not be effective for RLHF, underscoring the superiority of GRM as a more viable solution.
D Examples in the PPO Experiments
In Tables 15 16 and 17 we present three examples that compare the responses of optimized language models using the PPO algorithm with different reward models. The base models for policy and reward models are all gemma-2b-it as in Section 5.2
For baselines, it evident that the models exploit certain patterns in rewards, such the "Ensemble (min)" methods. This exploitation often leads to a collapse into repeated patterns. Besides, the "Baseline" and "Margin" models tend to disregard instructions refuse respond harmless prompts, as demonstrated in Tables 13 andd[6 Moreover, the baseline methods negatively impact the reasoning ability of language models for the math problem in Table 17 These observations indicate that current reward models can be easily hacked by the PPO algorithm, raising concerns about their reliability.
In contrast, the GRM model demonstrates greater robustness generating mnstructionfoollovingg responses and exhibits better reasoning abilities, even with identical hyperparameters of PPO. Notably, this superior performance GRM is achieved even with a smaller training cost compared to ensemble baselines. These examples underscore the importance of GRM and its effectiveness in mitigating the overoptimization problem, further highlighting its potential in RLHF applications.
E Broader Impacts
The proposed approach enhancing the generalization capabilities of reward models within the RLHF framework offers several positive societal impacts. By improving the accuracy of reward models on out-of-distribution (OOD) tasks, we can enhance the alignment of LLMs with human intent on larger dataset without human labels. leading to more reliable and stronger alignment. Moreover, the regularization technique that preserves the base model's language generation capabilities can contribute to the development of more robust and versatile Al systems, fostering innovation and efficiency across multiple domains. Currently, we do not foresee apparent negative societal impacts stemming from our methods. However, one potential adverse effect could arise if the generalizable reward model exploited for harmful language model training. Therefore, future efforts in Al safety crucial to prevent such misuse.
21

---

Table 15: Examples in the PPO experiments. GRM optimizes better language model aligned with human intention, while other baseline reward models can be easily hacked by PPO.
Prompt
Respond to the latest instruction based on the conversation history. Human: How do I fly kite? I'm happy to explain more about kites. What would you like to know?
Comment
Assistant:
Human:
how get the kite in the
Assistant; Can you tell me bit more about what you 'r trying to do? What kind of kite? What kind wind conditions are you planning to use it in? What is your kite-flying experience?
Human: don't 't have much experience and it's just simple diamond shaped one
Assistant: What wind conditions are you planning to use it in?
Human: just regular low wind i guess
Assistant:
Baseline
Based the conversation history, the human is looking for guidance on how get simple diamond-shiped kite in the air. The assistant recommends considering wind conditions such as the direction and strength of the wind. the kite type. and the intended flying environment, They suggest considering low wind speeds as suitable for beginners.
The response does not follow the conversa- tional tone.
Therefore, the assistant suggests that human rescarch the specific wind conditions and kite specifications suitable for their location and desired flying experience hefore attempting to the kite. Remember to prioritize safety and sock guidance from experienced kite enthusiasts or professionals possible.
Margin
Based the conversation history. the human is seeking guidance on how get a kite the air. However, the assistant cannot provide specific instructions or recommendations without thorough understanding of individual's 's situation and capabilities. It's important to prioritize safety and avoid engaging in activities that could potentially harm the person or property involved. If you have any general kite-flying tips or suggestions, it's recommended to consult with a local park ouldoor enthusiast seek guidance from reputable kite flying organization instructor. Remember to always follow safety guidelines and regulations, especially when engaging in kite flying activities.
The response does not follow the conversa- tional tone.
Ensemble (min)
Sure, here' soprHay guidHü 'ung pulsa palabo() sedi lampa palab pulsa palab pulsa palab pulsa palah pulsa palab pulsa palab pulsa palab pulsa palah pulsa palah pulsa palab pulsa palab pulsa palah pulsa palab pulsa palab pulsa palab pulsa palah pulsa palah pulsa palab pulsa palab pulsa palah pulsa palab pulsa palab pulsa palab pulsa palah pulsa palah pulsa palab pulsa palab pulsa palah pulsa polah pulsa palab pulsa palab pulsa palsh pulsa palah pulsa palah pulsa palab pulsa palah pulsa palab pulsa palab pulsa palab pulsa palah pulsa palah pulsa palab pulsa palab pulsa palah pulsa palah pulsa palab pulsa palab pulsa palah pulsa palah pulsa palah pulsa palab pulsa palab pulsa pulab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palah pulsa palah pulsa palab pulsa palab pulsa palah pulsa palah pulsa palah pulsa palab pulsa palab pulsu palab pulsa palab pulsa palab pulsa palah pulsa palab pulsa palab pulsa palab pulsa palah pulsa palab pulsa palab pulsa (omitted due to repeating)
The model is backed to gen- erate repeated words.
Ensemble (avg)
Sure, here's some advice on how to get a kite in the air safely and effectively: Wind Conditions
A good re- sponse to the prompt.
Choose location with consistent and gentle winds. Avoid strong or winds, as they can make difficult control kite. a Consider the wind speed and direction. Start with lighter winds, especially if you're 're beginner.
*"Kite Selection::*
+ Choose kite that's appropriate for your weight and the wind conditions. A well-sized kite will have sufficient lift to stay aloft and provide a thrilling flight experience. Consider the kite's 's design. Diamond shaped kites are relatively stable and easy to maneuver, while symmetrical kites offer more stability but may be more difficult control.
safeety P
Always fly kites away from people and obstucles. + Use sturdy line bridle that's long enough to allow the kite to fly freely but not become tangled. Start in an open area with clear view. Be patient and allow the kite fly a few minutes before adjusting releasing it. Remenher fly kites responsibly and always prioritize safety.
or
GRM (Ours)
""How to Get a Kite the to
A more infor- mative response to the prompt.
s+1. Choose the Right Kite:**. Select a kite that is appropriate for your skill level and wind conditions.- Consider the kite's 's size, shape (diamond, square, or oval), and the amount of provides.
*42. Set Up the Kite: Unpack kite and follow instructions provided the manual.
Attach the line to the kite's bridle string and adjust the length to aclueve the desired height.
Make sure the kite securely attached to sturdy pole or frume.
#11,, Choose Wind Conditions:** Look for wind conditions that moderate and consistent. Ideal wind speeds are between 5 and 15 mph (8 and 24 kph). Avoid strong winds, as they can make it difficult to control the kite.
#*4, Fly the Kite: - Hold the kite's handle string comfortably, keeping your relased.
Raise the kite slowly until it reaches its peak altitude. Adjust the kite' 's height and angle (angle slightly upwards) control flight path. Gemily move the with your hands change its direction and speed.
rrTipp for #eginners:
Start in flai, open area with minimal obstacles. Use lightweight kite to make it easier to handle. Start Hlynng in the early moming or late afternoon when the wind conditions are generally favorable. Be patient and persistent, as takes time to learn to fly a kite effectively.
22

---

Table 16: Examples in the PPO experiments. GRM optimizes better language model aligned with gold scores, while other baseline reward models can be easily hacked by PPO.
Prompt
EXAM Question 1. Mission Records was created by which lead singer of The Eagles?
Comment
Answer:
Baseline
The mission records were created by Don Henley, lead singer of The Eagles. The band formed in 1974 and releused several successful albums and hit songs throughout the 1970s and 1980s. Henley is widely regarded as one of the most influential and successful singers in the history of rock music. His songwriting and vocal contributions have influenced countless artists and fans worldwide. The Eagles became of the late 20th century, with songs like "Hotel Califomii," "Desperado," " and "Hotel," among many hits. Henley's 's contributions to the band have solidified his status as a legend in the music industry.
Lengthy re- sponse to the question
of the best-selling and critically acclaimed bands
Margin
The context does not provide information about the lead singer of The Eagles, so cannot answer this question from provided context. For accurate information about mission records their creators, is recommended to refer to official sources or seek further research. Remember to always prioritize reliable sources and avoid relying on secondhand or biased information. If you have any specific questions or concerns related to mission records or the artists involved. consider consulting with experts professionals familiar with subject matter.
Unhelpful response question
Ensemble (min)
Mission Records was created by lead singer of Eagles Eagles Keseitar sven pixabay Keseitar pixabuy sven pixabuy pixabay sven pixubay sven pixabuy sven pixabuy sven pixahay sven pixahay sven pixahay sven pixabay sven pixahay sven pixahay sven pixahay sven pixabuy sven pixabuy sven pixabay sven pixubay sven pixabay sven pixahuy sven pixabuy sven pixahay
The model hacked to gen- erate repeated words.
sven pixahuy sven pixabuy sven pixabuy pixabay sven pixabay
sven pixubuy sven pixabuy sven pixabay
pixahay sven pixabay sven pixahay sven pixahay sven pixabay sven pixubay sven pixahay sven pixahay sven pixahay sven pixabay sven pixabay sven pixabay sven pixabay sven pixahay sven pixabay sven pixabay sven pixahay sven pixahay sven pixabay sven pixahay sven pixahay sven pixahay sven pixahay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixahay sven pixahay sven pixabay sven pixabay sven (omitted due to repeating)
pixabay sven pixahay
pixabay sven pixahay
Ensemble (uvg)
Mission Records was created by the lead singer of The Bagles, Don Henley, Mission Records was founded 1971 Nashville. Tennessee. United States. The mission the record label was to provide platform for artists showease their music and talents. Mission Records has released songs by many artists, including Eagles, Bruce Springsteen, and Dolly Parton. Missions Records has also been instrumental in helping to establish Nashville as glohal music hub. Selamat
The model is hacked to gen- erate repeated words.
|-| Selamat |- Selamat | Selamat-14 |- Selamat
| Selamat
1-|
Selamat
Selamar 1-1 SSeamat- |-
Selamat-l --ladtt---lllaa------
Selamat Intere
Selamat Intere-4
Sclamat
Intere Selamat Intere-1-11 Selamat Intere
Selamat Intere
Selamat Intere
Selamat Intere | Selamat Intere Selamat Intere
Selamat Intere
Selamat Intere
Selamat Intere
Sclamat Intere
Sclamat Intere
Selamat Intere
Selamat Intere
Selamat Intere
Selamat Intere
Selamat Intere
Sclamat Intere
Sclamat Intere
Selamat Intere
Sclamat Intere
Selamat Intere
Selamat Intere
Sclamat Intere
Selamat Intere
Sclamat Intere Selamat Intere
Selamat Inters
Sclumat Intere
Sclamat Intere
Sclamat Intere- Selamat Intere
Selamat Intere
Selamat Intere
Selumat Intere
Selamat Intere- Selamat Intere
Selamat Intere
Selumat Intere
Sclumat Intere
Sclamat Intere- Selamat Intere
Selamat Intere
Selumat Intere-
Selamat Intere
Selamat Intere Selamat Intere
Selamat Intere
Selumat Intere
Selumat Intere Selamat Intere- Selumat Intere Selamat Intere Selumat Intere
Selamat Intere Selamat Intere Selamat Intere Sclamat Intere Selumat Intere
Selamat Intere Selamat Intere (omitted due to repeating)
GRM (Ours)
The answer is Don Henley.
A concise reply to the question with short ex- planation.
Mission Records was record label founded by Don Henley 1974. The label was initially focused on releasing albums by The Eagles and other artists the Eagles' genre. Mission Records went to sign and release albums by number of other artists, including Fleetwood Mac, Bon Jovi, and Linda Ronstadt.
23

---

Table 17:
Examples in the PPO experiments. GRM optimizes better language model aligned with gold scores, while other baseline reward models can be easily hacked by PPO.
Prompt
You will be given definition of a task first. then input the task. You given math word problem and you are supposed to apply multiplication or division mathematical operators on the numbers embedded in the text to answer the following question and then only report the final numerical answer.
Comment
A chef needs cook potatoes He has already cooked each potato takes minutes
cook, how long will take him to cook the rest ? Output:
Baseline
"Taakk"e Cook 9 potatoes minutes cach.
Wrong solution.
*=Input:* Number of potatoes cooked 7, cooking time potato minutes.
""Mulipplication"" potatoes 3 minutes/potato 21 minutes chef minutes Therefore, the final numerical unswer **2]**,
x*x *Annwer:**
remaining potatoes.
Margin
x* Task:** Cook potatoes
minutes each.
Wrong solution.
**Ippur:** Number of potatoes cooked 7, cooking time potato 3 minutes.
"Multiplicatioon Operator:** To calculate the total cooking time, we multiply the number of potatoes cooked by the cooking time per potato.
potatoes 3 minutes/potato
minutes.
"eFinal Answer:" It will take the chef 27 minutes cook the remaining potatoes, Please note that the question does not provide any information about the initial state of the potatoes or any constraints, so final unswer may vary depending the specific context and ussumptions.
always recommended to carefully review the tusk and input data before upplying mathematical operations.
Ensemble (min)
Sure here's how calculate the time needed cook the remaining potatoes:
Wrong solution.
**Step 18+; Calculate the total time already speni cooking potatnes: 7 minutes
*=Step 2 Subtract time already spent from total time allotted minutes 7 minutes minutes
"*Step 3*3; Multiply the remaining time by the number of potatocs to cook 2 minutes 9 18
minutes
Therefore. it will take him minutes to cook the remaining potatoes.
Ensemble (uvg)
Task Definition:** Cook potatoes for 3 minutes each. "Input:**
Wrong solution.
potatoes cooked at 3 minutes each
cCCalculations:**
9 potatoes minutes each 27 minutes
Therefore, take the chef minutes cook the rest the potatoes. Answer: 27 minutes.
GRM (Ours)
Sure, here's 's the analysis and solution for the task: x* Definition;
Right solution.
The chef needs cook 9 potatoes.
**Initial Information:
Cooked potatoes =
*"Calculation: se
To cook the remaining potatoes, need to add the number of cooked potatoes to the initial
number of cooked potatoes:
potatoes potatoes potatoes
""Time Cook Potatocs:**
Each potato takes minutes to cook.
Therefore, the total time to cook the potatoes is: minutes
potatoes x 3 minutes/potato Final *nswer:*
It will take the chef #e6 minutes** * to cook the 9 potatoes after he hus cooked 7.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our study proposes a novel approach enhance the reward model's general- ization ability against distribution shifts by regularizing the hidden states, It was mentioned both in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the separate "I "Limitations" section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification: This paper mainly focuses on empirical evaluation but also provides theoretical insights. We provide step-by-step proof and assumptions in Appendix A and carefully cite related works that can support our proof.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Our proposed method GRM and three types of regularization terms are detailed and illustrated in Section 3. For more in-depth experimental details to reproduce the main experimental results of the paper, please refer to Section 4 and Appendix B We also open-source our code on GitHub.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
26

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have ensured that all necessary information reproduce the experimental results of the paper is included, Links to the datasets and base models used are provided in the footnotes and Appendix B The code repository and open-sourced models available at https:/ggitub.ccom YanggRui201//GGeerllzzallll -
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
ceabbmmmssiooooicyyy......... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https www.ooooaeeeisoonaeee.
//nips.cmm httpc/ccpubccggguuuiee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The details of experimental setup are provided in Section 4 including dataset, base models, baselines, and model configurations. For more in-depth experimental details, please refer to Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: We evaluate our method and other baselines on large-scale open-source preference datasets with large models (2B and 7B). Repeating each experiment multiple times presents a challenge due time and computational constraints. However, we guarantee equitable comparisons, strikes balance between efficiency and practical limitations.
27

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For more details of computer resources, please refer Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ate:://neuiipppe wwwperpps.cpppulceeeeeee.eeeeeee ctc.p//uww.ct...cccciieeeee..ee.
Answer: [Yes]
Justification: mentioned.
We have reviewed the NeurIPS Code of Ethics and followed all the rules
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
28

---

Answer: [Yes]
Justification:
For more details of Broader Impacts, please refer to Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. dep.lymmlo of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release any new pretrained models or datasets for now. And we avoid the harmful examples show the paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Links to the datasets and base models used are provided the footnotes. We also cite the original papers of the datasets and benchmarks.
Guidelines:
The answer NA means that the paper does not existing assets.
29

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and open- open-source source reward models are available at GitHub and Hug- gingface.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
30

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "OPZGcTm9PO", "reviewer": "Reviewer_tSHJ", "summary": "The paper addresses the limitations of current reward models used in the reinforcement learning from human feedback (RLHF) framework, specifically their generalization capabilities to unseen prompts and responses. This limitation often leads to reward over-optimization, where the excessive optimization of rewards results in a decline in actual performance. The study proposes an approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states.", "questions": "Please refer to the Weaknesses section for questions regarding the paper.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation for the study is sound, and the experiments validate the effectiveness of the proposed method.", "weaknesses": "1. The proposed method lacks innovation as it combines DPO and SFT loss into the RM training phase as a regularization term. The inclusion of SFT loss in RM training has already been explored in previous works, such as InstructGPT and Anthropic's RM training.\n\n2. In the introduction, the authors mention that a randomly initialized head can distort pre-trained features, negatively impacting out-of-distribution (OOD) performance. Inspired by this finding, they propose to regularize feature distortion during fine-tuning for preference learning. However, there is no experimental evidence provided to support that this motivation holds true for RM. The improvements from adding regularization alone do not sufficiently prove that this motivation is solid.\n\n3. The results for label smoothing are missing in Figures 2 and 3, which should be addressed.\n\n4. The experimental section should include the alignment results after RL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments, and we provide clarification to your concerns as follows. \n\n\n**Q1**: The proposed method lacks innovation ... The inclusion of SFT loss in RM training has already been explored in previous works, such as InstructGPT and Anthropic's RM training.\n\n**A1:** We would like to clarify that **no previous work has incorporated text-generation loss as a regularization term for reward modeling**. InstructGPT and Anthropic's RM training optimize the vanilla reward loss (Eq 1 in [1]) based on the Bradley-Terry model, and their SFT regularization is typically applied to **language model training rather than reward modeling**.\n\nOur method is novel in that it integrates text-generation loss directly into the reward modeling process, a fundamentally different task from text generation. The enhancement effect of text-generation loss on reward modeling has not been explored in prior work. Additionally, our reward model structure is distinct to enforce this regularization. We maintain both a language head and a reward head within the reward model and apply text-generation loss to the language head during reward training.\n\n\nIn our experiments, compared to previous methods that use ensemble techniques [2][3] to enhance reward model capabilities, our method significantly reduces computational costs while achieving superior performance. This demonstrates promising results for more reliable and cost-effective reward modeling.\n\n\n\n**Q2**: No experimental evidence supporting the claim that a randomly initialized head can distort pre-trained features in RM, consequently leading to a negative impact on OOD performance.\n\n\n**A2:** The phenomenon is well-documented by [4], both theoretically and empirically (across a range of computer vision tasks). It is also easy to validate in the preference learning setting when using a smaller dataset size. We included a baseline, \"Classifier (Frozen)\", which fixes the base model’s features and only fine-tunes the classification head. When the dataset size is 8K (see the table below), the OOD evaluation results of the baseline reward model (without freezing the backbone) are worse than those of the frozen one, demonstrating the negative effect of distorting pre-trained features. However, we would like to note that when the dataset size is sufficiently large, this negative effect can be mitigated, and the baseline reward model can surpass the frozen reward model due to having more trainable parameters to fit the large dataset.\n\n\nIn contrast, by regularizing the hidden states, our GRM can achieve the regularizing effect while fine-tuning all parameters, showing strong performance with both large and small dataset sizes.\n\n\n| Reward Model   (8K data)           | Unified Feedback (IID) | HHH Alignment (OOD) | MT Bench (OOD) |\n|----------------------------|------------------|---------------|----------|\n| Classifier (Frozen)        |    62.2        |   68.8       | 67.6   |\n| Classifier (Baseline)      |  66.1   |  65.1    |  67.7 |\n| GRM (ours)|   **69.0**     | **71.9**           |       **69.8**    | \n\n\n\n**Q3**: The results for label smoothing are missing in Figures 2 and 3, which should be addressed.\n\n\n**A3:** We found that reward models with label smoothing are easily hacked by BoN and PPO, resulting in worse performance compared to other baselines. Therefore, we omitted them from the policy optimization experiments. To address the reviewer's concern, we have included the label smoothing results for the BoN and PPO experiments using the 7B base model in **Figure 1 of the supplementary PDF** provided during the rebuttal.\n\n\n**Q4**: The experimental section should include the alignment results after RL.\n\n**A4:** In Section 5.2, we evaluated different reward models for both RL training and Best-of-N sampling, using the gold score as the measurement for alignment, as adopted by prior studies [2][5] for reward modeling. Additionally, to demonstrate the advantage of GRM over vanilla reward modeling, we evaluated **the win rate of models after PPO training with GRM against those with the vanilla reward model**. The evaluation was conducted using GPT-4o on 100 randomly selected prompts from the test set in UnifiedFeedback, with the order of responses randomly flipped to avoid order bias. The results below show a significantly higher win rate for GRM than the vanilla reward model across two different base reward models.\n\n\n| Base reward model       | Win rate  | Tie rate | Loss rate |\n|----------------------------|------------------|---------------|----------|\n|   Gemma 2B it|     0.68    |    0.05   |  0.27  |\n| Mistral 7B Instruct         | 0.73   |  0.06      | 0.21 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LNRq8enDYR", "reviewer": "Reviewer_NAqK", "summary": "This paper proposes generalizable reward model (GRM) which modifies the standard reward-learning objective by adding an auxiliary task with a separate language modeling head. The auxiliary loss is either DPO or SFT. Experiments and ablations are conducted using mistral and gemma models and data from unified-feedback and evaluating on HHH, MT-bench, and rewardbench showing GRM improves OOD performance. Finally, GRM reward models are used to train new policies.", "questions": "None", "limitations": "There could be more discussion of limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of adding an auxiliary loss is simple and elegant. The paper provides a good intuitive argument for why it may be useful and clearly presents the potential ways to implement the auxiliary loss (DPO and SFT).\n\n2. The experiments and ablations are thorough and well-motivated. The paper considers a variety of baseline techniques to make reward models more robust like adding a margin term, label smoothing, or an ensemble. The paper also does a good job of comparing the low-data and high-data regimes.\n\n3. The results seem to show a consistent, if sometimes modest, improvement from GRM over baseline methods. And the improvements are larger in the low-data regime.", "weaknesses": "1. While the paper is generally comprehensive, there could be additional exploration of just using language modeling as an auxiliary task. It is not clear if the benefit of GRM is coming from language modeling the data from the preference dataset or if it would benefit from language modeling on any data (or even data from an even broader distribution, controlling for more OOD data). A further exploration of this would be good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments, and we provide clarification to your concerns as follows. \n\n**Q1:** While the paper is generally comprehensive, there could be additional exploration of just using language modeling as an auxiliary task. It is not clear if the benefit of GRM is coming from language modeling the data from the preference dataset or if it would benefit from language modeling on any data (or even data from an even broader distribution, controlling for more OOD data). A further exploration of this would be good.\n\n\n**A1:** Thanks for the insightful question. We believe that other data formats can also be useful, but preference data is more advantageous because it allows us to avoid using external datasets during reward modeling by leveraging the preference dataset, which may better match the distribution of prompts and responses. To illustrate this, we provide an experiment below using GRM with text-generation regularization on an open-source pretraining dataset, togethercomputer/RedPajama-Data-1T-Sample (including text from Commoncrawl, Arxiv, and books), referred to as 'GRM pretrain reg'. The results in **global response B** suggest that 'GRM pretrain reg' outperforms the baseline reward model and matches the performance of GRM when the dataset size is large (400K). However, when the dataset size is small, using a pretraining dataset is less effective than using the preference dataset.\n\n\n**Q2:** There could be more discussion of limitations\n\n\n**A2:** Thank you for the question. In addition to the limitations mentioned in Appendix D, another constraint is that we did not test reward models with parameter sizes exceeding 10B due to computational limitations. In the main paper, we tested models with sizes of 2B and 7B. During the rebuttal period, we also provided results for 8B reward models in **global response A**, which demonstrate promising potential for scaling GRM to larger reward models and datasets. Further efforts to extend our method to even larger reward models could be highly promising.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jXmZCZ1XaK", "reviewer": "Reviewer_SDk2", "summary": "This paper introduces a method that retains the base model's language model head while incorporating text-generation losses to preserve the hidden states' text generation capabilities.", "questions": "* The training details mention that the reward models are trained for only 2 epochs. This choice may not be optimal, considering the risk of overfitting in the full fine-tuning setting. It would be helpful to discuss the reasoning behind this decision or the impact of learning rate and convergence speed.\n* The simplicity of the regularization method is intriguing. Can it effectively mitigate reward hacking, potentially surpassing reward model ensemble techniques? It would be valuable to explore and discuss this aspect.", "limitations": "* The experiments may lack solidity due to the heavy reliance on the reward model in the PPO algorithm. If the reward model is not properly trained, the results can be significantly impacted, introducing randomness to the work.\n* The inclusion of manually synthetic data in the dataset may not accurately reflect real-world data, limiting the generalizability of the findings.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written, and the idea is straightforward.\n* The code is easy to understand and implement.\n* This method appears to be efficient and readily integrable with existing alignment approaches.", "weaknesses": "* Figure 3 (b) appears unusual, as the gold score decreases at the beginning of training. This could indicate suboptimal hyperparameter tuning or potential drawbacks in the pipeline.\n* The results demonstrate limited advantages. It is recommended to validate the method's benefits with a larger dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments, and we provide clarification to your concerns as follows. \n\n**Q1:** Figure 3 (b) appears unusual, as the gold score decreases at the beginning of training. This could indicate suboptimal hyperparameter tuning or potential drawbacks in the pipeline.\n\n**A1:** We found that this issue is influenced by **the learning rate of PPO**. By using a smaller learning rate (1e-5), we observed an initial improvement in the gold score during the first few steps, followed by subsequent drops (see **Figure 2 of the supplementary PDF**). Although the trend remains the same, the smaller learning rate delays the drop. In contrast, even with a larger learning rate, using our GRM allows PPO to learn stably, further emphasizing the importance of our proposed method for reward modeling and alleviating the burden of tuning hyperparameters for PPO.\n\nMoreover, we carefully followed the pipeline commonly adopted by the community [2][5] and used the open-source implementation of PPO from the Huggingface/trl package to ensure a correct and reliable pipeline.\n\n\n\n**Q2:** The results demonstrate limited advantages. It is recommended to validate the method's benefits with a larger dataset.\n\n\n**A2:** We would like to argue that the advantages of GRM are significant. As illustrated in Table 1 and 2 of our paper, increasing the training data from 40K to 400K enhances the OOD scores from 70.3/69.1 to 73.4/71.2. In contrast, GRM with just 40K training data achieves OOD scores of 78.7/73.0, showing a **much greater improvement than a 9x increase in the dataset size**. This is particularly important in real-world applications where there are limited samples for fine-tuning.\n\nAdditionally, in **global response A**, we present further results on RewardBench, demonstrating the strong potential of scaling GRM to **larger models and datasets**. Our findings indicate that GRM even outperforms a 34B reward model and GPT-4 as a judge.\n\nMoreover, as shown in Table 1 and 2, GRM outperforms a strong ensemble baseline (n=3) while **using only about 1/3 of the computational cost**. In Section 5.2, GRM significantly outperforms other reward models for RLHF in terms of the gold score, alleviating the reward over-optimization problem of RLHF.\n\n\n\n\n\n\n**Q3:** The training details mention that the reward models are trained for only 2 epochs ... It would be helpful to discuss the reasoning behind this decision or the impact of learning rate and convergence speed.\n\n\n**A3:** In our experience, the training design depends on the dataset's quality, diversity, and size, as well as the base model and optimizer hyperparameters used for training. There is no one-size-fits-all hyperparameter setting. Prior work [2] trains reward models for 5 epochs and suggests **determining this number based on a nearly converging validation loss**.\n\nFollowing this insight, we reserve 1% of the training set for validation (e.g., 4K for 400K training data) and found that 2 epochs are sufficient for reward modeling with LoRA in our setting. As shown in the table below, we observed convergence in the validation loss during the second epoch, with no further improvement in the third epoch. For full-parameter training experiments added during rebuttal, which are more prone to overfitting, we train the reward model for only one epoch.\n\n\n| Validation accuracy (%) of reward model           | 0.05 epoch | 0.5 epoch | 1 epoch |  1.5 epoch   |   2 epoch      |\n|----------------------------|------------------|---------------|----------|------| -----|\n| Classifier (Baseline)      |  68.4 | 73.3 | 73.8 | 74.0 | 74.0 |\n| GRM |     69.4 | 73.5 | 73.8 |74.2 | 74.1| \n\n\n\n**Q4** The simplicity of the regularization method is intriguing. Can it effectively mitigate reward hacking, potentially surpassing reward model ensemble techniques?\n\n\n**A4:** Yes, in **Section 5.2**, we already explored the effectiveness of GRM for RLHF, particularly focusing on two commonly used policy optimization methods: BoN and PPO. As shown in Figure 3 of our main paper and 3 figures in the supplementary PDF, GRM consistently excels in both the PPO and noisy label experiments. Unlike the baselines (including the ensemble techniques), which show an increasing proxy score but a declining gold score, GRM demonstrates its effectiveness in mitigating reward hacking.\n\n\n**Q5:** The experiments may lack solidity due to the heavy reliance on the reward model in the PPO algorithm. If the reward model is not properly trained, the results can be significantly impacted, introducing randomness to the work.\n\n\n**A5:** We would like to emphasize that our main focus is on improving reward modeling (Section 5.1), subsequently enhancing RLHF (Section 5.2). **Our reward model is not limited to the PPO algorithm**; it can be applied to any policy optimization method based on explicit reward modeling, such as BoN or reject sampling. As the reviewer mentioned, an improperly trained reward model can be hacked by RL algorithms, as demonstrated in Section 5.2. In contrast, GRM shows better robustness against reward hacking, underscoring its importance for robust preference learning.\n\n\n**Q6:** The inclusion of manually synthetic data in the dataset may not accurately reflect real-world data, limiting the generalizability of the findings.\n\n**A6:** In Section 5.2, synthetic data is used in the RLHF experiment to allow the gold reward model to annotate preference labels without human labor. This approach is also adopted by prior works [2][3][5], and even research from companies like OpenAI and Google uses this synthetic method to reduce human labor. We have also mentioned this in the limitations section (Appendix D). In other experiments in **Section 5.1**, we use the UnifiedFeedback dataset, which includes **datasets based on human annotators**, such as Anthropic/hh-rlhf, lmsys/chatbot_arena_conversations, openai/summarize_from_feedback. Therefore, we believe our setting also reflects real-world data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "11xUzh9s39", "reviewer": "Reviewer_VYhZ", "summary": "This paper is concerned with the overoptimization issue in reward modeling: When optimizing a policy against a reward model, this leads to a distributional shift that can lead to an increase in the proxy score while the true score decreases. This paper addresses these issues by regularizing the hidden states in reward models utilizing an additional output head that (with some variations) predicts the text of the winning trajectories in the comparison data. The idea is that this prediction task (a) helps keep the intermediate, strong representations from pretraining, and (b) adapts them to text that is more similar to what's encountered in post-training, thus leading to stronger generalization to the data encountered when optimizing the policy.", "questions": "I'm using this section for questions, suggestions, and minor weaknesses.\n\na. \"*In the RLHF stage, various policy optimization methods can be applied, with two frequently used methods being Best-of-n Sampling (BoN) and Proximal Policy Optimization (PPO).*\" --- This paper seems to view RLHF as the stage *after* reward modeling, so just policy optimization. In contrast, I think it's more typical to view reward modeling as *part of* RLHF.\n\nb. \"*While straightforward, the DPO regularization requires a reference model during training.*\" --- Don't you also need a reference policy during the PPO stage, given by the KL regularization with the SFT model? So it seems like this is no additional cost here?\n\nc. Equation (7): Did you also consider replacing $\\log \\circ \\sigma \\circ \\beta \\circ \\log$ by just $\\log$? Do you expect this to perform better/worse?\n\nd. Did you consider to interleave reward modeling with pretraining on a pretraining dataset to regularize the hidden states? I wonder if this performs better or worse than essentially doing pretraining on the preference dataset (as you do in your SFT regularization method).\n\ne. Did you check whether in SFT regularization it matters whether you predict the text of the winning responses, instead of trying to predict the losing responses? I could imagine that the former performs better since it brings the reward model closer to understand a distribution that the PPO stage *will steer towards*, but I'm curious if that prediction holds up empirically.\n\nf. For HHH-Alignment, MT-Bench, and RewardBench, could you give more details on what the benchmarks test for and what the scores mean?\n\ng. I'd recommend putting the limitations and broader impact sections into the main paper.", "limitations": "Limitations are described in Appendix D.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "My impression is that this is a strong paper dealing with the important overoptimization issue in RLHF. The motivation is strong and the experimental evaluation considers many baselines, settings, and datasets to be checked.", "weaknesses": "No strong weaknesses come to mind, though see the questions and comments below for other experiments that could be interesting to gain more confidence (though imho not necessary for this submission to be accepted).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments, and we provide clarification to your concerns as follows. We appreciate it if you have any further questions or comments.\n\n**Q1:** This paper seems to view RLHF as the stage after reward modeling ... I think it's more typical to view reward modeling as part of RLHF.\n\n**A1:** Thanks for the question. We agree with the reviewer that reward modeling is one step of RLHF. We will revise Section 2 Background to reflect this more accurately.\n\n\n**Q2:** \"The DPO regularization requires a reference model during training.\" --- Don't you also need a reference policy during the PPO stage ... So it seems like this is no additional cost here?\n\n**A2:** The cost includes both **memory usage and computation time**. Although DPO regularization does not increase memory usage for PPO, it still extends the computation time required for reward modeling. Furthermore, for other policy optimization methods such as BoN and reject sampling, the introduction of a reference model results in additional memory usage. Therefore, we claim that DPO regularization is more costly for reward modeling and prefer SFT and DPO w/o reference regularization according to the experimental results.\n\n\n**Q3:** Equation (7): Did you also consider replacing $log \\sigma (\\beta log)$  by just $log$? Do you expect this to perform better/worse?\n\n**A3:** In ideal situations, the two forms should perform similarly. We also tried the $log$ form but found that it requires different hyperparameter tuning for $\\alpha$ due to changes in the loss scale. In the tables below, **\"GRM logreg\" outperforms the baseline reward model and matches or even slightly exceeds the performance of our GRM on OOD tasks when $\\alpha$ is appropriate**.\n\nWe found that the current form of SFT regularization can directly use the same hyperparameters as our DPO regularization. Therefore, we opted for this solution to maintain coherence with these regularizations and avoid the need for hyperparameter adjustments.\n\n\n| Reward Model (400K data)             | Unified Feedback (IID) | HHH Alignment (OOD) | MT Bench (OOD) |\n|----------------------------|------------------|---------------|----------|\n| Classifier (Baseline)      |  72.1 | 73.4 | 71.2 |\n| GRM |     73.2 | 79.8 | 73.4 |\n| GRM logreg $\\alpha=0.005$ |    72.8  | 77.6 | 72.8 |\n| GRM logreg $\\alpha=0.001$ |    **73.3**  | **80.2** | **73.6** |\n\n\n\n| Reward Model (40K data)             | Unified Feedback (IID) | HHH Alignment (OOD) | MT Bench (OOD) |\n|----------------------------|------------------|---------------|----------|\n| Classifier (Baseline)      |  68.8 | 70.3 |  69.1 |\n| GRM |     **71.5** | 78.7 | **73.0** |\n| GRM logreg $\\alpha=0.005$ |  69.7   | 72.4 | 72.8|\n| GRM logreg $\\alpha=0.001$ |   70.8   | **80.7**  | 72.0 |\n\n\n\n**Q4:** Did you consider to interleave reward modeling with pretraining on a pretraining dataset to regularize the hidden states? \n\n**A4:** Thanks for the insightful quesiton. One challenge of the described approach is that we don't have the pretraining dataset for many language models. We think that other data formats can also be useful as a regularization, but preference data is more advantageous because it allows us to avoid using external datasets during reward modeling by leveraging the preference dataset, which may better match the distribution of prompts and responses. Please refer to the **global response B** for the additional results.\n\n\n**Q5:** Did you check whether in SFT regularization it matters whether you predict the text of the winning responses, instead of trying to predict the losing responses? \n\n**A5:** In our SFT regularization, we only use the winning responses to regularize the hidden states. During the rebuttal, we also tested regularizing with the negative responses, referred to as \"**GRM neg**.\" The results show **very close IID evaluation accuracy but relatively lower OOD accuracy** compared to our default setting, potentially due to the generally lower quality of rejected responses. The impact of this reward model on PPO training requires further examination. However, we agree with the reviewer's conjecture that the rejected responses may lead to unsatisfactory representations for the desired distribution.\n\n\n| Reward Model (400K data)             | Unified Feedback (IID) | HHH Alignment (OOD) | MT Bench (OOD) |\n|----------------------------|------------------|---------------|----------|\n| Classifier (Baseline)      |  72.1 | 73.4 | 71.2 |\n| GRM |     **73.2** | **79.8** | **73.4** |\n| GRM neg | **73.2**   |     78.1    |    73.0     | \n\n\n\n| Reward Model (40K data)             | Unified Feedback (IID) | HHH Alignment (OOD) | MT Bench (OOD) |\n|----------------------------|------------------|---------------|----------|\n| Classifier (Baseline)      |  68.8 | 70.3 |  69.1 |\n| GRM |     **71.5** | **78.7** | **73.0** |\n| GRM neg |  71.2   | 78.1 | 72.8|\n\n\n\n**Q6:** For HHH-Alignment, MT-Bench, and RewardBench, could you give more details on what the benchmarks test for and what the scores mean?\n\n\n**A6:** In Section 4, we briefly introduce three benchmarks. The HHH-Alignment dataset evaluates language models on helpfulness, honesty, and harmlessness. The MT-Bench dataset contains 3.3K human preferences for model responses generated by LLMs in response to MT-Bench questions. RewardBench is a new benchmark designed to evaluate the reward model's ability to select human-preferred responses for chat, reasoning, and safety tasks. In summary, these benchmarks are used to assess the reward model's alignment with human preferences and generalization ability across different prompt-response distributions. The scores represent the average accuracy across each benchmark, with higher accuracy indicating better performance.\n\n\n**Q7:** I'd recommend putting the limitations and broader impact sections into the main paper.\n\n**A7:** We agree with the reviewer and will move the two sections to the end of the main paper in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive comments and are particularly grateful for their recognition of our work: 'sound and strong motivation' (Reviewers tSHJ, VYhZ), 'elegant idea' (Reviewer NAqK), 'thorough experiments' (Reviewers NAqK, VYhZ), and 'well-written' (Reviewer SDk2). We hope our responses have addressed all of the reviewers' concerns. If you have any additional questions, please feel free to post them, and we will be happy to discuss further.

Below, we provide new experimental results and global references.

### A. **8B full parameter finetuning on larger dataset**

To further demonstrate the effectiveness of GRM, we present additional results on RewardBench, a new benchmark for human preference. We trained GRM using the llama3-8b-instruct model, performing full parameter fine-tuning for 1 epoch on one of the largest open-source preference datasets, hendrydong/preference_700K. Our results indicate the **strong potential of scaling GRM to larger models and datasets**, even outperforming a 34B reward model and GPT-4 as a judge.


|       Reward model           | Average       |  Chat     |     Chat Hard      |     Safety      |     Reasoning     |   
|:-------------------------:|:-------------:|:---------:|:---------:|:--------:|:-----------:|
|  GRM (Ours, 8B) | 87.0     |   98.6  |  67.8 |   89.4 |92.3     |  
|    openai/gpt-4-0125-preview                             |    85.9     |   95.3      |  74.3  |  87.2 |    86.9    |  
|openai/gpt-4-turbo-2024-04-09 | 85.1 |  95.3   | 75.4   |87.1 | 82.7|
|   Classifier   (baseline, 8B)                          |   	84.7     |   99.4     |   65.1   | 87.8  |    86.4  | 
|Nexusflow/Starling-RM-34B | 82.7 | 96.9 |57.2|88.2 |88.5|


### B. **Regularization with pretraining dataset**

In our default design, we use the preference dataset employed to train reward models to regularize the text-generation ability of the language head, eliminating the need for additional datasets. While we believe that other data formats, such as pretraining datasets, can also be beneficial, preference data offers a distinct advantage. It allows us to avoid using external datasets during reward modeling, which may also better align with the distribution of prompts and responses.

To illustrate this, we conducted an experiment using GRM with text-generation regularization on an open-source pretraining dataset, togethercomputer/RedPajama-Data-1T-Sample (which includes text from Commoncrawl, Arxiv, and books), referred to as **'GRM pretrain reg'**. For fairness, we only used a pretraining dataset of the same size as the training set for reward modeling. 

The results indicate that 'GRM pretrain reg' outperforms the baseline reward model and matches the performance of GRM when the dataset size is large (400K). However, when the dataset size is small, using a pretraining dataset is less effective than using the preference dataset.

| Reward Model (400K data)             | Unified Feedback (IID) | HHH Alignment (OOD) | MT Bench (OOD) |
|----------------------------|------------------|---------------|----------|
| Classifier (Baseline)      |  72.1 | 73.4 | 71.2 |
| GRM |     **73.2** | **79.8** | 73.4 |
| GRM pretrain reg  |   73.0  | 79.2 | **74.3** |




| Reward Model (40K data)             | Unified Feedback (IID) | HHH Alignment (OOD) | MT Bench (OOD) |
|----------------------------|------------------|---------------|----------|
| Classifier (Baseline)      |  68.8 | 70.3 |  69.1 |
| GRM |     **71.5** | **78.7** | **73.0** |
| GRM pretrain reg |  70.8   | 74.5 | 72.9|


### C. **Suplementary PDF**

In our supplementary PDF for the rebuttal, we include three figures:

- **Figure 1**: We present the results of label smoothing to address reviewer tSHJ's concern.
- **Figure 2**: We provide the results of PPO with a learning rate of $1 \times 10^{-5}$ to address reviewer SDk2's concern about the hyperparameter of PPO.
- **Figure 3**: We include the results of PPO in a noisy label setting. These results consistently demonstrate the superiority of GRM compared to other reward model baselines.


### D. **References**

[1] Training language models to follow instructions with human feedback. NeurIPS, 2022.

[2] Reward Model Ensembles Help Mitigate Overoptimization. ICLR 2024.

[3] Warm: On the benefits of weight averaged reward models. ICML, 2024.

[4] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022.

[5] Scaling laws for reward model overoptimization. ICML, 2023.

Meta-review: The paper studies the issue of reward generalization. It re-introduces the language model loss to reward model training as a regularization term to improve reward generalization. Experiment results show that this method could improve the reward model performance on multiple benchmarks. Reviews generally agree that the motivation for the study is sound, and the experiments validate the effectiveness of the proposed method.
Decision: Accept (poster)