Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs

Authors: rui  yang
ruomeng  ding
yong  lin
huan  zhang
tong  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:25:35

Abstract: Reward models trained on human preference data have been proven to effectively align Large Language Models (LLMs) with human intent within the framework of reinforcement learning from human feedback (RLHF). However, current reward models have limited generalization capabilities to unseen prompts and responses, which can lead to an unexpected phenomenon known as reward over-optimization, resulting in a decline in actual performance due to excessive optimization of rewards. While previous research has advocated for constraining policy optimization, our study introduces a novel approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states. Specifically, we retain the base model's language model head and incorporate a suite of text-generation losses to preserve the hidden states' text-generation capabilities, while concurrently learning a reward head behind the same hidden states. Our experimental results demonstrate that the introduced regularization technique markedly improves the accuracy of learned reward models across a variety of out-of-distribution (OOD) tasks and effectively alleviates the over-optimization issue in RLHF, offering a more reliable and robust preference learning paradigm.

Main paper: 
 introduction: 
 Pretrained large models have showcased impressive capabilities across diverse fields [1, 2, 3, 4, 5] . A notable trend in recent research is ensuring that large models align with human values and mitigate potentially harmful behaviors [6, 7, 8, 9, 10] . Alignment methods are crucial in achieving this objective, with two primary approaches being supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) [7, 8] . SFT directly finetunes the model using prompt and response pairs, proving to be a straightforward and efficient alignment technique [11, 12, 13] . Differently, RLHF begins by learning a reward model from user preferences and then employs reinforcement learning to optimize the language model to maximize rewards. A significant advantage of RLHF is its potential to generalize the reward model to unseen prompt-response pairs, effectively leveraging large volumes of unlabeled data [8, 14] .Despite the empirical success of RLHF, the challenge of training a reliable and generalizable reward model for unseen data remains an open problem. A well-known failure mode of reward model is known as "overoptimization" or "reward hacking" [15, 16, 17, 18] , where policy optimization seemingly improves the proxy reward model but actually degrades the true reward function. [17] demonstrated in a synthetic setup that increasing the size of the reward model and the volume of training data can mitigate this overoptimization issue. However, such scaling is not always feasible in many realistic scenarios. To address this, a series of studies have been conducted, focusing either on enhancing the reward model with ensemble techniques [18, 19, 20, 21] or on constrained policy optimization [22, 23, 24, 25] . The latter paradigm is related to the offline RL literature [26, 27, 28, 29, 30, 31] , which involves limiting the policy distribution to be close to the training data distribution. Among these, improving the generalization ability of reward models presents a fundamental and promising direction that can be studied independently from enhancements in policy optimization. Nevertheless, previous methods [18, 32] requiring training multiple reward models may be resource-intensive for the practical application of large models.In this study, we present a lightweight yet effective solution designed to enhance the reward model's generalization ability against distribution shifts. Previous research [33] has theoretically shown that a randomly initialized head can distort pre-trained features, thereby negatively impacting outof-distribution (OOD) performance. Inspired by this finding, we propose to regularize the feature during fine-tuning for preference learning using an adversarial regularizer, which derives a suite of text-generation losses. To this end, we introduce Generalizable Reward Model (GRM), which retains the base model's language model head and regularizes the hidden states of the reward model by incorporating text-generation losses. This approach makes better use of the preference learning data while preserving the text generation capabilities of the hidden states. Notably, GRM does not necessitate training multiple reward models or relying on additional training data.In our experiments, GRM substantially improves the evaluation accuracy of the reward model OOD evaluation datasets, demonstrating its superior ability to generalize learned preferences to unseen prompt and response pairs. Moreover, GRM consistently improves the performance of both 2B and 7B reward models, with a more pronounced improvement observed when the data size is limited. We also demonstrate that GRM can markedly enhance the performance of best-of-n (BoN) sampling and PPO [34] , effectively mitigating the overoptimization problem. These results highlight the potential of the GRM to serve as a more reliable proxy reward model for human preferences.To conclude, our primary contributions are as follows:• We propose GRM, a novel approach that employs text-generation regularization on the hidden states to enhance the generalization ability of reward models. • Our study validates the effectiveness of all three types of text-generation regularization for GRM, identifying the SFT regularization as the most effective and stable solution. • Our empirical results show that GRM significantly improves the accuracy of reward models across various OOD tasks. Furthermore, it consistently enhances the performance of RLHF, effectively alleviating the overoptimization problem.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our study proposes a novel approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states. It was mentioned both in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background: 
 Typically, reinforcement Learning from Human Feedback (RLHF) involves reward modeling and policy optimization, with Best-of-n Sampling (BoN) and Proximal Policy Optimization (PPO) being two commonly used methods for policy optimization.Reward Modeling. Generally, reward modeling is based on the Bradley-Terry model [35] , which aims to distinguish between the chosen response y c and the rejected response y r given the prompt x:L reward (θ) = -E (x,yc,yr)∼D [log (σ (r θ (x, y c ) -r θ (x, y r )))] ,where r θ (x, y) represents the reward score for prompt x and output y with model parameters θ. σ(•) is the sigmoid function. By minimizing this loss function, the reward model assigns higher scores to outputs preferred by humans. Subsequently, the trained reward model can be used to guide the optimization of the language model.Best-of-n Sampling (BoN). BoN generates n samples from the policy model, denoted as Y gen , and then selects the best one based on scores provided by a reward model. BoN can be used for inference-time improvement or iterative optimization [36, 37, 38] .y BON (x) = arg max y∈Ygen r θ (x, y).(EQUATIONProximal Policy Optimization (PPO). PPO is a widely adopted method for RLHF in optimizing language models [16, 8, 39] . PPO learns a policy by maximizing a reward objective with a KL divergence penalty with coefficient η:EQUATIONwhere the KL penalty ensures that the optimized policy does not deviate significantly from the SFT policy to maintain the reliability of the reward model.Overoptimization. Although the learned proxy reward model aims to approximate human preference, it may not consistently reflect authentic human preferences, potentially resulting in over-optimization [17, 18] . This issue emerges when the proxy reward model becomes overly optimized, causing the policy model to overfit certain erroneous patterns. Ultimately, this issue can diminish the model's alignment with actual human preferences, highlighting the need to ensure the reward model's robustness and reliability.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations are discussed in the separate "Limitations" section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In the common practice of training a reward model [8, 39, 40] , reward models are initialized using a pretrained or SFT finetuned backbone, along with a randomly initialized reward head to predict the scores for prompt-response pairs. It's important to note that the backbone and original language model head are trained on a diverse range of datasets for text generation, which is distinct from the preference learning tasks. Under the task shift, the randomly initialized reward head can distort the pretrained features, thereby reducing the OOD generalization performance, as observed by [33] . We also confirm this impact on preference learning in Appendix C.1.To improve the reward model's generalization capability against distribution shifts, we propose a lightweight yet effective solution, Generalizable Reward Model (GRM). This model employs a suite of text-generation regularizations for the hidden states. More specifically, GRM employs a structure as illustrated in Fig 1 , with one language model (LM) head and one reward head sharing the same hidden states. The reward head is trained to minimize the reward loss L reward in Eq 1, while the LM head is trained to maintain the text-generation ability of the hidden states during preference learning. Consequently, we define the overall loss function as follows:EQUATIONHere, α is the coefficient that balances the reward loss and the regularization. We will derive potential forms of the regularization term below.To derive the potential formulation of the regularization term, we consider the following adversarial optimization problem: learning a reward model against an adversarial policy.EQUATIONwhere γ > 0 is a coefficient. This objective is also considered by recent studies [24, 25] aiming to enhance DPO. Differently, we adopt it to learn a generalizable reward model.The insight of Eq 5 is that we can enhance the robustness of the reward model by considering an adversarial policy π from a certain policy class. The term for policy optimization J(θ, π) can have various formulations, but a KL divergence-regularized objective is generally used in training the policy [16, 8] . Moreover, it has an advantageous property that the inner optimization problem has an analytical solution, which can simplify the problem.EQUATIONwhere β > 0 is a regularization coefficient and π ref is the reference model. We denote the analytical solution of J(θ, π) as π * θ . Incorporating π * θ into Eq 5, we can transform the min-max optimization problem into a standard optimization problem under certain assumptions:EQUATIONDetailed derivation is deferred to Appendix A. Here, L DPO is the same as the DPO objective [41] and L SFT is the SFT objective that maximizes the probability of chosen responses. Notably, the two regularization terms originate from different sources: L DPO stems from the reward loss, while L SFT is derived from the adversarial term. This may explain why SFT regularization proves more beneficial than DPO regularization in our empirical results. Motivated by Eq 7, we relax the relationship between r θ and π * θ and propose learning a reward model parameterized by θ and a language model head parameterized by θ LM , both sharing the same hidden states. A discussion of this design can be found in Appendix A. Below, we detail three practical implementations.Inspired by Eq 7, we train the LM head to minimize text-generation losses, such as DPO and SFT losses, as the regularization term for GRM. To independently study the effectiveness of these two regularizations and reduce GPU memory usage, we introduce three practical implementations: DPO regularization, DPO without reference regularization, and SFT regularization. DPO Regularization. By setting α DPO = α and α SFT = 0 in Eq 7, we can directly adopt the DPO loss as a regularization term for GRM to regularize the hidden states:EQUATIONwhere π ref is the base model serving as the reference model, and π θLM is our optimized policy. β is a coefficient that controls the KL penalty between π θLM and π ref .Notably, π θLM shares the same base model with the reward model r θ , except for the output layer.DPO Regularization w/o Reference Model. While straightforward, the use of a reference model in DPO regularization can be memory-intensive for large models. To address this, and inspired by prior works that eliminate the need for reference model [42, 43] , we introduce the DPO regularization without a reference model, denoted as L DPO-noref . This method reduces the need for large GPU memory during training. The loss function L DPO-noref is defined as:EQUATION)SFT Regularization. By setting α DPO = 0 and α SFT = α in Eq 7, we can simplify the regularization term to SFT regularization, thereby reducing the computational cost. This method only maximizes the probability of the chosen responses:EQUATIONThis equation differs slightly from the standard SFT objective to maintain coherence with the above two cases within the regularization suite and avoid the need for hyperparameter adjustments for α.Please refer to Appendix C.3 for a discussion.In summary, GRM offers three key advantages: (1) Mitigating feature distortion. The application of text-generation loss helps maintain the text-generation ability of the base model and prevents excessive feature distortion. Simultaneously, it also adapts the model to the data distribution of preference learning.(2) Prevention of Overfitting. The text-generation regularization derived from an adversarial training objective helps prevent the reward model from overfitting to certain spurious features, which can be detrimental to OOD generalization. This effect becomes more pronounced when the preference data includes erroneous comparison pairs or when the dataset size is limited.(3) Efficiency. GRM is an efficient solution that does not require training multiple reward models or additional training data. Additionally, different choices of loss type entail varying memory and computational costs. Interestingly, we find that the simplest option, SFT regularization, proves to be the most stable choice.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: We have reviewed the NeurIPS Code of Ethics and followed all the rules mentioned.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental setup: 
 Datasets. For training reward models, we leverage the Unified-Feedback dataset 2 , which stands as one of the largest collections of pairwise feedback datasets. In Section 5.1, we train all reward models on a subset of 400K and 40K samples from the Unified-Feedback dataset and evaluate them on the hold-out 8K eval set. In addition, for evaluating model performance on out-of-distribution (OOD) preference data, we utilize datasets such as HHH-Alignment3 [44] , MT-Bench Human Judgements4 [45] , and RewardBench [46] . The HHH-Alignment dataset evaluates language models on helpfulness, honesty, and harmlessness, while the MT-Bench dataset contains human preferences for model responses to MT-bench questions. Besides, RewardBench is a new benchmark designed to evaluate the capabilities and safety of reward models. We consider HHH-Alignment, MT-Bench, and RewardBench as OOD evaluation tasks because the prompt and response distributions differ from our training distribution. For the RLHF experiments in Section 5.2 we downsample 20K data from Unified-Feedback for training reward models and optimizing the PPO policy, and another 1K data for evaluating BoN or the learned PPO policy.Base Models. In the preference learning experiments, our base models include gemma-2B-it [47] and Mistral-7B-Instruct-v0.2 [48] . For the RLHF experiments, gemma-2B-it serves as the policy model for both BoN and PPO experiments, whereas the gold reward model5 is a 7B human preference model finetuned using the entire Unified-Feedback dataset.Baselines. We compare the performance of GRM with several baselines, including Baseline Classifier trained using the original reward loss in Eq 1; Frozen Classifier that fixes the base model's feature and only finetunes a nonlinear classification head; Margin that adds an additional margin in the original reward loss [10, 39] ; Label Smooth that mitigate the overfitting problem by penalizing overconfident model outputs [39] ; Ensemble method with a group of 3 reward models [18] to calculate the average or minimum values as rewards. In addition, for RewardBench, we present the performance of several existing open-source state-of-the-art reward models for better reference, including PairRM [49] , Starling-RM-7B/34B [50] , and UltraRM-13B [51] . For more experimental details and additional results, please refer to Appendix B and Appendix C, respectively. 
evaluation results: 
 We present a comprehensive evaluation of GRM, utilizing both in-distribution (ID) and out-ofdistribution (OOD) datasets, as well as existing benchmarks for reward models. Furthermore, we explore the impact of GRM on the overoptimization issue in RLHF. Our primary findings can be summarized as follows:• GRM significantly enhances the generalization capability of reward models, resulting in substantial improvements on both ID and various OOD evaluation sets (Section 5.1). • All three types of text-generation regularization losses can improve the generalization performance, with the SFT regularization being the most effective and stable (Section 5.1). • GRM demonstrates robustness in the limited dataset setting, outperforming baselines by an even larger margin (Section 5.1).• GRM effectively mitigates the overoptimization issue in both BoN and PPO (Section 5.2).• GRM also exhibits robustness against label noise in the preference dataset (Section 5.2). Regarding other baseline models, the Frozen classifier, which maintains its base model's parameters, exhibits the lowest ID and OOD scores. This suggests that the pretrained features of the base model alone are insufficient for effective preference learning, emphasizing the importance of fine-tuning the base model's features to the preference task. Furthermore, the margin loss and label smoothing techniques do not consistently improve the ID and OOD tasks, whereas the ensemble baseline consistently enhances both ID and OOD scores. Despite requiring the training of multiple reward models, ensemble-based methods still do not surpass GRM, particularly when learning from a 40K training set. These results highlight the substantial improvement and generalization capability of GRM in preference learning.Comparison of Different Regularizations. As observed in Table 1 , when the training dataset is sufficiently large, GRM with three types of regularizations (namely GRM w/ dpo, GRM w/ dpo-noref, and GRM w/ sft) perform comparably. This demonstrates that GRM is robust to the choice of regularization type when the dataset is large. However, in Table 2 , where the training data is limited to 40K, a clear trend emerges: GRM w/ sft outperforms GRM w/ dpo-noref, which in turn outperforms GRM w/ dpo, on both the ID and OOD scores. Interestingly, the simplest form of regularization, SFT regularization, not only requires the lowest training cost but also yields the most stable overall results. Consequently, we adopt it as the default choice for our subsequent study.Results on RewardBench. In Table 3 and Table 4 , we evaluate GRM and various baselines on RewardBench across chat, chat-hard, safety, and reasoning task groups. We consider a variant of GRM with a linear reward head instead of the default nonlinear reward head as detailed in Appendix B. In Table 3 , the 7B baseline matches the score of Starling-RM-7B [50] , while GRM (linear) w/ sft demonstrates a considerable improvement, increasing the average score from 76.3 to 79.5. Comparing variants of GRM, we can conclude that: (1) SFT regularization performs better than the DPO w/o reference model regularization, and (2) GRM with a linear head achieves a better overall score than that with a nonlinear head, especially in the challenging reasoning task group. Regarding the baselines, consistent with previous results, the margin loss and label smoothing do not provide a coherent improvement over the baseline. While ensemble methods effectively improve upon the baseline, they still underperform GRM. Overall, these results demonstrate that GRM is a strong contender in reward modeling tasks, exhibiting superior performance across various benchmarks.Comparison of Different Dataset Sizes. Another noteworthy observation is that GRM exhibits greater robustness to the size of the training dataset compared to baselines. For instance, in trend is consistent in Table 3 and Table 4 . Specifically, for the Mistral 7B Instruct base model, the baseline's average score drops from 76.3 to 68.2 when learning from 40K training data, while GRM (linear) w/ sft only drops from 79.5 to 78.3. These findings suggest that the prior reward training paradigms are sensitive to smaller dataset sizes. In contrast, GRM is robust even with a limited dataset.Full Parameter Training Results on a Larger Dataset. To further demonstrate the effectiveness of GRM, we trained the GRM using the llama3-8b-instruct model [52] . We perform a full parameter fine-tuning for 1 epoch on one of the largest open-source preference datasets 6 . Our results, presented in Table 5 , highlight the considerable potential of scaling GRM to larger models and datasets. Especially, GRM outperforms a 34B reward model and even GPT-4 as a judge. It is worth noting that the GRM significantly improves the performance of the 8B reward model from 84.7 to 87.0, using the same base model and training data as FsfairX-LLaMA3-RM-v0.1 [40] . This improvement is particularly remarkable in the challenging 'Reasoning' section. [17, 18] , we conduct BoN sampling on a 1K held-out test set from n responses for each prompt, based on the scores of the trained proxy model. The selected responses are then scored using the gold reward model, and their gold scores are averaged over the 1K test prompts. The average gold score reveals the true quality of the responses selected by the proxy reward models. We set the KL Divergence from 0 to 5, corresponding to the number of responses n ranging from 1 to 405 for each prompt, according to the equation KL BoN = log n -n-1 n . Ideally, a good proxy reward model should yield larger average proxy and gold scores as the KL increases. However, the average gold scores of some baseline methods plateau or even drop after KL > 4, such as the baseline reward model in Fig 2(d) , despite their proxy scores continuing to increase in Fig 2(c ). This suggests that these reward models suffer from the overoptimization issue. In contrast, GRM consistently demonstrates an increase in both the proxy score and gold score, indicating that it effectively mitigates over-optimization. This advantage is more pronounced in the 7B base model, where GRM achieves an average gold score of 1.5, while the baseline reward model only reaches a score of 0.5. Regarding other baselines, we find that the margin loss and ensemble methods (especially the 'min' strategy) contribute to the robustness of the reward model. However, they still do not compare favorably with GRM. These results underscore the strong potential of GRM to serve as a reliable and robust proxy reward model for RLHF.Proximal Policy Optimization (PPO). Robustness to Label Noise. Human preference data typically contains around 20 to 30% noise, as highlighted in previous studies [39] . Such inconsistent preference data can render the reward model less generalizable [32, 53] and hinder policy learning [54, 55, 56] , leading to a performance decline.To evaluate the robustness of GRM against label noise, we incorporate a 25% label noise into the 20K training data for all proxy reward models. The results are depicted in indicating that those reward models are heavily overfitting under the noisy label setting. On the contrary, GRM exhibits superior robustness under noisy conditions, consistently achieving a peak gold score over 1.0 without a significant decline. This demonstrates that GRM is highly accurate and robust at measuring the sample quality, even in the presence of noise within the training data. 
related works: 
 Reward Modeling. Reward models, trained on human preference data, are crucial in guiding RLHF [8, 57] or prompt optimization [58] . Recent studies have concentrated on developing advanced reward models to improve the performance of LLMs in RLHF. One approach involves enhancing reward modeling by improving the quality or quantity of preference data [59, 60, 61] . Other strategies focus on learning token-wise dense rewards [62, 63] or multi-objective rewards [38] . Additionally, a series of works aim to enhance the robustness of reward models against preference inconsistencies. Techniques such as adaptive margin [10] , contrastive learning [39] , and meta-learning [64] are employed to improve the model's ability to differentiate between chosen and rejected responses.Mitigating Overoptimization in RLHF. Reward models tend to overfit and struggle to generalize beyond the training distribution, which often leads to the overoptimization issue [17] . One approach to mitigate this is to penalize overly confident model outputs using label smoothing [39] or SFT regularization [24, 25] . Alternatively, the model and data can be iteratively updated, replacing hard labels with soft labels [65] . Ensemble techniques, which train several reward models, can also help reduce reward hacking and manage shifts in distribution [18, 19, 20, 66, 32, 67, 68] . Adversarial Preference Optimization employs adversarial learning between reward models and an LLM agent to address the gap in generation distribution [69] . Recent studies have also utilized uncertainty to mitigate reward over-optimization, including the integration of an uncertainty penalty into rewards [70] , or the construction of a confidence interval for gold rewards based on uncertainty estimations [23] . 
conclusion: 
 In this study, we introduce an efficient approach aimed at enhancing the generalizability and robustness of reward learning for large language models. By incorporating regularization techniques on the hidden states of reward models, our method demonstrates substantial improvements in the generalization performance of reward models for unseen data. Moreover, our approach effectively mitigates the issue of overoptimization in RLHF. We believe that our findings hold promise in inspiring future research efforts towards the development of more robust reward models that can facilitate the alignment of large models through cost-effective solutions. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: For more details of Broader Impacts, please refer to Appendix E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not release any new pretrained models or datasets for now. And we avoid the harmful examples to show in the paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Links to the datasets and base models used are provided in the footnotes. We also cite the original papers of the datasets and benchmarks.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this study, we evaluate the robustness of GRM against label noise by introducing a 25% level of synthetic noise into the training data for all proxy reward models. This is achieved by randomly flipping chosen and rejected labels. Due to cost considerations, we conduct synthetic experiments in line with community practices [18, 39] , as using human-labeled data is not feasible for us. However, synthetic data may introduce biases that don't accurately reflect real-world scenarios. Future research should aim to mitigate this limitation by incorporating experiments with human-labeled data, providing a more thorough evaluation of the reward model's robustness. Another limitation of our study is the computational restriction preventing us from testing GRM with parameter sizes exceeding 10B. Further efforts to extend our method to larger reward models could be highly promising.To derive the potential formulation of the regularization term, we consider the following adversarial optimization problem: learning a reward model against an adversarial policy.EQUATIONThe term for policy optimization J(θ, π) can have different formulations, but a KL divergence regularized optimization objective is generally used in training the policy [16, 8, 71] . Moreover, it has an advantageous property that the inner optimization problem has an analytical solution, which can simplify the problem.EQUATIONwhere β > 0 is the coefficient controlling the regularization degree and π ref is the reference model. The analytical solution of J(θ, π) is formulated as follows:EQUATIONEquivalently, we can obtain the formulation of reward described by π * θ and π ref as in [41] :EQUATIONFollowing recent theoretical analysis [25] , we define a fixed calibration policy π cal that is independent of the algorithm, which has the calibration effect of centering the reward function while incorporating additional policy preferences into the objective.Definition 1 π cal is a fixed calibration policy for reward model r θ and the dataset D that satisfies:E x∼D,y∼π cal [r θ (x, y)] = 0Therefore, we can rewrite max π J(θ, π) as:EQUATIONThe second line is established because log Z θ (x) is independent of the distribution y. Besides, the last line just adopts the definition of π cal .Incorporating Eq 15 and Eq 14 into Eq 11, we can transform the min-max optimization problem into a standard optimization problem by considering the policy π * θ :EQUATION) Here, we use L DPO (π * θ ) to replace the second term, as it is the same as DPO objective [41] . In the second line, we put the reward described by Eq 14 into αL reward . In the final step, we remove the π ref term as it does not depend on the parameters of the reward r θ , unlike π * θ which is dependent on reward r θ .Interestingly, if we set the calibration policy π cal as the chosen responses y c from the dataset D, the last term becomes an SFT loss. Thus, we can derive the general regularization terms in our framework by renaming the coefficients for π * θ as α DPO and α SFT , and removing the constraint that α DPO = α.EQUATIONNotably, the two regularization terms come from different sources, where L DPO is from the reward loss and L SFT is derived from the adversarial term. This may be the reason why SFT regularization is more helpful than DPO regularization in our empirical results. Inspired by the objective in Eq 7, we relax the relationship between r θ and π * θ and propose to learn a reward model parameterized by θ and a language model head parameterized by θ LM , both sharing the same hidden states.Discussion. In Eq 17, we retain both the reward model r θ and the policy π * θ , and replace π * θ with a language head π θLM . A simpler solution is to keep only the reward model by replacing π * θ with r θ , which leads to the following objective:arg min θ {L reward (θ) -γE x,yc∼D [r θ (x, y c )] + γβE x∼D [log Z θ (x)]}.This approach can be understood as minimizing reward loss while applying regularization to maximize the rewards of selected responses relative to the overall rewards. However, this method is limited by the inefficient calculation of Z θ over the response distribution generated by π SFT . Therefore, we propose our solution, GRM, which involves a reward model that shares hidden states with a language head. This setup captures certain correlations through shared parameters, helps prevent feature distortion, and is both cost-effective and highly efficient.Baseline Details. All baseline reward models employ the "AutoModelForSequenceClassification" class from transformers [72] , which utilizes a randomly initialized linear head to derive rewards. We then train each reward model to minimize the loss function with the training data. For ensemble baselines, we train 3 reward models with different random seeds and aggregate their outputs via the 'average' or the 'minimum' strategy. We adopt the average value for the ensemble baseline in Section 5.1 as we find that the minimum value can decrease accuracy and underperform the average one. But for the RLHF experiments in Section 5.2, we report both results because we find some sometimes the 'minimum' strategy can work better than due to its pessimism.The margin loss function [10] is defined as below:L margin (θ) = -E (x,yc,yr)∼D [log (σ (r θ (x, y c ) -r θ (x, y r ) -m(r)))] ,which enhances the reward model by emphasizing the differences in rewards. We use the scores between chosen and rejected responses in the Unified-Feedback dataset to calculate m(r).Additionally, the label smooth loss is defined asL smooth (θ) = -E (x,yc,yr)∼D [(1 -ϵ) log (σ (r θ (x, y c ) -r θ (x, y r ))) -ϵ log (σ (r θ (x, y c ) -r θ (x, y r )))] ,where we set ϵ = 0.1. The label smooth loss function enhances the model's resilience to a certain degree of errors, thereby alleviating the problem of overfitting.GRM Details. For GRM, the default reward head is configured as a linear layer with shape (hidden size, 1024), followed by a ReLU activation function, and another linear layer of shape (1024, 1). The weight of the text-generation regularization α is set to 0.01 and the coefficient β in our regularizations is set to 0.1 by default. In the case of the GRM (linear) variant, the reward head is directly set as a linear layer of shape (hidden size, 1). We found a smaller α = 0.001 is better for the linear variant.Training and Evaluation Details. We implement all methods based on transformers [72] and trl [73] . More details are listed in Table 6 . To use the Unified-Feedback dataset, we downsample the training data from the 'all' set and use all the 8K test data for evaluation. For the HHH Alignment dataset, we adopt the average score of all four subsets as the result. For the main experiments trained with LoRA, we truncate the inputs for all reward models over 1024 tokens. All reward models are trained for two epochs using a learning rate of 1 × 10 -5 and a batch size of 16. We load the model with the bf16 precision. Regarding the full parameter training, we truncate the inputs over 4096 tokens and train the reward model for one epoch with a learning rate of 2 × 10 -6 and a batch size of 512 (with gradient accumulation). The effect of the random head for downstream finetuning of pretrained model is studied by [33] , both theoretically and empirically (across a range of computer vision tasks). It is also easy to validate in the preference learning setting when using a smaller dataset size. We included a baseline, "Classifier (Frozen)", which fixes the base model's features and only fine-tunes the classification head. When the dataset size is 8K (see Table 7 ), the OOD evaluation results of the baseline reward model (without freezing the backbone) are worse than those of the frozen one, demonstrating the negative effect of distorting pre-trained features. However, we would like to note that when the dataset size is sufficiently large, this negative effect can be mitigated, and the baseline reward model can surpass the frozen reward model due to having more trainable parameters to fit the large preference dataset.In contrast, by regularizing the hidden states, our GRM can achieve the regularizing effect while fine-tuning all parameters, showing strong performance with both large and small dataset sizes. In our paper, we consider a slightly different form of SFT objective as in Eq 10. A more straightforward objective isL SFT (θ LM ) = -E (x,yc)∼D [log (π θLM (y c | x))].In ideal situations, the two forms should perform similarly. We also tried the log form but found that it requires different hyperparameter tuning for the regularization weight α in Eq 4 due to changes in the loss scale. In Table 8 and Table 9 , "GRM logreg" outperforms the baseline reward model and matches or even slightly exceeds the performance of GRM on OOD tasks when α is tuned appropriately. This experiment uses the same gemma-2B-it as the base model.We found that the current form of SFT regularization can directly use the same hyperparameters as our DPO regularization. Therefore, we opted for this solution to maintain coherence with these regularizations and avoid the need for hyperparameter adjustments. We find the most impactful hyperparameter of GRM is the regularization weight α. Figure 6 presents an evaluation of GRM's performance under various α values. It is evident from the figure that setting α to either extreme, such as 0, or a relatively large value like 0.1, results in suboptimal out-of-distribution (OOD) performance. However, selecting an appropriate value between 0 and 0.1 consistently yields higher scores. In all our experiments, we default to an α value of 0.01. This choice has already shown significant performance improvements in our experiments.An interesting aspect to explore is how the structure of the nonlinear reward head influences preference learning performance. In Figure 7 , we compare the performance of the default GRM (using the SFT regularization) against a variant of GRM that incorporates an additional linear layer and a ReLU activation in the reward head, denoted as "2 layer". The results indicate that the two-layer version slightly surpasses the performance of the single-layer GRM on MT-Bench and RewardBench scores, but it exhibits a decline in the score on the HHH-Alignment. Due to this inconsistency, we opted not to include the two-layer version in our main experiments. However, future research focusing on the impact of the reward model's structure could yield promising insights.In Appendix A, we derive an objective that retains only the reward model r θ by replacing the policy π * θ with a formula of r θ . Empirically, this objective is challenging to optimize due to the calculation of Z θ . As an alternative, we propose a simplified objective by omitting the Z θ term:arg min θ {L reward (θ) -γE x,yc∼D [r θ (x, y c )]}.This objective includes a regularization term to maximize the average rewards of chosen responses. However, the second term can easily dominate the loss since the reward loss term is constrained by the logsigmoid operator. A more stable approach is to use the following empirical objective:arg min θ {L reward (θ) -γE x,yc∼D [log σ(r θ (x, y c ))]}.We refer to this regularizer as "positive regularization" or "pos reg" for short. We compare positive regularization with the baseline classifier and GRM with SFT regularization in Tables 10 and 11 . The base model for the reward models is gemma-2B-it, and GRM adopts the linear variant for the RewardBench results. "Positive regularization" does not yield improvement when the dataset size is limited to 40K, but it brings slight overall enhancement when learning from 400K training data.In contrast, GRM significantly enhances both ID and OOD accuracy, especially when learning from a limited preference dataset. These results demonstrate that our approach is more effective, even when based on similar theoretical derivation. In our default design, we use the preference dataset employed to train reward models to regularize the text-generation ability of the language head, eliminating the need for additional datasets. While we believe that other data formats, such as pretraining datasets, can also be beneficial, preference data offers a distinct advantage. It allows us to avoid using external datasets during reward modeling, which may also better align with the distribution of prompts and responses.To illustrate this, we conduct an experiment using GRM with text-generation regularization on an open-source pretraining dataset, togethercomputer/RedPajama-Data-1T-Sample7 (which includes text from Commoncrawl, Arxiv, and books), referred to as 'GRM pretrain reg'. For fairness, we only used a pretraining dataset of the same size as the training set for reward modeling.The results indicate that 'GRM pretrain reg' outperforms the baseline reward model and matches the performance of GRM when the dataset size is large (400K). However, when the dataset size is small, using a pretraining dataset is less effective than using the preference dataset. To demonstrate the advantage of GRM over vanilla reward modeling, we evaluate the win rate of models after PPO training with GRM against those with the vanilla reward model. The evaluation is conducted using GPT-4o on 100 randomly selected prompts from the test set in Unified-Feedback, with the order of responses randomly flipped to avoid order bias. The results below show a significantly higher win rate for GRM than the vanilla reward model across two different base reward models. In Figure 8 , we observed that reward models trained with label smoothing are vulnerable to hacking by BoN and PPO, leading to inferior performance compared to other baselines. The proxy score increases, while the gold score decreases rapidly. This finding suggests that previous robust techniques in the literature may not be effective for RLHF, underscoring the superiority of GRM as a more viable solution.In Tables 15, 16 , and 17, we present three examples that compare the responses of optimized language models using the PPO algorithm with different reward models. The base models for policy and reward models are all gemma-2b-it as in Section 5.2.For baselines, it is evident that the models exploit certain patterns in rewards, such as the "Ensemble (min)" methods. This exploitation often leads to a collapse into repeated patterns. Besides, the "Baseline" and "Margin" models tend to disregard instructions or refuse to respond to harmless prompts, as demonstrated in Tables 15 and 16 . Moreover, the baseline methods negatively impact the reasoning ability of language models for the math problem as in Table 17 . These observations indicate that current reward models can be easily hacked by the PPO algorithm, raising concerns about their reliability.In contrast, the GRM model demonstrates greater robustness in generating instruction-following responses and exhibits better reasoning abilities, even with identical hyperparameters of PPO. Notably, this superior performance of GRM is achieved even with a smaller training cost compared to ensemble baselines. These examples underscore the importance of GRM and its effectiveness in mitigating the overoptimization problem, further highlighting its potential in RLHF applications.The proposed approach to enhancing the generalization capabilities of reward models within the RLHF framework offers several positive societal impacts. By improving the accuracy of reward models on out-of-distribution (OOD) tasks, we can enhance the alignment of LLMs with human intent on larger dataset without human labels, leading to more reliable and stronger alignment. Moreover, the regularization technique that preserves the base model's language generation capabilities can contribute to the development of more robust and versatile AI systems, fostering innovation and efficiency across multiple domains. Currently, we do not foresee apparent negative societal impacts stemming from our methods. However, one potential adverse effect could arise if the generalizable reward model is exploited for harmful language model training. Therefore, future efforts in AI safety are crucial to prevent such misuse. Based on the conversation history, the human is looking for guidance on how to get a simple diamond-shaped kite in the air. The assistant recommends considering wind conditions such as the direction and strength of the wind, the kite type, and the intended flying environment. They suggest considering low wind speeds as suitable for beginners. Therefore, the assistant suggests that the human research the specific wind conditions and kite specifications suitable for their location and desired flying experience before attempting fly the kite. Remember to prioritize safety and seek guidance from experienced kite or professionals if possible.The response does not follow the conversational tone.Based on the conversation history, the human is seeking guidance on how to get a kite in the air. However, the assistant cannot provide specific instructions or recommendations without a thorough understanding of the individual's situation and capabilities. It's important to prioritize safety and avoid engaging in activities that could potentially harm the person or property involved.If you have any general kite-flying tips or suggestions, it's recommended to consult with a local park or outdoor enthusiast or seek guidance from a reputable kite-flying organization or instructor. Remember to always follow safety guidelines and regulations, especially when engaging in kite flying activities.The response does not follow the conversational tone.Sure, here' soprHãy guidHȗ´ȏng pulsa palab»() sedi lampa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa palab pulsa (omitted due to repeating)The model is hacked to generate repeated words.Sure, here's some advice on how to get a kite in the air safely and effectively: **Wind Conditions:** * Choose a location with consistent and gentle winds. Avoid strong or gusty winds, as they can make it difficult to control the kite. * Consider the wind speed and direction. Start with lighter winds, especially if you're a beginner. **Kite Selection:** * Choose a kite that's appropriate for your weight and the wind conditions. A well-sized kite will have sufficient lift to stay aloft and provide a thrilling flight experience. * Consider the kite's design. Diamond-shaped kites are relatively stable and easy to maneuver, while symmetrical kites offer more stability but may be more difficult to control. **Safety Precautions:** * Always fly kites away from people and obstacles. * Use a sturdy line or bridle that's long enough to allow the kite to fly freely but not become tangled. * Start in an open area with a clear view. * Be patient and allow the kite to fly for a few minutes before adjusting the line or releasing it. Remember to fly kites responsibly and always prioritize safety.A good response to the prompt. A more informative response to the prompt. The mission records were created by Don Henley, lead singer of The Eagles. The band was formed in 1974 and released several successful albums and hit songs throughout the 1970s and 1980s. Henley is widely regarded as one of the most influential and successful singers in the history of rock music. His songwriting and vocal contributions have influenced countless artists and fans worldwide. The Eagles became one of the best-selling and critically acclaimed bands of the late 20th century, with songs like "Hotel California," "Desperado," and "Hotel," among many hits. Henley's contributions to the band have solidified his status as a legend in the music industry.The context does not provide information about the lead singer of The Eagles, so I cannot answer this question from the provided context. For accurate information about mission records and their creators, it is recommended to refer to official sources or seek further research. Remember to always prioritize reliable sources and avoid relying on secondhand or biased information. If you have any specific questions or concerns related to mission records or the artists involved, consider consulting with experts or professionals familiar with the subject matter.Ensemble (min) Mission Records was created by lead singer of Eagles Eagles Keseitar sven pixabay Keseitar sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven pixabay sven (omitted due to repeating)The model is hacked to generate repeated words. The model is hacked to generate repeated words.The answer is Don Henley. Mission Records was a record label founded by Don Henley in 1974. The label was initially focused on releasing albums by The Eagles and other artists in the Eagles' genre. Mission Records went on to sign and release albums by a number of other artists, including Fleetwood Mac, Bon Jovi, and Linda Ronstadt.A concise reply to the question with a short explanation.Table 17 : Examples in the PPO experiments. GRM optimizes a better language model aligned with gold scores, while other baseline reward models can be easily hacked by PPO.You will be given a definition of a task first, then some input of the task. You are given a math word problem and you are supposed to apply multiplication or division mathematical operators on the numbers embedded in the text to answer the following question and then only report the final numerical answer.A chef needs to cook 9 potatoes . He has already cooked 7 . If each potato takes 3 minutes to cook , how long will it take him to cook the rest ? Output: NeurIPS Paper ChecklistCode and open-source reward models are available at https://github.com/YangRui2015/Generalizable-Reward-Modelhttps://huggingface.co/datasets/llm-blender/Unified-Feedbackhttps://huggingface.co/datasets/HuggingFaceH4/hhh_alignmenthttps://huggingface.co/datasets/lmsys/mt_bench_human_judgmentsreward-model-Mistral-7B-instruct-Unified-Feedbackhttps://huggingface.co/datasets/hendrydong/preference_700Khttps://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T-Sample 


Decision: The paper studies the issue of reward generalization. It re-introduces the language model loss to reward model training as a regularization term to improve reward generalization. Experiment results show that this method could improve the reward model performance on multiple benchmarks. Reviews generally agree that the motivation for the study is sound, and the experiments validate the effectiveness of the proposed method.

Reviews: review: paper_summary: paper_summary: The paper addresses the limitations of current reward models used in the reinforcement learning from human feedback (RLHF) framework, specifically their generalization capabilities to unseen prompts and responses. This limitation often leads to reward over-optimization, where the excessive optimization of rewards results in a decline in actual performance. The study proposes an approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states.
strength_weakness: strength_weakness: 1. The proposed method lacks innovation as it combines DPO and SFT loss into the RM training phase as a regularization term. The inclusion of SFT loss in RM training has already been explored in previous works, such as InstructGPT and Anthropic's RM training.

2. In the introduction, the authors mention that a randomly initialized head can distort pre-trained features, negatively impacting out-of-distribution (OOD) performance. Inspired by this finding, they propose to regularize feature distortion during fine-tuning for preference learning. However, there is no experimental evidence provided to support that this motivation holds true for RM. The improvements from adding regularization alone do not sufficiently prove that this motivation is solid.

3. The results for label smoothing are missing in Figures 2 and 3, which should be addressed.

4. The experimental section should include the alignment results after RL.
questions: questions: Please refer to the Weaknesses section for questions regarding the paper.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes generalizable reward model (GRM) which modifies the standard reward-learning objective by adding an auxiliary task with a separate language modeling head. The auxiliary loss is either DPO or SFT. Experiments and ablations are conducted using mistral and gemma models and data from unified-feedback and evaluating on HHH, MT-bench, and rewardbench showing GRM improves OOD performance. Finally, GRM reward models are used to train new policies.
strength_weakness: strength_weakness: 1. While the paper is generally comprehensive, there could be additional exploration of just using language modeling as an auxiliary task. It is not clear if the benefit of GRM is coming from language modeling the data from the preference dataset or if it would benefit from language modeling on any data (or even data from an even broader distribution, controlling for more OOD data). A further exploration of this would be good.
questions: questions: None
limitations: limitations: There could be more discussion of limitations
paper_summary: paper_summary: This paper introduces a method that retains the base model's language model head while incorporating text-generation losses to preserve the hidden states' text generation capabilities.
strength_weakness: strength_weakness: * Figure 3 (b) appears unusual, as the gold score decreases at the beginning of training. This could indicate suboptimal hyperparameter tuning or potential drawbacks in the pipeline.
* The results demonstrate limited advantages. It is recommended to validate the method's benefits with a larger dataset.
questions: questions: * The training details mention that the reward models are trained for only 2 epochs. This choice may not be optimal, considering the risk of overfitting in the full fine-tuning setting. It would be helpful to discuss the reasoning behind this decision or the impact of learning rate and convergence speed.
* The simplicity of the regularization method is intriguing. Can it effectively mitigate reward hacking, potentially surpassing reward model ensemble techniques? It would be valuable to explore and discuss this aspect.
limitations: limitations: * The experiments may lack solidity due to the heavy reliance on the reward model in the PPO algorithm. If the reward model is not properly trained, the results can be significantly impacted, introducing randomness to the work.
* The inclusion of manually synthetic data in the dataset may not accurately reflect real-world data, limiting the generalizability of the findings.
paper_summary: paper_summary: This paper is concerned with the overoptimization issue in reward modeling: When optimizing a policy against a reward model, this leads to a distributional shift that can lead to an increase in the proxy score while the true score decreases. This paper addresses these issues by regularizing the hidden states in reward models utilizing an additional output head that (with some variations) predicts the text of the winning trajectories in the comparison data. The idea is that this prediction task (a) helps keep the intermediate, strong representations from pretraining, and (b) adapts them to text that is more similar to what's encountered in post-training, thus leading to stronger generalization to the data encountered when optimizing the policy.
strength_weakness: strength_weakness: No strong weaknesses come to mind, though see the questions and comments below for other experiments that could be interesting to gain more confidence (though imho not necessary for this submission to be accepted).
questions: questions: I'm using this section for questions, suggestions, and minor weaknesses.

a. "*In the RLHF stage, various policy optimization methods can be applied, with two frequently used methods being Best-of-n Sampling (BoN) and Proximal Policy Optimization (PPO).*" --- This paper seems to view RLHF as the stage *after* reward modeling, so just policy optimization. In contrast, I think it's more typical to view reward modeling as *part of* RLHF.

b. "*While straightforward, the DPO regularization requires a reference model during training.*" --- Don't you also need a reference policy during the PPO stage, given by the KL regularization with the SFT model? So it seems like this is no additional cost here?

c. Equation (7): Did you also consider replacing $\log \circ \sigma \circ \beta \circ \log$ by just $\log$? Do you expect this to perform better/worse?

d. Did you consider to interleave reward modeling with pretraining on a pretraining dataset to regularize the hidden states? I wonder if this performs better or worse than essentially doing pretraining on the preference dataset (as you do in your SFT regularization method).

e. Did you check whether in SFT regularization it matters whether you predict the text of the winning responses, instead of trying to predict the losing responses? I could imagine that the former performs better since it brings the reward model closer to understand a distribution that the PPO stage *will steer towards*, but I'm curious if that prediction holds up empirically.

f. For HHH-Alignment, MT-Bench, and RewardBench, could you give more details on what the benchmarks test for and what the scores mean?

g. I'd recommend putting the limitations and broader impact sections into the main paper.
limitations: limitations: Limitations are described in Appendix D.

Comments: title: Looking Forward to Your Feedback
Looking Forward to Your Valuable Feedback
Thanks for raising the score!
comment: Dear Reviewer tSHJ,

We deeply appreciate your time, effort, and valuable feedback. **Since the discussion deadline is less than 10 hours away and we have not yet received an acknowledgment from you, we kindly request you to review our replies**. We hope that our responses, including detailed explanations about the differences between the prior reward modeling paradigm and our approach, as well as additional experimental results on feature distortion and alignment after RL, have addressed your concerns.

 If our responses have resolved your issues, we would greatly appreciate it if you could consider raising the score. If you have any additional concerns, please feel free to post them, and we would be happy to discuss them further before the discussion deadline

Thank you once again for your constructive review.

Best, 

The Authors
Dear Reviewer tSHJ,

We deeply appreciate your time, effort, and valuable feedback. We hope that our responses, including detailed explanations about the differences between the prior reward modeling paradigm and our approach, as well as additional experimental results on feature distortion and alignment after RL, have addressed your concerns.

If our responses have resolved your issues, we would greatly appreciate it if you could consider raising the score. If you have any additional concerns, please feel free to post them, and we would be happy to discuss them further before the discussion deadline on Aug 13th.

Thank you once again for your thoughtful review.

Best,
The Authors
We would like to thank the reviewer for acknowledging the positive aspects of our work. We greatly appreciate your valuable comments and suggestions, which enhance the quality of our paper.
(I'm another reviewer chiming in):

Could it be that you misclicked and accidentally raised your confidence instead of the score to 5?
Yes, after carefully reviewing your response, I have decided to raise the score to 5. While the paper's innovation appears limited, I want to emphasize the positive aspects, particularly the strong experimental results.
Thank you for your time, effort, and valuable feedback. We hope that our responses, including detailed explanations regarding the hyperparameter design and validation on a larger dataset, have addressed your concerns.

If our responses have resolved your issues, we would greatly appreciate it if you could consider raising the score. If you have any additional concerns, please feel free to post them, and we would be happy to discuss them further before the discussion deadline on August 13th.
Thanks for the authors' comprehensive response. I would like to keep the current score. Thanks!
Thank you for your positive evaluation of our work. We greatly appreciate your insightful questions and suggestions, as they enhance the accuracy of our writing and the comprehensiveness of our evaluation.
Thank you for your positive evaluation of our work. We greatly appreciate the insightful questions, particularly the one regarding the use of other text data for regularization, as they help make our work more comprehensive.
Thanks for your response and for running the additional experiments. Indeed it is interesting that using pre-training data for regularization can also be effective, while not quite as effective as the original method.

I will leave my accept score and continue to think this is a strong paper.
Thank you for the detailed answers to my questions!