Generative Reward Models

Authors: Dakota Mahan, Duy Van Phung, Alon Albalak, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: RLHF, reward model, LLM Judge, reasoning

Abstract:
Reinforcement Learning from Human Feedback (RLHF) has greatly improved the performance of modern Large Language Models (LLMs). The RLHF process is resource-intensive and technically challenging, generally requiring a large collection of human preference labels over model-generated outputs. Reinforcement Learning from AI Feedback (RLAIF) addresses this data collection challenge by leveraging synthetic preferences generated by an LLM. However, recent work has shown that synthetic preferences labels may not align well with human preference judgments.

To address this, we propose a hybrid approach that unifies RLHF and RLAIF methodologies. We introduce GenRM, an iterative algorithm that trains an LLM on self-generated reasoning traces, leading to synthetic preference labels matching human preference judgments. Empirically, we show that zero-shot LLM-based judgments under-perform compared to Bradley-Terry reward models on in-distribution tasks (between 9-36%). In contrast, GenRM achieves in-distribution accuracy comparable to Bradley-Terry models, while significantly outperforming them on out-of-distribution tasks (between 10-45%). Moreover, GenRM surpasses the performance of using LLMs as judges on both in-distribution (by 9-31%) and out-of-distribution tasks (by 2- 6%). Our results show that combining the strengths of RLHF and RLAIF offers a promising approach for improving the quality of synthetic preference labels.

Decision: reject
Reviewer scores: 5, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730181532139

thread_type:
official_review

<thread_object>
creator: reviewer_4sex

<content>
summary:
This paper proposes a Generative Reward Modeling (GenRM) approach that combines the main ideas of RLHF and RLAIF, aiming at fine-tuning LLMs with self-generated reasoning traces to generate synthetic preference labels that match human preference judgments. Experiments on Llama 3.1 8B confirm the effectiveness of the proposed method.

strengths:
This research topic is interesting.

weaknesses:
1. Mixing RLHF and RLAIF gets GenRM both their advantages, but also inherits both their disadvantages, and the paper lacks a discussion of that. And the idea doesn't seem very novel.

2. The foundation model in the experiment only uses Llama3.1 8B, which is not quite demonstrating the generality and robustness of the method. Experimenting on a more variety (e.g., mistral series) and size (e.g., 13B, 70B) of models would make this paper more solid.

3. Why does this method perform better on out-of-distribution (OOD) data? It's not very intuitive, not very easy to understand, and it would be better if there was more discussion and analysis of it.

questions:
1. Mixing RLHF and RLAIF gets GenRM both their advantages, but also inherits both their disadvantages. Would you mind analyzing it a little more?

2. The foundation model in the experiment only uses Llama3.1 8B, which is not quite demonstrating the generality and robustness of the method. Experimenting on a more variety (e.g., mistral series) and size (e.g., 13B, 70B) of models would make this paper more solid.

3. Why does this method perform better on out-of-distribution (OOD) data? It's not very intuitive, not very easy to understand, and it would be better if there was more discussion and analysis of it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730396540675

thread_type:
official_review

<thread_object>
creator: reviewer_yvkw

<content>
summary:
This paper tackles the problem that feedback from current RLAIF methods does not align with human preferences. It introduces a framework named GenRM, which integrates the strengths of RLHF and RLAIF. The authors utilize self-generated reasoning traces and iterative training loops, achieving results on in-distribution tasks that are comparable to Bradley-Terry models and outperforming them on out-of-distribution tasks. Additionally, GenRM surpasses the performance of using LLMs as judges for both types of tasks.

strengths:
The paper presents an iterative framework called GenRM designed to train an LLM using self-generated reasoning traces. This framework is novel to some extent and has been validated on both in-distribution and out-of-distribution tasks through various ablation settings. It demonstrates scalability and good efficiency. The experimental results indicate that integrating chain-of-thought reasoning within preference modeling enhances the model's reasoning ability, contributing valuable insights for future studies in RLAIF and RLHF.

weaknesses:
1. The evaluation of RM is inadequate, lacking experiments that assess agreement with human preferences. Additionally, only one out-of-distribution dataset, RewardBench, is utilized for evaluation.
2. The qualitative results presented in Figure 3 are unconvincing. Both STaR-DPO and LLM-as-a-judge use the same system prompt, as noted in Appendix A.1, which states that evaluations should consider factors such as helpfulness, relevance, accuracy, depth, creativity, and detail. However, your explanation for why STaR-DPO performs better than LLM-as-a-judge suggests that you prioritize instruction following over depth, helpfulness, and detail. While STaR-DPO includes an additional animal species, which is more detailed and helpful, it does not strictly follow the instructions. Furthermore, the system prompt is not always appropriate, as it can sometimes be contradictory.
3. The paper's title is somewhat unclear, making it difficult to discern your contribution. Additionally, some model settings are not sufficiently clear, such as the differences between GenRM and the STaR-SFT or STaR-DPO models.

questions:
1. Are STaR-SFT and STaR-DPO both trained on the same base model, rather than STaR-DPO being trained on STaR-SFT?
2. Where does the RLHF component fit into your framework? I see that you mention "incorporates human feedback in initial stages" in Section 7, but what exactly do you define as the initial stage, and how is human feedback utilized within this stage?
3. Could you provide a detailed summary of all the model settings, including the eight models depicted in Figure 2? Specifically, what is the base model, if it has been trained, what the training objective is, and what the dataset construction process entails? Thank you.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730569520925

thread_type:
official_review

<thread_object>
creator: reviewer_xwva

<content>
summary:
This paper proposes a reward modeling formulation GenRM that explicitly indicates the pairwise preference with $\pi_{\phi}(I | x, y_1, y_2)$, which allows a natural combination of RLHF and RLAIF approaches. By incorporating 1) CoT of Self-taught Reasoning traces of rationalization models, and 2) DPO-style optimization objective, the trained model STaR-DPO achieves on-par performance with Bradley-Terry reward models on IID settings and generalizes much better on OOD settings, e.g. safety and reasoning tasks.

strengths:
The strength of this paper is its potential significance and novelty. The proposed direct formulation of the preference indicator looks interesting to me, which may allow more types of approaches and data sources to be combined to train stronger reward models, especially in real-world scenarios that require strong generalization ability.

weaknesses:
The clarity and quality of the paper can be significantly improved. Specifically, for clarity, the organization of the experimental section can be re-structured to highlight the answers for the proposed questions in lines 296-302. Also, the accuracy numbers in Figure 2 and Figure 4 are recommended to be included in the paper, at least in the Appendix.

Concerning quality, it would be strongly recommended to have policy models trained on STaR-DPO to see whether this improvement in reward modeling can truly translate into gains in better policy models, such as better instruction-following/math/code LLMs. The detailed comments are listed as follows.

### Detailed comments
  * [Clarity] Section 4: Pseudo-code of the proposed algorithm is recommended, at least for STaR-DPO.
  * [Clarity] Section 5: it is recommended to provide separated tables/figures to highlight each point in lines 296-302. Some additional rewards that explicitly answer the question can greatly improve clarity.
  * **[Clarity]** line 306-323: the analysis here can be organized and summarized in several important remarks, especially for those relevant to the questions in lines 296-302. Others can be moved to the appendix.
  * [Clarity] line 224: typo, there is a redundant comma ","
  * [Clarity] line 235": typo, "fine tuning" -> "fine-tuning"
  * [Clarity] line 471: "by a a" -> "by a"
  * [Clarity] line 234, 244, 258: As $\pi$ normally indicates policies in RL, it would be recommended to change this with another symbol.

  * [Quality, Clarity] Section 5, Figures 2 & 4: better include the specific number of accuracies in Tables.
  * [Quality, Clarity] Section 5: a table that explicitly states the gain of the utilized techniques would be strongly encouraged, including GenRM formulation, CoT with other models, CoT with Self-Taught Reasoning, DPO, and test-time computing.
  * **[Quality]** It is strongly encouraged to include results that show the improvements in policy models trained with STaR-DPO.

questions:
I am wondering if there are results showing the improvements of policy models trained on STaR-DPO.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711623408

thread_type:
official_review

<thread_object>
creator: reviewer_674z

<content>
summary:
* The paper proposes GenRM (no CoT), which is similar to PairRM [1], focusing on direct pairwise preference prediction without the Bradley-Terry assumption. The core technical contribution is mainly about removing the explicit reward modeling design.
* Introduces CoT-GenRM (similar to Self-Rewarding LMs [2]) with two training approaches: bootstrapping and rationalization. Through experimental analysis, the authors found that while rationalization works well in-domain, it doesn't generalize effectively to OOD domains. The paper demonstrates this through comprehensive evaluations.
* A key advantage demonstrated is that self-consistency can be used as test-time compute scaling to improve preference modeling accuracy. The experimental results show that majority voting with 32 samples can significantly boost model performance across different evaluation settings.

[1] PairRM: https://arxiv.org/abs/2306.02561
[2] Self-Rewarding LMs: https://arxiv.org/abs/2401.10020

strengths:
* The paper successfully demonstrates that majority voting (self-consistency) at test time can substantially improve preference modeling accuracy. The results show consistent improvements across different datasets, providing a practical way to enhance preference modeling through additional compute at inference time.
* The evaluation for reward modeling covers both in-domain (UltraFeedback) and out-of-domain (RewardBench) datasets. The experimental results demonstrate robust performance improvements, particularly in OOD scenarios where traditional approaches often struggle.
* The paper provides strong empirical evidence that DPO training works effectively with CoT-GenRM. The experiments show that this training approach leads to better generalization compared to other training methods.

weaknesses:
* The evaluation is limited in demonstrating what the preference accuracy improvements mean for actual policy performance. While the paper shows improvements in preference modeling accuracy, this metric heavily depends on the policy sample distribution. Notably absent is an analysis of Best-of-N (BoN) performance with the proposed reward model, which would be crucial for understanding practical impact.
* The practical implementation raises significant concerns. The requirement for 32 majority votes and the limitation to pairwise comparison settings could make it challenging to implement in real-world scenarios, particularly in PPO. The paper doesn't adequately address how to handle these computational requirements in practice.
* The core concepts of GenRM (no CoT) and CoT-RM have been previously explored in literature, such as PairRM [1] and Self-Rewarding LMs [2]. The paper would benefit from more thorough comparisons with existing methods to better establish its novel contributions.

questions:
* Can you provide experimental results showing the Best-of-N performance with your reward model? This would help understand how the improvements in preference modeling translate to actual policy performance.
* Given the computational requirements of majority voting, how would you recommend implementing this approach practically in PPO? What modifications or optimizations might be needed?
* Could you elaborate on the comparisons with similar existing methods in the literature? Specifically, how does your approach differ from and improve upon previous work like PairRM [1] and Self-Rewarding LMs [2]?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The main goal of this paper is to propose a hybrid approach that combines RLHF and RLAIF approaches for reward modeling. The proposed method, STaR-DPO, shows on-par performance with Bradley-Terry reward models under IID settings and better generalization under OOD settings. Reviewers have significant concerns about the novelty, as pointed out by Reviewer 674z, and the practicality of the proposed model, given that it needs 32 models for majority voting. Reviewers also raised concerns about the clarity of the paper as well as the experimental setup. The authors are encouraged to incorporate these comments when preparing the next iteration of the paper.