Critique-out-Loud Reward Models

Authors: Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan Daniel Chang, Prithviraj Ammanabrolu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Reward Model, Chain of Thought, Alignment, Preference Modeling, Large Language Model, LLM, RLHF

Abstract:
Traditionally, reward models used for reinforcement learning from human feedback (RLHF) are trained to directly predict preference scores without leveraging the generation capabilities of the underlying large language model (LLM). This limits the capabilities of reward models as they must reason implicitly about the quality of a response, i.e., preference modeling must be performed in a single forward pass through the model. To enable reward models to reason explicitly about the quality of a response, we introduce Critique-out-Loud (CLoud) reward models. CLoud reward models operate by first generating a natural language critique of the assistant's response that is then used to predict a scalar reward for the quality of the response. We demonstrate the success of CLoud reward models for both Llama-3-8B and 70B base models: compared to classic reward models CLoud reward models improve pairwise preference classification accuracy on RewardBench by 4.65 and 5.84 percentage points for the 8B and 70B base models respectively. Furthermore, CLoud reward models lead to a Pareto improvement for win rate on ArenaHard when used as the scoring model for Best-of-N. Finally, we explore how to exploit the dynamic inference compute capabilities of CLoud reward models by performing self-consistency decoding for reward prediction.

Decision: reject
Reviewer scores: 3, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730485908639

thread_type:
official_review

<thread_object>
creator: reviewer_pirr

<content>
summary:
The paper proposes an interesting idea of joining training the LM head and RM head for reward model so that it first generates CoT-type natural-langauge-based critique, then let the reward model generate new scalar score. Experiments show that the performance greatly improves compared with normal reward training pipelines.

strengths:
The idea of training an LM head for critique appears neat and novel. The experiments are sound and look at both the performance on reward bench and arena hard for downstream performance of a trained model.

weaknesses:
I don't see major weakness from the paper and I like the proposed idea a lot. But I would appreciate if the authors could help clarify some of the questions I have below. Please refer to question section.

questions:
1. The authors use Llama 405B to generate oracle critique data for SFT, and replace the oracle data with fine-tuned model's self-genereated critique. Usually when you distill from larger models, there will be direct gain in model performance. How do we know that if the performance gain is mainly from distilling responses from large models, or from the methodology itself? If the critique SFT data is mostly self-generated (not on the fine-tuned one), would we still expect such high gain in downstream performance? This will be helpful for the case where we want to improve the largest model available with only binary human feedback.

2. The generation of SFT data is using llama 3.1 405B but the training model is llama 3 instead of llama 3.1, what's the reason behind this? Would the gain of the same methodology appear weaker when it comes to stronger base RM?

3. Does the gain mainly come from the diverse critiques generated, or from more clear instructions of what the RM should look like? For example, if we provide a very comprehensive fixed system prompt asking the response to focus on safety, helpfulness, comprehensiveness, (e.g. by modifying the prompt in Figure 9) and provide a combination of system prompt + user prompt + response to RM, would it achieve similar performance as user prompt + response + critique?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538366332

thread_type:
official_review

<thread_object>
creator: reviewer_spgi

<content>
summary:
The authors propose Critique-out-Loud (CLoud) reward models. CLoud reward models operate by first generating a natural language critique of the assistant's response, which is then used to predict a scalar reward for the quality of the response. Their effectiveness is demonstrated across different models and benchmarks.The motivation behind this paper is interesting and contributes to the ongoing advancements in RLHF fine-tuning for current LLMs.
The writing is clear and easy to understand.

strengths:
1. The motivation behind this paper is interesting and contributes to the ongoing advancements in RLHF fine-tuning for current LLMs.

2. The writing is clear and easy to understand.

weaknesses:
1. The authors constructed their dataset based on UltraFeedback and UltraInstruct, requiring Llama-3.1-405B-Instruct to generate critiques for chosen and non-chosen responses. This seems to introduce additional annotation costs and complexities, which is a drawback of the method. How effective would this pipeline be in a pure self-critique setting?

2. The authors test the method's effectiveness in the safety domain, where critique is currently widely used in LLM safety alignment scenarios. This is intriguing. However, the authors' safety evaluation of the method on RewardBench raises concerns, as my understanding is that RewardBench aggregates existing datasets with inconsistent safety classifications across different safety datasets. Consequently, the detailed assessments in safety and reasoning may lack credibility. I recommend that the authors evaluate the method on specific datasets for different categories, such as reasoning on GSM8K [5] and safety on BeaverTails [6], which are designed for specific LLM categories.

3. In validating the effectiveness of CLoud RM, apart from testing on RewardBench, the authors conducted only BoN experiments. What is the impact of Cloud RM on RLHF? This is a critical aspect that I would like to address, especially since Cloud RM adds the additional annotation cost of critiques. The authors claim that the scalar reward is more accurate but have not conducted any RLHF experiments [3].

4. This method inevitably increases the complexity and cost of the pipeline. I am particularly interested in novel uses of critique beyond merely improving the accuracy of scalar rewards. Since it incorporates additional information, it is not surprising that the scalar reward's accuracy improves compared to a pure reward model. However, its influence on RLHF could yield other applications, such as directly aligning models using critiques across multiple dimensions, such as Aligner [4], which trains based on correct data (a type of critique) to enable offline alignment. It would be interesting to explore whether learning from richer critique feedback is feasible [2]. Simply using a critique head and a scoring head is not particularly innovative, as discussed in relation to help steer.

5. Some intriguing ablation studies have not been conducted. For example, allowing the model to output a critique before producing a scalar reward could highlight the differences between training using the Cloud RM approach versus traditional methods. (I am unaware if such an experiment has been done; please correct me if I’m mistaken.)

[1] Self-critiquing models for assisting human evaluators

[2] Training Language Models with Language Feedback

[3] OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

[4] Aligner: Efficient Alignment by Learning to Correct

[5] https://huggingface.co/datasets/openai/gsm8k

[6] https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF

Overall, the motivation behind this paper is compelling. I welcome further discussions with the authors during the rebuttal period.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719891011

thread_type:
official_review

<thread_object>
creator: reviewer_bvce

<content>
summary:
This paper introduces “Critique-out-Loud” (CLoud) reward model, which aims to enhance reinforcement learning from human feedback (RLHF) by generating criticisms before giving rewards. This approach improves the performance and policy quality of preference classification, especially in LLM-based preference models, by unifying reward modelling with chain-of-thought reasoning.

strengths:
An innovative approach is proposed that combines critique generation with reward prediction to address the limitations of traditional reward models that lack explicit reasoning.
Empirical results show significant performance improvements on both preference modelling and BoN compared to traditional models.
Provides additional tests in strategy training and self-consistent decoding, demonstrating potential application scenarios.

weaknesses:
Criticism generation methods, while innovative, may introduce bias if the criticisms reflect some fixed bias of LLMs rather than true preferences.
The process of generating criticisms can be viewed as a sort of distillation from the larger model, and the comparisons provided in the paper are slightly unfair compared to scoring directly using the reward model. More ablation should be provided.
Critic generation and inference practices lack rational supervision. For GenRM, further discussion should be proposed on the construction of the critique process.

questions:
The approach in the paper can be viewed as incremental compared to a general reward model, so performance increases do not fully substantiate its value. More experiments are needed to prove that Critique-out-Loud works universally.
1. How does the performance of this type of operation compare to simply llm-judge to produce preference labeling?
2. Do authors try to output the reward scores before generating the corresponding critiques, and whether this format also improve the performance of the reward model？
3. How generalizable is the approach given that the existing training data is largely constructed on full domain samples? If this is the paradigm rather than a gain from the data itself, we should see an increase in scoring capabilities on OOD data when trained only on domain-specific data (e.g., trained on math data and tested on safety data)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730729394711

thread_type:
official_review

<thread_object>
creator: reviewer_t6ho

<content>
summary:
This paper proposes a reward modeling approach that combines next-token-prediction loss with binary rating loss, which improves the performance of the Llama3-8B and Llama3-70B language models on RewardBench and BoN. The authors conducted extensive ablation studies to analyze their proposed method.

strengths:
- Well written, the technical approach is very clear, and Figure 1 and Figure 2 provide a clear understanding of the CLoud reward model training process.
- Based on the experimental results published by the authors, the CLoud method seems to effectively improve the model's performance on RewardBench.
- The author's ablation studies are very comprehensive.

weaknesses:
- One of the most critical applications of the Reward Model is RLHF, yet the authors have not verified the improvement of CLoud RM over traditional RMs in RLHF. I believe this would greatly limit the widespread application of this method.
- It is not very clear what significant issues in Reward Modeling CLoud has addressed, such as reward overoptimization. From the experimental results, it appears to only enhance the reward model's ability to distinguish between binary responses.
- The experimental setup seems inconsistent. Based on my understanding from page 5, the authors' preference dataset and BoN dataset are derived from sampling Llama3-8B-Instruct. However, is it appropriate to use data generated by an 8B model for experiments with a 70B model? What is the rationale behind this choice? The authors should present this in the paper.
- The above point leads to my skepticism about the author's claim that "8B CLoud reward model even outperforms the 70B classic reward model."
- The experiment on self-consistency is interesting, but the author's research is not thorough enough. If they could investigate how the quality of critique generation affects scoring accuracy, and combine this with methods like COT, TOT, etc., for further exploration, it would be more impressive.

**Minors**

- I'm not particularly fond of the title "Results" for Section 3; terms like "Experiment" might be more appropriate.
- RQ2 in Section 3 seems a bit odd because there hasn't been any prior mention of on-policy/off-policy concepts, which makes it difficult to understand.

questions:
- How much more overhead does CLoud RM incur during inference compared to traditional RM? Given that CLoud RM needs to generate an entire sequence before scoring, it is foreseeable that it would be more time-consuming than traditional RM. My concern about the inference cost grew even more after discovering that the author used the self-consistency method for multiple sampling averages.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Critique-out-Loud reward models, which generate natural language critiques before predicting scalar rewards. This approach aims to improve the reasoning and preference classification capabilities of reward models. The authors present empirical results demonstrating performance improvements on RewardBench and ArenaHard benchmarks through Best-of-N scoring using CLoud reward model. 

Several reviewers found the contribution incremental, primarily combining critique generation and reward scoring without sufficient novelty. Concerns were raised regarding the lack of RLHF experiments to substantiate claims about downstream improvements. The reliance on critiques from larger models (e.g., Llama-3.1-405B) raises questions of whether the gains stem from methodology or model distillation. The authors addressed some concerns from reviewers but the doubts about the novelty and effectiveness of CLoud in RL settings beyond best-of-N remain.