SELF-EVOLVED REWARD LEARNING FOR LLMS

Authors: Chenghua Huang, Zhizhen Fan, Lu Wang, Fangkai Yang, Pu Zhao, Zeqi Lin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
Primary area: reinforcement learning
Keywords: RLHF, LLM, Self-learning

Abstract:
Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning language models with human preferences and is a key factor in the success of modern conversational models like GPT-4, ChatGPT, and Llama 2. A significant challenge in employing RLHF lies in training a reliable RM, which relies on high-quality labels. Typically, these labels are provided by human experts or a stronger AI, both of which can be costly and introduce bias that may affect the language model's responses. As models improve, human input may become less effective in enhancing their performance. This paper explores the potential of using the RM itself to generate additional training data for a more robust RM. Our experiments demonstrate that reinforcement learning from self-feedback outperforms baseline approaches.
We conducted extensive experiments with our approach on multiple datasets, such as HH-RLHF and UltraFeedback, and models including Mistral and Llama 3, comparing it against various baselines. Our results indicate that, even with a limited amount of human-labeled data, learning from self-feedback can robustly enhance the performance of the RM, thereby improving the capabilities of large language models.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729719383329

thread_type:
official_review

<thread_object>
creator: reviewer_ezma

<content>
summary:
This paper proposes self-evolved reward learning where the reward model is used to obtain more training data for iterative improvement. The algorithm is straightforward: (1) generate two responses per prompt and label using RM, (2) only keep response pairs where reward difference is large -- there are two filtering strategies (Status 1 and Status 2) and the authors alternate between them, (3) train RM with the filtered data; then, iterate the above. Finally, use RM for LM training. Findings: Only 15% human-annotated data is needed so that the new model archives performance similar to training on the entire human-annotation dataset; experiments are done on a few models (7B to 13B).

strengths:
The self-improvement paradigm could save lots of resources. 

I appreciate the fact that the authors not only tested on reward modeling benchmarks but also LMs (by doing PPO). 

Experiments are done on multiple models and multiple model sizes.

weaknesses:
On baselines:
- It’s unclear how specifically the authors trained “Loop 0” and “Full dataset” baseline models. What is the loss function? How are hyperparameters tuned? 
- Have the authors tried regular scalar-valued reward functions as baselines? 

On eval: Have the authors tested on common benchmarks like RewardBench (so we can easily compare to other scalar and generative RMs)?

In real LLM training, the same dataset would always contain examples of multiple domains. In the same dataset, lots of examples could be on simple helpfulness prompts, and lots of examples are on coding/reasoning prompts. In this case, how would thresholding work? Would the model first learn simple prompts and then coding/reasoning/complicated prompts? Or would the model give up on coding/reasoning/complicated prompts?

Eq. (3) can be clearer:
- It looks like Status 1 and Status 2 can be achieved at the same time? In that case, does S=Status1 or Status2? 
- Do we have a different S at every i? 
- Do we need to recompute Delta_p (avg through every i) after each gradient step? 
- When would the model hit “Stop”? What if the “otherwise” case never happens? An explanation in natural language would be great.

Details are lacking on data generation/filtering
- How specifically are responses generated?
- What percentage of data are filtered at different point in training? 
- Are there more conclusive evidence that alternating between status are beneficial? How are hyperparameters tuned -- based on what metrics? 


Other misc issues: 
- Eq. (4): missing right bracket
- Table 1 caption mentions “Our method” but I don’t see “Our method” in the table. Do you mean SER?
- Serious misuse of \citet and \citep. Most of the citations on the first page and the second page, for example, should be in \citep.

———

UPDATE: increased score to 5 but still concerned about paragraph 3 (and hopefully more detailed result on paragraph 2) above.

questions:
Are Llama models base models or instruction-tuned models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730252046683

thread_type:
official_review

<thread_object>
creator: reviewer_u5tr

<content>
summary:
The paper proposed a self-improvement RM training strategy to circumvent the challenges of collecting human preference data. The RM is initially warmed up using only 15% human preference, then it’s improved iteratively through a series of self-labeling and retraining. During the self-labeling the current RM is used to score unlabeled data (i.e., prompt and answer pairs), then two data filtering strategies are devised based on the predicted probability differences between answer pairs. The goal is for the RM to (1) first learn from pairs with clear distinction (trained on very high- and very low-scored response pairs) before (2) switching to refining its understanding of subtle differences (trained on pairs that are more similar). When no improvement is seen on this current data, training stops and new data is collected. 

The author tested their approach on multiple datasets including HH-RLHF, UltraFeedback as well as several model sizes and families including Llama and Mistral. They also evaluate how the improved RM helps during PPO.

strengths:
- Promising results on improving the RM performance matching or exceeding the baseline train on the full human labeled data
- The curriculum learning style for the RM which encourage learning from easy and then hard pairs seems an interesting finding in the context of RM optimization

weaknesses:
- The Experimental design is very narrow, and limited. (see questions for more details). E.g, I would expect to see results on downstream/standard benchmarks. However currently only preference accuracy is reported.
- Some claims in the results section are unsupported (mixed results which are generalized) → see detail below
- The paper clearly benefits from some rewriting and reorganization. In its current form several details are missing which makes it uneasy to follow.

questions:
1. Are the pairs of responses generated from the same base LLM? Or are the authors using the response pairs in the existing dataset. While the author mentioned generating two responses from the LLM (line 173), it is not clear which model they use to generate responses and if they do this on-policy data generation for all the iterations? More details on this is appreciated. 
2. Line 174-175: how generating two responses on-policy from the LLM "enriches" the training data with diverse answer qualities? What aspect of this setup is unique or does it guarantee quality/diversity?
3. Intuitively, what does the RM learn from its highly confident labeled data during stage 1? In other words, when the RM is too confident about a preference label (irrespective of whether it may be incorrect), what training signals it is gaining from it? Did authors try some ablations to test the contribution of each data state?
4. How the number of training steps differ in SER vs Full dataset in Table 1?
5. One would expect to see the performance of your PPO'ed model on downstream standard benchmarks like AlpacaEval, MMLU, BBH, MATH, CODEX, etc. So technically, start from a pre-trained base model doing supervised finetuning on some existing instruction-tuning data , then try doing PPO using your RM vs the baseline RM (trained on full data, e.g. on UltraFeedback) and compare performance over standard benchmarks. This will more realistically show the effectiveness of your approach as opposed to just narrowly evaluating models in their ability to choose the preferred response.
6. Line 343: The scaling claims are somewhat unsupported, based on Table 1 13b> 7b> 8b. more experiments specifically within the same model family is needed to make informative scaling trend conclusions.
7. Section 4.1.2, there's no explanation as to what Loop 1-3 refers to in section 4.1.2 and figure 2. It is not clear if loop is referring to one iteration of RM training (including both state 1 and 2), or something else? Line 373 states that "loop 2 is the least significant across all iterations" which makes me think loop != iteration
8. Line 418: does that mean Loop 1 and Loop 2 did not include data strategy of status 2? In general more details on the training is nice in the main paper.
9. Section 4.2: it would make sense to try PPO on a more general purpose preference dataset like ultrafeedback and see how it transfers to downstream tasks and not just alpacaeval style evaluation
10. Suggestion: the authors refer to their models as SER, or self RLHF. It’s nice to stick to one name and be consistent with it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730426468500

thread_type:
official_review

<thread_object>
creator: reviewer_vmdv

<content>
summary:
This paper introduces an approach called Self-Evolved Reward Learning (SER), which leverages an RM to autonomously curate its training data for continuous improvement. The authors conduct experiments across various models and datasets, demonstrating that the SER approach can train the reward model well using only 15% of the labeled data, which results in performance comparable to an RM trained on the full labeled dataset, both in terms of evaluation capability and integration within PPO training.

strengths:
**Motivation:** Given the high costs associated with acquiring human-annotated data for reward model training, the development of self-training approaches is highly beneficial. Such methods can leverage a minimal amount of human annotations for initial model training and subsequently enhance performance through self-improvement.

**Soundness:** The paper's various training strategies, which are based on different statuses, are well-conceived and intuitively sound.

**Substance:** The authors have conducted extensive experiments across multiple models and datasets, effectively demonstrating the efficacy of their Self-Evolved Reward Learning (SER) approach.

weaknesses:
**Clarity:** The paper lacks clarity in several places, with some descriptions being particularly confusing. For instance:

- **Lines 206-211:** It remains unclear how the status is determined, given that $p_i^1$ and $p_i^2$ are both single numbers. How are these individual numbers used to decide the training status? Additionally, how are the thresholds $\tau_\text{high}$, $\tau_\text{low}$, and $\tau_\Delta$ established?

- **Lines 248-249:** The method for determining the $\Delta$ term is not explained.

- **Section 3.2 (Theoretical Analysis in Appendix A):** This section appears to offer only speculative insights into how the algorithm might function, rather than providing concrete proofs.

- **Lines 325-326:** The statement "Larger parameter models can achieve higher self-improvement benefits" seems inconsistent, as Llama 13B appears to exhibit lower benefits compared to Llama 8B and Mistral 7B.

- **Section 4.1.2 (Fine-grained Analysis):** The paper does not clearly define what Loop0, Loop1, Loop2, and Loop3 training entail. What data are these loops trained on, and do they use the same subset of data? How is the transition to the next loop decided? Furthermore, why is the strategy of learning status 2 employed in Loop 3 not earlier?

**Meaningful Comparisons:** In Figure 4, the comparison using GPT-4 as a judge seems to employ an unconventional prompt for evaluating model responses (Appendix D.4). Why not use the AlpacaEval or MT-bench LLM-as-a-judge prompts, which have been shown to correlate well with human judgments?

**Implications:** The paper currently focuses on improving reward models that are not well trained (i.e., trained on only a small subset of the complete human-annotated dataset). Ultimately, the goal is to continuously enhance strong reward models with minimal human annotation. It is unclear whether this approach can be applied to strong reward models and whether these models have the potential for continuous self-improvement. Exploring this line of experimentation would be valuable.

questions:
See questions in the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731233187310

thread_type:
official_review

<thread_object>
creator: reviewer_8p3z

<content>
summary:
The manuscript introduces an approach to improving sample-efficiency of RLHF. The method enables using only a fraction of the original data through an iterative approach of labelling and ranking of the preference data based on the dynamics of the training process. The authors provide theoretical support of their empirical results, as well as extensive experiments (multiple models, multiple datasets).

strengths:
The manuscript, in my opinion, has several strengths that make it interesting to the ICLR audience.

* Tackles an important problem, namely of sample efficiency, for RLHF methods. The cost of human annotations is rather high and therefore studying ways to reduce the dependency on the entire datasets is useful overall.

* The recipe is well presented and supported theoretically. Although the steps seem obvious at first, the subtlety of amplifying the differences (second step) during the training loop seems rather novel and not extensively studied before. Effectively, it introduces the ability for the model to provide signal as to what samples provide most utility at current training stage.

* The extensive results, which utilize multiple models and multiple datasets. The performance across all these combinations of setups seems to support that the recipe is effective and general.

weaknesses:
The main weakness in current practices is the longer-term importance of annotated human datasets. The recipe introduces significant compute costs, trading off annotation costs for accelerators (through the recipe steps). However, the manuscript does not study the relationship between the two. If we assume human-annotated data goes away for preference labelling - having datasets created by LLMs for training other LLMs - do the benefits of the recipe still outweigh the costs?

I would like to understand the authors' position on this aspect, as well as a better comparison between labeling costs and accelerator costs of their recipe. Where does a tie come in (e.g. at the K=4 loops used in the paper, or for what value of K)?

questions:
Please refer to the weakness comment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732795724977

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
We sincerely thank all reviewers [R1 (8P3z), R2 (vMDv), R3 (U5TR), R4 (eZmA)] for their thoughtful and constructive feedback. We are delighted that the reviewers recognized the importance of the problem addressed [R1, R2], appreciated the well-supported motivation [R1, R2, R3], and found our experiments comprehensive with promising results [R1, R2, R3, R4].

Based on the reviewers' feedback, we have carefully revised the manuscript and uploaded a new version for review. All changes are highlighted in blue. Below is a summary of the key updates:

- **Section 3.1**: We provided a clearer explanation of Equation 3 and elaborated on the rationale behind its design [R2, R3, R4].
- **Appendix A**: A more rigorous theoretical analysis and proof have been added to strengthen the technical rigor of the paper [R2].
- **Appendix B**: The pseudo-code has been updated to better clarify the training process, incorporating feedback to address potential misunderstandings [R3, R4].
- **Appendix C.3**: The hyper-parameter tuning process has been updated to enhance the clarity [R2,R4].
- **Appendix E**: The cost of SER and the human-labeled methods has been updated, addressing the reviewers' requests for more detailed comparisons [R1]. 
- **Section 4.1.1**: Sentences were revised to improve clarity and avoid confusion, aligning with reviewers’ suggestions [R2, R3].
- Typos and minor errors throughout the manuscript have been corrected.

We deeply appreciate the reviewers’ insights, which have significantly enhanced the quality and presentation of our work. Thank you again for your thoughtful and valuable feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a new method to have a reward model to generate more data to "cover for its weaknesses". This is reminiscent of much work done in effective RL exploration but from the perspective of improving a learned human proxy reward instead of a policy. The idea is sound and well situated with the existing literature.

Many reviewer's raised questions initially about the validity of the claims made in light of the fact that only preference accuracy was reported in the evals which is rather non standard. After rather extensive discussions with the reviewers, the authors appear to have convinced a majority of them (and me) that the paper is suitable for publication by providing additional results on more indicative experimental settings. I'd highly encourage the authors to make the changes the reviewers especially those regarding clarity that vMDv and U5TR suggest and to include all of these results in the paper itself.