SELF-EVOLVED REWARD LEARNING FOR LLMS

## Abstract

Abstract Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning language models with human preferences, playing a pivotal role in the success of conversational models like GPT-4, ChatGPT, and Llama 2. A core challenge in employing RLHF lies in training a reliable reward model (RM), which relies on high-quality labels typically provided by human experts or advanced AI system. These methods can be costly and may introduce biases that affect the language model’s responses. As language models improve, human input may become less effective in further enhancing their performance. In this paper, we propose Self-Evolved Reward Learning (SER), a novel approach where the RM generates additional training data to iteratively improve itself. We conducted extensive experiments on multiple datasets such as HH-RLHF and UltraFeedback, using models like Mistral and Llama 3, and compare SER against various baselines. Our results demonstrate that even with limited human-annotated data, learning from self-feedback can robustly enhance RM performance, thereby boosting the capabilities of large language models (LLMs).

## 1 Introduction

Reinforcement Learning from Human Feedback (RLHF) is a well-established approach that aligns Large Language Models (LLMs) with human preference data Ouyang et al. (); Bai et al. (). The standard approach involves learning a reward model (RM) from human preferences and the learned RM is then frozen to train LLMs via Reinforcement Learning (RL) such as Proximal Policy Optimization (PPO) Schulman et al. (). Another common approach directly trains LLMs from the human preference data without learning an RM such as Direct Preference Optimiztion (DPO) Rafailov et al. (). Both approaches rely heavily on the size and quality of human-annotated preference data. However, the availability of such data is often limited and expensive to acquire, posing a significant bottleneck in the development and performance of RL approaches Yuan et al. (). This dependency on human-annotated data hinders the scalability of strong LLMs that require vast amounts of labeled data to achieve greater performance Kaplan et al. (); Muennighoff et al. (). To mitigate the dependency, recent works leverage the AI feedback to train RMs, referred to as Reinforcement Learning from AI Feedback (RLAIF) Bai et al. (); Lee et al. () , which reduces the reliance on human-annotated data. However, they hold heuristic assumptions that LLMs can provide high-quality feedback and they often requires stronger LLMs to provide feedback Pang et al. ().

Recent advancements suggest that LLMs have the potential to serve as world models to a certain degree, capable of understanding world knowledge and complex patterns independently of explicit human input Hao et al. (); Guan et al. (); Zhao et al. (). Leveraging this ability, LLMs can evaluate and provide feedback. In the context of RLHF and RLAIF, this capability of LLMs can be extended as the role of RMs, and RL approaches rely heavily on the RMs Dewey (); Li (). Focusing on training a better RM with limited human-annotated data, we propose a novel reward learning approach, which self-evolves the RM through a feedback loop using the RM itself. In our approach, the LLM serves as the RM, generating feedback on the dataset that is subsequently used to refine its own learning. This iterative ”feedback-then-train” loop allows the RM to self-evolve over time, gradually improving its performance, even with some noise in the initial self-labeled data. As the iteration progresses, however, similar data offers diminishing help and can even degrade performance. To address this, we identify the RM learning status in each iteration and introduce data filtering strategies to select high-confidence data that are later used for a more robust RM training.

Figure: Figure 1: The Self-Evolved Reward Learning (SER) pipeline. Our SER method consists of following steps: (1) Self-labeling: the reward model (RM) assigns labels to unlabeled data. (2) Identifying learning status and selecting data: high-confidence data is selected by assessing the learning status. (3) Retrain the RM: the RM trains itself using the self-labeled and selected data. (4) Train the Large Language Model (LLM): the LLM is trained under the guidance of the self-evolved RM. Note that steps (1)-(3) iterate multiple rounds to a converged RM.
Refer to caption: x1.png

By employing this self-evolved reward learning process, where the RM continually learns from its own feedback, we reduce dependency on large human-labeled data while maintaining, or even improving, the model’s performance. Our contributions are threefold:

- •
We introduce a novel self-evolved reward learning framework, demonstrating that only 15% of human-annotated seed data is required to achieve performance comparable to models trained with full human-labeled datasets, significantly reducing reliance on human data.
- •
We provide insights into the broader implications of self-learning paradigms in LLMs, particularly in improving reinforcement learning by enhancing RMs (see Section 4.1.2).
- •
Extensive experiments demonstrate that our self-evolved reward learning framework consistently improves performance across various LLMs, model sizes, and datasets.

We conducted experiments on multiple datasets and LLMs with their varied sizes to validate the generalization and effectiveness of our method. We find that, compared to the seed models that use only a small amount of human-labeled data, our method can robustly and significantly enhance model performance, with an average improvement of 7.88%. After multiple iterations, the final convergence can achieve or even surpass the performance of models using the entire human-annotated dataset, providing a potential solution for the self-improvement of models.

## 2 Related Work

### 2.1 Reinforcement Learning from External Feedback

Preference learning or now commonly referred to as reinforcement learning from human feedback (RLHF) Christiano et al. (); Ziegler et al. (); Stiennon et al. (); Ouyang et al. (); Bai et al. () train a fixed reward model (RM) from human preference data, and the trained RM is then used to train the Large Language Model (LLM) via RL, such as Proximal Policy Optimization (PPO) Schulman et al. (). In order to make RL training more stable and efficient, methods such as Direct Preference Optimization (DPO) Rafailov et al. () directly train the LLM using human preferences without training the RM. Other methods Zhao et al. (); Gulcehre et al. (); Yuan et al. () adjust the preference training schemes to improve the performance and stability. However, obtaining human preference data, especially high-quality data, is extremely expensive and time-consuming Köpf et al. (); Xu et al. (); Sun et al. (), and the data diversity is skewed to be low, containing few expert-annotated data which requires huge effort and expertise Peng et al. (); Zhang et al. (); Xu et al. (). The data quality and size sets the bottleneck of the performance of LLMs. Reinforcement Learning from AI Feedback (RLAIF) Bai et al. (); Lee et al. () employs LLMs to generate feedback for training RMs, reducing reliance on human-annotated data. However, it relies on the heuristic assumption that LLMs can provide high-quality and diverse feedback and often requires stronger LLMs to provide feedback Pang et al. (). In this paper, we leverage a small percentage of the human-annotated data to train an RM which achieves a comparable performance with the one trained with the full annotated data. The RM is further used in PPO to train the LLM.

### 2.2 Self-learning in LLMs

As the LLMs are developing towards superhuman-level, which may be bottlenecked by human performance level. Similar to the self-improvement in human reflection, self-learning is a new approch in improving LLM performance recently. Self-learning in LLMs focuses on enhancing capabilities without external supervision. SELF-ALIGN Sun et al. () demonstrates self-alignment through principle-driven reasoning, allowing models to adjust their outputs based on internal guidelines. ReSTEM Singh et al. () employs self-training to enhance problem-solving abilities. RLC Pang et al. () and SCoRe Kumar et al. () showcase methods for self-correction and improvement using self-generated data. Additionally, Huang et al. () illustrates how LLMs can refine reasoning through self-generated rationale-augmented answers, enhancing their explanatory depth. Math-Shepherd Wang et al. () and Self-Rewarding Language Models Yuan et al. () demonstrate self-rewarding mechanisms, where the model has the ability to provide high-quality rewards to itself. Our proposed approach falls in this self-learning paradigm by innovatively using the RM to generate feedback for itself, fostering robust RM training and improvement.

## 3 Self-Evolved Reward Learning for Large Language Models

In this section, we present our proposed Self-Evolved Reward Learning (SER) for LLMs. This approach enables the RM to iteratively improve itself by learning from its own high-confidence predictions, thereby reducing the need for extensive human-annotated data. Initially, the RM is trained with a small set of human-annotated data to provide a basic understanding of good and bad answers. From there, the RM evolves through self-labeling and iterative retraining. The enhanced RM is then employed to guide the LLM training via RL approaches. We detail each component of our method in the below section, including self-labeling, identifying learning status, data filtering, RM retraining and the LLM training via RL with improved and converged RM.

### 3.1 Overview

Figure 1 illustrates the overall pipeline of our SER method. This iterative process ensures that both the reward model and the LLM are continuously refined throughout the training cycle. Our method for Reward Model training consists of the following three iterative steps:

- 1.
Self-Label with Reward Model: The RM is initially trained with a small set of human-annotated data as a warm-up stage, then the RM performs self-labeling on the unlabeled data.
- 2.
Identify the Learning Status of the Reward Model and Select High-Confidence Data:
Evaluate the RM’s current ability to differentiate between good and bad answers or to amplify differences between similar answers. This status assessment guides the selection of high-confidence data.
- 3.
Retrain the Reward Model with Pairwise Loss:
After filtering, the selected high-confidence data are used to retrain the RM with pairwise loss, iteratively enhancing its understanding of answer quality.

With a few iterations of self-evolved reward learning, the RM training converges or meets the stopping criteria, such as when no further data can be filtered, the RM is then used to guide the training of the LLM via RL approaches. The modified PPO algorithm incorporates the evolved reward signals to optimize the LLM’s policy.

Our method relies on two distinct learning statuses for the RM: (1) the ability to distinguish between clearly good and bad answers, and (2) the ability to refine differences between answers of similar quality. These statuses are separated for the following reasons:
(a) Targeted Skill Development: by recognizing different learning statuses, the RM can focus on specific skill sets. Initially, the model focuses on clear distinctions (e.g., good vs. bad answers), and as training progresses, it refines its comparative abilities with more subtle distinctions.
(b) Adaptive Data Filtering: the data filtering process is driven by the current learning status, allowing the model to train on the most relevant data. This adaptive approach ensures the model always works on improving the appropriate aspect of its performance.
(c) Improved Self-Evaluation: by continuously monitoring its learning status, the RM can determine when to shift from one learning focus to another. This dynamic approach fosters self-driven, curriculum-like learning.

Furthermore, by allowing the LLM to generate two answers for each question, the RM is provided with paired examples that are key to both learning statuses. This setup enriches the training data with diverse answer qualities, enabling the RM to improve its discrimination and comparative abilities. Once the RM becomes proficient at handling both tasks, it is well-equipped to guide the LLM during reinforcement learning.

### Step 1: Self-Label with Reward Model

As shown in Figure 1, we first predict a reward score for all unlabeled data based on the current reward model (RM). This is formally expressed as follows:

$$ $r_{i}=RM(Q_{i},A_{i})$ (1) $$

This reward score may contain substantial noise, depending on the performance of the current state of the RM. We use these reward scores to determine the current training status and data selection strategy. Initially, we employ a small amount of human-annotated data to obtain a seed RM. In this study, the seed RM is trained using 15% of the entire dataset.

### Step 2: Identify the Learning Status of the Reward Model and Select High-Confidence Data

Each question $Q_{i}$ in our self-labeled dataset has two possible answers, $A_{i}^{1}$ and $A_{i}^{2}$, which can exhibit various relationships. The RM must differentiate between the following scenarios:
One answer is clearly better than the other (e.g., $A_{i}^{1}$ is good, $A_{i}^{2}$ is bad, or vice versa), or both answers are good, but one is better (or both are bad, but one is worse).
The RM assigns probabilities $p_{i}^{1}$ and $p_{i}^{2}$ that represent the likelihood that $A_{i}^{1}$ and $A_{i}^{2}$ are “good”. The goal is to distinguish the relative quality of the answers across these different cases.

Let $D_{\text{train}}=\{(Q_{i},A_{i}^{1},A_{i}^{2})\}_{i=1}^{N}$ be the training dataset. The learning status $\mathcal{S}$ is determined by the predicted probability differences between $A_{i}^{1}$ and $A_{i}^{2}$:

$$ $\Delta_{p}=\frac{1}{N}\sum_{i=1}^{N}|p_{i}^{1}-p_{i}^{2}|,$ (2) $$

We define the learning status $\mathcal{S}$ using thresholds $\tau_{\text{low}}$, $\tau_{\text{high}}$, and $\tau_{\Delta}$:

$$ $\mathcal{S}=\begin{cases}\text{Status}_{1},&\text{if }(p_{i}^{1}>\tau_{\text{ high}}\text{ and }p_{i}^{2}<\tau_{\text{low}})\text{ or }(p_{i}^{1}<\tau_{ \text{low}}\text{ and }p_{i}^{2}>\tau_{\text{high}}),\\ \text{Status}_{2},&\text{if }\Delta_{p}\geq\tau_{\Delta},\\ \text{Stop},&\text{otherwise}.\end{cases}$ (3) $$

In equation 3, Status 1 means the model can confidently distinguish between two answers when one has a high probability of being good ($p_{i}^{1}>\tau_{\text{high}}$) and the other a low probability ($p_{i}^{2}<\tau_{\text{low}}$), or vice versa. Status 2 means the model refines its judgment when the probability difference $\Delta_{p}$ exceeds $\tau_{\Delta}$, useful when both answers are either good or bad but differ in relative quality. Stop: no further self-training if neither condition is met, signaling the model has reached its capacity on the current data.
This approach allows the model to first learn clear distinctions between good and bad answers (Status 1) before focusing on more subtle differences (Status 2). When improvements from Status 1 are marginal, we shift to Status 2 to refine the model’s ability to handle nuanced cases.

Why capturing different statuses? The different statuses reflect the complexity of the relationships between $A_{i}^{1}$ and $A_{i}^{2}$. The reward model needs to manage both clear distinctions and subtler comparisons. By identifying the model’s learning status $\mathcal{S}$, we ensure that the model trains on the most relevant data for its current focus—strengthening its ability to handle clear distinctions or refining its comparative abilities.
This staged approach maximizes the model’s learning efficiency, first focusing on easier tasks (Status 1) and then moving on to more complex judgments (Status 2). Experiments validate that alternating between these statuses leads to improved performance overall.

### Step 3: Retrain the Reward Model with Filtered Data Using Pairwise Loss

Based on the state of the RM determined in Step 2, we select different data filtering strategies as outlined below:

$$ $\mathcal{F}(D_{\text{unlabeled}},\mathcal{S})=\begin{cases}\{(Q_{j},A_{j}^{1}, A_{j}^{2})\mid(RM(Q_{j},A_{j}^{1})>\tau_{\text{high}}\text{ and }RM(Q_{j},A_{j }^{2})<\tau_{\text{low}})\text{ or }\\ \qquad\qquad\qquad\qquad(RM(Q_{j},A_{j}^{1})<\tau_{\text{low}}\text{ and }RM(Q _{j},A_{j}^{2})>\tau_{\text{high}}),&\text{if }\mathcal{S}=\text{Status}_{1}, \\ \{(Q_{j},A_{j}^{1},A_{j}^{2})\mid|RM(Q_{j},A_{j}^{1})-RM(Q_{j},A_{j}^{2})|> \delta\},&\text{if }\mathcal{S}=\text{Status}_{2},\\ \emptyset,&\text{if }\mathcal{S}=\text{Stop}.\end{cases}$ (4) $$

Here, $D_{\text{unlabeled}}$ refers to the unlabeled data. In Status 1, the filter selects high-confidence data where $(RM(Q_{j},A_{j}^{1})>\tau_{\text{high}}\text{ and }RM(Q_{j},A_{j}^{2})<\tau_{
\text{low}})$ or $(RM(Q_{j},A_{j}^{1})<\tau_{\text{low}}\text{ and }RM(Q_{j},A_{j}^{2})>\tau_{
\text{high}})$, ensuring the model trains on reliable examples. In Status 2, the filter selects pairs where the reward difference $|RM(Q_{j},A_{j}^{1})-RM(Q_{j},A_{j}^{2})|$ exceeds a threshold $\delta$, focusing on refining comparative judgments. After filtering, the model is retrained using pairwise loss, allowing the model to compare answers relatively rather than relying on absolute labels. which consistently improves performance by focusing on relative comparisons rather than absolute classifications. The pairwise loss function is:

$$ $\mathcal{L}_{\text{pair}}=\frac{1}{|D_{\text{filtered}}|}\sum_{(Q_{j},A_{j}^{1 },A_{j}^{2})\in D_{\text{filtered}}}\max(0,\Delta-(RM(Q_{j},A_{j}^{1})-RM(Q_{j },A_{j}^{2}))),$ (5) $$

where $\Delta$ is the desired margin between reward scores. This iterative process, i.e., filtering data and retraining with pairwise loss, enables the RM to progressively refine its judgment until it converges.

### Step 4: Train the LLM via RL with Self-Evolved Reward Model

After self-evolving the RM, we use it to guide the training of the LLM via RL. To accommodate the refined reward signals from RM, we modify the PPO framework.
LLM training is framed as a policy optimization problem. The policy $\pi_{\phi}$ generates responses $A$ for inputs $Q$, and the objective is to optimize $\pi_{\phi}$ to maximize the rewards generated by the self-evolved RM: $r=RM(Q,A)$.
We maximize the expected reward from the self-evolved RM:
$\max_{\phi}\ \mathbb{E}_{Q\sim D_{\text{train}},A\sim\pi_{\phi}(\cdot\mid Q)}[
RM(Q,A)]$.

Using PPO (Schulman et al., ), we modify the policy updates to incorporate the refined reward signals from $R_{\theta}$, which better capture subtle differences in response quality. The policy is updated by maximizing the clipped surrogate objective:

$$ $\mathcal{L}_{\text{PPO}}=\mathbb{E}\left[\min\left(\frac{\pi_{\phi}(A\mid Q)}{ \pi_{\phi_{\text{old}}}(A\mid Q)}A^{\text{R}},\ \text{clip}\left(\frac{\pi_{ \phi}(A\mid Q)}{\pi_{\phi_{\text{old}}}(A\mid Q)},1-\epsilon,1+\epsilon\right) A^{\text{R}}\right)\right]$ (6) $$

Here, $A^{\text{R}}$ is the advantage function based on the rewards from RM. By leveraging the evolved reward model’s nuanced signals, the LLM’s policy updates align better with subtle distinctions in response quality. The detailed algorithm framework of our SER approach is provided in the Appendix B.

### 3.2 Theoretical Analysis

In this section, we analyze the theoretical feasibility of SER, focusing on the convergence properties of both RM training and PPO training. A detailed theoretical analysis supporting the effectiveness of our SER method is presented in Appendix A. The key conclusions of this analysis are as follows: (a) Convergence of the Reward Model: we demonstrate that, under reasonable assumptions, the RM iteratively improves by selecting high-confidence data based on predicted probabilities. This process ensures that its performance either improves or remains stable over time. (b) Convergence of PPO with a Learned Reward Model: we establish that PPO converges to a near-optimal policy even when the RM is trained through self-labeling, provided that reward estimation errors are small. These findings confirm that both the reward model and the LLM are capable of achieving high performance with minimal human supervision.

## 4 Experiment

In this section, we report our main experiment results, including reward modeling results and PPO results. We select multiple base models with different parameter sizes (Llama 3 8B (Dubey et al., ), Llama 2 13B (Touvron et al., ), Llama 3 70B (Dubey et al., ), Mistral 7B (Jiang et al., )) and conduct experiments on various datasets (StackOverflow (Lambert et al., ), HH-RLHF (Bai et al., ), UltraFeedback (Cui et al., ), Summarize (Stiennon et al., )) to verify the effectiveness of the method.
The experimental setup, statistical of datasets, evaluation metrics, and baselines are provided in the appendix C.

### 4.1 Reward Modeling Results

Our main experimental results are shown in Table 1. In all experimental setups, SER improves the model’s performance, ultimately achieving results close to those obtained using the full labeled dataset. In some experimental settings, it even exceeds the performance of models trained with the full human-labeled data, while using only 15% of the labeled data. This demonstrates the substantial potential of SER to enhance model performance in data-scarce scenarios.

**Table 1: The results of reward modeling on the HH-RLHF, Ultrafeedback, Summarize, and Stackoverflow. Loop 0 denotes the RM trained with 15% of the human data. Our method represents the results of iterative evolution based on the Loop 0 model. Full dataset denotes the results of the RM trained with the entire set of human-annotated data.**
|  | HH-RLHF | Summarize |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Llama3-8b | Mistral-7b | Llama2-13b | Llama3-8b | Mistral-7b | Llama2-13b |
| Loop 0 | 56.9 | 56.01 | 59.47 | 58.01 | 55.84 | 63.2 |
| SER | 68.56 | 68.1 | 70.26 | 68.42 | 65.49 | 69.19 |
| Full Dataset | 70.45 | 67.97 | 71.11 | 68.61 | 63.56 | 71.3 |
|  | Ultrafeedback | Stackoverflow |  |  |  |  |
|  | Llama3-8b | Mistral-7b | Llama2-13b | Llama3-8b | Mistral-7b | Llama2-13b |
| Loop 0 | 62.54 | 61.4 | 66.5 | 69 | 68.8 | 65.1 |
| SER | 74.46 | 70 | 72.69 | 70.8 | 70.1 | 69.2 |
| Full Dataset | 73.92 | 71.3 | 74.53 | 69 | 70.4 | 68.7 |

#### 4.1.1 Main Findings

SER consistently and effectively enhances model performance. As shown in Table 1, compared to the baseline that uses only 15% of the data, SER improves the model’s performance by incorporating self-labeled data for training. Through multiple iterations, the model achieves significant performance gains, resulting in an average 7.88% increase in accuracy. Furthermore, we find that as the model’s parameter size increases, the foundational capability strengthens, and the potential for SER’s self-improvement further enhances. Larger parameter models can achieve higher self-improvement benefits. Mistral 7B attains an average performance improvement of 7.74% during iterations, Llama 8B achieves an average improvement of 8.94%, and Llama 13B sees an average improvement of 6.76%. Across four datasets, Llama 8B surpasses Mistral 7B.

In data-rich scenarios (Stack Overflow), the performance gains from SER become smaller, averaging only 2.4%. In such data-rich contexts, a clear scaling trend with model parameter size is observed. The larger the model parameters, the greater the benefits of SER’s self-improvement. Mistral 7B achieves a performance improvement of 1.3%, Llama 8B achieves 1.8%, and Llama 13B achieves 4.1%.

SER can approach or even exceed the performance of full-scale human-labeled data. We compare our method with using the full human-labeled data. The results demonstrate that SER can achieve performance close to that of using the complete human-labeled dataset, with an average performance difference of 0.3%. For Mistral 7B on the HH-RLHF dataset, SER exceeds the baseline by 0.13%, and on the Summarize dataset, it surpasses the baseline by 1.93%. For LLaMA 8B on the UltraFeedback dataset, it achieves a performance advantage of 0.54% over the baseline. Additionally, a trend is observed where the disparity between the SER method and the full human-labeled data increases with the model parameter size. Specifically, Mistral 7B exceeds the baseline by an average of 0.12%, LLaMA 8B by 0.06%, while for LLaMA 13B, the baseline surpasses SER by an average of 1.07%. This indicates that as the model parameter size increases, its capacity to accommodate more labeled data also increases, thereby enhancing performance. This is a positive trend, suggesting that we can further enhance the performance ceiling of models by obtaining more labeled data through SER rather than manual labeling.

In data-rich scenarios, this trend becomes more pronounced. In the StackOverflow dataset, LLaMA 8B achieves performance very close to that of using the full dataset with only 15% of the human-label data. By employing SER, the model’s performance is further enhanced, surpassing the full human-labeled data by 1.8%. LLaMA 13B shows a 0.5% performance improvement compared to the full human-labeled data, while Mistral performs 0.3% lower than the baseline. This indicates that in cases of abundant data, the model’s self-evolved can lead to more diverse data distributions, thereby further raising the model’s performance ceiling.

Figure: Figure 2: Reward modeling improves in performance with iterative evolution. We demonstrate the performance variation of the model during the iterative process on the HH-RLHF, Ultrafeedback, and Summarize datasets. Baseline refers to the RM that uses the full dataset of human-annotated data. Due to the large size of the summarize test set, the results in the figure are based on a random sample of 1/10 of the test set.
Refer to caption: x2.png

#### 4.1.2 Fine-grained Analysis

Figure: Figure 3: The percentage of the total data used by the RM in each iteration is shown. (a)-(c) correspond to the HH-RLHF dataset, (d)-(f) correspond to the Ultrafeedback dataset, and (g)-(i) correspond to the Summarize dataset.
Refer to caption: x3.png

Figure: Figure 4: we use GPT-4 as a judge to evaluate the capabilities of the model trained with PPO. We employ the win rate as the evaluation metric. Left represents our SER method, SFT denotes the model fine-tuned with SFT, and Full refers to the PPO model guided by an RM trained on the full dataset.
Refer to caption: x4.png

In order to conduct a more detailed analysis of what occurs during the model’s iterations, we present the changes in the model’s accuracy on the validation set, as shown in Figure 2. Additionally, we illustrate the variations in the amount of training data across different iterations, as depicted in Figure 3. Our main conclusions are as follows:

The model can iteratively enhance its performance on self-labeled data, even if the self-labeled data contains noise. As shown in Figure 2, Loop1 consistently enhances the model’s performance in every experimental setup. During the Loop1 phase, the model’s performance is relatively weak, and there may be significant noise in the model’s self-feedback; therefore, it is essential to select high-confidence samples to improve the model’s performance(Status1). Generally, the performance improvement in Loop1 is the most significant among all loops, and it can filter out the largest number of training examples. As illustrated in Figure 2, on average, Loop1 provides a 4.54% enhancement to the model’s performance. This also verifies our Theory 1, which posits that when the model’s initial accuracy exceeds 50%, iterative training with high-confidence samples can further improve the model’s performance.

Similar data becomes marginally helpful after multiple iterations and may even harm the model’s performance. During the loop 2 phase, as the model’s capability increases, the benefits brought by the simple samples filtered out in state 1 become less significant. As shown in Figure 2, the performance improvement in loop 2 is the least significant across all iterations. This indicates that merely increasing the number of clearly defined samples offers limited performance enhancement for the model and may even lead to a decrease in model performance (as observed with llama 13b in the ultrafeedback dataset). By observing the training process, we should incorporate more ambiguous and difficult samples into the model’s training process, allowing the model to better discern the quality of two similar samples.

By adjusting the error reduction strategy, more diverse self-labeled data can be obtained, further enhancing the effectiveness of self-learning. Based on the analysis of the training process, to avoid overfitting the model on simple data, we need to focus the training objective on similar samples that are difficult to distinguish, enabling the model to identify differences between the two samples. During loop 3, we employ the data strategy of learning status 2, which enhances model performance by having the model learn to differentiate between more ambiguous hard samples. As illustrated in Figure 6, by modifying the data filtering strategy and introducing more diverse samples, the model in loop 3 increased the score differences between similar samples, thereby enhancing its discriminative ability.
As shown in Figure 2, in the loop 3 phase, training the model on hard samples further enhances its performance, approaching or even surpassing the results obtained using the full set of human-annotated data. Across multiple iterations, the total amount of training samples and the number of training steps are less than those required for the full dataset, typically representing only about 50% of the full data.

SER is more data and human-labor efficient than full fine-tuning. As shown in Figure 3, using the SER method, we utilize only 15% of the human-annotated data to train the initial model. Subsequently, we reselect the data based on the feedback from the initial model, achieving performance improvements. We conduct experiments on multiple datasets, demonstrating that SER effectively generalizes to various scenarios. Considering the high cost of human-annotated preference data, we provide a potentially effective solution to reduce this cost.

### 4.2 PPO Results

To validate the effectiveness of SER, we use the previously mentioned RM to guide PPO training, thereby optimizing the LLM. We conduct experiments on the Anthropic HH RLHF dataset and the Stackoverflow dataset, as shown in Figure 4. In the hh-rlhf dataset, all self RLHF models exceed the SFT baseline in terms of win rate, indicating that the self RLHF approach enhances the capabilities of the LLM. Compared to RMs trained with the full human-annotated data, the win rate in PPO experiments demonstrates a consistent trend with the performance of the RMs. For Mistral 7B, the accuracy of the self RLHF RM surpasses that of the RM trained with the full dataset, and in PPO experiments, the win rate also slightly exceeds that of the full model. Additionally, to verify the generalizability of our method, we conduct the same experiment on the StackOverflow dataset. As shown in Figure 4(b), the SER models outperform the full models to a certain extent, demonstrating a trend consistent with the accuracy of the RMs. In summary, our main findings are as follows:

SER enhances the capabilities of LLMs, and the degree of enhancement is positively correlated with the performance of the RMs. Through the self-evolved approach, we improve the performance of RMs using a limited amount of human-annotated data. Leveraging RMs to guide the learning of LLMs results in stronger LLMs. Theoretically, this process can be iterative, whereby stronger LLMs generate higher-quality responses, further enhancing the performance of RMs. However, due to the computational cost of PPO, we do not conduct related experiments. Additionally, we find that the performance gains in the PPO process are positively correlated with the performance of the RMs; stronger RMs generally guide the training of stronger LLMs.

## 5 Discussion

Our paper demonstrates empirical performance improvements through a self-evolved RM driven by intuitive motivations, though a rigorous theoretical analysis of its effectiveness is still needed. The data filtering strategies are empirical, yet it’s interesting that different datasets exhibit similar learning statuses in each iteration loop. Future work includes developing a more robust and autonomous method to identify learning statuses and filter self-labeled data. On the other hand, our method provides a feasible pathway to enhance reward modeling capabilities. An avenue worth exploring is generating more diverse responses through LLMs. By applying our method, a robust and general reward model can be developed to assist all existing feedback-based training methods. Additionally, integrating LLMs into the entire self-evolved reward learning loop is another future work, specifically by incorporating step 4 in each iteration and using LLMs to generate responses for the RM to perform self-labeling. Our work presents a potential solution to break through the performance ceiling of those strongest LLMs.

## 6 Conclusion

In this work, we introduce SER, a simple yet effective method of self-evolution that enhances model performance across various datasets and models. By allowing the model to generate its own labeled data and controlling the model’s learning state to select appropriate data, we achieve iterative evolution that ultimately converges to, or even exceeds, the performance ceiling. Extensive experiments indicate that our key design (consideration of different learning states) is essential, and we analyze the effects throughout the iterative process of SER, providing valuable insights for the self-improvement of LLMs.

## Appendix A THEORETICAL ANALYSIS

In this section, we provide a detailed theoretical analysis to support the effectiveness of our Self-Evolved Reward Learning (SER) method. The analysis focuses on two key aspects:

- 1.
Convergence of the Reward Model during Self-Training: We analyze conditions under which the RM improves through iterative self-training based on predicted probabilities.
- 2.
Convergence Properties of PPO with a Learned Reward Model: We discuss the theoretical properties of using PPO with an RM that has been trained using the self-evolved process.

### A.1 Convergence of the Reward Model during Self-Training

In our SER framework, the RM iteratively improves itself by filtering self-labeled data based on predicted probabilities and retraining using pairwise loss. The theoretical analysis focuses on the convergence of the RM’s predicted probabilities over time.

#### A.1.1 Assumptions

###### Assumption 1 (Initial Quality of Predicted Probabilities) .

The initial RM $R_{\theta}^{(0)}$ provides predictions where, for a pair of answers $(A_{i}^{1},A_{i}^{2})$, the probability difference $\Delta_{p}^{(0)}=|p_{i}^{1}-p_{i}^{2}|$ is meaningful, i.e., the probability difference reflects the relative quality of answers with reasonable accuracy.

###### Assumption 2 (High-Confidence Data Filtering) .

During self-training, only pairs of answers $(A_{i}^{1},A_{i}^{2})$ where the predicted probability difference exceeds a confidence threshold $\tau_{p}$ (e.g., $|p_{i}^{1}-p_{i}^{2}|>\tau_{p}$) are used for further training. This ensures that the model uses reliable self-labeled data.

###### Assumption 1 (Initial Quality of Predicted Probabilities) .

###### Assumption 2 (High-Confidence Data Filtering) .

#### A.1.2 Theoretical Result

###### Theorem 1 (Convergence of Reward Model during Self-Training) .

Under Assumptions 1 and 2, and with appropriate choices of confidence threshold $\tau_{p}$ and pairwise comparison margin $\delta$, the self-training process of the RM converges, and the predicted probabilities become increasingly reliable with each iteration, i.e., the RM improves over time.

###### Theorem 1 (Convergence of Reward Model during Self-Training) .

#### A.1.3 Proof Sketch

The key steps of the proof are as follows:

- 1.
Initial Quality of Predictions: From Assumption 1, the RM begins with predictions that, while not perfect, provide meaningful probability differences between answer pairs.
- 2.
High-Confidence Data Selection: By filtering the data based on the confidence threshold $\tau_{p}$ (Assumption 2), we ensure that only reliable, high-quality data points are used for retraining. This avoids the risk of propagating errors from uncertain self-labeled data.
- 3.
Iterative Improvement: Each iteration of the self-training process uses pairwise loss to refine the RM’s ability to distinguish between answer pairs, resulting in improved predicted probabilities with each training step.
- 4.
Convergence: As the self-training process continues, the RM’s predicted probabilities become more accurate, converging to a stable state where further self-training leads to diminishing returns in performance improvement.

#### A.1.4 Discussion

Theorem 1 shows that, under reasonable conditions, the self-training process for the RM leads to improved performance over time. The key is to carefully filter the data based on predicted probabilities to ensure that only reliable self-labeled examples are used for training. This filtering mechanism prevents error propagation and ensures that the RM improves iteratively.

### A.2 Convergence Properties of PPO with a Learned Reward Model

Proximal Policy Optimization (PPO) is used to train the LLM with reward signals generated by the self-evolved RM. The theoretical challenge lies in ensuring that PPO can converge effectively when using an RM that has been trained through self-labeling, which may introduce some inaccuracies in the reward estimates.

#### A.2.1 Assumptions

###### Assumption 3 (Bounded Reward Estimation Error) .

The RM provides reward signals where the error relative to the true reward function $R^{*}(Q,A)$ is bounded, i.e., $|RM(Q,A)-R^{*}(Q,A)|\leq\epsilon_{r}$, where $\epsilon_{r}$ is small.

###### Assumption 3 (Bounded Reward Estimation Error) .

#### A.2.2 Theoretical Result

###### Theorem 2 (Policy Improvement with Approximate Rewards) .

Under Assumption 3 and using PPO with appropriately chosen hyperparameters, the LLM’s policy $\pi_{\phi}$ converges to a near-optimal policy with respect to the true reward function $R^{*}$, with the error bounded by $\epsilon_{r}$.

###### Theorem 2 (Policy Improvement with Approximate Rewards) .

#### A.2.3 Proof Sketch

The proof proceeds as follows:

- 1.
Consistency of Policy Gradient: The policy gradient computed using the learned RM is close to the true policy gradient computed using $R^{*}$, with the error bounded by $\epsilon_{r}$.
- 2.
Effect of Reward Estimation Error: The bias introduced by the reward estimation error affects the gradient update but remains small, as long as $\epsilon_{r}$ is small.
- 3.
PPO Stability: PPO’s clipping mechanism prevents large updates, ensuring that policy updates remain stable even in the presence of small reward estimation errors.
- 4.
Convergence: By iteratively applying PPO updates, the policy converges to a near-optimal policy relative to the true reward function, with the error in the final policy bounded by $\epsilon_{r}$.

#### A.2.4 Discussion

Theorem 2 shows that PPO remains effective even when using a learned RM, provided the reward estimation errors are small. Our self-evolved RM minimizes $\epsilon_{r}$ over time by iteratively refining its predicted probabilities, ensuring that the reward signals used for policy optimization become more accurate. As a result, PPO can still guide the LLM toward a near-optimal policy.

## Appendix B Algorithm Summary

Algorithm 1 summarizes the entire SSRL-LLM method, including iterative self-evolved RM training and reinforcement learning for LLM policy optimization.

Figure: Algorithm 1 Self-Evolved Reward Learning for LLMs (SER)

## Appendix C Experimental Setup

SFT training: For each Base Model, we perform instruction fine-tuning using preference data. Similar to the setting by Rafailov et al. (), we sample higher quality responses from the preference data based on human annotations to use as training data (for instance, in the HH-RLHF dataset, we sample responses labeled as ‘chosen’ for instruction fine-tuning). We conduct standard instruction fine-tuning training on the base model using the sampled data. In our experiments, we refer to this as our SFT baseline.

Reward Model training: We perform reward modeling on the SFT baseline using preference data. In our method, we train an initial model with a small amount of human-annotated preference data (in our experiments, this constitutes 15% of the overall dataset size; details on the dataset split for SFT, PPO, etc., can be found in the appendix D.1). The initial model then assigns reward scores to unannotated responses, and based on these reward scores, we filter and obtain new training data for the next iteration of training.

### C.1 Dataset statistics

Our experiments explore four different preference datasets as show in Table 2. StackOverflow contains over 3,000K QA pairs collected from StackOverflow. Each question receives a score based on the number of upvotes, resulting in a comparison pair. HH-RLHF: we use human preference data, which consists of 118K helpful and 42K harmless instances as the training set. Similar to previous work, we select the last round of dialogues to construct the data into a single-turn dialogue format. UltraFeedback is constructed by large language models (LLMs). It collects 64K instructions from various sources, generates 256K responses using LLMs such as LLaMA, and has these responses annotated and scored by GPT4. From this process, we create a preference dataset containing 100K entries. TL;DR consists of 179K pairs of summarization and human preference annotations.

**Table 2: The statistics of datasets, types of tasks, and types of feedback are presented. We provide a detailed introduction to the datasets in the appendix C.1.**
| Dataset | Num | Task | Feedback type |
| --- | --- | --- | --- |
| Stackoverflow | 31,284,837 | QA | human |
| HH-RLHF | 169,352 | QA | human |
| UrtralFeedback | 63967 | QA | GPT4 |
| Summarize | 179000 | summarize | human |

### C.2 Evaluation Metrics and Baseline

Reward Modeling. The standard process of RLHF involves training an RM based on preference data to predict the preferences between human and model responses. Subsequently, reinforcement learning methods are used to optimize the language model based on the RM. Accuracy: We use accuracy to measure the performance of reward modeling. Specifically, for a given preference data, if the reward value assigned by the RM to the chosen response is higher than that to the rejected response, the prediction is considered correct. Baseline: considering that our method uses only a portion of the human-annotated data, we choose to use a model trained on the full dataset for reward modeling as a baseline.

PPO. For the RM obtained in the previous step, we apply it to the standard PPO process to optimize the LLM. We use LLM as a judge evaluate the performance of the model after PPO optimization. Specifically, we use GPT-4 as the evaluator to compare different responses to the same prompt. GPT-4 assesses the quality of the responses. We conduct the comparison in two different orders and if the results from these two orders are inconsistent, we consider the results as a tie. Baseline: We compare SER with the SFT baseline to intuitively demonstrate the improvement of our method in aligning model preferences. Additionally, we compare our approach with an RM trained using the full preference dataset, despite the fact that our method uses significantly less data.

### C.3 Training Details

SFT training. We use the following hyperparameters for instruction fine-tuning training. We employ a learning rate of 2e-5 with cosine decay, 2 warmup steps, and a batch size of 16. We calculate the loss only for the target tokens rather than the full input sequence, and we train for 3 epochs on the training data. For smaller parameter models (e.g., llama 8B, Mistral 7B, llama 13B), we conduct the training on 8 NVIDIA A100 80G GPUs. For the llama 70B model, we perform the training on 16 NVIDIA A100 80G GPUs.

Reward training. To enable the model to learn the relative ranking among different responses, we use a pair-wise loss. We employ the sigmoid function to normalize the reward scores to a range of 0-1. We utilize the LoRA method to train the RM on the SFT baseline, with a rank of 8, a LoRA alpha of 32, and a LoRA dropout of 0.1. The task type is sequence classification. We use a learning rate of 2e-5 with linear decay and the AdamW optimizer for training over 2 epochs, with a batch size of 4 (batch size of 2 for the LLaMA 70B model). We conduct the training on 8 NVIDIA A100 80G GPUs (32 NVIDIA A100 GPUs for the LLaMA 70B model).

PPO training. For PPO training, we use a learning rate of 1.4e-5 and set the generate sample length to 256. We employ a batch size of 8 and a mini-batch size of 1, with 4 PPO epochs and 1 gradient accumulation step. The target KL divergence is set to 0.1 and initial KL coefficient is set to 0.2. To ensure a more robust training process, we normalize the range of reward values to -0.5 to 0.5.

## Appendix D Implementation Details

### D.1 The split of the dataset

For the preference dataset, we split the training and testing sets according to the ratio of SFT:RM:PPO = 0.3:0.65:0.05. In this paper, SFT utilizes the chosen responses from the preference data for instruction fine-tuning. For the training of Reward Modeling, our approach randomly samples 15% of the RM data for training, while comprehensive comparison experiments train on the entire RM dataset. For the HH-RLHF dataset, it is divided into harmful and helpful subsets, and we only select the helpful subset.

### D.2 Statistical data of the iterative process

We quantify the amount of data filtered out during the iterative process, as shown in Figure 3. Loop 0 represents a fixed value, accounting for 15% of the overall dataset. We use this portion of the data to train the seed model, upon which all subsequent iterations are based for further evolution.

### D.3 PPO learning curve

Figure: Figure 5: The learning curve of the model on the HH-RLHF dataset, with the y-axis representing the reward score after scaling. The model reaches convergence after 1200 steps. The shaded area indicates the standard deviation.
Refer to caption: x5.png

As shown in Figure 5, we present the reward curves of Mistral 7B and LLaMA 8B on the HH-RLHF dataset. Both models reach convergence at around 1200 steps. We scale the reward scores to the range of $-1$ to $1$ using the following formula:

$$ $\mathcal{S}_{\text{Scaled}}=\left(\frac{\left((1+e^{-\mathcal{S}_{\textbf{ Original}}})^{-1}-t_{\text{clip}}\right)\times(NewMax-NewMin)}{1-t_{\text{clip }}}\right)+NewMin$ (7) $$

In this equation, $S_{original}$ represents the original reward score, $t_{clip}$ denotes the clipping value, $NewMin$ is the minimum value after scaling, which is $-1$, and $NewMax$ is the minimum value after scaling, which is $1$.

### D.4 GPT4 evaluation Prompt

A crucial element of our experimental framework is the evaluation of win rates using GPT-4. In this section, we provide the prompts utilized to generate win rates for both the summarization and dialogue experiments. All experiments were conducted using the gpt-4o-20240806 model. The sequence of responses was randomized for each evaluation to ensure unbiased results.

GPT-4 as judge system prompt:

User input prompt:

**Table 3: A case study for GPT4 evaluation**
| Input | Human: I’m trying to teach myself to have nicer handwriting. Can you help? Assistant: |
| --- | --- |
| Mistral 7B SFT | Sure, I can help! I’m a human handwriting expert. |
| \hdashlineGPT4-as-Judge | Score: 1<br>Evaluation evidence: The response is relevant in acknowledging the user’s request for help with handwriting, but it provides no substantial information or guidance related to improving handwriting skills. It also inaccurately claims to be from a ḧuman handwriting expert,ẅhich is misleading coming from an AI Assistant. The answer lacks depth and does not address the user’s question in a meaningful way. |
| Mistral 7B Our method | Sure, I can help you to learn to have nice handwriting. To do this, you will need to practice writing and drawing a lot, so that your handwriting will improve. |
| \hdashlineGPT4-as-Judge | Score: 2<br>Evaluation evidence: The response is relevant and provides basic advice related to improving handwriting, such as the importance of practice. However, it lacks depth and detail, offering only a very general suggestion without specific techniques or steps to actually enhance handwriting skills. Thus, it addresses part of the user’s question but does not offer a comprehensive or directly useful guide. |
| Mistral 7B Full dataset | Sure, I think I can! First, I need you to write a few sentences in a beautiful, swirly style, and then I can take a look at your handwriting and make some suggestions. |
| \hdashlineGPT4-as-Judge | Score: 2<br>Evaluation evidence: The response is relevant and provides some engagement with the user’s request by suggesting an activity. However, it doesn’t directly address the user’s question or offer actionable advice on improving handwriting. It lacks comprehensive guidance or practical tips, failing to fully answer the question or provide useful information on handwriting improvement techniques. |

Figure: Figure 6: The reward score distribution of the model on HH-RLHF, with the y-axis representing probability density and the x-axis representing pairwise score differences. Compared to other loops, loop 3 significantly increased the score differences between responses of similar quality by altering the error reduction strategy.
Refer to caption: x6.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Please refer to the weakness comment.
soundness: 3
strengths: The manuscript, in my opinion, has several strengths that make it interesting to the ICLR audience.

* Tackles an important problem, namely of sample efficiency, for RLHF methods. The cost of human annotations is rather high and therefore studying ways to reduce the dependency on the entire datasets is useful overall.

* The recipe is well presented and supported theoretically. Although the steps seem obvious at first, the subtlety of amplifying the differences (second step) during the training loop seems rather novel and not extensively studied before. Effectively, it introduces the ability for the model to provide signal as to what samples provide most utility at current training stage.

* The extensive results, which utilize multiple models and multiple datasets. The performance across all these combinations of setups seems to support that the recipe is effective and general.
summary: The manuscript introduces an approach to improving sample-efficiency of RLHF. The method enables using only a fraction of the original data through an iterative approach of labelling and ranking of the preference data based on the dynamics of the training process. The authors provide theoretical support of their empirical results, as well as extensive experiments (multiple models, multiple datasets).
weaknesses: The main weakness in current practices is the longer-term importance of annotated human datasets. The recipe introduces significant compute costs, trading off annotation costs for accelerators (through the recipe steps). However, the manuscript does not study the relationship between the two. If we assume human-annotated data goes away for preference labelling - having datasets created by LLMs for training other LLMs - do the benefits of the recipe still outweigh the costs?

I would like to understand the authors' position on this aspect, as well as a better comparison between labeling costs and accelerator costs of their recipe. Where does a tie come in (e.g. at the K=4 loops used in the paper, or for what value of K)?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See questions in the weaknesses part.
soundness: 3
strengths: **Motivation:** Given the high costs associated with acquiring human-annotated data for reward model training, the development of self-training approaches is highly beneficial. Such methods can leverage a minimal amount of human annotations for initial model training and subsequently enhance performance through self-improvement.

**Soundness:** The paper's various training strategies, which are based on different statuses, are well-conceived and intuitively sound.

**Substance:** The authors have conducted extensive experiments across multiple models and datasets, effectively demonstrating the efficacy of their Self-Evolved Reward Learning (SER) approach.
summary: This paper introduces an approach called Self-Evolved Reward Learning (SER), which leverages an RM to autonomously curate its training data for continuous improvement. The authors conduct experiments across various models and datasets, demonstrating that the SER approach can train the reward model well using only 15% of the labeled data, which results in performance comparable to an RM trained on the full labeled dataset, both in terms of evaluation capability and integration within PPO training.
weaknesses: **Clarity:** The paper lacks clarity in several places, with some descriptions being particularly confusing. For instance:

- **Lines 206-211:** It remains unclear how the status is determined, given that $p_i^1$ and $p_i^2$ are both single numbers. How are these individual numbers used to decide the training status? Additionally, how are the thresholds $\tau_\text{high}$, $\tau_\text{low}$, and $\tau_\Delta$ established?

- **Lines 248-249:** The method for determining the $\Delta$ term is not explained.

- **Section 3.2 (Theoretical Analysis in Appendix A):** This section appears to offer only speculative insights into how the algorithm might function, rather than providing concrete proofs.

- **Lines 325-326:** The statement "Larger parameter models can achieve higher self-improvement benefits" seems inconsistent, as Llama 13B appears to exhibit lower benefits compared to Llama 8B and Mistral 7B.

- **Section 4.1.2 (Fine-grained Analysis):** The paper does not clearly define what Loop0, Loop1, Loop2, and Loop3 training entail. What data are these loops trained on, and do they use the same subset of data? How is the transition to the next loop decided? Furthermore, why is the strategy of learning status 2 employed in Loop 3 not earlier?

**Meaningful Comparisons:** In Figure 4, the comparison using GPT-4 as a judge seems to employ an unconventional prompt for evaluating model responses (Appendix D.4). Why not use the AlpacaEval or MT-bench LLM-as-a-judge prompts, which have been shown to correlate well with human judgments?

**Implications:** The paper currently focuses on improving reward models that are not well trained (i.e., trained on only a small subset of the complete human-annotated dataset). Ultimately, the goal is to continuously enhance strong reward models with minimal human annotation. It is unclear whether this approach can be applied to strong reward models and whether these models have the potential for continuous self-improvement. Exploring this line of experimentation would be valuable.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Are the pairs of responses generated from the same base LLM? Or are the authors using the response pairs in the existing dataset. While the author mentioned generating two responses from the LLM (line 173), it is not clear which model they use to generate responses and if they do this on-policy data generation for all the iterations? More details on this is appreciated. 
2. Line 174-175: how generating two responses on-policy from the LLM "enriches" the training data with diverse answer qualities? What aspect of this setup is unique or does it guarantee quality/diversity?
3. Intuitively, what does the RM learn from its highly confident labeled data during stage 1? In other words, when the RM is too confident about a preference label (irrespective of whether it may be incorrect), what training signals it is gaining from it? Did authors try some ablations to test the contribution of each data state?
4. How the number of training steps differ in SER vs Full dataset in Table 1?
5. One would expect to see the performance of your PPO'ed model on downstream standard benchmarks like AlpacaEval, MMLU, BBH, MATH, CODEX, etc. So technically, start from a pre-trained base model doing supervised finetuning on some existing instruction-tuning data , then try doing PPO using your RM vs the baseline RM (trained on full data, e.g. on UltraFeedback) and compare performance over standard benchmarks. This will more realistically show the effectiveness of your approach as opposed to just narrowly evaluating models in their ability to choose the preferred response.
6. Line 343: The scaling claims are somewhat unsupported, based on Table 1 13b> 7b> 8b. more experiments specifically within the same model family is needed to make informative scaling trend conclusions.
7. Section 4.1.2, there's no explanation as to what Loop 1-3 refers to in section 4.1.2 and figure 2. It is not clear if loop is referring to one iteration of RM training (including both state 1 and 2), or something else? Line 373 states that "loop 2 is the least significant across all iterations" which makes me think loop != iteration
8. Line 418: does that mean Loop 1 and Loop 2 did not include data strategy of status 2? In general more details on the training is nice in the main paper.
9. Section 4.2: it would make sense to try PPO on a more general purpose preference dataset like ultrafeedback and see how it transfers to downstream tasks and not just alpacaeval style evaluation
10. Suggestion: the authors refer to their models as SER, or self RLHF. It’s nice to stick to one name and be consistent with it.
soundness: 3
strengths: - Promising results on improving the RM performance matching or exceeding the baseline train on the full human labeled data
- The curriculum learning style for the RM which encourage learning from easy and then hard pairs seems an interesting finding in the context of RM optimization
summary: The paper proposed a self-improvement RM training strategy to circumvent the challenges of collecting human preference data. The RM is initially warmed up using only 15% human preference, then it’s improved iteratively through a series of self-labeling and retraining. During the self-labeling the current RM is used to score unlabeled data (i.e., prompt and answer pairs), then two data filtering strategies are devised based on the predicted probability differences between answer pairs. The goal is for the RM to (1) first learn from pairs with clear distinction (trained on very high- and very low-scored response pairs) before (2) switching to refining its understanding of subtle differences (trained on pairs that are more similar). When no improvement is seen on this current data, training stops and new data is collected. 

The author tested their approach on multiple datasets including HH-RLHF, UltraFeedback as well as several model sizes and families including Llama and Mistral. They also evaluate how the improved RM helps during PPO.
weaknesses: - The Experimental design is very narrow, and limited. (see questions for more details). E.g, I would expect to see results on downstream/standard benchmarks. However currently only preference accuracy is reported.
- Some claims in the results section are unsupported (mixed results which are generalized) → see detail below
- The paper clearly benefits from some rewriting and reorganization. In its current form several details are missing which makes it uneasy to follow.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Are Llama models base models or instruction-tuned models?
soundness: 1
strengths: The self-improvement paradigm could save lots of resources. 

I appreciate the fact that the authors not only tested on reward modeling benchmarks but also LMs (by doing PPO). 

Experiments are done on multiple models and multiple model sizes.
summary: This paper proposes self-evolved reward learning where the reward model is used to obtain more training data for iterative improvement. The algorithm is straightforward: (1) generate two responses per prompt and label using RM, (2) only keep response pairs where reward difference is large -- there are two filtering strategies (Status 1 and Status 2) and the authors alternate between them, (3) train RM with the filtered data; then, iterate the above. Finally, use RM for LM training. Findings: Only 15% human-annotated data is needed so that the new model archives performance similar to training on the entire human-annotation dataset; experiments are done on a few models (7B to 13B).
weaknesses: On baselines:
- It’s unclear how specifically the authors trained “Loop 0” and “Full dataset” baseline models. What is the loss function? How are hyperparameters tuned? 
- Have the authors tried regular scalar-valued reward functions as baselines? 

On eval: Have the authors tested on common benchmarks like RewardBench (so we can easily compare to other scalar and generative RMs)?

In real LLM training, the same dataset would always contain examples of multiple domains. In the same dataset, lots of examples could be on simple helpfulness prompts, and lots of examples are on coding/reasoning prompts. In this case, how would thresholding work? Would the model first learn simple prompts and then coding/reasoning/complicated prompts? Or would the model give up on coding/reasoning/complicated prompts?

Eq. (3) can be clearer:
- It looks like Status 1 and Status 2 can be achieved at the same time? In that case, does S=Status1 or Status2? 
- Do we have a different S at every i? 
- Do we need to recompute Delta_p (avg through every i) after each gradient step? 
- When would the model hit “Stop”? What if the “otherwise” case never happens? An explanation in natural language would be great.

Details are lacking on data generation/filtering
- How specifically are responses generated?
- What percentage of data are filtered at different point in training? 
- Are there more conclusive evidence that alternating between status are beneficial? How are hyperparameters tuned -- based on what metrics? 


Other misc issues: 
- Eq. (4): missing right bracket
- Table 1 caption mentions “Our method” but I don’t see “Our method” in the table. Do you mean SER?
- Serious misuse of \citet and \citep. Most of the citations on the first page and the second page, for example, should be in \citep.

———

UPDATE: increased score to 5 but still concerned about paragraph 3 (and hopefully more detailed result on paragraph 2) above.

Meta-review: The paper presents a new method to have a reward model to generate more data to "cover for its weaknesses". This is reminiscent of much work done in effective RL exploration but from the perspective of improving a learned human proxy reward instead of a policy. The idea is sound and well situated with the existing literature.

Many reviewer's raised questions initially about the validity of the claims made in light of the fact that only preference accuracy was reported in the evals which is rather non standard. After rather extensive discussions with the reviewers, the authors appear to have convinced a majority of them (and me) that the paper is suitable for publication by providing additional results on more indicative experimental settings. I'd highly encourage the authors to make the changes the reviewers especially those regarding clarity that vMDv and U5TR suggest and to include all of these results in the paper itself.

Decision: Accept (Poster)