SELF-EVOLVED REWARD LEARNING FOR LLMS
Authors: Chenghua Huang
Zhizhen Fan
Lu Wang
Fangkai Yang
Pu Zhao
Zeqi Lin
Qingwei Lin
Dongmei Zhang
Saravan Rajmohan
Qi Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: RLHF
LLM
Self-learning
Primary area: reinforcement learning

Abstract: Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning language models with human preferences and is a key factor in the success of modern conversational models like GPT-4, ChatGPT, and Llama 2. A significant challenge in employing RLHF lies in training a reliable RM, which relies on high-quality labels. Typically, these labels are provided by human experts or a stronger AI, both of which can be costly and introduce bias that may affect the language model's responses. As models improve, human input may become less effective in enhancing their performance. This paper explores the potential of using the RM itself to generate additional training data for a more robust RM. Our experiments demonstrate that reinforcement learning from self-feedback outperforms baseline approaches.
We conducted extensive experiments with our approach on multiple datasets, such as HH-RLHF and UltraFeedback, and models including Mistral and Llama 3, comparing it against various baselines. Our results indicate that, even with a limited amount of human-labeled data, learning from self-feedback can robustly enhance the performance of the RM, thereby improving the capabilities of large language models.

Published as a conference paper at ICLR 2025
SELF-EVOLVED REWARD LEARNING FOR LLMs
, Pu Zhao Zeqi Lin Qingwei Lin Dongmei Zhang* ,Saravan Rajmohan" ,Qi Zhang
Chenghua Huang" Zhizhen
Zhizhen Fan ; Wang"/ Fangkai Yang",
"School of Computer Science, Fudan University
School of Computer Science, Peking University
Microsoft
huangch222m. fudan. edu. en, 2201210191@stu.pkK edu, cn
{wlu, fangkai puzhao, zelin, dongmeiz, saravar, qi zh mmicroooottcco
ABSTRACT
Reinforcement Learning from Human Feedback (RLHF) crucial technique for aligning language models with human preferences, playing pivotal role in the success of conversational models like GPT-4, ChatGPT, and Llama 2. A core challenge in employing RLHF lies in training reliable reward model (RM), which relies on high-quality labels typically provided by human experts or advanced AI system. These methods can be costly and may introduce biases that affect the language model's responses. As language models improve, human input may become less effective in further enhancing their performance. In this paper, we propose Self-Evolved Reward Learning (SER), novel approach where the RM generates additional training data iteratively improve itself. We conducted extensive experiments on multiple datasets such as HH-RLHF and UltraFeedback, using models like Mistral and Llama 3, and compare SER against various baselines. Our results demonstrate that even with limited human- annotated data, learning from self-feedback can robustly enhance RM performance, thereby boosting the capabilities of large language models (LLMss. Resources of this paper can be found at |attp::: // aka. mms/sess.....
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) is well-established approach that aligns Large Language Models (LLMs) with human preference data Ouyang et al. (2022); Bai et al. (2022b). The standard approach involves learning a reward model (RM) from human preferences and the learned RM is then frozen to train LLMs via Reinforcement Learning (RL) such as Proximal Policy Optimization (PPO) Schulman al. (017a). Another common approach directly trains LLMs from the human preference data without learning an RM such as Direct Preference Optimiztion (DPO) Rafailov et al. 20244). Both approaches rely heavily on the size and quality of human-annotated preference data. However, the availability of such data is often limited and expensive to acquire, posing a significant bottleneck in the development and performance of RL approaches Yuan al. (2024b). This dependency on annootatd data hinders the scalability of strong LLMs that require vast amounts labeled data to achieve greater performance Kaftormanne//paaann et al. (2020); MMennighoff et al. (2024). To mitigate the dependency, recent works leverage the AT feedback train RMs, referred to as Reinforcement Learning from Al Feedback (RLAIF) Bai al.) (2022b): Lee al. (2023). which reduces the reliance on can provide high-quality feedback and they often requires stronger LLMs to provide feedback Pang
data. However, they hold heuristic assumptions that LLMs
al. 20033).
Recent advancements suggest that LLMs have the potential to serve as world models to a certain degree, capable of understanding world knowledge and complex patterns independently of explicit human input Нao et al. ): Guan et al. (2023); Zhao et al. (2024). Leveraging this ability, L.MSs evaluate and provide feedback. In the context of RLHF and RLAIF, this capability of LLMs
"work is done during an internship at Microsoft 'corresponding author

---

Published as a conference paper at ICLR 2025
can be extended the role of RMs, and RL approaches rely heavily on the RMs Dewey (0014) (2017). Focusing on training a better RM with limited human-annotated data, we propose novel reward learning approach, which self-evolves the RM through feedback loop using the RM itself. In our approach, the L.M serves as the RM, generating feedback on the dataset that is subsequently used to refine its learning. This iterative "feedback-then-traiin loop allows the RM self-evolve over time, gradually improving its performance, even with some noise in the initial self-labeled data. As the iteration progresses, however, similar data offers diminishing help and can even degrade performance. To address this, we identify the RM learning status in each iteration and introduce data filtering strategies select high- high-confidence data that are later used for more robust RM training.
Li
RM,
[Q A, A21
Unlabeled Data
(8) Distnguish
Amplty
Self-Label with Reward Model (RM)
Select High- Confi-eCnndeeeeeeeee Self-Labeled Data
Next Iteration Reward Model
LLM
RM++3
(Q.AA,
Selected Sul-LLabueed Data
Train LLM with RM via Reinforcement Learning
Retrain RM with Selected Setf-Labeled Data
Figure The Self-Evolved Reward Learning (SER) pipeline. Our SER method consists of
following steps: (1) Self-labeling: the reward model (RM) assigns labels to unlabeled data. (2) Identifying learning status and selecting data: high-confidence data is selected by assessing the learning status. (3) Retrain the RM: the RM trains itself using the self-labeled and selected data. (4)
Train the Large Language Model (LLM)): the LLM trained under the guidance of the self-evolved RMM Note that steps (1)-(3 iterate multiple rounds converged RMM.
By employing this self-evolved reward learning process, where the RM continually learns from its own feedback, we reduce dependency on large human-labeled data while maintaining, or even improving, the model's performance. Our contributions are threefold:
We introduce novel self-evolved reward learning framework, demonstrating that only 15% luman-annotated seed data required to achieve performance comparable to models trained with full human-labeled datasets, significantly reducing reliance on human data.
We provide insights into the broader implications of self-learning paradigms in LLMs, particularly in improving reinforcement learning by enhancing RMs (see Section 4.1.2
Extensive experiments demonstrate that our self-evolved reward learning framework consis- tently improves performance across various LLMs, model sizes, and datasets.
We conducted experiments on multiple datasets and LLMs with their varied sizes to validate the generalization and effectiveness of our method. We find that, compared to the seed models that use only small amount of human-labeled data, our method can robustly and significantly enhance model performance, with average improvement of 7.88%. After multiple iterations, the final convergence can achieve or even surpass the performance models using the entire human-annotian annotated dataset, providing potential solution for the self-improvement of models.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 REINFORCEMENT LEARNING FROM EXTERNAL FEEDBACK
Preference learning or now commonly referred to as reinforcement learning from human feedback (RIHF) Christiano al. 2017); Ziegler et al. (2019); ; Stiennon al. (020b); Ouyang al. (2022) ;
Bai et al. (2022a) train a fixed reward model (RM) from human preference data, and the trained RM then used train the Large Language Model (LLM) via RL, such Proximal Policy Optimization (PPO) Schulman et al. (2017b). In order to make RL training more stable and efficient, methods such as Direct Preference Optimization (DPO) Rafailov al. (022) directly train the LLM using human preferences without training the RMM. Other methods Zhao et al. (2023); Gulcehre et al. (2023); Yuan et al. (2024a) adjust the preference training schemes to improve the performance and stability. However, obtaining human preference data, especially high-quality data, is extremely expensive and time-conuuning Köpf et al. (2024); Xu et al. (2023 y Sun et al. (2024), and the data diversity is skewed to be low, containing few expert-anme anprtt-nnottted data which requires huge effort and expertise Peng ett al. (2023); Zhang et al. (2023) ;; Xu et al. (2023). ). The data quality and size sets the bottleneck of the performance of LLMs. Reinforcement Learning from AI Feedback (RLAIF) [Bii al. (2022b); Lee et al. (2023) employs LLMs to generate feedback for training RMss, reducing reliance on human- annotated data. However, relies the heuristic assumption that LLMs can provide high-quality and diverse feedback and often requires stronger LLMs to provide feedback Pang et al. (2023). In this paper, we leverage a small percentage of the human-amma annotated data train an RM which achieves a comparable performance with the one trained with the full annotated data. The RM is further used in PPO to train the L.L..
2.2 SELF-LEARNING IN LLMS
As the LLMs developing towards superhuman-level, which may be bottlenecked by human performance level. Similar to the self-imppovement human reflection, self-learning is a new approch in improving LLM performance recently. Self-learning in LLMs focuses on enhancing capabilities without external supervision. SELF-ALIGN Sun et al. (202)) demonstrates self-alignment through principle-driven reasoning, allowing models adjust their outputs based on internal guidelines. ReSTEM (ingh et al.] (2023) employs self-training to enhance problem-solving abilities. RLC Pang al. 2023) and SCoRe Kumar et al. (2024) showcase methods for self-correction and improvement using gene-gene data. Additionally, Huang et al. (2022) illustrates how LLMs can refine reasoning through self-generated rationale- augmented answers, enhancing their explanatory depth. Math- Shepherd Wang et al. (2024) and Rewf-Renn Language Models Yuan et al. (2024b) demonstrate self-rewarding mechanisms, where the model has the ability to provide high-quality rewards to itself. Our proposed approach falls in this self-learning paradigm by innovatively using the RM to generate feedback for itself, fostering robust RM training and improvement.
3 SELF-EVOLVED REWARD LEARNING FOR LARGE LANGUAGE MODELS
In this section, we present our proposed Self-Evolved Reward Learning (SER) for LLMs. This approach enables the RM to iteratively improve itself by learning from its high-confidence predictions, thereby reducing the need for extensive human-an annotated data. Initially. the RM is trained with small set of human-annotated data provide a basic understanding of good and bad answers. From there, the RM evolves through self-labeling and iterative retraining. The enhanced RM then employed guide the LLM training via RL approaches. We detail each component of our method in the below section, including self-labeling, identifying learning status, data filtering, RM retraining and the LLM training via RL with improved and converged RM.
3.1 OVERVIEW
Figure Iillustrates the overall pipeline of our SER method. This iterative process ensures that both the reward model and the LLM are continuously refined throughout the training cycle. Our method for Reward Model training consists of the following three iterative steps:
3

---

Published as a conference paper at ICLR 2025
1. Self-Label with Reward Model: data as a warm-up stage, then the RM performs self-labeling on the unlabeled data.
The RM is initially trained with small set of
Identify the Learning Status of the Reward Model and Select High- High-Confidence Data: Evaluate the RM's current ability to differentiate between good and bad answers or to amplify differences between similar answers. This status assessment guides the selection of high-confidence data.
Retrain the Reward Model with Pairwise Loss: After filtering, the selected high-confidence data are used to retrain the RM with pairwise loss, iteratively enhancing its understanding of answer quality.
With a few iterations of self-evolved reward learning, the RM training converges or meets the stopping criteria, such as when further data can be filtered, the RM is then used to guide the training of the LLM via RL approaches. The modified PPO algorithm incorporates the evolved reward signals to optimize the LLM's policy.
Our method relies on two distinct learning statuses for the RM: (1) the ability to distinguish between clearly good and bad answers, and (2) the ability refine differences between answers of similar quality. These statuses are separated for the following reasons: (a) Targeted Skill Development: by recognizing different learning statuses, the RM can focus on specific skill sets. Initially, the model focuses on clear distinctions (e.g., good vs. bad answers), and as training progresses, it refines its comparative abilities with more subtle distinctions. (b) Adaptive Data Filtering: the data filtering process is driven by the current learning status, allowing the model to train on the most relevant data. This adaptive approach ensures the model always works on improving the appropriate aspect of its performance. (c) Improved Self-Er Self-Evalle vaau-tinn: by continuously monitoring its learning status, the RM can determine when to shift from one learning focus to another. This dynamic approach fosters self-driven, curriculum-like learning.
Furthermore, by allowing the RM to judge two answers for each question, the RM is provided with paired examples that are key to both learning statuses, enabling the RM to improve its discrimination and comparative abilities. Once the RM becomes proficient at handling both tasks, it is well-equipped guide the LLM during reinforcement learning.
STEP 1: SELF-LABEL WITH REWARD MODEL
we first predict reward score for all unlabeled data based on the current reward model (RM). This is formally expressed as follows:
As shown in Figure 1
Tt = RM Ar)
This reward score may contain substantial noise, depending on the performance of the current state of the RM. We use these reward scores to determine the current training status and data selection strategy. Initially, we employ a small amount of annottted data to obtain a seed RM. In this study, the seed RM is trained using 15% of the entire dataset.
STEP 2: IDENTIFY THE LEARNING STATUS OF THE REWARD MODEL AND SELECT CONF-CCNNE DATA
Each question Q in our self-labeled dataset has two possible answers, A and AJ, which can exhibit various relationships. The RM must differentiate between the following scenarios: One answer is clearly better than the other (c.g., A! is good, A? is bad, or vice versa), or both answers are good, but one is better (or both are bad, but one is worse). The RM assigns probabilities pl and P that represent the likelihood that A! and A? are "good". The goal is to distinguish the relative quality of the answers across these different cases.
Let Dumum = ((Q,. A/ A be the training dataset. The learning status S is determined by the predicted probability differences between A! and A?:
A = p přl. We define the learning status S using thresholds Tho Thigh. and TA:
(2)
4

---

Published as a conference paper at ICLR 2025
Status1,
if (p) > Thigh and pł < nlww or (p} Now and p3 Thigh),
S StatuS2, else if A 2 TA,
Stop, otherwise.
To determine the current status, use the reward model (RM) trained in the current iteration to predict on the unlabeled data. Both Status and Status 2 require a sufficient number of predictions meeting specific criteria ensure statistically meaningful assessment. (In this paper, selected Thigh = 0.55, Flow = 0.45, and Ta = 0.3 as they provided the most consistent improvements in the RM's ability)
Status (Easier Task): This status evaluates whether the RM can effectively distinguish between positive (good) and negative (bad) samples. The evaluation is based the predicted probabilities Př for each answer Af: If p > Thigh. the RM is confident that the answer is positive. If p < Tlow the RM confident that the answer is negative. A sufficient number of high-confidence predictions (e.g.. 600 in the HH dataset) indicates that the RM is proficient in distinguishing positive and negative samples, thereby satisfying the criteria for Status
Status 2 (Harder Task): This status assesses the RM's ability to discern subtle differences between answers of similar quality (e.g., both good both bad). It requires the RM evaluate paired answers to the same question and compute the absolute difference between their predicted probabilities:
p3 >T
If sufficient number of paired predictions meet this threshold, it indicates that the RM can amplify distinctions between similar-quality answers. This task is more challenging than Status I because requires the RM to recognize and quantify nuanced differences. Similar to Status 1, this determination requires a sufficient number of predictions on the unlabeled dataset (c.g., 600 predictions in the HH dataset).
We check the statuses in order-first first Status and then Status 2 because Status represents foundational capability that is necessary before tackling the more complex task in Status 2. Status 1 is the easier task, focusing on broad distinctions, while Status 2 is the harder task, requiring finer-grained analysis. If the RM does not meet the criteria for Status (i.e., few no samples satisfy the thresholds Thigh and Tow> ), we then check for Status 2. If the RM also fails to meet the criteria for Status 2, we interpret this the model reaching its convergence point, and we halt further training of the RMM
STEP RETRAIN THE REWARD MODEL WITH FILTERED DATA USING PAIRWISE LOSS
Based on the state of the RM determined in Step 2, select different data filtering strategies as outlined below:
((05,)),43) (H)( Thigh and RM (Q,,,) A3) Tww) (R, < and RM A3) > Thigh), ((Q), (1, |RM(Q,, A4) RM(Q). A3)1 > 6). B.
if Status
F((anlabeled 8)
if S = Status
if Stop.
(4)
Here, Dunlabeled refers to the unlabeled data. In Status 1, the filter selects high-confidence data where (RM(Qj A)) Thigh and (M, Af) < Tow) (Q, Tlww and (M, > Thigh), ensuring the model trains reliable examples. In Status 2, the filter selects pairs where the reward difference IRM(Q,, comparative judgments.
(M,,, A?)1 exceeds threshold focusing refining
After filtering. the model is retrained using pairwise loss, allowing the model compare answers relatively rather than relying on absolute labels, which consistently improves performance by focusing on relative comparisons rather than absolute classifications. The pairwise loss function is:
5

---

Published as a conference paper at ICLR 2025
pasi
Σ
max(0, A - -RRM(Q). 43) RM, 431))
(5)
Dittered
(Q:.A).AA) Dinerd
where the desired margin between reward scores, and Duttered D fittered DJ filtered where n denote the number of iterations of the loop. The training data for the current loop consists of the data filtered using Equation 4 in addition to the training data from the previous loops. This iterative process, i.c., filtering data and retraining with pairwise loss, enables the RM to progressively refine its judgment until it converges.
STEP 4: TRAIN THE LLM VIA RL WITH SELF-EVOLVED REWARD MODEL.
After self-evolving the RM, we use it to guide the training of the LLM via RL. To accommodate the refined reward signals from RM, we modify the PPO framework. LLM training framed as policy optimization problem. The policy T generates responses A for inputs Q, and the objective to optimize T to maximize the rewards generated by the self-evolved RM: r RM (Q, A). We maximize the expected reward from the self-evolved RM: max Sqq_Dem.A~T.(-10) [RM (Q, A)].
Using PPO (Schulman et al. 2017b), we modify the policy updates to incorporate the refined reward signals from Rg, which better capture subtle differences in response quality. The policy is updated by maximizing the clipped surrogate objective:
CPPO E min
(A
Q) ( (A
AR, clip
(A Ta (A Q))
11
aa)
(6)
Téaa
Here, A is the advantage function based on the rewards from RM. By leveraging the evolved reward model's nuanced signals, the LLM's policy updates align better with subtle distinctions in response quality. The detailed algorithm framework of our SER approach is provided in the Appendix
3.2 THEORETICAL ANALYSIS
In this section, we analyze the theoretical feasibility of SER, focusing on the convergence properties of both RM training and PPO training. A detailed theoretical analysis supporting the effectiveness of our SER method presented Appendix [A] The key conclusions of this analysis Convergence of the Reward Model: we demonstrate that, under reasonable assumptions, the RM iteratively improves by selecting high-confidence data based on predicted probabilities. This process ensures that its performance either improves or remains stable over time. (b) Convergence of PPO with a Learned Reward Model: we establish that PPO converges to a near-optimal policy even when the RM is trained through self-labeling, provided that reward estimation errors are small. These findings confirm that both the reward model and the LLM are capable of achieving high performance with minimal human supervision.
follows: (a)
EXPERIMENT
In this section, report main experiment results, including reward modeling results and PPO results. We select multiple base models with different parameter sizes (Llama 8B (Dubey et al.
2024). Llama 2 13B (Touvron al., 2023). Llama 3 70B (Dubey et al., (024). Mistral 7B (Jing et al. 2023)) and conduct experiments on various datasets (StackOverfloww (Lambert et al. 2023).
HH-RLHF (Bai al. 2022a), UltraFeedback (Cui al.) 2023), Summarize (Stiennon et al.) 2020a)
to verify the effectiveness of the method. The experimental setup, statistical of datasets, evaluation metrics, and baselines are provided in the appendix
4.1 REWARD MODELING RESULTS
Our main experimental results are shown in Table I In all experimental setups, SER improves the model's performance, ultimately achieving results close to those obtained using the full labeled dataset. In some experimental settings, it even exceeds the performance of models trained with the

---

Published as a conference paper at ICLR 2025
full numan-labeled data, while using only 15% of the labeled data. This demonstrates the substantial potential of SER to enhance model performance in data-scarce scenarios.
Table 1: The results of reward modeling on the HH-RLHF, Ultrafeedback, Summarize, and Stack- overflow. Loop 0 denotes the RM trained with 15% of the human data. SER represents the results of iterative evolution based on the Loop 0 model. Full dataset denotes the results of the RM trained data.
with the entire set
HH-RLHF Mistral-7b
Summarize Mistral-7b
Llama3-8b
Llama2-13b
Llama3-8b
Llama2-13b
Loop 0
56.9
56.01
59.47
58.01
55.84
63.2
SER
68.56
68.1
70.26
68.42
65.49
69.19
Full Dataset
70.45
67.97
71.11
68.61
63.56
71.3
Ultrafeedback Mistral-7b
Stackoverflow Mistral-7b
Llama3-8b
Llama2-13b
Llama3-8b
Llama2-130
Loop 0
62.54
61.4
66.5
69
68.8
65.1
SER
74.46
70
72.69
70.8
70.1
69.2
Full Dataset
73.92
71.3
74.53
69
70.4
68.7
4.1.1
MAIN FINDINGS
SER consistently and effectively enhances model performance. As shown in Table compared to the baseline that uses only 15% of the data, SER improves the model's performance by incorporating self-labeled data for training. Through multiple iterations, the model achieves significant performance gains, resulting in an average 7.88% increase in accuracy. Furthermore, we find that as the model's parameter size increases, the foundational capability strengthens, and the potential for SER's self- improvement further enhances. Larger parameter models typically achieve higher performance after undergoing self- gelf-improvement. In most experimental settings, the performance the LLama 13B model surpasses that of the other two smaller parameter models.
In data-rich scenarios (Stack Overflow), the performance gains from SER become smaller, averaging only 2.4%. In such data-rich contexts, clear scaling trend with model parameter size is observed. The larger the model parameters, the greater the benefits of SER's 's self-improvement. Mistral 7B achieves performance improvement of 1.3%. Llama 8B achieves 1.8%, and Llama 13B achieves 4.1%.
SER approach or exceed the performance of full-scale human-labeled data. We compare our method with using the full human-labeled data. The results demonstrate that SER can achieve performance close to that of using the complete human-labeled dataset, with average performance difference of 0.3%. For Mistral 7B on the HH-RLHF dataset, SER exceeds the baseline by 0.13%, and on the Summarize dataset, it surpasses the baseline by 1.93%. For LLaMA 8B on the UltraFeedback dataset, it achieves performance advantage of 0.54% over the baseline. A potential trend is observed where the difference between the SER method and the full human-labeled data increases with model size. Specifically, the average difference for Mistral 7B is +0.12%. for LLaMA 8B is +0.06%, and for LLaMA 13B -107%. This suggests that larger models better utilize labeled data, enhancing performance. This trend highlights the potential of SER to further elevate model performance by scaling labeled data through self- self-labeling rather than manual annotation.
In data-rich scenarios, this trend becomes more pronounced. In the StackOverflow dataset, LLaMA 8B achieves performance very close to that using the full dataset with only 15% of the human- label data. By employing SER, the model's performance is further enhanced, surpassing the full data by 1.8%. LLaMA 13B shows a 0.5% performance improvement compared to the full human- labeled data, while Mistral performs 0.3% lower than the baseline. This indicates that in cases of abundant data, the model's 's sell-evolved can lead to more diverse data distributions, thereby further raising the model's performance ceiling.
7

---

Published as a conference paper at ICLR 2025
(a) HH-RLHF
(a) LLama 8B
(b) Mistral 7B
(c) LLama 13B
Test accuracy (%6
Test accuracy (%)
Test accuracy (66
Loop
Loop
Loop 0
Loop
Loop
60
Loop
Loop
Loop
Loop
Loop
Loop
Loop Baseline
Baseline
Baseline
250
1000 Steps
1250
1500
50
202
600
1000
1200
1400 1802
50
200
600
8b0 1903 Steps
1200 1400 1600
Steps
(b) Ultrafeedback
(a) LLama 8B
(b) Mistral 7B
(c) LLama 13B
79
Test socuracy (96)
Test ccuracy (%)
Test acuracy (96)
Loop
Loop
Loop
Loop
Loop
Loop
Loop
Loop
Loop
Loop
Loop
Baseline
1800
2000
200
400
1000
4000
Steps
(c) Summarize
(a) LLama
70.0
(b) Mistral 78
(c) LLama 13B
75
Testt accurccy (%)
85.0
70
Test accuracy
80.0
Loop
Loop
57.5
Loop
Loop
55.0
Loop
Loop
Loop
Baseline
52.5
Baseline
Baseline
3000
3000
Steps
Steps
Figure 2: Reward modeling improves in performance with iterative evolution. We demonstrate the performance variation the model during the iterative process on the HH-RLHF, Ultrafeedback, and Summarize datasets. Baseline refers the RM that uses the full dataset of human-annotate data. Due to the large size of the summarize test set, the results in the figure are based on a random sample of 1/10 of the test set.
4.1.2 FINE-GRAINED ANALYSIS
In order to conduct more detailed analysis of what occurs during the model's iterations, we present the changes in the model's accuracy the validation set, shown in Figure Additionally, illustrate the variations in the amount of training data across different iterations, as depicted in Figure 3 Our main conclusions
follows:
The model can iteratively enhance its performance on self-labeled data, even if the self-labeled data contains noise. As shown in Figure Loopl consistently enhances the model's performance in every experimental setup. During the Loopl I phase, the model's performance is relatively weak, and there may be significant noise in the model's self-feedback; therefore, is essential select high-confidence samples to improve the model's 's CilooommnneeSStauuu 1). Generally, the performance improvement in Loopl is the most sigmificant among loops, and can filter out the largest number of training examples. As illustrated in Figure on average, Loop1 provides 4.54% enhancement to the model's performance. This also verifies our Theory which posits that when the model's initial accuracy exceeds 50%, iterative training with high-confidence samples can further improve the model's performance.
Similar data becomes marginally helpful after multiple iterations and may even harm the model's performance. During the loop 2 phase, as the model's 's capability increases, the benefits brought by the simple samples filtered out in state become less significant. As shown in Figure 2 the performance improvement loop is the least significant across all iterations. This indicates that merely increasing the number of clearly defined samples offers limited performance enhancement for
8

---

Published as a conference paper at ICLR 2025
Loop (human)
Loop 1
Loop 3
In] LLaraa s8
(ti Wenl
(o 138
Loop 3
Loop
Lusp
[ummnn)
15
20
30
40
50
10
20
30
60
50
10
20
30
40
50
Data Nums
Data Nums 151
Data Nums 196
Ii] Laara
(cl Wizol
(1) ULama 136
Loop 3-
Loop
Loap
(human)
10
20
30
40
50
10
30
60
10
Jg Wama
Loop
Loop
Loop
1961
Nums 1961
Figure The percentage of the total data used by the RM each iteration shown. (a)-(c)
correspond to the HH-RLHF dataset, (d)-(f) correspond to the Ultafeedback dataset, and (g)-(i correspond to the Summarize dataset.
Right wins
(a) hh-rihf
(b) stackoverflow
Mistral 78 hul
19%
65%
10%
Bamn 78 vs Full
30.5%
42.0%
27.5%
31.5%
LLama vs Full
laaaa 708 vs Full
78%
55%
24.5%
55.0%
20.5%
40
100
Figure 4: use GPT-4 as a judge to evaluate the capabilities of the model trained with PPO. We employ the win rate as the evaluation metric. Left represents our SER method, SFT denotes the model fine-tuned with SFT, and Full refers to the PPO model guided by an RM trained on the full dataset.
the model and may even lead decrease in model performance (as observed with Ilama 13b the ultrafeedback dataset). By observing the training process, should incorporate more ambiguous and difficult samples into the model's training process, allowing the model better discern the quality of two similar samples.
By adjusting the error reduction strategy, more diverse self-labeled data can be obtained, further enhancing the effectiveness of self-learning, Based the analysis of the training process, to avoid overfitting the model on simple data, we need to focus the training objective on similar samples that difficult distinguish, enabling the model to identify differences between the two

---

Published as a conference paper at ICLR 2025
samples. During loop 3, employ the data strategy of learning status 2, which enhances model performance by having the model learn to differentiate between more ambiguous hard samples. As illustrated in Figure 6 by modifying the data filtering strategy and introducing more diverse samples, the model in loop 3 increased the score differences between similar samples, thereby enhancing its discriminative ability. As shown in Figure _ in the loop 3 phase, training the model on hard samples further enhances its performance, approaching or even surpassing the results obtained using the full set of -anaaaa-d data, Across multiple iterations, the total amount of training samples and the number of training steps are less than those required for the full dataset, typically representing only about 50% of the full data.
SER is more data and human-labor efficient than full fine-tuning. As shown in Figure using the SER method, we utilize only 15% of the human-annotated data train the initial model. Subse- quently, we reselect the data based on the feedback from the initial model, achieving performance improvements. We conduct experiments on multiple datasets, demonstrating that SER effectively generalizes to various scenarios. Considering the high cost of human--nnotatee preference data, we provide a potentially effective solution to reduce this cost.
4.2
PPO RESULTS
To validate the effectiveness of SER, we use the previously mentioned RM to guide PPO training, thereby optimizing the LLM. We conduct experiments on the Anthropic HH RLHF dataset and the Stackoverflow dataset, as shown in Figure 4 In the hh-rlhf dataset, all SER models exceed the SFT baseline in terms of win rate, indicating that the SER approach enhances the capabilities of the LLM. Compared to RMs trained with the full human-annotated data, the win rate in PPO experiments demonstrates a consistent trend with the performance of the RMs. For Mistral 7B, the accuracy of the SER RM surpasses that of the RM trained with the full dataset, and in PPO experiments, the win rate also slightly exceeds that of the full model. Additionally, to verify the generalizability of our method, conduct the same experiment on the StackOverflow dataset. As shown in Figure ()), the SER models outperform the full models to a certain extent, demonstrating a trend consistent with the accuracy of the RMs. In summary, our main findings are follows:
SER enhances the capabilities of LLMs, and the degree of enhancement is positively correlated with the performance of the RMs. Through the self-evolved approach, we improve the performance of RMs using limited amount of human-annotated annotated data. Leveraging RMs to guide the learning of LLMs results stronger LLMs. Theoretically, this process can be iterative, whereby stronger LLMs generate higher-qualit responses, further enhancing the performance of RMM.. However, due to the computational cost of PPO, we do not conduct related experiments. Additionally, we find that the performance gains in the PPO process are positively correlated with the performance of the RMs; stronger RMs generally guide the training stronger LLMs.
5 DISCUSSION
Our paper demonstrates empirical performance improvements through self-evolved RM driven by intuitive motivations, though rigorous theoretical analysis of its effectiveness is still needed. The data filtering strategies are empirical, yet it's interesting that different datasets exhibit similar learning statuses in each iteration loop. Future work includes developing a more robust and autonomous method to identify learning statuses and filter self-labeled data. On the other hand, our method provides a feasible pathway to enhance reward modeling capabilities. An avenue worth exploring is generating more diverse responses through LLMs. By applying our method, a robust and general reward model can be developed to assist all existing feedback- integrating LLMs into the entire seol-ev reward learning loop is another future work, specifically by incorporating step 4 in each iteration and using LLMs to generate responses for the RM to perform self-labeling. Our work presents a potential solution to break through the performance ceiling of those strongest L.LMs.
training methods. Additionally,
10

---

Published as a conference paper at ICLR 2025
6 CONCLUSION
In this work, we introduce SER, a simple yet effective method of self-evolution that enhances model performance across various datasets and models. By allowing the model generate its own labeled data and controlling the model's 's learning state to select appropriate data, we achieve iterative evolution that ultimately converges to, or even exceeds, the performance ceiling. Extensive experiments indicate that our key design (consideration of different learning states) is essential, and we analyze the effects throughout the iterative process of SER, providing valuable insights for the self-improvement of L.Mss.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05860 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22 2022b.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-311, 2018.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
Daniel Dewey. Reinforcement learning and the reward engineering principle. In 2014 AAAI Spring Symposium Series, 2014.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek Kadian et al. The Ilama 3 herd of models. ArXiv, abs/2407.21783,, 2024.
Lin Guan, Karthik Valmeckam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre- trained large language models construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems, (6:79081-79099) 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint 2riiv:2308.09990 2023.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong. Zhen Wang, Daisy Zhe Wang. and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.1499 2023.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu. and Jiawei Han. Large language models can self-improva arXiv preprint arXiv:22 /6/0, 2022.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06822 2023.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar 2001 (00361 2020.
11

---

Published as a conference paper at ICLR 2025
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte. Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant invvessations-dmmorrattti large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John Co-Reyes. Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. 129/7, 2024.
Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 stack exchange preference dataset, 2023. URL hww.tttpss//gggggggaaaaeeee co/ dat/aattse///////
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint 1rxii::4003.378/ 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. Rlaif: Scaling reinforeement learning from human feedback with ai feedback. arXiv preprint arXiv:2309. 00267, 2023.
Tianle Li. Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-han hard and benchbuilder pipeline. arXiv preprint arXiv:2406. /939, 2024.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274 2017.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model seff-improvement by reinforcement learning contemplation. arXiv preprint arXiv:2305. 14483, 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXhv:2304.03277 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. ArXīv, 0ts/1707.063477 2017a. URL https://api semant semanticscholar. htw.aaar...gggroororrooieoeeeeee :B3bbpuu
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.066 2017b.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:23/2.0655 2023.
Nisan Stiennon, Long Ouyang, Jeff Wu. Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. NeurIPS, 2020a.
12

---

Published as a conference paper at ICLR 2025
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021,,0011,0000 2020b.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.09220 2023.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9426-9439, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024b.
Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou. Automl-gpt: Automatic machine learning with gpt. arXiv preprint arXiv:2305.02499, 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 1042: 2023.
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint wrXiv:199900859; 2019.
13

---

Published as a conference paper at ICLR 2025
A THEORETICAL ANALYSIS
In this section, we provide a rigorous theoretical analysis to support the effectiveness and convergence of our SER method. We focus two main aspects:
Convergence of the Reward Model during Self-Traine Training We provide formal conditions under which the reward model converges to a good solution through self-training.
Convergence Properties of PPO with a Learned Reward Model: We present theoretical
foundations supporting the use of PPO for training the LLM with the improved reward model, including convergence proofs.
A.1 CONVERGENCE OF THE REWARD MODEL DURING SELF-TRAINING
Self-training involves using model's own predictions to generate additional training data. While powerful, it can suffer from error amplification if not properly managed. We provide formal proofs for the convergence of the reward model under certain assumptions.
Definitions Let R, denote the reward model at iteration l. Let fillered be the filtered dataset used for retraining iteration I.
Assumptions
Assumption (Initial Model Accuracy). The initial reward model R, has an expected accuracy greater than random guessing:
P ((Q.A.))
R2 (Q,A) = (c) > 0.5,
where y {0, 1} denotes the true label (bad or good answer).
Assumption 2 Nrigh-Confidence Prediction Reliability). For data points where the reward model's s prediction confidence exceeds thresholds Tp and 1 Tp. the prediction accuracy is at least o, with
> 0.5:
P | (Q.A) (Q.A) = y 10 - - 6p
a,
(8)
where (Q, A) is the predicted probubility, and 6, = Tp 0.5, Tp equala to Thigh and equals to Tlow
Tp
Theoretical Result
Theorem (Convergence Reward Model during Self-Training). Under Assumptions [ and with appropriate choice of threshold Tp. the sequence of reward models {R²") converges to fixed point Rý with improved accuracy, i.e.,
lim (1) Acc 2 (cc!!)) 2 Acc > 0.5.
(9)
(+000
Proof We provide a proof by induction.
Base Case ( = O): By Assumption 1 (C)
(0) > 0.5.
Inductive Step: Assume that at iteration i, Acc > 0.5. The filtered dataset ((t) Filtered consists of
examples where the model's predicted probabilities are confident, i.e., (p()) (Q, A) 0.5| > &p. From Assumption the accuracy on (!)
fifered is at least > 0.5.
Retraining the model on (t) leads to updated model R R((++1) with improved accuracy due the iltered
following reasons: 1. Risk Minimization: Training minimizes the empirical risk on ((t)
ffterrer leading
to better performance on data similar to (t) 2. Data Distribution Shift: Since (1))
üittered is a subset
filtered
where the model is confident and likely correct, retraining this data reinforces correct predictions.
Therefore, the expected accuracy satisfies Acc
(!)
14

---

Published as a conference paper at ICLR 2025
Convergence: As Ace is bounded above by and forms non-decreaiing sequence, it converges to Acc" $ 1. Thus,
lim Acc Acce Acc 0.5.
(10)
The key to convergence the selection of high-cone nigh-confidence data that more likely correctly labeled. By ensuring >> 0.5, we guarantee that each retraining step is more likely to improve the model than degrade it.
A.2 CONVERGENCE PROPERTIES OF PPO WITH LEARNED REWARD MODEL
We now amalyze the convergence properties of PPO when using the learned reward model R2 obtained from self-training. PPO a policy gradient method that seeks to maximize the expected cumulative reward. The policy n is updated to maximize:
(() EQ, [ (Q].
(11)
Assumption 3 (Lipschitz Continuity of Reward Model). The learned reward model R2 A) is Lipschitz continuous with respect to A, ie., there exists LR > 0 such that for all A2,
[Q A1) Rg A2) S LR A1 A2l11
(12)
Assumption 4 (Bounded Policy Updates). The policy updates satisfy [|o((+1) - -60012 S 58 for some 66 > O.
Theorem (Convergence of PPO with Learned Reward Model). Under Assumptions Band 4 and given that the true reward function R' (Q, A) is approximated by (Q, A) with bounded error c,:
[R A) R2 (Q; A)] cr:
(13)
the PPO algorithm converges to a policy n that is within ((r)) of the optimal policy xt with respect to Rs AQ.
Proof
The proof follows from the performance difference lemma and properties of PPO.
Performance Difference Lemma(Kakade Kakade & Langford 2002):
The difference in expected rewards between the learned policy Ta and the optimal policy x² under the true reward function R" is:
J² - ('(T)) 1 EQuu [AAT (Q.)))
(14)
where ( AQ is the advantage function.
Since Rÿ approximates I² with error e,, the advantage estimates used in PPO are off by at most e,.
Impact on Policy Gradient: The policy gradient used in PPO is:
( RQ.A- V log म (A (AAR)((()))))) A)),
(15)
where (Q, A) is the advantage function computed using Ri..
Due to the bounded reward error c,, the gradient estimation error is also bounded:
()) ( S Cep,
(16)
where C' is a constant depending on the policy and reward model.
Convergence Analysis:
Under Assumptions 3 and 4 standard results from stochastic gradient descent convergence apply
(Bottou et al. 2018). The policy updates converge to stationary point of [()), and the error in the reward model introduces an O(e,)) bias.
Therefore, the final policy म satisfies:
[J*(x²) - J (((T)) Ker,
(17)
15

---

Published as a conference paper at ICLR 2025
for some constant K.
The convergence to near-optimal policy depends the accuracy of the learned reward model. As Er + O, the learned policy approaches the optimal policy under Rr.
Combining Theorems 1 and we conclude that: (1) The reward model improves over iterations, reducing the reward estimation error e,. (2) The improved reward model leads to better policy updates in PPO, resulting in an LLM that performs well with respect to the true reward function. (3) Our SER-LLM method is theoretically grounded, with formal proofs supporting its convergence and effectiveness.
B ALGORITHM
Algorithm summarizes the entire SER method, including iterative self-evolved RM training and reinforcement learning for LLM policy optimization.
Algorithm 1: Self-Exolved Reward Learning for LLMs (SER)
Input: Initial RM Rg, unlabeled data D unllabeled, tuman-labeled labeled data Dabbltd thresholds Tlow
Thigh T&. 5, learning rate
Output: Trained L.M policy Te
* Step 0: Pret ra in the Reward Model
Pretrain Rg on the human-labeled data Diabeled using pairwise loss Cpairi
while not converged do Identify the Learning Status of the Reward Model
Step
Evaluate Rg on unlaahetdd determine the learning status S using predicted probabilities and thresholds Tlw, Thigh, and TA:
S Stop then
break;
/+ Step 2: Filter Data Based on Learning Status
S Status; then
Filter samples where predicted probabilities P, satisfy Pj > Thigh (confidently good) or
Ps < 7lw (confidently bad) to construct Dittered
else if S Statuss then
Filter paired samples where the absolute difference in predicted probabilities satisfies
1p) - -P'l > 8 to construct Diltered:
Step 3: Update and Retrain the Reward Model
Update Dittered
D + Diiced where Dfiiered the newly filtered data and D is
the data from the previous iteration.;
Retrain Rg on the updated Diitered using pairwise loss Lpair with learning rate n7;
/* Step 4: Train the LLM via Reinforcement Learning
Train LLM Ve using Rg as the reward function and update T with modified PPO (Eq.
C EXPERIMENTAL SETUP
SFT training: For each Base Model, we perform instruction fine-tuning using preference data. Similar the setting by Rafailov al. (202)), we sample higher quality responses from the preference data based human annotations
as training data (for instance, in the HH- dataset, sample responses labeled as 'chosen' for instruction fine-tuning). We conduct standard instruction fine-tuning training the base model using the sampled data. In our experiments, refer this as our SFT baseline.
Reward Model training: We perform reward modeling the SFT baseline using preference data. In our method, we train an initial model with a small amount noman-annotate preference data (in our experiments, this constitutes 15% of the overall dataset size; details the dataset split for
16

---

Published as a conference paper at ICLR 2025
SFT, PPO, etc., be found in the appendix D.I) D.I). The initial model then assigns reward scores to unannotated responses, and based on these reward scores, we filter and obtain new training data for the next iteration of training.
C.I DATASET STATISTICS
Our experiments explore four different preference datasets as show in Table 2 StackOverflow contains over 3,000K QA pairs collected from StackOverflow. Each question receives a score based on the number of upvotes, resulting in a comparison pair. HH-RL we use human preference data, which consists of 1 18K helpful and 42K harmless instances as the training set. Similar to previous work, select the last round of dialogues to construct the data into a single-turn dialogue format. UltraFeedback is constructed by large language models (LLMs). collects 64K instructions from various sources, generates 256K responses using LLMs such LLaMA, and has these responses annotated and scored by GPT4. From this process, we create a preference dataset containing 100K entries. TL:DR consists of 179K pairs of summarization and human preference annotations.
Table 2: The statistics of datasets, types of tasks, and types of feedback presented. We provide detailed introduction to the datasets in the appendix C.1
Dataset
Num
Task QA
Feedback type
Response type
Stackoverflow 31,284,837
human
human response
HH-RLHF
169,352
QA
human
LLM response
UltraFeedback 63967
QA
GPT4
LLM response
Summarize 179000
summarize
human
LLM&human response
C.2 EVALUATION METRICS AND BASELINE
Reward Modeling. The standard process of RLHF involves training RM based on preference data to predict the preferences between human and model responses. Subsequently, reinforcement learning methods are used optimize the language model based on the RM. Accuracy: We use accuracy to measure the performance of reward modeling. Specifically, for a given preference data, the reward value assigned by the RM to the chosen response is higher than that to the rejected response, the prediction is considered correct. Baseline: considering that our method uses only portion of the modeling as baseline.
data, we choose use a model trained on the full dataset for reward
PPO. For the RM obtained in the previous step, we apply it the standard PPO process to optimize the LLM. We use LLM as judge evaluate the performance of the model after PPO optimization. Specifically, we use GPT-4 as the evaluator to compare different responses the same prompt. GPT-4 assesses the quality of the responses. We conduct the comparison in two different orders and if the results from these two orders are inconsistent, we consider the results as a tie. Baseline: We compare SER with the SFT baseline to intuitively demonstrate the improvement of our method in aligning model preferences. Additionally, we compare our approach with an RM trained using the full preference dataset, despite the fact that our method uses significantly less data.
C.3 TRAINING DETAILS
SFT training. We use the following hyperparameters for instruction fine-tuning training. We employ learning rate of 2e-5 with cosine decay, warmup steps, and a batch size of 16. We calculate the loss only for the target tokens rather than the full input sequence, and we train for 3 epochs on the training data. For smaller parameter models (e.g., llama 8B, Mistral 7B, llama 13B), we conduct the training on 8 NVIDIA A 100 80G GPUs. For the llama 70B model, we perform the training 16 NVIDIA 1000 80G GPUs.
Reward training. To enable the model to learn the relative ranking among different responses, use a pair-wise loss. We employ the sigmoid function to normalize the reward scores to a range of 0-1. We utilize the LoRA method to train the RM on the SFT baseline, with rank of 8, LoRA alpha of 32, and a LoRA dropout of 0.1. The task type is sequence classification. We use a learning rate of 2e-5 with linear decay and the AdamW optimizer for training over 2 epochs, with batch size
17

---

Published as a conference paper at ICLR 2025
of 4 (batch size of 2 for the LLaMA 70B model). We conduct the training 8 NVIDIA A100 80G GPUs (32 NVIDIA A100 GPUs for the LLaMA 70B model).
PPO training. For PPO training, to 256. We employ a batch size of and mini-batch size 1, with 4 PPO epochs and gradient accumulation step. The target KL divergence is set to 0.1 and initial KL coefficient is set to 0.2. To ensure a more robust training process, we normalize the range reward values to -1 to
learning rate of 1.4e-5 and set the generate sample length
Thresholds. The thresholds Thigh- Tlow and Ta determined through extensive hyper-paramene parameter tuning to balance precision and recall in the self-training process. Specifically, we experimented with the following values:
Thigh {0.55, 0.65, 0.75}
7lo E {0.45, 0.35, 0.25}
7A (0.3, 0.4, 0.5)
After evaluating the RM's performance with these parameters, we selected Thigh 0.55, 7low 0.45, and тΔ = 0.3 as they provided the most consistent improvements in the RM's ability to self-label effectively without introducing significant error amplification.
D IMPLEMENTATION DETAILS
D.1 THE SPLIT OF THE DATASET
For the preference dataset, we split the training and testing sets according to the ratio of PPP: 0.3:0.65:0.05. In this paper, SFT utilizes the chosen responses from the preference data for instruction fine-tuning. For the training of Reward Modeling, our approach randomly samples 15% of the RM data for training, while comprehensive comparison experiments train the entire RM dataset. For the HH-RLHF dataset, it is divided into harmful and helpful subsets, and we only select the helpful subset.
D.2 STATISTICAL DATA OF THE ITERATIVE PROCESS
We quantify the amount of data filtered out during the iterative process, as shown in Figure 3 Loop 0 represents fixed value, accounting for 15% of the overall dataset. We use this portion of the data to train the seed model, upon which all subsequent iterations are based for further evolution.
D.3 PPO LEARNING CURVE
PPO Training Curve: HH-RLHF
Our Method
Full Datasel
Scaled Reward
Scaled Reward
0.2
0.2
LLama BB our method
Mistral 7B Ful Dataset
0.0
0.0
Mietral our method
LLama 8B Full Datasel
800
1030
800
1000
1200
Steps
Figure 5: The learning curve of the model the HH-RLHF dataset, with the y-axis representing the reward score after scaling. The model reaches convergence after 1200 steps. The shaded area indicates the standard deviation.
18

---

Published as a conference paper at ICLR 2025
As shown in Figure 5 present the reward curves of Mistral 7B and LLaMA 8B the HH-RLHF dataset. Both models reach convergence at around 1200 steps. We scale the reward scores to the range of -1 to 1 using the following formula:
((1 Sorisiad) ) telip) (NewMar NewMin)
Sscaled =
+ New
(18)
felip
In this equation, Soriginal represents the original reward score, telip denotes the clipping value, N NewMin is the minimum value after scaling, which is -1, and NewMar is the minimum value after scaling, which 1.
D.4 GPT4 EVALUATION PROMPT
A crucial element of our experimental framework is the evaluation of win rates using GPT-4. In this section, we provide the prompts utilized generate win rates for both the summarization and dialogue experiments. All experiments were conducted using the gpt-4o-20240806 model. The sequence responses was randomized for each evaluation to ensure unbiased results.
GPT-4 as judge system prompt:
Review the user's question and the corresponding response using the additive S-pointscoring system described below. Points are accumulated ated based on the satisfaction of each criterion:
Add point if the response is relevant and provides some information related to the user's inquiry, even if it is \ incomplete or contains some irrelevant content.
Add another point if the response addresses a substantial \ portion of the user's s question, but does not completely resolve the query or provide a direct answer.
Award a third point if the response answers the basic \ elements of the user's question in useful way, regardless \ of whether it seems to have been written by an Al Assistant or if it has elements typically found in blogs or search results.
Grant fourth point if the response clearly written \ from an Al Assistant's perspective, addressing the user' s \
question directly and comprehensively. and is well-organized and helpful, even if there is slight room for improvement in \ clarity, conciseness or focus.
Bestow fifth point for a response that is impeccably \ tailcred to the user's question by AI Assistant, without \ extraneous information, refecting expert knowledge, and \
demonstrating a high-quality, engaging, and insightful answer.
After examining the user's instruction and the response, please first give the total score. Then provide a brief explanation of your total score, up to 100 words. Output. with the following format:
Score: <total score> Evaluation evidence: <your brief explanation here>
Remember to assess from the Al Assistant perspective, utilizing web search knowledge as necessary. To evaluate the \ response in al alignment with this additive scoring model,
19

---

Published as a conference paper at ICLR 2025
we'll systematically attribute points based on the outlined criteria.
User input prompt:
<Question>:
(question)
<response>:
(answer)
Table 3: A case study for GPT4 evaluation
Input
Human:
I'm trying to teach myself to have nicer handwriting. Can you help?
Assistant:
Mistral 7B SFT GPT4-as-Judge
Sure, can help! I'm a human handwriting expert.
Score: [
Evaluation evidence: The response is relevant in acknowledging the user's request for help with handwriting, but provides substantial information or guidance related to improving handwriting skills. It also inaccurately claims be from a human handwriting expert, Which is misleading coming from an AI Assistant. The answer lacks depth and does not address the user's question meaningful way.
Mistral method
7B Our
Sure, I can help you to learn to have nice handwriting. To do this, you will need to practice writing and drawing lot, so that your handwriting will improve.
GPT4-as-Judge
Score:
The response is relevant and provides basic advice re- lated to improving handwriting, such as the importance of practice. However, it lacks depth and detail, offering only very general suggestion without specific techniques or steps to actually enhance handwriting skills. Thus, addresses part of the user's question but does not offer comprehensive or directly useful guide.
Evaluation evidence:
Mistral dataset
7B Full
Sure, think can! First, need you to write a few sentences in a beautiful, swirly style, and then can take look your handwriting and make some suggestions.
GPT4-as-Judge
Score:
The response is relevant and provides some engage- ment with the user's request by suggesting an activity. However, it doesn't directly address the user's question offer actionable advice improv- ing handwriting. lacks comprehensive guidance or practical tips, failing to fully answer the question provide useful information on handwriting improvement techniques.
Evaluation evidence:
E COST AND PERFORMANCE OF SER VS. HUMAN LABELING
The cost of using the SER method is significantly lower than employing human labeling, while achieving comparable performance. According the following calculation method, the cost of the SER method is more than 6X lower than that of using human labeling.
E.
COST ESTIMATION
Human Labeling Cost. Google Cloud's ' human annotation service Saciceeecaarre charges approximately $0. 11 USD / 50 words for classification tasks at the time of writting. We assume that each classification task
http:::/oolood.goggle..ooo///
20

---

Published as a conference paper at ICLR 2025
llama 8b
b) mistral 7b
llama 13b
locp0
laop0
loopd
3.0
3.0
loop1
loop1
2.5
locp2
25
loop2
loop3
loop2
loop3
2.0
x=0.3
Denalty
2.0
x=0.3
Density
x=0.3
x=-0.3
1.6
1.0
1.0
05
0.0
-1.0
-0.5
0.0
1.0 -1.0
-0.5
0.0
0.5
-1.0
-0.5
0.0
0.5
Score J- Score K
Score J Score K
Score J Score
Figure 6: The reward score distribution of the model on HH-RLHF, with the y-axis representing probability density and the x-axis representing pairwise score differences. Compared to other loops, loop 3 significantly increased the score differences between responses of similar quality by altering the еπтоr reduction strategy.
only consists reading a document and two candidate summaries, which have combined average word length of 304 words. We estimate the human labeling cost per example to be $0.668 USD (304 words #$0.11 50 words) (Lee et al. 2023). The detailed calculation can found in Equation 19
304 words x 0.11 USD 0.668 USD/sample
Human labeling Cost
(19)
50 words
LLM Labeling Cost. We evaluated the use of GPT-4o in place of human labels for the SER method. The average input length for each annotation was 525 tokens, and the average output length was 104 tokens. For the GPT-4o model the input cost is $0.0025 per 1.000 tokens, and the output cost is $0.01 per 1,000 tokens. Fach response was scored three times by GPT-4o to provide a preference pair. Based on these parameters, the labeling cost per example was calculated as follows:
LLM labeling Cost = 6 x ((
525 x 0.0025)
104 X 0.01
))
+
0.0135,, USD/sample (20)
000
1, 000
Inference Cost. Based on estimations using GPU pricing from Amazon Cloud A machine equipped with A100 GPUs incurs cost of 32.77 USD per hour ), the average inference cost per testing, inferred 1,530 samples using
sample amounts to 1.338c-4 USD/sample. Based a single A GPU, which required 3 minutes. The detailed calculation of inference cost can be found in Equation 21 Our estimated SER cost per sample is $0. 10054 USD here we provide approximate estimation. Per sample cost = 0.67 + 3*1 4 USD. SER, inference is conducted per iteration, resulting in total of 3 additional inferences).
32.77/8 /8 USD/hour
Inference Cost
1.338e USD/saae
1, 530 examples/3 minutes x 20 303-nnute slots/hour
(21)
E.2 PERFORMANCE AND COST COMPARISON
SER utilized only 15% of the human-labeled data, resulting in significant reduction in data dependency. training stage, our computational costs are comparable to those incurred when using the full dataset, with additional inference costs being introduced solely during step I Self-label with Reward Model. We compared the performance and cost of SER and human labeling, as shown in Table We observe that:
https://platffot www.aoppena..... http://awwaaa..o.oaaoooacocca//a
https::/ /aw.. amazon. com/pricing
21

---

Published as a conference paper at ICLR 2025
Table Performance and cost comparison HH-RLHF dataset. addition comparing the 15% human-labeled data with SER, we further explored the effectiveness of replacing numan-labeled data with LLM-lleeled data. Furthermore, we investigated the impact of increasing the proportion of human-labeled data while incorporating the SER method. Our results demonstrate that this approach outperforms the full human-labeled dataset in terms of performance.
Method
Cost per Sample(USD)
Accuracy(%)
Full Human
0,668
70.45
15% Human-Labeled +SER
0.100
68.56
15% LLM-Labeled + SER
0.002
67.64
60% Human-Labeled SER
0.402
71.83
Cost-Eifectiveness of SER. The SER method achieves performance closely aligned with fully human-labeled data while significantly reducing costs. With only 15% human-labeled data and minimal compute costs, the SER method demonstrates that it is a practical and scalable approach to reducing annotation costs.
Trade-Off Between Annotation and Compute Costs. By leveraging accelerators for self-labeling, SER minimizes annotation costs without incurring prohibitive compute costs. At ( K = loops, SER achieves strong balance between performance and cost, suggesting it is viable alternative to full human labeling for various tasks.
Scalability, Uner lower cost, the (60% human laheled + SER) outperform the "Full Human" accuracy. The SER method provides cost-efficient framework for scaling reward models without significantly compromising performance, making it suitable for real-world applications where annotation budgets constrained.
BENCHMARK EVALUATION
To investigate in greater detail the impact of SER on reward models and preference alignment, evaluated both the reward models and the models after PPO across multiple benchmarks. Specifically. for the reward models, evaluations were conducted on RewardBench (Lambert al.) (024). For the preference alignment models, assessments were carried out on MT--Bench (Zheng et al. 2023) and Arena-Hard (Li al., 2024). Overall, we found that the evaluation results on the benchmarks were consistent with those obtained using LLM-as-a-judgee and that SER significantly enhanced model performance under Vata-limited conditions.
F.1 REWARD MODELING RESULTS IN REWARDBENCH
As shown in Table present results for the RewardBench evaluation of models trained the UltraFeedback dataset. This dataset offers comprehensive benchmark, allowing us to evaluate generalization across diverse tasks.
The results demonstrate that SER significantly enhances model performance compared to the baseline dataset. tasks Loop achieving results close to those obtained with the full UltraFeedbact dataset, SER related to dialogue (Chat and Chat-Hard), which are central to the achieves performance nearly identical to models trained on the full dataset.
F.2 PPO MODEL RESULTS IN MT-BENCH AND ARENA-HARD
To further evaluate the downstream performance of models trained with SER. we provide results from MT-Bench and Arena-Hard, comparing SER to both the Full Dataset and SFT baselines, as shown in Tadd Table 7
The results highlight that SER achieves competitive performance relative to models trained on the full dataset and often surpasses SFT-trained models (because Arena-Hard features more challenging test questions, it yields a relatively higher proportion of ties; however, improvements can still be observed). These improvements are particularly notable in dialogue-heavy tasks, further demonstrating the robustness of our approach across multiple tasks and domains.
22

---

Published as a conference paper at ICLR 2025
Table The table shows the performance of reward models trained on Ultrafeedback dataset, evaluated on RewardBench. Here, 'Avg.' denotes the average score. As demonstrated, under the same amount of numan-annotatee data, SER significantly outperforms Loop0 and achieves performance comparable that obtained using the full human-annotated dataset.
Model
Method
Avg.
Chat
Chat-Hard
Safety
Reasoning
Loop 0 SER
59.1
70.7
44.1
52.2
69.7
Llama 3 8B
72.3
97.2
58.2
67.8
75.0
Full Data
75.9
95.5
58.5
73.9
65.4
Loop 0
56.3
55.9
51.3
59.4
63.2
Mistral 7B
SER
72.0
85.5
57.1
64.5
62.0
Full Data
66.8
93.8
52.4
64.1
60.3
Loop 0
56.3
82.7
45.2
66.0
59.0
Llama 13B
SER
70.5
92.4
52.7
66.0
71.8
Full Data
74.1
95.5
54.1
68.7
63.7
Table 6: The performance of PPO models trained the HH-RLHF dataset in MT-Bench
Model
Comparison
Win
Tic
Lose
SER Full Dataset
96 (30.0%)
110 (34.3%)
114 (35.6%)
Llama 3 8B
SER vs SFT
116 (36.25%)
112 (35.0%)
92 (28.8%)
SER vs Full Dataset
61 (19.0%)
214 (66.9%)
45 (14.1%)
Mistral 7B
SER vs SFT
73 (22.8%)
199 (62.2%)
48 (15.0%)
Table 7: The performance of PPO models trained on the HH-RLHF dataset in Arena- Hardd-Hardd
Model
Comparison
Win
Tie
Lose
SER vs Full Dataset
62 (12.4%)
363 (72.6%)
75 (15.0%)
Llama 8B
SER vs SFT
70 (14.0%)
379 (75.8%)
51 (10.2%)
SER vs Full Dataset
34 (6.8%)
442 (88.4%)
24 (4.8%)
Mistral 7B
SER SFT
41 (8.2%)
435 (87.0%)
24 (4.8%)
23


Reviews:
[{"review_id": "x3AXsjAma7", "reviewer": "Reviewer_8P3z", "summary": "The manuscript introduces an approach to improving sample-efficiency of RLHF. The method enables using only a fraction of the original data through an iterative approach of labelling and ranking of the preference data based on the dynamics of the training process. The authors provide theoretical support of their empirical results, as well as extensive experiments (multiple models, multiple datasets).", "questions": "Please refer to the weakness comment.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The manuscript, in my opinion, has several strengths that make it interesting to the ICLR audience.\n\n* Tackles an important problem, namely of sample efficiency, for RLHF methods. The cost of human annotations is rather high and therefore studying ways to reduce the dependency on the entire datasets is useful overall.\n\n* The recipe is well presented and supported theoretically. Although the steps seem obvious at first, the subtlety of amplifying the differences (second step) during the training loop seems rather novel and not extensively studied before. Effectively, it introduces the ability for the model to provide signal as to what samples provide most utility at current training stage.\n\n* The extensive results, which utilize multiple models and multiple datasets. The performance across all these combinations of setups seems to support that the recipe is effective and general.", "weaknesses": "The main weakness in current practices is the longer-term importance of annotated human datasets. The recipe introduces significant compute costs, trading off annotation costs for accelerators (through the recipe steps). However, the manuscript does not study the relationship between the two. If we assume human-annotated data goes away for preference labelling - having datasets created by LLMs for training other LLMs - do the benefits of the recipe still outweigh the costs?\n\nI would like to understand the authors' position on this aspect, as well as a better comparison between labeling costs and accelerator costs of their recipe. Where does a tie come in (e.g. at the K=4 loops used in the paper, or for what value of K)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BWe5F2FqL2", "reviewer": "Reviewer_vMDv", "summary": "This paper introduces an approach called Self-Evolved Reward Learning (SER), which leverages an RM to autonomously curate its training data for continuous improvement. The authors conduct experiments across various models and datasets, demonstrating that the SER approach can train the reward model well using only 15% of the labeled data, which results in performance comparable to an RM trained on the full labeled dataset, both in terms of evaluation capability and integration within PPO training.", "questions": "See questions in the weaknesses part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Motivation:** Given the high costs associated with acquiring human-annotated data for reward model training, the development of self-training approaches is highly beneficial. Such methods can leverage a minimal amount of human annotations for initial model training and subsequently enhance performance through self-improvement.\n\n**Soundness:** The paper's various training strategies, which are based on different statuses, are well-conceived and intuitively sound.\n\n**Substance:** The authors have conducted extensive experiments across multiple models and datasets, effectively demonstrating the efficacy of their Self-Evolved Reward Learning (SER) approach.", "weaknesses": "**Clarity:** The paper lacks clarity in several places, with some descriptions being particularly confusing. For instance:\n\n- **Lines 206-211:** It remains unclear how the status is determined, given that $p_i^1$ and $p_i^2$ are both single numbers. How are these individual numbers used to decide the training status? Additionally, how are the thresholds $\\tau_\\text{high}$, $\\tau_\\text{low}$, and $\\tau_\\Delta$ established?\n\n- **Lines 248-249:** The method for determining the $\\Delta$ term is not explained.\n\n- **Section 3.2 (Theoretical Analysis in Appendix A):** This section appears to offer only speculative insights into how the algorithm might function, rather than providing concrete proofs.\n\n- **Lines 325-326:** The statement \"Larger parameter models can achieve higher self-improvement benefits\" seems inconsistent, as Llama 13B appears to exhibit lower benefits compared to Llama 8B and Mistral 7B.\n\n- **Section 4.1.2 (Fine-grained Analysis):** The paper does not clearly define what Loop0, Loop1, Loop2, and Loop3 training entail. What data are these loops trained on, and do they use the same subset of data? How is the transition to the next loop decided? Furthermore, why is the strategy of learning status 2 employed in Loop 3 not earlier?\n\n**Meaningful Comparisons:** In Figure 4, the comparison using GPT-4 as a judge seems to employ an unconventional prompt for evaluating model responses (Appendix D.4). Why not use the AlpacaEval or MT-bench LLM-as-a-judge prompts, which have been shown to correlate well with human judgments?\n\n**Implications:** The paper currently focuses on improving reward models that are not well trained (i.e., trained on only a small subset of the complete human-annotated dataset). Ultimately, the goal is to continuously enhance strong reward models with minimal human annotation. It is unclear whether this approach can be applied to strong reward models and whether these models have the potential for continuous self-improvement. Exploring this line of experimentation would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aTky2K2t1O", "reviewer": "Reviewer_U5TR", "summary": "The paper proposed a self-improvement RM training strategy to circumvent the challenges of collecting human preference data. The RM is initially warmed up using only 15% human preference, then it’s improved iteratively through a series of self-labeling and retraining. During the self-labeling the current RM is used to score unlabeled data (i.e., prompt and answer pairs), then two data filtering strategies are devised based on the predicted probability differences between answer pairs. The goal is for the RM to (1) first learn from pairs with clear distinction (trained on very high- and very low-scored response pairs) before (2) switching to refining its understanding of subtle differences (trained on pairs that are more similar). When no improvement is seen on this current data, training stops and new data is collected. \n\nThe author tested their approach on multiple datasets including HH-RLHF, UltraFeedback as well as several model sizes and families including Llama and Mistral. They also evaluate how the improved RM helps during PPO.", "questions": "1. Are the pairs of responses generated from the same base LLM? Or are the authors using the response pairs in the existing dataset. While the author mentioned generating two responses from the LLM (line 173), it is not clear which model they use to generate responses and if they do this on-policy data generation for all the iterations? More details on this is appreciated. \n2. Line 174-175: how generating two responses on-policy from the LLM \"enriches\" the training data with diverse answer qualities? What aspect of this setup is unique or does it guarantee quality/diversity?\n3. Intuitively, what does the RM learn from its highly confident labeled data during stage 1? In other words, when the RM is too confident about a preference label (irrespective of whether it may be incorrect), what training signals it is gaining from it? Did authors try some ablations to test the contribution of each data state?\n4. How the number of training steps differ in SER vs Full dataset in Table 1?\n5. One would expect to see the performance of your PPO'ed model on downstream standard benchmarks like AlpacaEval, MMLU, BBH, MATH, CODEX, etc. So technically, start from a pre-trained base model doing supervised finetuning on some existing instruction-tuning data , then try doing PPO using your RM vs the baseline RM (trained on full data, e.g. on UltraFeedback) and compare performance over standard benchmarks. This will more realistically show the effectiveness of your approach as opposed to just narrowly evaluating models in their ability to choose the preferred response.\n6. Line 343: The scaling claims are somewhat unsupported, based on Table 1 13b> 7b> 8b. more experiments specifically within the same model family is needed to make informative scaling trend conclusions.\n7. Section 4.1.2, there's no explanation as to what Loop 1-3 refers to in section 4.1.2 and figure 2. It is not clear if loop is referring to one iteration of RM training (including both state 1 and 2), or something else? Line 373 states that \"loop 2 is the least significant across all iterations\" which makes me think loop != iteration\n8. Line 418: does that mean Loop 1 and Loop 2 did not include data strategy of status 2? In general more details on the training is nice in the main paper.\n9. Section 4.2: it would make sense to try PPO on a more general purpose preference dataset like ultrafeedback and see how it transfers to downstream tasks and not just alpacaeval style evaluation\n10. Suggestion: the authors refer to their models as SER, or self RLHF. It’s nice to stick to one name and be consistent with it.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Promising results on improving the RM performance matching or exceeding the baseline train on the full human labeled data\n- The curriculum learning style for the RM which encourage learning from easy and then hard pairs seems an interesting finding in the context of RM optimization", "weaknesses": "- The Experimental design is very narrow, and limited. (see questions for more details). E.g, I would expect to see results on downstream/standard benchmarks. However currently only preference accuracy is reported.\n- Some claims in the results section are unsupported (mixed results which are generalized) → see detail below\n- The paper clearly benefits from some rewriting and reorganization. In its current form several details are missing which makes it uneasy to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bR6YPvZVYE", "reviewer": "Reviewer_eZmA", "summary": "This paper proposes self-evolved reward learning where the reward model is used to obtain more training data for iterative improvement. The algorithm is straightforward: (1) generate two responses per prompt and label using RM, (2) only keep response pairs where reward difference is large -- there are two filtering strategies (Status 1 and Status 2) and the authors alternate between them, (3) train RM with the filtered data; then, iterate the above. Finally, use RM for LM training. Findings: Only 15% human-annotated data is needed so that the new model archives performance similar to training on the entire human-annotation dataset; experiments are done on a few models (7B to 13B).", "questions": "Are Llama models base models or instruction-tuned models?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The self-improvement paradigm could save lots of resources. \n\nI appreciate the fact that the authors not only tested on reward modeling benchmarks but also LMs (by doing PPO). \n\nExperiments are done on multiple models and multiple model sizes.", "weaknesses": "On baselines:\n- It’s unclear how specifically the authors trained “Loop 0” and “Full dataset” baseline models. What is the loss function? How are hyperparameters tuned? \n- Have the authors tried regular scalar-valued reward functions as baselines? \n\nOn eval: Have the authors tested on common benchmarks like RewardBench (so we can easily compare to other scalar and generative RMs)?\n\nIn real LLM training, the same dataset would always contain examples of multiple domains. In the same dataset, lots of examples could be on simple helpfulness prompts, and lots of examples are on coding/reasoning prompts. In this case, how would thresholding work? Would the model first learn simple prompts and then coding/reasoning/complicated prompts? Or would the model give up on coding/reasoning/complicated prompts?\n\nEq. (3) can be clearer:\n- It looks like Status 1 and Status 2 can be achieved at the same time? In that case, does S=Status1 or Status2? \n- Do we have a different S at every i? \n- Do we need to recompute Delta_p (avg through every i) after each gradient step? \n- When would the model hit “Stop”? What if the “otherwise” case never happens? An explanation in natural language would be great.\n\nDetails are lacking on data generation/filtering\n- How specifically are responses generated?\n- What percentage of data are filtered at different point in training? \n- Are there more conclusive evidence that alternating between status are beneficial? How are hyperparameters tuned -- based on what metrics? \n\n\nOther misc issues: \n- Eq. (4): missing right bracket\n- Table 1 caption mentions “Our method” but I don’t see “Our method” in the table. Do you mean SER?\n- Serious misuse of \\citet and \\citep. Most of the citations on the first page and the second page, for example, should be in \\citep.\n\n———\n\nUPDATE: increased score to 5 but still concerned about paragraph 3 (and hopefully more detailed result on paragraph 2) above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)