CREAM: Consistency Regularized Self-Rewarding Language Models
Authors: Zhaoyang Wang
Weilei He
Zhiyuan Liang
Xuchao Zhang
Chetan Bansal
Ying Wei
Weitong Zhang
Huaxiu Yao
Venue: ICLR 2025 Poster (2025)
Keywords: alignment
self-rewarding
large language model
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Recent self-rewarding large language models (LLM) have successfully applied LLM-as-a-Judge to iteratively improve the alignment performance without the need of human annotations for preference data. These methods commonly utilize the same LLM to act as both the policy model (which generates responses) and the reward model (which scores and ranks those responses). The ranked responses are then used as preference pairs to train the LLM via direct alignment technologies (e.g. DPO). However, it is noteworthy that throughout this process, there is no guarantee of accuracy in the rewarding and ranking, which is critical for ensuring accurate rewards and high-quality preference data. Empirical results from relatively small LLMs (e.g., 7B parameters) also indicate that improvements from self-rewarding may diminish after several iterations in certain situations, which we hypothesize is due to accumulated bias in the reward system. This bias can lead to unreliable preference data for training the LLM. To address this issue, we first formulate and analyze the generalized iterative preference fine-tuning framework for self-rewarding language model. We then introduce the regularization to this generalized framework to mitigate the overconfident preference labeling in the self-rewarding process. Based on this theoretical insight, we propose a Consistency Regularized sElf-rewarding lAnguage Model (CREAM) that leverages the consistency of rewards across different iterations to regularize the self-rewarding training, helping the model to learn from more reliable preference data. With this explicit regularization, our empirical results demonstrate the superiority of CREAM in improving both reward consistency and alignment performance. The code is publicly available at https://github.com/Raibows/CREAM.

Published as a conference paper at ICLR 2025
CREAM: CONSISTENCY REGULARIZED LANGUAGE MODELS
Zhaoyang Wang' Weilei He?
Zhiyuan Liang²
Xuchao Zhang
Chetan Bansal* Ying Wei Weitong Zhang'
Huaxiu Yao
University of North Carolina at Chapel Hill
Nanyang Technological University
National University of Singapore *Microsoft Research
{ (zhacyang, huaxiu] unc.edu weitongz8unc.ccd edu
ABSTRACT
Recent self-rewarding large language models (LLM) have successfully applied JLM-as-a-Judge to iteratively improve the alignment performance without the need of human annotations for preference data. These methods commonly utilize the same LLM to act as both the policy model (which generates responses) and the reward model (which scores and ranks those responses). The ranked responses are then used as preference pairs to train the L.M via direct alignment technolo- gies (e.g. DPO). However, it is noteworthy that throughout this process, there is no guarantee of accuracy in the rewarding and ranking, which is critical for en- suring accurate rewards and high-quality preference data. Empirical results from relatively small LLMs (e.g.. 7B parameters) also indicate that improvements from self-rewarding may diminish after several iterations in certain situations, which we hypothesize is due to accumulated bias in the reward system. This bias can lead unreliable preference data for training the LLM. To address this issue, we first formulate and analyze the generalized iterative preference fine-tuning frame- work for self-rewarding language model. We then introduce the regularization to this generalized framework to mitigate the overconfident preference labeling in the self-rewarding process. Based on this theoretical insight, we propose a Consistency Regularized sElf-rewarding lAnguage Model (CREAM) that lever- ages the consistency of rewards across different iterations to regularize the self- rewarding training, helping the model learn from more reliable preference data. With this explicit regularization, our empirical results demonstrate the superior- ity of CREAM improving both reward consistency and alignment performance. The code is publicly available at
INTRODUCTION
Large language models (LLMs) have shown impressive capabilities across various tasks, including natural language understanding and generation (Radford et al., 2019). At the same time, I.LMs also face alignment challenges such as generating hallucinations and harmful outputs (Ji et al., 2023). To address these issues, a series of research works has explored preference learning methods such as Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) and direct align- ment techniques such Direct Preference Optimization (DPO) (Rafailov et al., 2023) to align the LLMs with human values and preferences. These aligmment methods often require large number of preference pairs which are indispensable in both RLHF and direct alignment training. However, collecting human-annotated preference pairs is time-consuming and labor-intensive, which seriously limits the scalability and efficiency of these alignment methods.
Recent advancements in self-rewarding language models (SRLMs) (Yuan et al., 2024) have attracted increasing attention in the field LLM alignment, which can efficiently synthesize preference data for iterative preference training. In this method, single LLM is required to act as two roles, the policy model and the reward model. Given unlabeled prompt data, the LLM first acts as the policy model generating several response candidates. Then, the same LLM acts the reward model, scoring and ranking these responses. These ranked responses are used as preference pairs to train the I.M with DPO, significantly reducing the reliance on human-annotated data. The above steps can

---

Published as a conference paper at ICLR 2025
be iteratively repeated to further enhance the performance. However, SRLMs still face challenges in generating reliable and accurate rewards for annotating the preference pairs, which is critical for ensuring both the quality of preference data and the alignment performance of LLMs.
To address these challenges, we first formulate a generalized iterative preference fine-tuning frame- work analyze the self-rewarding training, where this framework can also be adapted to other iterative preference tuning methods. Through this theoretical framework, we find that the re- warding bias issue in SRLMs arises from the overconfident preference labeling, which enforces the model to distinguish between responses of similar quality. For example, both responses in Figure I have high quality judgments from the human. The SRLM enforces the reward model to make a preference judgment, resulting in noisy and unreliable preference labeling. This can lead to negative impacts on preference tuning the model. Additionally, the iterative training man- ner can also accumulate the rewarding bias, further diminishing the benefits of self- eeff-improvement. From the insights of theoretical analysis, we propose Consistency Regularized sEIf- rewarding IAnguage Model (CREAM) to mit- igate the rewarding bias issue in SRLMs, par- ticularly for broadly accessible 7B-size LLMs. The core idea behind CREAM is that should not force the model to be overly confident in dis- tinguishing between responses of similar qual- ity. But how to tell the preference labeling is reliable or not? Out of the self-rewarding sce- nario, we may employ pool of external reward models to assist in ranking preferences. When two responses are of similar quality, these exter- nal models often produce inconsistent rankings. This inconsistency serves signal to indicate the level of confidence in the preference labeling. In self-rewarding scenarios, however, integrating such external reward models is not feasible. For- tunately, due the iterative nature of self-rewarding training, the previous iteration to rank preferences and then compare these rankings with those produced by the current model. This comparison provides estimate of such consistency rate. With this consis- tency rate, we can regularize the preference training to prevent the model from learning unreliable preference data, thereby mitigating the rewarding bias issue in SRLMs.
recemmend DIV project that acrompliched werkent!"
an propetu secemplodd wedkett
Rewarding
Macrame thaat Htseger Cheake teeleraas enperca-eepee plant hanger
ming wesiee
have
Aic
semo kin projuts to
wue wedkent
LBB Phat Doplay Couas uniqus plaze daelii valla audlandd mast, stta, chottppis This par purfict
Figure I: An example of both two responses are of high quality, which hard for human to distin- guish the preference. While the same model from different iterations have inconsistent rewarding.
use the reward model from
In summary, we first formulate a generalized iterative preference fine-tuning framework to analyze the rewarding bias issue in SRLMs. From the insights of theoretical analysis, we propose CREAM as the primary contribution of this paper. CREAM leverages the consistency of rewarding across different iterations for regularized preference training, which can effectively mitigate the reward- ing bias issue in SRLMs. Empirical results on series of natural language benchmarks validate the effectiveness of CREAM in mitigating the rewarding bias issue and enhancing the alignment performance of LLMs.
Notations.
Vectors are denoted by lowercase boldface letters, such as x, and matrices by uppercase boldface letters, such as A. For any positive integer k; the set 1, 2........ kk is denoted by [k]. Other general sets are denoted by calligraphic uppercase letters, such as D, with the cardinality of the set represented (D). Without ambiguity, we denote To the language model parameterized by 0, as the input prompt, and y as the output response from the language model. All other notations are defined prior to their first usage. We denote 1. [-] as the indicator function.
2 RELATED WORKS
LLM Alignment. Alignment lies at the core of LLM research and applications, aiming to ensure that LLMs adhere to human values and preferences. RLHF establishes the foundational alignment training paradigm (Leike et al., 2018; Ziegler et al., 2019; Ouyang et al., 2022a), where it leverages human preference feedback to train a reward model, and then uses this reward model to guide the LLM via reinforcement learning algorithms (Schulman et al., 2017). Recent efforts have focuses on developing direct alignment methods (Rafailov et al., 2023; Dong et al., 2023; Azar et al., 2023; Ethayarajh et al., 2024; Meng et al., 2024; Hong al., 2024), in order reduce the costs and
2

---

Published as a conference paper at ICLR 2025
complexity of RLHF and make it more efficient and accessible. DPO (Rafailov et al., 2023), representative direct alignment method, optimizes the LLM with annotated preference pairs, elimi- nating the need for training additional reward model. However, most RLHF and direct alignment methods heavily rely on human-annotated preference data, where the data collection commonly in- volves human distinguishing the "good" responses from the "bad" ones, which is time-consuning and labor-intensive (Ouyang et al., 2022a; Bai et al., 2022). Thus, synthesizing preference data with minimal human effort has become a valuable research direction.
Self-R Re-ardid Language Model. SRLM (Yuan et al., 2024) has emerged as promising approach to address the challenge of preference data synthesis in self-improvement manner. This method leverages the LLM itself to act as both the policy model and the reward model. The policy model can generate response candidates for unlabeled prompts, while the reward model uses LLMM-as-A- Judge (Zheng et al., 2023; Bai et al., 2023; Dubois et al., 2024) prompting to reward and rank these responses based on their quality. The ranked responses are then used as preference pairs to train the LLM via DPO (Rafailov et al., 2023). And this process can be iteratively repeated to improve the alignment performance without human intervention. However, having the same LLM serve as both the policy and the reward model, without any regularization, presents challenges in guaranteeing accurate rewards. This can lead to bias accumulation and noisy preference data, which ultimately harms the training. Other similar self-improvement methods (Huang al., 2022: Zelikman al., 2022; Chen et al., 2024; Guo et al., 2024b; Zhou et al., 2024) often either use the ground truth response to avoid annotation bias, introduce an additional reward model to reduce the noise in annotations. In contrast, our work neither requires labeled data nor relies on external LLMs. Instead, we propose to use the consistency of rewarding mitigate the rewarding bias SRLMs.
Reward Hacking. In both RLHF and SRLM scenarios, the reward model plays a crucial role in training LLMs (Ouyang et al., 2022b; Anwar al., 2024; Yuan et al., 2024; Fisch et al., 2024). For RLHF, reward hacking is a phenomenon where models exploit flaws or biases in reward models to maximize scores without aligning with the intended goals (Anwar et al., 2024). To mitigate this issue, various ensemble rewarding methods (Coste et al., 2023; Eisenstein et al., 2023; Ramé et al., conservative optimization (Coste et al., 2023) and
2024; Zhang et al., 2024) such as ensemble-base averaging rewards in the weight space (Ramé et al., 2024) have been proposed improve reliability and robustness. However, these works mainly focused on estimating the rewards, while CREAM in the self-rewand reffrdewar scenario only uses rewards for comparing the responses annotate preference pairs instead of maximizing the rewards. Besides, CREAM uses regularization instead of conser- vative value estimation mitigate the rewarding bias issue. Though applying ensemble rewarding methods such as robust preference optimization (Fisch et al., 2024) which proposes reward distil- lation and pessimistic ensemble is feasible, CREAM does not rely on ensemble rewarding which reduces the computational cost in SRLMs. Additionally, CREAM leverages the iterative nature of SRLM to estimate the consistency between iterations to regularize the preference pair, which is more suitable for iterative preference fine-tuning. Further discussion is in Appendix C.4.
METHODOLOGY
In this section, we first formulate the generalized iterative preference fine-tuning framework for self- rewarding, RL with Al feedback, and other iterative preference tuning methods. Next, we introduce the motivation behind the proposed consistency regularized self-rewarding method. Finally, we present the practical implementation algorithm of CREAM in detail.
3.1 GENERALIZED ITERATIVE PREFERENCE FINE-TUNING FRAMEWORK
We assume that we can access to the dataset with response Dş and the prompt dataset without response Du. The objective is to iteratively minimize the following loss with respect to the neural network parameter and a label function 2 as
C(O, 2) = (0 Ds) +
(-
O,y,y' y', =)].
(3.1)
where the first term CSFT (0; Ds) aligns the model лo to the SFT data. We note here that any potential
SFT methods (Ouyang et al., 2022b; Yuan et al., 2023; Dong et al., 2023; Chen et al., 2024), or the methods without SFT data (LsFT = 0) can be adapted in this framework. The second term EE Coro] corresponds to learning from the preference data pair {y.y'} generated by the current model 6,. The labeling function 2 y', x) (0, provides the preference judgment between y and y' for
3

---

Published as a conference paper at ICLR 2025
the DPO loss, where zyy y', x) =1 means y y' and yl,yy',)) =0 means y < <y'. The DPO
loss Lieo is defined as follows:
To xx)
To (y'lxx)
y = -y,, y' x) log
log
- log
myr (fer(y|x))
Tee(yy
log To(y' (x)
To (x)
(1 - z x)) log
log
(3.2)
Ty=r(y"(x))
Tree (y (x)
where Tee is the reference model for KL divergence regularization, and (-)) is the sigmoid function. The proposed loss [(O,z) )) in Eq. (3.1) represents all iterative preference fine-tuning algorithms. For the reinforcement learning (RL) with human feedback (Ouyang et al., 2022b), 2 the human preference comparing y and y'. For the RL with AI feedback, is the oracle reward model like GPT- (Achiam al., 2023). For the self-rewarding language model (Chen et al., 2024), is given by comparing the reward score generated from the language model itself, often with LLM-as-a-Judge prompting. However, as aforementioned, we note that such prompt rewarding method may only be feasible for larger and advanced LLMs such Llama-70B (Touvron al., 2023). For smaller models such as Llama-7B that do not have complex instruction following and reasoning abilitics, instead propose to leverage the intrinsic reward model (Rafailov al., 2023)
re & [log ( (y/x) log m [x]]
to reward and rank the responses for annotating preference pairs. Therefore, the choice of preference labeling function is closely connected with the language model parameter O. Then, introduce the following two-step optimization algorithm to solve Eq. (3.1).
Step 1. (Preference-labeling step) Keep 0 O, fixed, select function to minimize LDoo- In particular, letting 9 0, in Eq. (3.2), solution for =(y, y', x) = arg min, Loro (0;: ,',,,,',. x)
is
24 1y, x) [log To, (y|x) - log (x)(( 2 log To, TT, (yllx) - -log ( [x)]
(3.3)
Step 2. (Learning step) Keep as of Eq. (3.3), minimize loss function 2(11) with respect to 0 and get 0.+1 = arg min,
Different from existing methods, the proposed two-step optimization method directly uses the in- trinsic reward model to generate the preference data. This approach is particularly feasible for small LLMs, which lack the capacity to effectively use LLM-as-a-Judee prompts (Zheng al., 2023) for rewarding and ranking. We note that the proposed two-step method similar to the Expectation- Maximization algorithm and self-training paradigm (Zou al., 2019). This similarity is supported by the following theorem, which suggests the convergence of the proposed two-step algorithm.
Theorem 3.1. Suppose the optimization Oc+1 = arg ming L(0, 1 is solvable and the SFT loss (0 Ds) 2 for all 0 and Ds, the proposed two-step optimization method converges.
3.2 CONSISTENCY REGULARIZED RELLRREWA
The generalized framework presented in Eq. (3.1) assumes the human feedback or GPT-4 are all reliable so that the preference labeling function is trustworthy, However, for SRLMs, the accuracy of preference labeling is not always guaranteed. Therefore, treating all selected preference labels as "ground truth" by encoding them as hard labels can lead overconfident mistakes, potentially propagating biases and inaccuracies from the LLMs. Taking Figure 1 as an example, both the two responses y and y' are judged by humans to be of high quality. Forcing the model to be overly confident in distinguishing between these two responses y') of similar quality can negatively impact the performance of SRLMs during training.
This rewarding bias issue motivates us to mitigate such ambiguity by introducing consistency- regularized self-rewarding language model, CREAM. Specifically, for a pair of responses with very similar quality, their oracle reward scores should ideally be very close to each other. Particularly, when multiple reward models are available, it is likely that some models will rank one response superior, while others may rank the opposite response as better, resulting in high ranking in- consistency (i.e.. low ranking consistency) among these models. Based on this, CREAM aims to prevent the model from learning from preference pairs with low consistency. Instead, it focuses
4

---

Published as a conference paper at ICLR 2025
Sampling
Self Rewarding
Consistency Regularized Training
Vi > vi
DE
Last Iteration
PU9
Next Iteration
TTd,
yl 3 P....
Rs
Consistency Estimation
The flow of CREAM. In the response sampling stage. the policy model Te, generates N responses. After that, CREAM uses the reward model Rø.,, from previous iteration reward and rank these responses. Then, the rankings are compared with those generated by current reward model Rg, estimate the consistency rate. Finally, the policy model Te, is fine-tuned with consis- tency regularized preference training objective, resulting in the model 1Te4: for next iteration.
Figure 2:
solely on preference pairs with high consistency across different reward models, thereby mitigat- ing the rewarding bias issue and stabilize the learning process some extent. From the theoretical perceptive, we can introduce a regularization term to Eq. (3.1) as
[(0, LSFT (D; Ex-Duuy (-xx PPO y y', x2, 2) X (0:y, ,')), (3.4)
where the regularization term LReg (0: y. x) prevents the model To from overconfidence in dis- tinguishing the preference of {y.y' of similar quality, which is quantified in the following lemma.
Lemma 3.2. Let the random variable = 2(y.y' y', x) be defined as z(y, y',x) = 1[y > y' y/|x]. The Bradley- model (Bradley & Terry, 1952) for the probability of under parameter 0 given by
(= = P1 > y' xx]) (log(To) No (y/ TTe(y [ x)) log(To (y' (y'l))///////amr) merly [x)))
Letting the regularization LRey be defined by
Reeg 0: (yy,y', = log (log(=o (y) (y))))))))) log(To (y' x)/ T (x())))))))))))))))))))))))))(
log a (llog No ( xx) (/y /x)) (log no (y x)/ Tree (y (x))).
(3.5)
Then the expected regularized loss under the model 0, given by:
Ey.y' y' (reg 0 y.y.'. y' x) 2 KLL ( || Po(-)),
(3.6)
where u(=) is the uniform binary distribution, i.e., u 0) a(= 1) 0.5.
As Lemma 3.2 suggests, the LReg will regularize the preference between {y, that has similar quality to uniform distribution. Then the following theorem suggests that using LDFO XL corresponds to the soft-labeled DPO which we implemented in CREAM.
Theorem 3.3. For all y'y', minimizing
C(O, 2) LSFT(O::))) DSFT) + Exo-ooy [EDro xx2) + ACReg (0: equivalent with minimizing
2(0,2) 1+ 21
S (0; (0;
Ex ~ (xbx) Co xx,,) Con x, 2)],
(3.7)
where the 1 reverses the preference order of y x) and = (1+)) We choose to reverse the preference order if there is evidence that the annotated preference data is reversed. And the final form can also be viewed the label smoothing. Details are in Appendix C.2.
Theorem 3.3 suggests that instead of calculating the regularization term Leegs labeled DPO to train Eq. (3.7). In particular, when À = 0, C, = and Eq. (3.7) degenerates to Eq. (3.1). This represents the case where the preference label is trustworthy from human
the soft-
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Consistency- Self-Rewarding Language Model
Input: seed SFT dataset Dş; unlabeled prompt dataset Dy; initial model parameter Oo:
Input: number of iterations T; learning rate r
I: f SFT training */
2: Obtain 0₁ by taking the gradient steps over loss (0) (xxxie my x) from 6.
3: / Iterative Preference Training training "/
4: for t = to T do
5: Sample [yrgh== ~ no. (1)x)) for all x, E Du // Response Sampling
6: Compute reward riy = log No, (ysy log To. (yr](x,) for all i E [N],] E [[Pul]
7:
Obtain rank Jıj for all My using reward rij // Rank on model 0,
8:
(x. Compute reward r'g log _e._, (Vog) (x.) log ne, (y,)(x.) for all E [N], [Pol]
9:
Obtain rank Ka for all yhy using reward ríj // Rank on model 0,-
10:
Compute T, = = hi, (() according to Eq. (3.10) for all
[|Pe]]
11:
Compute consistency rate C [Dul ", (T) 1)/2 Adaptive consistency regularization
12:
Compose preference dataset Poro using pairs fx,, by according to Eq. (3.11)
13:
Compose preference dataset DRDPO using pairs Y y 35 according Eq. (3.12)
14:
Update 0.+1 by minimizing loss L()) C DDPO) + (1 C)CoOo DRDFO)
15:
end for
Output: aligned policy model ner
some oracle reward models (c.g., GPT-4). In other words, À represents the confidence of the label function z. Specially, since in our two-step optimization paradigm, the label function a is directly derived from the previous model me,, we can measure the performance of no, using the consistency between model 0g and the baseline model (e.g., external reward model) 6', defined by
X(x)
(-1x) 1 ly > y' x. 0.] 1yy
y' |x, 0,].
(3.8)
and when A + O, CA ~ 1 - À representing the consistency of model O, and O; 1 > y'|x, O.l means the response y is better than y' given the prompt x and language model parameter O,, i.e.,
[[oog(To, (y/x)/ (y)((((())))))))))))))>))) - log( (y' (e/x)//Trlly lx)),
and similar definition applies to 1 > y' |x, 01].
3.3 PROPOSED ALGORITIM
Equipped with the above two-stage optimization and the consistency-regularizedd self-rewarding. we are ready to present the implementation of CREAM in Algorithm 1. The whole framework of CREAM is also illustrated in Figure 2. The algorithm starts from the SFT training to obtain the first model parameter 0₂ in Line 2. A similar approach is applied in Yuan et al. (2024) for avoid calculating the Lart in the future optimization steps. Then for each x, in the unlabeled prompt set Dv. N response candidates { are sampled in Line 5. Then reward scores of these N candidates can be calculated according Rafailov et al. (2023) by
Tij = BBlo Ne, 1x,) - log me, Te, (x,y,,,)) + B log Z(x)),
(3.9)
where we use the initial model parameter 0n as the reference policy नंह. Since 3 2 and log ((x)) is constant across different response y; for the same input prompt Xj, we can drop these factors and calculate rewards in Line 6. Specially, when t = 1, the rank Kij is calculated based on the instead use the likelihood rij log Re, (y (x;) as the reward for this edge case. The rank for these N candidates are therefore obtained in Line 7, where Jij means response Yij in the Jıg-th th best in the preference list of x,.
reference policy On itself. Thus
Self-Rewarding. As discussed in Eq. (3.8), baseline model is re- quired to measure the consistency. In the self-rewarding scenario, it is infeasible to add an external reward model as the baseline model. Fortunately, as the baseline model 0% (i.e., last iteration's 's model) for evaluating the consistency of the model e, thanks to chances provided by iterative training manner. Such procedure helps mitigate the train- ing error introduced t 1-th step before obtaining 0,. Considering a pair of tied preference pair y, y' both performing well, as demonstrated in Figure 1. P(y y'|x, 0.] will be oscillating around
PrssistencyRRgullarie
can employ the model before last update 0,-1

---

Published as a conference paper at ICLR 2025
0.5 when grows due the random noise. Otherwise Ply > y/|x, 0, might consistently converge to or Due to this oscillation, the consistency between 0,- and 0, on this specific preference pair would be low, and the algorithm will learn less from this noninformative preference pair thus stabilize this oscillation.
Specifically, calculate the rank of these N candidates using 0.-1 in Line and then use the Kendall's s Tau coefficient (Kendall, 1938) denoted by
2 T N(N - 1) ISKiEN
Σ
[[ (k,)(Ka) Kvz) > 0J 1 [(Je) - Jeg)(Kig K2z) < O]].
(3.10)
Kendall's Tau coefficient is widely used coefficient (McLeod, 2005; Abdi, 2007) to measure the consistency of two ranking sequences. Basically, when two sequences perfectly aligns, T, = 1 and when two sequence never aligns, Tj = -. The following lemma draws the further connection between the Kendall's Tau and the regularization parameter À proposed in Section 3.2.
Lemma 3.4. Suppose the N response candidate (yij]; is i.i.d. given the prompt Xj, then
E[ry] = y' (+1x, !1 [y > y Xx, 0.] [y < < y 0.-1] = - 2X,
where the expectation taken over the randomness of sampling the N candidate set.
Given Lemma 3.4, can recover Cx = 1 x (1+ T;)/2 and we average all Ty for all x; E Du in Line 11. Finally, in Line 12, we compose the preference dataset by selecting the best response y Ysts and the worst response y Ye-j which is similar with (Yuan et al., 2024).
Doro {(x,. })}x, E Du, x = arg min Jij, i = arg max J:y} (3.11)
Following Theorem 3.3, we also prepare the reverse DPO dataset by switching the best response and the worst response by
DRIPO ((xj, j fx1g)ll E Pu, arg min Jay, i arg maxx31)
(3.12)
and update 4.11 by minimizing the empirical loss of Eq. (3.7) in Line 14. The detailed proof of theorems and lemmas provided in the Appendix B.
4 EXPERIMENT
4.1 EXPERIMENTAL. SETUP
Data. In our experiments, we use Open Assistant dataset (Köpf et al., 2024) and only reserve about 3.4K human-annotated examples the seed SFT data Dz. To construct the unlabeled prompt dataset Dr. we mix prompts of 'Ds with the train split of each downstream task (only reserve the prompts) including (1) ARC-Easy/Challenge (Clark et al., 2018), (2) OpenBookQA (Mihaylov et al., 2018), (3) SIQA (Sap et al., 2019), and (4) GSM8K (Cobbe et al., 2021). Finally, this process results in total of 21K prompts Du. which we distribute equally across iterative self-rewarding trainings.
Models. Due to limited computational resources, we mainly conduct experiments with two LLMs with about 7B parameters, including Llama-3 (Dubey et al., 2024) and Llama-2 (Touvron et al., 2023), both of which are widely used. Note that the proposed framework is designed for LLMs of any size, while validating our findings on other L.LMS will be our future work.
Baseline Methods. We mainly compare our method with SRLM (Yuan et al., 2024) which uses the same LLM to serve as both the policy and reward model to generate preference data for iterative training. Additionally, we introduce variant of RL with AI feedback (Guo et al., 2024a), referred to as "Oracle". In this variant, the reward model in SRLM replaced with an external reward model demonstrate the upper bound performance of SRLM. Specifically, et al., 2024). a specialized trained reward model, to provide the reward scores for the generated responses. We further enhance Oracle's rewarding by leveraging the labels from downstream tasks to improve the rewarding accuracy. To compare the regularization method, we introduce "SRLM
use InternLM2 (Cai
KL" variant which adds simple regularization term A [a(viz) mo(y/ x) 12 to the original DPO

---

Published as a conference paper at ICLR 2025
Table 1: Main results of each method on test sets of downstream tasks. The exact match accuracies are reported. The "1" and "I" indicate the performance improvement and degradation compared to the method's last iteration (e.g. MI + M2 and M2 + M3), respectively. The best performance among methods using self-reward self-rewarding is highlighted in bold.
Model
Melbod
Reward
lteration
Are EacyEary
Are Aa--Chalenge
OpenBookQA
SIQA
GSMSK
Averge
Iniiaa
MO
86.29
80.37
$6.00
68.58
76.01
79.85
86.78
80.14
56.40
69.50
78.39
80.82
Oracle
Exermal
89.31
81.31
90.20
73.75
76.04
82.12
80.38
87. 37.80
70.05
78.01
80.99
SKLM
$7.30
70.37
77.48
90.71
Llama
79.78
71.49
79.38 [
81
SRIM
M3
39.38
80.97
28.20
71.19
80.20
$1.81
M2
88.26
79.96
$6.801
69.55
79.98
$0.89
CREAM wio RC
M3
88.09
80.55
87.20
71.39
79.23
$1.29
80.89
58.001
69.79
81.04
81.72
CREAM
90.20
72.06
81.73
83.37
Initial
61,07
48.98
62.20
50.36
23.05
49.25
60.44
48.46
63.20
50.77
25.88
49.35
70.20
55,03
73:40
63.66
30.02
58.
Oncle
External
M3
71.72
55.80
77.20
02:44
29.57
59.35
SRLM
46.55
34.47
49.20
48.06
22.14
40.08
Llama-2
M2
57.45
46.93
62.40
51.022
24:18 ?
48. 48.40
SRLM
M3
57.20
46,42
61.60
49.08
26.08
48.08
M2
58.25
45.56
61.60
51.13
24.56
48.22
CREAM wio RC
60.06
49,23
65.40
49.44
25.62
49.95
24.41
CREAM
48.81
64.60
51.22
25.85
50.51
loss, where À E [0.1, ,100] is a hyperparameter. In order to validate the contribution of our automatically determined consistency rate via ranking correlation, we also introduce a variant "CREAM w/o RC" which replaces the ranking correlation (RC) with manually set value as the consistency rate, where this value searched in the range of [0. 0.3,.................... ,.....)).. 0.9].
Implementation Details. In our experiments, we fine-tune the initial model (M0) on the seed SFT data for 3 epochs with a learning rate of le 6, resulting in model M1. Following SRLM M approach, then iteratively fine-tune the model using the preference learning objective two additional iter- ations, producing models M2 and M3. In the preference training of each iteration, we set B = 0.1 of DPO, and fine-tune the model for 1 epoch with learning rate of 1e 6. All training processes use Adam W optimizer (Loshchilov & Hutter, 2019) with a warmup ratio of 0.1. For the response sampling stage of all SRLM methods,
decoding temperature of 0.8 and generate N = 5 re- sponses per prompt. For evaluating downstream tasks, we use greedy decoding to generate answers.
4.2
MAIN RESULTS
The main results are shown in Table 1. From these results, we have the following findings: (1) The Standard SRLM fails to achieve satisfactory per- formance, particularly with Llama-2 which has weaker foundation performance, which indicates its limitations for 7B-level LLMs. (2) Compared to SRLM, CREAM achieves significant improvement, showing the ad- vantage of introducing the proposed regularization. (3)
Lose
CREAM
37%
30%
27%
40% sk.N Hz
32%
CREAM
SRLM equipped with an oracle reward model (Oracle)
CREAM
achieves the best performance overall. This highlights a critical challenge for all methods with unreliable re-
caskó
38%
40%
22%
s MQ
34%
45%
21%
warding. Notably, for Llama-3, CREAM even outper- forms Oracle except on SIQA dataset. This superiority underlines the success of the proposed method in miti- gating the rewarding bias issue. (4) The consistent im- provements of CREAM across iterations show the effec- tiveness of the proposed regularization method in miti- gating the rewarding bias issue. Results of later iterations (up M6) in Appendix C.I also demon-
CREAM
0%
56%
75%
1024
Purcentage
Figure 3: Arena of CREAM v.s. SRLM and Oracle, judged by GPT-4o.
8

---

Published as a conference paper at ICLR 2025
strates that adding such regularization can help prevent the model from degeneration in the long term. (5) Though SRLM + KL method improves performance of SRLM via regularization, its over- all performance across iterations is still inferior CREAM, indicating CREAM performs better than directly restricting the KI. divergence. (6) CREAM consistently outperforms "CREAM w/o RC" which manually set the consistency rate. And using the dynamically calculated consistency rate can eliminate the costs of hyperparameter search.
Alignment Arena. To more directly compare the alignment performance of CREAM, we further show the Arena win-rate of our method in Figure 3. We can find that CREAM can beat baseline methods with the same iteration, which confirms the superiority of the proposed regularized self- rewarding. Also, the win and tie rates of CREAM increase with the iteration against stronger model, i.e. Oracle, indicating the consistent performance improvements across iterations.
4.3 ANALYSIS
4.3.1 ANALYSIS OF REWARDING
Rewarding Consistency. We first examine the consistency of rewarding of different methods using their corresponding models from the last iteration in Table 2. Here, we use the pro- posed Consistency Rate C, Kendall correlation coefficient Spearman correlation coefficient, and TopOrder metrics to measure the consistency, where this metric evaluates whether the final paired preference data remains the same, calculated by: TopOrder,
[arg min Jj arg min Ký] 1 [arg max Jj- arg max Kj] where Jj and K, are the rankings provided by current model and the last iteration's 's model, respectively. This metric assesses whether both the least preferred and most preferred responses are consistently ranked across iterations. The results confirm that SRLMs ex- hibit a rewarding consistency issue, indicating that the generated preference data contains noise. In contrast, CREAM keeps the ranking consistency across iterations thanks the regularized training.
Prompt Rewarding v.s. DPO Rewarding, As aforementioned, 7B level LLMs struggle with generating accurate rewards when using LLM- as-a-Judge prompting due to their limited capa- bility. Both Figure 5 and Figure 4 clearly show that prompt rewarding is not effective for SRLM with small LLMs, the performance starts to decrease at the first iteration (MI - M2) when
Table 2:
Ranking consistency of CREAM and SRLM across iterations using Llama-3.
Inerations
Method
Comisstenc Ct
Kendall
Spearman
TopOrder
M2 M1
SRLM CREAM
0.39 0.21
a 0.15
-0.22 =
a.46 0.36
0.36 0:34
0.77
0.03 x 1.18 0.19
M3 vs M2
SRIM
CREAM
0.66 0.10
0.92 0.09
-008 0.38
8.30
0.95 0,07
0.50
0.59 x IL49
0.33
trained on the self-rewarded preference data. In contrast, the adopted DPO rewarding method can be more suitable, since it is intrinsically aligned with the model's learning objective.
Ranking Accuracy. The ranking accuracy is crucial for self-rewarding methods, as it directly affects the quality of the self-generated preference data for training. Figure 4 provides an intuitive compar- ison of the rewarding performance. The results include the ranking accuracy self-generated pref- erence data and the RewardBench (Lambert et al., 2024), both of which is formulated to predict the preferred one between two responses. We use the preference data obtained by self-rewarding with ground truth ranking labels, for testing the model's ' in-domain ranking performance. RewardBench used assess models' generalization beyond the training domain. CREAM consistently achieves higher ranking accuracy than baseline methods, which promises more reliable data for training. Even though the overall ranking accuracy (< 70%) is not satisfying, the introduced consistency regularization can help mitigate the negative impacts of potentially noisy ranking annotations.
RewardBench
Curated Preference Data
Average Performance on five Datasets
Ranking Acturacy
68
P-SRLM
SRLM
famkkin Acouracy
59
P-SLLM M
SRUM
03.37
G6
Accuracy
64
55
10.24
13.99
$0.24
50:24
M1
M2 Iteration
M3
M1
M3
Renation
SRLM
P-SRLM
CREAM
Figure Pairwise ranking accuracy on Reward- Bench and a curated Preference Data. P-SRLM is SRLM with prompt rewarding (LudM).s-a-JJdge).
Figure 5: Average performance of SRLM, P-SRLM, and CREAM. The detailed per- formance is shown in Table Table 13.

---

Published as a conference paper at ICLR 2025
4.3.2 RELIABILITY OF SELF- CONSISTENCY
The most straightforward way to enhance the rewarding and ranking accuracy is by incorporating external reward models, such as the SRLM variant "Oracle" used in our experiments. The theoretical analy- mit-
Table Comparison of CREAM using oracle reward model and last iteration's model. BRM denotes the choice of baseline reward model.
sis in Eq. (3.8) suggests that igate the rewarding bias issue by calculat- ing the ranking consistency between cur-
Method
BRM
Are-E
Arc-C
OBQA
SIQA
GSM8K
Liama-3 M1
80.14
78.39
CREAM M2
MO
88,89
80.89
88.00
69.79
81.04
CREAM M2
Oracle
88.51
81.06
86.20
72.21
79.91
rent model and other available baseline re-
Llama-2 M1
60.44
48.46
63.20
50.77
23.88
ward models (BRMs). However, it is not always feasible to have access to such ex- ternal reward models in practice, such as the self-rewarding scenario. Thus, we instead propose to use the last iteration's 's model as the BRM to measure the consistency of rewarding.
CREAM M2
MO
58.97
47.53
62.80
50.43
24.41
CREAM M2
Oracle
62.42
48.72
66.00
51.13
22.52
Choice of Baseline Reward Model. To measure the impacts on CREAM of using different BRMs, we fine-tune the M1 model using CREAM with two different BRMs: the rewarding function of Oracle and the model from the last iteration (MO). As shown in Table 3, using strong reward model as the BRM can bring better regularization effect, especially for Llama-2. However, we find that the last iteration's 's model also provides reasonably reliable consistency signal for Llama-3. We attribute this to Llama-3's 3's inherently stronger foundational alignment and better internal consistency, which allows it to effectively utilize itself without needing external reward model.
4.3.3 CONSISTENCY MEASUREMENT
Besides the adopted Kendall coefficient, other metrics can also be used to mea- sure the consistency between two prefer- ence ranking lists, such as Spearman co- efficient (Spearman, 1904) and the afore- mentioned TopOrder method. We conduct a comparison experiments of using differ- ent consistency measurement methods in Table 4. We can observe that: (1) All these measurements are effective with CREAM, indicating the generalization and applica- bility of our regularized training approach. (2) Kendall correlation coefficient generally yields higher scores across various datasets compared to Spearman and TopOrder methods. (3) The differences in performance highlight the sensitivity of these consistency measurements. Specifically, the Spear- man coefficient appears slightly less robust than Kendall's Is T, as analyzed in Croux & Dehon (2010). Meanwhile, TopOrder focuses only on top-1 and bottom-1 rankings, limiting its evaluation scope.
Table 4: Performance of CREAM using different con- sistency measurements with Llama-3.
Iteration
Method
Are-E
Arc-C'
OBQA
SIQA
GSMNK
86.78
86.40
Spearman
86.95
82,00
$5.40
70.05
78.77
M2
TopOrder
87.25
86.88
70.83
79.75
Kendall (Ours)
88.89
80.89
88.00
69.79
81.04
Spearman
90.00
TopOrder
88.51
80.37
87.40
71.03
79.76
Kendall (Ours)
89.52
83,36
90.20
72.06
81.73
5 CONCLUSION
In this paper, we first formulate a generalized iterative preference fine-tuning framework for self- rewarding language models (SRLMs), which can also be applied to other iterative preference train- ing methods. Then, we highlight the rewarding bias that arises from overconfident preference la- beling. which is particularly problematic for small LLMs, such as those with 7B parameters. This rewarding bias results in the accumulation of noisy and unreliable preference data, harming the pref- erence training and hindering alignment performance of LLMss. To address this issue, we proposed the Consistency Regularized sElf- Rewardenn IAnguage Model (CREAM), which leverages the con- sistency of rewarding across different iterations as regularization signal. This approach allows the model to learn more selectively, emphasizing reliable preference data and avoiding overconfidence in preference labeling. Extensive experimental results and analysis on various natural language benchmark datasets demonstrate the effectiveness of the proposed method in mitigating the reward- ing bias issue and improving the alignment performance of SRLMs. We believe that these findings can provide valuable insights for future research on self-improvement methods of L.LM alignment.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Hervé Abdi. The kendall rank correlation coefficient. Encyclopedia of measurement and statistics, 2:508-510, 2007.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv: 2023.
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:224.005662 2022.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. Benchmarking foundation models with language-model-as- -an-eel---------miiii an-examiner. In Thirty-sevnnth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, and al. Internlm2 technical report, 2024.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg. and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017.
2017, Long Beach, CA, USA, pp.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the A12 reasoning challenge. CoRR, abs/1800 hts/1803.05457. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10. 14168, 2021.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arX02:3310.02743, 2023.
Christophe Croux and Catherine Dehon. Influence functions of the spearman and kendall correlation measures. Statistical methods & applications, 19:497-515, 2010.

---

Published as a conference paper at ICLR 2025
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. ArXiv preprint, abs/2304.06767, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and et al. The llaaa herd of models, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Perey S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Newral Information Processing Systems, 36, 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint 23X::22312.09244, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024.
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2455.193 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Ramé, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online AI feedback. CoRR, abs/2402.04492. 2024a.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.0477 2024b.
Geoffrey Hinton. 2015.
Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531,
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5, 2024.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210. /16110000111,,, 2022.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv,, 55:248:1 1--8:334, 2023.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1 (0(1-2):81-93, 1938.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis. Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations comversttons-demorrttinnggg large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Ha- jishirzi. Rewardbench: Evaluating reward models for language modeling. CoRR, abs/2403.13787, 2024.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.088) 2018.
12

---

Published as a conference paper at ICLR 2025
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https:://///. http:///pneeeeeeeeeeeeee wtw.://wwwweeeooooo....... Bkg6RiCqY7.
lan McLeod. Kendall rank correlation and mann-kendall trend test. R package Kendall, 602:1-10, 2005.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint MxXv:2245..14334, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Seyed Iman Mirzadeh, Mchrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 17730-27744 2022a.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- fion Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022b.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1:9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. In Advances in Neural Information Processing Systems 36: Anmual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.1218) 2024.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. CoRR, abs/1904.00923 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347, 2017.
C Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 1511 1904.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, and et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307. 0bbb//337.00288, 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 2 21-27, 2024. OpenReview.net, 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint 3rXiv:2304.05302. 2023.
13

---

Published as a conference paper at ICLR 2025
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35: 55:15476-15444 2022.
Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble, 2024. URL org/ alss 2401 6635.
https:/ //axxiv
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623 2023.
Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint arXiv: 240.:2405.14622 /4622, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: (rXvv:1999008593, 2019.
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF international conference computer vision, pp. 5982-5991, 2019.
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
In this section, we discuss the limitations of our work and propose potential solutions for future research. First, our method requires full fine-tuning of the model over multiple iterations, which is both time-consuming and computationally intensive. For future research, we plan to explore seamneter-fffiieen Fine-Tuning (PEFT) methods reduce training costs. Second, our approach primarily focuses on small (7B level) language models, which may limit its applicability. However, the proposed method is designed to be general and can be applied to models of any size. Third, it is worth investigating more complex how to assign different weights to various models during the self-rewarding process based on their importance, and enable these models to collaborate improve the rewarding accuracy.
self-rewarding scenarios, such as
B PROOF OF THE THEOREMS AND LEMMAS
B. PROOF OF THEOREM 3.1
Proof of Theorem 3.1. We denote iteration of the two-step algorithm as f. The algorithm starts from (0r, z1,, and obtains 1 = =0, according to Eq. (3.3) in the preference-labeling step and then obtains 0 through the learning step. Since 2++1 arg min, LDoo according to Eq. (3.3), we have that
(Dro (f x, for any y. y.
[(O,, 2+11 S L(0,,
(B.1)
And the learning step suggests that 0.+1 arg ming L(O, 1411), yielding that
[(0++1 2411) S 2++1).
(B.2)
Connecting Eq. (B.1) with Eq. (B.2) yields that the loss function C(O, 2) is monotonically decreas- ing, i.c.
< L(0,+1, 2+11 S L(O,, #++1 < L(O,, =1) < .....
(B.3)
Since L(O, 2) is upper bounded by 0, it suggests that the sequence of [(0,, will converge w.r.t the growth of t.
B.2 PROOF OF LEMMA 3.2
Proof of Lemma 3.2. We start by expanding By
(+xx) Cre y0: y x)) as
Ey.y' y' (1x) freg (f: (0:yy',a x) = Eys'' (1x) (log Po(2 = 1) + log Po = 0))
wTo (1)) [log Po = 1) + log Po = 0)]
+ y' mo. (1x) [log Pa( 1) log Po(z = 0)]
E y' woa (x) Po. (a 0) [log Po 1) log Po(z = 0)]
By ~o, (1x) Pe. (a 1) [log Po 1) log Po(= 0)],
(B.4)
where the second equation decompose the expectation Eyy''mo,((1)) into two expectation Ey sy'noTo (-Ix) (-Ix) + Ty (+1x). the third equation extract the event y 2 y' as distribution (z).
Then Eq. (B.4) can be further written by
Ey F (-1x) y,y
Po. (2 1) Dlog Po(z 1) + log Po(: 0)]
+ (2 0) [log Po(z 1) + log Po(2 0)]
(B.5)
since both y, y' are generated from ne, T0, (-1x), Po. 0) -Po, (a 1) 0.5. Thus Eq. (B.5) becomes
Ey y'yy'o (ey (0:,,'' (rr = (Po(() || Po. (x)) = 2KT. m(=)),
(B.6)
where u(=) is the uniform binary distribution with u(= 0) u(= 1) 0.5.
15

---

Published as a conference paper at ICLR 2025
B.3 PROOF THEOREM 3.3
Proof of Theorem 3.3. We start by writing down each components in ((0, =) defined in Eq. (3.1) by
,,,,,,,)) C(0,2) (0r (s) + ExwD'ars''mp, (.1e) ( y,y', x, + ACneg y y,y, y'x))
(0: Ds)
+ Ex iy ~oa (:1x)
-2 yy, voy.y'x, logo
log
Tierly /xx) ) - log
TT(r)xx)
To (x)
Teerly' (x)))))))))))))
TOO (
TV (///)//))/ x)) Tiat (y (x)
- (1 z(y, xx) log
log
log
To (y(2))
(TO (xx)
-Alogo
log
log
Teer [yer(y| xx)
Teee (y' (x)
no (y'Ixx
no (ylxx)
-Alogo
log
log
Tyer(y"(x))
Trer(y (x)
= Ls Ds)
To (x)
Bx- y. wao (-xx)
-(A =(y,y' y' xx)) logor
log
llog
(yr (xc)
(yr (x)
-(1 À 1 ,(y,y',))) log
x))
(log
To (
log
To ( (x) (Ter(y/x))
Ty xx)
where the third equation absorbs the regularization into the DPO loss. Noticing that X+= ,))+ (1 + À =(y, y', ,',x)) + 2A, by dividing (1 2X) we have
C(0,2)
Lsn (0; Dsrt)
1+ 2X
1+2X 21
A + 2(y,y' xx) log log To(y)xx) (ra(y(x))
To(y'Ix))
+ E D'y (-xx)
-log
1 12
(Tr(y'(x))
1 1+ =(y.y',
To (x)
To x))
logo log Tyer(y'(xx)
log
1+22 22
Tee
When :2y,,,yyy,, y'x) = (1+=(y,,y'' ())) = 1-1/(1+2A) and y =
a/(1 2A). Therefore, letting C^ 1/(1 2λ) yields that
2(0,2)
CSFT(O: DSFT)
1 2X
1 2λ
D' (1x)) CDro x,,,))) Cx x, =)],
which completes the proof since minimizing 2(0,2)/(1+ A) is equivalent with minimizing C(O, 2)
itself.
B.4 PROOF OF LEMMA 3.4
Proof of Lemma 3.4. To begin with, according to the ranking of Jıg, the sufficient and necessary condition for Jıj JØz 0 that riy < Tęj. Similarly, the sufficient and necessary condition for Kij > Krÿ is that r'g < rlgg As result, the indicator becomes
xgy [[(Jr Jrz) Key)) < [[ - Tey (r -ry) < 0j.
[[(}; (Kij K2₂) > 0, 1 [(ri) rw))((( -
(B.7)
(B.8)
Since ray > Try yields Voj > Yey under the input prompt rby and language model 0,, Eq. (B.7) becomes
[[Je - JKi) - K2T) > 0] -1(ry > re.l 1// > rhyl + 1[r < rey] < <tyll
-1(ys; > ysylx,, 0, 1 > yy.lly, 6:-1]
+1 < 0,1 11 1y Kyrg/z,, 0-11
(B.9)
16

---

Published as a conference paper at ICLR 2025
As result, since Vij are i.... given x,. the expectation of first part of the Kendall's ' Tau coefficient is
E| [[n - (K Kæy) > 0| [[[(J, -e. K2y) <0] Bys (11x))
> X,,,,,
[v5]2,,0--11
+ By mo. 0 (x,)
[1[y < Vy)1x,00.111 - <yrs 0--1]] By m (- (x,)
[1 < <y,y|x, 1 > ~Yry/x, 02 02-1
By me, ( [1 " <yr |xy, 01-1]]
Bys (1x)) [1 [yو > 0x yry/,,, 01-1] 1[y; < <yr) 0--1))
+ By Mey me, (xx [1[y < 0. < <y,, 0--1 1 > 0r
= Ey Mey mo ( (x)) [1 [y > 01 - yy,) 0.-1))
By Me (1xx)
[1[y <yæ,|x, 21 < <yes [yy] 01-1
Bys Me, mo (1 (1x) [1] [y 0, 1yy, <y,;|xj; 0,]]
+ 2 Vir. mm, (11x,) [1 < Mry 0.-
< >yrylx, 0,] 1 > yry|x, 0.])]
= + 2 y., vy. ~ (1x,) [ < y,, 1, 22 21 > yry 0:1)
(B.10)
where the second equation merge the terms together, and the third equation due to the fact 1l [y. x Yeyi [y > y2yl 1, the forth equation reorganize the term and the fifth equation is due to the fact that Ey "e,
~xt, (-Ix,) [1 [yij > yesl 0,] [yij - yes Xxj, 0.]] = 0 due to symmetry. Similarly by reversing the < and > we can write Eq. (B.10) by
[[1 (K,) K2y) 0] [[ [Ki K2₂) < <0]
0 y ~o, (1x,) [1 y. 0.-](1 21 < yeyl 0.1]]
(B.11)
Adding Eq. (B.10) and Eq. (B.11) together yields
[ Joy (Kij K2₂) 0]] []] J. [K, K < O]]
2 Me, no Mor (1x)) [y:z > y,,|,,,,,,,,,- + [y - X. 0--1]]
y (1xx, [yi > N,ylx, [yy;; < xyrglx;00 01-1]
4.y (1x;) [1| [ıo <yyy|x, 0.] 11 [y > yrylx,, 01-1]]
8E y Nv, 0, [1 [y: > yry|x, 0,] 1 < <y,s| 0
(B.12)
where the final equation is because [ < xy, 0.] 1 > Yey xy, 0-11] B| Yry x,, 0r] 1[yss < yvy 1--1] due to symmetry. Divide Eq. (B.12) by 2 yields the claimed result.
C ADDITIONAL RESULTS
C.I SUSTAINABILITY OF REWA-REWARDING
To explore continuous improvements and further validate the effectiveness of the introduced reg- ularization, we conduct additional experiments with SRLM (baseline method) and CREAM (ours) using Llama-3 for the 4th, 5th, and 6th iterations. From the results in Table 5, we have the fol- lowing findings: (1) CREAM converges at the 4th iteration (M4) while the baseline method SRLM started performance degradation much earlier at the 2nd iteration (M2). This shows that CREAM helps stabilize the ceff-improeement training by mitigating the rewarding bias issue. (2) CREAM does not seriously harm the performance after convergence (i.e., during M5 and M6), while SRLM drastically drops the performance. This suggests that adding this consistency-bee regularization is beneficial to preventing model from degeneration in the long term.
C.2 DIFFERENCE BETWEEN CREAM LOSS AND WEIGHTED DPO LOss
We would like to emphasize that the final form of CREAM, which combines a normal DPO and a reversed DPO, cannot simply be replaced by reducing the weight in normal DPO (weight DPO
17

---

Published as a conference paper at ICLR 2025
Table 5: Results for SRLM and CREAM using Llama-3 for six iterations. and &. indicate the performance improvement and degradation compared to the method's 's last iteration.
Method
Iteration
Arc-Easy
Are-Challenge
OpenBookQA
SIQA
GSM8K
Average
MO
86.29
80.37
86.00
68.6
78.01
79.85
SFT
MI
86.78
14
86.40
69.50
78.39
80.24
M2
87.79
80.38
87.80 [
70.95
78.01 4
80.99 1
M3
87.17
81.23
87.30 |
70.37
77.48 1
80.71 4
SRLM
M4
86.07
78.33
87.80
68.58 1
75.83 1
79.32 1
M5
84.34
76.53
85.80
66.84 L
64.22 L
75.55 L
M6
76.22
72.36
76.00
59.06 1
59.29 L
68.59 J
M2
88.89
80.89
88.00 f
69.79 t
81.04 1
81.72 1
M3
89.52 f
83.36
90.20 f
72.06 1
81.73 1
83.37 1
CREAM
M4
89.56
82.68
90.80
72.93
82.26 T 82,49 1
83.65 1
M5
89.35
82.08
90.20
72.06
81.73 L
83.08
M6
88.85
81.57
89.60
71.14
82.73 4
Loss). Specifically, the loss adopted by CREAM is C' r(y+)) differs fundamentally from a weighted DPO loss loo (r(()((()))))))))))))))))))))))) r(y-))- From the opti- mization perceptive, the latter weighted DPO loss behaves similar with regular DPO loss with smaller learning rate. addition, since the preference probability P(y+ < y-) = o (r(y+)-r(y- and P(1+ > yー) (r((y))= r(y+)), we would highlight that the CREAM loss can be kind of viewed as a cross-entropy loss C' log P(y+ > y-) (1 (1-6 Cog P(y- > 4+) with sabel-smoothing factor C. Thus, The weighted DPO loss cannot deliver such a observation.
log (rg-(r(v-)-
Detailed Derivation. The loss used by our CREAM can be expressed as:
C + C) RD,
where C' is the consistency rate, and D and RD are the DPO loss and ReverseDPO loss, respectively.
For the DPO loss, log a - b), where the sigmoid function a converts the reward gap b) into probability P. For RD, the reward gap is (b a). Recall that the sigmoid function has property that a (a)) 1 a (-æ). Using this property, we can convert the sigmoid (gap part) of RD to (1 P). Thus, the RD loss is log(1 - P). and the D loss is log P.
Combining these, we can derive
C. (1- RD log P
C) log(1 - P);
which corresponds to a cross-entropy loss for a binary classification task (binary preference judg- ment). The proposed loss acts as label smoothing regularize the training. This regularization is expected to enhance the model's generalization ability and performance during training.
Gradient Analysis. To further understand the impact of the sum normal DPO loss and reversed DPO loss, we analyze the gradient of the adopted loss function. Recall that CREAM uses the
= C (1- RD
for training, where C' is the consistency rate, and D and RD are the DPO loss and ReverseDPO loss, respectively.
We can simplify the DPO loss by substituting the reward gap
Te(y²) log
Te (verly))
log
TTe((y+)
.o. Then, the D loss is written log logo((t)) and the RD loss is written as log (-00) Recall that the sigmoid function a has the property that (ιx) = 1 o(-x). Then, RD can be converted to log(1 (xr)). Our regularized loss L is
C' (2o)
log(1 (zr)))
18

---

Published as a conference paper at ICLR 2025
Table Comparison of CREAM and SRLM using weighted DPO loss on Llama-3, "NA" indicates that the method cannot generate fluent sentences.
Method
Iteration
Are- Arc-Easy
Are- Are-Challenge
OpenBookQA
SIQA
GSM8K
Average
SRL M
Weight 1100
M2
NA
NA
NA
NA
NA
NA
SRLM
Weight = 0.55
NA
NA
NA
SRLM
Weight -0.55
M2
NA
NA
NA
NA
SRLM
Weight 0.25
M2
88,97
80.38
87.00
71.39
78.7
81.29
SRLM
Weight= 0.50
M2
86.49
79.61
87.60
70.42
79.00
80.62
SRLM
Weight= 1.00
M2
87.79
80.38
87.80
70.95
78.01
80.99
CREAM
M2
88.89
80.89
88,00
69.79
81.04
81.72
SRLM
Weight -1.0
M3
24.16
22.53
27.6
30,91
1.74
21.39
SRL M
Weight -0.5
M3
NA
NA
NA
NA
NA
SRLM
Weight -0.25
NA
NA
NA
SRLM
Weight 0.25
M3
88,93
81.40
89.20
71.34
75.74
81.32
SRL M
Weight 0.50
M3
88.13
81.74
89.00
70.73
75.89
81.10
SRLM
Weight=
87.17
81.23
87.30
70.37
77.48
80.71
CREAM
M3
89.52
83,36
90.20
72.06
81.73
83.37
The gradient of L with respect to model parameters A is
ÖL
O.o
00 = (C - a (xx))
a0
In particular, when C' = 2. it pushes the model to learn (xr)) = 1.4 i.e., encouraging the model to have
To( To To(
(y
So far, there no theoretical evidence that the likelihoods of both y and y will decrease. How- ever, based on empirical results from existing works (Rafailov et al., 2023; Azar et al., 2023; Hong et al., 2024; Ethayarajh et al., 2024), the özg 06 term will decrease the likelihoods of both yt and y while increasing their gap. Thus, the combined optimization will potentially lead to: (1) de- crease the likelihoods both y and y (2) an increase in the gap between yt and y while maintaining
To (z(+
To Teer(y))
Tier (y)
Empirical Results. We also take additional experiments to validate the SRLM using weight DPO Loss, where weight [-1.0. 1.0). For SRLM + Weight method, M3 is trained based on the best checkpoint of M2 across different weights. The results are shown in Table 6. Note that "NA" means the negative of the DPO loss leads to catastrophic forgetting, where the LLM fails generate fluent sentences. According to the results, we observe that CREAM outperforms the SRLM+weight method, indicating its effectiveness and irreplaceability.
C.3 CONSISTENCY TREND
In CREAM, if the consistency rate C reaches to O, the preference order of the training data will be totally reversed, which may lead to overconfidence in the reversed order. However, We would like to highlight that the consistency between 0,-1 and 6g can rarely be as low as C z 0. This is because C means for any preference pair y. y', two consecutive model 0 and 0 give totally reversed prefernence. For the scaled Kendal Tau, that means if the ranking for 5 generated responses is > B C > E, to reach C O, we need change the ranking to > > > > A, which is very rare in SRLM. Since 0, is trained using the prefernece data given by 0,-1, they tend to have similar behavior and thus such a circumstance making C can hardly happen in practice.
Further, analyze the distribution of the consistency and supply the results for 4th, 5th and 6th iterations as follows. The results in Table 7 show the consistency trend for CREAM. We can find that the initial consistency rate 0.39 is acceptable, which will not result in heavily relying verse preferences. Besides, the regularization rankings and maintain stability in the later rounds of training.
CREAM shown to encourage the consistency of
19

---

Published as a conference paper at ICLR 2025
Table The distribution of the training samples' consistency rate C' in the six iterations of CREAM.
lieration
Consistency Rate
0.2404
0.400.6
0.6-0.8
08-110
Total
Arg. C
M2
1829 (13 5666)
(32.628)
(1.7(17
(5.525)
9592 (100%)
0.39
(0.50%8)
(94.71%)
9592 (100%)
M4
(0.03%
24 (0.255%)
195 (2.039)
(1.009)))
8118 (34
(1002))
0.87
MI M5
2 (0.02%
10 (0.10%)
121 (1.26%)
858 (9.25%)
8571 (89.36%)
9592 (100%)
0.89
M5 M6
6 (0.1665)
109 (1.14%)
573 (5.97%)
2129 (22.399)
6775 (70.6356)
9592 (1009)
0.81
C.4 COMPARISON OF ENSEMBLE METHODS
we also include an ablation study on using conservative value estimation methods (Coste et al., 2023) (EnsembleWWorst) and Ensemble-Mean SRLMs. Specifically, ensemble method trains different models with 3 different learning rates [7e-7, le-6, 3e-6] to serve as the ensemble reward models in each iteration, since no external models can be involved in the self-rewarding scenario. Then, these three models would independently reward and rank the responses:
Ensemble-Wone Worst selects the minimum reward for ranking the response.
Ensemble-Mest selects the average reward for ranking the response.
Table suggests that CREAM has advantage against Ensemble methods in the alignment perfor- mance cross iterations. Additionally, compared with the ensemble model which requires batch of reward models, CREAM only requires the model from the last one iteration and thus would be more efficient than the ensemble methods.
Further Discussion on Robust Preference Optimization In this paragraph, we would like to take in-depth diseussion the related work robust preference optimization (Fisch al., 2024), in order to better understand the contribution of CREAM in the self-rewarding scenario. In details, Fisch et al. (2024) suggested injecting pessimism by ensemble or introducing the KL divergence KL(TT, || Tref). However, we would like to highlight that this method cannot be directly applied in our task scenario, due to the nature of the self-rewarding setting.
First, regarding the reward distillation, Fisch et al. (2024) suggested distilling the (ensembled) re- ward information into the language models. However, knowledge distillation usually assumes distill- ing information from a large model into small model (Hinton, 2015; Buciluă et al., 2006; Mirzadeh al., 2020). This makes perfect sense when the reward given (e.g., from human preference, an oracle reward model, given by the language model the last iteration. The model capacity does not change over time, so the distillation may not help improve the model. For instance, if we extract x* from language model Tg the reward distillation (Eq. 7 in Fisch et al. 2024) becomes
more advanced model). However, in SRLMs, the reward preference is
0. = arg min EE log To (++ a)) log To(U+ xx) 2 To. (y- x))
To x)
and trivial minimizer is 02 = f with loss equal to zero. This indicates that if we use the same, single model for reward distillation, the model can hardly be improved.
Second, we would like to acknowledge that the ensembled model used in Fisch et al. (2024) and other works Coste et al. (2023); Eisenstein et al. (2023); Ramé et al. (2024) can provide pessimism, especially when we have a batch of reward models. For example, Fisch et al. (2024) used 5 reward models in their Pessimistic Ensemble DPO (c-DPO) variant, which has the best performance. How- ever, in the self-rewarding setting, each reward model is trained by ourselves, so obtaining a reward model would be costly.
For example, let's assume Nei is the reference model, is the model trained by SFT, is the first round of self-rewarding training, and яæ is the second round, etc. we would see such ensemble will not work for training T2, and for т3, will ensemble 2 models; T44 will ensemble 3 models. Note that though we can use different data subsets or hyperparameters to obtain multiple reward models in the earlier iterations, it can bring additional training costs. It is obvious that this algorithm needs to store all models T ......., R as the iteration progresses. In the case where more models required to improve the ensemble performance (e.g., need to ensemble separate कi for
20

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of SRLM, Ensemble- Worst, Ensemble-Mean and CREAM on Llama-3. The best performance among methods in each iteration is highlighted in bold.
Method
Iteration
Arc Arc-Easy
Arc- Are-Challenge
OpenBookQA
SIQA
GSMBK
Average
SRLM
M2
87.79
80.38
87.80
70.95
78.01
80.99
Ensemble-Word
M2
88.00
79.78
87.00
69.96
78.54
80,66
Ensemble-MMe
M2
88.47
80.80
88.00
69.50
80.67
81.49
CREAM
M2
88.89
80.89
88.00
69.79
81.04
81.72
SREM
M3
87.17
81.23
87.30
70.37
77.48
80.71
Ensemble-Wors
M3
87.75
80.89
86.80
70.78
78.70
80.98
Ensemble-Mean
M3
88.85
80.89
87.00
69.96
79.98
81.34
CREAM
M3
89.52
83.36
90.20
72.06
81.73
83.37
Pessimistic Ensemble DPO), this storage and computation consumption increase significantly. In contrast, CREAM only requires storing and evaluating the latest two checkpoints, which is more efficient compared with the standard ensemble methods.
Finally, we would like to put more theoretical remarks on the regularization which is also discussed in Section 3.3 in Fisch al. (2024). In detail, Fisch et al. (2024) considers the KL divergence between π and Teet as
[ mealy) æ)) log (t dy
Eg
Ex log Tty x)),
(1)
which is the same as conducting SFT with respect to Tie. Ours considers the KL divergence for the preference model, denoted by
KL(u || (a > y22) x E2.
log Pr(y > y22 = Ensy.m logo (A (11, w2))
(2)
[o(vel with (11, v22 log (x1l)))) log mar( are)" mar( (ar)
One can find the fundamental differences between (1) and (2):
1. The regularization of (1) is to control the behavior of π similar to the reference policy Tee-- It is more suitable for one iteration of DPO. In the SRLM where the LMs are expected to iteratively improve their performance by themselves, intuitively should hypothesize that Te becomes more and more distant from Tet. As a result, using (1) in SRLM would be too conservative for the LM to improve itself.
2. The regularization of (2) is to prevent the preference model from being overconfident on preference pairs from the last iteration model. Obviously, it fits the SRLM better by let- ting the SRLM improve from the correct self-labeled preference pairs but also not be too conservative on the previous iteration or the baseline model.
To summarize, believe CREAM and Fisch et al. (2024) fall into different categories: in SRLM and CREAM, we are more focused on regularizing the preference pairs, while performing model ensemble would be challenging since the model is updating and depends each other (iteration). We also acknowledge that in regular DPO or with different reward models, the ensemble methods or the regularization methods are feasible and useful.
C.5 CREAM WITH DATA CONSISTENCY
For CREAM, The introduced (in)consistency serves as the uncertainty quantification for the model to prevent the model itself being overconfident due the stochastic training noise incorrectly labeled prefernece data. Such a uncertainty, usually referred to as epistemic uncertainty, is a property for each model. We further add two variants to explore treating the consistency as a property of the data instead of the model: (1) Threshold x: This variant uses the normal DPO for samples (confident dataset) whose consistency rate > x, and uses ours C*DPO (1-C)*DDPO to regularize the training for other samples (unconfident dataset). (2) CREAM w/ Dynamic Consistency: Instead of original CREAM using the average consistency rate across the dataset, this method uses dynamic consistency rate for each sample, using each sample's 's own consistency rate regularize the training.
21

---

Published as a conference paper at ICLR 2025
Table 9: Results for treating the consistency as property of the data instead of the model using
CREAM on Llama-3. The best performance among methods in each iteration is highlighted in bold.
Method
lteration
Are Easy
Arc Challenge
OpenBookKQA
SIQA
GSMSK
Average
SRLM (Threshold 0.0)
M2
87.79
80.38
87,80
78.01
80.99
Threshold 0.1
M2
88.13
80.89
58.20
70.52
78.70
81.29
Threshold 0.3
M2
88.38
80,29
89,20
68.99
79.83
81.34
Threshold= = 0.5
M2
89.10
80.89
87.20
69.70
80.52
81.48
Threshold 0.7
M2
88.72
80,20
88.00
69.24
80.82
81.4
Threshold = 0.9
M2
88.55
81.48
88,40
70,78
79.45
81.73
CREAM (Threshold = 1.0)
M2
88.89
80.89
88.00
69.79
81.04
81.72
CREAM dynamic
M2
88.13
80.80
88.00
69.50
79.38
81,16
SRLM (Threshold 0.0)
81.23
Threshold 0.1
M3
88.38
80.29
87.20
70.98
79.83
81.34
Threshold
Threshold = 0.5
M3
89.10
61.48
89.80
71.08
79.91
82:27
Threshold=
81.31
Threshold = 0.9
M3
89.27
83.02
90.80
72.31
81.50
83.38
(Threshold=
89.52
CREAM + dynamic
M3
88.85
81.57
89,20
71.65
79.91
82.24
Table 10: Results of SRLM and CREAM with Llama-2-131
Method
Arc-Easy
Are-Challenge
OpenBookQA
SIQA
GSM8K
Average
MO
67.47
56.31
67.00
47.54
35.03
$4.67
68.27
57.42
67.40
47.85
36.09
55.41
SRLM M2
69.61
57.00
64.00
52.10
31 31.69
54.88
SRLM M3
62.08
53.67
61.20
48.93
20.77
49,33
CREAM M2
69.19
57.17
69.60
48.57
36.01
$6.11
CREAM M3
68.56
59.04
73.20
49.90
36.62
57.46
As shown Table 9, we observe that as the threshold increases beyond a certain value (including more samples for regularization), the performance gains converge. Actually, CREAM is compat- ible with using a threshold to fine-grained eeeigraaiddly select the data to regularize, however, this inevitably introduces additional hyperparameter. Besides, CREAM already achieves sufficiently good per- formance, without this added complexity. Compared to the "CREAM w/ Dynamic Consistency" variant, our method, which utilizes the average consistency rate, significantly reduces the variance in estimating dataset uncertainty, resulting in improved performance.
C.6 APPLICABILIIYY OF CREAM
Applicability to Larger Models. Due to the limited computational resources, our experiments are mainly conducted on 7B-level LLMs such as Llama-2 and Llama-3. However, we believe this still meaningful democratize the self-rewarding paradigm LLMs of community affordable size, such as 7B models. Advanced and larger LLMs (e.g., ChatGPT) are not always accessible, especially specific application scenarios like those involving medical or privacy-sensitive data. It's important to note that our method is theoretically applicable to LLMs of any size (not limited to 7B). To validate our method with larger LLMs, we test the CREAM on Llama-2-13B (Touvron et al., 2023) model without any hyperparameter tumings. The results in Table 10 confirm that method remains effective for the 13B LLM, demonstrating the generalizability our approach across different model sizes.
Applicability to unaligned Models. First, our method CREAM does require the model to have some initial alignment capability, as the adopted DPO rewarding relies on the model being aligned, otherwise the rewards would not be meaningful (Rafailov et al., 2023). Note that in the self- rewarding scenario, the original SRLM (Yuan et al., 2024) also requires the model to be "post- trained" version, where they adopt Llama-2-P7 2-BB 70B-Chat (Touvron et al., 2023) as the initial model (MO). We conduct additional experiments to test the effectiveness of CREAM with LLMs that not post-trriine trained. Specifically, we use Llama-3-8P --ama-3--B-No B.-8-----Cha Version (Dubey et al., 2024) as the base model (MO). The results in Table 11, reveal that: (1) MO performs poorly due to its lack of instruc-
22

---

Published as a conference paper at ICLR 2025
Table 11: Results of applying SRLM and CREAM to unaligned Llama-3-8B-NO-Chat -oa-------Caa--essiioon.
Method
Arc-Easy
Are-Challenge
OpenBookQA
SIQA
GSM8K
Average
MO
30.64
28.50
20.80
23.95
4.02
21.58
MI
31.23
30.21
33:40
26.51
51.25
34.52
SRLM M2
29.80
28.67
36.00
23.44
50.19
33.62
SRLM M3
31.82
30.03
39.00
28.40
48.52
35.55
CREAM M2
31.61
28.75
35.60
28.66
53.00
35.52
CREAM
39.18
35.07
33.62
56.48
42.67
Table 12: Results for P-CREAM and Distilled DPO (Fisch et al., 2024).
Method
Iteration
Are- Are-Easy
Are-Challenge
OpenBookQA
SIQA
GSM8K
Average
SRLM
87.79
80.38
87.80
70.95
78.01
80.99
P-SRLM
84.64
76.79
80.40
67.81
78.47
77.62
Distilled
87.20
80.12
85.40
79.76
80.43
P-CREAM
M2
87.67
78.58
86.40
68.47
79.76
80.18
CREAM
M2
88.89
80.89
88.00
69.79
81.04
81.72
SRLM
87.17
81.23
87.30
70.37
77.48
80.71
P-SRLM
83.75
76.28
80.20
78.99
77.17
Distilled DPO
86.15
79.01
85.00
68.68
78.62
79.49
P-CREAM
M3
86.49
78.33
87.40
69.70
81.05
80.59
CREAM
M3
89.52
83.36
90.20
72.06
81,73
83.37
tion following ability. However, after training on a small amount of seed SFT data, M1 demonstrates improved performance, especially the reasoning task GSMSK. (2) Both SRLM and CREAM effectively boost the performance, while CREAM often provides greater gains. (3) We find that the performance gains of CREAM in the 3rd iteration (M2 - M3) exceed those in the 2nd iteration (M1 - M2). This may be because M2 has better alignment than MI, and the effectiveness of our method relies on the model's alignment capability. Based on these findings, conclude that SRLM like methods can still be applied to pre-trained models, provided they have undergone slight alignment (e.g., through SFT training).
C.7 PROMPT-REWARDING FOR REGULARIZATION
In this section, discuss the usage of prompt-rewarding with regularization, and compare its ef- fectiveness of the DPO rewarding. We mainly consider two methods: (1) P-CREAM: CREAM with prompt-rewarding method. (2) Distilled DPO (Fisch al., 2024) which leverages the prompt- rewarding to apply regularization to the DPO training. The results are shown in Table 12. We can find that (1) prompt-rewarding is not effectivve DPO rewarding for both SRLM and CREAM. (2) The degradation of Distilled DPO in M3 iteration indicates its inapplicability for continuous improvements, compared to our method CREAM.
Table 13: Results for different self-rewarding methods using Llama-3.
Method
MI
M2
M3
Dataset
SFT
SRLM
P-SRLM
CREAM
SRLM
P-SRLM
CREAM
Are- Ere-Easy
86.78
87.79 f
84.64 1
88.89 1
87.17
83.75 L
89.52
Arc-Challenge
80.14
80,38
76.79 1
80.89 1
81.23
76.28 1
83.36
OpenBookQA
86.40
87.801
80.40 1
88.00
87.30
80.20 1
90.20
SIQA
69.50
70.95
67.81
69.79
70.37
66.63
72.06
GSM8K
78.39
78.01
78.47 1
81.04
77.48
78.99 1
81.73
23


Reviews:
[{"review_id": "MFrw53hKYY", "reviewer": "Reviewer_1A7Q", "summary": "The authors propose the Consistency Regularized self-rewarding Language Model (CREAM), which incorporates regularization to improve reward consistency in order to alleviate the risks of inaccurate rewards and biased training data over time. The basic idea of the method si to add a regularization of optimizing towards a balanced binary distribution, where the weight of the regularization is determined by the  self reward ranking correlation between different optimization steps. The author provides some theoretical proof and empirical results show that CREAM enhances both alignment performance and the reliability of preference data.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is interesting and there are theoretical and empirical proofs of effectiveness.\n- The method uses the ranking correlation to evaluate the consistency or uncertainty in self rewarding.", "weaknesses": "- I think first the author should add some baselines such as the original form of the method which just uses the KL constraint toward the Bernoulli distribution. It would be good to break down where the improvement comes from and how much the ranking correlation helps the methods.\n - Then another very important experiment I think is that previous self-rewarding methods can not maintain the improvement beyond 3 or 4 iterations, how will your method help this? Will there still be significant improvement for the 5th or 6th iteration?\n - Another thing I am thinking is that given the final form, in the training you use a normal DPO and a reversed DPO, and weight the loss functions. Then this is basically the same as just weight the normal DPO with a weight maybe ranging from -1<->1. Why don't the author try this for example shifting the C weight and only keeping DPO loss? This would be much more efficient in implementation I assume?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RHbOVTMSQq", "reviewer": "Reviewer_Nh1y", "summary": "This paper attempts to propose a more consistent reward learning strategy within self-rewarding LLM settings where the same model acts as the policy generating sample responses as well as a reward model that ranks those responses. Their framework called  Consistency Regularized sElf-Rewarding lAnguage Models (CREAM) leverage reward consistency signals to make their models learn from more reliable preference\ndata. Their experiments suggests that CREAM-trained models outperform SRLMs as well as external reward models especially with smaller models such as LLaMa2 and 3.", "questions": "Questions:\n1. In the line #445 - 447, It describes as “SRLM with prompt rewarding is not effective”, but\nthe performances are increasing on the GSM8K dataset in the Table 3. Can you explain\nwhy this could be the case?\n2. In Table 4, the performance of CREAM dis not improve when we compared to Llama2\nM1 on the Arc-E, Arc-C, OBQA, and SIQA datasets. What are your thoughts on why it\ndidn’t improve, even though CREAM showed better performance compared to Llama3\nM1?\n\n3. Regarding the reversal of the DPO dataset, if \\( C = 0 \\) indicates an inverse correlation between $\\theta_{t-1}$ and $\\theta_{t}$, isn’t learning from this strategy placing too much confidence in the assumption that reverse preferences are meaningful? This concern arises particularly when the initial inconsistency between $\\theta_{t-1}$ and $\\theta_{t}$ might be due to stochasticity rather than an actual flaw in the self-rewarding model being trained. As such, did the authors conduct experiments to analyze how the trend of inconsistency evolves between consecutive iterations, especially at later stages of training (e.g., around $t = 5 $ or  $t = 6$), to see if this inconsistency stabilizes later in training?\n\n\n\npresentation/typos:\n\nAbstract: “there is no guarantee on the accurate of the rewarding and ranking” —> accuracy \nLine 082: “we first formulates a generalized iterative preference” —> formulate\nLine 139: “The objective is to iteratively minimizing the following loss” —> minimize\nLine 522: “emphasizing reliable preference data and avoiding overconfident\nin preference labeling.” —> overconfidence", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The methodology section is well described with mathematical analyses for readers.\n2. Formulating the challenges and corresponding analysis is organized and explained well.\n3. Various experiments are conducted to show the performance of CREAM with models like LLama2 and 3, including comparison with reasonable baselines like SRLMs and external reward models.", "weaknesses": "1. In the line #244, there is a lack of information about the reason for using reversed\npreference order.\n2. Once again, in the line #328, The reasons for preparing a reverse DPO dataset and swapping the\nbest response with the worst response are somewhat unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uUUOrpwR8O", "reviewer": "Reviewer_AuNi", "summary": "This paper proposes a modified DPO objective that considers multiple reward judgements, and penalizes disagreement across multiple reward models. The new objective is studied in the context of self-rewarding LMs, an iterative process based on DPO.\n\n(Note: score increased from 3 to 6 based on the discussion)", "questions": "See weaknesses for questions.\n\nThere are many grammar mistakes that would be good to resolve, e.g.:\n* Abstract - “the accurate of the” -> “the accuracy of the”\n* Intro - “we formulates” -> “we formulate”\n* Many places - “the rewarding consistency” -> “the consistency reward”?\n* Section 3.2 - “prevents the model… from overconfident”\n\nOther nits:\n* Notation - You define `[k]` as a set, but the next sentence says “Sets are denoted by calligraphic uppercase letters”. Maybe say “Other sets are…” to avoid a contradiction?\n* Equation 3.4 - My understanding is that typically SFT is performed first, and then DPO, rather than optimizing both objectives jointly, as this equation implies.\n* The RewardBench citation is missing a year.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The idea of considering diversity across an ensemble of RMs to prevent reward hacking is sound and well motivated.\n* Consideration of such a signal is supported by the experimental results.", "weaknesses": "* The biggest weakness of this paper is that it is not contextualized well with prior work on reward hacking and using ensembles to mitigate this. Given that proposing variants of DPO and related algorithms is increasingly crowded, this is especially important. Beyond discussing and comparing with this prior work, it is also important to understand how pessimism-based approaches compare, both theoretically and empirically, with the proposed methods based on agreement metrics. Similarly to this paper, there is a growing body of work that aims to mitigate reward hacking through reward model ensembles (e.g. https://arxiv.org/abs/2310.02743, https://arxiv.org/abs/2312.09244, etc.). These papers provide analysis showing various correlation metrics during RLHF. “Disagreement” between reward models can be penalized by considering pessimistic aggregations of rewards, e.g. by taking the min or some percentile (<50) over the set of RM rewards. Such concepts have also previously been integrated into DPO (e.g. https://arxiv.org/abs/2405.19316). It is not clear the relation between the proposed approach of weighting by RM agreement to this prior work which aggregates RM scores pessimistically, or whether there is any advantage. \n* The motivation for introducing the framework in 3.1 was not clear to me. How is it different from prior work, e.g. Yuan et al., 2024? Why is it necessary to introduce this new framework to evaluate the key contribution of the paper, i.e. the impact of the consistency reward? This made the experiments more difficult to analyze, because it was unclear whether the only change between \"SRLM\" and \"CREAM\" was the consistency reward or whether other factors contributed to the difference.\n* The overly formal exposition of the proposed method makes the intuition difficult to follow. For example, the definition of the regularization loss, introduced in equation 3.5, does not include any intuition for why this term was chosen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D6zuTiNu5Q", "reviewer": "Reviewer_LeV2", "summary": "This paper proposes a form of regularization for self-rewarding language models, which involves creating a loss that is a weighted combination of the LM's implied preferences and the reverse preferences. The weighting is determined by the rank correlation between implied rewards across adjacent training steps: if these rewards are perfectly rank-correlated then the reverse preferences are not included; to the extent that they are uncorrelated, the self reward model is deemed less trustworthy and so the reverse preferences are given more weight. The specific form of the regularizer is justified by theoretical connections to entropic regularization. The method yields improvements over the unregularized SRLM approach of Yuan et al (2024) on benchmark datasets and in head-to-head win rate (e.g., 34/21 win-loss ratio). Evaluations focus at the 7B scale.", "questions": "The method seems to be applied to LMs that have already been post-trained (LLAMA 2 & 3), which is perhaps why the impact is less than in Yuan et al. Would it be possible to apply the method to a model immediately after pretraining or instruction tuning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- While the proposed approach seems somewhat heuristic (e.g., training on both (y, y') and (y', y)), the steps of the algorithm are justified by theoretical support.\n- The \"oracle\" component of the evaluation helps to clarify the headroom for better reward modeling on these tasks.\n- The ablations help clarify the role of the specific form of rank correlation and \"prompt rewarding\" vs the implicit DPO RM", "weaknesses": "It's not clear to me why one would want to use self-rewarding LMs at the 7B scale, as it would not be too difficult to get preference annotations from a larger model and train on them instead.\n\nThe algorithm appears to treat consistency as a property of the model \\theta_t, but I wonder whether it would be better to treat it as a property of the *data*. That is, for any model, there will be some examples for which preference can be assessed confidently, and others for which it cannot. It would be ideal to use the full DPO loss for the first type of examples, and regularize strongly on the second type. Instead, the proposed approach seems to set the regularization parameter based on some property of the optimization path: as long as the LM has not changed its rankings from time t-1, it will use the standard DPO objective. (Thanks for the additional experiments on this.)\n\nI would like to see more analysis of the sum of DPO losses. In particular, what is the effect to the policy of applying the update in step 14 of the algorithm when C=1/2? My guess is that the probability for both y+ and y- decreases, but it would be great to have some formal analysis because this situation could be relevant, e.g., at the beginning of training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)