Cream: Consistency Regularized Self-Rewarding Language Models

## Abstract

Abstract Recent self-rewarding large language models (LLM) have successfully applied LLM-as-a-Judge to iteratively improve the alignment performance without the need of human annotations for preference data. These methods commonly utilize the same LLM to act as both the policy model (which generates responses) and the reward model (which scores and ranks those responses). The ranked responses are then used as preference pairs to train the LLM via direct alignment technologies (e.g. DPO). However, it is noteworthy that throughout this process, there is no guarantee of accuracy in the rewarding and ranking, which is critical for ensuring accurate rewards and high-quality preference data. Empirical results from relatively small LLMs (e.g., 7B parameters) also indicate that improvements from self-rewarding may diminish after several iterations in certain situations, which we hypothesize is due to accumulated bias in the reward system. This bias can lead to unreliable preference data for training the LLM. To address this issue, we first formulate and analyze the generalized iterative preference fine-tuning framework for self-rewarding language model. We then introduce the regularization to this generalized framework to mitigate the overconfident preference labeling in the self-rewarding process. Based on this theoretical insight, we propose a C onsistency R egularized s E lf-rewarding l A nguage M odel ( CREAM ) that leverages the rewarding consistency across different iterations to regularize the self-rewarding training, helping the model to learn from more reliable preference data. With this explicit regularization, our empirical results demonstrate the superiority of CREAM in improving both reward consistency and alignment performance. The code is publicly available at https://github.com/Raibows/CREAM .

## 1 Introduction

Large language models (LLMs) have demonstrated impressive capabilities across a range of tasks, including natural language understanding and generation (Radford et al., ). However, LLMs also face alignment challenges, such as generating hallucinations and harmful outputs (Ji et al., ). To address these issues, a series of research efforts have explored preference learning methods like Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., ) and direct alignment techniques such as Direct Preference Optimization (DPO) (Rafailov et al., ) to better align LLMs with human values and preferences. These alignment methods often rely heavily on large amounts of preference pairs, which are essential for both RLHF and direct alignment training. However, collecting human-annotated preference pairs is time-consuming and labor-intensive, posing significant challenges to the scalability and efficiency of these alignment approaches.

Recent advancements in self-rewarding language models (SRLMs) (Yuan et al., ) have attracted significant attention in the field of LLM alignment due to their ability to efficiently synthesize preference data for iterative preference training. In this approach, a single LLM takes on two roles: the policy model and the reward model. Given unlabeled prompt data, the LLM first acts as the policy model, generating several response candidates. It then functions as the reward model, scoring and ranking these responses. These ranked responses serve as preference pairs to train the LLM using DPO, significantly reducing the reliance on human-annotated data. This process can be iteratively repeated to further enhance performance. However, SRLMs still face challenges in generating reliable and accurate rewards for annotating preference pairs, which is crucial for ensuring the quality of preference data and the alignment performance of LLMs.

To address these challenges, we first formulate a generalized iterative preference fine-tuning framework to analyze the self-rewarding training, where this framework can also be adapted to other iterative preference tuning methods. Through this theoretical framework, we identify that the rewarding bias issue in SRLMs arises from overconfident preference labeling, which forces the model to differentiate between responses of similar quality. For example, both two responses in Figure 1 receive high-quality judgments from human evaluators. However, the SRLM requires the reward model to make a preference judgment, resulting in noisy and unreliable preference labeling. This can negatively impact the model’s preference tuning. Additionally, the iterative training process can further accumulate rewarding bias, diminishing the potential benefits of self-improvement.

Figure: Figure 1: An example of both two responses are of high quality, which is hard for human to distinguish the preference. While the same model from different iterations have inconsistent rewarding.
Refer to caption: x1.png

Based on insights from our theoretical analysis, we propose Consistency Regularized sElf-rewarding lAnguage Model (CREAM) to mitigate the rewarding bias issue in SRLMs, particularly for widely accessible 7B-size LLMs. The core idea behind CREAM is to avoid forcing the model to be overly confident when distinguishing between responses of similar quality. But how can we determine if the preference labeling is reliable? Outside the self-rewarding scenario, we could employ a pool of external reward models to assist in ranking preferences. When two responses are of similar quality, these external models often yield inconsistent rankings, which signal the confidence level of the preference labeling. In self-rewarding scenarios, however, integrating such external reward models is not feasible. Fortunately, due to the iterative nature of self-rewarding training, we can use the reward model from the previous iteration to rank preferences and then compare these rankings with those produced by the current model. This comparison provides an estimate of such consistency rate. By leveraging this consistency rate, we can regularize the preference training to prevent the model from learning from unreliable preference data, thereby mitigating the rewarding bias issue in SRLMs.

In summary, we first formulate a generalized iterative preference fine-tuning framework to analyze the rewarding bias issue in SRLMs. From the insights of theoretical analysis, we propose CREAM as the primary contribution of this paper. CREAM leverages the rewarding consistency across different iterations for regularized preference training, which can effectively mitigate the rewarding bias issue in SRLMs. Empirical results on a series of natural language benchmarks validate the effectiveness of CREAM in mitigating the rewarding bias issue and enhancing the alignment performance of LLMs.

Notations. Vectors are denoted by lowercase boldface letters, such as $\mathbf{x}$, and matrices by uppercase boldface letters, such as $\mathbf{A}$. For any positive integer $k$, the set ${1,2,\dots,k}$ is denoted by $[k]$. Sets are denoted by calligraphic uppercase letters, such as $\mathcal{D}$, with the cardinality of the set represented as $|\mathcal{D}|$. Without ambiguity, we denote $\pi_{\bm{\theta}}$ as the language model parameterized by $\bm{\theta}$, $\mathbf{x}$ as the input prompt, and $\mathbf{y}$ as the output response from the language model. All other notations are defined prior to their first usage. We denote $\operatorname{\mathds{1}}[\cdot]$ as the indicator function.

## 2 Related Works

This paper primarily focuses on addressing the rewarding bias issue in self-rewarding language models (SRLMs) (Yuan et al., ), a type of self-improvement method for LLM alignment. In this section, we provide an overview of recent advancements in LLM alignment and delve into a detailed discussion of SRLMs.

LLM Alignment.
Alignment is at the core of LLM research and applications, with the goal of ensuring that LLMs adhere to human values and preferences. RLHF established the foundational alignment training paradigm (Leike et al., ; Ziegler et al., ; Ouyang et al., ), where it leverages human preference feedback to train a reward model, and then use this reward model to guide the LLM via reinforcement learning algorithms (Schulman et al., ). Recent efforts have been made to develop direct alignment methods (Rafailov et al., ; Dong et al., ; Azar et al., ; Ethayarajh et al., ; Meng et al., ; Hong et al., ; Zhou et al., ), in order to reduce the costs and complexity of RLHF and make it more efficient and accessible. Representatively, DPO (Rafailov et al., ) as a representative direct alignment method, optimizes the LLM with annotated preference pairs, eliminating the need of training an additional reward model. However, most RLHF and direct alignment methods heavily rely on human-annotated preference data, where the data collection commonly involves human to distinguish the “good” responses from the “bad” ones, which is time-consuming and labor-intensive (Ouyang et al., ; Bai et al., ). Thus, synthesizing preference data with minimal human efforts has become a valuable research direction.

Self-Rewarding Language Model. SRLM (Yuan et al., ) has emerged as a promising approach to address the challenge of preference data synthesis in a self-improvement manner.
This method leverages the LLM itself to act as both the policy model and the reward model. The policy model can generate response candidates for unlabeled prompts, while the reward model uses LLM-as-A-Judge (Zheng et al., ; Bai et al., ; Dubois et al., ) prompting to reward and rank these responses based on their quality. The ranked responses are then used as preference pairs to train the LLM via DPO (Rafailov et al., ).
And this process can be iteratively repeated to improve the alignment performance without human intervention.
However, having the same LLM serve as both the policy and reward model, without any regularization, presents challenges in guaranteeing accurate rewards. This can lead to accumulated bias and noisy preference data, which ultimately harms the training.
Other similar self-improvement methods (Huang et al., ; Zelikman et al., ; Chen et al., ; Guo et al., ; Zhou et al., ; Wang et al., ) often either use the ground truth response to avoid annotation bias, or introduce an additional reward model to reduce the noise in annotations. In contrast, our work neither requires labeled data nor relies on external LLMs. Instead, we propose to use the rewarding consistency to mitigate the rewarding bias in SRLMs.

## 3 Methodology

In this section, we first formulate the generalized iterative preference fine-tuning framework for self-rewarding, RL with AI feedback, and other iterative preference fine-tuning methods.
Next, we introduce the motivation behind the proposed consistency regularized self-rewarding method.
Finally, we present the practical implementation algorithm of CREAM in detail.

### 3.1 Generalized Iterative Preference Fine-Tuning Framework

We assume that we can access the dataset with response $\mathcal{D}_{\text{S}}$ and the prompt dataset without response $\mathcal{D}_{\text{U}}$. The objective is to iteratively minimize the following loss with respect to the neural network parameter $\bm{\theta}$ and a label function $z$ as

$$ $\displaystyle\mathcal{L}(\bm{\theta},z)$ $\displaystyle=\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{S}})+ \mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{U}};\mathbf{y},\mathbf{y}^{\prime }\sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}[\mathcal{L}_{\text{DPO}}(\bm{ \theta};\mathbf{y},\mathbf{y}^{\prime},\mathbf{x},z)].$ (3.1) $$

where the first term $\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{S}})$ aligns the model $\pi_{\bm{\theta}}$ to the SFT data. We note here that any potential SFT methods (Ouyang et al., ; Yuan et al., ; Dong et al., ; Chen et al., ), or the methods without SFT data ($\mathcal{L}_{\text{SFT}}=0$) can be adapted in this framework. The second term $\mathbb{E}[\mathcal{L}_{\text{DPO}}]$ corresponds to learning from the preference data pair {$\mathbf{y},\mathbf{y}^{\prime}$} generated by the current model $\bm{\theta}_{t}$. The labeling function $z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})\in\{0,1\}$ provides the preference judgment between $\mathbf{y}$ and $\mathbf{y}^{\prime}$ for the DPO loss, where $z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})=1$ means $\mathbf{y}\succ\mathbf{y}^{\prime}$ and $z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})=0$ means $\mathbf{y}\prec\mathbf{y}^{\prime}$. The DPO loss $\mathcal{L}_{\text{DPO}}$ is defined as follows:

$$ $\displaystyle\mathcal{L}_{\text{DPO}}(\bm{\theta};\mathbf{y},\mathbf{y}^{ \prime},\mathbf{x},z)$ $\displaystyle=-z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})\log\sigma\left( \log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})}{\pi_{\text{ref}}( \mathbf{y}|\mathbf{x})}\right)-\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}^{ \prime}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x})}\right)\right)$ $\displaystyle\quad-(1-z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x}))\log\sigma \left(\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}^{\prime}|\mathbf{x})}{\pi_{ \text{ref}}(\mathbf{y}^{\prime}|\mathbf{x})}\right)-\log\left(\frac{\pi_{\bm{ \theta}}(\mathbf{y}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})} \right)\right),$ (3.2) $$

where $\pi_{\text{ref}}$ is the reference model for KL divergence regularization, and $\sigma(\cdot)$ is the sigmoid function.
The proposed loss $\mathcal{L}(\bm{\theta},z)$ in Eq. (3.1) represents all iterative preference fine-tuning algorithms. For the reinforcement learning (RL) with human feedback (Ouyang et al., ), $z$ is the human preference comparing $\mathbf{y}$ and $\mathbf{y}^{\prime}$.
For the RL with AI feedback, $z$ is the oracle reward model like GPT-4 (Achiam et al., ).
For the self-rewarding language model (Chen et al., ), $z$ is given by comparing the reward score generated from the language model itself, often with LLM-as-a-Judge prompting.
However, as aforementioned, we note that such prompt rewarding method may only be feasible for larger and advanced LLMs such as Llama-70B (Touvron et al., ). For smaller models such as Llama-7B that do not have complex instruction following and reasoning abilities, we instead propose to leverage the intrinsic reward model (Rafailov et al., )

$$ $\displaystyle r_{\bm{\theta}}(\mathbf{x},\mathbf{y})\propto[\log\pi_{\bm{ \theta}}(\mathbf{y}|\mathbf{x})-\log\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})]$ $$

to reward and rank the responses for annotating preference pairs.
Therefore, the choice of preference labeling function $z$ is closely connected with the language model parameter $\bm{\theta}$. Then, we introduce the following two-step optimization algorithm to solve Eq. (3.1).

Step 1. (Preference-labeling step) Keep $\bm{\theta}=\bm{\theta}_{t}$ fixed, select function $z$ to minimize $\mathcal{L}_{\text{DPO}}$. In particular, letting $\bm{\theta}=\bm{\theta}_{t}$ in Eq. (3.2), solution for $z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})=\operatorname*{arg\,min}_{z}
\mathcal{L}_{\text{DPO}}(\bm{\theta}_{t};\mathbf{y},\mathbf{y}^{\prime},
\mathbf{x},z)$ is

$$ $\displaystyle z_{t+1}(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})=\operatorname {\mathds{1}}\left[\log\pi_{\bm{\theta}_{t}}(\mathbf{y}|\mathbf{x})-\log\pi_{ \text{ref}}(\mathbf{y}|\mathbf{x})\geq\log\pi_{\bm{\theta}_{t}}(\mathbf{y}^{ \prime}|\mathbf{x})-\log\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x})\right].$ (3.3) $$

Step 2. (Learning step) Keep $z$ as of Eq. (3.3), minimize loss function $\mathcal{L}(\bm{\theta},z_{t+1})$ with respect to $\bm{\theta}$ and get $\bm{\theta}_{t+1}=\operatorname*{arg\,min}_{\bm{\theta}}\mathcal{L}(\bm{\theta
},z_{t+1})$.

Different from existing methods, the proposed two-step optimization method directly uses the intrinsic reward model to generate the preference data.
This approach is particularly feasible for smaller LLMs, which lack the capacity to effectively use LLM-as-a-Judge prompts (Zheng et al., ) for rewarding and ranking.
We note that the proposed two-step method is similar to the Expectation-Maximization algorithm and self-training paradigm (Zou et al., ). This similarity is supported by the following theorem, which suggests the convergence of the proposed two-step algorithm.

###### Theorem 3.1 .

Suppose the optimization $\bm{\theta}_{t+1}=\operatorname*{arg\,min}_{\bm{\theta}}\mathcal{L}(\bm{\theta
},z_{t+1})$ is solvable and the SFT loss $\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{S}})\geq 0$ for all $\bm{\theta}$ and $\mathcal{D}_{\text{S}}$, the proposed two-step optimization method converges.

###### Theorem 3.1 .

### 3.2 Consistency Regularized Self-Rewarding

The generalized framework presented in Eq. (3.1) assumes the human feedback or GPT-4 are all reliable so that the preference labeling function $z$ is trustworthy.
However, for SRLMs, the accuracy of preference labeling is not always guaranteed.
Therefore, treating all selected preference labels as “ground truth” by encoding them as hard labels can lead to overconfident mistakes, potentially propagating biases and inaccuracies from the LLMs.
Taking Figure 1 as an example, both the two responses $\mathbf{y}$ and $\mathbf{y}^{\prime}$ are judged by humans to be of high quality.
Forcing the model to be overly confident in distinguishing between these two responses $\{\mathbf{y},\mathbf{y}^{\prime}\}$ with similar quality can negatively impact the performance of SRLMs during training.

This rewarding bias issue motivates us to mitigate such ambiguity by introducing a consistency-regularized self-rewarding language model, CREAM.
Specifically, for a pair of responses with very similar quality, their oracle reward scores should ideally be very close to each other.
Particularly, when multiple reward models are available, it is likely that some models will rank one response as superior, while others may rank the opposite response as better, resulting in high ranking inconsistency (i.e., low ranking consistency) among these models.
Based on this, CREAM aims to prevent the model from learning from preference pairs with low consistency. Instead, it focuses solely on preference pairs with high consistency across different reward models, thereby mitigating the rewarding bias issue and stabilizing the learning process to some extent.
From the theoretical perceptive, we can introduce a regularization term to Eq. (3.1) as

$$ $\displaystyle\mathcal{L}(\bm{\theta},z)$ $\displaystyle=\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{S}})+ \mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{U}};\mathbf{y},\mathbf{y}^{\prime }\sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}[\mathcal{L}_{\text{DPO}}(\bm{ \theta};\mathbf{y},\mathbf{y}^{\prime},\mathbf{x},z)+\lambda\mathcal{L}_{\text {Reg}}(\bm{\theta};\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})],$ (3.4) $$

where the regularization term $\mathcal{L}_{\text{Reg}}(\bm{\theta};\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})$ prevents the model $\pi_{\bm{\theta}}$ from overconfident in distinguishing the preference of $\{\mathbf{y},\mathbf{y}^{\prime}\}$ with similar quality, which is quantified in the following lemma.

###### Lemma 3.2 .

Let the random variable $z=z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})$ be defined as $z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})=\operatorname{\mathds{1}}[\mathbf
{y}\succ\mathbf{y}^{\prime}|\mathbf{x}]$. The Bradley-Terry model (Bradley & Terry, ) for the probability of $z$ under parameter $\bm{\theta}$ is given by

$$ $\displaystyle P_{\bm{\theta}}(z)=P_{\bm{\theta}}(\operatorname{\mathds{1}}[ \mathbf{y}\succ\mathbf{y}^{\prime}|\mathbf{x}])=\sigma\left(\log(\pi_{\bm{ \theta}}(\mathbf{y}|\mathbf{x})/\pi_{\text{ref}}(\mathbf{y}|\mathbf{x}))-\log( \pi_{\bm{\theta}}(\mathbf{y}^{\prime}|\mathbf{x})/\pi_{\text{ref}}(\mathbf{y}^ {\prime}|\mathbf{x}))\right),$ $$

Letting the regularization $\mathcal{L}_{\text{Reg}}$ be defined by

$$ $\displaystyle\mathcal{L}_{\text{Reg}}(\bm{\theta};\mathbf{y},\mathbf{y}^{ \prime},\mathbf{x})$ $\displaystyle=-\log\sigma\left(\log(\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})/ \pi_{\text{ref}}(\mathbf{y}|\mathbf{x}))-\log(\pi_{\bm{\theta}}(\mathbf{y}^{ \prime}|\mathbf{x})/\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x}))\right)$ $\displaystyle\quad-\log\sigma\left((\log\pi_{\bm{\theta}}(\mathbf{y}^{\prime}| \mathbf{x})/\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x}))-(\log\pi_{\bm{ \theta}}(\mathbf{y}|\mathbf{x})/\pi_{\text{ref}}(\mathbf{y}|\mathbf{x}))\right).$ (3.5) $$

Then the expected regularized loss under the model $\bm{\theta}_{t}$ is given by:

$$ $\displaystyle\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t }}(\cdot|\mathbf{x})}\mathcal{L}_{\text{reg}}(\bm{\theta};\mathbf{y},\mathbf{y }^{\prime},\mathbf{x})=2\,\mathbb{KL}(u(\cdot)\parallel P_{\bm{\theta}}(\cdot)),$ (3.6) $$

where $u(z)$ is the uniform binary distribution, i.e., $u(z=0)=u(z=1)=0.5$.

As Lemma 3.2 suggests, the $\mathcal{L}_{\text{Reg}}$ will regularize the preference between
$\{\mathbf{y},\mathbf{y}^{\prime}\}$ that has similar quality to a uniform distribution.
Then the following theorem suggests that using $\mathcal{L}_{\text{DPO}}+\lambda\mathcal{L}_{\text{Reg}}$ corresponds to the soft-labeled DPO which we implemented in CREAM.

###### Theorem 3.3 .

For all $\mathbf{y},\mathbf{y}^{\prime},\mathbf{x},z$, minimizing

$$ $\displaystyle\mathcal{L}(\bm{\theta},z)=\mathcal{L}_{\text{SFT}}(\bm{\theta}; \mathcal{D}_{\text{SFT}})+\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{U}}; \mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})} \left[\mathcal{L}_{\text{DPO}}(\bm{\theta};\mathbf{y},\mathbf{y}^{\prime}, \mathbf{x},z)+\lambda\mathcal{L}_{\text{Reg}}(\bm{\theta};\mathbf{y},\mathbf{y }^{\prime},\mathbf{x})\right]$ $$

is equivariant with minimizing

$$ $\displaystyle\mathcal{L}(\bm{\theta},z)$ $\displaystyle=\frac{1}{1+2\lambda}\mathcal{L}_{\text{SFT}}(\bm{\theta}; \mathcal{D}_{\text{S}})$ $\displaystyle\quad+\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{U};\mathbf{y}, \mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}[\mathcal{C}_{ \lambda}\mathcal{L}_{\text{DPO}}(\bm{\theta};\mathbf{y},\mathbf{y}^{\prime}, \mathbf{x},z)+(1-\mathcal{C}_{\lambda})\mathcal{L}_{\text{DPO}}(\bm{\theta}; \mathbf{y},\mathbf{y}^{\prime},\mathbf{x},1-z)],$ (3.7) $$

where the $1-z$ reverses the preference order of $z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})$ and $\mathcal{C}_{\lambda}=(1+\lambda)/(1+2\lambda)$.

Theorem 3.3 suggests that instead of calculating the regularization term $\mathcal{L}_{\text{Reg}}$, we can use the soft-labeled DPO to train Eq. (3.7). In particular, when $\lambda=0$, $\mathcal{C}_{\lambda}=0$ and Eq. (3.7) degenerates to Eq. (3.1). This represents the case where the preference label $z$ is trustworthy from human or some oracle reward models (e.g., GPT-4). In other words, $\lambda$ represents the *confidence* of the label function $z$.
Specially, since in our two-step optimization paradigm, the label function $z$ is directly derived from the previous model $\pi_{\bm{\theta}_{t}}$, we can measure the performance of $\pi_{\bm{\theta}_{t}}$ using the consistency between model $\bm{\theta}_{t}$ and the baseline reward model (e.g., external reward model) $\bm{\theta}_{t}^{\prime}$, defined by

$$ $\displaystyle\lambda(\mathbf{x})=2\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime} \sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}\operatorname{\mathds{1}}[\mathbf{ y}\succ\mathbf{y}^{\prime}|\mathbf{x},\bm{\theta}_{t}]\operatorname{\mathds{1} }[\mathbf{y}\succ\mathbf{y}^{\prime}|\mathbf{x},\bm{\theta}_{t}^{\prime}],$ (3.8) $$

and when $\lambda\rightarrow 0$, $\mathcal{C}_{\lambda}\approx 1-\lambda$ representing the consistency of model $\bm{\theta}_{t}$ and $\bm{\theta}_{t}^{\prime}$. $\operatorname{\mathds{1}}[\mathbf{y}\succ\mathbf{y}^{\prime}|\mathbf{x},\bm{
\theta}_{t}]$ means the response $\mathbf{y}$ is better than $\mathbf{y}^{\prime}$ given the prompt $\mathbf{x}$ and language model parameter $\bm{\theta}_{t}$, i.e.,

$$ $\displaystyle\operatorname{\mathds{1}}[\log(\pi_{\bm{\theta}_{t}}(\mathbf{y}| \mathbf{x})/\pi_{\text{ref}}(\mathbf{y}|\mathbf{x}))-\log(\pi_{\bm{\theta}_{t} }(\mathbf{y}^{\prime}|\mathbf{x})/\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf {x}))],$ $$

and similar definition applies to $\operatorname{\mathds{1}}[\mathbf{y}\succ\mathbf{y}^{\prime}|\mathbf{x},\bm{
\theta}_{t}^{\prime}]$.

Figure: Figure 2: The flow of CREAM. In the response sampling stage, the policy model $\pi_{\bm{\theta}_{t}}$ generates $N$ responses. After that, CREAM uses the reward model $R_{\bm{\theta}_{t-1}}$ from the previous iteration to reward and rank these responses. Then, the rankings are compared with those generated by current reward model $R_{\bm{\theta}_{t}}$ to estimate the consistency rate. Finally, the policy model $\pi_{\bm{\theta}_{t}}$ is fine-tuned with consistency regularized preference training objective, resulting in the model $\pi_{\bm{\theta}_{t+1}}$ for next iteration.
Refer to caption: x2.png

Figure: Algorithm 1 Consistency-Regularized Self-Rewarding Language Model

###### Lemma 3.2 .

###### Theorem 3.3 .

### 3.3 Proposed Algorithm

Equipped with the above two-stage optimization and the consistency-regularized self-rewarding, we are ready to present the implementation of CREAM in Algorithm 1.
The whole framework of CREAM is also illustrated in Figure 2.
The algorithm starts from the SFT training to obtain the first model parameter $\bm{\theta}_{1}$ in Line 2.
A similar approach is applied in Yuan et al. () to avoid calculating the $\mathcal{L}_{\text{SFT}}$ in the future optimization steps.
Then for each $\mathbf{x}_{j}$ in the unlabeled prompt set $\mathcal{D}_{\text{U}}$, $N$ response candidates $\{\mathbf{y}_{i}\}_{i=1}^{N}$ are sampled in Line 5. Then reward scores of these $N$ candidates can be calculated according to Rafailov et al. () by

$$ $\displaystyle r_{ij}=\beta[\log\pi_{\bm{\theta}_{t}}(\mathbf{y}_{ij}|\mathbf{x }_{j})-\log\pi_{\bm{\theta}_{0}}(\mathbf{y}_{ij}|\mathbf{x}_{j})]+\beta\log Z( \mathbf{x}_{j}),$ (3.9) $$

where we use the initial model parameter $\bm{\theta}_{0}$ as the reference policy $\pi_{\text{ref}}$.
Since $\beta\geq 0$ and $\log Z(\mathbf{x}_{j})$ is a constant across different response $\mathbf{y}_{i}$ for the same input prompt $\mathbf{x}_{j}$, we can drop these factors and calculate rewards in Line 6.
Specifically, when $t=1$, the rank $K_{ij}$ is calculated based on the reference policy $\bm{\theta}_{0}$ itself. Thus we instead use the likelihood $r_{ij}=\log\pi_{\bm{\theta}_{0}}(\mathbf{y}_{ij}|\mathbf{x}_{j})$ as the reward for this edge case.
The rank for these $N$ candidates are therefore obtained in Line 7, where $J_{ij}$ means response $\mathbf{y}_{ij}$ is in the $J_{ij}$-th best in the preference list of $\mathbf{x}_{j}$.

##### Consistency-Regularized Self-Rewarding.

As discussed in Eq. (3.8), a baseline reward model is required to measure the consistency.
In the self-rewarding scenario, it is infeasible to add an external reward model as the baseline reward model.
Fortunately, we can employ the model before last update $\bm{\theta}_{t-1}$ as the baseline reward model $\bm{\theta}_{t}^{\prime}$ (i.e., last iteration’s model) for evaluating the consistency of the model $\bm{\theta}$, thanks to chances provided by iterative training manner.
Such a procedure helps mitigate the training error introduced in $t-1$-th step before obtaining $\bm{\theta}_{t}$. Considering a pair of tied preference pair $\mathbf{y},\mathbf{y}^{\prime}$ both performing well, as demonstrated in Figure 1. $P[\mathbf{y}\succ\mathbf{y}^{\prime}|\mathbf{x},\bm{\theta}_{t}]$ will be oscillating around $0.5$ when $t$ grows due to the random noise. Otherwise $P[\mathbf{y}\succ\mathbf{y}^{\prime}|\mathbf{x},\bm{\theta}_{t}]$ might consistently converge to 0 0 or $1$. Due to this oscillation, the consistency between $\bm{\theta}_{t-1}$ and $\bm{\theta}_{t}$ on this specific preference pair would be low, and the algorithm will learn less from this noninformative preference pair thus stabilize this oscillation.

Specifically, we calculate the rank of these $N$ candidates using $\bm{\theta}_{t-1}$ in Line 9 and then use the Kendall’s Tau coefficient (Kendall, ) denoted by

$$ $\displaystyle\tau_{j}=\frac{2}{N(N-1)}\sum_{1\leq i<i^{\prime}\leq N}\left[ \operatorname{\mathds{1}}\left[(J_{ij}-J_{i^{\prime}j})(K_{ij}-K_{i^{\prime}j} )>0\right]-\operatorname{\mathds{1}}\left[(J_{ij}-J_{i^{\prime}j})(K_{ij}-K_{i ^{\prime}j})<0\right]\right].$ (3.10) $$

Kendall’s Tau coefficient is a widely used coefficient (McLeod, ; Abdi, ) to measure the consistency of two ranking sequences. Basically, when two sequences perfectly aligns, $\tau_{j}=1$ and when two sequence never aligns, $\tau_{j}=-1$. The following lemma draws a further connection between Kendall’s Tau and the regularization parameter $\lambda$ proposed in Section 3.2.

###### Lemma 3.4 .

Suppose the $N$ response candidate $\{\mathbf{y}_{ij}\}_{i}$ is i.i.d. given the prompt $\mathbf{x}_{j}$, then

$$ $\mathbb{E}[\tau_{j}]=1-4\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm {\theta}_{t}}(\cdot|\mathbf{x}_{j})}\operatorname{\mathds{1}}[\mathbf{y}\succ \mathbf{y}^{\prime}|\mathbf{x}_{j},\bm{\theta}_{t}]\operatorname{\mathds{1}}[ \mathbf{y}\prec\mathbf{y}^{\prime}|\mathbf{x}_{j},\bm{\theta}_{t-1}]=1-2\lambda,$ $$

where the expectation is taken over the randomness of sampling the $N$ candidate set.

Given Lemma 3.4, we can recover $\mathcal{C}_{\lambda}\approx 1-\lambda=(1+\tau_{j})/2$ and we use average all $\tau_{j}$ for all $\mathbf{x}_{j}\in\mathcal{D}_{\text{U}}$ in Line 11. Finally, in Line 12, we compose the preference dataset by selecting the best response $\mathbf{y}^{+}_{j}=\mathbf{y}_{i^{+}j}$ and the worst response $\mathbf{y}^{-}_{j}=\mathbf{y}_{i^{-}j}$ which is similar with (Yuan et al., ).

$$ $\displaystyle\mathcal{D}_{\text{DPO}}=\{(\mathbf{x}_{j},\mathbf{y}_{i^{+}j}, \mathbf{y}_{i^{-}j})|\mathbf{x}_{j}\in\mathcal{D}_{\text{U}},i^{+}= \operatorname*{arg\,min}_{i}J_{ij},i^{-}=\operatorname*{arg\,min}_{i}J_{ij}\}$ (3.11) $$

Following Theorem 3.3, we also prepare the reverse DPO dataset by switching the best response and the worst response by

$$ $\displaystyle\mathcal{D}_{\text{RDPO}}=\{(\mathbf{x}_{j},\mathbf{y}_{i^{-}j}, \mathbf{y}_{i^{+}j})|\mathbf{x}_{j}\in\mathcal{D}_{\text{U}},i^{+}= \operatorname*{arg\,min}_{i}J_{ij},i^{-}=\operatorname*{arg\,min}_{i}J_{ij}\}$ (3.12) $$

and update $\bm{\theta}_{t+1}$ by minimizing the empirical loss of Eq. (3.7) in Line 14.
The detailed proof of theorems and lemmas are provided in the Appendix A.

###### Lemma 3.4 .

## 4 Experiment

### 4.1 Experimental Setup

Data.
In our experiments, we use Open Assistant dataset (Köpf et al., ) and only reserve about 3.4K human-annotated examples as the seed SFT data $\mathcal{D}_{\text{S}}$.
To construct the unlabeled prompt dataset $\mathcal{D}_{\text{U}}$, we mix prompts of $\mathcal{D}_{\text{S}}$ with the train split of each downstream task (only reserve the prompts) including
(1) ARC-Easy/Challenge (Clark et al., ), a dataset of genuine grade-school level, multiple-choice science questions-answering dataset;
(2) OpenBookQA (Mihaylov et al., ), a dataset of elementary-level science questions designed to assess understanding of realistic scenario core facts;
(3) SIQA (Sap et al., ), a question-answering benchmark for testing models’ abilities to reason about the social implications of everyday events and situations;
(4) GSM8K (Cobbe et al., ), a primary school level mathematical dataset in which the questions take between 2 and 8 steps to solve.
Finally, this process results in a total of 21K prompts in $\mathcal{D}_{\text{U}}$, which we distribute equally across iterative self-rewarding trainings.

Models.
Due to limited computational resources, we mainly conduct experiments with two LLMs with about 7B parameters, including Llama-3 (Dubey et al., ) and Llama-2 (Touvron et al., ), both of which are widely used.
Note that the proposed framework is designed for LLMs with any sizes, while validating our findings on other LLMs will be our future work.

Baseline Methods.
To validate our findings, we mainly compare our method with SRLM (Yuan et al., ) which uses the same LLM to serve as both the policy and reward model to generate preference data for iterative training.
Additionally, we introduce a variant of RL with AI feedback (Guo et al., ), referred to as “Oracle”. In this variant, the reward model in SRLM is replaced with an external reward model to demonstrate the upper bound performance of SRLM.
Specifically, we use InternLM2 (Cai et al., ), a specialized trained reward model, to provide the reward scores for the generated responses. We further enhance Oracle’s rewarding by leveraging the labels from downstream tasks to improve the rewarding accuracy.

Implementation Details.
In our experiments, we fine-tune the initial model (M0) on the seed SFT data for $3$ epochs with a learning rate of $1e-6$, resulting in model M1.
Following SRLM approach, we then iteratively fine-tune the model using DPO in two additional iterations, producing models M2 and M3.
In the preference training of each iteration, we set $\beta=0.1$ of DPO loss, and fine-tune the model for $1$ epoch with a learning rate of $1e-6$.
All training processes use the AdamW optimizer (Loshchilov & Hutter, ) with a warmup ratio of $0.1$.
For the response sampling stage of all SRLM methods, we use a decoding temperature of $0.8$.
For evaluating downstream tasks, we use greedy decoding.
All experiments are conducted primarily on NVIDIA A6000 GPUs.

### 4.2 Main Results

**Table 1: Main results of each method on test sets of downstream tasks. The exact match accuracies are reported. The “↑” and “↓” indicate the performance improvement and degradation compared to the method’s last iteration (e.g., M1 $\rightarrow$ M2 and M2 $\rightarrow$ M3), respectively. The best performance among methods using self-rewarding is highlighted in bold.**
| Model | Method | Reward | Iteration | Arc-Easy | Arc-Challenge | OpenBookQA | SIQA | GSM8K |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3 | Initial | - | M0 | 86.29 | 80.37 | 86.00 | 68.58 | 78.01 |
| SFT | - | M1 | 86.78 | 80.14 | 86.40 | 69.50 | 78.39 |  |
| Oracle | External | M2 | 89.60 ↑ | 82.17 ↑ | 90.00 ↑ | 72.88 ↑ | 80.82 ↑ |  |
| M3 | 89.31 ↓ | 81.31 ↓ | 90.20 ↑ | 73.75 ↑ | 76.04 ↓ |  |  |  |
| SRLM | Self | M2 | 87.79 ↑ | 80.38 ↑ | 87.80 ↑ | 70.95 ↑ | 78.01 ↓ |  |
| M3 | 87.17 ↓ | 81.23 ↑ | 87.30 ↓ | 70.37 ↓ | 77.48 ↓ |  |  |  |
| CREAM | Self | M2 | 88.89 ↑ | 80.89 ↑ | 88.00 ↑ | 69.79 ↑ | 81.04 ↑ |  |
| M3 | 89.52 ↑ | 83.36 ↑ | 90.20 ↑ | 72.06 ↑ | 81.73 ↑ |  |  |  |
| Llama-2 | Initial | - | M0 | 61.07 | 48.98 | 62.20 | 50.36 | 23.65 |
| SFT | - | M1 | 60.44 | 48.46 | 63.20 | 50.77 | 23.88 |  |
| Oracle | External | M2 | 70.20 ↑ | 55.03 ↑ | 75.40 ↑ | 63.66 ↑ | 30.02 ↑ |  |
| M3 | 71.72 ↑ | 55.80 ↑ | 77.20 ↑ | 62.44 ↓ | 29.57 ↓ |  |  |  |
| SRLM | Self | M2 | 58.67 ↓ | 46.67 ↓ | 59.80 ↑ | 49.69 ↓ | 25.17 ↑ |  |
| M3 | 46.55 ↓ | 34.47 ↓ | 49.20 ↓ | 48.06 ↓ | 22.14 ↓ |  |  |  |
| CREAM | Self | M2 | 58.97 ↓ | 47.53 ↓ | 62.80 ↓ | 50.43 ↓ | 24.41 ↑ |  |
| M3 | 62.08 ↑ | 48.81 ↑ | 64.60 ↑ | 51.22 ↑ | 25.85 ↑ |  |  |  |

The main results are shown in Table 1. From these results, we observe the following:
(1) The Standard SRLM fails to achieve satisfactory performance, particularly with Llama2 which has relatively weaker foundation performance even after SFT fine-tuning (M0 $\rightarrow$ M1), which indicates its limitations for 7B-level LLMs. (2) Compared to SRLM, CREAM achieves a significant improvement across almost all downstream tasks, showing the advantage of introducing the proposed regularization method. (3) SRLM equipped with an external oracle reward model (“Oracle”) can ensure high rewarding accuracy for annotations of self-generated preference data, thereby achieving the best performance overall. This highlights a critical challenge for all methods with unreliable rewarding. Notably, for Llama3, CREAM even outperforms Oracle except on SIQA dataset, showcasing the superior performance of CREAM. This superiority underlines the success of the proposed regularization in mitigating the rewarding bias issue.

Figure: Figure 3: Arena of CREAM v.s. SRLM and Oracle with Llama3, judged by GPT-4o.
Refer to caption: x3.png

(4) The consistent performance improvements of CREAM across iterations (M1 $\rightarrow$ M2, M2 $\rightarrow$ M3) validate the effectiveness of our regularization in mitigating the rewarding bias issue. This also demonstrates the benefits of introducing such internal ranking consistency as a regularization signal.

Alignment Arena. To more directly compare the alignment performance of CREAM, we further show the Arena win-rate of our method in Figure 3. We can find that CREAM can beat baseline methods with the same iteration, which confirms the advantage of the proposed regularized self-rewarding. Also, the win and tie rates of CREAM increase with the iteration against stronger model Oracle, indicating the consistent performance improvements across iterations.

### 4.3 Analysis

In this section, we provide a detailed analysis to validate and understand the contribution of CREAM in improving the performance via iterative preference fine-tuning manner.

#### 4.3.1 Analysis of Rewarding

Rewarding Consistency. We first examine the rewarding consistency of different methods using their corresponding models from the last iteration in Table 2.
Here, we use the proposed Consistency Rate $\mathcal{C}$, Kendall correlation coefficient $\tau$, Spearman correlation coefficient, and TopOrder metrics to measure the consistency, where the TopOrder metric evaluates whether the final paired preference data remains the same, calculated as follows:

$$ $\displaystyle\text{TopOrder}_{j}=\operatorname{\mathds{1}}\left[\operatorname* {arg\,min}J_{j}=\operatorname*{arg\,min}K_{j}\right]\cdot\operatorname{\mathds {1}}\left[\operatorname*{arg\,max}J_{j}=\operatorname*{arg\,max}K_{j}\right],$ $$

where $J_{j}$ and $K_{j}$ are the rankings of the responses provided by the current model and the last iteration’s model, respectively. This metric assesses whether both the least preferred and most preferred responses are consistently ranked across iterations. The results confirm that SRLMs exhibit a rewarding consistency issue, indicating that the generated preference data may not be reliable for training. In contrast, CREAM can keep the ranking consistency across iterations thanks to the explicit regularization in the training.

**Table 2: Ranking consistency of CREAM and SRLM across iterations using Llama3.**
| Iterations | Method | Consistency $\mathcal{C}$↑ | Kendall $\tau$ ↑ | Spearman ↑ | TopOrder ↑ |
| --- | --- | --- | --- | --- | --- |
| M2 vs M1 | SRLM | 0.39 ± 0.21 | -0.22 ± 0.41 | 0.36 ± 0.24 | 0.03 ± 0.18 |
| CREAM | 0.73 ± 0.18 | 0.46 ± 0.35 | 0.77 ± 0.19 | 0.19 ± 0.39 |  |
| M3 vs M2 | SRLM | 0.46 ± 0.19 | -0.08 ± 0.38 | 0.50 ± 0.22 | 0.12 ± 0.33 |
| CREAM | 0.92 ± 0.09 | 0.84 ± 0.19 | 0.95 ± 0.07 | 0.59 ± 0.49 |  |

Prompt Rewarding v.s. DPO Rewarding.
As aforementioned, 7B-level LLMs struggle with generating accurate rewards when using LLM-as-a-Judge prompting (Zheng et al., ) due to their limited capacities.
We validate this finding in Figure 4 and Figure 5 which compare the ranking accuracy and model performance with prompt rewarding (P-SRLM) against DPO rewarding (SRLM and CREAM) using Llama3, respectively.
The ranking accuracy results include the performance on a curated in-domain Preference Data and out-of-domain RewardBench (Lambert et al., ) dataset, both of which are formulated as a ranking task to predict the preferred one between two responses.
The results suggest that prompt rewarding is not effective for smaller LLMs, as the model performance starts to decrease at the first iteration when trained on the self-rewarded preference data, though the ranking accuracy raises slightly.
In contrast, DPO rewarding methods, particularly CREAM, show better performance in both ranking accuracy and five downstream tasks.
This is because DPO rewarding is intrinsically aligned with the model’s learning objective.

Ranking Accuracy. The ranking accuracy is crucial for self-rewarding methods, as it directly affects the quality of the preference data for training.
Figure 4 directly compares the ranking accuracy of different methods.
The Preference Data is curated by collecting self-generated preference pairs, followed by using GPT-4o to annotate the ranking labels. This dataset is used to evaluate the model’s in-domain performance.
The RewardBench dataset is used to assess the generalizability of the models beyond the training domain.
CREAM consistently achieves higher ranking accuracy than baseline methods, which promises more reliable preference data for training.
Even though the overall ranking accuracy ($\leq 70\%$) is not satisfying, the introduced consistency regularization can help mitigate the negative impacts of potentially noisy ranking annotations during preference fine-tuning.

Figure: Figure 4: Pairwise ranking accuracy on RewardBench and a curated Preference Data. P-SRLM is SRLM with prompt rewarding (LLM-as-a-Judge).

Figure: Figure 5: Average performance of SRLM, P-SRLM, and CREAM. The detailed performance is shown in Table 1 and Table 6.

#### 4.3.2 Reliability of Self-Consistency

**Table 3: Comparison of CREAM using oracle reward model and last iteration’s model. BRM denotes the choice of baseline reward model.**
| Method | BRM | Arc-E | Arc-C | OBQA | SIQA | GSM8K |
| --- | --- | --- | --- | --- | --- | --- |
| Llama3 M1 | - | 86.78 | 80.14 | 86.40 | 69.50 | 78.39 |
| CREAM M2 | M0 | 88.89 | 80.89 | 88.00 | 69.79 | 81.04 |
| CREAM M2 | Oracle | 88.51 | 81.06 | 86.20 | 72.21 | 79.91 |
| Llama2 M1 | - | 60.44 | 48.46 | 63.20 | 50.77 | 23.88 |
| CREAM M2 | M0 | 58.97 | 47.53 | 62.80 | 50.43 | 24.41 |
| CREAM M2 | Oracle | 62.42 | 48.72 | 66.00 | 51.13 | 22.52 |

The most straightforward way to enhance the rewarding and ranking accuracy is by incorporating external reward models, such as the SRLM variant “Oracle” used in our experiments. Recall the theoretical analysis in Eq. (3.8) suggests that we can mitigate the rewarding bias issue by calculating the ranking consistency between the current model and other available baseline reward models (BRMs).
However, it is not feasible to have access to external reward models acting as BRMs in the self-rewarding scenario.
Thus, we instead propose to use the last iteration’s model as the BRM to measure the rewarding consistency.

Choice of Baseline Reward Model.
To measure the impacts on CREAM of using different BRMs, we fine-tune the M1 model using CREAM with two different BRMs: the rewarding function of Oracle and the model from the last iteration (M0).
As shown in Table 3, using a strong reward model as the BRM can bring a better regularization effect, especially for Llama2.
However, we find that the last iteration’s model also provides a reasonably reliable consistency signal for Llama3.
We attribute this to Llama3’s inherently stronger foundational alignment and better internal consistency, which allows it to effectively utilize itself without needing an external reward model.

#### 4.3.3 Consistency Measurement

**Table 4: Performance of CREAM using different consistency measurements with Llama3.**
| Iteration | Method | Arc-E | Arc-C | OBQA | SIQA | GSM8K |
| --- | --- | --- | --- | --- | --- | --- |
| M1 | - | 86.78 | 80.14 | 86.40 | 69.50 | 78.39 |
| M2 | Spearman | 86.95 | 82.00 | 85.40 | 70.05 | 78.77 |
| TopOrder | 87.25 | 80.12 | 86.88 | 70.83 | 79.75 |  |
| Kendall (Ours) | 88.89 | 80.89 | 88.00 | 69.79 | 81.04 |  |
| M3 | Spearman | 88.76 | 81.83 | 90.00 | 70.98 | 79.15 |
| TopOrder | 88.51 | 80.37 | 87.40 | 71.03 | 79.76 |  |
| Kendall (Ours) | 89.52 | 83.36 | 90.20 | 72.06 | 81.73 |  |

Besides the adopted Kendall $\tau$ coefficient, other metrics can also be used to measure the consistency between two preference ranking lists, such as Spearman coefficient (Spearman, ) and the aforementioned TopOrder method.
Table 4 compares the performance of CREAM using different consistency measurements.
We can observe that:
(1) Compared to M1, all these measurements are effective with CREAM, indicating the generalization and applicability of our regularized training.
(2) Kendall correlation coefficient generally yields higher scores across various datasets compared to Spearman and TopOrder measurement methods.
(3) The differences in performance highlight the sensitivity of these consistency measurements. Specifically, the Spearman coefficient appears slightly less robust than Kendall’s $\tau$, as analyzed in Croux & Dehon (). Meanwhile, TopOrder focuses only on top-1 and bottom-1 rankings, limiting its evaluation scope.

#### 4.3.4 Case Study

**Table 5: Case study of the generated response sampled from SRLM and CREAM models.**
| Question | Today is the 5th of March, what is the date in 6 weeks? |  |
| --- | --- | --- |
| Method | Response sampled from M2 | Ranking |
| SRLM | Today is March 5th.\nIn 6 weeks, we can count the days as follows:\n* March: 5, 6, 7, 8, 9, 10, $\cdots$, 31 = 31 days\n* April: 1, 2, 3,$\cdots$, 12 = 12 days\n* Total days = 31 + 12 = 43 days\nSince there are 7 days in a week, we can divide the total days by 7 to find the number of weeks:\n* 43 days / 7 days/week = 6.14 weeks\nSince we can’t have a fraction of a week, we round up to the nearest whole number. Therefore, 6 weeks from March 5th would be:\n* March: 31\n* April: 1, 2, 3, 4, 5\nThe date in 6 weeks would be April 5th. Human Correctness Judgment: ✗ | M2: 1st<br>$\mathbf{\neq}$<br>M3: 3rd |
| CREAM | Let’s count the days:\nToday is March 5th.\nIn 6 weeks, we’ll add:\n* 6 weeks = 6 x 7 days/week = 42 days\nSo, we’ll add 42 days to March 5th:\nMarch 5th + 42 days = April 16th\nThe date in 6 weeks is April 16th! Human Correctness Judgment: ✔ | M2: 1st<br>$\mathbf{=}$<br>M3: 1st |

Table 5 presents a case study to show the quality of the generated responses and the consistency of rankings across iterations.
We find that the quality of top-ranked response generated by CREAM is significantly better than that of SRLM, showing more accurate and concise reasoning.
It also shows that CREAM provides consistent rankings between M2 and M3, while SRLM exhibits inconsistent rankings, possibly due to M3 being fine-tuned on noisy preference data annotated by M2.

## 5 Conclusion

In this paper, we first formulate a generalized iterative preference fine-tuning framework for self-rewarding language models (SRLMs).
Then, we highlight the rewarding bias that emerges from overconfident preference labeling, which is particularly problematic for 7B level smaller LLMs.
This rewarding bias results in the accumulation of noisy and unreliable preference data, harming the preference training and hindering alignment performance of LLMs.
To address this issue, we proposed the CREAM, which leverages the rewarding consistency across different iterations as a regularization signal. This approach allows the model to learn more selectively, emphasizing reliable preference data and avoiding overconfidence in preference labeling. Experimental results on five natural language benchmark datasets demonstrate the effectiveness of the proposed method in mitigating the rewarding bias issue of SRLMs and improving the alignment performance of LLMs.

## Appendix A Proof of the theorems and lemmas

### A.1 Proof of Theorem 3.1

###### Proof of Theorem 3.1 .

We denote iteration of the two-step algorithm as $t$. The algorithm starts from $(\bm{\theta}_{t},z_{t})$, and obtains $z_{t+1}=z_{\bm{\theta}_{t}}$ according to Eq. (3.3) in the preference-labeling step and then obtains $\bm{\theta}_{t+1}$ through the learning step. Since $z_{t+1}=\operatorname*{arg\,min}_{z}\mathcal{L}_{\text{DPO}}(\bm{\theta}_{t};
\mathbf{y},\mathbf{y}^{\prime},\mathbf{x},z)$ for any $\mathbf{y},\mathbf{y}^{\prime},\mathbf{x}$ according to Eq. (3.3), we have that

$$ $\displaystyle\mathcal{L}(\bm{\theta}_{t},z_{t+1})\leq\mathcal{L}(\bm{\theta}_{ t},z_{t}).$ (A.1) $$

And the learning step suggests that $\bm{\theta}_{t+1}=\operatorname*{arg\,min}_{\bm{\theta}}\mathcal{L}(\bm{\theta
},z_{t+1})$, yielding that

$$ $\displaystyle\mathcal{L}(\bm{\theta}_{t+1},z_{t+1})\leq\mathcal{L}(\bm{\theta} _{t},z_{t+1}).$ (A.2) $$

Connecting Eq. (A.1) with Eq. (A.2) yields that the loss function $\mathcal{L}(\bm{\theta},z)$ is monotonically decreasing, i.e.

$$ $\displaystyle\cdots\leq\mathcal{L}(\bm{\theta}_{t+1},z_{t+1})\leq\mathcal{L}( \bm{\theta}_{t},z_{t+1})\leq\mathcal{L}(\bm{\theta}_{t},z_{t})\leq\cdots.$ (A.3) $$

Since $\mathcal{L}(\bm{\theta},z)$ is upper bounded by 0 0, it suggests that the sequence of $\mathcal{L}(\bm{\theta}_{t},z_{t})$ will converge w.r.t. the growth of $t$.
∎

###### Proof of Theorem 3.1 .

### A.2 Proof of Lemma 3.2

###### Proof of Lemma 3.2 .

We start by expanding $\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|
\mathbf{x})}\mathcal{L}_{\text{reg}}(\bm{\theta};\mathbf{y},\mathbf{y}^{\prime
},\mathbf{x})$ as

$$ $\displaystyle\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t }}(\cdot|\mathbf{x})}\mathcal{L}_{\text{reg}}(\bm{\theta};\mathbf{y},\mathbf{y }^{\prime},\mathbf{x})$ $\displaystyle=\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{ t}}(\cdot|\mathbf{x})}\left[\log P_{\bm{\theta}}(z=1)+\log P_{\bm{\theta}}(z=0 )\right]$ $\displaystyle=\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{ t}}(\cdot|\mathbf{x}),\mathbf{y}\prec\mathbf{y}^{\prime}}\left[\log P_{\bm{ \theta}}(z=1)+\log P_{\bm{\theta}}(z=0)\right]$ $\displaystyle\quad+\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{ \theta}_{t}}(\cdot|\mathbf{x}),\mathbf{y}\succeq\mathbf{y}^{\prime}}\left[\log P _{\bm{\theta}}(z=1)+\log P_{\bm{\theta}}(z=0)\right]$ $\displaystyle=\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{ t}}(\cdot|\mathbf{x})}P_{\bm{\theta}_{t}}(z=0)\left[\log P_{\bm{\theta}}(z=1)+ \log P_{\bm{\theta}}(z=0)\right]$ $\displaystyle\quad+\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{ \theta}_{t}}(\cdot|\mathbf{x})}P_{\bm{\theta}_{t}}(z=1)\left[\log P_{\bm{ \theta}}(z=1)+\log P_{\bm{\theta}}(z=0)\right],$ (A.4) $$

where the second equation decompose the expectation $\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|
\mathbf{x})}$ into two expectation $\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|
\mathbf{x})}$ + $\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|
\mathbf{x})}$, the third equation extract the event $\mathbf{y}\geq\mathbf{y}^{\prime}$ as distribution $P_{\bm{\theta}_{t}}(z)$. Then Eq. (A.4) can be further written by

$$ $\displaystyle\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t }}(\cdot|\mathbf{x})}\mathcal{L}_{\text{reg}}(\bm{\theta};\mathbf{y},\mathbf{y }^{\prime},\mathbf{x})$ $\displaystyle=P_{\bm{\theta}_{t}}(z=1)\left[\log P_{\bm{\theta}}(z=1)+\log P_{ \bm{\theta}}(z=0)\right]$ $\displaystyle\quad+P_{\bm{\theta}_{t}}(z=0)\left[\log P_{\bm{\theta}}(z=1)+ \log P_{\bm{\theta}}(z=0)\right],$ (A.5) $$

since both $\mathbf{y},\mathbf{y}^{\prime}$ are generated from $\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})$, $P_{\bm{\theta}_{t}}(z=0)=P_{\bm{\theta}_{t}}(z=1)=0.5$. Thus Eq. (A.5) becomes

$$ $\displaystyle\mathbb{E}_{\mathbf{y},\mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t }}(\cdot|\mathbf{x})}\mathcal{L}_{\text{reg}}(\bm{\theta};\mathbf{y},\mathbf{y }^{\prime},\mathbf{x})=2\mathrm{KL}(P_{\bm{\theta}}(z)\parallel P_{\bm{\theta} _{t}}(z))=2\mathrm{KL}(P_{\bm{\theta}}(z)\parallel u(z)),$ (A.6) $$

where $u(z)$ is the uniform binary distribution with $u(z=0)=u(z=1)=0.5$.
∎

###### Proof of Lemma 3.2 .

### A.3 Proof Theorem 3.3

###### Proof of Theorem 3.3 .

We start by writing down each components in $\mathcal{L}(\bm{\theta},z)$ defined in Eq. (3.1) by

$$ $\displaystyle\mathcal{L}(\bm{\theta},z)$ $\displaystyle=\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{S}})+ \mathbb{E}_{\mathbf{x}\sim\mathcal{D}^{\prime};\mathbf{y},\mathbf{y}^{\prime} \sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}[\mathcal{L}_{\text{DPO}}(\bm{ \theta};\mathbf{y},\mathbf{y}^{\prime},\mathbf{x},z)+\lambda\mathcal{L}_{\text {Reg}}(\bm{\theta};\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})]$ $\displaystyle=\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{S}})$ $\displaystyle\quad+\mathbb{E}_{\mathbf{x}\sim\mathcal{D}^{\prime};\mathbf{y}, \mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}\left[-z( \mathbf{y},\mathbf{y}^{\prime},\mathbf{x})\log\sigma\left(\log\left(\frac{\pi_ {\bm{\theta}}(\mathbf{y}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})} \right)-\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}^{\prime}|\mathbf{x})}{\pi _{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x})}\right)\right)\right.$ $\displaystyle\qquad\qquad\qquad\qquad\qquad\left.-(1-z(\mathbf{y},\mathbf{y}^{ \prime},\mathbf{x}))\log\sigma\left(\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{ y}^{\prime}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x})} \right)-\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})}{\pi_{\text{ ref}}(\mathbf{y}|\mathbf{x})}\right)\right)\right.$ $\displaystyle\qquad\qquad\qquad\qquad\qquad\left.-\lambda\log\sigma\left(\log \left(\frac{\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf {y}|\mathbf{x})}\right)-\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}^{\prime}| \mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x})}\right)\right)\right.$ $\displaystyle\qquad\qquad\qquad\qquad\qquad\left.-\lambda\log\sigma\left(\log \left(\frac{\pi_{\bm{\theta}}(\mathbf{y}^{\prime}|\mathbf{x})}{\pi_{\text{ref} }(\mathbf{y}^{\prime}|\mathbf{x})}\right)-\log\left(\frac{\pi_{\bm{\theta}}( \mathbf{y}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})}\right)\right)\right]$ $\displaystyle=\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{S}})$ $\displaystyle\quad+\mathbb{E}_{\mathbf{x}\sim\mathcal{D}^{\prime};\mathbf{y}, \mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}\left[-(\lambda +z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x}))\log\sigma\left(\log\left(\frac{ \pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}|\mathbf{ x})}\right)-\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}^{\prime}|\mathbf{x})} {\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x})}\right)\right)\right.$ $\displaystyle\qquad\qquad\qquad\qquad\qquad\left.-(1+\lambda-z(\mathbf{y}, \mathbf{y}^{\prime},\mathbf{x}))\log\sigma\left(\log\left(\frac{\pi_{\bm{ \theta}}(\mathbf{y}^{\prime}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}^{\prime} |\mathbf{x})}\right)-\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})} {\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})}\right)\right)\right]$ $$

where the third equation absorbs the regularization into the DPO loss. Noticing that $\lambda+z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})+(1+\lambda-z(\mathbf{y},
\mathbf{y}^{\prime},\mathbf{x}))=1+2\lambda$, by dividing $(1+2\lambda)$ we have

$$ $\displaystyle\frac{\mathcal{L}(\bm{\theta},z)}{1+2\lambda}$ $\displaystyle=\frac{\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{ SFT}})}{1+2\lambda}$ $\displaystyle\quad+\mathbb{E}_{\mathbf{x}\sim\mathcal{D}^{\prime};\mathbf{y}, \mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}\left[-\frac{ \lambda+z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x})}{1+2\lambda}\log\sigma \left(\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}|\mathbf{x})}{\pi_{\text{ref }}(\mathbf{y}|\mathbf{x})}\right)-\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y} ^{\prime}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}^{\prime}|\mathbf{x})}\right )\right)\right.$ $\displaystyle\qquad\qquad\qquad\qquad\qquad\left.-\frac{1+\lambda-z(\mathbf{y} ,\mathbf{y}^{\prime},\mathbf{x})}{1+2\lambda}\log\sigma\left(\log\left(\frac{ \pi_{\bm{\theta}}(\mathbf{y}^{\prime}|\mathbf{x})}{\pi_{\text{ref}}(\mathbf{y} ^{\prime}|\mathbf{x})}\right)-\log\left(\frac{\pi_{\bm{\theta}}(\mathbf{y}| \mathbf{x})}{\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})}\right)\right)\right].$ $$

When $z(\mathbf{y},\mathbf{y}^{\prime}\mathbf{x})=1$, $(\lambda+z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x}))/(1+2\lambda)=1-\lambda/
(1+2\lambda)$ and $(1+\lambda-z(\mathbf{y},\mathbf{y}^{\prime},\mathbf{x}))(1+2\lambda)=\lambda/(
1+2\lambda)$. Therefore, letting $\mathcal{C}_{\lambda}=\lambda/(1+2\lambda)$ yields that

$$ $\displaystyle\frac{\mathcal{L}(\bm{\theta},z)}{1+2\lambda}$ $\displaystyle=\frac{\mathcal{L}_{\text{SFT}}(\bm{\theta};\mathcal{D}_{\text{ SFT}})}{1+2\lambda}$ $\displaystyle\quad+\mathbb{E}_{\mathbf{x}\sim\mathcal{D}^{\prime};\mathbf{y}, \mathbf{y}^{\prime}\sim\pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x})}[(1-\mathcal{C} _{\lambda})\mathcal{L}_{\text{DPO}}(\bm{\theta};\mathbf{y},\mathbf{y}^{\prime} ,\mathbf{x},z)+\mathcal{C}_{\lambda}\mathcal{L}_{\text{DPO}}(\bm{\theta}; \mathbf{y},\mathbf{y}^{\prime},\mathbf{x},1-z)],$ $$

which completes the proof since minimizing $\mathcal{L}(\bm{\theta},z)/(1+\lambda)$ is equivalent with minimizing $\mathcal{L}(\bm{\theta},z)$ itself.
∎

###### Proof of Theorem 3.3 .

### A.4 Proof of Lemma 3.4

###### Proof of Lemma 3.4 .

To begin with, according to the ranking of $J_{ij}$, the sufficient and necessary condition for $J_{ij}-J_{i^{\prime}j}>0$ is that $r_{ij}<r_{i^{\prime}j}$. Similarly, the sufficient and necessary condition for $K_{ij}>K_{i^{\prime}j}$ is that $r^{\prime}_{ij}<r^{\prime}_{i^{\prime}j}$. As a result, the indicator becomes

$$ $\displaystyle\operatorname{\mathds{1}}[(J_{ij}-J_{i^{\prime}j})(K_{ij}-K_{i^{ \prime}j})>0]$ $\displaystyle=\operatorname{\mathds{1}}[(r_{ij}-r_{i^{\prime}j})(r^{\prime}_{ ij}-r^{\prime}_{i^{\prime}j})>0]$ (A.7) $\displaystyle\operatorname{\mathds{1}}[(J_{ij}-J_{i^{\prime}j})(K_{ij}-K_{i^{ \prime}j})<0]$ $\displaystyle=\operatorname{\mathds{1}}[(r_{ij}-r_{ij^{\prime}})(r^{\prime}_{ ij}-r^{\prime}_{i^{\prime}j})<0].$ (A.8) $$

Since $r_{ij}>r_{i^{\prime}j}$ yields $\mathbf{y}_{ij}\succ\mathbf{y}_{i^{\prime}j}$ under the input prompt $xb_{j}$ and language model $\bm{\theta}_{t}$, Eq. (A.7) becomes

$$ $\displaystyle\operatorname{\mathds{1}}[(J_{ij}-J_{i^{\prime}j})(K_{ij}-K_{i^{ \prime}j})>0]$ $\displaystyle=\operatorname{\mathds{1}}[r_{ij}>r_{i^{\prime}j}]\operatorname{ \mathds{1}}[r^{\prime}_{ij}>r^{\prime}_{i^{\prime}j}]+\operatorname{\mathds{1} }[r_{ij}<r_{i^{\prime}j}]\operatorname{\mathds{1}}[r^{\prime}_{ij}<r^{\prime}_ {i^{\prime}j}]$ $\displaystyle=\operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y}_{i^{ \prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}]\operatorname{\mathds{1}}[\mathbf{y}_ {ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}]$ $\displaystyle\quad+\operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i ^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}]\operatorname{\mathds{1}}[\mathbf{y }_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}].$ (A.9) $$

As a result, since $\mathbf{y}_{ij}$ are i.i.d. given $\mathbf{x}_{j}$, the expectation of first part of the Kendall’s Tau coefficient is

$$ $\displaystyle\mathbb{E}[\operatorname{\mathds{1}}[(J_{ij}-J_{i^{\prime}j})(K_{ ij}-K_{i^{\prime}j})>0]]-\mathbb{E}[\operatorname{\mathds{1}}[(J_{ij}-J_{i^{ \prime}j})(K_{ij}-K_{i^{\prime}j})<0]]$ $\displaystyle=\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{\bm {\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ij} \succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}]\operatorname{ \mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{ \theta}_{t-1}]]$ $\displaystyle\quad+\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi _{\bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y} _{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}] \operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t-1}]]$ $\displaystyle\quad-\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi _{\bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y} _{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}] \operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t-1}]]$ $\displaystyle\quad-\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi _{\bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y} _{ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}] \operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t-1}]]$ $\displaystyle=\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{\bm {\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ij} \succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}](\operatorname{ \mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{ \theta}_{t-1}]-\operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{ \prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}])]$ $\displaystyle\quad+\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi _{\bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y} _{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}]( \operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t-1}]-\operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ \mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}])]$ $\displaystyle=\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{\bm {\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ij} \succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}](1-2\operatorname {\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{ \theta}_{t-1}])]$ $\displaystyle\quad-\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi _{\bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y} _{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}](1-2 \operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t-1}])]$ $\displaystyle=\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{\bm {\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ij} \succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}]-\operatorname{ \mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{ \theta}_{t}]]$ $\displaystyle\quad+2\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim \pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf {y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}]( \operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t}]-\operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ\mathbf {y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}])]$ $\displaystyle=0+2\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{ \bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}](1-2 \operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t}])]$ (A.10) $$

where the second equation merge the terms together, and the third equation is due to the fact $\operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}]+
\operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y}_{i^{\prime}j}]=1$, the forth equation reorganize the term and the fifth equation is due to the fact that $\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{\bm{\theta}_{t}}(
\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y
}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}]-\operatorname{\mathds{1}}[
\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}]]=0$ due to symmetry. Similarly by reversing the $\prec$ and $\succ$, we can write Eq. (A.10) by

$$ $\displaystyle\mathbb{E}[\operatorname{\mathds{1}}[(J_{ij}-J_{i^{\prime}j})(K_{ ij}-K_{i^{\prime}j})>0]]-\mathbb{E}[\operatorname{\mathds{1}}[(J_{ij}-J_{i^{ \prime}j})(K_{ij}-K_{i^{\prime}j})<0]]$ $\displaystyle=0+2\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{ \bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}](1-2 \operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t}])].$ (A.11) $$

Adding Eq. (A.10) and Eq. (A.11) together yields

$$ $\displaystyle 2\mathbb{E}[\operatorname{\mathds{1}}[(J_{ij}-J_{i^{\prime}j})(K _{ij}-K_{i^{\prime}j})>0]]-\mathbb{E}[\operatorname{\mathds{1}}[(J_{ij}-J_{i^{ \prime}j})(K_{ij}-K_{i^{\prime}j})<0]]$ $\displaystyle=2\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{ \bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}]+ \operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t-1}]]$ $\displaystyle\quad-4\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim \pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf {y}_{ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}] \operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t-1}]]$ $\displaystyle\quad-4\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim \pi_{\bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf {y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}] \operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf {x}_{j},\bm{\theta}_{t-1}]]$ $\displaystyle=2-8\mathbb{E}_{\mathbf{y}_{ij},\mathbf{y}_{i^{\prime}j}\sim\pi_{ \bm{\theta}_{t}}(\cdot|\mathbf{x}_{j})}[\operatorname{\mathds{1}}[\mathbf{y}_{ ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t}]\operatorname{ \mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{ \theta}_{t-1}]],$ (A.12) $$

where the final equation is because $\mathbb{E}[\operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{y}_{i^{\prime
}j}|\mathbf{x}_{j},\bm{\theta}_{t}]\operatorname{\mathds{1}}[\mathbf{y}_{ij}
\succ\mathbf{y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}]]=\mathbb{E}[
\operatorname{\mathds{1}}[\mathbf{y}_{ij}\succ\mathbf{y}_{i^{\prime}j}|\mathbf
{x}_{j},\bm{\theta}_{t}]\operatorname{\mathds{1}}[\mathbf{y}_{ij}\prec\mathbf{
y}_{i^{\prime}j}|\mathbf{x}_{j},\bm{\theta}_{t-1}]$ due to symmetry. Divide Eq. (A.12) by 2 yields the claimed result.
∎

###### Proof of Lemma 3.4 .

## Appendix B Additional Results

Table 6 includes the detailed results of SRLM with prompt rewarding method (P-SRLM).
Compared to the performance of methods using DPO rewarding (SRLM and CREAM), we find that DPO rewarding, particularly CREAM, can generally achieve better performance than P-SRLM.
This indicates the limitations of applying LLM-as-a-Judge to 7B level LLMs in self-rewarding scenarios.

**Table 6: Results for different self-rewarding methods using Llama3.**
| Dataset<br>Method | M1 | M2 | M3 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SFT | SRLM | P-SRLM | CREAM | SRLM | P-SRLM | CREAM |  |
| Arc-Easy | 86.78 | 87.79 ↑ | 84.64 ↓ | 88.89 ↑ | 87.17 ↓ | 83.75 ↓ | 89.52 ↑ |
| Arc-Challenge | 80.14 | 80.38 ↑ | 76.79 ↓ | 80.89 ↑ | 81.23 ↑ | 76.28 ↓ | 83.36 ↑ |
| OpenBookQA | 86.40 | 87.80 ↑ | 80.40 ↓ | 88.00 ↑ | 87.30 ↓ | 80.20 ↓ | 90.20 ↑ |
| SIQA | 69.50 | 70.95 ↑ | 67.81 ↓ | 69.79 ↑ | 70.37 ↓ | 66.63 ↓ | 72.06 ↑ |
| GSM8K | 78.39 | 78.01 ↓ | 78.47 ↑ | 81.04 ↑ | 77.48 ↓ | 78.99 ↑ | 81.73 ↑ |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: - The proposed method is interesting and there are theoretical and empirical proofs of effectiveness.
- The method uses the ranking correlation to evaluate the consistency or uncertainty in self rewarding.
summary: The authors propose the Consistency Regularized self-rewarding Language Model (CREAM), which incorporates regularization to improve reward consistency in order to alleviate the risks of inaccurate rewards and biased training data over time. The basic idea of the method si to add a regularization of optimizing towards a balanced binary distribution, where the weight of the regularization is determined by the  self reward ranking correlation between different optimization steps. The author provides some theoretical proof and empirical results show that CREAM enhances both alignment performance and the reliability of preference data.
weaknesses: - I think first the author should add some baselines such as the original form of the method which just uses the KL constraint toward the Bernoulli distribution. It would be good to break down where the improvement comes from and how much the ranking correlation helps the methods.
 - Then another very important experiment I think is that previous self-rewarding methods can not maintain the improvement beyond 3 or 4 iterations, how will your method help this? Will there still be significant improvement for the 5th or 6th iteration?
 - Another thing I am thinking is that given the final form, in the training you use a normal DPO and a reversed DPO, and weight the loss functions. Then this is basically the same as just weight the normal DPO with a weight maybe ranging from -1<->1. Why don't the author try this for example shifting the C weight and only keeping DPO loss? This would be much more efficient in implementation I assume?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Questions:
1. In the line #445 - 447, It describes as “SRLM with prompt rewarding is not effective”, but
the performances are increasing on the GSM8K dataset in the Table 3. Can you explain
why this could be the case?
2. In Table 4, the performance of CREAM dis not improve when we compared to Llama2
M1 on the Arc-E, Arc-C, OBQA, and SIQA datasets. What are your thoughts on why it
didn’t improve, even though CREAM showed better performance compared to Llama3
M1?

3. Regarding the reversal of the DPO dataset, if \( C = 0 \) indicates an inverse correlation between $\theta_{t-1}$ and $\theta_{t}$, isn’t learning from this strategy placing too much confidence in the assumption that reverse preferences are meaningful? This concern arises particularly when the initial inconsistency between $\theta_{t-1}$ and $\theta_{t}$ might be due to stochasticity rather than an actual flaw in the self-rewarding model being trained. As such, did the authors conduct experiments to analyze how the trend of inconsistency evolves between consecutive iterations, especially at later stages of training (e.g., around $t = 5 $ or  $t = 6$), to see if this inconsistency stabilizes later in training?



presentation/typos:

Abstract: “there is no guarantee on the accurate of the rewarding and ranking” —> accuracy 
Line 082: “we first formulates a generalized iterative preference” —> formulate
Line 139: “The objective is to iteratively minimizing the following loss” —> minimize
Line 522: “emphasizing reliable preference data and avoiding overconfident
in preference labeling.” —> overconfidence
soundness: 3
strengths: 1. The methodology section is well described with mathematical analyses for readers.
2. Formulating the challenges and corresponding analysis is organized and explained well.
3. Various experiments are conducted to show the performance of CREAM with models like LLama2 and 3, including comparison with reasonable baselines like SRLMs and external reward models.
summary: This paper attempts to propose a more consistent reward learning strategy within self-rewarding LLM settings where the same model acts as the policy generating sample responses as well as a reward model that ranks those responses. Their framework called  Consistency Regularized sElf-Rewarding lAnguage Models (CREAM) leverage reward consistency signals to make their models learn from more reliable preference
data. Their experiments suggests that CREAM-trained models outperform SRLMs as well as external reward models especially with smaller models such as LLaMa2 and 3.
weaknesses: 1. In the line #244, there is a lack of information about the reason for using reversed
preference order.
2. Once again, in the line #328, The reasons for preparing a reverse DPO dataset and swapping the
best response with the worst response are somewhat unclear.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See weaknesses for questions.

There are many grammar mistakes that would be good to resolve, e.g.:
* Abstract - “the accurate of the” -> “the accuracy of the”
* Intro - “we formulates” -> “we formulate”
* Many places - “the rewarding consistency” -> “the consistency reward”?
* Section 3.2 - “prevents the model… from overconfident”

Other nits:
* Notation - You define `[k]` as a set, but the next sentence says “Sets are denoted by calligraphic uppercase letters”. Maybe say “Other sets are…” to avoid a contradiction?
* Equation 3.4 - My understanding is that typically SFT is performed first, and then DPO, rather than optimizing both objectives jointly, as this equation implies.
* The RewardBench citation is missing a year.
soundness: 2
strengths: * The idea of considering diversity across an ensemble of RMs to prevent reward hacking is sound and well motivated.
* Consideration of such a signal is supported by the experimental results.
summary: This paper proposes a modified DPO objective that considers multiple reward judgements, and penalizes disagreement across multiple reward models. The new objective is studied in the context of self-rewarding LMs, an iterative process based on DPO.

(Note: score increased from 3 to 6 based on the discussion)
weaknesses: * The biggest weakness of this paper is that it is not contextualized well with prior work on reward hacking and using ensembles to mitigate this. Given that proposing variants of DPO and related algorithms is increasingly crowded, this is especially important. Beyond discussing and comparing with this prior work, it is also important to understand how pessimism-based approaches compare, both theoretically and empirically, with the proposed methods based on agreement metrics. Similarly to this paper, there is a growing body of work that aims to mitigate reward hacking through reward model ensembles (e.g. https://arxiv.org/abs/2310.02743, https://arxiv.org/abs/2312.09244, etc.). These papers provide analysis showing various correlation metrics during RLHF. “Disagreement” between reward models can be penalized by considering pessimistic aggregations of rewards, e.g. by taking the min or some percentile (<50) over the set of RM rewards. Such concepts have also previously been integrated into DPO (e.g. https://arxiv.org/abs/2405.19316). It is not clear the relation between the proposed approach of weighting by RM agreement to this prior work which aggregates RM scores pessimistically, or whether there is any advantage. 
* The motivation for introducing the framework in 3.1 was not clear to me. How is it different from prior work, e.g. Yuan et al., 2024? Why is it necessary to introduce this new framework to evaluate the key contribution of the paper, i.e. the impact of the consistency reward? This made the experiments more difficult to analyze, because it was unclear whether the only change between "SRLM" and "CREAM" was the consistency reward or whether other factors contributed to the difference.
* The overly formal exposition of the proposed method makes the intuition difficult to follow. For example, the definition of the regularization loss, introduced in equation 3.5, does not include any intuition for why this term was chosen.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The method seems to be applied to LMs that have already been post-trained (LLAMA 2 & 3), which is perhaps why the impact is less than in Yuan et al. Would it be possible to apply the method to a model immediately after pretraining or instruction tuning?
soundness: 3
strengths: - While the proposed approach seems somewhat heuristic (e.g., training on both (y, y') and (y', y)), the steps of the algorithm are justified by theoretical support.
- The "oracle" component of the evaluation helps to clarify the headroom for better reward modeling on these tasks.
- The ablations help clarify the role of the specific form of rank correlation and "prompt rewarding" vs the implicit DPO RM
summary: This paper proposes a form of regularization for self-rewarding language models, which involves creating a loss that is a weighted combination of the LM's implied preferences and the reverse preferences. The weighting is determined by the rank correlation between implied rewards across adjacent training steps: if these rewards are perfectly rank-correlated then the reverse preferences are not included; to the extent that they are uncorrelated, the self reward model is deemed less trustworthy and so the reverse preferences are given more weight. The specific form of the regularizer is justified by theoretical connections to entropic regularization. The method yields improvements over the unregularized SRLM approach of Yuan et al (2024) on benchmark datasets and in head-to-head win rate (e.g., 34/21 win-loss ratio). Evaluations focus at the 7B scale.
weaknesses: It's not clear to me why one would want to use self-rewarding LMs at the 7B scale, as it would not be too difficult to get preference annotations from a larger model and train on them instead.

The algorithm appears to treat consistency as a property of the model \theta_t, but I wonder whether it would be better to treat it as a property of the *data*. That is, for any model, there will be some examples for which preference can be assessed confidently, and others for which it cannot. It would be ideal to use the full DPO loss for the first type of examples, and regularize strongly on the second type. Instead, the proposed approach seems to set the regularization parameter based on some property of the optimization path: as long as the LM has not changed its rankings from time t-1, it will use the standard DPO objective. (Thanks for the additional experiments on this.)

I would like to see more analysis of the sum of DPO losses. In particular, what is the effect to the policy of applying the update in step 14 of the algorithm when C=1/2? My guess is that the probability for both y+ and y- decreases, but it would be great to have some formal analysis because this situation could be relevant, e.g., at the beginning of training.

Meta-review: The paper addresses challenges in self-rewarding LLMs, where the same model functions as both the policy and the reward model. It highlights issues with ranking accuracy and reward systems, which lead to unreliable preference data. The authors propose CREAM, a method that introduces regularization to improve reward consistency and prevent overconfident preference labeling, thereby generating more reliable training data and enhancing alignment performance.

After reviewing the rebuttal, the authors responded effectively to the reviewers' concerns. The reviewers noted that the paper could benefit from additional background information and clearer explanations, which would improve its readability. I hope the authors will enhance the final version of the paper. My recommendation is acceptance.

Decision: Accept (Poster)