Bootstrapping Language Models with DPO Implicit Rewards

## Abstract

Abstract Human alignment in large language models (LLMs) is an active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO, after training, provides an implicit reward model. In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM. Our approach is to use the rewards from a current LLM model to construct a preference dataset, which is then used in subsequent DPO rounds. We incorporate refinements that debias the length of the responses and improve the quality of the preference dataset to further improve our approach. Our approach, named self-alignment with D PO I mpli C it r E wards (DICE), shows great improvements in alignment and achieves superior performance than Gemini Pro on AlpacaEval 2, reaching 27.55 % percent 27.55 27.55\% 27.55 % length-controlled win rate against GPT-4 Turbo, but with only 8B parameters and no external feedback. Our code is available at https://github.com/sail-sg/dice .

## 1 Introduction

Direct preference optimization (DPO) [] presents a compelling alternative to reinforcement learning from human feedback (RLHF) in large language models (LLMs). By circumventing the complexity of learning a reward model from given human preferences, DPO is simpler to implement and train compared to the RLHF approaches. Importantly, DPO, once trained, implicitly specifies a reward model. Mathematically, the reward for a response $\mathbf{y}$ to the prompt $\mathbf{x}$ can be expressed in terms of the optimal policy $\pi^{\star}$ and the reference policy $\pi_{\textrm{ref}}$:

$$ $\hat{r}(\mathbf{x},\mathbf{y})=\beta\log\frac{\pi^{\star}(\mathbf{y}|\mathbf{x })}{\pi_{\textrm{ref}}(\mathbf{y}|\mathbf{x})}+\beta\log Z(\mathbf{x})\textrm{,}$ $$

for parameter $\beta$ and normalizing constant $Z$.
Further, an *implicit reward* $r(\mathbf{x},\mathbf{y})=\beta[\log\pi^{\star}(\mathbf{y}|\mathbf{x})-$ $\log\pi_{\textrm{ref}}(\mathbf{y}|\mathbf{x})]$ is defined in DPO where the normalizing constant term can be ignored as it will be canceled out in the DPO objective, which only involves the difference of the rewards for the same prompt.
In this work, we explore whether the above readily available implicit reward model after DPO training provides an opportunity to further improve the language model.

This paper answers the research question in the affirmative, by using the above implicit rewards in a bootstrapping fashion to further improve the LLM alignment with human preferences. Specifically, our approach follows the iterative DPO framework [], where the implicit rewards are provided as the preference signals, as illustrated in Figure 1.
We start with a model that has been through one round of DPO using human preference data. We then use the implicit rewards induced by itself to rank outputs from the current LLM, thereby, yielding a new preference dataset cheaply. We run DPO again with this newly generated preference dataset to obtain an updated LLM and then repeat the process. However, the above approach still needs further refinement to address practical issues. One is the known issue of length exploitation [] where LLMs generate long responses when the same content could be provided more succinctly. Another issue is that the implicit reward model is an approximate proxy for human preferences, hence relying on it strongly can result in corruption of the initial knowledge inbuilt into the LLM.

Figure: Figure 1: (Left) Iterative DPO [] with various preference signals: under the iterative DPO framework, the policy model is iteratively trained on a newly generated preference dataset. This dataset can be constructed using various preference signals. A common source is a scalar reward model (RM) [], denoted as $r_{\phi}$. Alternatively, the dataset can be created by prompting a LLM to judge the responses. This LLM can either be an external model $\pi_{\phi}$ or the policy model itself from the previous iteration $\pi_{\theta^{(t-1)}}$. In this context, ${\mathbf{x}}$ and ${\mathbf{y}}$ are processed through a LLM-as-a-judge template $g(\cdot,\cdot)$ []. We propose utilizing the length-regularized (LR) implicit rewards introduced in Section 4.2, where $\pi_{\theta^{(t-1)}}$ and $\pi_{\mathrm{ref}}^{(t-1)}$ represent the policy model and reference model from the previous DPO iteration, respectively. Our experiment excludes approaches requiring external models, such as $r_{\phi}$ and $\pi_{\phi}$, as they are beyond this work’s scope. (Right) Our method, which leverages implicit rewards, further improves DPO-tuned models by a large margin, resulting in superior performance compared to the prompting counterpart.
Refer to caption: x1.png

We address the length exploitation issue by length-regularized reward shaping, which discourages long responses from being preferred. To fix the overreliance on the implicit reward we use insights from continual learning [] and replay high quality human preference data that was used in the first round of DPO (this round was before we start our bootstrapping). Our method, named self-alignment with DPO ImpliCit rEwards (DICE), significantly improves LLM alignment quality with different base models. On AlpacaEval 2, we achieve $8.02\%$ length-controlled win rate improvement with the Zephyr-based model and $9.35\%$ improvement with the Llama3-based model. Our best model outperforms Gemini Pro, but with only 8B parameters and does not require any in-house data or external reward model.

To summarize, our main contributions are as follows:

- •
We propose to utilize the implicit reward model readily available in a DPO-ed LLM. The implicit reward model enables us to evaluate the responses generated by the current policy and construct a preference dataset for future rounds of DPO without any external feedback;
- •
We propose to apply two techniques together with our above proposed approach, length-regularized reward shaping and experience replay;
- •
Empirical results show that our approach DICE enables significant improvement in alignment with different base models, and our best bootstrapped model achieves better performance than Gemini Pro but with only 8B parameters.

## 2 Related Work

Self-Improving Fine-Tuning. Many efforts have been made to investigate ways of fine-tuning language models without a large amount of human annotation [, , , , ]. Starting from an SFT model, Sun et al. [] collect the preference labels by prompting the SFT model itself to choose the preferred response given a principle and train a principle-driven reward model with these preference labels. Afterwards, they optimize the policy by PPO with their reward model. Yuan et al. [] construct the preference dataset by their own supervised fine-tuned model trained on instruction following data and evaluation fine-tuning data, followed by DPO training on the preference dataset. Our work differs from this line of work in the motivations and assumptions. This line of work aims to align a language model with a small amount of seed data by eliciting the internal knowledge that is learned during the pretraining phase of LLMs. In contrast, our goal is to further improve a DPO-ed model in a bootstrapping manner by utilizing its implicit rewards.

On-Policy Sampling in Preference Tuning. DPO and its variants are popular due to their simplicity in training and implementation. However, research indicates that the offline nature of these direct alignment from preference (DAP) algorithms often prevents them from learning a good policy []. Guo et al. [] have shown that the offline DPO will quickly overfit the preference dataset, while it performs much better and is more stable if online feedback can be provided to their on-policy samples(^1^11On-policy samples: the data collected while following the current policy that is being optimized.). Tajwar et al. [] systematically discussed the properties of different preference fine-tuning approaches and the role of on-policy sampling. Their actionable takeaways for practitioners tell on-policy sampling generally improves performance and efficiency. If not being able to perform the pure on-policy sampling, using the data that is closer to on-policy samples also helps as it can be seen in iterative DPO [] and self-rewarding language models []. Similarly, our approach enables us to train the policy model with the preference data which is closer to on-policy samples than the offline dataset without any external reward models. We hypothesize this is one of the main gain sources of our approach.

DPO Implicit Rewards. Rafailov et al. [] recently study DPO from a token-level MDP perspective, and reveal that DPO-ed models implicitly parameterize token-wise dense reward functions. They therefore conduct beam search using the implicit rewards to improve the inference quality. Our work is inspired by this observation but focuses on utilizing the DPO-ed model’s implicit rewards to bootstrap itself for self-alignment.
Another independent work to ours [] pretrains a DPO model to serve as a standalone dense reward generator for PPO training, instead of further improving the DPO model.

## 3 Preliminaries

We provide a brief review of the standard RLHF [] and DPO algorithms []. Through the review, we demonstrate the implicit reward model induced by DPO, which will be used in our work.

In preference tuning, the preference data typically takes the form of pairwise preferences. Each prompt ${\mathbf{x}}$ is paired with two possible responses, ${\mathbf{y}}_{1}$ and ${\mathbf{y}}_{2}$. The human annotator [] or AI annotator [] provides the preference feedback $o(\mathbf{y}_{1}\succ\mathbf{y}_{2}|\mathbf{x})\in\{0,1\}$, indicating whether $\mathbf{y}_{1}$ is preferred over $\mathbf{y}_{2}$. The preferred response is denoted as ${\mathbf{y}}_{w}$, while the other is denoted as ${\mathbf{y}}_{l}$. A common assumption is that the ground-truth human preferences follow the Bradley-Terry model []. Based on this assumption, we can train a parameterized reward model $r_{\phi}(\mathbf{x},\mathbf{y})$ using maximum likelihood:

$$ $\mathcal{L}_{R}\left(r_{\phi},\mathcal{D}\right)=-\mathbb{E}_{\left(\mathbf{x} ,\mathbf{y}_{w},\mathbf{y}_{l}\right)\sim\mathcal{D}}\left[\log\sigma\left(r_{ \phi}\left(\mathbf{x},\mathbf{y}_{w}\right)-r_{\phi}\left(\mathbf{x},\mathbf{y }_{l}\right)\right)\right]\textrm{,}$ (1) $$

where $\sigma$ is the logistic function.

### 3.1 Reinforcement Learning from Human Feedback

The standard RLHF algorithm treats the LLM as a policy and optimizes the policy using the reward model $r_{\phi}$. The optimization objective is represented by the following equation:

$$ $\displaystyle\max_{\pi_{\theta}}\mathbb{E}_{\mathbf{x}\sim\mathcal{D},\mathbf{ y}\sim\pi_{\theta}(\mathbf{y}|\mathbf{x})}{\left[r_{\phi}(\mathbf{x},\mathbf{y })\right]}-\beta\cdot\mathbb{D}_{\mathrm{KL}}\left[\pi_{\theta}(\mathbf{y}| \mathbf{x})\|\pi_{\textrm{ref}}(\mathbf{y}|\mathbf{x})\right]\textrm{,}$ (2) $$

where $\pi_{\textrm{ref}}(\mathbf{y}|\mathbf{x})$ denotes a reference distribution,
and $\beta$ is a hyper-parameter. This objective balances the maximization of the reward $r_{\phi}(\mathbf{x},\mathbf{y})$ and divergence from the fixed reference distribution. The divergence term, given by the KL divergence (i.e., $\mathbb{D}_{\mathrm{KL}}\left[\pi_{\theta}(\mathbf{y}|\mathbf{x})\|\pi_{
\textrm{ref}}(\mathbf{y}|\mathbf{x})\right]$) acts as a regularizer to prevent the policy $\pi_{\theta}$ from drifting too far away from the initial distribution $\pi_{\textrm{ref}}(\mathbf{y}|\mathbf{x})$. This objective is then optimized using a general-purpose RL algorithm, such as PPO [].

### 3.2 Direct Preference Optimization

DPO [] starts with the same objective as Eq. 2. However, DPO
derives an analytical closed-form relation between the reward and the resulting optimal policy. This relation can be used to reparameterize the ground truth reward in terms of the corresponding optimal policy. This reparameterized formulation can be substituted back into the reward optimization objective in Eq. 1, enabling direct training of the optimal model on the feedback data using the following objective:

$$ $\mathcal{L}_{\mathrm{DPO}}\left(\pi_{\theta};\pi_{\textrm{ref}}\right)=- \mathbb{E}_{\left(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\right)\sim\mathcal{ D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}\left(\mathbf{y}_{w}| \mathbf{x}\right)}{\pi_{\textrm{ref}}\left(\mathbf{y}_{w}|\mathbf{x}\right)}- \beta\log\frac{\pi_{\theta}\left(\mathbf{y}_{l}|\mathbf{x}\right)}{\pi_{\text{ ref }}\left(\mathbf{y}_{l}|\mathbf{x}\right)}\right)\right].$ (3) $$

In this context, the parameter $\beta$ is the same as in Eq. 2, balancing the expected reward and divergence from the initial model. The DPO objective is particularly advantageous as it facilitates the recovery of the optimal model through a standard classification loss, without the need for on-policy sampling or extensive hyper-parameter tuning. Observe that Eq. 3 resembles the reward modeling objective in Eq. 1 under the parameterization

$$ $~{}r(\mathbf{x},\mathbf{y})=\beta\log\frac{\pi_{\theta}(\mathbf{y}|\mathbf{x}) }{\pi_{\textrm{ref}}(\mathbf{y}|\mathbf{x})}.$ (4) $$

This reward function is commonly referred to as an “implicit reward” [, ]. Theorem 1 in Rafailov et al. [] demonstrates that this parameterization of a reward model is indeed valid without loss of generality. If we substitute this form of $r_{\theta}(\mathbf{x},\mathbf{y})$ into the RL objective in Eq. 2, we can derive the optimal solution in a closed form, which is $\pi_{\theta}$. Consequently, once DPO optimization is completed, we obtain an “implicit reward model” as defined by Eq. 4.

## 4 Bootstrapping with DPO Implicit Rewards

DPO is an attractive alternative to RLHF as it largely simplifies the implementation and training process of language model alignment.
However, recent evidences [, ] have shown that continuing DPO training on a fixed offline dataset results in inferior policy, while the policy can be further improved if one can collect new responses generated by the updated policy and provide preference labels to perform another round of DPO training.
This can be understood as being closer to the on-policy sampling, which is generally preferred in preference fine-tuning [, ]. In Section 4.1, we first outline the iterative DPO training framework adopted in this work, and provide a theoretical analysis similar to Xie et al. [] to show that learning with on-policy samples can be more effective than utilizing an offline dataset.
Next, in Section 4.2, we introduce the proposed Length-Regularized Implicit Rewards, which augment the vanilla implicit rewards with a length-regularized reward shaping, to judge the on-policy sampled responses for constructing the preference dataset.
Notably, by fine-tuning a DPO-ed LM on the constructed dataset, we essentially align it without relying on any external preference feedback (e.g., RLHF or RLAIF), hence in a bootstrapping fashion. Furthermore, to mitigate the
potential catastrophic forgetting in the continual fine-tuning, we propose experience replay (Section 4.3) that mixes the generated data with the offline data for better performance.
We refer to our method as iterative self-alignment (bootstrapping) with DPO ImpliCit rEwards (DICE).

### 4.1 Iterative DPO with On-policy Sampling

We employ the iterative DPO preference tuning framework, where we start from a base language model (a base policy) $\pi_{\theta^{(0)}}$ that is DPO-tuned from an initial reference model $\pi_{\text{ref}}^{(0)}$, commonly an SFT model. In each round $t\in\{1,2,\dots\}$, we sample $K$ on-policy responses from the latest policy $\pi_{\theta^{(t-1)}}(\cdot\mid{\mathbf{x}})$ given a prompt ${\mathbf{x}}$. We then label the response with the highest and the lowest implicit rewards as winning and losing responses respectively, thus constructing a new preference dataset ${\mathcal{D}}_{t}$. We further fine-tune the policy with DPO’s objective (Eq. 3) to obtain the updated policy $\pi_{\theta^{(t)}}$ with reference model $\pi^{(t)}_{\text{ref}}=\pi_{\theta^{(t-1)}}$. This process is repeated to iteratively improve the language model.

To investigate the effect of on-policy sampling, we make a generalized notation for the sampling policy at the $t$-th round as $\pi^{(t)}$, and compare the on-policy sampling ($\pi^{(t)}=\pi_{\theta^{(t-1)}}$) with sampling from an offline dataset ($\pi^{(t)}=\pi_{\mu}$). For a prompt ${\mathbf{x}}$, we denote its optimal response as ${\mathbf{y}}^{\star}$ and suboptimal ones as ${\mathcal{S}}=\{{\mathbf{y}}^{-}_{i}\}$.
We also denote its preferences sampled from $\pi^{(t)}$ as $({\mathbf{y}}^{(t)}_{w},{\mathbf{y}}^{(t)}_{l})$. By Eq. 3 and the definition of the logistic function, we can rewrite the DPO loss at round $t$ for the sample $({\mathbf{x}},{\mathbf{y}}^{(t)}_{w},{\mathbf{y}}^{(t)}_{l})$ as:

$$ ${\mathcal{L}}^{(t)}_{\mathrm{DPO}}(\pi_{\theta^{(t)}};\pi_{\mathrm{ref}}^{(t)} )=-\log\frac{\pi_{\theta^{(t)}}\left({\mathbf{y}}_{w}^{(t)}\mid{\mathbf{x}} \right)^{\beta}}{\pi_{\theta^{(t)}}\left({\mathbf{y}}_{w}^{(t)}\mid{\mathbf{x} }\right)^{\beta}+\pi_{\theta^{(t)}}\left({\mathbf{y}}_{l}^{(t)}\mid{\mathbf{x} }\right)^{\beta}R^{\beta}}\textrm{,}$ (5) $$

where $R={\pi_{\mathrm{ref}}^{(t)}({\mathbf{y}}_{w}^{(t)}\mid{\mathbf{x}})}/{\pi_{
\mathrm{ref}}^{(t)}({\mathbf{y}}_{l}^{(t)}\mid{\mathbf{x}})}$ is a constant.
Observe that Eq. 5 can be minimized to zero by just minimizing $\pi_{\theta^{(t)}}({\mathbf{y}}_{l}^{(t)}|{\mathbf{x}})$ to be zero, without minimizing the likelihood of
other suboptimal responses $\pi_{\theta^{(t)}}({\mathbf{y}}^{-}|{\mathbf{x}})$ for ${\mathbf{y}}^{-}\neq{\mathbf{y}}_{l}^{(t)}$. After $t$ rounds of optimization, we are interested in the probability of outputting the optimal response:

$$ $\pi_{\theta^{(t)}}({\mathbf{y}}^{\star}\mid{\mathbf{x}})=1-\sum_{{\mathbf{y}}^ {-}_{i}\in{\mathcal{S}}}\pi_{\theta^{(t)}}({\mathbf{y}}_{i}^{-}\mid{\mathbf{x} }).$ (6) $$

Eq. 6 allows us to reveal the deficiency of training on a fixed offline dataset.
If there exists a suboptimal response ${\mathbf{y}}^{-}\in{\mathcal{S}}$ that lies in the high likelihood region of $\pi_{\theta^{(t)}}$, say $\pi_{\theta^{(t)}}({\mathbf{y}}^{-}\mid{\mathbf{x}})\geq p$ for some $p\in[0,1]$ that is close to $1$, and ${\mathbf{y}}^{-}$ is never sampled from $\pi^{(t)}=\pi_{\mu}$ thus not optimized as ${\mathbf{y}}_{l}^{(t)}$ during all $t$ rounds, we have:

$$ $\pi_{\theta^{(t)}}({\mathbf{y}}^{\star}\mid{\mathbf{x}})\leq 1-\pi_{\theta^{(t )}}({\mathbf{y}}^{-}\mid{\mathbf{x}})\leq 1-p.$ (7) $$

Since $\pi_{\mu}$ is zero except points that appear in ${\mathcal{D}}_{\mathrm{offline}}$, it is highly likely to find such a ${\mathbf{y}}^{-}$ not being sampled from $\pi_{\mu}$ (hence never optimized) and therefore $\pi_{\theta^{(t)}}({\mathbf{y}}^{\star}\mid{\mathbf{x}})$ can be very low with a large $p$.

On the other hand, conducting on-policy sampling can alleviate the “never-sampled” issue and promote convergence to the optimal policy. This is because whenever $\pi_{\theta^{(t-1)}}({\mathbf{y}}_{i}^{-}\mid{\mathbf{x}})$ is high, it is likely to sample ${\mathbf{y}}_{i}^{-}$ from $\pi^{(t)}=\pi_{\theta^{(t-1)}}$ and thus it can be optimized such that $\pi_{\theta^{(t)}}\left({\mathbf{y}}_{l}^{(t)}={\mathbf{y}}^{-}_{i}\mid{
\mathbf{x}}\right)\approx 0$. In this way, the subtrahend of Eq. 6 is decreased per round, hence we can gradually improve the language model policy towards the optimal policy.

Figure: Figure 2: Distribution of the length difference between the winning and losing examples ($|\mathbf{y}_{w}|-|\mathbf{y}_{l}|$). (Top) Distribution of the first round on-policy generated dataset. With LR reward shaping defined by Eq. 8, the length bias is mitigated and the length difference becomes more evenly distributed. The average length difference decreases from $1031$ to $-21$ by setting $\alpha$=$0.023$. (Bottom) Distribution of the high quality UltraFeedback preference dataset [] is almost unbiased.
Refer to caption: x2.png

### 4.2 Length-Regularized Implicit Rewards

It is a known issue in the literature that preference tuning may introduce length bias (or length exploitation) [] , which is likely caused by the fact that the preference labels collected from human annotators favor more verbose responses. This problem is further compounded by the iterative self-alignment scheme such as the one in Yuan et al. [], because the generated responses that are long and preferred will be reinforced in the next round of DPO, leading the language model to generate increasingly longer responses.

Inevitably, the vanilla DPO implicit rewards as in Eq. 4 would also exhibit length bias when generating preference dataset. In Figure 2, we show the distribution of the difference in string length of the winning and losing responses. We can see from the top figure that vanilla implicit rewards yield a skewed distribution (in green), with an average length difference $1031$. In stark contrast, the length difference of a high quality preference dataset is almost normally distributed (as in the bottom figure). This observation motivates us to debias the distribution induced by vanilla implicit rewards so as to mitigate the length exploitation. We resort to reward shaping [] for this purpose. In particular, we introduce a length-regularized (LR) reward shaping term in the implicit reward that penalizes the length of the response to obtain the shaped reward:

$$ $r_{\mathrm{LR}}(\mathbf{x},\mathbf{y};\alpha)=\beta\log\frac{\pi_{\theta}( \mathbf{y}\mid\mathbf{x})}{\pi_{\textrm{ref}}(\mathbf{y}\mid\mathbf{x})}- \alpha\left|\mathbf{y}\right|,$ (8) $$

where $\alpha$ controls the penalty strength and $\left|\mathbf{y}\right|$ is the string length of the response. Based on the shaped rewards, we can construct many versions of the preference dataset ${\mathcal{D}}(\alpha)$, following the principle that the response with the higest $r_{\mathrm{LR}}({\mathbf{x}},{\mathbf{y}}_{i};\alpha)$ is labeled as ${\mathbf{y}}_{w}$ and the one with the lowest reward is labeled as ${\mathbf{y}}_{l}$. To find the most suitable $\alpha$ such that ${\mathcal{D}}(\alpha)$ is (approximately) unbiased, we optimize $\alpha$ with the objective to minimize the average absolute difference in response length:

$$ $\alpha^{\star}=\operatorname*{arg\,min}_{\alpha}\mathbb{E}_{({\mathbf{y}}_{w}, {\mathbf{y}}_{l})\sim{\mathcal{D}}(\alpha)}\left(\Big{|}|{\mathbf{y}}_{w}|-|{ \mathbf{y}}_{l}|\Big{|}\right).$ (9) $$

We can employ any black-box optimizer to solve this non-differentiable objective function. In this work we find a simple random search suffices, and its solution effectively transforms the dataset into a more evenly distributed one (shown in the orange curve in the top of Figure 2). We will output ${\mathcal{D}}(\alpha^{\star})$ for the next round of DPO training. Details of the optimization can be found in Appendix B.

Importantly, despite the resemblance of Eq. 8 to Park et al. [] where they incorporate the token length as a regularizer in the training objective, our reward shaping is conducted during the dataset construction stage, thereby avoiding the need for expensive hyper-parameter tuning.

Figure: Algorithm 1 Bootstrapping with DPO Implicit Rewards (DICE)

### 4.3 Experience Replay

Though DICE enables us to learn from the response of the current policy, we know that the implicit reward model from the DPO training is not a perfect proxy for human preferences. Solely relying on the implicit reward model may result in forgetting the knowledge inbuilt in the initial policy at the first DPO stage. Inspired by the technique of experience replay [] in continual learning for preventing catastrophic forgetting and making a good balance between old and new data, we propose to use a mixture of the generated data and the offline preference dataset. While offline preference data is considered to be high-quality, it is off-policy samples; the generated data is closer to the on-policy samples, but the imperfect implicit reward model may introduce noise in labeling the preference. Combining the two can make for a good balance.

Our complete algorithm is summarized in Algorithm 1. During each iteration $t$, we generate $K$ responses ${\mathbf{y}}_{1},{\mathbf{y}}_{2},\ldots,{\mathbf{y}}_{K}$ from the policy $\pi_{\theta^{(t-1)}}$ for each prompt ${\mathbf{x}}$ (3). The preference dataset ${\mathcal{D}}(\alpha)$ is constructed based on the LR rewards $r_{\mathrm{LR}}$, where the response with the highest LR reward is labeled ${\mathbf{y}}_{w}$ and the one with the lowest reward is labeled ${\mathbf{y}}_{l}$. In this process, $\pi_{\theta^{(t-1)}}$ serves as the target policy, and $\pi_{\mathrm{ref}}^{(t-1)}$ serves as the reference policy. At 4, we construct the debiased dataset ${\mathcal{D}}(\alpha^{\star})$ by minimizing the average absolute difference in response length, referring to Eq. 9. Subsequently, a mixed dataset ${\mathcal{D}}_{t}$ is created by sampling $\gamma$ proportion of data from ${\mathcal{D}}_{\mathrm{offline}}$ and $(1-\gamma)$ proportion of data from ${\mathcal{D}}(\alpha^{\star})$ (5). DPO training is then conducted on ${\mathcal{D}}_{t}$ using $\pi_{\theta^{(t-1)}}$ as both the initial policy and the reference policy, resulting in the updated policy $\pi_{\theta^{(t)}}$ (6). Finally, at 7, we set $\pi_{\theta^{(t-1)}}$ as the new reference policy $\pi^{(t)}_{\mathrm{ref}}$.

## 5 Experiments

This section empirically investigates DICE. Our findings highlight several key points: (1) DICE significantly improves the model performance on the widely used leaderboard AlpacaEval 2.0 [], increasing length-controlled win rate by up to 9.35$\%$; (2) our best model bootstrapped from a 8B base model (Llama-3-8B-DPO) achieves a better performance than Gemini Pro without any extra human annotations (or external reward model) other than the preference dataset that is used in the initial DPO training of base model; (3) the two proposed techniques in Sections 4.2 and 4.3 are shown to be critical for DICE.

### 5.1 Experiment Setup

Base Models and Datasets. In this paper, we adopt Llama-3-8B-DPO(^2^22[https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT-DPO](https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT-DPO)) and zephyr-7B-beta(^3^33[https://huggingface.co/HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta)) as our base models. Both models are trained following the pipeline of Zephyr []. Llama-3-8B-DPO is trained based on meta-llama/Meta-Llama-3-8B(^4^44[https://huggingface.co/meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)), developed by Meng et al. []. zephyr-7B-beta is fine-tuned based on mistralai/Mistral-7B-v0.1(^5^55[https://huggingface.co/mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)). We adopt UltraFeedback preference dataset [] and randomly sample a subset of around 10k preference pairs as the offline dataset ${\mathcal{D}}_{\mathrm{offline}}$. Our experiment aims to show how much the language model can improve from a DPO-ed model and a subset of the preference dataset that was used to conduct the initial DPO training.

Response Generation and Dataset Construction. At the beginning of each round, we first sample responses from the current policy, with temperature $T=0.9$, $p=1.0$ for the Llama3 setting and $T=0.7$, $p=0.9$ for the Zephyr setting. We sample with different random seeds to get $K=16$ diverse responses for each prompt. We then reward each prompt-response pair by the implicit reward model (Eq. 4) and incorporate length-regularized reward shaping (Eq. 8) to get the debiased dataset ${\mathcal{D}}(\alpha^{\star})$ with the optimal regularization strength $\alpha^{\star}$. The final dataset is a mixture of ${\mathcal{D}}(\alpha^{\star})$ and ${\mathcal{D}}_{\mathrm{offline}}$.

Training Details. All experiments are conducted on 8 Nvidia A100 GPUs. For DICE, we trained two rounds in total. In each round, we train the model for 300 steps on a preference dataset with 9.6k preference pairs (either a solely generated dataset, or a mixture of the generated dataset and the offline preference dataset). The global training batch size is set to 32 and the learning rate is $5\textrm{e-}{7}$ with a constant schedule and a warm-up of 50 steps. We hypertune $\beta$ for each method and report their best performance. For our approach, we additionally hypertune the experience replay ratio $\gamma$.

Baselines. We evaluate the following baseline methods that are applicable to the setting of this paper:

- •
Offline DPO: continue conducting DPO training with the offline preference dataset.
- •
Offline DPO w/ new ref: similar to offline DPO but we assign the current policy as the new reference model, while we use a fixed reference model in Offline DPO. This corresponds to $\gamma=1$.
- •
DPO with prompted rewards: similar to Self-rewarding LM [], where they prompt the LLM itself as a preference judge to construct new preference pairs and iteratively fine-tune the LLM with the DPO algorithm. In their case, the judge capability is learned by supervised fine-tuning on an evaluation fine-tuning dataset. We exploit our base model to perform LLM-as-a-Judge directly as it can follow the judge instructions well. In the experiment, we call it LLM-as-a-Judge. The LLM-as-a-Judge prompt template can be found in Appendix A.

**Table 1: AlpacaEval 2.0 evaluation of various models (detailed in Baselines in Section 5.1) in terms of both normal win rates (WR) and length-controlled win rates (LC) in percentage (%).**
| Method | zephyr-7B-beta | Llama-3-8B-DPO |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| LC | WR | Avg. Len. | LC | WR | Avg. Len. |  |
| Base | 12.69 | 10.71 | 1444 | 18.20^∗ | 15.50^∗ | - |
| Offline DPO Iter 1 | 13.40 | 11.10 | 1544 | 20.22 | 18.33 | 1750 |
| Offline DPO Iter 2 | 4.96 | 5.47 | 2178 | 21.04 | 19.21 | 1761 |
| Offline DPO (w/ new ref) Iter 1 | 13.40 | 11.10 | 1544 | 22.29 | 19.96 | 1775 |
| Offline DPO (w/ new ref) Iter 2 | 4.58 | 5.27 | 2296 | 22.50 | 20.18 | 1737 |
| LLM-as-a-Judge Iter 1 | 17.81 | 15.36 | 2572 | 20.30 | 21.31 | 2029 |
| LLM-as-a-Judge Iter 2 | 14.14 | 17.89 | 2916 | 21.80 | 22.42 | 2051 |
| DICE Iter 1 | 19.03 | 17.67 | 1848 | 25.08 | 25.77 | 2201 |
| DICE Iter 2 | 20.71 | 20.16 | 1879 | 27.55 | 30.99 | 2427 |

Evaluation. We evaluate our method by AlpacaEval 2.0 []. AlpacaEval 2.0 is an LLM-based automatic evaluation benchmark. It employs AlpacaFarm [] as its prompts set composed of general human instructions. The model responses and the reference responses generated by GPT-4-Turbo are fed into a GPT-4-Turbo-based annotator to be judged. We follow the standard approach and report both the win rate (WR) and the Length-Controlled win rate (LC) [] over the reference responses.

### 5.2 Main Results

**Table 2: AlpacaEval 2.0 leaderboard results.**
| Model | LC | WR |
| --- | --- | --- |
| GPT-4 0613 | 30.18 | 15.76 |
| Mistral Medium | 28.61 | 21.86 |
| Claude 2 | 28.15 | 17.19 |
| DICE-Llama3 8B Iter 2 | 27.55 | 30.99 |
| Snorkel (Mistral-PairRM-DPO) | 26.39 | 30.22 |
| Gemini Pro | 24.38 | 18.18 |
| Mixtral 8×7B v0.1 | 23.69 | 18.26 |
| Llama 3 8B Instruct | 22.92 | 22.57 |
| GPT-3.5 Turbo 0613 | 22.35 | 14.10 |
| Tulu 2+DPO 70B | 21.24 | 15.98 |
| DICE-Zephyr 7B Iter 2 | 20.71 | 20.16 |
| GPT-3.5 Turbo 1106 | 19.30 | 9.18 |
| Llama-3-8B-DPO | 18.20 | 15.50 |
| Vicuna 33B v1.3 | 17.58 | 12.71 |
| zephyr-7B-beta | 12.69 | 10.71 |

DICE Effectively Improves a DPO-ed Model. With only a DPO-ed model and a preference dataset that was used to train this model, one can choose to further improve the current policy via Offline DPO, or construct a new preference dataset via LLM-as-a-Judge. In Table 1, we compare the performance of the model fine-tuned by DICE in two rounds with the base model and other baselines. It shows all methods can improve the LC win rate on AlpacaEval 2.0 over the base model while DICE leads to the most significant enhancement in both Zephyr and Llama3 settings, increasing by 8.02$\%$ and 9.35$\%$ respectively. We found that the LLM-as-a-Judge leads to good performance in the Zephyr setting while it has minor improvement in the Llama3 setting. We hypothesize this may be caused by the coarse rewards which are not able to provide effective preference signals when responses are of high quality (the prompt template requires LLM judge to provide a discrete score from 0 to 5, referring to Appendix A).
We note that training on a fixed offline dataset for multiple rounds leads to even worse performance than the base model, possibly due to the increased data staleness and overfitting.

Compare with the models on the leaderboard. Compared with the models on the public leaderboard shown in Table 2, DICE-Llama3 8B performs better than the official instruct version of Llama3 by a non-trivial margin, $4.63\%$. Regarding the closed-source models, it achieves better performance than Gemini Pro with only 8B parameters and does not require any in-house data or external reward model.

### 5.3 DICE Is Compatible with Other Direct Alignment from Preference Algorithms

Though DICE works best with DPO as it makes the iterative training possible (because the implicit reward model for the next round can be naturally derived using the updated policy), we would like to check if the dataset generated by DICE can also improve the base model with other DAP algorithms. In the Zephyr setting, we tune the policy model using DICE-generated dataset (at the first round) with KTO [], IPO [], and Hinge loss proposed in Zhao et al. []. The training follows the protocol described in Section 5.1. The results in Table 3 show that all DAP algorithms benefit from the newly generated data by the current policy and DPO implicit reward model with LR reward shaping, demonstrating LC win rates higher than the offline DPO. Meanwhile, DICE works best with DPO while IPO is the runner-up. It should be noted that DICE-IPO has a serious length exploitation issue even on the dataset where LR reward shaping has been applied.

Table: Table 3: DPO rewards are compatible with other direct preference optimization algorithms.

| Method | LC | WR | Avg. Len. |
| --- | --- | --- | --- |
| Offline DPO | 13.40 | 11.10 | 1544 |
| DICE-IPO | 18.51 | 19.49 | 2729 |
| DICE-KTO | 14.88 | 12.16 | 1541 |
| DICE-Hinge | 15.92 | 15.57 | 1970 |
| DICE-DPO | 19.03 | 17.67 | 1848 |

Table: Table 4: Effects of the Length Regularized Weight.

| $\gamma$ | $\alpha$ | LC | WR | Avg. Len. |
| --- | --- | --- | --- | --- |
| 0.0 | 0.000 | 13.32 | 15.37 | 2570 |
| 0.023 ($\alpha^{\star}$) | 18.88 | 19.31 | 2109 |  |
| 0.046 ($2\alpha^{\star}$) | 14.40 | 9.08 | 876 |  |
| 0.5 | 0.000 | 15.92 | 19.08 | 2600 |
| 0.023 ($\alpha^{\star}$) | 19.03 | 17.67 | 1848 |  |
| 0.046 ($2\alpha^{\star}$) | 14.91 | 10.80 | 1185 |  |

### 5.4 Ablation Study

Figure: Figure 3: AlpacaEval 2.0 LC Win Rate across different experience replay ratio ($\gamma$) in the Zephyr setting. The highest LC Win Rate is reported via text.
Refer to caption: x3.png

In this section, we investigate the effects of LR reward shaping and experience replay.

Effects of LR reward shaping. LR reward shaping (Eq. 8) penalizes responses for being too verbose, and guides the construction of a debiased dataset with the optimal penalty strength $\alpha^{\star}$ found by optimizing Eq. 9. To validate the effectiveness of the propose LR reward shaping as well as the $\alpha$-searching procedure, we run experiments in the Zephyr setting with different mixture ratios ($\gamma=0$ and $\gamma=0.5$) and ablate three design choices: (1) no LR reward shaping ($\alpha=0$); (2) LR reward shaping with penalty strength found by Eq. 9, i.e., $\alpha=\alpha^{\star}=0.023$; (3) LR reward shaping with slightly larger penalty strength ($\alpha=2\alpha^{\star}$). Results are presented in Table 4. For all values of the $\gamma$, we observe that $\alpha=0.0$ does lead to serious length exploitation.
So, even if the policy can get a high win rate, it will suffer in the LC win rate due to the length exploitation issue (responses with longer average length get a lower LC win rate), e.g., the low LC win rate with $\gamma=0.5,\alpha=0.0$. In contrast, a larger $\alpha$ seemingly mitigates the length exploitation issue even better, but it may adversely affect the response quality. Our proposed approach of finding $\alpha^{\star}$ using the objective of minimizing the absolute difference of response length does provide the best performance.

Effects of experience replay. The experience replay results in a new mixed dataset in which $\gamma$ fraction of the data is from the offline dataset and $(1-\gamma)$ fraction of the data is from the generated dataset. E.g., we use data only from the generated dataset if $\gamma=0.0$. We run experiments in the Zephyr setting with $\gamma\in\{0.0,0.25,0.5,0.75,1.0\}$ and conduct DICE in total of two rounds. From the results shown in Figure 3, we find $\gamma=0.5$ provides the best performance. The results satisfy our expectations. With only offline preference data, the DPO optimizes the current policy with off-policy samples that are further away from its distribution. With only its own generated data, the model may keep reinforcing its current “belief”, potentially leading to catastrophic forgetting. An intermediate value $\gamma=0.5$ finds a good balance. Additional results in the Llama3 setting can be found in Appendix C.

## 6 Limitations, Future Work and Broader Impacts

Limitations. One of the primary limitations of our work is the reliance on the DPO training prior to bootstrapping. If the implicit reward model is not well-trained, it can lead to a collapse of the training pipeline. This challenge is not unique to our approach; the classic RLHF pipeline is also struggling when its reward model is not well-trained. Another limitation is the lack of continued improvement over many iterations. Similar to other research, such as the work by [], which enhances the policy model without an external reward model, we did not observe continuous improvement in our model beyond three iterations. This issue highlights an open question within this field regarding the iterative enhancement of policy models.

Future Work. Future research could explore the rewarding capabilities of models trained using other DPO variants, such as KTO and IPO. Investigating whether these variants can offer general rewards similar to those provided by DPO-ed policy would be valuable. Another promising direction involves developing methods that enable continuous improvement of the policy model over iterations without degradation. Additionally, investigating a theoretical understanding of the policy learned through self-bootstrapping could provide deeper insights into the mechanics of our approach and facilitate further advancements.

Broader Impacts. Due to its simplicity in implementation and its ability to enhance the policy model effectively, our method has the potential to be misused, for instance, in generating harmful content. The ease of use and the capability to produce stronger policy models might attract malicious actors who seek to exploit these features for unethical purposes. It is crucial to address these concerns by developing safeguards and ethical guidelines to prevent misuse.

## 7 Conclusion

In this paper, we introduce DICE, a novel approach that leverages the implicit reward model from DPO to further align LLMs with human preferences. Our method stands out in the current landscape of LLM alignment research, as it uses the implicit reward model to iteratively refine the policy model. Empirical results demonstrate that DICE surpasses the closed-source language models such as Gemini Pro, despite utilizing a model with only 8B parameters. This efficiency in parameter usage does not come at the cost of additional human annotations or external reward models, underscoring the practicality and accessibility of our approach.

## Appendix A Prompts Used by LLM-as-a-Judge

Figure: Figure 4: We follow the prompt template used by Yuan et al. [] to use LLMs to judge model responses and construct paired dataset for further preference tuning.

## Appendix B Optimization for Length Regularized Reward Shaping

Figure: Figure 5: The objective landscape for (Left) our implicit reward model and (Right) the LLM-as-a-Judge reward model.
Refer to caption: x4.png

We solve the objective in Eq. 9 using a simple Bayesian optimization toolkit based on Gaussian process(^6^66[https://scikit-optimize.github.io/stable/modules/generated/skopt.gp_minimize.html](https://scikit-optimize.github.io/stable/modules/generated/skopt.gp_minimize.html)). The objective landscape with respect to $\alpha$ is depicted in Figure 5, where we compare the proposed implicit reward model with the LLM-as-a-Judge reward model. With our length-reguarlized implicit rewards, the optimizer is able to find the optimal solution quickly that debiases the length difference of the winning and losing responses. For LLM-as-a-Judge rewards, the optimal solution is obtained with $\alpha=0$, hence we do not explicitly debias the dataset for all the experiments.

## Appendix C Extended Ablation Study

Figure: Figure 6: AlpacaEval 2.0 LC Win Rate across different experience replay ratio ($\gamma$) in the Llama3 setting.
Refer to caption: x5.png

In the Llama3 setting, we also conduct a coarse sweeping for the experience ratio $\gamma$, and present the AlpacaEval 2.0 LC win rate in Figure 6 for two self-alignment rounds. We observe similar trends to those in the Zephyr setting, which further justify the effectiveness of the proposed experience replay: it helps to keep a balance between the more on-policy generated data and the curated offline data. The best identified value of the mixture ratio is $\gamma=0.1$.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: N/A
soundness: 3
strengths: The writing is clear and easy to follow, and the experiments provide support for the claim that repeated use of the implicit reward model could enhance performance.
summary: This paper introduces a self-alignment approach, DICE, which uses the implicit reward model from DPO in a bootstrapping manner to further align LLMs, achieving over 8% improvement in alignment on AlpacaEval 2 without external feedback.
weaknesses: The results offer a useful insight, though the approach itself may lack significant novelty, and the improvements seem to be marginal. Additionally, length-regularized reward shaping is already a widely adopted technique. While the paper seeks to improve model performance through Direct Preference Optimization (DPO) by leveraging the implicit reward model, it would benefit from a theoretical explanation of why repeated use of the implicit reward model could enhance performance. Moreover, it might be worth considering whether repeatedly using the implicit reward model offers more advantages than utilizing an additional reward model (RM) to label subsequent iterations.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. Are the base models both trained on the same UltraFeedback dataset from which the offline dataset is sampled? It seems likely yes, but it would be good to clarify this in Section 5.1, since it affects how to interpret the comparison with the base and offline DPO models.
2. What is the length bias like for LLM-as-a-Judge? If it is too large, an ablation that uses a length penalty with this method would seem appropriate.
soundness: 2
strengths: The method is natural and straightforward to apply: using the current policy to generate both new on-policy samples and rewards provides and easy way to do iterative policy improvement with no additional models or data. I expect the method to be of great interest to practitioners who value this benefit, which is similar to the benefit of DPO compared to PPO for RLHF.

The experiments are thorough and promising enough that I expect others will want to try out the method in their own settings, although I discuss a major weakness below. The ablations are also well-chosen and interesting. In particular, the ablation for the choice of alpha* mostly satisfied me that there is a real benefit to the method beyond the length penalty, although see my question 2 on this point.

I thought the hypothesis about on-policy data helping was a good one, but I would suggest an additional hypothesis for why the method helps, namely that using the implicit rewards is similar to training a reward model, so as with PPO there may be a benefit from discrimination being easier than generation.
summary: The paper introduces a new approach to iterative DPO. After each iteration, synthetic comparisons are generated by the current policy: given a prompt, 16 completions are sampled and then labeled using the implicit reward from the DPO formalism; a length penalty is applied to make the comparison dataset approximately length-unbiased; and the best and worst completions are used for the synthetic comparison. These synthetic comparisons are then used for the subsequent iteration. The method outperforms baselines on two different benchmarks across two different models.
weaknesses: A glaring methodological weakness is in the treatment of the hyperparameter gamma (1 minus the fraction of offline data used): if I am understanding correctly, the tuning was done using the final reported results. This is contrary to standard machine learning best practices, whereby hyperparameters are tuned using a validation set. This gives the method a significant advantage if the difference between different values of gamma has a significant noise component. The same applies to the hyperparameter beta, but the issue is not as important there because the same tuning was done for the baselines too (although I still think this should be fixed – in particular, the Base model does not get to tune beta). Regardless, some clarification should be given as to how the tuning for beta was done (is it the same across models/methods/iterations/benchmarks?). I do not believe the hyperparameter K was tuned, but it could be good to clarify how K = 16 was chosen.

It looks like the same value of gamma was used across benchmarks, which is good since this reduces the selection effect, but different values of gamma were used between different models (and moreover, this fact is not clearly disclosed, I attempted to infer this by comparing Table 1 with Appendix C). Since it would take a lot of work to re-run all the experiments with fairly-tuned hyperparameters, my suggested remedy is as follows: **report results for the same value of gamma (say gamma = 0.5) across all experiments from both models**. I think this would go a long way towards presenting a fairer picture of the algorithm's performance compared to other methods.

**I have rated the paper and 8 in spite of this scientific mis-step in anticipation of some sort of correction or explanation.** Without the above suggested change or some other correction or explanation, I may lower my score, which would be a shame, because it is otherwise a great paper. Similarly, I may also raise my score to a 10 if I become convinced that this problem has been adequately addressed.

Another limitation is a lack of comparison to reward model-based methods such as PPO, but I think it is reasonable to leave this to future work. I think there are some interesting comparisons to be made here because the use of the implicit rewards is similar to training a reward model and using the rewards from that.

The presentation was generally pretty clear, although I found Algorithm 1 somewhat hard to follow (and it would be even harder for someone glancing at the paper), for a couple of reasons:
- It seems worth stating that the comparison dataset is created by taking the best and worst samples according to the (length-penalized) reward.
- I believe there is a typo where it says to use "pi_ref^(t-1) as reference policy", and it should be pi_ref^(t) (unless I am missing some subtlety that deserves further explanation). But since pi_ref^(t) is equal to pi_{theta^(t-1)}, I think it would be simpler and clearer to remove the notation pi_ref^(t) from the pseudocode entirely.

Minor line-by-line comments:
- Abstract: "refinements that debias the length of the responses and enhance the quality of the preference dataset" - this phrase is convoluted and unclear, suggest replacing with "a length penalty to make the preference dataset length-unbiased" or just "a length penalty" may suffice.
- Line 046: typo "by itself" -> "by DPO itself"
- Line 216: might be good to clarify where does pi_mu comes from in practice
- Line 227: "without minimizing the likelihood of other suboptimal responses" - I didn't understand what point you were trying to make, since the loss doesn't depend on these other responses by definition?
- Line 245: "subtrahend" - I had never seen this word before but it made sense after I Googled it :)
- Line 291.5: "normally distributed" - It does not look normally distributed at all, there is a big spike at 0. It looks more like a Laplace distribution (although I wouldn't want to make this claim without statistical evidence, and in any case it doesn't seem important to the exposition).
- Equation (9) - It would seem more natural to put the absolute value outside the expectation instead of inside, since this would directly optimize for the dataset being unbiased in the sense of the mean length of winning completions roughly matching the mean length of losing completions. That said, this probably gives a very similar value of alpha* and so this change would mostly be for improved theoretical soundness.
- Tables 3 and 4: I assume this is for AlpacaEval 2, but maybe good to clarify in the table captions.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - If there is anything the authors can do to convince me that using the adjusted DPO implicit rewards has an advantage over training a separate reward model for iterative finetuning, this would be greatly appreciated (see Weakness #1 above for some ideas). (I am aware that using the DPO implicit reward is computationally more efficient, but I do not believe that advantage alone is sufficient, at least not without fully understanding the compute-return trade-off here).
soundness: 2
strengths: Iterative finetuning on self-supervised data is an important direction, which has received a reasonable amount of attention, e.g., with Anthropic's Constitutional AI / RLAIF, and others. The DPO approach has also gained significant adoption, so improving DPO with iterative finetuning would be of interest to the community. 

The proposed procedure is straightforward and the results are good. The paper as a whole is well written and presented. I believe I could reimplement the method/experiments based on the presented details. The implementation is novel, as is the use of implicit rewards for iterative finetuning as far as I am aware.
summary: This paper proposes a self-supervised iterative finetuning procedure for LMs trained with DPO, which, after the first round of DPO (with human data), uses the Implicit Reward Model (as used by DPO) to label new self-supervised preference data, and then trains again. This process can be repeated. To obtain good performance, the authors propose 2 techniques that improve the nth stage of finetuning and show that these techniques are critical for thei good results: adjusting the Implicit Rewards by a length penalty, and using an experience replay to mix in the human labeled data. The authors find that this iterative procedure obtains better results than several baselines on AlpacaEval and ArenaHard.
weaknesses: Main concern: 
- I remain unconvinced this is the best approach to iterative finetuning a DPO model, since the evaluation only tests downstream performance and other potential reward models that are known to be empirically stronger than implicit DPO rewards are ignored: If you look at RewardBench, the implicit rewards of DPO models are quite far down the list in terms of their strength as a reward model. Furthermore, reward models themselves are not very difficult to train, so the overhead as compared to using the implicit DPO reward is not that large. See also Lin et al. 2024, On the Limited Generalization Capability of the Implicit Reward Model... 
So for these reasons, my prior that using a trained RM for iteratively improving a DPO model (e.g., as done by He et al. 2024, Semi-Supervised Reward Modeling via Iterative Self-Training...) is unchanged after reading this paper... i.e., I do not think the proposed approach is competitive with simple alternatives, which do not appear as baselines.  
To convince me here (or alternatively, to contribute a "full" empirical study that determines what is still an open question in my mind), I would want to see the iterative DPO w/ trained RM show up in Table 1, OR alternatively, I would want to see that the implicit reward model that is learned via this iterative process (and adjusted with the LC normalization) is competitive vs other RMs on RewardBench or similar. 
Side note: esp. given the length regularizer, avg log prob reward (see simpo) would be a good baseline to have

Concern: 
- Limited Conceptual Novelty: While the use of implicit rewards for iterative finetuning is new, neither iterative finetuning, nor the implicit reward model are new concepts... and it is in some sense "obvious" that one can use any reward model for iterative finetuning, including the implicit reward model. Therefore, the main novelty in my view is limited to (a) empirical results (which are good, but see point above), and (b) the length-normalization penalty (although others have also investigated different variations of this) and experience replay (not sure I would call this new, but the empirical demonstration of its usefulness in this context is good). 

Lesser concern:
- I think Section 4.1 detracts from the paper, rather than adds to it. E.g., Equation (6) seems wrong to me (unless you mean S to be ALL possible completions besides the "optimal" one, but this is also wrong, because "optimal" is ill defined here (the optimal response out of all possible completions under the implicit reward will likely be overoptimized / OOD / poor). I'm not sure this adds much intuition, and perhaps even supports the wrong intuitions (e.g., just because a high likelihood suboptimal response is not sampled, that doesn't mean it's probability doesn't change... and this criticism applies to finite on-policy data as well). 

Minor/Irrelevant to score:
- I would call the length penalty reward specification/augmentation/engineering rather than reward shaping, which in my experience is reserved for rewards that are meant to improve the learning process but maintain the same desired policy (see Ng et al. 1999, Policy Invariance...) 
- Tortured acronym that is already used elsewhere. Consider something more unique that actually maps to the method name, e.g., Iterative Finetuning with Implicit Reward, or if you insist on a "word", Finetuning with Implicit REwards gives you FIRE, so you can do FFIRE (Finetuning Further) or IFIRE (Iterative FIRE).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Is foregoing the partition function Z principled in this setting? This is an important design choice that is not discussed in the current state.

IRPO (Pang et al, 2024) [1] should be mentioned (maybe when discussing iterative DPO)

Authors could draw a parallel between the replay buffer approach and the learning from demonstrations literature (e.g. DQfD, Hester et al, 2017 [2])

What is the granularity of the search for $\alpha^*$ ? What is the cost of conducting such experiments?

Self-rewarding DPO (Yuan et al, 2024) [3] seems very much connected: is it a baseline here? It should be.

[1] - Iterative Reasoning Preference Optimization, arxiv preprint, Pang et al, 2024 

[2] - Deep Q-learning from Demonstrations, AAAI, Hester et al, 2017

[3] - Self-Rewarding Language Models, arxiv preprint, Yuan et al, 2024
soundness: 3
strengths: Writing is mostly good and clear (but see problems below).

Length bias is clearly illustrated and mitigated by the length penalty (though not a technical contribution as discussed below).

Table 4 presents some good results: one can have short responses scoring rather high (higher than the $\alpha=0$ baseline).

Results are good on AlpacaEval and Arena Hard.

EDIT: I raised my score significantly after authors addressed major concerns of mine. I thank authors for the diligent work on making the submission better and fostering proper credit assignment.
summary: The authors propose to use the implicit reward model learned via DPO to create a preference dataset and further improve the policy learned by an extra DPO step on the newly created preference dataset. They propose two additional techniques: 1) length regularization to prevent the known issue of response length increasing through DPO and 2) incorporating human preference data among the preference data collected with the final policy from the previous iteration and annotated by the implicit RM from the previous iteration.
weaknesses: A key weakness in the current state is that authors should connect their approach for length bias mitigation to existing approaches: it seems to me that this is the exact same approach as in Park et al, 2024 (which is cited but credit is not sufficiently assigned as of now). While I understand that the use of the length penalty is new, it currently reads as if authors came up with the penalty from Park et al, which is unfair to them. I make a non-exhaustive list of the places where credit / claims must be clarified:
* “We address the length exploitation issue by length-regularized reward shaping, which discourages long responses from being preferred”
    * I understand this sentence as the length regularization being an actual contribution, which is not the case. Please fix.
* “We propose utilizing the length regularized (LR) implicit rewards introduced in Section 4.2”
    * these are the same as in Park et al, so again this sentence is misleading
* “We propose to apply two techniques together with our above proposed approach, length regularized reward shaping and experience replay”
    * same: applying an existing technique (which was developed in a very close context) is not a technical contribution here
* “Next, in Section 4.2, we introduce the proposed Length-Regularized Implicit Rewards, which augment the vanilla implicit rewards with a length-regularized reward shaping, to judge the on-policy sampled responses for constructing the preference dataset”
    * same comment
* “Importantly, despite the resemblance of Eq. (8) to Park et al. (2024) where they incorporate the token length as a regularizer in the training objective, our reward shaping is conducted during the dataset construction stage, thereby avoiding the need for expensive hyper-parameter tuning” -> this needs to be explicited before! and does not prevent from toning down the novelty claims on the length bias mitigation

In the same vein, why not using the regularized DPO from Park et al as a baseline?? If the length bias mitigation proposed works better, this would make for a good contribution of the paper, but then this should be backed by explicit comparisons.

I also find that the authors did not get to the bottom of the lack of improvement past iteration 2 yet:
* “we did not observe continuous improvement in our model beyond three iterations” -> why is this not presented in graphs? that is important data that is currently (afaict) not in the paper
* Unclear why there is no improvement after K iterations (K > 2), is it because the policy becomes too deterministic? any root cause analysis would be valuable here 

Mathematical notations and statements
* I do not understand why using $\hat{r}$ instead of $r^*$ (this is the optimal reward!) l. 34
* “For a prompt x, we denote its optimal response as y^⋆”
    * in general there can be multiple optimal responses (as there can be multiple optimal policies), please adapt the statement
    * but here authors mean preferred response? this really needs clarification

Writing
* “In this way, the subtrahend of Eq. (6)”
* “To validate the effectiveness of the propose LR reward”

Clarity
* Section 4.1 as a whole is really confusing in the current state.
* citation missing for RLHF (Christiano et al) l. 28
* when mentioning the length exploitation issue, authors could quickly explain why the models exploit length (which is because longer responses are correlated with higher probability of being preferred by human or machine-based preferences)
* no citation for Gemini Pro! also the version used should be stated (1.0 vs 1.5 vs 1.5 002)
* “Offline DPO w/ new ref: similar to offline DPO but we assign the current policy as the new reference model, while we use a fixed reference model in Offline DPO. This corresponds to γ = 1” -> misleading as my understanding is that the reference is fixed (it is just the end policy of the first round of DPO) 

Experiments / ablations missing
* What about using all the K samples with a Plackett-Luce model? Does this bring any benefit? 
* Replaying human preference data seems similar to KL regularization to the previous policy -> would be interesting to compare
* Baseline missing: offline AI feedback annotations instead of implicit RM (+ online would be nice but harder/costlier to setup)

Meta-review: This paper provides a new approach for self-improvement in LLMs using implicit DPO rewards. The idea is to first perform DPO on a preference data, and then generate online samples, and use the learned implicit rewards to generate extra preference data and keep training. This idea makes sense and to my knowledge is novel.

Strengths:
1. Novel idea for self-improvement that avoids training a separate reward model or relying on it.
2. Results show that it does better than using LLM judge for selecting preference
3. Surprisingly, DICE does better than training a reward model on the preference data (IntlRM) and an external reward model. 

Weakness:
1. Unclear how it would fair against a big space of preference learning literature, especially literature on reward learning. Authors do present useful baselines, particularly, using LLM judge and training a reward model.

Overall, I think the idea makes sense and the results are positive. I especially like that DICE does better than using LLM judge and doesn't require training in another reward model which is important from a practical point of view. Further, it is possible that implicit rewards are better for in-distribution tasks over general-purpose rewards. For this reason, I am recommending acceptance. 

That said, I have a concern that as there are a lot of self-improvement learning methods using preferences and we don't quite have a grip on the best strategies given the early stages of this field. However, I think it is hard for a single paper outside the frontier labs to do an exhaustive evaluation of all proposed methods on a variety of tasks.

Decision: Accept (Poster)