Bootstrapping Language Models with DPO Implicit Rewards

Authors: Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakantham, Min Lin
Primary area: foundation or frontier models, including LLMs
Keywords: Alignment, Direct Preference Optimization, Large Language Models

Abstract:
Human alignment in large language models (LLMs) is an active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO, after training, provides an implicit reward model. In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM. Our approach is to use the rewards from a current LLM to construct a preference dataset, which is then used in subsequent DPO rounds. We incorporate two refinements to further improve our approach: 1) length-regularized reward shaping to make the preference dataset length-unbiased; 2) experience replay to enhance the quality of the preference dataset. Our approach, named self-alignment with DPO ImpliCit rEwards (DICE), shows great improvements in alignment. It achieves an increase of more than 8$\\%$ in lengthcontrolled win rate on AlpacaEval 2 for all the different base models that we tried, without relying on external feedback. Our code is available at https://github.com/sail-sg/dice.

Decision: accept (poster)
Reviewer scores: 3, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1730457098861

thread_type:
official_review

<thread_object>
creator: reviewer_3zep

<content>
summary:
The authors propose to use the implicit reward model learned via DPO to create a preference dataset and further improve the policy learned by an extra DPO step on the newly created preference dataset. They propose two additional techniques: 1) length regularization to prevent the known issue of response length increasing through DPO and 2) incorporating human preference data among the preference data collected with the final policy from the previous iteration and annotated by the implicit RM from the previous iteration.

strengths:
Writing is mostly good and clear (but see problems below).

Length bias is clearly illustrated and mitigated by the length penalty (though not a technical contribution as discussed below).

Table 4 presents some good results: one can have short responses scoring rather high (higher than the $\alpha=0$ baseline).

Results are good on AlpacaEval and Arena Hard.

EDIT: I raised my score significantly after authors addressed major concerns of mine. I thank authors for the diligent work on making the submission better and fostering proper credit assignment.

weaknesses:
A key weakness in the current state is that authors should connect their approach for length bias mitigation to existing approaches: it seems to me that this is the exact same approach as in Park et al, 2024 (which is cited but credit is not sufficiently assigned as of now). While I understand that the use of the length penalty is new, it currently reads as if authors came up with the penalty from Park et al, which is unfair to them. I make a non-exhaustive list of the places where credit / claims must be clarified:
* “We address the length exploitation issue by length-regularized reward shaping, which discourages long responses from being preferred”
    * I understand this sentence as the length regularization being an actual contribution, which is not the case. Please fix.
* “We propose utilizing the length regularized (LR) implicit rewards introduced in Section 4.2”
    * these are the same as in Park et al, so again this sentence is misleading
* “We propose to apply two techniques together with our above proposed approach, length regularized reward shaping and experience replay”
    * same: applying an existing technique (which was developed in a very close context) is not a technical contribution here
* “Next, in Section 4.2, we introduce the proposed Length-Regularized Implicit Rewards, which augment the vanilla implicit rewards with a length-regularized reward shaping, to judge the on-policy sampled responses for constructing the preference dataset”
    * same comment
* “Importantly, despite the resemblance of Eq. (8) to Park et al. (2024) where they incorporate the token length as a regularizer in the training objective, our reward shaping is conducted during the dataset construction stage, thereby avoiding the need for expensive hyper-parameter tuning” -> this needs to be explicited before! and does not prevent from toning down the novelty claims on the length bias mitigation

In the same vein, why not using the regularized DPO from Park et al as a baseline?? If the length bias mitigation proposed works better, this would make for a good contribution of the paper, but then this should be backed by explicit comparisons.

I also find that the authors did not get to the bottom of the lack of improvement past iteration 2 yet:
* “we did not observe continuous improvement in our model beyond three iterations” -> why is this not presented in graphs? that is important data that is currently (afaict) not in the paper
* Unclear why there is no improvement after K iterations (K > 2), is it because the policy becomes too deterministic? any root cause analysis would be valuable here 

Mathematical notations and statements
* I do not understand why using $\hat{r}$ instead of $r^*$ (this is the optimal reward!) l. 34
* “For a prompt x, we denote its optimal response as y^⋆”
    * in general there can be multiple optimal responses (as there can be multiple optimal policies), please adapt the statement
    * but here authors mean preferred response? this really needs clarification

Writing
* “In this way, the subtrahend of Eq. (6)”
* “To validate the effectiveness of the propose LR reward”

Clarity
* Section 4.1 as a whole is really confusing in the current state.
* citation missing for RLHF (Christiano et al) l. 28
* when mentioning the length exploitation issue, authors could quickly explain why the models exploit length (which is because longer responses are correlated with higher probability of being preferred by human or machine-based preferences)
* no citation for Gemini Pro! also the version used should be stated (1.0 vs 1.5 vs 1.5 002)
* “Offline DPO w/ new ref: similar to offline DPO but we assign the current policy as the new reference model, while we use a fixed reference model in Offline DPO. This corresponds to γ = 1” -> misleading as my understanding is that the reference is fixed (it is just the end policy of the first round of DPO) 

Experiments / ablations missing
* What about using all the K samples with a Plackett-Luce model? Does this bring any benefit? 
* Replaying human preference data seems similar to KL regularization to the previous policy -> would be interesting to compare
* Baseline missing: offline AI feedback annotations instead of implicit RM (+ online would be nice but harder/costlier to setup)

questions:
Is foregoing the partition function Z principled in this setting? This is an important design choice that is not discussed in the current state.

IRPO (Pang et al, 2024) [1] should be mentioned (maybe when discussing iterative DPO)

Authors could draw a parallel between the replay buffer approach and the learning from demonstrations literature (e.g. DQfD, Hester et al, 2017 [2])

What is the granularity of the search for $\alpha^*$ ? What is the cost of conducting such experiments?

Self-rewarding DPO (Yuan et al, 2024) [3] seems very much connected: is it a baseline here? It should be.

[1] - Iterative Reasoning Preference Optimization, arxiv preprint, Pang et al, 2024 

[2] - Deep Q-learning from Demonstrations, AAAI, Hester et al, 2017

[3] - Self-Rewarding Language Models, arxiv preprint, Yuan et al, 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1730492360687

thread_type:
official_review

<thread_object>
creator: reviewer_a3uc

<content>
summary:
This paper proposes a self-supervised iterative finetuning procedure for LMs trained with DPO, which, after the first round of DPO (with human data), uses the Implicit Reward Model (as used by DPO) to label new self-supervised preference data, and then trains again. This process can be repeated. To obtain good performance, the authors propose 2 techniques that improve the nth stage of finetuning and show that these techniques are critical for thei good results: adjusting the Implicit Rewards by a length penalty, and using an experience replay to mix in the human labeled data. The authors find that this iterative procedure obtains better results than several baselines on AlpacaEval and ArenaHard.

strengths:
Iterative finetuning on self-supervised data is an important direction, which has received a reasonable amount of attention, e.g., with Anthropic's Constitutional AI / RLAIF, and others. The DPO approach has also gained significant adoption, so improving DPO with iterative finetuning would be of interest to the community. 

The proposed procedure is straightforward and the results are good. The paper as a whole is well written and presented. I believe I could reimplement the method/experiments based on the presented details. The implementation is novel, as is the use of implicit rewards for iterative finetuning as far as I am aware.

weaknesses:
Main concern: 
- I remain unconvinced this is the best approach to iterative finetuning a DPO model, since the evaluation only tests downstream performance and other potential reward models that are known to be empirically stronger than implicit DPO rewards are ignored: If you look at RewardBench, the implicit rewards of DPO models are quite far down the list in terms of their strength as a reward model. Furthermore, reward models themselves are not very difficult to train, so the overhead as compared to using the implicit DPO reward is not that large. See also Lin et al. 2024, On the Limited Generalization Capability of the Implicit Reward Model... 
So for these reasons, my prior that using a trained RM for iteratively improving a DPO model (e.g., as done by He et al. 2024, Semi-Supervised Reward Modeling via Iterative Self-Training...) is unchanged after reading this paper... i.e., I do not think the proposed approach is competitive with simple alternatives, which do not appear as baselines.  
To convince me here (or alternatively, to contribute a "full" empirical study that determines what is still an open question in my mind), I would want to see the iterative DPO w/ trained RM show up in Table 1, OR alternatively, I would want to see that the implicit reward model that is learned via this iterative process (and adjusted with the LC normalization) is competitive vs other RMs on RewardBench or similar. 
Side note: esp. given the length regularizer, avg log prob reward (see simpo) would be a good baseline to have

Concern: 
- Limited Conceptual Novelty: While the use of implicit rewards for iterative finetuning is new, neither iterative finetuning, nor the implicit reward model are new concepts... and it is in some sense "obvious" that one can use any reward model for iterative finetuning, including the implicit reward model. Therefore, the main novelty in my view is limited to (a) empirical results (which are good, but see point above), and (b) the length-normalization penalty (although others have also investigated different variations of this) and experience replay (not sure I would call this new, but the empirical demonstration of its usefulness in this context is good). 

Lesser concern:
- I think Section 4.1 detracts from the paper, rather than adds to it. E.g., Equation (6) seems wrong to me (unless you mean S to be ALL possible completions besides the "optimal" one, but this is also wrong, because "optimal" is ill defined here (the optimal response out of all possible completions under the implicit reward will likely be overoptimized / OOD / poor). I'm not sure this adds much intuition, and perhaps even supports the wrong intuitions (e.g., just because a high likelihood suboptimal response is not sampled, that doesn't mean it's probability doesn't change... and this criticism applies to finite on-policy data as well). 

Minor/Irrelevant to score:
- I would call the length penalty reward specification/augmentation/engineering rather than reward shaping, which in my experience is reserved for rewards that are meant to improve the learning process but maintain the same desired policy (see Ng et al. 1999, Policy Invariance...) 
- Tortured acronym that is already used elsewhere. Consider something more unique that actually maps to the method name, e.g., Iterative Finetuning with Implicit Reward, or if you insist on a "word", Finetuning with Implicit REwards gives you FIRE, so you can do FFIRE (Finetuning Further) or IFIRE (Iterative FIRE).

questions:
- If there is anything the authors can do to convince me that using the adjusted DPO implicit rewards has an advantage over training a separate reward model for iterative finetuning, this would be greatly appreciated (see Weakness #1 above for some ideas). (I am aware that using the DPO implicit reward is computationally more efficient, but I do not believe that advantage alone is sufficient, at least not without fully understanding the compute-return trade-off here).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730495292676

thread_type:
official_review

<thread_object>
creator: reviewer_3rhj

<content>
summary:
The paper introduces a new approach to iterative DPO. After each iteration, synthetic comparisons are generated by the current policy: given a prompt, 16 completions are sampled and then labeled using the implicit reward from the DPO formalism; a length penalty is applied to make the comparison dataset approximately length-unbiased; and the best and worst completions are used for the synthetic comparison. These synthetic comparisons are then used for the subsequent iteration. The method outperforms baselines on two different benchmarks across two different models.

strengths:
The method is natural and straightforward to apply: using the current policy to generate both new on-policy samples and rewards provides and easy way to do iterative policy improvement with no additional models or data. I expect the method to be of great interest to practitioners who value this benefit, which is similar to the benefit of DPO compared to PPO for RLHF.

The experiments are thorough and promising enough that I expect others will want to try out the method in their own settings, although I discuss a major weakness below. The ablations are also well-chosen and interesting. In particular, the ablation for the choice of alpha* mostly satisfied me that there is a real benefit to the method beyond the length penalty, although see my question 2 on this point.

I thought the hypothesis about on-policy data helping was a good one, but I would suggest an additional hypothesis for why the method helps, namely that using the implicit rewards is similar to training a reward model, so as with PPO there may be a benefit from discrimination being easier than generation.

weaknesses:
A glaring methodological weakness is in the treatment of the hyperparameter gamma (1 minus the fraction of offline data used): if I am understanding correctly, the tuning was done using the final reported results. This is contrary to standard machine learning best practices, whereby hyperparameters are tuned using a validation set. This gives the method a significant advantage if the difference between different values of gamma has a significant noise component. The same applies to the hyperparameter beta, but the issue is not as important there because the same tuning was done for the baselines too (although I still think this should be fixed – in particular, the Base model does not get to tune beta). Regardless, some clarification should be given as to how the tuning for beta was done (is it the same across models/methods/iterations/benchmarks?). I do not believe the hyperparameter K was tuned, but it could be good to clarify how K = 16 was chosen.

It looks like the same value of gamma was used across benchmarks, which is good since this reduces the selection effect, but different values of gamma were used between different models (and moreover, this fact is not clearly disclosed, I attempted to infer this by comparing Table 1 with Appendix C). Since it would take a lot of work to re-run all the experiments with fairly-tuned hyperparameters, my suggested remedy is as follows: **report results for the same value of gamma (say gamma = 0.5) across all experiments from both models**. I think this would go a long way towards presenting a fairer picture of the algorithm's performance compared to other methods.

**I have rated the paper and 8 in spite of this scientific mis-step in anticipation of some sort of correction or explanation.** Without the above suggested change or some other correction or explanation, I may lower my score, which would be a shame, because it is otherwise a great paper. Similarly, I may also raise my score to a 10 if I become convinced that this problem has been adequately addressed.

Another limitation is a lack of comparison to reward model-based methods such as PPO, but I think it is reasonable to leave this to future work. I think there are some interesting comparisons to be made here because the use of the implicit rewards is similar to training a reward model and using the rewards from that.

The presentation was generally pretty clear, although I found Algorithm 1 somewhat hard to follow (and it would be even harder for someone glancing at the paper), for a couple of reasons:
- It seems worth stating that the comparison dataset is created by taking the best and worst samples according to the (length-penalized) reward.
- I believe there is a typo where it says to use "pi_ref^(t-1) as reference policy", and it should be pi_ref^(t) (unless I am missing some subtlety that deserves further explanation). But since pi_ref^(t) is equal to pi_{theta^(t-1)}, I think it would be simpler and clearer to remove the notation pi_ref^(t) from the pseudocode entirely.

Minor line-by-line comments:
- Abstract: "refinements that debias the length of the responses and enhance the quality of the preference dataset" - this phrase is convoluted and unclear, suggest replacing with "a length penalty to make the preference dataset length-unbiased" or just "a length penalty" may suffice.
- Line 046: typo "by itself" -> "by DPO itself"
- Line 216: might be good to clarify where does pi_mu comes from in practice
- Line 227: "without minimizing the likelihood of other suboptimal responses" - I didn't understand what point you were trying to make, since the loss doesn't depend on these other responses by definition?
- Line 245: "subtrahend" - I had never seen this word before but it made sense after I Googled it :)
- Line 291.5: "normally distributed" - It does not look normally distributed at all, there is a big spike at 0. It looks more like a Laplace distribution (although I wouldn't want to make this claim without statistical evidence, and in any case it doesn't seem important to the exposition).
- Equation (9) - It would seem more natural to put the absolute value outside the expectation instead of inside, since this would directly optimize for the dataset being unbiased in the sense of the mean length of winning completions roughly matching the mean length of losing completions. That said, this probably gives a very similar value of alpha* and so this change would mostly be for improved theoretical soundness.
- Tables 3 and 4: I assume this is for AlpacaEval 2, but maybe good to clarify in the table captions.

questions:
1. Are the base models both trained on the same UltraFeedback dataset from which the offline dataset is sampled? It seems likely yes, but it would be good to clarify this in Section 5.1, since it affects how to interpret the comparison with the base and offline DPO models.
2. What is the length bias like for LLM-as-a-Judge? If it is too large, an ablation that uses a length penalty with this method would seem appropriate.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709915005

thread_type:
official_review

<thread_object>
creator: reviewer_6mfr

<content>
summary:
This paper introduces a self-alignment approach, DICE, which uses the implicit reward model from DPO in a bootstrapping manner to further align LLMs, achieving over 8% improvement in alignment on AlpacaEval 2 without external feedback.

strengths:
The writing is clear and easy to follow, and the experiments provide support for the claim that repeated use of the implicit reward model could enhance performance.

weaknesses:
The results offer a useful insight, though the approach itself may lack significant novelty, and the improvements seem to be marginal. Additionally, length-regularized reward shaping is already a widely adopted technique. While the paper seeks to improve model performance through Direct Preference Optimization (DPO) by leveraging the implicit reward model, it would benefit from a theoretical explanation of why repeated use of the implicit reward model could enhance performance. Moreover, it might be worth considering whether repeatedly using the implicit reward model offers more advantages than utilizing an additional reward model (RM) to label subsequent iterations.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1731451628982

thread_type:
official_comment

<thread_object>
creator: reviewer_3rhj

<content>
title:
Further comment from reviewer 3RHj

comment:
Reviewer 3RHj here – since my score is an outlier, I want to add a brief comment in support of the paper. While I agree with reviewer A3Uc that a comparison with the method of He et al. [1] would strengthen the paper, I think that the method presented remains a helpful contribution even without this comparison. This is because it does not involve any secondary models such as reward models, making it easier for practitioners to use. In my opinion, the experiments presented provide enough evidence for practitioners who are looking for simple extensions of DPO to be interested in trying this approach (modulo the main weakness mentioned in my review).

[1] Semi-Supervised Reward Modeling via Iterative Self-Training, https://arxiv.org/abs/2409.06903
</content>
</thread_object>

</thread>
<thread>
creation date:
1732131036418

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper Revision

comment:
We thank all reviewers for their constructive feedback, and we have responded to each reviewer individually. We have also uploaded a **Paper Revision** including additional results and clarifications:

$\\textrm{\\color{blue}Section 5.4}$ (Page 9): Comparison of implicit reward model (ours) with scalar reward model (Reviewer **6mfr,A3Uc**).

$\\textrm{\\color{blue}Appendix B}$ (Page 14-15): Results of AlpacaEval 2 across two base models over three iterations (Reviewer **3Zep**).

$\\textrm{\\color{blue}Table. 7}$ (Page 16): Comparison with length-regularized DPO (Reviewer **3Zep**).

$\\textrm{\\color{blue}Appendix A}$ (Page 14): We have reorganized the paper by moving Section 4.1 to Appendix A. (Reviewer **A3Uc,3Zep**).

$\\textrm{\\color{blue}Table 8}$ (Page 17): Results of PPO with implicit rewards on AlpacaEval 2 (Reviewer **3RHj**). 

$\\textrm{\\color{blue}Table 9 \\& 10}$ (Page 18): Hyperparameter tuning for $\\gamma$ (Reviewer **3RHj**).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper provides a new approach for self-improvement in LLMs using implicit DPO rewards. The idea is to first perform DPO on a preference data, and then generate online samples, and use the learned implicit rewards to generate extra preference data and keep training. This idea makes sense and to my knowledge is novel.

Strengths:
1. Novel idea for self-improvement that avoids training a separate reward model or relying on it.
2. Results show that it does better than using LLM judge for selecting preference
3. Surprisingly, DICE does better than training a reward model on the preference data (IntlRM) and an external reward model. 

Weakness:
1. Unclear how it would fair against a big space of preference learning literature, especially literature on reward learning. Authors do present useful baselines, particularly, using LLM judge and training a reward model.

Overall, I think the idea makes sense and the results are positive. I especially like that DICE does better than using LLM judge and doesn't require training in another reward model which is important from a practical point of view. Further, it is possible that implicit rewards are better for in-distribution tasks over general-purpose rewards. For this reason, I am recommending acceptance. 

That said, I have a concern that as there are a lot of self-improvement learning methods using preferences and we don't quite have a grip on the best strategies given the early stages of this field. However, I think it is hard for a single paper outside the frontier labs to do an exhaustive evaluation of all proposed methods on a variety of tasks.