Bootstrapping Language Models with DPO Implicit Rewards
Authors: Changyu Chen
Zichen Liu
Chao Du
Tianyu Pang
Qian Liu
Arunesh Sinha
Pradeep Varakantham
Min Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Alignment
Direct Preference Optimization
Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: An approach uses implicit rewards from DPO to bootstrap and further align LLMs.

Abstract: Human alignment in large language models (LLMs) is an active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO, after training, provides an implicit reward model. In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM. Our approach is to use the rewards from a current LLM to construct a preference dataset, which is then used in subsequent DPO rounds. We incorporate two refinements to further improve our approach: 1) length-regularized reward shaping to make the preference dataset length-unbiased; 2) experience replay to enhance the quality of the preference dataset. Our approach, named self-alignment with DPO ImpliCit rEwards (DICE), shows great improvements in alignment. It achieves an increase of more than 8$\\%$ in lengthcontrolled win rate on AlpacaEval 2 for all the different base models that we tried, without relying on external feedback. Our code is available at https://github.com/sail-sg/dice.

Published as a conference paper at ICLR 2025
BOOTSTRAPPING LANGUAGE MODELS WITH DPO IMPLICIT REWARDS
Changyu Chen" Zichen Liu Chao Du Tianyu Pang", Qian Liu?, Arunesh Sinhal4, Pradeep Varakantham Min Lin 'Singapore Management University "Sea Al Lab, Singapore
Nattiona University of Singapore *Rutgers University
{chency, liuze, duchao, tianyupang, 1 liugian, 1 linmin J com; arunesh. sinha@rutgeen edu; pradeepvêsmu edu. sg
ABSTRACT
Human alignment in large language models (LLMs) is active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feed- back (RLHF) by bypassing the reward learning stage in RLHE. DPO, after train- ing, provides an implicit reward model. In this work, we make a novel obser- vation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM. Our approach is to use the rewards from a current LLM to construct preference dataset, which is then used in subsequent DPO rounds. We incorporate two refinements to further improve our approach: 1) length-regularize reward shaping to make the preference dataset length-unbiased: 2) experience replay to enhance the quality of the preference dataset. Our ap- proach, named self-alignment with DPO ImpliCit rwwards (DICE), shows great improvements in alignment. It achieves an increase of more than 8% in length- controlled win rate AlpacaEval 2 for all the different base models that we tried, without relying on external feedback. Our code is available at https: //gi com/ sa 1-sgl/s/ao /ecce..
INTRODUCTION
Direct preference optimization (DPO) (Rafailov et al., 2024b) presents a compelling alternative to reinforcement learning from human feedback (RLHF) (Stiennon et al., 2020) in large language models (LLMMs). By circumventing the complexity of learning a reward model from given human preferences, DPO is simpler to implement and train compared to the RLHF approaches. Importantly, DPO, once trained, implicitly specifies reward model. Mathematically, the reward for response y the prompt can be expressed in terms of the optimal policy x² and the reference policy Teet
π (ylx) [))((y|x)
r* *xx y) Blog
Blog (xx).
for parameter 3 and normalizing constant Z. Further, an implicit reward r(x. 3-[log x -y//x)- log [xx]] is defined in DPO where the normalizing constant term can ignored as will be canceled out in the DPO objective, which only involves the difference of the rewards for the same prompt. In this work, we explore whether the above readily available implicit reward model after DPO training provides an opportunity to further improve the language model.
This paper answers the research question in the affirmative, by using the above implicit rewards in a bootstrapping fashion to further improve the LLM alignment with human preferences. Specifically, our approach follows the iterative DPO framework (Tran et al., 2023), where implicit rewards serve as the preference signals, as illustrated in Figure 1. We start with a model that has been through one round of DPO using human preference data, referred to as a DPO-tuned model. We then use the im- plicit rewards induced by DPO itself rank outputs from the current LLM, thereby, yielding new
Equaa contribation. The project was done during Changyu Chen's 's internship at Sea Al Lab. "Correspondence Chao Du, Arunesh Sinha, and Pradeep Varakantham.

---

Published as a conference paper at ICLR 2025
AlpacaEval LC Win Rate (%)
Preference Signal
Sampled prompt X,
(x, Yun NI
[yitie(A) 11
rPR
rs
DPO Training
Eehhy
+2.67
PPI
PLR
(1)))
+8.02
8(t-1) g(t)
Prompting External LLM
External Scalar RM
18.20
TPE T o g(x
rs = (x(,,))
+3.60
+9.35
Prompting LLM Itself
LR Implicit RM
FLR Blog We (y x)
TPI = 1) g(x y)
lyl
1) (y x)
ref
Bosc
Procapt
Implicit
Figure 1: (Left) Iterative DPO (Tran et al., 2023) with various preference signals: under the iterative DPO framework, the policy model is iteratively trained on a newly generated preference dataset. This dataset can be constructed using various preference signals. A common source is scalar reward model (RM) (Ouyang et al., 2022), denoted as T Alternatively, the dataset can be created by prompting a LLM judge the responses. This LLM can either be external model T or the policy model itself from the previous iteration T))t-1). In this context, x and are processed through LLM-as- as-juudg template g(:--) (Yuan al., 2024). We propose utilizing the length- regularized (LR) implicit rewards introduced in Section 3.1, where T) and Tzef ((t-1) represent the policy model and reference model from the previous DPO iteration, respectively. Our experiment excludes approaches requiring external models, such as r and as they are beyond this work's s scope. (Right) Our method, which leverages implicit rewards, further improves DPO-tuned models by a large margin, resulting in superior performance compared to the prompting counterpart.
preference dataset cheaply. We run DPO again with this newly generated preference dataset to obtain an updated LLM and then repeat the process. However, the above approach still needs further finement to address practical issues. One is the known issue of length exploitation (Park et al., 2024) where LLMs generate long responses when the same content could be provided more succinetly. Another issue is that the implicit reward model is an approximate proxy for human preferences, hence relying on strongly can result in corruption of the initial knowledge inbuilt into the LLM.
Inspired by (Park et al., 2024), we address the length exploitation issue by length-regularized re- ward shaping, which discourages long responses from being preferred. Notably different from Park et al. (2024), who incorporate response length into the loss function, we directly search for a length- unbiased dataset, avoiding the expensive hyper- saper-parameter tuning. To fix the overreliance on implicit rewards, we leverage insights from continual learning (Rolnick et al., 2019) and replay high qual- ity human preference data that was used in the first round of DPO (the round before bootstrapping began). Our method, named self-alignment with DPO ImpliCit rEwards (DICE), significantly im- proves LLM alignment quality with different base models. On AlpacaEval 2, we achieve 8.02% length-controlled (LC) win rate improvement with the Zephyr-based model and 9.35% improve- ment with the Llama3-based model.
To summarize, main contributions follows:
We propose to utilize the implicit reward model readily available in a DPO-tuned LLM. The implicit reward model enables to evaluate the responses generated by the current policy and construct preference dataset for future rounds of DPO without any external feedback;
We propose to apply two techniques together with our above proposed approach, length- regularized reward shaping and experience replay;
Empirical results show that our approach DICE enables significant (more than 8% LC win rate increase AlpacaEval 2) improvement in alignment with different base models; thus, believe that the core idea of using DPO Implicit Reward in DICE is a general purpose approach that can improve alignment for any single DPO-tuned base model.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
We provide brief review of the standard RLHF (Stiennon et al., 2020; Ouyang et al., 2022) and DPO algorithms (Rafailov et al., 2024b). Through the review, we demonstrate the implicit reward model induced by DPO, which will be used our work.
In preference tuning, the preference data typically takes the form of pairwise preferences. Each prompt paired with two possible responses, Y1 and Y2- The human annotator (Christiano al., 2017) Al annotator (Lee et al., 2023) provides the preference feedback o'y1 y2(x) E {0, 1}. indicating whether Y1 is preferred over Y2. The preferred response denoted as Yw while the other is denoted as yı. A common assumption is that the ground-truth human preferences follow the Bradley-Tere model (Bradley Terry. 1952). Based on this assumption, we can train a parameter- ized reward model rg rx( y) using maximum likelihood:
L (re. D) = - (x.. [log a (ra (x,Yw) To (x,)
(1)
where a is the logistic function.
2.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
The standard RLHF algorithm treats the LLM as policy and optimizes the policy using the reward model rg. The optimization objective is represented by the following equation:
max Bxw.D.yn.aaaa...... Dyw. y)] DKk. [To (e)|/ [x)] 7
where [x)(( denotes a reference distribution, and 33 is hyper-parameter. This objective balances the maximization of the reward (x. y) and divergence from the fixed reference distribution. The divergence term, given by the KL divergence (i.c., DKL (y xx) (y)((y/))) acts as regularizer to prevent the policy Ty from drifting too far away from the initial distribution ( (x). This objective is then optimized using a general-punte purpose RL algorithm, such as PPO (Schulman et al., 2017).
2.2 DIRECT PREFERENCE OPTIMIZATION
DPO (Rafailov al., 2024b) starts with the same objective Eq. (2) but derives an analytical closed-form relation between the reward and the resulting optimal policy. This relation can be used reparameterize the ground truth reward in terms of the corresponding optimal policy. This repa- rameterized formulation can be substituted back into the reward optimization objective in Eq. (I), enabling direct training of the optimal model the feedback data using the following objective:
LDPO (Toi Teer) -xx.. log (Blog (ywlx) B log Tee (y x))
Wref (yr(x) )
Te (yx(x)
(3)
In this context, the parameter (3 is the same as in Eq. (2), balancing the expected reward and diver- gence from the initial model. The DPO objective is particularly advantageous as it facilitates the recovery of the optimal model through standard classification loss, without the need for on-policy sampling or extensive hyper-parameter tuning. Observe that Eq. (3) resembles the reward modeling objective in Eq. (1) under the parameterization
r(x, = - Blgg [y(x)x
(4)
Tret Tror(ylo [x)
This reward function is commonly referred to as an "implicit reward" (Rafailov et al., 2024a; Zhong et al., 2024). Theorem I in Rafailov et al. (2024b) demonstrates that this parameterization of reward model is indeed valid without loss of generality. If we substitute this form of ro(x, y) into the RL objective in Eq. (2), Tg. Consequently, once DPO optimization is completed, we obtain an "implicit reward model" as defined by Eq. (4).
derive the optimal solution in closed form, which is
3

---

Published as a conference paper at ICLR 2025
3 BOOTSTRAPPING WITH DPO IMPLICIT REWARDS
DPO is an attractive alternative to RLHF as it largely simplifies the implementation and training process of language model alignment. However, recent evidences (Guo et al., 2024; Tran et al., 2023) have shown that continuing DPO training on fixed offline dataset results in inferior policy, while the policy can be further improved if one can collect new responses generated by the updated policy and provide preference labels to perform another round of DPO training. This can be understood as being closer to the on-policy sampling, which is generally preferred in preference fine-tuning (Tajwar al., 2024; Tang al., 2024).
In this work, we employ such iterative DPO preference tuning framework, where we start from a (0) base language model (a hase policy) woto; that is DPO-tuned from initial reference model Ted commonly an SFT model. In each round f E (1,2....), ). we sample K on-policy responses from the latest policy Tge (- | x) given prompt x. We then label the response with the highest and the lowest implicit rewards as winning and losing responses respectively, thus constructing a new preference dataset De- We further fine-tune the policy with DPO's s objective (Eq. (3)) obtain the
updated policy *gte with reference model -(0) = Tet This process is repeated to iteratively improve the language model. Notably, by fine-tuning DPO-tuned LM on the its own constructed dataset, we essentially align it without relying on any external preference feedback (e.g.. RLHF or RLAIF), hence in a bootstrapping fashion. We thus refer to our method as iterative self-alignment (bootstrapping) with DPO ImpliCit rEwards (DICE).
We next introduce two ingredients that are proven critical in the self-alignment process. In Sec- tion 3.1. we present Sength-Regulann wards with a length-regularized reward shaping. to judge the on-policy sampled responses for con- structing the preference dataset. Furthermore, to mitigate the potential catastrophic forgetting in the continual fine-tuning, we propose experience replay (Section 3.2) that mixes the generated data with the offline data for better performance.
Implicit Rewards, which augment the vanilla implicit re-
3.1 LENGTII-REGUE REGULARIZED IMPLICIT REWARDS
is a known issue in the literature that preference tuning may introduce length bias (or length exploitation) (Park et al., 2024), , which is likely caused by the fact that the preference labels collected from human annotators favor more verbose responses. This problem is further compounded by the iterative self-alignment scheme such as the one in Yuan et al. (2024), because the generated responses that are long and preferred will be reinforced in the next round of DPO, leading the language model to generate increasingly longer responses.
Inevitably, the vanilla DPO implicit rewards as in Eq. (4) would also exhibit length bias when gen- erating preference dataset. In Figure we show the distribution of the difference in string length of the winning and losing responses. We can see from the top figure that vanilla implicit rewards yield skewed distribution (in green). with an average length difference 1031. In stark contrast, the length difference of a high quality preference dataset is almost normally distributed (as in the bottom figure). This observation motivates us to debias the distribution induced by vanilla implicit rewards so to mitigate the length exploitation. We resort to reward shaping (Sutton & Barto, 2018) for this purpose. In particular, we introduce a length-regullaized (LR) reward shaping term in the implicit
Length Difference Distrihation
Figure 2: Distribution of the length difference be- tween the winning and losing examples (yyw ly:)). (Top) Distribution of the first round on on-policy generated dataset. With LR reward shaping de- fined by Eq. (5), the length bias is mitigated and the length difference becomes more evenly distributed.
vanilla
regulatized
dataset
The average length difference decreases from 1031 to -21 by setting a=0.023. (Bottom) Distribu- tion of the high quality UltraFeedback preference dataset (Cui et al., 2023) is almost unbiased.
6000
1000
-2000
2000
1000
6000
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Bootstrapping with DPO Implicit Rewards (DICE)
1: Input: prompt set X' extracted from Doffine. initial DPO-tuned policy Too,, initial reference
policy Tref = T)(-1), number of generated samples K, regularization weight 3, experience replay weight 7 E (0,1)
2: for = 1,2.... do
3: Generate responses by sampling X' and YI:K Tgtt-1)3
4: Create preference dataset D(aa)) ) by optimizing Eq. (6);
// ((a) is constructed by taking the best and worst y based on (x, Yı; yx;a), k [K]:
// Evaluate (x, Ys: a) given Tg(e-1) and A 23 as the target policy and reference policy
5: Create the mixed dataset via experience replay Dj {(xx, Yun Wi.iieen
I/ Sample (x, Yu. yi) PD,, PD. (1 ()pp())) + Damai Optimize my according to DPO loss, Eq. (3):
6:
(0 +
arg min E E(x.y...))D, log o (Blgg TT(t-1) (Fwlx)
To (ywlx)
3 Blog To (y: (x)
1) (y/(x)
7: end for
reward that penalizes the length of the response to obtain the shaped reward:
(x. = B log Ty x) x)
lyl
(5)
T
where controls the penalty strength and ly] is the string length of the response. Based on the shaped rewards, we can construct many versions of the preference dataset D(a), following the principle that the response with the higest FLR (x, a))a) is labeled as y and the one with the lowest reward is labeled as yt. To find the most suitable a such that D(a) is (approximately) unbiased, we optimize with the objective minimize the average absolute difference in response length:
a' = arg min Bly (ly lyr
(6)
We can employ any black-box optimizer to solve this non-differentiable objective function. In this work we find a simple random search suffices, and its solution effectively transforms the dataset into more evenly distributed one (shown in the orange curve in the top of Figure 2). We will output D(a") for the next round of DPO training. Details of the optimization can be found in Appendix C.
Importantly. despite the resemblance of Eq. (5) to Park et al. (2024) where they incorporate the token length as regularizer in the training objective, our reward shaping is conducted during the dataset construction stage, thereby avoiding the need for expensive hyper-parameter tuning.
3.2 EXPERIENCE REPLAY
Though DICE enables to learn from the response of the current policy, know that the implicit reward model from the DPO training is not perfect proxy for human preferences. Solely relying on the implicit reward model may result in forgetting the knowledge inbuilt in the initial policy the first DPO stage. Inspired by the technique of experience replay (Rolnick et al., 2019) in continual learning for preventing catastrophic forgetting and making a good balance between old and new data, we propose to use a mixture of the generated data and the offline preference dataset. While offline preference data considered be high-quality, it is off-policy samples; the generated data is closer to the on-policy samples, but the imperfect implicit reward model may introduce noise in labeling the preference. Combining the two can make for good balance. This idea also related to deep Q-learning with demonstrations (Hester et al., 2018), which demonstrates that incorporating supervised learning from an offline dataset can accelerate online reinforcement learning.
Our complete algorithm is summarized in Algorithm 1. During each iteration t, we generate K responses y2......... YK from the policy Tgge-1) for each prompt x (Line 3). The preference dataset D(a) is constructed based on the LR rewards FLR, where the response with the highest LR reward is labeled yu and the one with the lowest reward is labeled yı. In this process, Tate-1) serves as the target policy, and Tgre-2) serves as the reference policy. For t 1, denote the reference policy in the implicit reward model as Tgc-1), corresponding to the reference policy used in the initial DPÒ training. At Line 4, construct the debiased dataset D(a*) by minimizing the average absolute
5

---

Published as a conference paper at ICLR 2025
difference in response length, referring to Eq. (6). Subsequently, mixed dataset Dj is created by sampling Y proportion of data from Doffine and (1 - 7) proportion of data from D(a*) (Line 5). Finally, DPO training is conducted D, using T) both the initial policy and the reference policy, resulting in the updated policy Toco (Line 6).
EXPERIMENTS
This section empirically investigates DICE. Our findings highlight several key points: (1) DICE significantly improves the model performance on the widely used leaderboard AlpacaEval 2 (Li et al., 2023b), increasing length-controlled win rate by more than 8% for all the different base models; (2) our best model bootstrapped from 8B base model (1.1 -ama-3-88------ better performance than Gemini Pro 1.0 (Team et al., 2023) without any extra human annotations (or externl reward model) other than the preference dataset that is used in the initial DPO training of base model; (3) the two proposed techniques in Sections 3.1 and 3.2 are shown to be critical for DICE; (4) DICE demonstrates competitive performance relative scalar reward models trained exclusively on the same seed data.
achieves
4.1 EXPERIMENT SETUP
Base Models and Datasets. We adopt base models. Both models are trained following the pipeline of Zephyr (Tunstall et al., 2023) on the UltraFeedback (Cui et al., 2023) dataset. 22lama33-8B-20 Luma/MMta-Llama-33 on mistrala/MMiitrll--B-000
our and sephyr-7B-betad
is trained based meta- developed by Meng et al. (2024). zephyr- -B---t---tttae---- fine-tuned based We randomly sample subset of around 10k preference pairs from UltraFeedback as the offline dataset Doffline for our fine-tuning experiments. Our experiments aim show how much the language model can improve from DPO-tuned model and subset of the preference dataset that was used to conduct the initial DPO training.
Response Generation and Dataset Construction. At the start of each round, sample responses from the current policy, with temperature T = 0.9, p 1.0 for the Llama3 setting and T = 0.7, 0.9 for the Zephyr setting. We sample with different random seeds to get K 16 diverse responses for each prompt. We then reward each prompt-response pair by the implicit reward model (Eq. (4)) and incorporate length-regularizer reward shaping (Eq. (5)) to get the debiased dataset D(a") ) with the optimal regularization strength a*. The final dataset is a mixture of D(a*) and Doffline
Training Details. All experiments are conducted on 8 Nvidia A100 GPUs For DICE, we trained two rounds in total. In each round, we train the model for 300 steps on a preference dataset with 9.6k preference pairs (either a solely generated dataset, or mixture of fline preference dataset). The global training batch size is set to 32 and the learning rate is 5e-7 with a constant schedule and a warm-up of 50 steps. We hypertuned 3 E (0.01, 0.1} based on the model performance AlpacaEval 2 for each method and model separately. For approach, we addi- tionally hypertuned the experience replay ratio using cross-validation to ensure fair assessment. Further details hyperparameter tuning can be found in Appendix F.
generated dataset and the of-
Baselines. We evaluate the following baseline methods applicable to the setting of this paper:
Offline DPO: continue conducting DPO training with the offline preference dataset.
Offline DPO w/ new ref: similar to offline DPO but we assign the trained policy after each round as the new reference model for the next round. This corresponds to 1.
DPO with prompted rewards: similar to self-rewarding LM (Yuan et al., 2024), where they prompt the LLM itself as a preference judge construct new preference pairs and iteratively fine-tune the LLM with the DPO algorithm. In their case, the judge capability is learned by supervised fine-tuning on evaluation fine-tuning dataset. We exploit our base model to perform -Ldd directly as it can follow the judge instructions well. In the experiment, we call LLM- LLM-as-a-uudge. The LLM-as-a-Judge prompt template can be found in Appendix E.
'https://hugging co/ /prr/rrii necnnnnnnnnnn 10annnnnppppp amaa-maaa----mmmm Phttps: hhttpss//hugginggfaae co/ Huggi ingh aceH4/ //oe///eze //phyr-ppppppeepe 7btbea
ama- 3- Base- 88- 38-SFT-DPO

---

Published as a conference paper at ICLR 2025
Table 1: Results of AlpacaEval 2 and Arena-Hard across two base models. LC and WR denote length- and raw win rate in percentage (%) respectively.
zephyr-78-bete
Method
AlpacaEval? 2
Arena-Hae Hard
AlpacaEval 2
Arena-Hard
LC
WR
LC
LC
WR
LC
Base
12.69
10.71
10.17
18.20
15,50"
22.32
Offline DPO Iter
13.40
11.10
14.13
20.22
18.33
24.78
Offline DPO Iter 2
4.96
5.47
1.89
21.04
19.21
24.04
Offline DPO (w/ new ref) Iter
13.40
11.10
13.10
22.29
19.96
22.2
Offline DPO (w/ new ref) Iter 2
4.58
5.27
5.15
22.50
20.18
23.55
L.LM-as-a-Judge 1
15.36
17.81
13.96
20.30
21.31
26.46
ILM-as-a-Judge lter
14.14
17.89
13.47
21.80
22,42
27.99
DICE Iter
19.03
17.67
18.15
25.08
25.77 30.99
33.52
DICE Iter 2
20.71
20.16
18.45
27.55
39.13
Wc note that the results of
base are obtained from Meng et al. (2024).
Evaluation. We evaluate our method by AlpacaEvva 2 (Li et al., 2023b) and Arena-Hard (Li et al., 2024). Both LLM-bbsed automatic evaluation benchmarks and have been widely adopted by the community. AlpacaEval 2 employs AlpacaFarm (Dubois et al., 2023) as its prompts set composed of general human instructions. The model responses and the reference responses generated by GPT- 4-Turbo are fed into GPT-4-Turbo based annotator to be judged. We follow the standard approach and report both the win rate (WR) and the Length- Controlled win rate (L.C) (Dubois et al., 2024) over the reference responses. Arena-Hard is recently released benchmark, incorporating 500 well- defined technical problem-solving queries. Due to the expensive evaluation cost Arena-Hard, we follow the official guidance and use strong open-source model mistralai/MMistrllLLana -ntttttt 2407 (123B)' as the judge model.
4.2
MAIN RESULTS
DICE Effectively Improves DPO-tuned Model. With only a DPO-tuned model and a preference dataset that was used train this model, one can choose to further improve the current policy via Offline DPO. construct new preference dataset via LLMMass-a-uudge In Table 1, compare the performance of the model fine-tuned by DICE in two rounds with the base model and other baselines. It shows all methods can improve the LC win rate on AlpacaEval 2 over the base model while DICE leads to the most significant enhancement in both Zephyr and Llama3 settings, increasing by 8.02% and 9.35% respectively. We found that the LLM-as-a-Judge leads to good performance in the Zephyr setting while it has minor improve- ment in the Llama3 setting. We hypothesize this may be caused by the coarse rewards which not able to provide effective preference signals when responses are of high quality (the prompt template requires LLM judge to provide discrete score from 0 to 5, referring to Appendix E). We note that training on a fixed offline dataset for multi- ple rounds leads even worse performance than the base model, possibly due the increased data staleness and overfitting.
Table 2: AlpacaEval leaderboard results.
Model
LC
WR
GPT-4 0613
30.18 88
15.76
Mistral Medium
28.61
21.86
Claude 2
28.15
17.19
DICE-Llama3 8B Iter 2
27.55
30.99
Snorkel Mistral-PairRM--a Gemini Pro Mixtral 8x7B v0.1 Llama 3 8B Instruct
26.39
30.22
24.38
18.18
23.69
18.26
22.92
22.57
GPT-3.5 3.5 Turbo 0613
22.35
14.10
Tulu 2+DPO 70B
21.24
15.98
DICE-Zephyr GPT-3.5 Turbo 1106
20.71
20.16
19.30
9.18
Bastma3--BB--- Vieuna 33B v1.3
18.20
15.50
17.58
12.71
cephyr-7B-boc
12.69
10.71
Phttps://hugginggfaas st /Misssaaaaaai -large- Inst

---

Published as a conference paper at ICLR 2025
Table 3: AlpacaEval 2 results showing DPO Table 4: AlpacaEval results showing effects of rewards are compatible with other DAP algo- the Length Regularized Weight. rithms.
I.C.
WR
Avg. Len.
Offline
DICE
Method
0.000
13.32
15.37
2570
LC
WR
LC
WR
0.0
0.023 (a*)
18.88
19.31
2109
0.046 (2a")
14.40
9.08
876
DPO
13.40
11.10
19.03
17.67
IPO
14.83
16.16
18.51
19.49
0,000
13.92
19.08
2600
KTO
13.92
10.65
14.88
12.16
0.5
0.023 (a*)
19.03
17.67
1848
Hinge
13.51
12.45
15.92
15.57
0.046 (2a")
14.91
10.80
1185
Comparison with the models on the leaderboard. Compared with the models on the public leaderboard shown in Table 2, DICE-Llama3 8B performs better than the official instruct version of Llama3 by a non-trivial margin, 4.63%. Regarding the closed-source models, it achieves better performance than Gemini Pro with only 8B parameters and does require any in-house data or external reward model.
4.3 DICE IS COMPATIBLE WITH OTHER DIRECT ALIGNMENT FROM PREFERENCE ALGORITIMMS
Though DICE works best with DPO as it makes the iterative training possible (because the implicit reward model for the next round can be naturally derived using the updated policy), we would like check if the dataset generated by DICE can also improve the base model with other Direct Aligmment from Preference (DAP) algorithms. using
the Zephyr setting, we tune the policy model dataset (at the first round) and the offline dataset with KTO (Ethayarajh al., 2024), IPO (Azar et al., 2024), and Hinge loss proposed in Zhao et al. (2023). The training follows the protocol described in Section 4.1. The results in Table 3 show that all DAP algorithms benefit from the newly generated data by the current policy and DPO implicit reward model with LR reward shaping, demonstrating LC win rates higher than their offline counterparts. Notably, DICE shows the greatest improvement for DPO.
4.4 COMPARING DICE WITH INTERNAL REWARD MODEL
Our experiment setup assumes access to an offline preference dataset and a DPO-tuned language model. To further improve the language model, one may choose to train new scalar reward model and then conduct iterative DPO. As this scalar reward model is trained solely on the offline prefer- ence dataset without utilizing the external resources, call it the internal reward model (InURM) In this section, we compare the IntlRM with implicit reward model to evaluate their performance in self-alignment setting.
Base Models and Datasets.
We utilize mi mistxaal-7--sft--eta. as the base model. We adopt the full Feerbeee preference dataset (Cui et al., 2023), comprising 60k preference pairs.
Training Details. For the implicit reward, we use zephyr- Seephyr-B--bee 7B-beta along with its reference model mistrr ra trrb-sft-ber
The scalar reward model is trained using OpenRLHF (Hu et al., 2024) framework, adhering to the recommended training parameters This setup ensures that both the implicit reward model and IntRM use the same base model (mistral -apaal-bb--- -7b-sf sft -etttt and are trained on the same dataset (UlrraFeedback).
Evaluation. We evaluate the reward models based on the alignment rate of the model with the preference labels provided by GPT-4o. The alignment rate is defined as m n, where n is the total number of prompt-response pairs and is the number of labels matching GPT-4o. As our goal is to provide preference labels for the model-generated responses to enable the subsequent DPO training. we sample 500 (x, y2) tuples from the preference dataset during the first iteration of the DICE experiment under the Zephyr setting. Different reward models are then queried to provide the preference labels.
hhtpps: httpgg/inggaac..co.ooo Huggi /wttt//isttaaaa-aa/e ftebeaa Pittps://g) thub. wwtp:/ppew.ttt//oeennoeeooo /blob//lmm// http:///llmmm//////a/////// es/scriii wwriitttae itrriitt//ttt pts/ttain. 11ama
8

---

Published as a conference paper at ICLR 2025
Table 5: Alignment rate of the reward model with the preference labels provided by GPT-4o.
Method
IntRR
ERM-555K
DPO Implicit Reward
Alignment Rate
0.624
0.656
0.698
In the experiment, we include an external reward model, ERM-555K", trained by OpenRLHF main- tainers on 555k preference data, as a stronger baseline. In Table 5, we observe that the DPO implicit reward achieves higher accuracy than the IntRM trained an equivalent amount of preference data. Furthermore, it surpasses the performance of the ERM-555k trained on substantially more data. This implies that the implicit reward model offers advantages when evaluating its own gen- erated data, though the scalar reward models in general excel on a wider range of tasks when the preference data is abundant, as demonstrated by IM preference data. Based on these findings, we argue that the implicit reward is competitive option in self-alignment settings.
which was trained
4.5 ABLATION STUDY
In this section, we investigate the effects of LR reward shaping and experience replay.
Effeets of LR reward shaping. LR reward shaping (Eq. (5)) penalizes responses for being too verbose, and guides the construction of a debiased dataset with the optimal penalty strength at found by optimizing Eq. (6). To validate the effectiveness the propose LR reward shaping as well as the a-seerching procedure, we experiments in the Zephyr setting with different mixture ratios (y = 0 and 7 = 0.5) and ablate three design choices: (1) no LR reward shaping (a = 0): (2) LR reward shaping with penalty strength found by Eq. (6), i.e., a* 0.023; (3) LR reward shaping with slightly larger penalty strength (o = 2xx") ). Results are presented in Table 4. For all values of the y, we observe that a 0.0 does lead to scrious length exploitation. So, even if the policy can get a high win rate, it will suffer in the LC win rate due to the length exploitation issue (responses with longer average length get lower LC win rate), e-g., the low LC win rate with ? 0.5, a 0.0. In contrast, a larger seemingly mitigates the length exploitation issue even better, but it may adversely affect the response quality. Our proposed approach of finding a* using the objective of minimizing the absolute difference of response length does provide the best performance.
Effects of experience replay. The experience replay results in a new mixed dataset in which fraction of the data is from the offline dataset and (i - 7) frac- tion of the data is from the generated dataset. E.g., we use data only from the generated dataset if y = 0.0. We experiments in the Zephyr setting with Y E (0.0, 0.25, 0.5, 0.75, 1.0} and conduct DICE in total of two rounds. From the results shown in Figure 3, we find = 0.5 provides the best performance. The re- sults satisfy our expectations. With only offline prefer- ence data, the DPO optimizes the current policy with off- policy samples that are further away from its distribution. With only its own generated data, the model may keep re- inforcing its current "belief", potentially leading to catas- trophic forgetting. An intermediate value = 0.5 finds a good balance. Additional results in the Llama3 setting can be found in Appendix D.
Win Rate
20.71
20
18,88
87
15-
33
16.9
12.60
10
base
iter
ier
0.0
0.5
1.0
0.25
0.75
Figure 3: AlpacaEval 2 LC Win Rate
across different experience replay ratios (r) the Zephyr setting. The highest LCC Win Rate is reported via text.
5 LIMITATION AND FUTURE WORK
Limitations. One of the primary limitations of our work is the reliance the DPO training prior to bootstrapping. the implicit reward model not well-trained, it can lead to a collapse of the training pipeline. This challenge is not unique to our approach; the classic RLHF pipeline is also
htttps:/huugiiggee wttp:/aaaaccoooeiaaaataaaaa
wtw.cee..coopeeeee opennppe / /1laamm-.3. 3-8b- rm-m xture

---

Published as a conference paper at ICLR 2025
struggling when its reward model is not well-trained. Another limitation is the lack of continued improvement over many iterations. Similar to other research, such as the work by Yuan et al. (2024), which enhances the policy model without external reward model. we did not observe continuous improvement in our model beyond three iterations. This issue highlights an open question within this field regarding the iterative enhancement of policy models.
Future Work. Future research could explore the rewarding capabilities of models trained using other DPO variants, such as KTO and IPO. Investigating whether these variants can offer general rewards similar to those provided by DPO-tuned policy would be valuable. Another promising direction involves developing methods that enable continuous improvement of the policy model over iterations without degradation. Additionally, investigating a theoretical understanding of the policy learned through self- saff-bootstrappng could provide deeper insights into the mechanics of our approach and facilitate further advancements.
6 RELATED WORK
Self-Improvig Fine- Fine-Tuning. Many studies explore fine-tuning language models without a large amount of human annotation (Huang al., 2022; Li et al., 2023a; Sun et al., 2023; 2024; Yuan et al., 2024: Chen et al., 2024b; Zhang et al., 2025). Starting from an SFT model, Sun et al. (2023) collect the preference labels by prompting the SFT model itself to choose the preferred response based on principles, training a principle-driven reward model, and optimizing via PPO. Yuan et al. (2024) con- struct a preference dataset by their supervised fine-tuned model trained on instruction following data and evaluation fine-tuning data, followed by DPO training. Different from these approaches, which align models by leveraging pretrained knowledge with minimal seed data, our work focuses on further improving a DPO-tuned model through bootstrapping with its implicit rewards.
On-Policy Sampling in Preference Tuning. DPO and its variants are widely used for their sim- plicity, but their offline nature often limits policy learning (Guo et al., 2024). Guo et al. (2024) show that offline DPO quickly overfits the preference dataset, while it performs much better and is more stable if online feedback can be provided to their on on-policy samples'. Tajwar et al. (2024) further analyze the properties of different preference tuning methods, concluding that on-policy sampling enhances performance and efficiency. When pure on-policy sampling is infeasible, using data closer to on-policy samples, such as in iterative DPO with external rewards (Tran al., 2023; Dong et al., 2024: Xiong et al., 2024; Ding al., 2024: Pang et al., 2024; Muldrew et al., 2024) or internal wards (Yuan et al., 2024; Furuta et al., 2025). can still be beneficial. Similarly, our approach enables us train the policy model with the preference data which is closer on-policy samples than the offline dataset without any external reward models. We hypothesize this is one of the main gain sources of our approach.
DPO Implicit Rewards. Rafailov et al. (2024a) recently studied DPO from a token-level MDP per- spective, revealing that DPO-tuned models implicitly parameterize token-wise dense reward func- tions. Building on similar theoretical fundations, Zhong et al. (2024) pretrains a DPO model to serve as a standalone dense reward generator for PPO training. Unlike these works, we leverage the implicit reward model as an outcome reward model to bootstrap itself for self-alignment. Implicit reward models have also been used for selecting the pairwise data (Yang et al., 2024; Chen al., 2024a) to improve the annotation efficiency in preference tuning.
7 CONCLUSION
In this paper, we introduce DICF, a novel approach that leverages the implicit reward model from DPO to further align LLMs with human preferences. Our method stands out in the current landscape of LLM alignment research, as it uses the implicit reward model to iteratively refine the policy model. Empirical results show that our approach DICE enables significant (more than 8% LC win rate increase on AlpacaEval 2) improvement in alignment across different base models, without relying on any external feedback.
Onnppllicy samples: the data collected while following the current policy that is heing optimized.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
REFERENCES
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Yifang Chen, Shuohang Wang, Ziyi Yang, Hiteshi Sharma, Nikos Karampatziakis, Donghan Yu, Kevin Jamieson, Simon Shaolei Du, and Yelong Shen. Cost-effective proxy reward model con- struction with on-policy and active learning, arXiv preprint arXiv:2407.02211, 2024a.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning verts weak language models to strong language models. In Forty-first International Conference Machine Learning, 2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- fems, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Alec Koppel, Mengdi Wang, Am- rit Bedi, and Furong Huang. Sail: Self self-improving efficient online alignment of large language models. arXiv preprint arXiv:2406.15567 2024.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint 2:x.x::240007866 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.044475 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:240..01306 2024.
Hiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, and Izzeddin Gur. Geometric-averager preference optimization for soft preference labels. Ad- vances Neural Information Processing Systems, 71:57076-571144 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.047990 2024.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations, In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jian Hu, Xibin WU, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An easy-to-use, scalable and perfoperfor rihf framework. arXiv preprint arXiv:2405. 1/43, 2024.

---

Published as a conference paper at ICLR 2025
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprini arXiv:2309.0000 00266,, 2023.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and lon Stoica. From live data high-quality benchmarks: The arena-hard pipeline, 2024.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.06254 2023a.
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloingg models, 2023b.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Proceedings of the 41st International Conference on Machine Learning, pp. 36577-36590, 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-277447 2022.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404. 19733, 2024.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arX1v:2403.19159 2024.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q": Your language model is secretly a q-function. arXiv preprint arXiv:2404.12355 12358, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: alifmanignment with principle-follwinng reward models. arXiv preprint arXhv:2310.05910, 2023.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from seratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
12

---

Published as a conference paper at ICLR 2025
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14366 2024.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offine alignment algorithms. arXiv preprint arXiv: 2405.08448, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Hoang Tran, Chris Glaze, and Braden Hancock. Iterative dpo alignment. Technical report, Snorkel AI, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405. 00675, 2024.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.0045) 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang. Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024.
Sen Yang, Leyang Cui, Deng Cai, Xinting Huang, Shuming Shi, and Wai Lam. Not all preference pairs are created equal: A recipe for annotation-efficient iterative preference learning. arXiv preprint arXiv:2406.1733 17312, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint 24XI::2400..02200 2024.
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference opti- mization: Improving chain-of ohfingof- reasoning in llms. Advances in Neural Information Pro- cessing Systems, 37:333-356, 2025.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2023.
Han Zhong, Guhao Feng. Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv: 18922, 2024.
13

---

Published as a conference paper at ICLR 2025
A BENEFITS OF ON-POLICY SAMPLING
In this section, we provide theoretical analysis similar Xie et al. (2024) to show that learning with on-policy samples can be more effective than utilizing an offline dataset. We make a generalized notation for the sampling policy at the f-th round as x(0, and compare the on-policy sampling (x(1) = Tate 1)) with sampling from an offline dataset (z7(1 =xp²) ²). For prompt x, we denote its optimal response" as y* and suboptimal ones as = {y, }. We also denote its preferences sampled from r(1) as (y (f) )) By Eq. (3) and the definition of the logistic function, we can rewrite the DPO loss at round for the sample (x
(1) Yt (t) as:
B
(1) y
Tgtt)
x
C (T : = log
To(s)
Y1 x R
where R = x red (y al/TTee(y (yl x) is a constant. Observe that Eq. (7) can be minimized to zero
by just minimizing Tigro) (y) x) be zero, without minimizing the likelihood of other suboptimal responses Teen [y][x) for y * y!" After i rounds of optimization, we are interested in the probability of outputting the optimal response:
Toco (y" |x) =1
Σ Y; ES
Tote (y; x).
(8)
Eq. (8) allows us to reveal the deficiency of training on fixed offline dataset. If there exists a suboptimal response y E S that lies the high likelihood region of agte), say Togo (y x) 2 p for some p E [0, 1] that is close 1, and y is never sampled from x(() TH thus not optimized as y during all / rounds, we have:
Tyto (y" x) S Tgto (y
x p.
Since Ty is zero except points that appear in Dotfline it is highly likely to find such y not being sampled from n. (hence never optimized) and therefore Jace (y* x) can be very low with large p.
On the other hand, conducting on-policy sampling can alleviate the "never-sampled" issue and pro- mote convergence the optimal policy. This is because whenever Tg (y,"
x) is high, it is likely
to sample y. from (() TT and thus it can be optimized such that Toce) yy) y; x ss 0.
In this way, the subtrahend of Eq. (8) is decreased per round, hence we can gradually improve the language model policy towards the optimal policy.
B EXPERIMENT BEYOND TWO ITERATIONS
We conduct DICE and LLLM-as-a-judge baseline (the strongest baseline) for three iterations, evalu- ating them on AlpacaEval 2 benchmark (see Table 6).
From the table, we observe a drop in LC win rate the third iteration for both our approach and the L.MM-as-a-Judee baseline. This is known challenge in the literature; both self-rewarding language model (Yuan et al., 2024) and SPPO (Wu al., 2024) show successful improvement only up to three iterations, which corresponds to two iterations in our approach, our method begins after one initial iteration of DPO training. Additionally, we use a fixed prompt set in every iteration, whereas other works employ different prompt sets across iterations. We hypothesize that using different prompt sets across iterations could help mitigate the performance drop observed the third iteration.
T is the hehavior policy that collects the offline dataset, which can be mixture of existing L.Mss human experts.
Optimality is measured by the underlying BT reward model, instead of the implicit rewards. In general there can be multiple optimal responses, but here assume single optimal response for simplicity.
14

---

Published as a conference paper at ICLR 2025
Table 6: Results of AlpacaEval across base models over three iterations. LC and WR denote length-controlled and raw win rate in percentage (%) respectively.
zephyr-
Sephyr-/B-oeta /B-beta
1Lama-3-93--1
Method
AlpacaEval 2
AlpacaEval
LC
WR
LC
WR
Base
12.69
10.71
18.20
15.50"
LLM-as-a-Jude Iter
15.36
17.81
20,30
21.31
LLM-as-a-Judge Iter
14.14
17.89
21.80
22.42
LLM-as-a-Judge Iter 3
13.95
17.23
16.86
19.78
DICE Iter 1
19.03
17.67
25.08
25.77
DICE Iter 2
20.71
20.16
27.55
30.99
DICE Iter 3
14.02
15.30
20.61
26.18
note that results of 1atmam---8---000 hase are obtained
from Meng et al. (2024).
C OPTIMIZATION FOR LENGTH REGULARIZED REWARD SHAPING
We solve the objective in Eq. (6) using simple Bayesian optimization toolkit based on Gaussian process The objective landscape with respect to depicted in Figure 4, where we compare the proposed implicit reward model with the LLM-as-a-Judde reward model. With our length- reguarlized implicit rewards, the optimizer able find the optimal solution quickly that debiases the length difference of the winning and losing responses. For LLM-as-a-Judde rewards, the optimal solution obtained with a 0, hence we do not explicitly debias the dataset for all the experiments.
Average Absolute Length Difference
1250
1000
1000
800
750
600
500
400
250
0.00
00.05
0.10
0.00
0.05
0.10
Figure 4: The objective landscape for (Left) our implicit reward model and (Right) the LLM-as-a- Judge reward model.
D EXTENDED ABLATION STUDY
Effects of experience replay with Llama3 backbone model. In the Ilama3 setting, we also con- duct a coarse sweeping for the experience ratio y, and present the AlpacaEval LC win rate in Figure 5 for two self-alignment rounds. We observe similar trends to those in the Zephyr setting, which further justify the effectiveness the proposed experience replay: helps to keep a balance between the more on-policy generated data and the curated offline data. The best identified value of the mixture ratio is = 0.1.
httpps: //ac////aii//// paackiitoooo i ze. thub. 10/ st/bt e/modut /s/ggeeeatee///keeeeee.e.e gP_ minimi html
15

---

Published as a conference paper at ICLR 2025
LC Win Rate
21
22
18
bo00
iter
iter 2
0.25
0.75
0.5
Figure 5: AlpacaEval 2 LC Win Rate across different experience replay ratio (y) in the Llama3 setting.
Comparison between Reggtt-Regularized DPO and LR reward shaping. The length- regularized DPO proposed by Park al. (2024) incorporates penalty term into the DPO objective to explicitly penalize length exploitation. In contrast, our approach mitigates this issue by applying reward shaping during preference dataset construction. Since both methods target the same prob- lem, we compare their performance in the Zephyr setting with y = O. To clarify notation, we denote the length regularization coefficient in Park et al. (2024) X. rength-regulariied DPO uses the generated dataset D(0) without reward shaping, and À is hypertuned from (0.02, 0.05) as suggested length-regularized DPO paper. comparison, our method employs the dataset D(a*). All other settings remain consistent across both approaches.
As shown Table 7, we observe that our approach achieves better performance while effectively mitigating the length exploitation. In contrast, regularized DPO exhibits a quality-length trade-off, where larger regularization coefficients successfully control response length but lead decline in response quality.
Table 7: Results of AlpacaEval comparing techniques for mitigating length exploitation. LC and WR denote congtlecontoolled and raw win rate in percentage (%) respectively. Len denotes the average string length of the model response.
AlpacaEval 2
Method
A
LC
WR
Len
No mitigation
13.32
15.37
2570
0.02
16.30
19.16
2629
Regularized DPO
0.05
16.03
16.14
2030
Ours
18.88
19.31
2109
PPO with Implicit Reward. To assess whether implicit rewards can be effectively utilized in reward model-based methods such as PPO (Schulman al., 2017), conduct experiments using PPO with an implicit reward model. Specifically, the zepäyr-7B-bon
model and its SFT model are used construct the implicit reward model. Additionally, zephyr-75 7B---B----e---- abtt serves as the base model to be optimized. We adopt the implementation in OpenRLHF (Hu et al., 2024), adhering to the recommended training parameters. Evaluation results on AlpacaEval are presented in Table
The results indicate a slight improvement in length-controlled win rate for PPO, although the im- provement is less pronounced compared to DPO. While PPO's performance can potentially be en- hanced with additional hyperparameter tuning, such efforts would require substantial resources and time. We leave this for future work.
E PROMPT USED BY LLM-AS-A-JUDGE
We provide the prompt used by L.LM-as-a-judge in Figure 6.
16

---

Published as a conference paper at ICLR 2025
Table 8: Results of AlpacaEval 2 for Zephyr backbone. LC and WR denote length-controlled and raw win rate in percentage (%) respectively.
AlpacaEval 2
Method
LCC
WR
Base
12.69
10.71
PPO
13.32
8.55
DPO
19.03
17.67
LLM-as-a-Judge Prompt Template
Review the user's question and the corresponding response using the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:
Add point if the response is relevant and provides some information related to the user's inquiry, even it is incomplete or contains some irrelevant content.
Add another point if the response addresses a substantial portion of the user's 's ques- tion, but does not completely resolve the query or provide direct answer.
Award third point if the response answers the basic elements of the user's ques- tion in a useful way, regardless whether it seems to have been written by an Al Assistant or if it has elements typically found in blogs search results.
Grant fourth point if the response is clearly written from AI Assistant's 's per- spective, addressing the user's question directly and comprehensively, and is well- organized and helpful, even if there is slight room for improvement in clarity, con- ciseness or focus.
Bestow fifth point for a response that is impeccably tailored to the user's question by an AI Assistant, without extraneous information, reflecting expert knowledge, and demonstrating a high-quality, engaging, and insightful answer.
User: {instruction) Response: {response}
After examining the user's instruction and the response, provide brief step-by-step justifications and conclude with score between 0 to 5. You must follow the format below:
Evaluation: <evaluation
Score: <score>>
Figure 6: We follow the prompt template used by Yuan et al. (2024) to LLMs to judge model responses and construct paired dataset for further preference tuning.
F HYPERPARAMETER TUNING
In our experiments, the hyperparameters, including 7 and 8, were selected based on the model performance on AlpacaEval 2 (AE2). These parameters were tuned for each method and model separately (with y specific to our approach) but remained fixed across iterations. Specifically, / was tuned during the first iteration from the set (0.01, 0.1). while was tuned using the ranges (0.0, 0.25, 0.50, 0.75, 1.0} for the Zephyr backhone and {0.0, 00, 0.25, 0.50, 0.75, 1.0] for the Llama-3 backbone. The best hyperparameter setting identified in AlpacaEval 2 was directly applied in Arena-Hard (AH) for all methods and models without any further tuning.
Compared to baseline approaches, our approach has the additional hyperparameter 7 which may offer potential advantages during evaluation when hypertuning is performed on AE2. To address this concern, treat the two benchmarks (AE2 and AH) as proxies for validation and test sets to ensure fair comparisons. Specifically, we validate on AE2 while testing on AH, and then reverse the roles of the two benchmarks. Below show the results in Tables 9 and 10:
17

---

Published as a conference paper at ICLR 2025
Table 9: Model performance on AlpacaEval 2 (AE2) and Erena-Hard (AH) across different for the Zephyr Backbone. LC denotes the length-controlled win rate. CI denotes confidence interval.
7
LC. AE2
LC.AH: Score (95% CI)
0.00
18.88
17.6 (-1.4, 1.7)
0.25
17.87
16.1 (-1.2, 1.4)
0.50
20.71
18.4 (-1.3, 1.6)
0.75
16.90
14.9 (-1.6, 1.4)
1.00
13.40
13.0 (-1.3, 1.3)
Table 10: Model performance on AlpacaEval 2 (AE2) and Arena-Hard (AH) across different 7 for the Llama-3 Backbone. LC denotes the length win rate. CI denotes confidence interval.
LC. AE2
LC.AH: Score (95% CI)
0.00
26.93
39.1 (-2.3, 2.5)
0.10
27.55
(-2.0, 2.4)
0.25
24.78
33.3 (-2.1,
0.50
23.59
32.2 (-2.3, 2.2)
0.75
22.57
27.4 (-2.0, 1.8)
1.00
22.50
24.5 (-1.7, 1.8)
The two benchmarks and two backbone models constitute four validation-test instances. We observe that the selected from the validation set/benchmark consistently achieves the best performance the test set across all instances, demonstrating the optimal value of 7 is robust to the choice of the val- idation set/benchmark. Meanwhile, we do observe that the optimal y for different backbone models is varied, which is expected, as different backbones produce responses of varying quality. Higher- quality generated data reduces the need for replayed experience, aligning with our observations for the Llama-3 backbone.
18


Reviews:
[{"review_id": "MF8Rx9Tix9", "reviewer": "Reviewer_6mfr", "summary": "This paper introduces a self-alignment approach, DICE, which uses the implicit reward model from DPO in a bootstrapping manner to further align LLMs, achieving over 8% improvement in alignment on AlpacaEval 2 without external feedback.", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The writing is clear and easy to follow, and the experiments provide support for the claim that repeated use of the implicit reward model could enhance performance.", "weaknesses": "The results offer a useful insight, though the approach itself may lack significant novelty, and the improvements seem to be marginal. Additionally, length-regularized reward shaping is already a widely adopted technique. While the paper seeks to improve model performance through Direct Preference Optimization (DPO) by leveraging the implicit reward model, it would benefit from a theoretical explanation of why repeated use of the implicit reward model could enhance performance. Moreover, it might be worth considering whether repeatedly using the implicit reward model offers more advantages than utilizing an additional reward model (RM) to label subsequent iterations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tdwdvJ5NwG", "reviewer": "Reviewer_3RHj", "summary": "The paper introduces a new approach to iterative DPO. After each iteration, synthetic comparisons are generated by the current policy: given a prompt, 16 completions are sampled and then labeled using the implicit reward from the DPO formalism; a length penalty is applied to make the comparison dataset approximately length-unbiased; and the best and worst completions are used for the synthetic comparison. These synthetic comparisons are then used for the subsequent iteration. The method outperforms baselines on two different benchmarks across two different models.", "questions": "1. Are the base models both trained on the same UltraFeedback dataset from which the offline dataset is sampled? It seems likely yes, but it would be good to clarify this in Section 5.1, since it affects how to interpret the comparison with the base and offline DPO models.\n2. What is the length bias like for LLM-as-a-Judge? If it is too large, an ablation that uses a length penalty with this method would seem appropriate.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "The method is natural and straightforward to apply: using the current policy to generate both new on-policy samples and rewards provides and easy way to do iterative policy improvement with no additional models or data. I expect the method to be of great interest to practitioners who value this benefit, which is similar to the benefit of DPO compared to PPO for RLHF.\n\nThe experiments are thorough and promising enough that I expect others will want to try out the method in their own settings, although I discuss a major weakness below. The ablations are also well-chosen and interesting. In particular, the ablation for the choice of alpha* mostly satisfied me that there is a real benefit to the method beyond the length penalty, although see my question 2 on this point.\n\nI thought the hypothesis about on-policy data helping was a good one, but I would suggest an additional hypothesis for why the method helps, namely that using the implicit rewards is similar to training a reward model, so as with PPO there may be a benefit from discrimination being easier than generation.", "weaknesses": "A glaring methodological weakness is in the treatment of the hyperparameter gamma (1 minus the fraction of offline data used): if I am understanding correctly, the tuning was done using the final reported results. This is contrary to standard machine learning best practices, whereby hyperparameters are tuned using a validation set. This gives the method a significant advantage if the difference between different values of gamma has a significant noise component. The same applies to the hyperparameter beta, but the issue is not as important there because the same tuning was done for the baselines too (although I still think this should be fixed – in particular, the Base model does not get to tune beta). Regardless, some clarification should be given as to how the tuning for beta was done (is it the same across models/methods/iterations/benchmarks?). I do not believe the hyperparameter K was tuned, but it could be good to clarify how K = 16 was chosen.\n\nIt looks like the same value of gamma was used across benchmarks, which is good since this reduces the selection effect, but different values of gamma were used between different models (and moreover, this fact is not clearly disclosed, I attempted to infer this by comparing Table 1 with Appendix C). Since it would take a lot of work to re-run all the experiments with fairly-tuned hyperparameters, my suggested remedy is as follows: **report results for the same value of gamma (say gamma = 0.5) across all experiments from both models**. I think this would go a long way towards presenting a fairer picture of the algorithm's performance compared to other methods.\n\n**I have rated the paper and 8 in spite of this scientific mis-step in anticipation of some sort of correction or explanation.** Without the above suggested change or some other correction or explanation, I may lower my score, which would be a shame, because it is otherwise a great paper. Similarly, I may also raise my score to a 10 if I become convinced that this problem has been adequately addressed.\n\nAnother limitation is a lack of comparison to reward model-based methods such as PPO, but I think it is reasonable to leave this to future work. I think there are some interesting comparisons to be made here because the use of the implicit rewards is similar to training a reward model and using the rewards from that.\n\nThe presentation was generally pretty clear, although I found Algorithm 1 somewhat hard to follow (and it would be even harder for someone glancing at the paper), for a couple of reasons:\n- It seems worth stating that the comparison dataset is created by taking the best and worst samples according to the (length-penalized) reward.\n- I believe there is a typo where it says to use \"pi_ref^(t-1) as reference policy\", and it should be pi_ref^(t) (unless I am missing some subtlety that deserves further explanation). But since pi_ref^(t) is equal to pi_{theta^(t-1)}, I think it would be simpler and clearer to remove the notation pi_ref^(t) from the pseudocode entirely.\n\nMinor line-by-line comments:\n- Abstract: \"refinements that debias the length of the responses and enhance the quality of the preference dataset\" - this phrase is convoluted and unclear, suggest replacing with \"a length penalty to make the preference dataset length-unbiased\" or just \"a length penalty\" may suffice.\n- Line 046: typo \"by itself\" -> \"by DPO itself\"\n- Line 216: might be good to clarify where does pi_mu comes from in practice\n- Line 227: \"without minimizing the likelihood of other suboptimal responses\" - I didn't understand what point you were trying to make, since the loss doesn't depend on these other responses by definition?\n- Line 245: \"subtrahend\" - I had never seen this word before but it made sense after I Googled it :)\n- Line 291.5: \"normally distributed\" - It does not look normally distributed at all, there is a big spike at 0. It looks more like a Laplace distribution (although I wouldn't want to make this claim without statistical evidence, and in any case it doesn't seem important to the exposition).\n- Equation (9) - It would seem more natural to put the absolute value outside the expectation instead of inside, since this would directly optimize for the dataset being unbiased in the sense of the mean length of winning completions roughly matching the mean length of losing completions. That said, this probably gives a very similar value of alpha* and so this change would mostly be for improved theoretical soundness.\n- Tables 3 and 4: I assume this is for AlpacaEval 2, but maybe good to clarify in the table captions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lpxWv9CoEA", "reviewer": "Reviewer_A3Uc", "summary": "This paper proposes a self-supervised iterative finetuning procedure for LMs trained with DPO, which, after the first round of DPO (with human data), uses the Implicit Reward Model (as used by DPO) to label new self-supervised preference data, and then trains again. This process can be repeated. To obtain good performance, the authors propose 2 techniques that improve the nth stage of finetuning and show that these techniques are critical for thei good results: adjusting the Implicit Rewards by a length penalty, and using an experience replay to mix in the human labeled data. The authors find that this iterative procedure obtains better results than several baselines on AlpacaEval and ArenaHard.", "questions": "- If there is anything the authors can do to convince me that using the adjusted DPO implicit rewards has an advantage over training a separate reward model for iterative finetuning, this would be greatly appreciated (see Weakness #1 above for some ideas). (I am aware that using the DPO implicit reward is computationally more efficient, but I do not believe that advantage alone is sufficient, at least not without fully understanding the compute-return trade-off here).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Iterative finetuning on self-supervised data is an important direction, which has received a reasonable amount of attention, e.g., with Anthropic's Constitutional AI / RLAIF, and others. The DPO approach has also gained significant adoption, so improving DPO with iterative finetuning would be of interest to the community. \n\nThe proposed procedure is straightforward and the results are good. The paper as a whole is well written and presented. I believe I could reimplement the method/experiments based on the presented details. The implementation is novel, as is the use of implicit rewards for iterative finetuning as far as I am aware.", "weaknesses": "Main concern: \n- I remain unconvinced this is the best approach to iterative finetuning a DPO model, since the evaluation only tests downstream performance and other potential reward models that are known to be empirically stronger than implicit DPO rewards are ignored: If you look at RewardBench, the implicit rewards of DPO models are quite far down the list in terms of their strength as a reward model. Furthermore, reward models themselves are not very difficult to train, so the overhead as compared to using the implicit DPO reward is not that large. See also Lin et al. 2024, On the Limited Generalization Capability of the Implicit Reward Model... \nSo for these reasons, my prior that using a trained RM for iteratively improving a DPO model (e.g., as done by He et al. 2024, Semi-Supervised Reward Modeling via Iterative Self-Training...) is unchanged after reading this paper... i.e., I do not think the proposed approach is competitive with simple alternatives, which do not appear as baselines.  \nTo convince me here (or alternatively, to contribute a \"full\" empirical study that determines what is still an open question in my mind), I would want to see the iterative DPO w/ trained RM show up in Table 1, OR alternatively, I would want to see that the implicit reward model that is learned via this iterative process (and adjusted with the LC normalization) is competitive vs other RMs on RewardBench or similar. \nSide note: esp. given the length regularizer, avg log prob reward (see simpo) would be a good baseline to have\n\nConcern: \n- Limited Conceptual Novelty: While the use of implicit rewards for iterative finetuning is new, neither iterative finetuning, nor the implicit reward model are new concepts... and it is in some sense \"obvious\" that one can use any reward model for iterative finetuning, including the implicit reward model. Therefore, the main novelty in my view is limited to (a) empirical results (which are good, but see point above), and (b) the length-normalization penalty (although others have also investigated different variations of this) and experience replay (not sure I would call this new, but the empirical demonstration of its usefulness in this context is good). \n\nLesser concern:\n- I think Section 4.1 detracts from the paper, rather than adds to it. E.g., Equation (6) seems wrong to me (unless you mean S to be ALL possible completions besides the \"optimal\" one, but this is also wrong, because \"optimal\" is ill defined here (the optimal response out of all possible completions under the implicit reward will likely be overoptimized / OOD / poor). I'm not sure this adds much intuition, and perhaps even supports the wrong intuitions (e.g., just because a high likelihood suboptimal response is not sampled, that doesn't mean it's probability doesn't change... and this criticism applies to finite on-policy data as well). \n\nMinor/Irrelevant to score:\n- I would call the length penalty reward specification/augmentation/engineering rather than reward shaping, which in my experience is reserved for rewards that are meant to improve the learning process but maintain the same desired policy (see Ng et al. 1999, Policy Invariance...) \n- Tortured acronym that is already used elsewhere. Consider something more unique that actually maps to the method name, e.g., Iterative Finetuning with Implicit Reward, or if you insist on a \"word\", Finetuning with Implicit REwards gives you FIRE, so you can do FFIRE (Finetuning Further) or IFIRE (Iterative FIRE).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KD1HK1dZbM", "reviewer": "Reviewer_3Zep", "summary": "The authors propose to use the implicit reward model learned via DPO to create a preference dataset and further improve the policy learned by an extra DPO step on the newly created preference dataset. They propose two additional techniques: 1) length regularization to prevent the known issue of response length increasing through DPO and 2) incorporating human preference data among the preference data collected with the final policy from the previous iteration and annotated by the implicit RM from the previous iteration.", "questions": "Is foregoing the partition function Z principled in this setting? This is an important design choice that is not discussed in the current state.\n\nIRPO (Pang et al, 2024) [1] should be mentioned (maybe when discussing iterative DPO)\n\nAuthors could draw a parallel between the replay buffer approach and the learning from demonstrations literature (e.g. DQfD, Hester et al, 2017 [2])\n\nWhat is the granularity of the search for $\\alpha^*$ ? What is the cost of conducting such experiments?\n\nSelf-rewarding DPO (Yuan et al, 2024) [3] seems very much connected: is it a baseline here? It should be.\n\n[1] - Iterative Reasoning Preference Optimization, arxiv preprint, Pang et al, 2024 \n\n[2] - Deep Q-learning from Demonstrations, AAAI, Hester et al, 2017\n\n[3] - Self-Rewarding Language Models, arxiv preprint, Yuan et al, 2024", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Writing is mostly good and clear (but see problems below).\n\nLength bias is clearly illustrated and mitigated by the length penalty (though not a technical contribution as discussed below).\n\nTable 4 presents some good results: one can have short responses scoring rather high (higher than the $\\alpha=0$ baseline).\n\nResults are good on AlpacaEval and Arena Hard.\n\nEDIT: I raised my score significantly after authors addressed major concerns of mine. I thank authors for the diligent work on making the submission better and fostering proper credit assignment.", "weaknesses": "A key weakness in the current state is that authors should connect their approach for length bias mitigation to existing approaches: it seems to me that this is the exact same approach as in Park et al, 2024 (which is cited but credit is not sufficiently assigned as of now). While I understand that the use of the length penalty is new, it currently reads as if authors came up with the penalty from Park et al, which is unfair to them. I make a non-exhaustive list of the places where credit / claims must be clarified:\n* “We address the length exploitation issue by length-regularized reward shaping, which discourages long responses from being preferred”\n    * I understand this sentence as the length regularization being an actual contribution, which is not the case. Please fix.\n* “We propose utilizing the length regularized (LR) implicit rewards introduced in Section 4.2”\n    * these are the same as in Park et al, so again this sentence is misleading\n* “We propose to apply two techniques together with our above proposed approach, length regularized reward shaping and experience replay”\n    * same: applying an existing technique (which was developed in a very close context) is not a technical contribution here\n* “Next, in Section 4.2, we introduce the proposed Length-Regularized Implicit Rewards, which augment the vanilla implicit rewards with a length-regularized reward shaping, to judge the on-policy sampled responses for constructing the preference dataset”\n    * same comment\n* “Importantly, despite the resemblance of Eq. (8) to Park et al. (2024) where they incorporate the token length as a regularizer in the training objective, our reward shaping is conducted during the dataset construction stage, thereby avoiding the need for expensive hyper-parameter tuning” -> this needs to be explicited before! and does not prevent from toning down the novelty claims on the length bias mitigation\n\nIn the same vein, why not using the regularized DPO from Park et al as a baseline?? If the length bias mitigation proposed works better, this would make for a good contribution of the paper, but then this should be backed by explicit comparisons.\n\nI also find that the authors did not get to the bottom of the lack of improvement past iteration 2 yet:\n* “we did not observe continuous improvement in our model beyond three iterations” -> why is this not presented in graphs? that is important data that is currently (afaict) not in the paper\n* Unclear why there is no improvement after K iterations (K > 2), is it because the policy becomes too deterministic? any root cause analysis would be valuable here \n\nMathematical notations and statements\n* I do not understand why using $\\hat{r}$ instead of $r^*$ (this is the optimal reward!) l. 34\n* “For a prompt x, we denote its optimal response as y^⋆”\n    * in general there can be multiple optimal responses (as there can be multiple optimal policies), please adapt the statement\n    * but here authors mean preferred response? this really needs clarification\n\nWriting\n* “In this way, the subtrahend of Eq. (6)”\n* “To validate the effectiveness of the propose LR reward”\n\nClarity\n* Section 4.1 as a whole is really confusing in the current state.\n* citation missing for RLHF (Christiano et al) l. 28\n* when mentioning the length exploitation issue, authors could quickly explain why the models exploit length (which is because longer responses are correlated with higher probability of being preferred by human or machine-based preferences)\n* no citation for Gemini Pro! also the version used should be stated (1.0 vs 1.5 vs 1.5 002)\n* “Offline DPO w/ new ref: similar to offline DPO but we assign the current policy as the new reference model, while we use a fixed reference model in Offline DPO. This corresponds to γ = 1” -> misleading as my understanding is that the reference is fixed (it is just the end policy of the first round of DPO) \n\nExperiments / ablations missing\n* What about using all the K samples with a Plackett-Luce model? Does this bring any benefit? \n* Replaying human preference data seems similar to KL regularization to the previous policy -> would be interesting to compare\n* Baseline missing: offline AI feedback annotations instead of implicit RM (+ online would be nice but harder/costlier to setup)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)