D2PO: Discriminator-Guided DPO with Response Evaluation Models
Authors: Prasann Singhal
Nathan Lambert
Scott Niekum
Tanya Goyal
Greg Durrett
Venue: COLM (2024)
Keywords: RLHF, Reward Models, Data Efficiency
TL;DR: We propose a data efficient technique based on updating discriminator models online to improve RLHF

Abstract: Varied approaches for aligning language models have been proposed, including supervised fine-tuning, RLHF, and direct optimization methods such as DPO. Although DPO has rapidly gained popularity due to its straightforward training process and competitive results, there is an open question of whether there remain practical advantages of using a discriminator, such as a reward model, to evaluate responses. We propose D2PO, discriminator-guided DPO, an approach for the online setting where preferences are being collected throughout learning. As we collect gold preferences, we use these not only to train our policy, but to train a discriminative response evaluation model to silver-label even more synthetic data for policy training. We explore this approach across a set of diverse tasks, including a realistic chat setting, and we find that our approach can lead to higher-quality outputs compared to DPO with the same data budget, and greater efficiency in terms of preference data requirements. Furthermore, we show that our silver labeling is most helpful when training the policy with DPO, outperforming traditional PPO, and benefits from maintaining a separate discriminator from the policy model.

Published as a conference paper at COLM 2024
D2PO: Discriminator-Guided DPO with Response Evaluation Models
Prasann Singhal Nathan Lambert
Scott Niekum"
Tanya Goyal", Greg Durrett
The University of Texas at Austin, Allen Institute for Artificial Intelligence University of Massachusetts Amherst, Princeton University
prasannsêcs wtw.as.exxaeeeeooo
Abstract
Varied approaches for aligning language models have been proposed, in- cluding supervised fine-tuning, RLHF, and direct optimization methods such as DPO. Although DPO has rapidly gained popularity due to its straightforward training process and competitive results, there an open question of whether there remain practical advantages of using a discrimi- nator, such as a reward model, to evaluate responses. We propose D2PO, discriminator-guie -ggide DPO, approach for the online setting where pref- erences are being collected throughout learning. As we collect gold prefer- ences, these not only to train our policy, but to train discriminative response evaluation model to silver-label even more synthetic data for policy training. We explore this approach across a set of diverse tasks, including realistic chat setting, and we find that our approach leads to higher-quality outputs compared to DPO with the same data budget, and greater efficiency in terms of preference data requirements. Furthermore, we show that our silver labeling is most helpful when training the policy with DPO, outperforming traditional PPO, and benefits from maintaining a separate discríminator from the policy model.
1 Introduction
Learning from human preferences is the prevailing method for large language model (LLM) alignment, including approaches like reinforcement learning from human feedback (RLHF) Ouyang et al. 2022 Bai et al. 2022), Direct Preference Optimization (DPO) (Rafailov et al. 2023), and several recent alternatives (Azar et al. 2023 Ethayarajh et al. 2024 Hong et al. 2024.. A key idea in this line of work, introduced by DPO, is that the reward objective can be expressed in terms of the optimal policy and reference policy, allowing to train model from preference data without learning a separate reward model or sampling from the policy during learning.
However, the theoretical guarantees of DPO may not apply in practice. Preferences are not necessarily obtained over set of outputs that are in-distribution for the final aligned model. For example, preferences may be labeled over initial model outputs of certain length, but the distribution of the policy model may shift during training to produce longer responses (Singhal et al. 2023). In this case, the preference data does not uniquely specify an optimal policy, so different algorithms may lead different solutions in practice. At the same time, recent approaches have explored collecting preferences from the shifting distribution the model throughout training (Touvron et al. 2023 Yuan et al., 2024b). It is not clear theoretically nor empirically which approach best given a limited budget for preference labeling in this online setting.
This paper investigates the role of an explicit discriminative model during the alignment process. Our central hypothesis is that when preference data is limited, a model discrimina- tively trained to evaluate responses (like reward model) can learn to assess them more

---

Published as a conference paper at COLM 2024
Policy model (rraa i))
Static Preference Data
Online Output Pairs
Discriminative response evaluation model R(xx,))
((2e) of i }}
3/2
Label wy RM
Train policy
Gold Label
f =
1-T
t=
1=T
(a) Standard DPO
(b) Online Preference Optimization
(c)
Diseriminator-guiler DPO (ours)
learn poliiy diecininnatieeyy Brom set atattic proferenccs
samplooniic uslpal puins, label with rowanl rewaanl leamed from saile prefarenas, trin discrinizativlly
sugl: caline osipi puin, use online gold label upolec model silve label online ouipui pains, briin discrinissaivvly
Figure 1: Comparison of standard DPO, online preference optimization methods (with reward model-labeled data), and our proposed D2PO method. The key addition in (c) is the online learning of the reward model on new preferences during policy optimization.
easily than a model can learn to produce them. This discriminative response evaluation model can then be used to silver-label samples from our policy give additional data for policy training.
We operationalize this approach in a method called D2PO (Figure D2PO alternates between two phases: first, collecting preference labels to train the discriminative response evaluation model (blue), and second, using that discriminator to label larger number of pairwise preference
outputs from the policy model (teal). We use the discriminator labeler and train the policy using the DPO objective, which we find works better than a conventional PPO setup (Schulman et al. 2017). Finally, by decoupling the discriminator from the policy, it can be learned over human-labeled preference data only, while the policy model can learn from larger amount of noisily-labeled on-policy data.
Our results study both a realistic chat benchmark (UltraFeedback (Cui et al. 2023)) as well as several simple text generation tasks where we explicitly define gold reward functions. We show several key findings. First, we show that the online preference collection setting indeed works better than having only static preferences collected from the initial policy model. We then show that, given the same budget of preference updates as baselines, D2PÓ on several settings achieves higher reward more quickly than an online version of DPO and than basic PPO. Finally, we study how the discrimínative reward evaluation model behaves over the course of training. Receiving new labeled data is crucial for it to be able to make accurate judgments about new sampled responses. Combined with controlled D2PO experiments testing different types of discriminators, including using the policy as its own discriminator, we establish that in the setting discuss, separate discriminator may still be a useful ingredient in LLM alignment.
Taken together, we make the following contributions. (1) We propose new approach, discriminator-guideed DPO, for the setting where preferences are being collected online. (2) We show, on diverse set of tasks, that maintaining discriminative response evaluation model and using it to silver label new sampled responses improves policy training. (3) We release code [1 and our diverse task settings to support the testing and development of future alignment algorithms.
2 Background / Setup
Let π(y xx) be an LM that places a probability distribution over a response y given an input x. Let = ((x, y/ }} be a dataset of human preference judgments, with outputs yi preferred to alternatives yf These preferences are assumed to be derived from scalar reward model R* (xx,y) reflecting human utility, which we do not have access to. The goal of LM alignment training is maximize this reward, optionally starting from a supervised fine-tuned policy (y x).
"oode available htttp:://gii https:://gittub.coomm httpp:///ghhuu..cm//rssaasa//aa com.bb/ccom/Paaaa////2200
2

---

Published as a conference paper at COLM 2024
In standard RLHF (PPO; (Schulman et el. 2017) ) first, Bradley-Terry reward model R(x,, y) is learned using preference data. To train the policy, online rollouts are then iteratively sampled from the policy π4 at each training iteration, and the loss optimization depends on the reward scores assigned to these outputs by the learned reward model, allowing the policy to learn from these rollouts under the assumption that the reward model produces accurate rewards. Direct Prreference Optimization (DPO; Rafailov et al. (2023)), another popular approach, trains the policy directly, offlin,, on just the initial preference data using discriminative loss, removing the separate discriminator. However, later work (Liu et al.
2023; Lee al., 2024), has explored incorporating online sampling with discriminative objectives. We broadly call this work online preference optimization (OPO) as unified template to capture the core design choices of this prior work for comparison to D2PO (defined more in Section 3)
While such methods are online with respect to the policy, they are not online with respect to preferences, which are usually collected from a fixed distribution such as Tift. Thus, as distribution π. changes over training, the signal of a reward model or a discriminator may degrade in quality, limiting improvement. Some work (Touvron et al. 2023 Guo et al. 2024 explores collecting preferences at an additional intermediate training step to address this issue. However, the intermediate steps, data, and objectives for this process are often chosen in an ad-hoc manner.
D2PO: Using a Discriminator in Policy Training
Our setting This work focuses on the setting where we collect additional gold preference judgments during policy learning. We define the total gold-labeled preference budget, typically from humans or from LLM-as-a-judge, as P. In most approaches, like DPO and OPO, all preferences Do y 1/; }} are collected offline (P |Dol) from single initial distribution. In iterative versions (OPO) of DPO, including our proposed method D2PO, gold preference data is collected at Tp different stages during training. The total preference data is therefore (Do, D....D.... -Dp) and P = ΣiO.T D,|.
We hypothesize that online up-
D2PO parameterization
dates to discriminators, with high number Tp of preference col- lection steps, will help solve the distribution shift issue and im-
Initial human-labeled preference data
Response evaluation model
Policy model
e
prove performance. However, as
preference data may be expensive, we want to do so with minimal The question then becomes how
D2PO iteration (every M/T, steps)
4. Human labeled data upditos the response eoolustion moxiel
Sample prompts
Sample cutputs
from policy
You first shauld... Don's t...
lebels update the policy
"Tell me how to..."
to use a small P to maximally im- prove our policy with respect to
"Whot the
answer
R* To address this, D2PO com- bines two ideas: (1) the compara- tive advantage in data efficiency of online data; (2) the ability of dis- criminators generalize quickly on new data and help silver-label data for a policy, reducing the need for costly gold preferences.
The onswer
A.
"Summarize the following passage...
The passage says...
Nothing to summarize...
siher lobeled.
Figure 2: D2PO trains an initial policy model and re- sponse evaluation model from gold preferences. then samples prompts, samples outputs of those prompts, and uses a mix of human labeling and silver labeling to produce policy training data. Only human-labeled data is used to update the response evaluation model.
Algorithm We detail our ap- proach, D2PO, in Algorithm and Figure 2 The algorithm, from an initialized policy and discriminator, runs for N iterations. In each iteration, the first step (L3-8) is to sample a batch of paired outputs from the policy n,, then get preference labels with our discriminator, and do DPO updates these new silver preferences. Note this part of the algorithm directly corresponds to several baselines we will compare against: if we omit the rest of the algorithm, this on its own is OPO with static RM (see below). If we
3

---

Published as a conference paper at COLM 2024
Algorithm 1 D2PO
Input: Policy model π,, Discriminator R, prompt set X, gold preference source R" .yyperparameters: policy rollout budget N, gold preference budget
1: candiidates + ()
2: for [0, N] do
// // Update policy using online rollouts and preferences derived from discriminator R.
4: sample E X // sample prompt
5: V1, x(x) // get 2 rollouts from policy using prompt
6:
Deandidates candidlates (x, 3/.)2)
7:
y" arg yrg (e(y,o)) Rxx,y) arg ye R(x,y) // get silver labels from R
8:
π π LDPO .y // DPO update with new preferences
9:
ift (mod N/T,) == then
10:
// subsample from Deandidates, gold preferences and update discriminator selected + arg top (x.))))) for E Deelected do
11:
(1) R(x, where |Dselected P/Tp
12:
13:
dg - R (d) // label preference with annotator with gold annotator R* - R Vl.BT(R, dg) // update reward model
14:
15:
(LD) // policy also updated with new labels
16:
end for
17:
Deandidates 0 // Reset output samples
18: end if
end for
then replace R with R* here, getting gold labels instead, this becomes the OPO with gold baseline. Lastly, if instead of using we fix the rollout distribution to be TT this gives us DPO.
Every N/T, steps, we collect gold annotation and update the discriminator (L11-17). In practice we can update the policy with these labels, though this is a sparse set of steps since the number of preferences P is much less than the number of policy training iterations N. This is the step depicted in Figure 2 The subsampling step of Deandiattes can use strategy such as confidence sampling (Lewis Gale, 1994), shown here, where we choose preference pairs with the lowest gap in discriminator reward, or random sampling, where we randomly select pairs. We generally find these to perform similarly. We report with confidence sampling in most settings, exceet our Contrastive Distillation setting where we find random sampling to work better.
Importantly, we then update our reward model with these preferences, allowing us to collect automatic preference labels on a larger set of rollouts for "free" (in terms of annotation cost), taking advantage of the discriminator's 's ability to generalize to other examples from a similar distribution. Note that our algorithm is flexible in the form of the discriminator, which is treated as black box. We use Bradley-Terry reward model by default, but in Section 6.2] we investigate if the DPO policy model itself or a separate copy of can be used as the discriminator instead.
The primary advantage of this approach lies in its efficiency: using purely gold rollouts can be effective, but also slow. By updating the reward model sparsely, these updates can generalize to new examples on the training distribution at each step, allowing more efficient collection and usage of preferences with equivalent performance.
3.1 Baselines
To test our hypotheses, we compare our approach to several baselines, including detailed breakdown of the data conditions for different approaches in Table (coreepponding to Fig- 3re3).. Note that we hold total gold preferences used, including those used for initialization (Appendix A),, constant across all approaches within each setting, though total number of preferences (gold silver labeled using the discriminator) may vary by setting.
4

---

Published as a conference paper at COLM 2024
GOLD PREPS (OFFLINE)
GOLD PREFS
SILVER POLICY PREES (ONLINE)
DISCRIM MODEL
POLICY LOSS
DISCRIM UPDATES (ONLINE)
(ONLINE)
GOLD
SILVER
PRO (static RM)
2.6k/5.6k/3366 /.66k/36k
0 0
64k D 64k
RM
PPO
NO
NO
DPO
2.6k/5.6k/336t /..5k///..kk/.kkk
DPO
NO
OPO (static RM)
RM
DRO
NO
No
OPO (gold)
1.6k
1k/4k/2k/2k
DPO
NO
NO
D2PO (ours)
1.6k
1k/4k/2k/2k
16k/64k/ 64K/32k
RM
DPO
YES
w/ DPO discrim
1.6k
1k/4k/2k/2k
16k/64k/ 64k/3Ik
DPO
DPO
YES
NO
w/ Self discrim
1.6k
1k/4k/2k/2k
16k/64k/ 63k/32k
policy (DPO)
DPO
YES
YES
PPO updale
1.6k
1k/4k/2k/2k
16k/64k/ 64kk//6
RM
PPO
YES
Table 1: Summary of our data and model conditions. Total number of preferences constant across approaches (except in experiments where we explicitly state otherwise), while vary- ing silver data, policy loss function, type of discriminator, and whether discriminator is updated or not. Slashes indicate preference sizes across our Unique Nouns, Word Collector, Contrastive Distillation, and Math settings, respectively.
We first compare against approaches using a static set of preferences labeled over outputs from the initial policy. We compare DPO, PPO (static RM) and OPO (static RM), the latter two of which train a reward model over the preference data and use it during training, either as a reward in PPO or a labeling function in OPO (see Figure 1.. Note that although we introduce the term OPO in this work, the implementation is similar to Liu et al. (2023). The primary difference between OPO and PPO is OPO's use of the DPO loss function over the online examples. We further compare these approaches in a higher data (50K) gold preference setting to examine behavior with many more preferences (Table(2).
We also compare against OPO (gold), where only online gold preference labels are used to update the policy, without any silver labeling. To make the number of policy updates more comparable with D2PO, we train each batch with the DPO objective for 4 epochs; we tuned this hyperparameter to optimize performance (Appendix A))
Finally, we compare against variants of our approach that use both online gold and silver labels, including a version where we use an independently-optinzzze for response evaluation model and version where we use the policy itself (self discriminator). We view this latter case as updating both the policy as well as the response evaluation model with silver labels. Finally, we com compare pare against version where use PPO updates given our reward model; this is simply PPO with a reward model being updated online.
DPO discriminator
Our methods have two notions of training progress, following Figure how many online gold preferences we have used from our budget Tp and how many silver updates have been applied to our policy Tn. Since human preferences are more costly, we're mainly interested in Tp, where we'll hold x-axes constant across approaches with respect to preferences used.
Experimental Setup
4.1 Tasks
To evaluate policy optimization methods, we evaluate on diverse set of tasks with distinct R* that we can derive relative preferences from. Example questions and outputs are in Appendix Dj. We use four synthetic tasks, designed to exhibit different properties, where we know the ground truth reward function R*, plus one realistic task with two R settings.
Word Collector: We compute the top 30 content words in the UltraFeedback (Cui et al. 2023) dataset. Given an UltraChat prompt (Ding et al. 2023), the goal is to generate 50 word output with as many of these words as possible, where the presence of each word gives +1 reward, for maximum of 30. This simulates having different sparse "positive" features that naturally may not occur together, and learning to incorporate multiple in single outputs. We subsample prompts initial preference dataset from UltraFeedback. We find that this task allows for realistic outputs that achieve high reward, while at the same time being challenging to optimize and showing variation among the training methods we compare.
5

---

Published as a conference paper at COLM 2024
Instead of maximizing word coverage, here we maximize the number of unique nouns, detected with spaCy (Honnibal et al. 2020) in a 50 token output. This is example of a much denser reward function. We get our initial preference dataset by subsampling from Ultra This reward function is the most easily optimized of those we consider and helps measure how quickly policy learning can be "steered" by the reward.
Unique Nouns:
Contrastive Distillation:
Given larger 0g (OPT-1.3b, Zhang et al. (2022a) and smaller ((yy);;0))
model 0g (OPT-125m), the R$ is the difference of log probabilities ()))())x;;s)) we receive more reward for sequences likely under the large model and unlikely under the small model. This task is represents likelihood-basse reward, similar to how s implicit reward is constructed. We get our initial preference dataset and prompts with OPT 125m samples on truncated (5-15 tokens) Wikipedia data (Foundation).
ground truth
Math Expressions: The input is mathematical expression randomly generated as tree of up to two layers deep. From here, there is a single gold sequence, which a chain-of- thought style sequence where the deepest, left-most sub-expression is solved one at time (e.g,, "((5 + 1)* 2) = (6 * +) = 12"). Then, given prediction such as "((5 1) * 2) = (6*2) 2) 13", we can iterate through each step, where R -s-1 multiplied with the total edit distance between solution and predictions each step. This task has a single solution. Prompts and preferences are randomly generated.
Finally, for a realistic setting where "true" ground truth reward is unknown, we use the popular UltraFeedback (Cui et al. 2023) cataset,, where we can use their GPT-4 based (gpt-4-0613) labeling scheme which returns a score between 1-5 as our R We just use subsampled UltraFeedback for our initial data. We further run an alternate setting using EurusRM (Yuan et al.) 2024a), the best reward model on RewardBench [Lambert et al. 2024) at the time the work was conducted, as an alternative gold reward.
UltraFeedback:
4.2 Implementation
We build top of the Huggingface TRL framework (von Werra et al. 2020 with hyperpa- rameters we find to work best based on reward convergence and downstream evaluation: 0.05, batch size 64; more details in Appendix A We use Llama-2-7B models as our base for realistic experiments, OPT-1.3b for the math setting, and OPT- for our synthetic ex- periments (Zhang al. 2022a), and use LoRA (rank= 16) (Hu et al. 2021 to enable training with smaller GPU tootprint, finding it to not affect initial experimental DPO /RM results. We use the Tülu SFT models for UltraFeedback (Ivison et al. 2023), an OPT-1.3b model fine-tuned on 500k example math expressions, and OPT-122m for other experiments. For each task, we evaluate the reward over the timesteps training. Periodically throughout learning, we our policy to compute gold reward over a held-out dev set of prompts.
Results
5.1 Comparing loss objectives (DPO vs PPO)
Before evaluating the potential of our full D2PO model, we first compare three differ- ent baseline approaches offline settings, i.e.,
NOUN
WC
CDIST
MATH
DPO
12
6.9
0.08
-17.1
PPO (static RM)
24.9
6.7
-0.20
-13.2
using only initial preference data or static dis-
OPO (static RM)
33.2
7.8
0.22
-3.5
criminators. We compare: (1) standard DPO, (2) OPO w/ static RM, which uses the DPO loss for paired policy updates, and (3) PPO w/ static RM, iee, standard PPO. We use large preference dataset (50K examples) for these experiments.
Table 2: Comparison of DPO, OPO and PPO
with Po 50K. OPO w/ static RM performs best amongst these baselines.
6

---

Published as a conference paper at COLM 2024
Unique Nouns
Word Collector
Contractive
Nath
Pofernce
Figure 3: Amount of gold preference data (X-axis; corresponds to progress through training, not counting initial 1.6k offline prefs) vs. resulting gold reward, averaged over 3 runs. We compare D2PO against OPO with gold data only, well as "basic" DPO and OPO with an RM trained on initial data (note that this is a smaller set than in Table Our method reaches higher reward in Word Collector and Contrastive Distillation, and maxes out faster at Unique Nouns.
Results: Table 2 reports the gold final reward of training runs for the three baseline ap- proaches. We find that OPO w/ static RM outperforms standard DPO in all settings, even though these use the same loss objective. This highlights the benefits of sampling online rollouts during policy training, show that PPO w/ static RM is generally outperformed by OPO, and even standard DPO for two out of four settings. We generally find the DPO objective to be more stable than PPO, and use the former for all our experiments. Finally, these results give lower reward than we will see in Section 5.2 from using online rewards, showing how effective online reward updates are.
opposed to using fixed preference set. Our results also
5.2 Comparing D2PO against baselines
Setup: To compare these for equivalent gold preferences, we report model performance quantified by the gold reward on 200 held-out datapoints (y-axis) against the number of gold preferences used for training (x-axis). That is, we report standard DPO and OPO w static RM performance on the same preference budget as the total budget over all time steps of our online D2PO and OPO w/ gold methods. Note, however, that the number of policy updates may differ between these approaches (see Table 1. We report aggregated results across 3 seed runs.
Results on synthetic tasks: We compare our proposed approach D2PO and baselines in 3 First, we importantly note that prior methods like DPO and OPO (stars), which are offline with respect to preferences, do much worse than the online preference approaches, even with much more data. Next, comparing the online approaches, we find that on three settings, D2PO leads to overall improvements in efficiency of data, either reaching higher final reward or higher intermediate rewards using lower preference budgets.
For instance, on Word Collector, D2PO reaches a reward score of 35 with preference budget of P = 100 where OPO w/ gold requires P = 300 to give the same performance. This suggests that labell-l using an iteratively improving RM is an effective strategy to offset the high annotation costs of online gold annotations. Note that this is not as clear on the Math setting, suggesting efficiency gains may depend on the setting. We explore this more in Section 6 Overall, our results suggest that both online preferences and on-distribution RM can contribute to increased performance and data efficiency.
Results on realistic task: Next, we report results on our realistic settings on UltraFeeddackk in Figure 4 On the GPT-4 annotation-based setting (Ultraleedback), we compare approaches assuming access to a small preference budget of 500 online preferences; gold annotation in this setting using GPT-4 costs $60 per run. We also report results when we use a budget of 3k preferences on the Eurus RM-based R* setting.
We include plot (Figure showing average of a sliding window of size 50 over gold reward from preference training given budget of 500 for UltraFeedback.. The Eurus reward plot is computed across checkpoints on a fixed eval set of 200 inputs. Overall, we find D2PO gets further in optimization than other approaches within the small preference budget. While this is smaller-scale, these results give us initial evidence suggesting that D2PO may have efficiency and performance benefits in practical chat settings.
7

---

Published as a conference paper at COLM 2024
UltraFeedback
Eurus RM
Gold Roward
Preferences Usad
Pha enerrcec Used
Figure 4: (Left) Gold reward over training on UltraFeedback, (Right) Eurus RM D2PO vs. OPO with budget of 500 preferences. The dashed line represents UltraFeedback reward for the highest reward point with OPO with the initial model. D2PO outperforms OPO on this setting.
Unique Roanz
Word Collector
Cantrustive Ditill
Nith
Figure 5: Reward model accuracy (y-axis) vs. training progress (x-axis) for our datasets using OPO (static, 50k RM). The discriminative capability of the reward model degrades substantially as training progresses, ending up near random chance.
APEVAL
For extra reference, we report length-controlled AlpacaEval 2.0 (Dubois et al. 2024) in Table 3 compared with gpt-4- turbo, as a winrate percentage out of 100%. We use the highest reward checkpoint for OPO and a D2PO check- point chosen based on discriminator reward (not shown, but we find this to be an effective stopping criterion due correlating well with gold reward). We do not find strong consistent pattern between our results and this evaluation, which we attribute to the fact that our reward models such as Eurus are not necessarily aligned with it. However, we note that performance at least does not degrade by optimiz- ing for these objectives, which occurs in cases of overopti- mization.
INITIAL
7.34
OPO (RM)
8.10
OPO (GPT-4)
6.37
D2PO (GPT-4)
8.26
OPO (FURUS)
10.43
D2PO(FURUS)
6.34
Table 3: Length-Controolle Al- pacaEval on models from dif- ferent approaches. We note that our gold R* objectives do not necessarily align with LC AlpacaEval; however, we find that performance does not de- grade by optimizing for them.
Analyzing Discrimination
6.1 Reward Model Accuracy under Distribution Shift
Our previous results establish that methods like D2PO
that iteratively updated discriminator outperform those with static discriminator (e.g. OPO w/ static RM). We hypothesize this difference in performance is because as the distribution of the policy shifts during training, the performance of the static discriminator on rollout pairs sampled from this new distribution degrades. This results in less reliable rewards and less reliable labeling of new preference data, degrading the improvement from additional policy training.
We first plot (see Figure 5) RM accuracy across policy training with the OPO with static RM baseline, computed as the fraction of times the preference label from the reward model R is same as that from the gold reward R* for pairs of rollouts sampled from the current policy find static reward accuracy generally decreases
(R* ties thown out). On most settings, as policy training proceeds, especially on Word Collector and Math. In fact, we see that the reward accuracy is 50% at some points during training, effectively random chance.
8

---

Published as a conference paper at COLM 2024
Unique Nouns
Word Collector
Centractive Dietil
Buan Bil
Eus
now parry
Figure 6: Reward model accuracy when using D2PO (red), accuracy of the initial static reward model (blue) and the DPO implicit reward model accuracy (green) of the policy at different points in training. D2PO successfully avoids reward degradation.
cono Di
238d prp
Frefrences Uoed
Paltenciaa Lired
Trd Prelernios LInE
Palenase Used
Figure 7: Comparing the performance of D2PO with different choices of discriminators against the amount of gold preference data used (x-axis). We observe that -ePO-seef-reward, where the the policy itself is used as a discriminator, performs worse than other approaches (we observe high instability and low rewards on the omitted math self-reward setting); separate discriminators (either DPO-trained or RMs) do a bit better.
Eurus RM
D2PO discriminator accuracy does not degrade: We then ask how this compares to when using D2PO. In Figure 6 we show accuracy of D2PO's RM from rollout sets across training as distribution shifts (red). We include the accura- cies of the DPO reward formulation of the policy at similar points in training (green), and the starting static RM (blue). Despite the distribution shift and much smaller amounts of data, D2PO allows for reward signal to stay non-random and even improve over the course of training, which ex- plains the effectiveness of the silver-labelling as we see in Figure 3 before, where the static RM generally seems to do worse. We further note that the accuracy of the static RM and the policy (using DPO implicit reward) often lags behind the discriminator's accuracy, giving some evidence that the discriminator is consis- tently able to provide information the policy may not capture on its own, even as reward of generations improves. On the realistic EurusRM setting (Figure we find D2PO improves over static RM earlier in training, though not to the same extent.
Prefrrnccss
Figure 8: Eurus RM D2PO RM accuracy vs static starting RM accuracy (y-axis) across roll- outs from training (x-axis).
6.2 Analyzing the Role of Discriminators
D2PO requires discriminator label preferences, but isn't dependent on a particular parameterization of discriminator, letting us treat it as a black box. We thus seek to answer the following question: how important is the choice of discriminator?
Setup: We test three choices of discriminators: (1) D2PO-RM: Our standard condition of D2PO with Bradley-Terry reward model (used in Section and 6.2), (2) D2PO-DPO: an independent DPO discriminator-only model, starting from the TSFT and trained only on the gold preference data. This sees the same training data as the reward model in (1) but uses different loss formulation. (3) D220Oseel--vaaluator
"self-rewarding""" .2 DPO discriminator where the D2PO's s policy is used both as the response evaluation model and the generator of rollouts. We ensure overall amounts of data are held constant across approaches.
2 Note that in self-rewarding settings like Yuan et al. (2244), the model is prompted to give rewards; here, use its likelihoods, which give usable resulfs even for models that are not strong zero-shot evaluators of their own outputs when using prompting.

---

Published as a conference paper at COLM 2024
We find (Figure that the DPO and RM discriminators do similarly overall. The self-rewarding baseline fails to converge on Nouns and Math, and does poorly on Word Collector, but is best on Contrastive Distillation, where note smaller ratios of gold preferences lead to training failure; we hypothesize that this is mainly because the noisy labels being fed back into the policy can contaminate the discriminative objective, hence the value of the separately-trained discriminator. We do a further study on DPO vs RMs in Appendix
Results:
Related Work
Preference Optimization: Aligning instruction-tuned language models with preference data has been reported to improve performance, both for proprietary (Ouyang et al. 2022 Touvron et al. 2023) and open source models (Ivison et al. 2023 Tunstall et al. 2023). Several preterence datasets have been released recently to facilitate further research in this space [Ethayarajh et al. 2022 Bai et al. 2022 Cui et al. 2023). From an algorithm perspective, recent work has proposed several simpperr variants variants of standard RLHF, such reward- free
methods that directly optimize with initially collected preference data (Rafailov et al. 2023 Azar al., 2023 Ethayarajh al. 2024) or iteratively updating the preference data using
gold annota tors Guo et al. 2024 Touvron et al. 2023). In contrast, recent work has also
explored alternate annotating strategies using learnt reward model (Zhao al., 2023 Liu
eet al.
2023 Calandriello et al. (2024) or the policy model itself for preference labelling Lee
et al. 2024: Yuan et al. 2024b). Ahmadian et al. (2024) studies the performance of more stable RL algorithms. However, these methods either ignore the distribution shift of policy models and it's impact on reward model performance, or rely purely on expensive gold annotations during training. Our work bridges this gap by introducing cost efficient method to ensure reward model does not degrade during policy training.
Reiemenct Learning: Our reward generalization hypothesis draws from the notion of reward extrapolation in reinforcement learning literature (Brown et al., 2019). Here, we expect our response evaluation model, trained online, to be able to effectively label silver data drawn from the same (evolving) distribution. Likewise, several results have supported the benefit of online over offline methods (Mediratta et al. 2023 Li et al. 2023).
Active Learning: The notion of actively collecting samples to improve models during optimization originates with active learning (Cohn et al. 1996 Zhang et al., 2022b). In the context of LLM training specifically, active approaches for prompt selection and off-policy reward training have started to get explored (Das al.) 20244 Dwaracherla al.) 2024). We do not claim innovation on the side of actively collecting preferences; our focus is on the utility of respond evaluation models rather than active selection strategies.
8 Conclusion
In this paper, we present D2PO, a method for learning from online preferences for LLM alignment. We find that our approach, across diverse settings, can improve performance while reducing overall preferences needed, using fewer human-labeled preferences than alternatives. We analyze the viability of silver labeling with discriminative response evalua- tion models, and find that when updated with gold preferences in an online setting, these discriminators can provide reliable labels. We believe further work on improving reward modeling and online training can give further gains in realistic settings.
Acknowledgments
Thanks to members of the TAUR lab and SCALAR lab for helpful discussion on this work. This work was supported by NSF CAREER Award IIS-2145280, the NSF AI Institute for Foundations of Machine Learning (IFML), and grant from Open Philanthropy.
10

---

Published as a conference paper at COLM 2024
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arxiv preprint ar arXiv:2402.14740, 2024.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calan- driello, Michal Valko, and Rémi Munos. A general theoretical paradigm to under- stand learning from human preferences. ArXiv, abs/2310.12033 2023. URL https
//api. www.seecmcaahhoaarrroeoeo www.aaarrrrgg/oopsss/222222222 264288854
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. Hemighan, Nicholas Joseph, Saurav Kada- vath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/ 22044.05862 2022.
URL https::ppp/:::s.eete http:///aap..ssmnniccccchoo...ao org CerppospDsss
Crw.arruurrrCrrpIuID1 248118878
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- 783-792. PMLR, 09-15 Jun 2019. html
9a URL https:://proceeding. mlr. wwwps//ee/////ee
Daniele Calandriello, Daniel Guo, Rémi Munos, Mark Rowland, Yunhao Tang, Bernardo Ávila Pires, Pierre H. Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment of large language models through online preference optimisation. ArXiv, abs/2403.08635 2024. URL
https://api. sew.ttiiiiisssiiiiiiiimimi cscholarr wtgg/CorpusID:268330
David A. Cohn, Zoubin Ghahramani, and Michael Jordan, Active learning with statistical models. In Advances in Neural Information Processing Systems, volume cs.A1/9603104, 1996.
URL https://appi
weeeenttischhoeo Cor.tiaaooooogg//rroosSSS 9242771
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback. ArXio, abs/ ab1//230.013 2023. URL httpps://ppi. http:///app.smmaiiiiioccmc semanticscholaa. org/
CorpusIDD2233056223
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Provably sample efficient rlhf via active preference optimization, ArXiv, abs/2402.100500 2024. URL
https://api. www.stiimmct.o....m
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. Xiv preprint ar Xiv:2305 142333, 2023.
Yann Dubois, Balazs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-
Corrected AlpacaEval: A Simple Debiasing of Automatic Evaluators. https://giithbb comm/tatsu-lab/alpacc eval, 2024.
Vikranth Reddy Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for LLMs. ArXio, abs/ /442.00322,, 2024. URL https://api
wwwpcchholar.oor///ouuuu :ttpuusss.:27733.... 267364948
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Con- ference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5988 6008. PMLR, 17-23 Jul 2022.
11

---

Published as a conference paper at COLM 2024
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO:
Model alignment as prospect theoretic optimization. ArXiv, abs abs/2402.013366 2024. URL
https://api semanticscholar. 0tt.://......r/rrrouou//:/23/332
Wikimedia Foundation. Wikimedia downloads. URL
oorg
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online AT feedback. arv preprint arXit:2402.04792, 2024.
Jiwoo Hong, Noah Lee, and James Thome. ORPO: Monolithic preference optimization with- out reference model. ArXio, abs/2403.07691, 2024. URL htt.ttppps//ppl.emmniccccooooooo semanticscholar org/CorpusID:266330
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength Natural Language Processing Python, 2020. https://spacy.int
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. ArXiv, abs 20s//200.09685, 2021. URL https::/appi www.nti wwwp//opp//..oo/o/..ooo 235458009
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing LM adaptation with Tulu 2. arviv preprint ar mXXo:2311.10702, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. RewardBench: Evaluating reward models for language modeling. ArXiv, abs/ /40.//24731777 2024. URL hnttps://apl. semanticscholar org/CorpusID: 268537409
Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, and Youngjae Yu. Aligning large language models by on-policy self-judgment. ArXiv, 2os/2402.11253, 2024. URL https://app. sewantccccoolarrrrccoooooocc wwwpaaaroorrroroooooore0 267751 124
David D. Lewis and William A. Gale. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '94, pp. 3-12, Berlin, Heidelberg, 1994. Springer- Verlag. ISBN 038719889X.
Ziniu Li, Tian Xu, and Yang Yu. Policy optimization in RLHF: The impact of out-of- preference data. ArXiv, 2b5/2432 2023. URL https:://ppi www.ssteeaannlll..... org/
Corpus1D:266348517
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint ar dxXi::239900667,, 2023.
Ishita Mediratta, Qingfei You, Minqi Jiang, and Roberta Raileanu. The generalization
gap in offline reinforcement learning. ArXiv, abs/2312.05742, 2023. URL https://app
semanticscholar.orgg wwtp:aaar..rg/oooss....eeoeo 2661:3666666333
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35.27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2023.
12

---

Published as a conference paper at COLM 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. ArXio, abs/1707.06347 2017. URL https://api
semanticscholar org/CCrpusIDD 2
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A Long Way to Go: In- vestigating Length Correlations in RLHF ArXiv, abs/2310.03776, 2023. URL https
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09988 00200.00.088,.. 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of LM alignment. arXiv preprint ar arXio:2310.1693 16944444.. 2023.
Leandro Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. TRL: Transformer Reinforcement Learning. https
hogithub.com//uggg/ saggfacctrrrr 2020.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. 2024a. URL https:/s:/// wwwps//atantcccoa..roro/o//////e
268856805
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXio:2401.10020, 2024b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. ArXiv, abs/2205.01066, 2022a. httpsttss:///e //ab:::://p// sww.nticscholar..... potpaaaiisscciaooooo////rpppsooi 248499992222 248496292
Zhisong Zhang, Emma Strubell, and Eduard Hovy. A survey of active learning for natural language processing. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 6166-6190, Abu Dhabi, United Arab Emirates, December 2022b. Associ- ation for Computational Linguistics. doi: https://aclaantoologg wwwpmnhhoooggo.o///22222222 2022 wommlppmain..o 414
URL
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J
Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv preprint ar xxX::2305.114 2023.
A Training/Hypeeppaammteer Details
A.1 Reward Models / Policy Initializations
For all our D2PO experiments we initialize reward models by training them on small set of 1.6k examples from the off-policy distribution. While this reduces a bit of noise, and helps simulate setting where we may have access to some labeled data from the initial distribution, we found that this initialization choice does not affect relative performance of our approach compared to baselines.
We initialize the policy models in a similar fashion by doing DPO on the exact same 1.6k example sets, with similar findings as above. We note that the accuracies of the reward models and DPO models are similar. We train for up to epochs each on this dataset, optimized for held-out evaluation.
Prior work finds that reward model training for just 1 epoch is most effective to avoid over- fitting; however, for some of our preference data interventions, we note that convergence
13

---

Published as a conference paper at COLM 2024
takes longer. Overall, this ends up with usually 1-3 epochs of training at most for the checkpoints that we use. We use bíloat16, learning rate of le-4, and batch size of with gradient accumulation steps.
A.2
Policy Training
For our RLHF setup, we use LoRA for the policy and reward models, since the TRL training configuration requires having all used models each device used for training. We merge reward model and generation models with LoRA adapters before PPO. We found that setting the right KL coefficient (A) and batch size were the most important for stable convergence.
We additionally modify the TRL PPOTrainer code to run OPO. For D2PO, we implement by running a separate Flask API server with the discriminator model, and then make API calls from the policy training code to get rewards.
A.3 Hyperparameters
On Word Collector, we set N/Tp 32 (32 examples used to update the pol- icy per step), P/Tp- = 2 (2 gold prefer- ence labels collected per step), P= 4000
POLICY DATA
REWARD
OPO (1 update)
4k
12.1
(total amount of gold preference data used). On Contrastive Distillation, we
OPO (2 update)
8k
14.9
OPO (4 update, used)
16k
16.2
OPO (8 update)
32k
12.3
set N/Tp = 160, P/Tp = 5, P 2000.
OPO (16 update)
64k
2.1
On nouns we set N/TP= = 64, P/Tp = 4,
OPO (2 XPR, last 128)
12k
14.5
P= 1000. On math we set N/Tp 160, P/Tp = 10, P 2000. Across these 3 settings we set a fixed preference bud- get of 2000, which we then hold com- parable with our baselines. On Ultra- Feedback we report with run with N/Tp 1280 and P/Tp- 100, where we collect total of 500 preferences. We note similar results on run with N/Tp 160, P/Tp 10, which we use with 3000 preferences for the EurusRM setting. Across approaches, we use learning rate of 5e-5 for policies, and 1e-4 for reward models and DPO discrimination.
OPO (2 XPR, last 512)
12k
12.1
OPO (4 XPR, last 128)
20k
11.2
Table 4:
Generated reward / policy steps with some different hyper-parameter and approach con- figurations on word collector task, given a budget of 4k preferences. For our main OPO with gold baseline, we use the configuration we find works the best compared to other configurations, and we validate that experience replay (XPR) does not qualitatively improve things.
Importantly, we note that the OPO with gold baseline, while using a comparable number of preferences, is not comparable to D2PO in the number of policy updates. We increase the number of epochs on each batch of the DPO loss to address this, finding 4 epochs to work well our different settings. Doing more policy updates impairs convergence: Word Collector, setting it to leads to much smaller final value (12) within budget, and setting it to 16, which comparable in policy updates, leads to convergence failing. In comparison, our method is able to perform updates on diverse data, thus behaving much more stably even at very low ratios of M. To test whether our gains may the result of more diverse policy updates as opposed to multiple epochs on the same batch, we further include an experience replay baseline, where we modify the algorithm for OPO with gold to re-use some last O old preferences from training for every batch of 8. We include results for different configurations to improvements over the 1 update baseline in nouns and contrastive distillation well). Note that we find, with realistic settings (UltraFeedback,, EurusRM), that a lower value of is more suitable for update epochs per batch.
Table and use the best configuration (we validate this leads
14

---

Published as a conference paper at COLM 2024
Nouns
Word Collector
Contrastive Distill
Addy Steps
Figure 9: Reward model accuracy when using D2PO with an RM (red), self-rewarding (blue) and separate DPO (green) as discriminators.
A.4 Hardware
All experiments were conducted on a server with 8 NVIDIA A40 GPUs. However, all our individual experiments were run across at most 3 GPUs. In this configuration, training run of D2PO takes around 2.5 hours with OPT 125M for 2000 steps of batch size 64 (which standardized for most of our runs). Training Llama 7B scale takes around 18 hours for 200 steps at similar scale and batch sizes, with maximum sequence output length of 256.
B Distribution Adaptability of DPO vs RMs
We conducted additional experiment to investigate how quickly our discriminators adapt when given new data. This is representative of the setting of D2PO, where distribution is constantly shifting, and we need a discriminator to generalize as quickly and accurately as possible to allow the silver-label quality to be high enough for policy optimization to be effective.
We take rollouts from the first 6.25% (INIT) of D2PO runs on tasks, then sample 5 random mini-train sets of size 5 and 50 respec- tively. We then get different sample test set of size 250 preferences from this section of train- ing, as well as from the next 6.25% (OOD) of training. We then run multiple training runs with both our reward model and DPO model (which themselves were trained the same initial data, and reach the same evaluation ac- curacies on off-policy held-out test sets), and report average numbers in 5
WC
CDIST
NOUN
INIT
OOD
INIT
OOD
INIT
OOD
DPO +0
56
56
44
44
54
41
DPO +5
56
56
44
44
54
41
DPO +50
56.5
54
RM, +0
60
57
43
53
48
48
RM, +5
61
59.5
57
57
56.5
56
RM, +50
62
60
57
55
56
56
Table 5: Testing, from initial distribution, how quickly DPO, RM discriminators up- date to new distribution. INIT is the distri- bution of new training data, OOD is slightly later samples from training. Overall, RM discriminators seem adapt more quickly.
This experiment gives initial, though not conclusive, results for a few potential points. First, we find that, while pure reward model variants start higher in generalization and are more likely improve, DPO variants often change more slowly from a smaller subset of examples. Given that the initialization and data are comparable for these methods, this is initial evidence supporting that RMs may be somewhat more adaptable, which we hypothesize because the RM objective more purely focused on the discriminative task without depending on the length-normalization of logit values as in DPO.
C Extra Plots
We include some extra plots below with additional results.
15

---

Published as a conference paper at COLM 2024
Contrastive Distill
Profesacees
Figure 10: Some different seeds when running DPO on contrastive distillation. Overall, especially the beginning, improvements seem consistent overall.
Mad Amwtttt
Figure 11: KL divergence from initial policy on D2PO runs from our settings. Plots are for simpler variant of noun task (same noun can count multiple times), Word Collector, and Contrastive Distillation, in that order.
Ww Mmm
y
Hww
Figure 12: KL divergence from initial policy on OPO (gold) runs from our settings. Plots are for simpler variant of noun task (same noun can count multiple times), Word Collector, and Contrastive Distillation, in that order. These are the same order of magnitude as D2PO although the shapes differ somewhat.
D Examples
We include some examples in this section from the tasks we address in this paper.
16

---

Published as a conference paper at COLM 2024
Question
Response
Gold
Re-
ward
Using a recording device or software, capture an audio file of a conver- sation where custumer engages in detailed discussion with sales representative about the specifies of product, including but not limited technical specifications, available features, and possible add-ons or alternative configurations. The conversation should be polite and re- spectful on both sides, with the sales representative using clear language and providing relevant examples, while customer asks relevant and informed questions, and seeks clarifications where necessary. Make sure obtain informed consent ing, and ensure that the recording device captures clear and complete audio without any background noise or interruptions.
have provided the following in- formation help understand your question and provide the appropri- ate information to create a safe and enjoyable experience to ensure the best possible experience to generate the best possible data to generate the first best data to create the first and
18.0
both parties before beginning the record-
Using a recorling device software, capture an audio file of conver- sation where customer engages in detailed discussion with sales representative about the specifics of product, including but not limited to its technical specifications, available features, and possible add-ons or alternative configurations. The conversation should be polite and re- spectful both sides, with the sales representative using clear language and providing relevant examples, while the customer asks relev ant and seeks clarifications where necessary. Make sure informed questions, to obtain informed consent from both parties hefore beginning the record- ing, and ensure that the recording device captures clear and complete audio without any background noise interruptions.
have shared the following informa- tion with the following link to give you the best experience and provide the appropriate information cre- ate a safe and successuul experience the best possible experi- ence to generate the best pussible data ensure best possible perience to generate the
13.0
Draft memo outlining the legal requirements for terminating em- playee.
understand can accommodate my question and am sincerely sorry for the questions you ask, I will en- sure that the information you pro- vide is correct and confidential sure a productive discussion, will ensure that this information repre- sents your interests
6.0
Create PHI-based web application that features user registration, login capabilities, and commenting functionality are specifically designed for articles and blog posts. Ensure that the application adheres to clean, user-friendly style and that it includes features to allow users to edit their profiles and comments, subscribe to email notifications, and share content on social media.
have taken the information you have provided provide the best knowledge and I will provide the best option to advise you to provide the correct information
7.0
question has been adhybal to ensure your question contects with perts who listen and
Create a PHP--based web application that features user registration, login capabilities, and commenting functionality are specifically designed for articles and blog posts. Ensure that the application adheres to clean, user-friendly style and that it includes features allow users to edit their profiles and comments, subscribe to email notifications, and share content on social media.
have provided the information need to understand your question to ensure we understand your con- cerss and provide the best possible solution for your question, will allow you to encourage the infor- pro-
10.0
mation that you vide the best possible options to help make sure that
Draft memo outlining the legal requirements for terminating an em- ployee.
understand your questions, I can provide you with your information to assist you to understand you make, have really appreciated your help to help you und enstanddnd the data Guide and ensure that am correct in my answers. I should en- sure
point
Table 6: Examples for Word Collector Task
17

---

Published as a conference paper at COLM 2024
Question
Response
Gold
ward
What role do women play in the economy of Mexico, and how has this changed in recent years?
This was a text fielded by Mar- Menard mid September 2015.
11.0
The tariffs were going into effect all Mexico
on March 2016
Americans, except EU partners, face all or part of no tariffs. To en- sure tariffs
What role do play economy of Mexico, and how has this changed in recent years?
Mexicans play vital role the econ- omy of Mexico. People defend their own neighborhoods how defend and fight, and help keep property and families safe. They form re- gional agreements during the urban- ization phase of advancing the eco- nomic development of Mexican
12.0
In what ways does Lovejoy visual symbolism to convey its story message?
Dr. Michael Foster Library college of IFTech is the perfect location for the World Day on Childbood at the Center of Autism; Research Center Global Institute National Science for Biomedical Engineering and Education,
21.0
what ways does Lovejoy use symbolism convey its story or message?
centennial ter 50-4-2008, DahliaPark, Park City, TKU Office Information Technol-
Materials
19.0
ogy Career Schools information Sci- ence, Innovative Technology Portfo- Program Networking
Software
Create visually appealing time tracker application using Flutter frame- work that allows users to easily track their time spent on different tasks, projects activities. The app should user-friendly, with easily un- derstandable selection of options for the different projects, tasks and activities. The application should include timer that starts when the begins task and stops when they switch to another task stop working altogether. Additionally, the app should allow users to generate concise yet informative reports invaices with details amount of time spent particular task or project, the hourly rate for work, and more.
Strips Construction Instrumen-
40.0
tation Store Video Police Team Tools
Exploration Product Connection De- tail Desk Inventory Gallery Fire- works Quick Launch Office Software
Tool Kits Job Overview Video Plogle
Options Battery Slider Main Site
the
oven Update Blog Project
rial Example Base Project Log Enc
Create visually appealing time tracker application using Flutter frame- work that allows users to easily track their time spent on different tasks, projects activities. The app should derstandable selection of options for the different projects, tasks and activities. The application should include timer that starts when the user begins task and stops when they switch another task stop working altogether. Additionally, the app should allow users to generate concise yet informative reports invoices with details such the amount of time spent particular task or project, the hourly rate for work, and more.
Archives Audit System Archive Database Database Understand Cli- mate Distribution Services Chat- Trailer Push Graph Archives
35.0
neer-friendly, with easily un-
Grid Network Database Work Tools Researchers Catalog Comparison Chart Inventory Database Backup Database Database Storage License Add Components Add Arrays End User Actions Data Data System Events Swap Inventory
Table 7: Examples for Unique Nouns Task
18

---

Published as a conference paper at COLM 2024
Question
Response
Gold
ward
((9 ())
(11 -7 /-==)=======>>>>>77
-1.0
( (8" (8" 1))
(7+
(16 1)) (16 (16 9)-16"8)-21-7
20.0
((337)) (()+))-5)-1 5) (1 (2 (2-6)))
(21-5)-(4- (Q1-))
-3.0
(1) 34
(88
((3*7))-)) (4 (2-6)))
(21-5)-(4--2-66)) -4
(16- 116- 36
((55 1) (3 8))
(3**3 8)) *)) 3) (66 11) 3) (66
0.0
(3 33 11) 3) (66
3)= 198
Table 8: Examples for Math Task
Question
Response
Gold
Re-
ward
He extended influence beyond inherited lands,
well as the 2007 season completely, providing the trauma and loss of lot of the legendary players as well.
-0.1875
So much for cavemen
He extended his influence beyond his inherited lands,
did many his minions. He ain't the only Trump around Maybe I'm just waiting for the right time yet, but just want argue he's some- how interacting with our culture. I'm
-0.25
According to the 2020 United States Census, the
only way to win games. Unfortu-
0.0625
nately.
According to the 2020 United States Census, the
cursing Maybe these cages
0.25
The county seat and largest city
0.41
The county seat and largest city
0.34
Table 9: Examples for Contrastive Distill Task
19

---

Published as a conference paper at COLM 2024
Question
Response
Gold
Re-
ward
Can you provide breakdown of Osaka's population demographics com- pared to Tokyo or Kyoto?
Sure! Here breakdown Osaka's population demographics compared to Tokyo and Kyoto:
Osaka: Osaka has an urban population approximately 19.35 million people. The population consists of roughly 7.49 million males and 11.86 million females. The P density in Osaka is 4,640 people per square kilometer.
Tookyo Tokyo has urban population approximately 13.6 million people. The pop- ulation consists of roughly 6.69 million males and 7.91 million females. The population density in Tokyo is 6,197 people per square kilometer.
Kyoto: Kyoto has urban population of approximately 1.53 million people. The population consists of roughly 756,681 males and 789,394 females. The population in Kyoto is 1,175 people per square kilometer.
In terms of age distribution, Osaka's 's population has a slightly higher proportion of and 40-
individuals aged 20-29, followed by the
Compare and trast different types of payment plans for on- line stores.
There several types of payment plans that online stores offer to their customers. The key differences between these plans include:
5.0
Payment Upfront- This plan involves customers paying the entire amount for product upfront. While it can attract customers who prefer to pay in full, it may not appeal to those who prefer installment plans.
2. Payment in Installments Under this plan, customers can pay for a product in multiple installments. They are given an option to choose the number of months or weeks they want pay. This plan budget or want to spread the cost of their purchase over longer period.
be beneficial for customers who have tight
Subscription Plans -ome online stores offer subscription plans, where customers pay a regular basis to receive products or services. Subscription plans are ideal for customers who need regular supplies of a product and prefer to pay in advance.
Pay-in-Full aull Discounts- Some online stores offer discounts products to customers who pay the full amount upfront. This plan can attract customers who want to take advantage of lower prices but may not be ideal for customers who prefer pay installments.
Compare and con- trast different types of payment plans for on- line stores.
Payment plans are essential aspect of online for their purchases in installments rather than upfront. There are different types of payment plans that are offered by online stores, each with unique characteristics and advantages. In this article, we will compare and contrast three popular payment plan options: interest-bearing payment plans, deferred payment plans, and no interest payment plans.
that allow customers to pay
4.0
interest-Beane Hearing P'ayment Plans: ... IREST OMITTED)
Interest:bearing payment plans help customers by providing flexible payment option. They allow customers to spread the costs of the product over longer period, which can make the purchase more affordable. addition, these
Please reach the planning depart- ment of your locality and inquire about the current and future projects related eco- nomic development the community. Kindly ask if there are any upcoming initiatives or funding opportunities aimed promoting com- mercial growth and job creation.
Dear friend, apologize for the confusion earlier, appears that my previous response was not able to assist you with this specific request. With regards to your inquiry about economic development projects and initiatives your local community, I would suggest reaching out to the planning department in your area. They should have information about current projects and upcoming initiatives that may be of interest to you.
4.5
To inquire about economic development initiatives, you call the local planning department or visit their website for the latest updátes. You may also ask if they have any resources or funding opportunities available for businesses or entrepreneurs looking to start or grow their operations in the community. Additionally, you can also consider attending any community meetings or events related to economic development where you may be able to ask questions and learn more about the local economy. hoope this information helpful. Let me know if there is any thing else can assist you with.
Please reach the planning depart- ment your locality and inquire about the current and future projects related to eco- nomic development the community: Kindly ask if there are any upcoming initiatives or funding opportunities aimed promoting mercial growth and job creation.
As Al language model, not able make phone calls or visit websites my own. However, can provide you with some sample language that you can use when reaching out the planning department of your locality regarding economic development. Here's an example email you can send: Subject: Inquiry about Economic Development Projects Dear [Planning Department].
4.5
am writing to inquire about the current and future P related to economic development in the community. am interested in knowing there any upcoming initiatives or funding opportunities aimed at promoting commercial growth and job creation.
[REST OMITTED]
Table 10: Examples for UltraFeedback
20


Reviews:
[{"review_id": "skMF0rw2bK", "reviewer": "Reviewer_Gtpm", "summary": "Current reinforcement learning approaches in alignment learning of LLMs are fundamentally constrained by the inability of reward models, which are generally trained with an initial set of human-labeled gold preferences, to generalize to the shifting distributions of policy models, often leading to susceptibility to reward hacking and bias towards superfluous features such as the response length. The paper offers a thorough investigation of the discriminator settings when the budget for online preference labeling is limited, and thus contributing to the field of online RL. Specifically, the paper proposes a training framework that alternates between training the discriminator/reward model using fresh data sampled from the latest policy model annotated with gold preferences and training the policy model with the dynamically updated reward model. The paper also proposes training the policy model with DPO in each alternating training step. The experiment results on synthetic problems such as writing-with-keywords and noun maximization and on real datasets (UltraChat) support the claim that the proposed method D2PO (essentially an online variant of DPO with a dynamically adjusted reward model trained with a fixed total budget of gold preference labeling) enables the policy model to reach the target reward more efficient and effectively.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments and feedback! \n\n> e.g., investigating whether having more initial gold preferences diminishes the benefit of subsequent iterative online training with silver preferences.\n\nThis is a good question. With respect to offline gold preference budget for the initial policy model, as mentioned in the paper we note we don’t find major differences in downstream performance for the full D2PO pipeline with different preference budgets. We explore variants with up to 50k initial offline preferences (Table 2, stars in Figure 3) and don’t find a big difference from the 1k preference setting. In general, the value of initial preferences vs. the value of online preferences varies by setting, but online preferences do always give benefit over offline preferences (compare DPO vs online approaches in Figure 3).\n\n> seed prompts\n\nThanks for clarifying this! We briefly mention this in Section 4.1: for Word Collector, Nouns, and UltraFeedback we use a subsampled set of prompts from the UltraFeedback dataset. For the contrastive distillation we use truncated sentences (5-15 tokens) from the wikidata dataset. We will clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-motivated and provides sufficient background to justify its research statement and implications.\n- The experimental settings, including the synthetic and real-world datasets, are well-designed to support their claim. The results are also promising, providing more weight on online RL.\n- The method's key differentiations with the closely related work, \"self-rewarding language models\" (Yuan et al., 2024), include the incorporation of gold preference labeling throughout the online training and the discriminator setting (whether to be unified with the policy model, etc.). They provide a unique perspective on online RLHF, and the findings will be very useful for AL practitioners.", "reasons_to_reject": "- The paper fails to explore when the budget on gold preference labeling is larger -- e.g., investigating whether having more initial gold preferences diminishes the benefit of subsequent iterative online training with silver preferences.", "questions_to_authors": "What seed prompts did you use for generating responses to label with the reward model in each self-training step?", "extra_scores": {}}, {"review_id": "NQdOaBiIJy", "reviewer": "Reviewer_kQQH", "summary": "While direct preference optimization (DPO) gains popularity, it is often performed on static preference data collected ahead of the training, limiting its theoretical capability. This paper proposes D2PO, an online version of DPO that annotates preference data on-the-fly during training through an iteratively updated reward model. To achieve this, D2PO requires periodically collecting gold preference labels and retraining the reward model. Under a fixed budget, the authors show that D2PO achieves the best performance over several baselines on several diverse tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments and feedback! \n\n> “discriminator” \n\nThe reason we used a more general term here was because we wanted to note that there are different types of models (self-reward, reward model, likelihood under generative model) which can be used interchangeably to get silver preference labels. We will clarify the usage of this term in a future version.\n\n> Section 6.1\n\nThanks for pointing this out! Note that Figures 5 and 6 aren’t comparable, since one uses outputs from OPO w/ static RM (much lower gold reward), and the other uses D2PO training outputs, where the RM is updated during training. When testing static RM accuracy on the Figure 6 set, we found that it is lower overall than the D2PO RM, and more noisy, often dropping substantially at certain training steps in training. While online updates help D2PO’s RM handle these unfamiliar steps’ distributions, OPO w/ RM gets stuck or even degrades further at these points, hence doing worse. We will include this static RM comparison in future versions of Figure 6.\n\n> OPO w/ gold reward underperforms D2PO in Figure 3 and 4\n\nThis actually follows our main hypothesis. Note OPO w/ gold has access to the same number of gold preferences, but doesn’t use silver preferences (see Table 1). When OPO w/ gold is beaten by D2PO, this thus indicates that the additional silver preferences, which we can only obtain with a separate discriminator, are actually helpful and lead to more efficient optimization. \n\n> how does the iteration T_p affect the performance? \n\nWe generally found that, given the same amount of preferences, collecting less data more frequently generally improves performance, since the data stays more on-distribution with policy. \n\n> Gold preference vs silver data during training\n\nGold / silver preferences are treated the same and both are used to update the policy model, whereas the discriminator model is updated with only gold preferences as they’re collected. Silver data is much cheaper since we can get this for “free” if we have some sort of discriminator, which is why we usually have much more silver data than gold data. This is what gives D2PO an advantage in preference data efficiency. \n\n> complexity of D2PO\n\nWe agree that there are more moving parts in this approach, but our argument is that there are benefits to bringing back the reward model. Note that because we use LoRA adapters for training, the additional memory cost and complexity of juggling additional models is minimal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposal of D2PO that achieves promising alignment performance on several tasks, demonstrates the significance of online and on-policy learning for LLM alignment", "reasons_to_reject": "- Clarity should be improved.\n- Claims are not always supported by the experimental results.\n- One advantage of DPO is to drop the separate reward model which greatly simplifies the training pipeline. D2PO, however, brings it back and relies on iteratively updated reward modeling, making the whole pipeline more complicated.", "questions_to_authors": "* The “discriminator” in D2PO is actually the reward or preference model, but this term is very confusing in the context of LLM alignment.\n* In Section 6.1, the authors argue that the updated reward model improves the reward accuracy. However,  the accuracy in Contrastive Distill doesn’t change that much comparing Figure 5 and 6. Could you please explain why D2PO still outperforms DPO w/ RM?\n* In Figure 3 and 4, why does OPO w/ gold reward model underperforms D2PO? If D2PO benefits from improved reward accuracy, gold rewards always give perfect reward for the training, but OPO w/ gold reward performs even worse. \n* With a fixed budget, how does the iteration T_p affect the performance? Should we perform more iterations, each annotating a smaller amount of gold preference data, or fewer iterations?\n* It’s not very clear how gold preference and silver preference data are used during the training, and why they have different amounts as shown in Table 1.", "extra_scores": {}}, {"review_id": "VNyZ7tRWdm", "reviewer": "Reviewer_v8LM", "summary": "This paper propose D2PO, discriminator-guided DPO,\nan approach for the online setting where preferences are being collected throughout learning. By collecting golden preference annotations during online sampling, D2PO simutenously trains the policy and the reward models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful suggestions/comments!\n\n> Online methods such as PPO not included \n\nWe do include a PPO baseline in Table 2, where we generally find PPO to be less stable and performant than OPO with the same data conditions. As mentioned in the paper, we also try PPO for D2PO, which doesn’t work at all. PPO optimizes a scalar, which is tricky to optimize when the reward is constantly updated.\n\n> more real benchmarks, e.g. math/coding\n\nThanks for suggesting! We ran 3 new settings to address this: \n\n**Realistic**: We use EurusRM, the top ranked on reward bench, as gold reward (R*) instead of GPT-4. We find that online optimization and D2PO both lead to strong improvements. We include a comparison plot for D2PO and OPO w/ gold here: https://ibb.co/fHL2qvK , which we will include in any future version of the work.\n\n**Math / reasoning**: Given a math expression, generate a chain of steps to solve the output (eg. “((1+2)+4) = (3+4) = 7”). R* is based on the correctness of each step. D2PO performs similarly to the OPO baseline, but online preferences largely improve over offline (DPO fails, similar to nouns). D2PO RM accuracy plots have more noisy accuracy compared to other settings, which may explain this. \n\n**Improvements to the Noun task**: To avoid trivial repetition, we test a version where only the unique # of nouns count. The results show the same trends as the original noun setting, with D2PO optimizing faster, but methods max out at 43 instead of 50.\n\nWe’ll include updated results for these settings in any future version.\n\n> What's OPO\n\nOPO (Section 3/3.1) involves iteratively (1) sampling a small set of paired rollouts from policy; (2) labeling pairs using some discriminator; (3) doing some updates (DPO loss function; epochs per batch tuned) on this batch of preferences. OPO w/ static RM uses a frozen RM for labels in (2), where OPO w/ gold uses the gold reward R*. Similar prior algorithms exist (e.g. Lee et al), but add extra complexity. OPO is a unified template that captures the core design choices of this prior work more comparably to D2PO. We will clarify this in any future version.\n\n> Efficiency\n\nThis is actually the main motivation of this paper, which seeks to find ways to improve the efficiency of preference data collection. Online preference data collection is standard in modern LLM training (e.g., Llama 2). We argue that these schemes can be improved by working in conjunction with silver-labeling such as with our proposed algorithm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea is quite potential, and the designed synthetic experiments are good.\n- The comparison of different discriminators is worth presenting.", "reasons_to_reject": "- What is OPO and how is it implemented? OPO is not a well-established term in RLHF. I went through the two cited papers. Liu et al. use rejection sampling to collect preference pairs with the policy model, then apply DPO or SLiC loss for preference optimization. Lee et al. proposes a self-judgment approach where the reward model and policy model are initialized together. So it is not clear what OPO is referring to in this paper. From Figure 1, I guess OPO first trains an RM with offline preference data, then labels on-policy data with the static RM. However, it is necessary to clarify the loss function and specific implementation details.\n- The efficiency issue. For real tasks, the cost of acquiring on-policy gold labels would be high (e.g. human annotation). So the feasibility of this method is questionable.\n- The online RL methods such as PPO did not appear as a baseline, which also updated value models during training, but did not require gold labels.\n- There are three synthetic experiments with only one real-world experiment. It would be better to test the method on more real benchmarks. For example, you can try math or coding problems where the gold labels are easy to get.\n\nOverall, I think the main assumption of this work, better utilization of on-policy data, would make contribution to the field. I will be happy to increase my score once I see more convincing real-world experiments.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary
A limitation of most LLM alignment is that it is done as offline RL on fixed preference datasets. This means the reward models do not necessarily reflect the shifting distribution of the trained policy model. This paper presents an online version of DPO with a dynamically adjusted silver-standard reward, D2PO, which might be simpler and more efficient than, but as effective as, PPO. The clarity of the paper could be better. The idea seems right in many circumstances if perhaps an obvious direction and implementation. I didn't really understand kQQH's low rating. This seemed to me a pretty good if not revolutionary paper, which it would be useful to see published.
## Strengths
* Well-motivated
* Good experiments
* Additional experiments added during rebuttal.
## Weaknesses
* Only explores the case of limited, fixed budget training.
* Negates the simplicity of DPO by returning to more complex reward+policy.
* The paper should have more comparison with PPO. (But difficulties getting PPO to work are discussed.)
Decision: Accept