D2PO: Discriminator-Guided DPO with Response Evaluation Models

Authors: prasann  singhal
nathan  lambert
scott  niekum
tanya  goyal
greg  durrett
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 18:14:44

Abstract: Varied approaches for aligning language models have been proposed, including supervised fine-tuning, RLHF, and direct optimization methods such as DPO. Although DPO has rapidly gained popularity due to its straightforward training process and competitive results, there is an open question of whether there remain practical advantages of using a discriminator, such as a reward model, to evaluate responses. We propose D2PO, discriminator-guided DPO, an approach for the online setting where preferences are being collected throughout learning. As we collect gold preferences, we use these not only to train our policy, but to train a discriminative response evaluation model to silver-label even more synthetic data for policy training. We explore this approach across a set of diverse tasks, including a realistic chat setting, and we find that our approach can lead to higher-quality outputs compared to DPO with the same data budget, and greater efficiency in terms of preference data requirements. Furthermore, we show that our silver labeling is most helpful when training the policy with DPO, outperforming traditional PPO, and benefits from maintaining a separate discriminator from the policy model.

Main paper: 
 introduction: 
 Learning from human preferences is the prevailing method for large language model (LLM) alignment, including approaches like reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Bai et al., 2022) , Direct Preference Optimization (DPO) (Rafailov et al., 2023) , and several recent alternatives (Azar et al., 2023; Ethayarajh et al., 2024; Hong et al., 2024) . A key idea in this line of work, introduced by DPO, is that the reward objective can be expressed in terms of the optimal policy and reference policy, allowing us to train a model from preference data without learning a separate reward model or sampling from the policy during learning.However, the theoretical guarantees of DPO may not apply in practice. Preferences are not necessarily obtained over a set of outputs that are in-distribution for the final aligned model. For example, preferences may be labeled over initial model outputs of a certain length, but the distribution of the policy model may shift during training to produce longer responses (Singhal et al., 2023) . In this case, the preference data does not uniquely specify an optimal policy, so different algorithms may lead to different solutions in practice. At the same time, recent approaches have explored collecting preferences from the shifting distribution of the model throughout training (Touvron et al., 2023; Yuan et al., 2024b) . It is not clear theoretically nor empirically which approach is best given a limited budget for preference labeling in this online setting. This paper investigates the role of an explicit discriminative model during the alignment process. Our central hypothesis is that when preference data is limited, a model discriminatively trained to evaluate responses (like a reward model) can learn to assess them more O v Y K j 2 4 9 p m n I I q C C K F V 1 7 A R q O Z H A q W B t 0 0 0 V S w i 9 I Q 1 W 1 R i R k K l a 3 o 2 g j S 2 t 1 H E Q S 3 0 i w F 3 1 + 0 R O Q q W y 0 N e d n X 3 V b 6 8 j / u d V U w h 2 a j m P k h R Y R H s P B a n A E O N O n r j O J a M g M g 2 E S q 7 / i m m T S E J B p 2 7 q E J z f K / + F 8 / W y s 1 X e P N k o 7 u 3 3 4 x h D C 2 g R l Z C D t t E e O k L H q I I o u k f P 6 B W 9 G Q / G i / F u f P R a B 4 z + z D z 6 U c b n F 7 R Q q 5 M = < / l a t e x i t > Y S Y J J K Q M l o y E 4 g x k b o A y J c x f C Z t Q R R m Y + L E J w X 2 + 8 j L 0 t j v u X m f 3 2 0 7 z 6 L i K Y x V t o o + o h V y 0 j 4 7 Q V 3 S G u o i h n + g e / UY P 1 o 3 1 y / p j P S 5 a a 1 Y 1 8 w H 9 U 7 X 6 E 7 T g r V w = < / l a t e x i t > y 1 , y 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 G easily than a model can learn to produce them. This discriminative response evaluation model can then be used to silver-label samples from our policy to give additional data for policy training.v N u 1 P e W u x U Z r s l E 2 j i M t j h U K Y = " > A A A C U 3 i c b V H N T 9 s w H H W y b u u 8 M c o 4 7 m J R R S t a V y V o Y 7 t U q o A D R 5 h o Q W q 6 y H G d x s P 5 k P 3 L 1 i j q / 4 i Q O P C P c O E A 7 s e B 0 T 3 J 8 t N 7 7 y f b z 2 E u h Q b X v b X s F 7 W Xf P / 3 C l R Z a e Q Z n z U U I n q Y g E o 2 C k o P H b 8 R M K c R h V 8 e w T d q B 7 h h 0 / F w G 0 S u I n Y k y m u 9 g 5 I l 3 i V 6 Q 1 D U S b l I H 4 9 X m 5 f d k l / g w 7 P 1 v T d m m o E p M Y q F L Z X 6 K w M 8 V O G X j t M t j D y 0 D Q a L o d d w G y T r w V a a I V T o L G t T / O W J H w F J i k W g 8 9 N 4 d R R R U I J v k M + 4 X m O W W X d M K H h q Y 0 4 X p U L T q Z E c c o Y x J l y q w U y E J 9 O l H R R O s y C U 1 y X o B + 7 s 3 F / 3 n D A q I f o 0 q k e Q E 8 Z c u D o k I S y M i 8 Y D I W i j O Q p S G U K W H u S l h M F W V g v g G b E r z n T 1 4 n g 7 2 O t 9 / 5 d v q 1 2 T t Y 1 V F H H 9 E O a i E P f U c 9 d I x O U B 8 x d I X u 0 I O F r B v r 3 r b t 2 j J q W 6 u Z b f Q P 7 I 1 H W D 2 s n g = = < / l a t e x i t > R(x, y)We operationalize this approach in a a method called D2PO (Figure 1 ). D2PO alternates between two phases: first, collecting preference labels to train the discriminative response evaluation model (blue), and second, using that discriminator to label a larger number of outputs from the policy model (teal). We use the discriminator as a pairwise preference labeler and train the policy using the DPO objective, which we find works better than a conventional PPO setup (Schulman et al., 2017) . Finally, by decoupling the discriminator from the policy, it can be learned over human-labeled preference data only, while the policy model can learn from a larger amount of noisily-labeled on-policy data.Our results study both a realistic chat benchmark (UltraFeedback (Cui et al., 2023) ) as well as several simple text generation tasks where we explicitly define gold reward functions. We show several key findings. First, we show that the online preference collection setting indeed works better than having only static preferences collected from the initial policy model. We then show that, given the same budget of preference updates as baselines, D2PO on several settings achieves higher reward more quickly than an online version of DPO and than basic PPO. Finally, we study how the discriminative reward evaluation model behaves over the course of training. Receiving new labeled data is crucial for it to be able to make accurate judgments about new sampled responses. Combined with controlled D2PO experiments testing different types of discriminators, including using the policy as its own discriminator, we establish that in the setting we discuss, a separate discriminator may still be a useful ingredient in LLM alignment.Taken together, we make the following contributions.(1) We propose a new approach, discriminator-guided DPO, for the setting where preferences are being collected online.(2)We show, on a diverse set of tasks, that maintaining a discriminative response evaluation model and using it to silver label new sampled responses improves policy training. (3) We release code1 and our diverse task settings to support the testing and development of future alignment algorithms.The answer is 10The answer is 12The passage says… We hypothesize that online updates to discriminators, with a high number T P of preference collection steps, will help solve the distribution shift issue and improve performance. However, as preference data may be expensive, we want to do so with minimal P.The question then becomes how to use a small P to maximally improve our policy with respect to R * . To address this, D2PO combines two ideas: (1) the comparative advantage in data efficiency of online data; (2) the ability of discriminators to generalize quickly on new data and help silver-label data for a policy, reducing the need for costly gold preferences.We detail our approach, D2PO, in Algorithm 1 and Figure 2 . The algorithm, from an initialized policy and discriminator, runs for N iterations. In each iteration, the first step (L3-8) is to sample a batch of paired outputs from the policy π t , then get preference labels with our discriminator, and do DPO updates on these new silver preferences. Note this part of the algorithm directly corresponds to several baselines we will compare against: if we omit the rest of the algorithm, this on its own is OPO with a static RM (see below then replace R with R * here, getting gold labels instead, this becomes the OPO with gold baseline. Lastly, if instead of using π t we fix the rollout distribution to be π sft , this gives us DPO.Every N/T p steps, we collect gold annotation and update the discriminator (L11-17). In practice we can update the policy with these labels, though this is a sparse set of steps since the number of preferences P is much less than the number of policy training iterations N. This is the step depicted in Figure 2 . The subsampling step of D candidates can use a strategy such as confidence sampling (Lewis & Gale, 1994) , shown here, where we choose preference pairs with the lowest gap in discriminator reward, or random sampling, where we randomly select P pairs. We generally find these to perform similarly. We report with confidence sampling in most settings, except our Contrastive Distillation setting where we find random sampling to work better.Importantly, we then update our reward model with these preferences, allowing us to collect automatic preference labels on a larger set of rollouts for "free" (in terms of annotation cost), taking advantage of the discriminator's ability to generalize to other examples from a similar distribution. Note that our algorithm is flexible in the form of the discriminator, which is treated as a black box. We use a Bradley-Terry reward model by default, but in Section 6.2, we investigate if the DPO policy model itself or a separate copy of it can be used as the discriminator instead.The primary advantage of this approach lies in its efficiency: using purely gold rollouts can be effective, but also slow. By updating the reward model sparsely, these updates can generalize to new examples on the training distribution at each step, allowing more efficient collection and usage of preferences with equivalent performance. 
background / setup: 
 Let π(y | x) be an LM that places a probability distribution over a response y given an input x. Let D = {(x i , y + i , y i )} be a dataset of human preference judgments, with outputs y + i preferred to alternatives y i . These preferences are assumed to be derived from a scalar reward model R * (x, y) reflecting human utility, which we do not have access to. The goal of LM alignment training is to maximize this reward, optionally starting from a supervised fine-tuned policy π sft (y | x).In standard RLHF (PPO; (Schulman et al., 2017) ), first, a Bradley-Terry reward model R(x, y) is learned using preference data. To train the policy, online rollouts are then iteratively sampled from the policy π t at each training iteration, and the loss optimization depends on the reward scores assigned to these outputs by the learned reward model, allowing the policy to learn from these rollouts under the assumption that the reward model produces accurate rewards. Direct Preference Optimization (DPO; Rafailov et al. (2023) ), another popular approach, trains the policy directly, offline, on just the initial preference data using a discriminative loss, removing the separate discriminator. However, later work (Liu et al., 2023; Lee et al., 2024) , has explored incorporating online sampling with discriminative objectives. We broadly call this work online preference optimization (OPO) as a unified template to capture the core design choices of this prior work for comparison to D2PO (defined more in Section 3).While such methods are online with respect to the policy, they are not online with respect to preferences, which are usually collected from a fixed distribution such as π sft . Thus, as distribution π t changes over training, the signal of a reward model or a discriminator may degrade in quality, limiting improvement. Some work (Touvron et al., 2023; Guo et al., 2024) explores collecting preferences at an additional intermediate training step to address this issue. However, the intermediate steps, data, and objectives for this process are often chosen in an ad-hoc manner. 
d2po: using a discriminator in policy training: 
 Our setting This work focuses on the setting where we collect additional gold preference judgments during policy learning. We define the total gold-labeled preference budget, typically from humans or from LLM-as-a-judge, as P. In most approaches, like DPO and OPO, all preferences D 0 = {(x i , y To test our hypotheses, we compare our approach to several baselines, including a detailed breakdown of the data conditions for different approaches in Table 1 (corresponding to Figure 3 ). Note that we hold total gold preferences used, including those used for initialization (Appendix A), constant across all approaches within each setting, though total number of preferences (gold + silver labeled using the discriminator) may vary by setting. We first compare against approaches using a static set of preferences labeled over outputs from the initial policy. We compare DPO, PPO (static RM) and OPO (static RM), the latter two of which train a reward model over the preference data and use it during training, either as a reward in PPO or a labeling function in OPO (see Figure 1 ). Note that although we introduce the term OPO in this work, the implementation is similar to Liu et al. (2023) . The primary difference between OPO and PPO is OPO's use of the DPO loss function over the online examples. We further compare these approaches in a higher data (50K) gold preference setting to examine behavior with many more preferences (Table 2 ).We also compare against OPO (gold), where only online gold preference labels are used to update the policy, without any silver labeling. To make the number of policy updates more comparable with D2PO, we train each batch with the DPO objective for 4 epochs; we tuned this hyperparameter to optimize performance (Appendix A).Finally, we compare against variants of our approach that use both online gold and silver labels, including a version where we use an independently-optimized DPO discriminator for our response evaluation model and a version where we use the policy itself (self discriminator). We view this latter case as updating both the policy as well as the response evaluation model with silver labels. Finally, we compare against a version where we use PPO updates given our reward model; this is simply PPO with a reward model being updated online.Our methods have two notions of training progress, following Figure 1 : how many online gold preferences we have used from our budget T P and how many silver updates have been applied to our policy T N . Since human preferences are more costly, we're mainly interested in T P , where we'll hold x-axes constant across approaches with respect to preferences used. 
tasks: 
 To evaluate our policy optimization methods, we evaluate on a diverse set of tasks with distinct R * that we can derive relative preferences from. Example questions and outputs are in Appendix D). We use four synthetic tasks, designed to exhibit different properties, where we know the ground truth reward function R * , plus one realistic task with two R * settings.We compute the top 30 content words in the UltraFeedback (Cui et al., 2023) dataset. Given an UltraChat prompt (Ding et al., 2023) , the goal is to generate a 50 word output with as many of these words as possible, where the presence of each word gives +1 reward, for a maximum of 30. This simulates having different sparse "positive" features that naturally may not occur together, and learning to incorporate multiple in single outputs. We subsample prompts initial preference dataset from UltraFeedback. We find that this task allows for realistic outputs that achieve high reward, while at the same time being challenging to optimize and showing variation among the training methods we compare.Unique Nouns: Instead of maximizing word coverage, here we maximize the number of unique nouns, detected with spaCy (Honnibal et al., 2020) in a 50 token output. This is an example of a much denser reward function. We get our initial preference dataset by subsampling from UltraFeedback. This reward function is the most easily optimized of those we consider and helps measure how quickly policy learning can be "steered" by the reward.Contrastive Distillation: Given a larger θ L (OPT-1.3b, Zhang et al. (2022a) ) and smaller model θ S (OPT-125m), the R * is the difference of log probabilities log(p(y|x; θ L ))log(p(y|x; θ S )); we receive more reward for sequences likely under the large model and unlikely under the small model. This task is represents a likelihood-based ground truth reward, similar to how DPO's implicit reward is constructed. We get our initial preference dataset and prompts with OPT 125m samples on truncated (5-15 tokens) Wikipedia data (Foundation).The input is a mathematical expression randomly generated as a tree of up to two layers deep. From here, there is a single gold sequence, which is a chain-ofthought style sequence where the deepest, left-most sub-expression is solved one at a time (e.g., "((5 + 1) * 2) = (6 * 2) = 12"). Then, given a prediction such as "((5 + 1) * 2) = (6 * 2) = 13", we can iterate through each step, where R * is -1 multiplied with the total edit distance between solution and predictions at each step. This task has a single solution. Prompts and preferences are randomly generated.UltraFeedback: Finally, for a realistic setting where a "true" ground truth reward is unknown, we use the popular UltraFeedback (Cui et al., 2023) dataset, where we can use their GPT-4 based (gpt-4-0613) labeling scheme which returns a score between 1-5 as our R * . We just use subsampled UltraFeedback for our initial data. We further run an alternate setting using EurusRM (Yuan et al., 2024a) , the best reward model on RewardBench (Lambert et al., 2024) at the time the work was conducted, as an alternative gold reward.We build on top of the Huggingface TRL framework (von Werra et al., 2020) with hyperparameters we find to work best based on reward convergence and downstream evaluation: λ = 0.05, batch size 64; more details in Appendix A. We use Llama-2-7B models as our base for realistic experiments, OPT-1.3b for the math setting, and OPT-125m for our synthetic experiments (Zhang et al., 2022a) , and use LoRA (rank= 16) (Hu et al., 2021) to enable training with a smaller GPU footprint, finding it to not affect initial experimental DPO/RM results. We use the T ülu SFT models for UltraFeedback (Ivison et al., 2023) , an OPT-1.3b model fine-tuned on 500k example math expressions, and OPT-125m for other experiments. For each task, we evaluate the reward over the timesteps of training. Periodically throughout learning, we use our policy to compute gold reward over a held-out dev set of prompts. 
comparing loss objectives (dpo vs ppo): 
 NOUN WC CDIST MATH DPO 12 6.9 0.08 -17.1 PPO (static RM) 24.9 6.7 -0.20 -13.2 OPO (static RM) 33.2 7.8 0.22 -3.5 Results: Table 2 reports the gold final reward of training runs for the three baseline approaches. We find that OPO w/ static RM outperforms standard DPO in all settings, even though these use the same loss objective. This highlights the benefits of sampling online rollouts during policy training, as opposed to using a fixed preference set. Our results also show that PPO w/ static RM is generally outperformed by OPO, and even standard DPO for two out of four settings. We generally find the DPO objective to be more stable than PPO, and use the former for all our experiments. Finally, these results give lower reward than we will see in Section 5.2 from using online rewards, showing how effective online reward updates are.Setup: To compare these for equivalent gold preferences, we report model performance quantified by the gold reward on 200 held-out datapoints (y-axis) against the number of gold preferences used for training (x-axis). That is, we report standard DPO and OPO w/ static RM performance on the same preference budget as the total budget over all time steps of our online D2PO and OPO w/ gold methods. Note, however, that the number of policy updates may differ between these approaches (see Table 1 ). We report aggregated results across 3 seed runs.We compare our proposed approach D2PO and baselines in Figure 3 . First, we importantly note that prior methods like DPO and OPO (stars), which are offline with respect to preferences, do much worse than the online preference approaches, even with much more data. Next, comparing the online approaches, we find that on three settings, D2PO leads to overall improvements in efficiency of data, either reaching higher final reward or higher intermediate rewards using a lower preference budgets.For instance, on Word Collector, D2PO reaches a reward score of ∼ 35 with a preference budget of P = 100 where OPO w/ gold requires P = 300 to give the same performance. This suggests that silver-labelling using an iteratively improving RM is an effective strategy to offset the high annotation costs of online gold annotations. Note that this is not as clear on the Math setting, suggesting efficiency gains may depend on the setting. We explore this more in Section 6. Overall, our results suggest that both online preferences and an on-distribution RM can contribute to increased performance and data efficiency.Results on realistic task: Next, we report results on our realistic settings on UltraFeedback in Figure 4 . On the GPT-4 annotation-based setting (UltraFeedback), we compare approaches assuming access to a small preference budget of 500 online preferences; gold annotation in this setting using GPT-4 costs $60 per run. We also report results when we use a budget of 3k preferences on the Eurus RM-based R * setting.We include a plot (Figure 4 ) showing average of a sliding window of size 50 over gold reward from preference training given a budget of 500 for UltraFeedback. The Eurus reward plot is computed across checkpoints on a fixed eval set of 200 inputs. Overall, we find D2PO gets further in optimization than other approaches within the small preference budget. While this is smaller-scale, these results give us initial evidence suggesting that D2PO may have efficiency and performance benefits in practical chat settings. For extra reference, we report length-controlled AlpacaEval 2.0 (Dubois et al., 2024) in Table 3 compared with gpt-4turbo, as a winrate percentage out of 100%. We use the highest reward checkpoint for OPO and a D2PO checkpoint chosen based on discriminator reward (not shown, but we find this to be an effective stopping criterion due to correlating well with gold reward). We do not find a strong consistent pattern between our results and this evaluation, which we attribute to the fact that our reward models such as Eurus are not necessarily aligned with it. However, we note that performance at least does not degrade by optimizing for these objectives, which occurs in cases of overoptimization. 
reward model accuracy under distribution shift: 
 Our previous results establish that methods like D2PO that use an iteratively updated discriminator outperform those with static discriminator (e.g. OPO w/ static RM). We hypothesize this difference in performance is because as the distribution of the policy shifts during training, the performance of the static discriminator on rollout pairs sampled from this new distribution degrades. This results in less reliable rewards and less reliable labeling of new preference data, degrading the improvement from additional policy training.We first plot (see Figure 5 ) RM accuracy across policy training with the OPO with static RM baseline, computed as the fraction of times the preference label from the reward model R is same as that from the gold reward R * for pairs of rollouts sampled from the current policy (R * ties thown out). On most settings, we find static reward accuracy generally decreases as policy training proceeds, especially on Word Collector and Math. In fact, we see that the reward accuracy is ∼ 50% at some points during training, effectively random chance. D2PO discriminator accuracy does not degrade: We then ask how this compares to when using D2PO. In Figure 6 we show accuracy of D2PO's RM from rollout sets across training as distribution shifts (red). We include the accuracies of the DPO reward formulation of the policy at similar points in training (green), and the starting static RM (blue). Despite the distribution shift and much smaller amounts of data, D2PO allows for reward signal to stay non-random and even improve over the course of training, which explains the effectiveness of the silver-labelling as we see in Figure 3 before, where the static RM generally seems to do worse. We further note that the accuracy of the static RM and the policy (using DPO implicit reward) often lags behind the discriminator's accuracy, giving some evidence that the discriminator is consistently able to provide information the policy may not capture on its own, even as reward of generations improves. On the realistic EurusRM setting (Figure 8 ), we find D2PO improves over a static RM earlier in training, though not to the same extent.D2PO requires a discriminator to label preferences, but isn't dependent on a particular parameterization of discriminator, letting us treat it as a black box. We thus seek to answer the following question: how important is the choice of discriminator?Setup: We test three choices of discriminators: (1) D2PO-RM: Our standard condition of D2PO with a Bradley-Terry reward model (used in Section 5.2 and 6.2), (2) D2PO-DPO: an independent DPO discriminator-only model, starting from the π SFT and trained only on the gold preference data. This sees the same training data as the reward model in (1) but uses a different loss formulation.(3) D2PO-self-evaluator: a "self-rewarding"2 DPO discriminator where the D2PO's policy is used both as the response evaluation model and the generator of rollouts. We ensure overall amounts of data are held constant across approaches.We find (Figure 7 ) that the DPO and RM discriminators do similarly overall. The self-rewarding baseline fails to converge on Nouns and Math, and does poorly on Word Collector, but is best on Contrastive Distillation, where we note smaller ratios of gold preferences lead to training failure; we hypothesize that this is mainly because the noisy labels being fed back into the policy can contaminate the discriminative objective, hence the value of the separately-trained discriminator. We do a further study on DPO vs RMs in Appendix B. 
related work: 
 Preference Optimization: Aligning instruction-tuned language models with preference data has been reported to improve performance, both for proprietary (Ouyang et al., 2022; Touvron et al., 2023) and open source models (Ivison et al., 2023; Tunstall et al., 2023) . Several preference datasets have been released recently to facilitate further research in this space (Ethayarajh et al., 2022; Bai et al., 2022; Cui et al., 2023) . From an algorithm perspective, recent work has proposed several "simpler" variants of standard RLHF, such as reward-free methods that directly optimize with initially collected preference data (Rafailov et al., 2023; Azar et al., 2023; Ethayarajh et al., 2024) or iteratively updating the preference data using gold annotators (Guo et al., 2024; Touvron et al., 2023) . In contrast, recent work has also explored alternate annotating strategies using a learnt reward model (Zhao et al., 2023; Liu et al., 2023; Calandriello et al., 2024) or the policy model itself for preference labelling (Lee et al., 2024; Yuan et al., 2024b) . Ahmadian et al. (2024) studies the performance of more stable RL algorithms. However, these methods either ignore the distribution shift of policy models and it's impact on reward model performance, or rely purely on expensive gold annotations during training. Our work bridges this gap by introducing a cost efficient method to ensure reward model does not degrade during policy training.Reinforcement Learning: Our reward generalization hypothesis draws from the notion of reward extrapolation in reinforcement learning literature (Brown et al., 2019) . Here, we expect our response evaluation model, trained online, to be able to effectively label silver data drawn from the same (evolving) distribution. Likewise, several results have supported the benefit of online over offline methods (Mediratta et al., 2023; Li et al., 2023) .The notion of actively collecting samples to improve models during optimization originates with active learning (Cohn et al., 1996; Zhang et al., 2022b) . In the context of LLM training specifically, active approaches for prompt selection and off-policy reward training have started to get explored (Das et al., 2024; Dwaracherla et al., 2024) . We do not claim innovation on the side of actively collecting preferences; our focus is on the utility of respond evaluation models rather than active selection strategies. 
conclusion: 
 In this paper, we present D2PO, a method for learning from online preferences for LLM alignment. We find that our approach, across diverse settings, can improve performance while reducing overall preferences needed, using fewer human-labeled preferences than alternatives. We analyze the viability of silver labeling with discriminative response evaluation models, and find that when updated with gold preferences in an online setting, these discriminators can provide reliable labels. We believe further work on improving reward modeling and online training can give further gains in realistic settings. 
 
 Appendix: For all our D2PO experiments we initialize reward models by training them on a small set of 1.6k examples from the off-policy distribution. While this reduces a bit of noise, and helps simulate a setting where we may have access to some labeled data from the initial distribution, we found that this initialization choice does not affect relative performance of our approach compared to baselines.We initialize the policy models in a similar fashion by doing DPO on the exact same 1.6k example sets, with similar findings as above. We note that the accuracies of the reward models and DPO models are similar. We train for up to 5 epochs each on this dataset, optimized for held-out evaluation.Prior work finds that reward model training for just 1 epoch is most effective to avoid overfitting; however, for some of our preference data interventions, we note that convergence takes longer. Overall, this ends up with usually 1-3 epochs of training at most for the checkpoints that we use. We use bfloat16, learning rate of 1e-4, and batch size of 2 with 2 gradient accumulation steps.For our RLHF setup, we use LoRA for the policy and reward models, since the TRL training configuration requires having all used models on each device used for training. We merge reward model and generation models with LoRA adapters before PPO. We found that setting the right KL coefficient (λ) and batch size were the most important for stable convergence.We additionally modify the TRL PPOTrainer code to run OPO. For D2PO, we implement it by running a separate Flask API server with the discriminator model, and then make API calls from the policy training code to get rewards. Table 4 : Generated reward / policy steps with some different hyper-parameter and approach configurations on word collector task, given a budget of 4k preferences. For our main OPO with gold baseline, we use the configuration we find works the best compared to other configurations, and we validate that experience replay (XPR) does not qualitatively improve things.On Word Collector, we set N/T P = 32 (32 examples used to update the policy per step), P/T P = 2 (2 gold preference labels collected per step), P = 4000 (total amount of gold preference data used). On Contrastive Distillation, we set N/T P = 160, P/T P = 5, P = 2000. On nouns we set N/T P = 64, P/T P = 4, P = 1000. On math we set N/T P = 160, P/T P = 10, P = 2000. Across these 3 settings we set a fixed preference budget of 2000, which we then hold comparable with our baselines. On Ultra-Feedback we report with a run with N/T P = 1280 and P/T P = 100, where we collect a total of 500 preferences. We note similar results on a run with N/T P = 160, P/T P = 10, which we use with 3000 preferences for the EurusRM setting. Across our approaches, we use learning rate of 5e-5 for policies, and 1e-4 for reward models and DPO discrimination.Importantly, we note that the OPO with gold baseline, while using a comparable number of preferences, is not comparable to D2PO in the number of policy updates. We increase the number of epochs on each batch of the DPO loss to address this, finding 4 epochs to work well on our different settings. Doing more policy updates impairs convergence: on Word Collector, setting it to 8 leads to much smaller final value (12) within budget, and setting it to 16, which is comparable in policy updates, leads to convergence failing. In comparison, our method is able to perform updates on diverse data, thus behaving much more stably even at very low ratios of P to M. To test whether our gains may be the result of more diverse policy updates as opposed to multiple epochs on the same batch, we further include an experience replay baseline, where we modify the algorithm for OPO with gold to re-use some last O old preferences from training for every batch of 8. We include results for different configurations in Table 4 , and use the best configuration (we validate this leads to improvements over the 1 update baseline in nouns and contrastive distillation as well).Note that we find, with realistic settings (UltraFeedback, EurusRM), that a lower value of 2 is more suitable for update epochs per batch. We include some examples in this section from the tasks we address in this paper.Question Response Gold Reward Using a recording device or software, capture an audio file of a conversation where a customer engages in a detailed discussion with a sales representative about the specifics of a product, including but not limited to its technical specifications, available features, and possible add-ons or alternative configurations. The conversation should be polite and respectful on both sides, with the sales representative using clear language and providing relevant examples, while the customer asks relevant and informed questions, and seeks clarifications where necessary. Make sure to obtain informed consent from both parties before beginning the recording, and ensure that the recording device captures clear and complete audio without any background noise or interruptions.I have provided the following information to help understand your question and provide the appropriate information to create a safe and enjoyable experience to ensure the best possible experience to generate the best possible data to the first best data to create the first and 18.0 Using a recording device or software, capture an audio file of a conversation where a customer engages in a detailed discussion with a sales representative about the specifics of a product, including but not limited to its technical specifications, available features, and possible add-ons or alternative configurations. The conversation should be polite and respectful on both sides, with the sales representative using clear language and providing relevant examples, while the customer asks relevant and informed questions, and seeks clarifications where necessary. Make sure to obtain informed consent from both parties before beginning the recording, and ensure that the recording device captures clear and complete audio without any background noise or interruptions.I have shared the following information with the following link to give you the best experience and provide the appropriate information to create a safe and successful experience to ensure the best possible experience to generate the best possible data to ensure the best possible experience to generate the 13.0Draft a memo outlining the legal requirements for terminating an employee.I understand you can accommodate my question and I am sincerely sorry for the questions you ask, I will ensure that the information you provide is correct and confidential to ensure a productive discussion, I will ensure that this information represents your interests 6.0 Create a PHP-based web application that features user registration, login capabilities, and commenting functionality that are specifically designed for articles and blog posts. Ensure that the application adheres to a clean, user-friendly style and that it includes features to allow users to edit their profiles and comments, subscribe to email notifications, and share content on social media.I have taken the information you have provided to provide the best knowledge and I will provide the best option to advise you to provide the correct information to ensure no question has been adhybal to ensure your question connects with our experts who listen and 7.0 Create a PHP-based web application that features user registration, login capabilities, and commenting functionality that are specifically designed for articles and blog posts. Ensure that the application adheres to a clean, user-friendly style and that it includes features to allow users to edit their profiles and comments, subscribe to email notifications, and share content on social media.I have provided the information I need to understand your question to ensure we understand your concerns and provide the best possible solution for your question, I will allow you to encourage the information that you provided to provide the best possible options to help make sure that 10.0 Draft a memo outlining the legal requirements for terminating an employee.I understand your questions, I can provide you with your information to assist you to understand the point you make, I have really appreciated your help to help you understand the data Guide and ensure that I am correct in my answers. I should ensure 8.0 There are several types of payment plans that online stores can offer to their customers.The key differences between these plans include: 1. Payment Upfront -This plan involves customers paying the entire amount for a product upfront. While it can attract customers who prefer to pay in full, it may not appeal to those who prefer installment plans. 2. Payment in Installments -Under this plan, customers can pay for a product in multiple installments. They are given an option to choose the number of months or weeks they want to pay. This plan can be beneficial for customers who have a tight budget or want to spread the cost of their purchase over a longer period.3. Subscription Plans -Some online stores offer subscription plans, where customers pay on a regular basis to receive products or services. Subscription plans are ideal for customers who need regular supplies of a product and prefer to pay in advance. 4. Pay-in-Full Discounts -Some online stores offer discounts on products to customers who pay the full amount upfront. This plan can attract customers who want to take advantage of lower prices but may not be ideal for customers who prefer to pay in installments.Compare and contrast different types of payment plans for online stores.Payment plans are an essential aspect of online stores that allow customers to pay for their purchases in installments rather than upfront. There are different types of payment plans that are offered by online stores, each with unique characteristics and advantages. In this article, we will compare and contrast three popular payment plan options: interest-bearing payment plans, deferred payment plans, and no interest payment plans. Interest-Bearing Payment Plans: ..Interest-bearing payment plans help customers by providing a flexible payment option. They allow customers to spread the costs of the product over a longer period, which can make the purchase more affordable. In addition, these Code available at https://github.com/PrasannS/d2poNote that in self-rewarding settings likeYuan et al. (2024b), the model is prompted to give rewards; here, we use its likelihoods, which give usable results even for models that are not strong zero-shot evaluators of their own outputs when using prompting. 


Decision: ## Summary
A limitation of most LLM alignment is that it is done as offline RL on fixed preference datasets. This means the reward models do not necessarily reflect the shifting distribution of the trained policy model. This paper presents an online version of DPO with a dynamically adjusted silver-standard reward, D2PO, which might be simpler and more efficient than, but as effective as, PPO. The clarity of the paper could be better. The idea seems right in many circumstances if perhaps an obvious direction and implementation. I didn't really understand kQQH's low rating. This seemed to me a pretty good if not revolutionary paper, which it would be useful to see published.
## Strengths
* Well-motivated
* Good experiments
* Additional experiments added during rebuttal.
## Weaknesses
* Only explores the case of limited, fixed budget training.
* Negates the simplicity of DPO by returning to more complex reward+policy.
* The paper should have more comparison with PPO. (But difficulties getting PPO to work are discussed.)

Reviews: review: paper_summary: paper_summary: Current reinforcement learning approaches in alignment learning of LLMs are fundamentally constrained by the inability of reward models, which are generally trained with an initial set of human-labeled gold preferences, to generalize to the shifting distributions of policy models, often leading to susceptibility to reward hacking and bias towards superfluous features such as the response length. The paper offers a thorough investigation of the discriminator settings when the budget for online preference labeling is limited, and thus contributing to the field of online RL. Specifically, the paper proposes a training framework that alternates between training the discriminator/reward model using fresh data sampled from the latest policy model annotated with gold preferences and training the policy model with the dynamically updated reward model. The paper also proposes training the policy model with DPO in each alternating training step. The experiment results on synthetic problems such as writing-with-keywords and noun maximization and on real datasets (UltraChat) support the claim that the proposed method D2PO (essentially an online variant of DPO with a dynamically adjusted reward model trained with a fixed total budget of gold preference labeling) enables the policy model to reach the target reward more efficient and effectively.
strength_weakness: strength_weakness: - The paper fails to explore when the budget on gold preference labeling is larger -- e.g., investigating whether having more initial gold preferences diminishes the benefit of subsequent iterative online training with silver preferences.
questions: questions: What seed prompts did you use for generating responses to label with the reward model in each self-training step?
paper_summary: paper_summary: While direct preference optimization (DPO) gains popularity, it is often performed on static preference data collected ahead of the training, limiting its theoretical capability. This paper proposes D2PO, an online version of DPO that annotates preference data on-the-fly during training through an iteratively updated reward model. To achieve this, D2PO requires periodically collecting gold preference labels and retraining the reward model. Under a fixed budget, the authors show that D2PO achieves the best performance over several baselines on several diverse tasks.
strength_weakness: strength_weakness: - Clarity should be improved.
- Claims are not always supported by the experimental results.
- One advantage of DPO is to drop the separate reward model which greatly simplifies the training pipeline. D2PO, however, brings it back and relies on iteratively updated reward modeling, making the whole pipeline more complicated.
questions: questions: * The “discriminator” in D2PO is actually the reward or preference model, but this term is very confusing in the context of LLM alignment.
* In Section 6.1, the authors argue that the updated reward model improves the reward accuracy. However,  the accuracy in Contrastive Distill doesn’t change that much comparing Figure 5 and 6. Could you please explain why D2PO still outperforms DPO w/ RM?
* In Figure 3 and 4, why does OPO w/ gold reward model underperforms D2PO? If D2PO benefits from improved reward accuracy, gold rewards always give perfect reward for the training, but OPO w/ gold reward performs even worse. 
* With a fixed budget, how does the iteration T_p affect the performance? Should we perform more iterations, each annotating a smaller amount of gold preference data, or fewer iterations?
* It’s not very clear how gold preference and silver preference data are used during the training, and why they have different amounts as shown in Table 1.
paper_summary: paper_summary: This paper propose D2PO, discriminator-guided DPO,
an approach for the online setting where preferences are being collected throughout learning. By collecting golden preference annotations during online sampling, D2PO simutenously trains the policy and the reward models.
strength_weakness: strength_weakness: - What is OPO and how is it implemented? OPO is not a well-established term in RLHF. I went through the two cited papers. Liu et al. use rejection sampling to collect preference pairs with the policy model, then apply DPO or SLiC loss for preference optimization. Lee et al. proposes a self-judgment approach where the reward model and policy model are initialized together. So it is not clear what OPO is referring to in this paper. From Figure 1, I guess OPO first trains an RM with offline preference data, then labels on-policy data with the static RM. However, it is necessary to clarify the loss function and specific implementation details.
- The efficiency issue. For real tasks, the cost of acquiring on-policy gold labels would be high (e.g. human annotation). So the feasibility of this method is questionable.
- The online RL methods such as PPO did not appear as a baseline, which also updated value models during training, but did not require gold labels.
- There are three synthetic experiments with only one real-world experiment. It would be better to test the method on more real benchmarks. For example, you can try math or coding problems where the gold labels are easy to get.

Overall, I think the main assumption of this work, better utilization of on-policy data, would make contribution to the field. I will be happy to increase my score once I see more convincing real-world experiments.

Comments: title: Continued Response
Reply
comment: Thanks for following up! We definitely agree that clarity of the original version needs improvement, and have been working on a much clearer version (especially on explanations of the algorithm and data condition) based on your feedback. 

> Section 6.1

We also found it interesting that contrastive distillation seems to improve despite having lower overall reward accuracy than other tasks. It’s worth noting that in standard RLHF, even on large datasets, RMs are known to have low (~60%) accuracy.  Our hypothesis is that the noisy features reward models learn do, in aggregate, correlate enough with the true objective for optimization to succeed, and the online setting helps this feature learning in aggregate. Referring to Figure 3 (comparable with Fig 6), note that the beginning and the end of policy training (where Fig 6 shows highest stable accuracy) is where policy improvement occurs the most, which makes sense.  

> T_p 

We can include more complete results in a future version. Some reference results (high T_p means more online): 

Word collector: 
- T_p=2000 ->  max R ~=20
- T_p=500 ->  max R ~=17.9
- T_p=8 ->  max R ~=15.3

Contrastive Distill: 
- T_p=400 ->  max R ~=0.7
- T_p=20 ->  max R ~=0.2

Nouns: 
- T_p=200 -> max R ~=49 (with 200 gold prefs)
- T_P=20 -> max R ~=49 (with 600 gold prefs)

 > OPO w/ gold

This is an important thing we’ve tried to improve clarity on in our new version. We can explain further: for example, in OPO w/ gold, we may sample 32 preferences in a step, and label these with the gold reward function. With D2PO, we also get 32 gold labels, and can update the policy on these, but since we have an RM, we can *also* generate 128 (or more) new outputs on fresh prompts, and get more preference labels using the RM, without needing any new gold labels. This thus gives DPO its advantage in gold preference data efficiency.
Thanks for the response!

Re "discriminator"

In my opinion, the term "reward or preference" model explains things well. I didn't see the necessity of using "discriminator" here.

Re "Section 6.1"

Without comparing to Figure 5, Figure 6, the Contrastive Distill particularly, doesn't show clear benefit from online updates. It would be great to explain why the policy model still improves in this condition.

Re "T_p"

Please include the corresponding experimental results.

Re "OPO w/ gold"

In the paper, the authors explain that OPO (gold): only online gold preference labels are used to update the policy. Doesn't OPO (gold) adopts perfect reward function during training since it's online? I may misunderstand gold/silver preference data, but as I mentioned before, clarity needs improved.

I prefer to keep my original judgement.
Thanks for your reply. I think the additional results and explanation are beneficial. I will maintain my score and lean positive to the paper.