Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

Authors: hamish  ivison
yizhong  wang
jiacheng  liu
zeqiu  wu
valentina  pyatkin
nathan  lambert
noah a smith
yejin  choi
hannaneh  hajishirzi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:12:43

Abstract: Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories.

Main paper: 
 introduction: 
 Modern language models (LMs) commonly undergo a final stage of training, called learning from preference feedback, 4 before deployment to end-users. This stage of training has been applied to many prominent language models like ChatGPT [45] , Llama 3 [35] , and Claude [4] , and has been shown to improve performance across a wide number of capabilities, including instruction following [22] , code [28] , math [54] , and summarisation [67] . Despite the widespread use and potential of this learning paradigm, applications of preference-based learning vary wildly, both in the data and learning algorithm used. As such, it is unclear what aspects of learning from preferences matter most for Figure 1 : Performance improvements resulted by changing different components in the preference training of TÜLU. Left: Accuracy on GSM [9] , for testing math capabilities. Right: Overall performance, aggregated over the 11 benchmarks described in §2.2. downstream model performance, particularly in relation to the two most popular preference-based learning algorithms, PPO and DPO, which take different approaches to preference-based learning.As seen in Figure 2 , both DPO and PPO rely on training on preference data-DPO for directly training the model, and PPO for training a reward model. In PPO, this reward model is then used to score generations from the main model generated using a set of policy training prompts (unlabelled prompts used for eliciting generations). This provides us with four important aspects of learning from preferences, including the choice of learning algorithm itself (PPO vs. DPO).In this work, we aim to systematically investigate these key components of learning from preferences, exploring the effect of varying each component on downstream model performance. Starting from an initial strong open supervised finetuning (SFT) model, we investigate each aspect in turn. As seen in Figure 1 , we find that all aspects are important for performance, albeit to varying degrees. Our findings are as follows:• When comparing 14 popular existing preference datasets across a diverse set of evaluations, we find that synthetic, diverse data annotated with per-aspect preferences works best for learning from preferences ( §3.1). We find that the quality of the preferences (choice of chosen/rejected pairs) matters more than the quality of the actual generations being considered. • PPO outperforms DPO across varied datasets in our evaluation suite, even when using exactly the same models and initial training data ( §3.2). • Increasing reward model size or dataset size used to train the reward model results in improved reward model performance on benchmarks directly testing reward model performance. Examining their effect on policy model performance when used during PPO training, we find that these improved reward models have a large effect on GSM performance, but give marginal to no improvements across all other evaluations considered ( §3.3). • Using unlabelled prompts that better match the test setting during policy can further improve model performance in domain-specific settings (e.g., when focusing on improving math performance), but has limited to no effect when targeting overall performance ( §3.4).Overall, we suggest a recipe for learning from preference feedback ( §4): using synthetic preference datasets and training using PPO with a large reward model performs best overall. Additionally, targeted prompts should be used if one only cares about a single particular downstream task. 
setup: 
 We first describe the core aspects of PPO and DPO before moving into our experimental setup. We summarise both approaches in Figure 2 .Figure 2 : The core aspects of learning from preference feedback. For DPO (solid line), preference data is directly used to train a policy model. For PPO (dashed line), preference data is used to train a reward model, which is then used to score model-generated responses during PPO training.PPO. PPO-based approaches for learning from preference feedback involve first training a reward model on preference data, and then training a policy model by scoring responses produced by the policy model itself during training ("online" learning) as shown in Fig. 2 . First, the policy model is prompted to generate responses, which are then scored with the reward model. These scores (i.e., scalar rewards) are then used to guide the optimization of the policy model, following the PPO algorithm. Additionally, a KL penalty term is applied to avoid model degeneration.• Preference data. A preference dataset D R is used for training a reward model, and it typically consists of prompts, responses, and rankings. Each prompt x comes with a pair of responses y c , y r , and a preference ranking between them (denoted as y c ≻ y r | x, where y c is the chosen response and y r is the rejected one). Both the responses and the rankings can be either humanannotated, generated/evaluated by other language models, or derived from examining relative votes on publicly posted online comments (e.g., comparing high and low-upvoted comments from Reddit). While recent work has relied much on synthetic, model-generated data [53, 22] , it is unclear if high-quality human data can provide similar or better performance, or if the significantly larger size of online-scraped datasets allows for improved performance. • Reward model. The reward model R ψ (x, y) is a scalar function, and can be parameterized with a transformer where a regression head replaces the language modeling head. The reward model is usually trained to minimize the following loss: L R (ψ) = -E (x,yc,yr)∼D R log σ R ψ (x, y c ) -R ψ (x, y r ) .(1) • Policy training prompts. The policy training data D π has a set of policy training prompts. Instead of the pre-generated responses, here for each prompt we sample a response y from the policy model being actively trained, y ∼ π θ (y|x). It is then scored by the reward model to get a sequence-level reward, r = R ψ (x, y). Intuitively, this suggests that more accurate rewards should improve model performance. Additionally, we note that most existing works typically use either the prompts used during reward training [66] or a generic pool of anticipated user queries [37] . We investigate if better targeting the prompts used during policy training can further improve performance. • Policy training. The goal of policy training is to maximize the reward of policy-generated responses, subject to a soft KL constraint that prevents degeneration: maxEQUATIONwhere π ref is the reference policy (usually the same SFT policy that initializes policy training). We find tuning the KL penalty coefficient β is important for performance and explore the effect of varying it in App. I. Since directly optimizing Eq. 2 can be unstable, it is common to reformulate language generation as a Markov decision process (MDP) and optimize using an RL algorithm. PPO is one of the most widely adopted RL algorithms for this problem. See App. F.2 for additional details about PPO.DPO. DPO is an offline RL approach for performing learning from preference feedback. It allows one to directly optimize the policy on preference data, without building reward models or needing to sample online from the active policy. As such, DPO is an offline training algorithm. In simple terms, DPO aims at increasing the margin between the log-likelihood of the chosen responses and the log-likelihood of the rejected ones, while ensuring the model does not stray far from the initial policy.• Preference data. The structure of preference data is identical to that of PPO.• Policy training. By following a closed-form solution to Equation 2, DPO re-writes any reward model R ψ in terms of its corresponding optimal policy π θ * (ψ) , R ψ (x, y) = β logπ θ * (ψ) (y|x)πref(y|x) + β log Z(x), where Z(x) is a partition function. Consequently, the policy model can be trained by directly optimizing the reward objective in Eq. 1, and hence the DPO loss:EQUATIONComparing DPO and PPO. Compared with PPO, DPO is more efficient in terms of compute, speed, and engineering efforts. DPO does not need the extra stage of training a reward model, and during policy training it does not need to decode online responses (which is usually slow) or train an additional value model. Meanwhile, PPO trains on online data generated by the current policy, while DPO trains on static, pre-generated offline data. This may limit exploration in DPO and thus hurt the training quality. While concurrent work has compared DPO and PPO [60, 48] , comparisons are typically limited to constrained domains and evaluations, using ground-truth rewards [60] or primarily examine smaller synthetic settings [48] . We complement such studies by comparing the downstream performance of models trained with DPO and PPO across a wide variety of datasets and evaluations and consider additional potential factors in PPO performance, such as improved reward models and policy training prompts.We base our exploration off TÜLU 2 13B [22] , a popular openly released model. TÜLU 2 is a series of Llama 2 [52] finetunes across all sizes with publicly released weights and data, the largest of which achieved state-of-the-art performance on AlpacaEval and Chatbot Arena. As such, we are curious how much further we can push the performance of TÜLU 2 models through exploring alternative datasets, training algorithms, reward models, and policy training prompts. We use this model as a base policy when training policy and reward models, following Ouyang et al. [38] for PPO and Rafailov et al. [39] for DPO. We provide additional details for each in App. F.We extend the TÜLU [55] evaluation, aiming to cover a diverse range of skills and behaviours. We evaluate models on 11 different benchmarks, covering skills such as factuality (MMLU [20] ), reasoning (GSM8k [9] , Big Bench Hard [5, 47] ), truthfulness (TruthfulQA [29] ), coding (HumanEval+ [6, 32] , MBPP+ [2, 32] ), safety (ToxiGen [19] , XSTest [42] ), and instruction following (AlpacaEval 1 and 2 [27, 13] , IFEval [65] ). We report the per-category average of evaluations and the overall average across categories. We provide further details in App. D. 
exploring learning from preference feedback: 
 We now explore each aspect of learning from preferences: preference data, learning algorithm, reward models, and finally policy training prompts.We compare the performance of models trained with DPO across 14 different preference datasets in Table 1 . We choose datasets that represent various potential sources of data: human-annotation (HH-RLHF [4] , HelpSteer [56] , Chatbot Arena 2023 [64] and 2024 [7] , AlpacaFarm Human [14] , PRM800k [28] ), web-scraping (SHP-2 [15] , StackExchange [25] ), and synthetic generation (Ultra-Feedback [11] , Nectar [66] , Orca [34] , Capybara [12] , AlpacaFarm GPT-4 [14] ). For UltraFeedback, we consider both using the 'overall' score provided in the dataset and taking an average of the per-aspect scores ('fine-grained'). We provide further detail on each dataset in App. B. We find that:Preference-based learning with existing datasets has the strongest effect on instruction following and truthfulness performance. Our best models improve on the SFT model by over 8 points in these categories. In contrast, preference-based learning does not aid factuality, with all models remaining with 1 point of each other. This suggests that when using existing publically-available datasets, preference-based learning is most useful for improving chat-related abilities (instruction following, truthfulness) and learning stylistic features, but less strong at teaching new facts to a Synthetic data with per-aspect annotations performs best. Synthetic datasets generally outperform human-annotated and web-scraped datasets, especially in truthfulness. Additionally, using datasets collected by first getting per-aspect annotations (i.e., annotations from a human or model that score the helpfulness, harmlessness, etc. of the response independently) and then averaging across these scores tend to outperform datasets that rely only on overall judgements (i.e., just asking the annotator for an overall score instead of a per-aspect score). The two datasets that use this method, HelpSteer and UltraFeedback, display stronger or similar performance to datasets up to 15 times larger (e.g. HelpSteer vs HH-RLHF). We investigate the performance of varied sub-splits of UltraFeedback in App. E, which suggests that the use of per-aspect annotations is more important for performance than the quality of the models used to generate completions for the dataset. Table 3 : Reward model evaluation: (a) Direct evaluation: reward models when directly evaluated using RewardBench [26] and Best-of-N (left two columns); for BoN, we report both raw average performance, and the difference in performance over the base SFT model in brackets. (b) Downstream evaluation: models trained using PPO and the given reward model (right three columns). We report GSM and AlpacaEval 2 performance along with average performance across the entire evaluation suite defined in §2.2. Directly comparing reward models indicate increasing scale and data improves reward models, but these only minimally impact downstream performance.We now compare algorithms for learning from preferences, comparing the performance of DPO and PPO when the same base models and data are used (Table 2 ). We use exactly the same data for training DPO and PPO models, 5 PPO improves on DPO in reasoning, coding and safety capabilities most. PPO improves over DPO by an average of 1.3, 2.9, and 2.3 points for reasoning, coding, and safety respectively, while truthfulness tends to degrade by an average of 2.5 points. Instruction following and factuality remain largely the same. Interestingly, we find that models trained with PPO are far more likely than DPO-trained models to perform chain-of-thought reasoning when prompted with reasoning or math problems, even when not given in-context examples using chain-of-thought. This suggests that reasoning improvements from PPO may be due to increased chain-of-thought abilities. Additionally, while overall instruction following ability remains similar, we find that PPO-trained models tend to perform better at AlpacaEval 1 and 2, with PPO-trained models outperforming DPO-trained ones on AlpacaEval 2 by an average of 3.4 points.Next, we focus on PPO and study reward models both directly, and on downstream tasks with PPO training (Table 3 ). We first consider two ways to improve a reward model:1. Scaling up the training data for the reward model. We construct a data mixture of the topperforming datasets in Table 1 from each section: UltraFeedback, HelpSteer, Nectar, StackExchange, HH-RLHF, and additionally add PRM800k for math data. We compare reward models trained on this data mixture (called Mix RM) with reward models trained only on UltraFeedback (UltraF. RM) -the top-performing dataset from prior sections. 2. Scaling up the reward model size. We train reward models at two scales of 13B and 70B starting from TÜLU 2.Direct evaluation of reward models. To isolate the performance of the differing reward models, we first evaluate them with best-of-N (BoN): we sample 16 generations from TÜLU 2 13B SFT, score them using the given reward model, and then use the top-scoring output as the model output. Notably, we ensure model outputs are identical between runs, meaning that the only difference is the reward model scores. We perform evaluation on the subset of evaluations in our suite that require long-form generation, 7 and report overall average performance. We additionally evaluate our reward models on RewardBench [26] , a popular evaluation for reward models, which involves evaluating if the relative scores given by reward models match a test set of chosen-rejected pairs from diverse sources. We provide further details in Appendix H. 13 , the largest gaps between the 13B Mix RM and the 70B Mix RM is in reasoning (mostly code), suggesting that the larger model may not benefit much from the additional (somewhat noisy) data 8 .Downstream evaluation of reward models. We then test if our improved reward models lead to improved downstream policy models when used during PPO training. We perform PPO training using the UltraFeedback prompts during training and report our results on the right side of Table 3 . Surprisingly, we find that improvements in reward models result in surprisingly small improvements in overall downstream performance. We see the largest (and only) overall improvement when using the 70B UltraFeedback RM, despite the fact that all improved RMs performed significantly better than the 13B UltraFeedback RM in RewardBench and Best-of-N evaluations. Additionally, the improvement from the 70B RM is largely driven by a large performance jump in GSM, as shown in Table 3 , while other metrics stay largely similar. This suggests that it is difficult to translate improvements in reward models to the underlying policy. We note that most prior work examining reward models tends to examine either direct reward model performance [26, 61] or proxy reward [17, 40] , rather than downstream performance. Our findings suggest that improving such evaluations may not necessarily translate to improved downstream performance. Additionally, we find that using the larger 70B UltraF. RM is less sensitive and performs well with lower KL penalty coefficient values than using the 13B UltraF. RM, and further examine the effect of the KL penalty coefficient on performance in App. I.While it may seem unintuitive that an extreme increase in reward model size does not lead to extreme improvements in performance, we note that prior work has similarly noted that much larger reward models do not necessarily lead to significant improvements in performance, and smaller reward models can lead to similar performance (Ouyang et al. [37] §C.2, Wang et al. [56] §4.3), although we are the first, to our knowledge, to ablate this and explicitly report results on downstream evaluations.We additionally explore how further training and potentially overoptimizing against the reward model affects performance across different evaluations in App. L. Importantly, we find that different evaluations show different trends -while some evaluations such as AlpacaEval benefit from continued training, other evaluations such as IFEval or GSM8k drop with continued training. This highlights the importance of evaluating over a diverse set of test tasks, including both 'traditional' benchmarks and LLM-as-a-judge evaluations.We finally examine the effect of using varied policy training prompts, first when targeting a particular capability (math performance as evaluated by GSM), and then for improving overall performance. This is in contrast to prior work that just re-use the prompts used for reward model training [66] or a generic pool of anticipated user queries [37] . We first examine the effect of using policy training prompts targeted towards a particular downstream setting-specifically, math capabilities as evaluated by GSM. We construct three different prompt sets: prompts sourced from the GSM train set, math-related prompts from varied datasets, and 20K random prompts from UltraFeedback 9 . We further detail the approach used to identify math-related prompts in Appendix J. We then train models using PPO with each reward model from the previous section on each prompt set. Results in Fig. 3 demonstrate that:Larger reward models perform better when closely matching train prompts to test settings. When using prompts from the GSM train set, we find using either 70B reward models during training leads to significant improvements in GSM, with our best performing model improving over the SFT model by 16 points (46%→62%).Weaker reward models struggle to make use of prompts differing from those they are trained on. Surprisingly, we find that training with the 13B UltraFeedback reward model actually performs worse when using prompts apart from UltraFeedback, potentially due to the reward model generalising poorly to prompts not seen during training. Similarly, the 13B Mix and 70B UltraFeedback reward models struggle to make use of math prompts, which are out-of-domain for the reward models, but in-domain for the task.Overall, these results suggest that targeted prompt distributions can improve performance when coupled with powerful reward models (e.g., using prompts from the GSM train set and then evaluating on GSM). This highlights a strength of PPO: it can make use of unlabelled prompts to better target downstream tasks.Inspired by the success of the targeted prompts, we construct a new remixed prompt set by finding 20K math and 20K code-related prompts using the same method as in the previous subsection (see App. J). We then combine the math, code, and UltraFeedback prompts to create a larger prompt pool that we downsample randomly to 60,908 prompts. This rebalances the prompt pool to focus more on coding and math tasks, which we hope yields improvements on math and code respectively while maintaining overall performance. We present our results in Table 4 .Surprisingly, we observe that using mixed prompts does not seem to improve performance in the generalist setting. When looking at code and math results specifically, we do not see consistent improvements using the altered prompt mixture, and our overall performance tends to drop. This is likely due to the already diverse nature of UltraFeedback, such that when looking at the whole dataset (i.e., not just the 20K subset in Figure 3 ), we are able to reach strong performance on math and coding evaluations. Altering the distribution away from other tasks slightly hurts the overall performance. We additionally found that training all the mined prompts in additional to all of UltraFeedback (i.e., not downsampling the combined prompt set) did not yield further improvements over the results shown in Table 4 . 
a recipe for learning from preferences: 
 Putting together all our findings from previous sections, we suggest a recipe for training a strong model using learning from preferences, as shown in Figure 1 and in Table 5 . We take a highquality, synthetic preference dataset, a large reward model, 10 and train it using PPO. If we additionally wish to focus on a specific domain, we can additionally collect domain-specific prompts for policy training. We find that our best model (TÜLU 2+PPO 13B trained with the 70B UltraF. RM) outperforms our best DPO model and other popular models based off Llama 2 13B, including Llama 2 Chat, which has also undergone SFT and PPO training. Additionally, incorporating task-specific prompts into policy training may further improve performance when the prompts align closely with downstream evaluations, as shown in Figure 3 . Finally, we also experiment with Llama 3.0 8B [35] , finetuning on the TÜLU2 Mix, and then training it using DPO and PPO with the same hyperparameters. We find that overall performance is significantly improved, and we observe similar trends as with our other experiments (DPO performing better than PPO, using a larger reward model improving performance, using mixed prompts not improving performance).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Best efforts have been made to make sure the results are reproducible, and we will provide code and model artifacts for all models discussed in the paper after the review period. All base datasets and models used in this paper are publicly available. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
related work: 
 Learning from Preferences for LMs.Initial approaches to learning from preferences used reinforcement learning from human feedback (RLHF) [8, 67] , a method that first trains a reward model to capture human preferences and then optimizes against it using reinforcement learning algorithms such as PPO [44] . Recent work has additionally questioned the need for PPO, and has found that similar but simpler approaches such as REINFORCE [46] with LM-specific adjustments work well [1, 50] . Unlike prior work, we instead focus on examining the effect of varying the data and models used in PPO (i.e., the reward model, preference data, initial policy model, and prompts used for sampling outputs). We believe that our results should transfer to similar approaches such as REINFORCE, since they share many commonalities with PPO (e.g., reliance on a well-tuned reward model and an unlabelled prompt set for eliciting generations during training).Another line of recent work has also attempted to simplify the PPO algorithm and remove the online generation component, with the most popular algorithm following this approach being DPO [39] . DPO's ease of implementation and use has made it widely popular among open-source community. Notably, several high-performing models have been trained using DPO for learning from preferences, including TÜLU 2 [22] , Zephyr [53] , and Mixtral-Instruct [24] . Much recent work [3, 21, 59, 16, inter alia] has attempted to further improve the DPO algorithm. However, comparisons of these approaches so far have found that they largely perform similarly [41, 43] . As such, in this work we focus on the most popular variant, DPO, and examine what data works best for it and how it compares to a popular online RL approach, PPO.Comparing Approaches for Learning from Preferences. Recent concurrent work has compared the properties and performance of DPO, PPO, and other approaches for learning from preference feedback. Xu et al. [60] suggest DPO performs poorly when using data out-of-distribution from the initial base model, while PPO (and a semi-online DPO variant) perform better both in such cases and overall when evaluated on safety and code capabilities. However, they do not investigate the impact of reward models and focus on core algorithmic details of PPO that lead to improvements. Tajwar et al. [48] identify on-policy sampling and negative gradients as two important aspects of preference-based learning when optimal reward values do not lie 'close' to the base model's distribution and the preference data is skewed. Tang et al. [49] study how the IPO [3] algorithm performs in the static offline setting versus various ways of updating or ordering the data in an online manner. In this work, we focus on empirically examining the impact of core aspects of learning from preference feedback, including the effects of varied rewards and data. 
conclusion: 
 In this work, we have systematically explored the core components of learning from preference feedback and examined the relative impacts of each in turn on model performance across a wide range of evaluations. Our results suggest the following ordering of importance: preference data quality, algorithm choice, reward model quality, and finally targeted policy training prompts. Additionally, we find that using larger reward models can significantly improve math capabilities, but have marginal effects on other capabilities we evaluate in this work. Overall, we suggest a recipe for learning from preference feedback with currently available resources: best performance can be achieved by using a strong synthetic dataset (UltraFeedback), and using PPO training with a large reward model. Our work suggests that further exploring how to make better use of improved reward models is an important direction for further improving PPO-style approaches to learning from preference data. We plan to release models and code related to this paper and hope that our settings provide a firm base for future work further exploring learning from preferences for language models. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report results from single runs due the computational cost of experiments, and make sure we word our claims accordingly. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Dataset details (including citations and licenses) are provided in Appendix B.We provide model and code details in Appendix C. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] We provide links to released assets and code in our abstract, and the assets themselves provide details about the hyperparameters, datasets, etc. used to produce them, and further documentation about the released assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Limitations. As an empirical study with limited compute, our results are largely based on an indepth examination over a single model suite (TÜLU 2), using two base models (Llama 2 and 3). Additionally, our evaluation may not reflect all potential downstream use-cases -for example, we do not test multilingual performance. As such, examining how our results carry to multilingual settings or greatly differing base models is interesting future work. However, we have done our best to explore a wide variety of evaluations and datasets, using publicly available resources to aid in reproducibility.In terms of computational efficiency, we note that, while PPO does well, it comes with a significantly increased computational cost compared to DPO. We only consider performance in this study and don't explicitly measure the relative computational cost of different methods (e.g., by measuring FLOPs cost of training with DPO vs PPO). There are also other aspects to keep improving PPO (e.g., doing new rounds of preference annotations to update the reward model for the changing distributions of the policy model), which we cannot exhaust in this paper, and we leave the exploration of them for future work.Broader Impacts. Language models have recently been deployed extensively, but there are few public studies on the impact of different algorithms for learning from preference feedback and datasets on these models. We hope to shed light on the impact of this stage of LM training and aid in improving future LMs. We explicitly consider safety evaluations as part of our evaluation and hope that our findings help inform how to improve the safety of future LMs, limiting potential harms and enhancing potential benefits. However, we note that focussing too much on learning from single preference datasets may result in LMs being aligned to only relatively small subsections of the global population.We examine the following datasets in this work. We provide details on where we source the data and the license associated with each dataset.• SHP-2 [15] : We use the publically available HuggingFace train split, and randomly downsample to 500,000 samples. The stackexchange portion of the dataset is licensed under the CC BY-SA license, and the reddit portion made available in accordance with the Reddit API terms of use. See the HuggingFace dataset card for more details. • StackExchange [25] : We use the publically available HuggingFace train split, and randomly downsample to 500,000 samples. The dataset is licensed under the CC by-SA license. • PRM800k [28] : We use the data from the second phase of collection. We consider prompts where the model generations achieved the correct answer once and failed to find the right answer once. We then randomly choose one correct and one incorrect generation as chosen and rejected respectively. The dataset is licensed under the MIT license. • Chatbot Arena Conversations (Chatbot Arena 2023) [64] : We use the publically available HuggingFace train split at https://huggingface.co/datasets/lmsys/chatbot_ arena_conversations. We remove all multi-turn samples as these diverge after the first turn (but users still rate the entire conversation), and filter out ties. The prompts are licensed under CC-BY-4.0 and the outputs under CC-BY-NC-4.0. • Chatbot Arena Preferences (Chatbot Arena 2024) [7] : We use the publically available HuggingFace train split at https://huggingface.co/datasets/lmsys/ lmsys-arena-human-preference-55k. We remove all multi-turn samples as these diverge after the first turn (but users still rate the entire conversation), and filter out ties. The dataset is licensed under the Apache 2.0 license. • AlpacaFarm Human and GPT-4 Preferences [14] : We use the publicly available huggingface 'preference' splits for each dataset. The dataset is licensed under CC-BY-NC-4.0. • HH-RLHF [4] : We use the publically available HuggingFace train split. The dataset is licensed under the MIT license. • HelpSteer [56] : We use the publically available HuggingFace train split. We average the fine-grained scores (except verbosity) for an overall score to decide chosen and rejected pairs. The dataset is licensed under the CC-BY-4.0 license. where the chosen and rejected samples are chosen using the overall score (Overall/OV) and one where they are chosen using an average of the fine-grained scores (Fine-grained/FG).The dataset is licensed under the MIT license.We additionally create splits of HH-RLHF, StackExchange, and Nectar downsampled to 60,908 (matching the size of our UltraFeedback split) examples for size-equal comparisons of algorithms across different dataset types (i.e., Table 2 ). We also filter out any malformed examples we find 13 .We additionally construct our 'mix' used for expanding the reward model training by a number of examples from each dataset as listed in Table 6 . Overall, our 'mix' prefence set contains roughly 260k samples, over four times larger than just using UltraFeedback.We primarily build off TÜLU 2 13B and 70B [22] , which themselves are based on Llama 2 [52] . TÜLU 2 models are licensed under the AI2 low-risk license and available at https://huggingface.co/ collections/allenai/tulu-v2-suite-6551b56e743e6349aab45101. Llama 2 is licensed under a custom license and available at https://huggingface.co/collections/meta-llama/ llama-2-family-661da1f90a9d678b6f55773b.For code, we build off EasyLM [18] , specifically the fork used to train TÜLU 2 [23] , which is licensed under Apache 2.We use the following evaluations. We provide the category we map each evaluation to in brackets.• MMLU (factuality) [20] : We use the official MMLU evaluation script and prompts available at https://github.com/hendrycks/test, with modifications to allow for batch processing. We evaluate using 0 few-shot examples, following the original setup of MMLU. We report average accuracy across test examples. • GSM8k (reasoning) [9] : We evaluate models on the test set of GSM. Following Wei et al.[57], we evaluate with chain-of-thought. We use 8 few-shot in-context examples. Because all answers in GSM are numbers, we extract the last number in the model response as the final answer. We report average accuracy across test examples.• Big Bench Hard (BBH; reasoning) [5, 47] : We follow the setup described in the original paper and evaluate with chain-of-thought. The officially provided prompts, which have 3 few-shot in-context examples are used. For the CoT setup, we extract the first word after the phrase 'So the answer is', or the entire response if there is no such substring present. We report average accuracy over sub-tasks (all of which use accuracy as the primary metric). • TruthfulQA (truthfulness) [29] : Following Touvron et al. [52] , we mainly use the generation setting. We follow the official script in their official implementation 14 to do greedy decoding and answer postprocessing. We also follow their instruction to train two GPT-based classifiers to judge the truthfulness and informativeness of the model response. We report the % Informative and Truthful. • AlpacaEval (instruction following) [27, 13] : We use the package provided by Li et al. [27] ,following the default setup for both AlpacaEval 1 and length-controlled AlpacaEval 2 [13] .We allow the evaluated model to generate up to 8192 tokens, without specifying special stop sequences. • IFEval (instruction following) [65] : IFEval benchmarks whether models can follow instructions that contain verifiable constraints, such as "write in more than 400 words". We use the official evaluation code released with the original paper 15 , and report the "Loose Accuracy" metric at the prompt level (i.e., a response is counted as correct only if all the constraints in the prompt are detected to be satisfied after normalizing the response). • HumanEval+ (coding) [6, 32] : We use the augmented form of the HumanEval dataset introduced by Liu et al. [32] , which contains additional test cases. We additionally use the instructions provided by HumanEvalPack [36] when prompting instruction-tuned models. We report pass@10 and sample with a temperature of 0.8. We use v0.2.1 of the released Eval+ package. • MBPP+ (coding) [2, 32] : For coding, we also evaluate using MBPP+, for which we prompt models to complete a program program given a natural language description and function signature. Similar to HumanEval+, We report pass@10, sampling with a temperature of 0.8. We use the v0.2.1 of the released Eval+ package. • ToxiGen (safety) [19] : We follow the setup in Touvron et al. [52] , but use the original set of prompts from Hartvigsen et al. [19] , which are designed to elicit toxic generations for certain groups. We take only the prompts designed to produce toxic language ('hateful' prompts) and use 500 prompts per group to reduce evaluation costs. For base language models, we pass in the original ToxiGen prompts unchanged and greedily decode up to the first new line (or a maximum of 512 tokens). For instruction-tuned models, we place the prompt in the corresponding template, and ask the model to complete the prompt, until the model generates a stop token (or a maximum of 512 tokens). We pass the generated text into a roberta-large model trained to detect toxic content finetuned as part of Hartvigsen et al. [19] (available at https://huggingface.co/tomh/toxigen_roberta). We then report the percentage of generations deemed toxic by the classifier. • XSTest (safety) [42] : XSTest evaluates whether a model can correctly refuse to follow malicious instructions, while still being able to correctly identify similar but safe instructions and follow those ones correctly. We use the same set of test prompts provided in their official repository 16 , which includes a mixture of 200 unsafe prompts and 250 safe prompts. Similar to the original paper, we tried using both GPT4 and heuristic rules to detect whether the model responses are refusal. We found that GPT4 provides more reliable detection, as modern models may respond in various ways that exceed the coverage of those pre-defined rules. For GPT4-based detection, we found the originally provided prompt in the official repository has a confusing definition of "full refusal" and "partial refusal", so we removed that distinction and rewrote the prompt. In this paper, we report the F1 metric, which aggregates the precision and recall of model refusals. We will release all our evaluation codes for reproduction. Intrigued by the strong performance of UltraFeedback, the overall best performing dataset, we further compare the performance of differing subsets of UltraFeedback, and show our results in Table 7 . First, we compare using only generations from the highest-scoring, middle-scoring, and lowest-scoring models (as judged by GPT-4). Surprisingly, we find that there is little difference in overall performance between these splits. This suggests the preference annotations from GPT-4 (i.e., the decisions about chosen and rejected pairs) are more important than the strength of the underlying models used to sample responses. Then, we compare models only trained on samples from the same original dataset (as UltraFeedback was constructed by sampling prompts from various sources). We find that each prompt source performs best in at least one category. This suggests that selecting a diverse set of prompts when generating synthetic data is important, and UltraFeedback already contains a diverse set of prompts covering various downstream applications.Subset Construction Details. We construct the 'top', 'middle', and 'bottom' sets of UltraFeedback by using the average score of responses from each model according to the average of fine-grained scores. We then bucket the models into three groups accordingly. We provide the groups and average scores in Table 8 . For constructing the splits, we then filter UltraFeedback to only include responses from the given models, pick the highest-scoring response per prompt as the chosen, and a random lower-scoring prompt as rejected. We construct a subset of 10,000 prompts for each group of models.For the prompt subsets, we simply use source dataset annotations provided in the UltraFeedback data itself.We performed hyperparameter search for both DPO and PPO, sweeping across core hyperparameters in pilot experiments on HH-RLHF and UltraFeedback. We provide further details for both algorithms below.We testing multiple values for β (0.1, 0.01, 0.001) and for learning rate (5e-6, 5e-7, 5e-8) in pilot experiments on HH-RLHF and UltraFeedback. Ultimately, we found that the hyperparameters suggested by Tunstall et al. [53] and followed by Ivison et al. [22] worked best (learning rate of 5e-7, β of 0.01). This additionally involves 3 epochs of training with a learning rate of 5 × 10 More on the algorithm. PPO formulates language generation as a Markov decision process (MDP), where each response y is an episode, an action is a token y t , and a state is the concatenation of a prompt and a partial response (x • y <t ). To construct token-level rewards r t that guide the RL training, the sequence-level reward r is applied only to the last token, and each token is penalized by a KL term -β log π θ (y t | x • y <t ) -log π ref (y t | x • y <t ) . Note that π θ is the model we are training and π ref is a reference model (in our case, the initial SFT model we start training from). Formally, the token-level reward r t for each response token y t is defined asEQUATIONIn addition to the policy model, PPO also trains a value model V ϕ (x • y <t ) that estimates the expected value function of incomplete responses under the active policy. The value model typically shares the same architecture as the reward model, while the regression head can be applied on any response token. The value model helps with estimating the advantage on each token,A t = -V ϕ (x • y <t ) + G t , where G t = |y| t ′ =t γ t ′ -t r t ′is the empirical return. 18 PPO trains the policy model by minimizing loss 19L π (θ) = -E x∈Dπ,y∼π θ (y|x),t∈ [1,|y|] π θ (yEQUATIONand trains the value model by minimizing the MSE loss against the empirical return:EQUATION)The models can be jointly optimized by linearly combining the two losses:L PPO (θ, ϕ) = L π (θ) + α • L V (ϕ).Implementation details. PPO comes with many implementation details. We made a simplistic implementation that results in stable training. Notably:• We initialize the value model from the reward model. This follows from InstructGPT [37] . Some other implementations initialize from the SFT model [33] or the base model [30, 58] , while replacing the LM head with a regression head. • For truncated completions, we set the reward to a large negative number (e.g., -10.0), which is referred to as the EOS trick. • We do not perform normalization on the rewards. This follows from AlpacaFarm [14] .Although reward models trained under different settings can have very different output ranges, we found our PPO experiments quite robust to such variation. • We do not whiten the step-level rewards within each batch. We do whiten the step-level advantages within each batch, following other implementations. • We use a fixed KL penalty coefficient. The original PPO algorithm [44] has an adaptive KL controller, but most recent implementations have moved away from this [37, 52] .See Table 10 for a comparison with other open-source implementations.Hyperparameters. The prompt and continuation each has at most 1024 tokens. We use a batch size of 64 and the same for minibatch size, and train for 1 epoch over the prompt dataset. On each batch of prompts and rollouts, we train for 1 inner epoch. We use a sampling temperature τ = 0.7 for rollout, and a fixed KL penalty coefficient β = 0.05. We optimize with AdamW with learning rate η = 1 × 10 -6 with a linear warmup for 10% of iterations. We provide additional details on the hyperparameters used for PPO in Table 9 . In experiments, we found that training with larger (70B) reward models benefited from a reduced KL penalty coefficient β, but using smaller reward models did not benefit (see App I for further discussion). As such, for runs using larger reward models, we use a KL penalty of β = 0.0325 instead of β = 0.05. This is in line with prior work suggesting that improved reward models are more difficult to overoptimize against, and so can benefit from lower KL penalties [17, 40] . We found these hyperparameters through experimenting in pilot experiments over HH-RLHF and UltraFeedback. We experimented with KL penalty coefficients of 0.01, 0.025, 0.0325, 0.05, as well as varying the batch size (including taking more samples per gradient step, and taking multiple steps over the same generations). We additionally explored training for up to 3 epochs and found that training beyond 1 epoch did not yield improved performance and occasionally resulted in the training collapsing entirely, and as such stuck to training for only 1 epoch.We additionally provide a more detailed comparison of our PPO implementation to other work in Table 10 .For training the reward model, we follow prior work [11, 37] in only training the reward model for one epoch, using a learning rate of 1 × 10 -5 that warms up for the first 3% of training steps and linearly decays to 1 × 10 -6 by the end of training. We use a batch size of 512.Choice of hyperparameters. We found that training beyond one epoch did not yield improved performance and occasionally resulted in the training collapsing entirely, and thus only trained for 1 epoch in our experiments (E = 1 in Table 9 ). We also found that training multiple inner epochs on each batch destabilizes PPO training, and thus we use 1 inner epoch (e = 1).Accelerating training with larger prompt batch size. Rollout (i.e., generating online responses from the policy) is the most time-consuming step in PPO, taking up more than 95% of the total training time. We explored speeding up training by increasing the prompt batch size from 64 to 512, such that we can obtain more responses with a small latency overhead. We kept the minibatch size at 64 due to memory constraint, and because we need about 900 gradient steps to make the training converge.This setup implies that rollouts in later minibatches become slightly off-policy when the model trains on them. In most implementations of PPO, the forward passes are also done on the batch level and Table 9 : PPO hyperparameters. We use values listed here unless otherwise noted.Quark Rainier/Crystal FG-RLHF AlpacaFarm OursInit value from reward ✗ ✗ ✗ ✓ ✓ EOS trick ✗ ✗ ✗ ✗ ✓ Reward normalization ✓ ✓ ✓ ✗ ✗ Reward whitening ✓ ✓ ✓ ✓ ✗ Advantage whitening ✓ ✓ ✓ ✓ ✓ Adaptive KL controller ✓ ✗ ✗ ✗ ✗ KL clamping ✗ ✗ ✗ ✓ ✗ Multiple rollouts per prompt ✗ ✗ ✓ ✗ ✗Table 10 : Variations in implementation of the PPO algorithm. Compared with serveral open-source PPO implementations: Quark [33] , Rainier [30] , Crystal [31] , FG-RLHF [58] , and AlpacaFarm [14] .thus the resulting logprobs and values are also obtained from a slightly stale version of the models. In our experiments, this severely destabilized training, and a closer investigation shows that the value model loss did not converge fast enough to supply policy training with reliable advantage signals. As a remedy, we made the forward passes to be carried out on the minibatch level so that these logprobs and values are obtained from the most current models. Note that this slightly deviates from most PPO implementations.In Table 11 , we compare the training time and performance of our default PPO setup and the larger prompt batch size setup. Increasing the prompt batch size by 8x reduces the training time by 5x (60 hours → 12 hours), while also decreasing the overall performance by 0.6% (61.7% → 61.1%). To remedy the performance loss, we experimented with generating multiple (up to 4) rollouts for each prompt, which increased the effective batch size for gradient updates while keeping the total number of gradient steps fixed. When using a rollout multiplier of 4, most performance loss can be recovered, while the training speedup is also less dramatic.Our conclusion from this set of experiments is that, it is important for the forward passes to be performed online, and to get the extra mile offered by PPO, the rollouts should also be generated fully online. Deviating from this may speed up training, but at some performance cost. Since we Best-of-N Details For best-of-N, we sample 16 responses from TÜLU 2 13B with a temperature of 0.7 for each evaluation task we examine. We then pass these responses (along with the prompt used for generation) into the given reward model, and use the top-scoring response as the final output.Given a list of these top-scoring outputs, we then pass these to the rest of the given evaluation setup. We examine only a subset of evaluations, focussing on the evaluations that rely on long-form model generations (as we are most interested in the reward model's ability to judge these outputs during PPO training). In particular, we look at GSM8k, BBH, Codex-Eval+, MBPP+, AlpacaEval 1 and 2, IFEval, XStest. For Codex-Eval+ and MBPP+, we use Pass@1 instead and just evaluate the top-scoring example (unlike other tables in which we use Pass@10). When reporting the average score in Table 3 , we first calculate an average score per evaluation task category (following the same evaluation categories as in App. D), and then report the average over categories.We report the full best-of-N results across each evaluation in Table 12 . RewardBench Details We report the subset scores of our reward models on RewardBench in Table 13 .ChatAs seen in Figure 4 , we observe that the optimal KL penalty coefficient (β in Eq. 2) value changes with the reward model used. While using the smaller 13B reward model results in large drops in overall performance as β shrinks, the larger 70B RM is more robust and actually achieves higher performance at a smaller β than the 13B UltraF. RM. This is in line with prior work suggesting that larger reward models are less prone to overoptimization [17, 10] , and as such are more robust to lower β (which encourages more optimization against the RM). This highlights an additional potential benefit of larger RMs: they may be easier to tune due to their increased robustness to choices of β.We also note that some evaluations do not suffer as much from reduced β. For example, AlpacaEval 2 performance (Fig. 4c ) actually is highest at the lowest β shown. This is likely due to the match between AlpacaEval and PPO training: as the reward models are trained on GPT-4 preferences, more closely optimizing against them (via a lower β) also improves the rate at which the model generations are preferred by GPT-4. However, it also comes with significant reductions in other evaluations (as seen in the drop in GSM performance in Fig 4) , which is undesirable when training a generalist multi-task model. This is in line with prior work observing that there appears to be an 'alignment tax' (i.e., reduced scores on various capabilities including reasoning and math) when performing learning from preference feedback [37, 4] , and that this tax can be partially controlled through tuning β (e.g., Fig. 34 in Ouyang et al. [37] ).In order to find additional unlabelled prompts, we mine prompts from UltraFeedback [11] , Wild-Chat [62] , and LMSYS-1M [63] . We only consider the prompts and throw away the accompanying model responses. We categorise unlabelled prompts by prompting TÜLU 2 70B. We first prompt TÜLU 2 70B to tag the prompts with various categories using the prompt shown in Figure 5 . We then sample Code and Math prompts by picking prompts that include 'Coding' or 'Math' tags and as few other tags as possible. We remove all prompts with 'unclear' and 'multilingual' tags as we wish to focus on clear queries made in English. The authors examined a small portion of the mined prompts by hand and found this resulted in a solid collection of high-quality prompts for math and code respectively.Please label the task tags for the user query. We train all models on 256 or 128-size TPUv3 pods. PPO training with a 13B reward model and 13B policy on the UltraFeedback set takes 3 days on a 256-size pod, with the reward model training taking roughly 10 hours to train. DPO training on UltraFeedback (roughly 60,000 samples) for a 13B model takes 9 hours. We note that we conducted additional smaller-size runs that we do not report results from in this paper as part of our research to further explore dataset and hyperparameter choices.We investigate how the performance of our best model (PPO training TÜLU 2 13B with the 70B UltraFeedback RM and UltraFeedback prompts), and continue training beyond the 1 epoch result reported in the main text of the paper. We show these evaluations in Figure 6 . We find that how performance changes over training is quite distinct between each evaluation: while some evaluations continuously improve (e.g., AlpacaEval 2), others improve and then degrade (IFEval, GSM8k), or remain relatively unchanged over training (MMLU). We show the AlpacaEval 2, IFEval, and GSM8k performances individually in Figure 7 . This highlights the need to measure a broad variety of benchmarks: while just examining AlpacaEval 2 would suggest that training for 3 epochs (or longer) is best, examining IFEval we would find that the model after 500 steps (0.5 epochs) is best, and GSM8k peaks at the 1 epoch mark. As such, we also broadly caution against evaluating model performance only on llm-as-judge benchmarks such as AlpacaEval, as we observe that improved AlpacaEval performance may come at the cost of other desirable properties.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.That is, we compare DPO models trained directly on each dataset with PPO models trained using reward models trained directly on that dataset and using prompts from the same dataset. Note that PPO uses additional generations from the model during training, but both approaches use the same amount of labelled data.p < 0.05 in a two-tailed paired t-test.This is because best-of-N with multiple-choice evaluations does not test the ability of the reward model to pick model generations, but merely picks the correct multiple choice answer.As the RewardBench reasoning subset is made up mainly of coding tasks, we hypothesise the presence of noisy StackExchange data may harm the 70B model, which has more capacity to fit to the larger mixture dataset than the 13B RM.Note that this is only a third of all UltraFeedback data. We reduce the size to fairly compare to the small number of GSM8k prompts.However, a smaller 13B RM performs almost as well if one is compute-constrained.https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairsManually cleaned and TruthfulQA prompts removed.For example, conversations with empty turns.https://github.com/sylinrl/TruthfulQA/https://github.com/google-research/google-research/tree/master/instruction_following_evalhttps://github.com/paul-rottger/exaggerated-safetyhttps://github.com/hamishivi/EasyLMIn practice, a generalized advantage estimation is used.In practice, we clip the ratio νt(θ) = π θ (y t |x•y <t ) π ref (y t |x•y <t ) by 1 ± ϵ, and minimize Lπ(θ) = -E min νt(θ) • At, clip(νt(θ), 1 -ϵ, 1 + ϵ) • At 


Decision: The primary concerns raised by the reviewers related to the limitations in the breadth of the experiments and the depth of analysis. However, these concerns were largely addressed by the additional experiments during the rebuttal period.  The authors have argued that the constraints were due to computational limitations, which were managed effectively to extract meaningful insights from the experiments conducted. Moreover, the empirical results obtained are still highly valuable. The observations made in this paper are practical and provide a good foundation for further research. We recommend the paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper disentangles core components of current learning from preference feedback algorithms in alignment, conducts comprehensive experiments on the individual effect of each component, and provides a recipe of learning from preference feedback based on experiment results.
strength_weakness: strength_weakness: 1. The experiment testing the influence of training dataset size for the reward model is not sound enough. The dataset is a mixture of different high-quality datasets, which may affect the reward model's performance on different tasks, making it difficult to attribute changes in performance solely to the dataset size. Besides, the conclusion that "both increasing the reward model dataset (‘Mix’) and reward model size (from 13B to 70B) improve performance" is not evident, as the dataset size shows little influence in the 70B model.
2. The limited number of models tested in the paper may restrict the impact of this work, as results could vary among different models, as the authors have mentioned in their "Limitations" section.
questions: questions: Here are some questions that I'd like to discuss with the authors to enhance my understanding of this work:
1. What is the data ratio of Mix and UltraF in the reward model experiment?
2. What is the codebase used in this paper?
3. On the right side of Table 3, the GSM scores indicate that the 70B UltraF.RM with UltraF.prompts performs best, whereas Figure 3 shows a different result. Is this discrepancy due to different experimental settings?
limitations: limitations: none
paper_summary: paper_summary: This work concentrates on methods for LLM learning from preference feedback and conducts a lot of experiments to identify key aspects of the preference-based methods. The work gives an ordering for the  importance of the core aspects: preference data quality, algorithm choice, reward model quality, and finally targeted policy training prompt.
strength_weakness: strength_weakness: Although a lot of observations are provided in this work, it seems that authors don't provide a deeper and more systematic understanding of the whole process of preference learning. Indeed, this is not an easy task, but I think it would be really helpful if we could draw some deeper insights from all these observations.

  Since there are many  elements involved in the preference training process, it is not easy to control variates. I found that some results might be improved with more ablations.
questions: questions: Here I have some detailed questions and I would be thankful if authors can give some further explanations.
1. On page 5 line 153, the quality of pairwise judgements is mentioned but there is not much explanation for this. Can we give a relatively quantitative method to judge the quality of pairs?
2. On page 5 line 161, it is mentioned that PPO outperforms DPO. Since there is a lot of DPO related research recently, I'm willing to see some more discussion about PPO and DPO. Is the offline property of DPO the main reason?
3. On page 6 table 3, it seems quite strange that 70B Mix RM has a lower RewardBench Score than 13B Mix RM. Is there any reason for this result?
4. On page 7 line 201, it is mentioned that there are no improvements in overall downstream performance. I noticed that the PPO is trained on UltraF prompts. Since the two Mix RMs are trained on more data, I might expect the two models can achieve better performance if the PPO is trained on the Mix prompts. In Fig.3, we observed that Mix RMs gain higher acc on GSM train. Also, I noticed that in table 4, Mix RMs trained on the code-and-math-mixed prompts have poor performance. Can we get some principles for dataset selection for RM?
5. On page 7 table 4, I found that the set of UF prompts and the set of Mixed prompts have the size. Since more code and math prompts are used in the Mixed prompts, some UF prompts are not used. Could this lead to the degradation of the Avg. Across all Evals? What if we use all UF prompts and some further code and math prompts to train a PPO?
6. On page 18 table 7, the UF dataset is split into Strong, Middle and Weak sets. Are the four datasets (UF all, Strong, Middle and Weak) downsampled to the same size for comparision?
7. On page 19 equation (5), is there any typo with the \pi_{ref}？Does it refer to the SFT policy?
8. The work shows that preference data is most important. I noticed that effect of dataset size is not well discussed. Is there any observation about this? Can more data provide more improvements?
9. I am curious about the training process during these experiments. For example, how the training reward changes, how the KL divergence between PPO policy and SFT policy changes, how the entropy of the policy changes, and how the length of the responses changes during the whole training processes. I would be grateful if authors could share some interesting observations during training.
limitations: limitations: N/A
paper_summary: paper_summary: The work summarizes the area of learning from preferences for optimizing language models. Specifically, the analyze four aspects: preference data, learning algorithm, reward  model, and policy training prompts. They empirically  answer questions on the downstream  improvement by improvements in each of these axis. Overall, they observe that the largest improvement arises from higher quality preference data, followed by choice of learning algorithm (PPO over DPO) with relatively smaller gains on reward model capabilities followed by training prompts.
strength_weakness: strength_weakness: * Novelty: While the work helps answers some excellent questions, the overall scope of the work seems limited. It would be interesting if this could be expanded to a more comprehensive study, see the question below.
questions: questions: How do these insights scale to other paradigms of language models such as in the  multimodal regime?
limitations: limitations: Yes, Section A in the Appendix.
paper_summary: paper_summary: The paper identifies four core components in training from preference feedback (RLHF): labeled preference data, learning algorithm, reward model, and unlabeled training prompts; and conducts a study on the effect of each component separately to disentangle their contributions to performance.
The authors experiment with variations of each component and make several claims:
1. There is a clear ordering of the impact of each component on performance: preference data quality > algorithm choice > reward model quality > targeted policy prompts.
2. Synthetic data with (aggregated) per-aspect preferences works best.
3. PPO outperforms DPO
4. Increasing the reward model size or its training dataset size improves its performance on reward benchmarks, but has marginal effects on downstream tasks overall.
5. targeted online training prompts can improve performance in domain-specific settings, but not overall performance.

Hence, they suggest a recipe with the best of each component.
strength_weakness: strength_weakness: I find that the claims made in the paper often deviate from the experimental support provided, they are either overstated compared to the scope or the experiments, or inaccurately reflect the scope of the experiments. Specifically,

**Algorithm choice:**

Although comparing components such as prompt datasets is possible when keeping all else the same (algo, hyperparameters, etc), the claim "PPO outperforms DPO" cannot be made without carefully considering hyperparameters.
The authors have a single hyperparameter configuration for each model, and although they may have been validated in other settings, they have not been optimized (within a fixed but same budget) for each algorithm on the author's setting.
The DPO hyperparameters were originally used by Zephyr-Beta, and taken without tuning for TULU 2 [Appendix F.1, 22].
For PPO, apart from the number of epochs that have been cut to 1, it's not clear how the authors obtained the rest of the hyperparameters.

Furthermore, from Table 2, "PPO outperforms DPO" _on average_ on _pre-selected_ datasets. When looking at individual benchmarks DPO models outperform PPO models in quite a number of tasks, and Table 2 only includes a subset of datasets which has not been motivated in the paper.

Therefore I believe that the claim is misleading.

**Preference dataset**:

Only Table 1 with DPO is used to formulate the conclusions on the best preference dataset. For the claim to hold in the scope of the paper (considering its limitations with a single pre-trained model) the preference datasets have to be compared with at least multiple algorithms (and ideally hyperparameters).
Table 2 provides some results for PPO but on all datasets, so it fails to complement Table 1.

The authors do not provide insights about why synthetic data performs best. It is appreciated to observe such difference in performance, but the claim would have more value and impact with some analysis or intuition as to why it would hold.

The claim "Preference-based learning has the strongest effect on instruction following and truthfulness performance." may only be valid in when the preference data itself mainly includes preferences based on instruction following and truthfulness, which is the case in the datasets considered in the paper. However, it may be very valid, but inefficient, to train factuality using preferences.
The scope of such a claim should be carefully explicited.

**Reward model**:

Although the paper aims to disentangle the effects of each component in RLHF, the experiments it conducts on some components tend to depend on the results from the previous components.
E.g. the reward model experiments are only carried out on the preference datasets providing the best performance, giving a biased view on the effects of the reward model and does not allow to formulate an ordering on the performance contribution of each component independently.

Also, the average improvement of 0.6 from the 13B to the 70B UltraF. reward model has been qualified as marginal while the improvement of 0.7 from DPO to PPO has been qualified as significant.
questions: questions: How did the authors select the preference datasets to compare DPO vs PPO in Table 2? One would expect to have them compared on all datasets as in Table 1 or to have a justification for the datasets that have been dropped/kept.
limitations: limitations: Although in the appendix, the authors adequately state the limitations of their work and its broader impacts.

Comments: title: Acknowledging Rebuttal
comment: I totally understand the computational budget limits and did not imply that the budget was objectively low, nor did I imply that its use was not efficient. What I mean is that there is a misfit between the budget and the initial research question addressed or the claims expected from them, especially the "Disentangling" part, which didn't really end up being a disentangling.

(a), (b), (c), and (d) can all be answered by observations and that's why I said that the numbers will be useful to readers, but I believe that they lack some research depth providing an understanding of why a number is better than another. The authors blamed the computational budget which I then raised as a limitation and a misfit, but I also believe that the current results can be analyzed further to answer other research questions with enough depth.
For this reason, I prefer to keep my score.
Thank you for responding! We are happy to clarify any further questions or address concerns if you have other reasons for not raising the score beyond the original weaknesses and questions asked (which we hope we have addressed appropriately above).
Thank you for your response, I decide to keep my score.
Thank you very much for your response and raising your score!

As for research questions, we believe that we have examined a number of useful and interesting areas, including (a) what public datasets works well for DPO, (b) relative performance of DPO and PPO across different datasets (and models, with our Llama 3 results), (c) the effect (or rather, surprising lack thereof) of the size of the RM during PPO training (with similar results from increasing dataset size using well-performing datasets from prior steps), (d) the effect of using more domain-specific prompts during PPO training. We hope and believe these are interesting and useful results for researchers in the RLHF space, and supported by the experimental results we have reported.

While it would be great to further explore these effects by doing a more thorough grid search over {dataset, RM size, prompt} combinations, we note that running a PPO experiment takes \~54 hours on a v3-256, and so running all possible dataset and RM size combinations (14 x 2) would take 1,512 hours (ignoring extra experiments exploring mixing datasets or varying prompt sets, which would incur further costs). Based on the google cloud calculator (https://cloud.google.com/products/calculator?hl=en) as of the 12th of august, 700 hours on a v3-256 (rather than newer, more expensive options) costs \\$394,240 in europe-west4, and so running these additional experiments would take \~\$800,000, and take 63 days to run (and if we wanted to e.g. explore the effect of using a different prompt set in each case, this would further multiply the cost). As such, we first ablated datasets in cheaper experiments (DPO), and then examined the more promising ones in PPO, and then additionally further explored key aspects of PPO using the best dataset found. This allows us to explore interesting and promising aspects of PPO without extreme computational costs.
I thank the authors for the clarifications. Their reply addresses all my points but also acknowledges a major limitation due to computational costs.

I understand that the authors aimed to provide the best empirical recommendations they could given a limited budget and acknowledge that many of the numbers in the paper will be relevant to readers. However, I believe that under this limited computational budget, not enough research questions have been tackled and answered in satisfactory manner. I find this a necessary condition for a publication at the venue.

I, therefore, increase my score but maintain it below the acceptance threshold.
Thanks for taking the time answering the questions, keeping my score.
Thank you for your response, most of my concerns are addressed during rebuttal and I decide to keep my score.