Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
Authors: Hamish Ivison
Yizhong Wang
Jiacheng Liu
Zeqiu Wu
Valentina Pyatkin
Nathan Lambert
Noah A. Smith
Yejin Choi
Hannaneh Hajishirzi
Venue: NeurIPS 2024 poster (2024)
Keywords: RLHF
learning from preferences
PPO
DPO
Language Models
Primary area: natural_language_processing

Abstract: Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories.

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
Hamish Ivison *
Yizhong Wang
Jiacheng Liu
Zeqiu Wu
Valentina Pyatkin *
Nathan Lambert
Noah A. Smith
Yejin Choi
Hannaneh Hajishirzi
Allen Institute for Al University of Washington hamishivecs. www.ingtwssinnnoncooooomomoo edu
Abstract
Learning from preference feedback has emerged essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, identify four aspects preference- based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% math and 1.2% in general domains. High-qualin preference data leads to improvements of up to 8% instruction following and truthfulness. Despite significant gains of up to 5% mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements other categories.
We publicly release the code used for training' and evaluating? our models, along with the models and datasets themselves?.
Introduction
Modern language models (LMs) commonly undergo a final stage of training, called learning from preference feedback, before deployment to end-users. This stage of training has been applied many prominent language models like ChatGPT [45], Llama 3 [35], and Claude [4], and has been shown to improve performance across wide number of capabilities, including instruction following [22], code [28], math [54], and summarisation [67]. Despite the widespread use and potential of this learning paradigm, applications of preference-based learning vary wildly, both in the data and learning algorithm used. As such, it is unclear what aspects of learning from preferences matter most for
'Sometimes this stage called reinforcement learning from human feedhack (RLHF). However, the human reinforcement learning aspects are not always present, while the learning from preferences aspect is always present. We will use the terms 'learning from preferences' and "preference-based learning' interchangeably.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

70
PPO
PPO
Domain-
specific
70
Bigger
w:
PPO w.
Bigger
Prompts
Porl. on All Evals
DPO
Reward
Mixed
GSM Accuracy
60
DPO
Reward
Model
60.5
65-
DPO
Better
PPO
Model
Prompts
1
DPO Pref. Data
55-
DPO Better
DPO
PPO
Aggregated Porf.
60
Pref. Data
Weak
61.0
62.2
62.4
Weak Prof. Data
53.0
50-
Pref. Data
50.5
58.1
55
45
46.0
45.0
40
Initial SFT
DPO Runs
PPO Runs
Initial SFT
DPO Runs
PPO Runs
Figure 1: Performance improvements resulted by changing different components in the preference
training of TüLU. Left: Accuracy on GSM [9], for testing math capabilities. Right: Overall performance, aggregated over the 11 benchmarks described in $2.2.
downstream model performance, particularly in relation the two most popular preference-based learning algorithms, PPO and DPO, which take different approaches preference-based learning.
As seen in Figure 2, both DPO and PPO rely on training on preference data Iata-DPO for directly training the model, and PPO for training reward model. In PPO, this reward model is then used to score generations from the main model generated using a set of policy training prompts (unlabelled prompts used for eliciting generations). This provides us with four important aspects of learning from preferences, including the choice of learning algorithm itself (PPO vs. DPO)
In this work, aim systematically investigate these key components of learning from preferences, exploring the effect of varying each component on downstream model performance. Starting from an initial strong open supervised finetuning (SFT) model, investigate each aspect in turn. As in Figure 1, we find that all aspects are important for performance, albeit to varying degrees. Our findings follows:
When comparing 14 popular existing preference datasets across a diverse set of evaluations, we find that synthetic, diverse data annotated with per-aspect preferences works best for learning from preferences ($3.1). 1). We find that the quality of the preferences (choice of chosen/rejected pairs) matters more than the quality of the actual generations being considered.
PPO outperforms DPO across varied datasets in our evaluation suite, even when using exactly the same models and initial training data ($3.2).
Increasing reward model size or dataset size used to train the reward model results in improved reward model performance on benchmarks directly testing reward model performance. Examining their effect on policy model performance when used during PPO training, find that these improved reward models have large effect on GSM performance, but give marginal to no improvements across all other evaluations considered ($3.3).
Using unlabelled prompts that better match the test setting during policy can further improve model performance in fomain--pecifit settings (e.g., when focusing improving math performance), but has limited to no effect when targeting overall performance ($3.4).
Overall, suggest recipe for learning from preference feedback ($4): using synthetic preference datasets and training using PPO with a large reward model performs best overall. Additionally, targeted prompts should be used if one only cares about single particular downstream task.
2 Setup
We first describe the core aspects PPO and DPO before moving into our experimental setup. We summarise both approaches in Figure 2.

---

DPO Training
PPO Training
Preference Data: Dr
Policy Training Data: D
Reward Model (x(x,y)
Policy
Prompt: x
Policy Training, Prompt:
Decodl
Model Ty ( (x)
Responses (offline): yo.k
Respoese (online):
Ranking: y/ y/ xx
Score
Reward score:
Train
Figure 2:
The core aspects of learning from preference feedback. For DPO (solid line), preference data is directly used to train policy model. For PPO (dashed line), preference data used to train reward model, which then used to score model-generated responses during PPO training.
2.1 PPO and DPO
PPO. PPO-based approaches for learning from preference feedback involve first training a reward model on preference data, and then training policy model by scoring responses produced by the policy model itself during training ("online" learning) as shown in Fig. 2. First, the policy model prompted to generate responses, which are then scored with the reward model. These scores (i.e., scalar rewards) are then used to guide the optimization of the policy model, following the PPO algorithm. Additionally, a KL penalty term is applied to avoid model degeneration.
Preference data. A preference dataset DR is used for training reward model, and it typically consists of prompts, responses, and rankings. Each prompt x comes with a pair of responses Mes yr. and preference ranking between them (denoted as Me > response and y/- is the rejected one). Both the responses and the rankings can be either human- annotated, generated/evaluated by other language models, or derived from examining relative votes on publicly posted online comments (e.g.. comparing high and low-upvoted comments from Reddit). While recent work has relied much synthetic, model-generated data [53, 22], it is unclear if high-quality human data can provide similar or better performance, or the significantly larger size of scrape datasets allows for improved performance.
where Ve is the chosen
Reward model. The reward model R.g (2r, y) is a scalar function, and can parameterized with a transformer where regression head replaces the language modeling head. The reward model is usually trained to minimize the following loss:
Ca (R(D) ((2.,...-))DDD. log a (Ry Me) (x, v-))]-
(1)
Policy training prompts. The policy training data D, has a set of policy training prompts. Instead of the pre-generated responses, here for each prompt we sample a response y from the policy model being actively trained, y (y(2x.) It is then scored by the reward model to get a sequence-level reward, r = Rap (x. y). Intuitively, this suggests that more accurate rewards should improve model performance. Additionally, we note that most existing works typically use either the prompts used during reward training [66] or a generic pool of anticipated user queries [37]. We investigate better targeting the prompts used during policy training further improve performance. Policy training. The goal policy training is to maximize the reward of policy-generater responses, subject soft KL constraint that prevents degeneration:
max Ex--D..grrttlll (I, y)] 3l0kI. (TellFret),
(2)
where Tee is the reference policy (usually the same SFT policy that initializes policy training). We find tuning the KL penalty coefficient / important for performance and explore the effect of varying in App. Since directly optimizing Eq. can be unstable. it is common reformulate language generation as a Markov decision process (MDP) and optimize using an RL algorithm. PPO is one of the most widely adopted RL algorithms for this problem. See App. F.2 for additional details about PPO.
DPO. DPO is offline RL approach for performing learning from preference feedback. It allows one to directly optimize policy on preference data, without building reward models or needing to sample online from the active policy. As such, DPO is an offline training algorithm. In simple terms, DPO aims at increasing the margin between the log-likelihood of the chosen responses and the liglliild of the rejected ones, while ensuring the model does not stray far from the initial policy.

---

Preference data. The structure of preference data is identical to that of PPO.
Policy training. By following closed-form solution to Equation 2. DPO re- writts any reward
model Ry in terms of its corresponding optimal policy (e"()))
3 log [xd[y]a]
(e)
log (x)), where (x) is partition function. Consequently, the policy model can be trained by directly optimizing the reward objective in Eq. 1, and hence the DPO loss:
(0) -((.......) log a (3 log
(B x) [ef x))
To x)) Tref (Ur xx)
B log
(3)
Comparing DPO and PPO. Compared with PPO, DPO is more efficient in terms of compute, speed, and engineering efforts. DPO does not need the extra stage of training a reward model, and during policy training it does need decode online responses (which usually slow) or train an additional value model. Meanwhile, PPO trains on online data generated by the current policy, while DPO trains on static, pre-generated offline data. This may limit exploration in DPO and thus hurt the training quality. While concurrent work has compared DPO and PPO [60, 48), comparisons are typically limited constrained domains and evaluations, using ground-trui rewards [60] or primarily examine smaller synthetic settings [48]. We complement such studies by comparing the downstream performance of models trained with DPO and PPO across wide variety of datasets and evaluations and consider additional potential factors in PPO performance, such as improved reward models and policy training prompts.
2.2 Experimental and Evaluation Setup
We base our exploration off TüLU 2 13B [22]. a popular openly released model. TüLU 2 is series of Llama 2 [52] finetunes across all sizes with publicly released weights and data, the largest of which achieved state-of-the-art performance on AlpacaEval and Chatbot Arena. As such, we are curious how much further we can push the performance of TULU 2 models through exploring alternative datasets, training algorithms, reward models, and policy training prompts. We use this model as a base policy when training policy and reward models, following Ouyang et al. [38] for PPO and Rafailov al. [39] for DPO. We provide additional details for each in App.
Evaluation We extend the TŪLU [55] evaluation, aiming to cover a diverse range of skills and behaviours. We evaluate models on 11 different benchmarks, covering skills such factuality (MMLU [20]), reasoning (GSM8K [9]. Big Bench Hard [5, 47J). truthfulness (TruthfulQA [29]). coding (HumanEval+ [6, 32), MBPP+ [2, 32]), safety (ToxiGen [19], XSTest [42]), and instruction following (AlpacaEval 1 and 2 [27, 131, IFEval [65]). We report the per-category average of evaluations and the overall average across categories. We provide further details in App. D.
3 Exploring Learning from Preference Feedback
We now explore each aspect of learning from preferences: preference data, learning algorithm, reward models, and finally policy training prompts.
3.1 Preference Data
We compare the performance of models trained with DPO across 14 different preference datasets in Table 1. We choose datasets that represent various potential sources of data: human-annotation
(HH-RLHF [4], HelpSteer [56), Chatbot Arena 2023 [64] and 2024 [7], AlpacaFarm Human [14],
PRMS00K [28]), web-scraping (SHP-2 [15]. StackExchange [25]), and synthetic generation (Ultra-
Feedback [11]. Nectar [66], Orca [34], Capybara [12], AlpacaFarm GPT-4 [14]). For UltraFeedback, we consider both using the 'overall' score provided in the dataset and taking an average of the per-aspect scores ('fine-grained"). We provide further detail on each dataset in App. B. We find that:
Preference-based learning with existing datasets has the strongest effect on instruction following and truthfulness performance. Our best models improve on the SFT model by over 8 points in these categories. contrast, preference-base learning does not aid factuality, with all models remaining with I point of each other. This suggests that when using existing publically-availan datasets, preference-bar based learning most useful for improving chat-related abilities (instruction following, truthfulness) and learning stylistic features, but less strong at tcaching new facts to

---

Source
Samples
Factuality
Reasoning
Coding
Truthfulness
Safety
Inst. Following
Average
Llama 2 base
52.0
37.0
30.7
32.7
32.7
TOLLU 2 (SFT)
35.4
47.8
45.1
56.6
91.8
44.2
56.8
500,000
45.6
Weh
StnckExchange
500,000
55.7
46.8
39.6
67.4
92.6
44.6
57.8
PRMS00K
6,949
55.3
49.7
46.6
54.7
91.9
43.4
Chatbol Arena (2023)
20,465
55.4
50.2
45.9
58.5
67.3
50.8
54.7
Human
Chatbot Arena (2024)
34,269
55.7
50.4
37.7
56.7
58.1
50.7
51.5
AlpacaF. Human Pref
9,686
55.3
47.6
43.3
56.1
90.7
44.5
$6.2
HH-RLIHF
158,530
54.7
46.0
43.6
65.6
93.1
45.4
58.1
HelpSteer
9,270
55.2
48.2
46.5
60.3
92.5
45.2
58.0
Alpacal. GPT-4 Pref
19,465
55.3
49.1
43.4
57.7
89.5
46.3
56.9
Capybara
7,563
57.5
91.5
Orea Pairs
12,899
55.5
46.8
46.0
$7.9
90.5
46.2
57.2
Symthetic
Nectar
UltraF: (overall)
60,908
55.6
48.8
46.5
67.6
92.1
51.1
60.3
Ultra.. (fine grained)
Table 1: Preference data: Performance TÜLU 2 13B models trained on various preference
datasets using DPO. Blue indicates improvements over the SFT baseline, orange degradations. Overall, synthetic data works best. DPO training improves truthfulness and instruction-folloiing most, with limited to improvements in factuality and reasoning.
Data / Model
Alg.
Factuality
Reasoning
Coding
Truthfulness
Safety
Inst. Foll.
Average
Llama base TOLU (SFT)
52.0
37.0
30.7
32.7
32.7
55.4
47.8
45.1
56.6
91.8
44.2
56.8
DPO
55.3
47.8
42.4
56.2
92.0
46.7
56.7
StackExchange
PPO
55.1
47.8
46.4
54.2
92.6
47.4
57.3
DPO
55.4
50.2
45.9
58.5
67.3
50.8
54.7
ChatArena (2023)
PPO
55.2
49.2
46.4
55.8
79.4
49.7
DPO
55.2
47.6
44.2
60.0
93.4
46.6
57.8
HH-RLHF
PPO
54.9
48.6
45.9
58.0
92.8
47.0
57.9
DPO
55.6
45.8
39.0
68.1
93.3
48.4
Nectar
PPO
55.2
51.2
45.6
60.1
92.6
47.4
58.7
DPO
55.3
50.9
45.9
69.3
91.9
52.8
61.0
UltraFeedback (FG)
PPO
56.0
52.0
47.7
71.5
91.8
54.4
62.2
Avg. b/w PPO & DPO
-0.1
+1.3
+2.9
-2.5
+2.3
+0.11
+0.7
Table 2: DPO PPO: Average performance of 13B models trained using DPO and PPO across different datasets, along with the performance difference between DPO and PPO (A). Blue indicates improvements over the SFT baseline, orange degradations. All datasets are downsampled 60,908 examples (except ChatArena, which is made up of 20,465 responses). PPO outperforms DPO by an average 0.7 points, where most improvements are reasoning, coding, and chat capabilities.
model. Interestingly, we observe that training on the Chatbot Arena data performs poorly on safety, suggesting Chatbot Arena volunteers generally prefer more toxic completions.
Synthetic data with per-aspect annotations performs best. Synthetic datasets generally outperform numan-annotated and web-scraped datasets, especially in truthfulness. Additionally, using datasets collected by first getting per-aspect annotations (i.e., annotations from human or model that score the helpfulness, harmlessness, etc. of the response independently) and then averaging across these scores tend outperform datasets that rely only on overall judgements (i.e., just asking the annotator for overall score instead of per-aspect score). The two datasets that use this method, HelpSteer and Fledbakee display stronger or similar performance datasets up to 15 times larger (e.g. HelpSteer vs HH-RLHH).. We investigate the performance of varied sub-splits of UltraFeedback in App. E, which suggests that the use per-aspect annotations more important for performance than the quality of the models used to generate completions for the dataset.
5

---

Direct Eval.
PPO Training Perf. (w. UltraF. prompts)
Reward Model
RewardBench Score
Best-of-N over SFT
GSM
AlpacaEval2 winrate
Avg. on All Evals.
Avg. Perf. (A)
Acc.
13B UltraF. RM
61.0
56.9 (+5.8)
53.0
26.1
62.2
13B Mix RM
79.8
58.3 (+7.3)
51.0
25.7
61.6
70B UltraF. RM
73.6
61.1 (+10.3)
58.0
26.7
62.8
70B Mix RM
73.9
60.6 (+9.5)
51.5
31.6
61.8
Table 3: Reward model evaluation: (a) Direct evaluation: reward models when directly
evaluated using RewardBench [26] and Best-of-N (left two columns); for BoN, we report both raw average performance, and the difference in performance over the base SFT model in brackets. (b) Downstream evaluation: models trained using PPO and the given reward model (right three columns). We report GSM and AlpacaEval 2 performance along with average performance across the entire evaluation suite defined in $2.2. Directly comparing reward models indicate increasing scale and data improves reward models, but these only minimally impact downstream performance.
3.2 Preference Learning Algorithm: DPO vs. PPO
We now compare algorithms for learning from preferences, comparing the performance of DPO and PPO when the same base models and data used (Table 2). We use exactly the same data for training DPO and PPO models, and subsample larger datasets to 60,908 examples due to computational resources and only use these subsampled datasets during reward model and PPO training. For dataset choice, we use the top-performing dataset from each source type in Table (StackExchange from Web, HH-RLHF from human, Ultrafeedback from synthetic). We also include the second-best performing dataset overall (Nectar) and an additional human-annotated dataset from popular evaluation platform (Chatbot Arena 2023). Results in Table indicate that:
PPO outperforms DPO. Across all datasets, models trained with PPO outperform models trained with DPO. In fact, PPO is able to provide improvements over the SFT model in cases where DPO training did not, such as when using StackExchange. On average, PPO significantly improves over DPO performance.
PPO improves on DPO in reasoning, coding and safety capabilities most. PPO improves over DPO by an average of 1.3, 2.9, and 2.3 points for reasoning, coding, and safety respectively, while truthfulness tends to degrade by an average of 2.5 points. Instruction following and factuality remain largely the same. Interestingly, we find that models trained with PPO are far more likely than DPO-trained models to perform chain-of-thougl! reasoning when prompted with reasoning or math problems, even when not given in-context examples using thain-of-thought This suggests that reasoning improvements from PPO may be due to increased -f-nh-of-thought abilities. Additionally, while overall instruction following ability remains similar, find that PPO-trained models tend to perform better at AlpacaEval I and 2, with PPO-trained models outperforming DPO-trainer ones on AlpacaEval 2 by an average of 3.4 points.
3.3 Reward Models
Next, we focus on PPO and study reward models both directly, and on downstream tasks with PPO training (Table 3). We first consider two ways to improve reward model:
Scaling up the training data for the reward model. We construct data mixture of the top- performing datasets in Table 1 from each section: UltraFeedback, HelpSteer, Nectar, StackEx- change, HH-RLHF, and additionally add PRM800K for math data. We compare reward models trained on this data mixture (called Mix RM) with reward models trained only on UltraFeedback (UltraF. RM) the per-errfnnnnn dataset from prior sections.
2. Scaling up the reward model size. We train reward models at two scales of 13B and 70B starting from TULU2 2.
'That is, compare DPO models trained directly on each dataset with PPO models trained using reward models trained directly on dataset and using prompts from the same dataset. Note that PPO uses additional generations from the model during training, but both approaches use the same amount of labelled data.
"p < 0.05 in wootailld paired t-test.

---

Direct evaluation of reward models. To isolate the performance of the differing reward models, we first evaluate them with best-of-N (BoN); we sample 16 generations from TüLU 13B SFT, score them using the given reward model, and then use the top-scoring output as the model output. Notably, we ensure model outputs are identical between runs, meaning that the only difference is the reward model scores. We perform evaluation on the subset of evaluations in our suite that require long-form generation," 7 and report overall average performance. We additionally evaluate our reward models on RewardBench [26]. popular evaluation for reward models, which involves evaluating if the relative scores given by reward models match a test set of chosen-rejected pairs from diverse sources, We provide further details in Appendix H.
Results in Table indicate that either increasing the reward model dataset ('Mix') or reward model size (from 13B to 70B) improves direct RM performance. Surprisingly, we find that our 70B reward models perform best on BoN evaluation, while the 13B mixture model performs best on RewardBench. Both evaluations show that increasing the dataset mixture and increasing the dataset size can help, although increasing the dataset mixture is not as useful for further improving the 70B reward model. Although scaling model size helps with best-of-N, it does not improve RewardBench performance. Examining the per-split performance on RewardBench in App. H Table 13, the largest gaps between the 13B Mix RM and the 70B Mix RM is in reasoning (mostly code), suggesting that the larger model may not benefit much from the additional (somewhat noisy) data".
Downstream evaluation of reward models. We then test if our improved reward models lead to improved downstream policy models when used during PPO training. We perform PPO training using the UltraFeedback prompts during training and report our results on the right side of Table 3. Surpris- ingly, we find that improvements in reward models result in surprisingly small improvements in overall downstream performance. We see the largest (and only) overall improvement when using the 70B UltraFeedback RM, despite the fact that all improved RMs performed significantly better than the 13B UltraFeedbcck RM in RewardBench and Best-of-N evaluations. Additionally, the improvement from the 70B RM is largely driven by a large performance jump in GSM, as shown in Table 3, while other metrics stay largely similar. This suggests that it is difficult to translate improve- ments in reward models to the underlying policy. We note that most prior work examining reward models tends to examine either direct reward model performance [26, 61] or proxy reward [17, 40], rather than downstream performance. Our findings suggest that improving such evaluations may not necessarily translate to improved downstream performance. Additionally, the larger 70B UltraF. RM is less sensitive and performs well with lower KL. penalty coefficient values than using the 13B UltraF. RM, and further examine the effect of the KL penalty coefficient on performance in App. I.
find that using
While it may seem unintuitive that an extreme increase in reward model size does not lead to extreme improvements in performance, we note that prior work has similarly noted that much larger reward models do not necessarily lead to significant improvements in performance. and smaller reward models can lead to similar performance (Ouyang et al. [37] $C.2. Wang et al. [56] $4.3), although we are the first, to our knowledge. to ablate this and explicitly report results on downstream evaluations.
We additionally explore how further training and potentially overoptimizing against the reward model affects performance across different evaluations in App. L. Importantly, we find that different evaluations show different trends while some evaluations such as AlpacaEval benefit from continued training, other evaluations such as IFEval or GSM8k drop with continued training. This highlights the importance of evaluating over a diverse set of test tasks, including both 'traditional" benchmarks and LLM--a-a-judge evaluations.
3.4 Policy Training Prompts
We finally examine the effect of using varied policy training prompts, first when targeting a particular capability (math performance as evaluated by GSM), and then for improving overall performance. This is in contrast to prior work that just re-use the prompts used for reward model training [66] or a generic pool of anticipated user queries [37].
"This is because best-of-N of-N with multiple-choice evaluations does not test the ability of the reward model to pick model generations, but merely picks the correct multiple choice answer. As the RewardBench reasoning subset is made up mainly of coding tasks, we hypothesise the presence of noisy StackExchange data may harm the 70B model, which has more capacity to fit to the larger mixture dataset than the 13B RMM
7

---

62.5
Tols 2 SFT Basellne
Reward Model
Prompts
GSM
Coding
Avg. Across All Evals
PBO 13B UhaaF
60.0-
PPOw. 13B Mix RM
PrO 7018 Uhall
Tulu 2 SFT
46.0
45.1
56.8
GSM Accursyy
PPO W. 70B Mix RM
13B Ultra.
UF
53.0
47.7
62.2
55.0
UltraF.
Mixed
47.8
61.9
52.5
13B Mix
UF
51.0
46.8
61.6
13B Mix
Mixed
50.5
43,8
60.9
50.0
70B UltraF:
UF
58.0
47.3
62.8
47.5
70B UltraF.
Mixed
56.5
48.4
62.4
Mix
46.1
70B Mix
Mixed 52.0
44.9
61.1
Ulma
Prompts
Math
GSM Train
PPO Training Prongts (++ More Targeted for GSM)
Table 4: Policy training prompt overall eval- uation: Performance of PPO policy models trained with the given reward models on 60K prompts from either UltraFeedback or the remixed prompt set that adds additional unlabeled math and coding-related prompts. Using the remixed prompt set does not improve performance, either
Figure 3: Policy training prompt math evalu- ation: Performance of models trained on 20K prompts from varying sources using PPO and evaluated on GSM. Training with larger RMs trained on more data benefits more from in- domain prompts (i.e., prompts directly from the GSM train set), while weaker RMs struggle to generalize beyond their training prompts.
specific evaluations (math, code) in terms of overall performance.
3.4.1 The effect of targeted prompts.
We first examine the effect of using policy training prompts targeted towards particular downstream math capabilities as evaluated by GSM. We construct three different prompt sets: prompts sourced from the GSM train set, math-related prompts from varied datasets, and 20K random prompts from UltraFeedback', We further detail the approach used to identify math related prompts in Appendix We then train models using PPO with each reward model from the previous section on each prompt set. Results in Fig. 3 demonstrate that:
Larger reward models perform better when closely matching train prompts to test settings. When using prompts from the GSM train set, we find using either 70B reward models during training leads to significant improvements GSM, with our best performing model improving over the SFT model by 16 points
Weaker reward models struggle to make use of prompts differing from those they trained on. Surprisingly, we find that training with the 13B UltraFeedback reward model actually performs worse when using prompts apart from UltraFoedback, potentially due to the reward model generalising poorly to prompts not seen during training. Similarly, the 13B Mix and 70B UltraFeedback reward models struggle to make use of math prompts, which are out-of-dommin for the reward models, but in-domain for the task.
Overall, these results suggest that targeted prompt distributions can improve performance when coupled with powerful reward models (e.g., using prompts from the GSM train set and then evaluating on GSM). This highlights a strength of PPO: it can make use of unlabelled prompts to better target downstream tasks.
3.4.2 Altering prompts to improve overall performance.
Inspired by the success of the targeted prompts, we construct a new remixed prompt set by finding 20K math and 20K code-related prompts using the same method in the previous subsection (see App. J). We then combine the math, code, and UltraFeedback prompts to create a larger prompt pool that we downsample randomly to 60,908 prompts. This rebalances the prompt pool to focus more coding and math tasks, which we hope yields improvements on math and code respectively while maintaining overall performance. We present our results in Table 4.
Surprisingly, we observe that using mixed prompts does not seem to improve performance in the generalist setting. When looking at code and math results specifically, we do not see consistent
Note that this is only third of all UltraFeedback data. We reduce the size to fairly compare to the small number of GSM8k prompts.
8

---

Model
Factuality
Reasoning
Coding
Truthfulness
Safety
Instr. Foll.
Averuge
Llama 13B Base
52.0
37.0
30.7
32.7
32.7
Llaraa
[52]
24.7
88.0
91.9
51.2
57.7
Nous Hermes 138 [51]
53.2
43.5
47.7
80.5
43,9
38.7
51.3
Vicuna 1.5 13B [64]
54.5
39.3
38.5
62.8
92.4
45.8
55.6
TüLU 13B SFT
55.4
47.8
45.1
56.6
91.8
44.2
36.8
TULU 2+DPO 1318
55.3
50.9
45.9
69.3
91.9
52.8
61.0
TŪLU 2+PPO 13B (13B UFRM)
56.0
$2.0
47.7
71.5
54.4
62.2
TOLU 2+PPO 13B (70B UPRM)
55.4
53.9
47.3
72.3
91.9
55.8
62.8
TŪLU 2+PPO 13B (70B UFRM+MP)
55.3
53.1
48.4
71.0
92.7
$4.0
62.4
L3+TOLU 8B SFT
58.0
58.6
56.4
59.2
92.8
42.6
61.3
L3+TŌLU 2+DPO
59.4
56.2
55.6
71.4
91.7
50.4
64.1
L34TOLU 2+PPO 8B (8B UFRM)
59.5
57.0
55.9
69.6
91.4
56.0
64,9
1.3+TOLE 2+PPO 8B (70B UFRM)
58.5
60.8
55.0
72.8
91.8
55.8
65.8
L3+TOLU 2+PPO 8B (70B UIRM+MP)
58.3
40.6
48.2
62.4
91.0
53.0
58.9
Table 5: Putting together recipe for preference-based learning: Performance of our best- performing models along with popular open models based on Llama 13B. 'MP' refers to using the mixed prompt set described in $4. 'L3' stands for experiments using Llama base model. Using PPO with large reward model performs best overall.
improvements using the altered prompt mixture, and our overall performance tends to drop. This is likely due to the already diverse nature of Feeaakee such that when looking at the whole dataset (i.e., not just the 20K subset Figure 3), are able reach strong performance evaluations. Altering the distribution away from other tasks slightly hurts the overall performance. We additionally found that training all the mined prompts in additional all of UltraFeedback (i.e., not downsampling the combined prompt set) did not yield further improvements over the results shown in Table 4.
math and coding
4 A Recipe for Learning from Preferences
Putting together all our findings from previous sections, we suggest recipe for training strong model using learning from preferences, as shown Figure and Table We take a high- quality, synthetic preference dataset, large reward model, and train it using PPO. If we additionally wish to focus on a specific domain, we can additionally collect domain-specific prompts for policy training. We find that our best model (TOLU 2+PPO 13B trained with the 70B UltraF. RM) outperforms our best DPO model and other popular models based off Llama 2 13B, including Llama 2 Chat, which has also undergone SFT and PPO training. Additionally, incorporating task-specific prompts into policy training may further improve performance when the prompts align closely with downstream evaluations, shown in Figure 3. Finally, we also experiment with Llama 3.0 8B [35], finetuning on the TüLU2 Mix, and then training using DPO and PPO with the same hyperparameters. We find that overall performance is significantly improved, and observe similar trends with our other experiments (DPO performing better than PPO, using a larger reward model improving performance, using mixed prompts not improving performance).
Related Work
Learning from Preferences for LMs.
Initial approaches to learning from preferences used reinforcement learning from human feedback (RLHF) [8, 67), a method that first trains a reward model to capture human preferences and then optimizes against it using reinforcement learning algorithms such PPO [44]. Recent work has additionally questioned the need for PPO, and has found that similar but simpler approaches such REINFORCE [46] with LM-specific adjustments work well [1, 50]. Unlike prior work, instead focus on examining the effect of varying the data and models used in PPO (i.e., the reward model, preference data, initial policy model, and prompts used for sampling outputs). We believe that our results should transfer to similar approaches such as REINFORCE, since they share many
10 However, smaller 13B RM performs almost well if is compute-constrained.

---

commonalities with PPO (e.g., reliance on well-tuned reward model and an unlabelled prompt set for eliciting generations during training).
Another line of recent work has also attempted to simplify the PPO algorithm and remove the online generation component, with the most popular algorithm following this approach being DPO [39]. DPO's case of implementation and use has made it widely popular among open-source community, Notably, several nigh-performing models have been trained using DPO for learning from preferences, including TŌLU [22], Zephyr [53], and Mixtral-Instruc [24]. Much recent work [3, 21, 59, 16, inter alia] has attempted to further improve the DPO algorithm. However, comparisons of these approaches so far have found that they largely perform similarly [41, 43]. As such, in this work we focus on the most popular variant, DPO, and examine what data works best for it and how compares to popular online RL approach, PPO.
Comparing Approaches for Learning from Preferences. Recent concurrent work has compared the properties and performance of DPO, PPO, and other approaches for learning from preference feedback. Xu et al. [60] suggest DPO performs poorly when using data out-of-distribtiion from the initial base model, while PPO (and a semi-online DPO variant) perform better both in such cases and overall when evaluated on safety and code capabilities. However, they do not investigate the impact of reward models and focus on core algorithmic details of PPO that lead to improvements. Tajwar et al. [48] identify on-policy sampling and negative gradients as two important aspects of preference-bant based learning when optimal reward values do not lie 'close" to the base model's distribution and the preference data skewed. Tang al. [49] study how the IPO [3] algorithm performs in the static offline setting versus various ways of updating ordering the data in an online manner. In this work, we focus on empirically examining the impact of core aspects of learning from preference feedback, including the effects of varied rewards and data.
6 Conclusion
In this work, we have systematically explored the core components of learning from preference feedback and examined the relative impacts of each in turn on model performance across a wide range of evaluations. Our results suggest the following ordering of importance: preference data quality, algorithm choice, reward model quality, and finally targeted policy training prompts. Additionally, we find that using larger reward models can significantly improve math capabilities, but have marginal effects on other capabilities we evaluate in this work. Overall, we suggest recipe for learning from preference feedback with currently available resources: best performance can be achieved by using strong synthetic dataset (UltraFeedback), and using PPO training with large reward model. Our work suggests that further exploring how to make better use of improved reward models is an important direction for further improving PPO-style approaches to learning from preference data. We plan to release models and code related to this paper and hope that our settings provide a firm base for future work further exploring learning from preferences for language models.
Acknowledgements
This research was funded part with funding from the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8650-23-C0 23-C--316. DARPA MCS program through NIWC Pacific (N66001-19 19-24011,--, and NSF I0-2004660. Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). We thank members of AI2 and UW NLP for useful feedback throughout this project.
References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadace. Kreutzer, O. Pietquin. A. Üstün, and
S. Hooker. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, 2024.
[2] Austin, A. Odena, M. Nye, M. Bosma, Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry,
Q Le, and C. Sutton. Program Synthesis with Large Language Models, 2021.
10

---

[3] M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. /2036, 2023.
[4] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprini arXiv:2204. 05862, 2022.
[5] B. bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreviest
[6] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 0333774 2021.
[7] W.-L. Chiang, L. Zheng, Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, 2024.
[8] Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
[9] K. Cobbe, Kosaraju, M. Bavarian, M. Chen, Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.
2021.
[10] T. Coste, U. Anwar, R. Kirk, and D. Krueger. Reward model ensembles help mitigate overopti- mization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevies
met/forummidddeee tMKkxxx.
[11] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
[12] L. Daniele and Suphavadecprasit. Amplify-Instruct Synthetically Generated Diverse Multi-turn Conversations for Effecient LLM Training. arXiv preprint arXiv:(coming soon), 2023. URL https://huggingfe
co/ hat.aataess////eeeeeooeeaaaae LttLL/DDDD///aaaaa//
[13] Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-Controlled AlpacaEval: Simple Way to Debias Automatic Evaluators. arXiv preprint arXiv:2404.04475. 2024.
[14] Y. Dubois, C. X. Li, R. Taori, T. Zhang. I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information P Processing Systems, 36, 2024.
[15] K. Ethayarajh. Y. Choi, and S. Swayamdipta. Understanding Dataset Difficulty with V-Usable Information. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988-6008 PMLR, 17-23 Jul 2022.
[16] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint 24Xx:22402.01300 2024.
[17] L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, ICML' '3. JMLR.org, 2023.
[18] X. Geng. EasyLM: Simple And Scalable Training Framework for Large Language Models, 2023. URL htttts://ttth/tt com/young- gengggnngggggee Meang/E
[19] T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar. TOXIGEN; Controlling Language Models to Generate Implied and Adversarial Toxicity. ACI., 2022. URL https: / //arxiv. wwg.oogg/////o/oooooooooooeeeert 2203. 09509.

---

[20] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring Massive Multitask Language Understanding. In International Conference on Learning Representations (ICLR), 2020.
[21] Hong, N. Lee, and Thome. ORPO: Monolithic Preference Optimization without Reference Model, 2024.
[22] H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi. Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2023.
[23] H. Ivison, Y. Wang, V. Pyatkin, J. Liu, J. Lu, and Z. Wu. EasyLM-Fork: A Simple And Scalable Training Framework for Large Language Models, 2023. URL https://gtttbbb. http//amishivi/easee
[24] A. Q- Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, F. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L.. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antomiak, T. L. Scao,
T. Gervet, T. Lavril, T. Wang, Т. Lacroix, and W. E. Sayed. Mixtral of Experts, 2024.
[25] N. Lambert, I.. Tunstall, N. Rajani, and T. Thrush. HuggingFace H4 Stack Exchange Preference Dataset, 2023. URL https::/hugggnggfcc..oooo hwtp.///attttttt/ggggggaaaaa..ee stack-
[26] N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, N. A. Smith, and H. Hajishirzi. RewardBench: Evaluating Reward Models for Language Modeling, 2024.
[27] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. AlpacaEval: An Automatic Evaluator of Instruction-following Models. Github repository, 2023. URL https: /tt///gtttuu//b thub. conn/ttsu-llab/ alpaca_evall.
[28] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, Schulman, I. Sutskever, and K. Cobbe. Let's Verify Step by Step, 2023.
[29] S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252, 2022.
[30] J. Liu, S. Hallinan, X. Lu, P. He, S. Welleck, H. Hajishirzi, and Y. Choi. Rainier: Reinforced knowledge introspector for commonsense question answering. Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 8938-8958, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10. 18653/v1/2022.emmllg http:/0222.mmmmmmmmm..111.. URL https: atttps://acanme aalaaalaalnnoooyyyyoooooo org/2022. emnlp-main. 611.
[31] J. Liu, R. Pasunuru, H. Hajishirzi, Y. Choi, and A. Celikyilmaz. Crystal: Introspective Reasoners Reinforced with feffFeeedbac. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://app. semanticscholar. org/ CorpusID: forpuSIDD26383031. 263830316.
[32] Liu, C. Xia, Y. Wang, and Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL net//forum7id-qqz61100
[33] X. Lu, S. Welleck, L. Jiang, Hessel, L. Qin, P. West, P. Ammanabrolu, and Y. Choi. Quark: Controllable Text Generation with Reinforced Unlearning- ArXiv, abs/2205.136m 2022. URL https://api. semanticscholar, org//orpps// 249152301.
[34] K. Lv, W. Zhang, and H. Shen. Supervised Fine-Tuning and Direct Preference Optimiza- tion on Intel Gaudi2, 2023. URL. https: wwweeddiiimommmiteeell.l cs-software
a1 a1197d8a3cd3,
12

---

can [35] Meta. Introducing Meta Llama 3: The most capable openly available LLM to date, May 2024. URL attps://aa. meta. com/blog/metteeeeoooooooooooooee wowgg//mmtaaaaaee 11ama--a-------- 3/.
[36] N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, Singh, X. Tang, L. von Werra, and S. Longpre. OctoPack: Instruction Tuning Code Large Language Models. arXiv preprint arXiv:2308.077 07124, 2023.
[37] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. E. Miller, M. Simens, A. Askell, Welinder, P. Christiano, J. Leike, and R. J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022. URL https:
/ //api. swmanticschoolarr Srrg/CorpusID:220 :26422604..
[38] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, et al. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[39] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model secretly reward model. arXiv preprint arXiv:2305.18200 2023.
[40] A. Ramé, N. Vieillard, L.. Hussenot, R. Dadashi, G. Cideron, O. Bachem, and J. Ferret. WARM: On the Benefits of Weight Averaged Reward Models, 2024.
[41] K. Rasul, E. Beeching, L. Tunstall, L. von Werra, and O. Sanseviero. Preference Tuning LLMs with Direct Preference Optimization Methods. https://huggingfaee httph/gggiin..ac....oooo/gg/ pref - tuning, 2024. Accessed: 2024-05-17.
[42] P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2024.
[43] A. Saeidi, S. Verma, and C. Baral. Insights into Alignment: Evaluating DPO and its Vari- ants Across Multiple Tasks. 2024. URL htttps://ap/ www.nticscooolaatto..ooeo wwwg/rorpssDDDD.... 269303161.
[44] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint 1rXii:1707.06344 2017.
[45] J. Schulman, B. Zoph, C. Kim, and more. ChatGPT: Optimizing Language Models for Dialogue. https://openai. com/blog/ wwwtgg////////.//////////oo 2022. Accessed: 2023-02-12.
[46] R. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In S. Solla, T. Leen, and K. Müller, editors, Advances Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https://proceedinggs newr.is...e.....eee..... /LL1//86822500000885600000363333 -aaeer... pdf.
[47] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Tay, H. W. Chung, A. Chowdhery, Q. Le, E. H. Chi, D. Zhou, al. Challenging BIG-Bench tasks and whether fatesrffllssppppp///eeee solve them. arXiv preprint arXiv:2210.09261, 2022.
[48] F. Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, S. Ermon, C. Finn, and A. Kumar. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, 2024.
[49] Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. Ávila Pires, M. Valko, Y. Cheng, and W. Dabney, Understanding the performance gap between online and offline alignment algorithms, 2024.
[50] G. Team. Gemma: Open Models Based on Gemini Research and Technology, 2024.
[51] "Teknium". Nous-hermes--lamm2-1133 URL Nous Hermes- LLamaa- 13bb] MousResearcch/ooss--eee Hermes L1ama2- 11b] (https:
co/NousResearch/
/ htt.://gaaeeeeoeoooooeearrrrrraa Hermes Llama2-1111 -1b)m.
13

---

[52] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov. S. Batra, Bharggva, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[53] L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, et al. Zephyr: Direct Distillation of LM Alignment. arXiv preprint
Nttv:23100119444 2023.
[54] P. Wang, L. Li, Z. Shao, R. X. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. Shth-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations, 2024.
[55] Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. R. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, et al. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. arXiv preprint arXiv:2306.
2023.
[56] Z. Wang, Y. Dong, J. Zeng, V. Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. P. Scowcroft, N. Kant, A. Swope, and O. Kuchaiev. HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM, 2023.
[57] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting clicits reasoning in large language models. arXiv preprint arXiv:2201 11903, 2022.
[58] Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi. Fine-Grained Human Feedback Gives Better Rewards for Language Model Train- ing. ArXiv, abs/2306.01693 2023. URL pattps///apl. sewant semanticacholar .rgg Porg/orppusI: 259064099.
[59] H. Xu, Sharaf, Y. Chen, W. Tan, Shen, B. Van Durme, K. Murray, and Y. J. Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.088: 17, 2024.
[60] Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y. Wu. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, 2024.
[61] L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun. Advancing Ilm reasoning generalists with preference trees, 2024.
[62] W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. (InThe) 570K ChatGPT Interaction Logs In The Wild. In The Twelfth International Conference on Learning Representations, 2024. URL https: http::::////peeevieeee
www//frruue http://formmmddd/llllme BdB8887ZRI.M
[63] I. Zheng, W.-L. Chiang, Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, 7. Li, 7.. Lin, P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang. LMSYS-Chat- 1M: A Large-Scale Real-Worke LLM Conversation Dataset, 2023.
[64] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-juue with MT-Bench and Chatbot Arena. In NewrlPS Datasets and Benchmarks Track, 2023.
[65] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction- Following Evaluation for Large Language Models, 2023.
[66] B. Zhu, E. Frick, Wu, H. Zhu, and Jiao. Starling-7B. Improving LLM Helpfulness Harmlessness with RLAIF, November 2023.
[67] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0885 2019.
14

---

A Limitations & Broader Impacts
Limitations. As an empirical study with limited compute, our results are largely based on an in- depth examination over a single model suite (TOLU 2), using two base models (Llama 2 and 3). Additionally, our evaluation may not reflect all potential downstream use-cases- for example, we do not test multilingual performance. As such, examining how our results carry to multilingual settings or greatly differing base models is interesting future work. However, we have done our best to explore a wide variety of evaluations and datasets, using publicly available resources to aid in reproducibility. note that, while PPO does well, it comes with significantly
In terms of computational efficiency, increased computational cost compared to DPO. We only consider performance in this study and don't 't' explicitly measure the relative computational cost of different methods (e.g.. by measuring FLOPs cost of training with DPO vs PPO). There are also other aspects to keep improving PPO (e.g.. doing new rounds of preference annotations update the reward model for the changing distributions of the policy model), which we cannot exhaust in this paper, and we leave the exploration of them for future work.
Broader Impacts. Language models have recently been deployed extensively, but there are few public studies on the impact of different algorithms for learning from preference feedback and datasets on these models. We hope to shed light on the impact of this stage of LM training and aid in improving future LMs. We explicitly consider safety evaluations as part of our evaluation and hope that our findings help inform how to improve the safety of future LMs, limiting potential harms and enhancing potential benefits. However, note that focussing too much on learning from single preference datasets may result in LMs being aligned to only relatively small subsections of the global population.
B Dataset Details
We examine the following datasets this work. We provide details on where we source the data and the license associated with each dataset.
SHP-2 [15]: We use the publically available HuggingFace train split, and randomly down- sample to 500,000 samples. The stackexchange portion of the dataset is licensed under the CC BY-SA license, and the reddit portion made available in accordance with the Reddit API terms of use. See the HuggingFace dataset card for more details.
StackExchange [25]: We use the publically available HuggingFace train split, and randomly downsample to 500,000 samples. The dataset is licensed under the CC by-SA license.
PRMS00K [28]: We use the data from the second phase of collection. We consider prompts where the model generations achieved the correct answer once and failed to find the right answer once. We then randomly choose one correct and one incorrect generation as chosen and rejected respectively. The dataset is licensed under the MIT license.
Chatbot Arena Conversations (Chatbot Arena 2023) [64]: We use the publically avail- able HuggingFace train split at https: / //wuggiiggacceee co /ataaatts/nnsss
chatbot_ arena_ cllae_covvecaate ions. We remove all multi-turn samples as these diverge after the first turn (but users still rate the entire conversation), and filter out ties. The prompts are licensed under CC-BY-4.0 and the outputs under CC-BY-NC-4.0.
Chatbot Arena Preferences (Chatbot Arena 2024) [7]: We use the publically available HuggingFace train split at
We remove all multi-turn samples as these diverge after the first turn (but users still rate the entire conversation). and filter out ties. The dataset is licensed under the Apache 2.0 license.
imays-arenc, human-pree ppreference-5552
AlpacaFarm Human and GPT-4 Preferences [14]: We use the publicly available hugging- face 'preference' splits for each dataset. The dataset is licensed under CC-BY-NC-4.0.
HH-RLHF [4]: We use the publically available HuggingFace train split. The dataset is licensed under the MIT license.
HelpSteer [56]: We use the publically available HuggingFace train split. We average the fine-grained scores (except verbosity) for an overall score to decide chosen and rejected pairs. The dataset is licensed under the CC-BY-4.0 license.
Capybara 7k [12]: We use the publically available HuggingFace train split co/ co//ataastt/aarillaa
released by Argilla available at https://huggiin
15

---

Dataset
# Samples
UltrFeedback (FG)
60,908
Stack Exchange
60,908
HH-RLHF
60,908
HelpSteer
9,270
PRM800K
6,949
Nectar
60,908
Total
259,851
Table 6: Size of the subsplits of our RM mixture data.
distilabel-caaybaaaa 7k- 77k-binarized. The dataset is licensed under the Apache 2.0 license.
Orea Pairs [34]: We use the publically available HuggingFace train split cleaned by Argilla", as found it had better performance in initial experiments. The dataset is licensed under the Apache 2.0 license.
Nectar [66]: We use the publically available HuggingFace train split. The dataset is licensed under the Apache 2.0 license.
UltraFeedback [11]: We use the split released by Argilla and consider versions: where the chosen and rejected samples are chosen using the overall score (OveralI/OV) and where they chosen using an average of the fine-grained scores (Fine-graineed (Fine-grained/FG)). The dataset is licensed under the MIT license.
We additionally create splits of HH-RLHI, StackExchange, and Nectar downsampled to 60,908 (matching the size our UltraFeedback split) examples for size-equal comparisons of algorithms across different dataset types (i.e., Table 2). We also filter out any malformed examples find'
We additionally construct our 'mix' used for expanding the reward model training by number of examples from each dataset as listed in Table 6. Overall, our 'mix" prefence set contains roughly 260k samples, over four times larger than just using UltraFeedback.
C Model and Code details
We primarily build off TÈLU 2 13B and 70B [22], which themselves are based on Llama 2 [52]. TULU 2 models are licensed under the A12 low-risk license and available at https //ugg//gg/ggegacceeeeeee co/ collections, under a custom license and available at https: http://///ggggggg...oooooooo//oa
Ilama 2 is licensed
1lama- 2 amalm 1at.aaa222f111111....00000000000 2aa.-66.....11100006666666777771
For code, we build off EasyLM [18], specifically the fork used train TŪLU [23], which is licensed under Apache 2.
D Evaluation Details
We the following evaluations. We provide the category map each evaluation to in brackets.
MMLU (factuality) [20]: We use the official MMLU evaluation script and prompts avail- able at https://githn
com cww/momd/eieeeeeeee hendrycks/testt with modifications to allow for batch processing. We evaluate using 0 few-shot examples, following the original setup of MMLU. We report average accuracy across test examples.
GSM8k (reasoning) [9]: We evaluate models on the test set of GSM. Following Wei et al. [57], evaluate with bfain-of-thoughe
We use 8 few-shot in-context examples. Because all answers GSM are numbers, we extract the last number in the model response as the final answer. We report average accuracy across test examples.
hhtpps: http://hhgggggaaaceeeee httpe//eaaaateeeegaaaaaaaaaaaaaa harillabbll-nnlllleeeeeeee -or--orca-ppp sacca-dpo-paire 12 Manually cleaned and TruthfulQA prompts removed. For example, conversations with empty turs..
16

---

Big Bench Hard (BBH; reasoning) [5, 47]: We follow the setup described in the original paper and evaluate with Ohain-of-thought. The officially provided prompts, which have few-shot in-context examples are used. For the COT setup, we extract the first word after the phrase 'So the answer is', , or the entire response if there is no such substring present. We report average accuracy over sub-tasks (all of which use accuracy as the primary metric). TruthfulQA (truthfulness) [29]: Following Touvron et al. [52], we mainly use the gener- ation setting. We follow the official script in their official implementation" to do greedy decoding and answer postprocessing. We also follow their instruction to train two GPT-based classifiers to judge the truthfulness and informativeness of the model response. We report the % Informative and Truthful.
AlpacaEval (instruction following) [27, 13]: We use the package provided by Li et al. [27], following the default setup for both AlpacaEval and length-controlled AlpacaEval 2 [13]. We allow the evaluated model to generate up 8192 tokens, without specifying special stop sequences.
IFEval (instruction following) [65]: IFEval benchmarks whether models can follow in- structions that contain verifiable constraints, such as "write in more than 400 words". We use the official evaluation code released with the original paper and report the "Loose Accuracy" metric at the prompt level (i.e., a response is counted as correct only if all the constraints in the prompt are detected be satisfied after normalizing the response).
Hvla (coding) [6. 32]: We use the augmented form of the HumanEval dataset introduced by Liu et al. [32], which contains additional test cases. We additionally use the instructions provided by HumanEvalPadd [36] when prompting instruction-tunee models. We report pass@ 10 and sample with a temperature of 0.8. We use v of the released Eval+ package.
MBPP+ (coding) [2, 32]: For coding, we also evaluate using MBPP+, for which we prompt models complete a program program given a natural language description and function signature. Similar to HumanEval+, We report pass@ 10, sampling with a temperature of 0.8.
We use the v of the released Eval+ package.
ToxiGen (safety) [19]: We follow the setup in Touvron et al. [52], but use the original set of prompts from Hartvigsen et al. [19], which are designed to elicit toxic generations for certain groups We take only the prompts designed to produce toxic language ('hateful' prompts) and use 500 prompts per group to reduce evaluation costs. For base language models, we pass in the original ToxiGen prompts unchanged and greedily decode up to the first new line (or maximum of 512 tokens). For instruction-tuner prompt in the corresponding template, and ask the model to complete the prompt, until the model generates a stop token (or maximum of 512 tokens). We pass the generated text into a roberta-large model trained to detect toxic content finetuned as part of Hartvigsen et al. [19] (available at https: htap::://uugee htt.//uggggfacccceeee/o
models, we place the
co/tomh/ monnhh/txxigeee wwoxigennrbbbrttete report the percentage of generations deemed toxic by the classifier.
We then
XSTest (safety) [42]: XSTest evaluates whether a model can correctly refuse to follow malicious instructions, while still being able to correctly identify similar but safe instructions and follow those ones correctly. We use the same set of test prompts provided in their official repository 16 which includes a mixture of 200 unsafe prompts and 250 safe prompts. Similar to the original paper, we tried using both GPT4 and heuristic rules to detect whether the model responses are refusal. We found that GPT4 provides more reliable detection, as modern models may respond in various ways that exceed the coverage of those pre-defined rules. For GPT4-based detection, we found the originally provided prompt in the official repository has a confusing definition of "full refusal" and "partial refusal", so we removed that distinction and rewrote the prompt. In this paper, report the Fl metric, which aggregates the precision and recall of model refusals. We will release all our evaluation codes for reproduction.
17

---

# Samples
Factuality
Reasoning
Coding
Truthfulness
Safety
Inst. Foll.
Average
IJama 2 base
$2.0
37.0
30.7
32.7
32.7
Tulu 2 (SFT)
55.4
47.8
45.1
56.6
91.8
44.2
56.8
UlraFcodback (FG; All)
60.908
55.3
50.9
45.9
69.3
91.9
52.8
61.0
Strong Model Gen.s
10,000
55.3
46.5
45.6
58.9
90.1
47.6
57.3
Middle Model Gen.s
10,000
55.2
48.5
44.5
60.6
91.5
48.2
58.1
Weak Model Gen.s
10,000
55.2
47.3
45.4
60.5
90.4
46.2
57.5
FalseQA Prompts
2,339
55.2
49.2
44.0
60.8
92.2
45.5
57.8
Evol Instruct Prompts
10,000
55.5
47.8
47.5
58.4
48.7
58.1
TruthfulQA Prompts
811
55.4
47.6
44.5
55.7
91.6
45.4
56.7
Flan V2 Prompis
2,0939
55.3
49.4
46.3
62.4
91.3
48.9
58.9
ShareGPT Prompts
19948
55.4
49.8
43.9
60.2
87.5
50.3
57.9
UltrChht Prompes
9929
55.4
50.6
47.5
59.9
89.9
48.1
58.6
Table 7: Performance of varied sub splits of Utraaeeddbcck, considering both samples using genera- tions from models varying strength (weak, middle, strong model gen.) strong and only considering samples using prompts from single datasets (e.g., FalseQA, Flan V2). Sampling from different--uality models makes relatively little difference, while prompt choice matters, Additionally, different source datasets provide improvements in different evaluations.
E
UltraFeedback Subset Study
Intrigued by the strong performance of UltraFeedback, the overall best performing dataset, further compare the performance of differing subsets of Ultraeeedback, and show our results in Table 7. First, we compare using only generations from the highest-scoring middle-scoring, and lowest-scoring models (as judged by GPT-4). Surprisingly, we find that there is little difference in overall performance between these splits. This suggests the preference annotations from GPT-4 (i.e., the decisions about chosen and rejected pairs) are more important than the strength of the underlying models used sample responses. Then, compare models only trained on samples from the same original dataset (as UltraFeedback was constructed by sampling prompts from various sources). We find that each prompt source performs best in least one category. This suggests that selecting a diverse set of prompts when generating synthetic data is important, and UltraFeedback already contains a diverse set of prompts covering various downstream applications.
Subset Construction Details. We construct the 'top', 'middle", and 'bottom' sets UltraFeedback by using the average score of responses from each model according the average of fine-grained scores. We then bucket the models into three groups accordingly. We provide the groups and average scores in Table For constructing the splits, then filter UltraFeedback to only include responses from the given models, pick the highest-scoring response per prompt as the chosen, and a random lower-scoring prompt as rejected. We construct a subset of 10,000 prompts for each group of models. For the prompt subsets, we simply use source dataset annotations provided in the UltraFeedback data itself.
F Additional Details for PPO and DPO
We performed hyperparameter search for both DPO and PPO, sweeping across core hyperparameters in pilot experiments on HH-RLHF and UltraFeedback. We provide further details for both algorithms below.
F1 DPO
We testing multiple values for 3 (0.1, 0.01, 0.001) and for learning rate (5e-6, 5e-7, 5e-8) in pilot experiments HH-RLHF and UltraFeedback. Ultimately, suggested by Tunstall et al. [53] and followed by Ivison et al. [22] worked best (learning rate of 5e-7, 3 of 0.01). This additionally involves 3 epochs of training with learning rate of 5 x 10-?, with
found that the hyperparameters
"https//githuh..cmm// http:///hhhbb.ooomlli/i///ttoooo
18

---

Group
Model
Avg. Score
gpt-4
4.50
gpt-3.5-turbo
4.48
Top
wizardlm-70b
4.14
bard
4.12
vicuna-33b
3.95
mpt-30b-chat
3,92
llama-2-70b-cha
3.91
Middle
wizardlm-133 13b
3.91
hama-2-13b-chal
3.78
ultralm-65b
3.69
ultralm-136
3.63
wizardlm-7b
3.44
llama-2-7b-chat
3.39
starchat
3.13
Bottom
alpaca-7b
2.97
pythia-12b
2.60
-iboon--Ob-inttuuc
2.57
Table 8: Model splits and average UltraFeedback fine-grained score used for construct UltraFeedback subsets used in Table 7.
linear warmup for 10% of training and linear cooldown for the duration. We use the open-sourced codebase used for training TüLU 2+DPO.
F.2
PPO
More on the algorithm. PPO formulates language generation as a Markov decision process (MDP), where each response is an episode, an action is token Vt, and state is the concatenation of construct token-level rewards r; that guide the RL training, the sequence-level reward r is applied only to the last token, and each token is penalized by KI. term (3 log (y x y<e) log (Br I Vet)). Note that Wa the model we are training and Tee is reference model (in our case, the initial SFT model we start training from). Formally, the token-level reward r, for each response token y/: defined as
prompt and partial response (x (ee)..).
- 3 log (v
оoce) log Teer x (ooD))) x V<t)) +r
(where 1 < < (y]) (where / (vl)
r2
- log (v
yoce) log mee
(4)
In addition to the policy model, PPO also trains value model (r that estimates the expected value function of incomplete responses under the active policy. The value model typically shares the same architecture as the reward model, while the regression head can be applied on any response token. The value model helps with estimating the advantage on each token, A, -V Gr, where G El n loss!"
rr is the empirical return. PPO trains the policy model by minimizing
T Fer(
vct) x v<t)
C (0) -E suno
-
(5)
and trains the value model by minimizing the MSE loss against the empirical return:
(v b (V G.)
(6)
I htatge://gttaab..oom/mmssss/// htt.://ammssssis//aags////
In practice, a generalized advantage estimation is used.
Inn practice, we clip the ratio (0) selullovved
Tarlod moyee) by + e, and minimize (x) -E(min
min (p: (0)
(v)
As, clip(px(0)) c, +c At)]
19

---

The models can be jointly optimized by linearly combining the two losses: (ero 6) (x) + (o)...
Implementation details. PPO comes with many implementation details. We made a simplistic implementation that results in stable training. Notably:
We initialize the value model from the reward model. This follows from InstructGPT [37]. Some other implementations initialize from the SFT model [33] or the base model [30, 58), while replacing the LM head with a regression head.
For truncated completions,
set the reward to large negative number (e.g.. -1.00, which is referred to as the EOS trick.
We do not perform normalization the rewards. This follows from AlpacaFarm [14]. A Although reward models trained under different settings can have very different output ranges, we found our PPO experiments quite robust such variation.
We do not whiten the step-level rewards within each batch. We do whiten the step-level advantages within each batch, following other implementations.
We use fixed KL penalty coefficient. The original PPO algorithm [44] has an adaptive KL controller, but most recent implementations have moved away from this [37, 52].
See Table 10 for a comparison with other open-source implementations.
Hyperparameters. The prompt and continuation each has at most 1024 tokens. We use batch size of 64 and the same for minibatch size, and train for 1 epoch over the prompt dataset. On each batch of prompts and rollouts, we train for inner epoch. We use sampling temperature for rollout, and a fixed KL penalty coefficient 3 = 0.05. We optimize with Adam with learning rate 17 = 1 10-6 with linear warmup for 10% of iterations. We provide additional details on the hyperparameters used for PPO in Table 9. In experiments, we found that training with larger (70B) reward models benefited from reduced KL penalty coefficient B, but using smaller reward models did not benefit (see App I for further discussion). As such, for runs using larger reward models, we use KL penalty of 0.0325 instead of B 0.05. This is in line with prior work suggesting that improved reward models are more difficult to overoptimize against, and so can benefit from lower KL penalties [17, 40]. We found these hyperparameters through experimenting in pilot experiments over HH-RLHF and UltraFeedback. We experimented with KL penalty coefficients of 0.01, 0.025, 0.0325, 0.05, as well as varying the batch size (including taking more samples per gradient step, and taking multiple steps over the same generations). We additionally explored training for up to 3 epochs and found that training beyond 1 epoch did not yield improved performance and occasionally resulted in the training collapsing entirely, and as such stuck to training for only epoch.
0.7
We additionally provide more detailed comparison of our PPO implementation other work in Table 10.
Reward model hyperparameters. For training the reward model, we follow prior work (7] in only training the reward model for one epoch, using a learning rate of 1 x 10-5 that warms up for the first 3% of training steps and linearly decays to x 10 by the end of training. We use a batch size of 512.
G Additional Observations with PPO
Choice of hyperparameters. We found that training beyond one epoch did not yield improved performance and occasionally resulted in the training collapsing entirely, and thus only trained for 1 epoch in our experiments (F. = in Table 9). We also found that training multiple inner epochs on each batch destabilizes PPO training. and thus we use inner epoch (e 1).
Accelerating training with larger prompt batch size. Rollout (i.c., generating online responses from the policy) is the most time-consuming step in PPO, taking up more than 95% of the total training time. We explored speeding up training by increasing the prompt batch size from 64 to 512, such that 64 due to memory constraint, and because we need about 900 gradient steps to make the training converge.
obtain more responses with small latency overhead. We kept the minibatch size
This setup implies that rollouts later minibatches become slightly off off-policy when the model trains on them. In most implementations of PPO, the forward passes are also done the batch level and
20

---

Symbol
Value
Description
TRAINING
B
64
Prompt batch size (for rollout).
Number of rollouts for each prompt.
Minibatch for forward-backe backwrdd passes.
Gradient accumulation (in number minibatches).
E
Training epochs.
Inner epochs trained for each batch.
Lpp
1024
Max number of tokens in the prompt.
1024
Max number tokens continuation.
0.0
Dropout
Temperature sampling rollouts.
0.05,
0.0325
KI. penalty coefficient.
1.0
Discount factor for rewards.
0.95
Parameter for generalized advantage estimation.
Clipping range policy and value losses.
0.1
Value loss coefficient.
OPTIMIZATION (ADAMW)
x 10 88
Learning rate.
10%
Percentage of iterations for warmup.
(3,, 32)
(0.9, 0.95)
AdamW optimizer hyperparameters.
x 10-5
Adam W optimizer hyperparameter.
0.0
Weight decay.
1.0
Max global norm for gradient clipping.
Table 9: PPO hyperparameters. We use values listed here unless otherwise noted.
Quark
Rainier/Crystal
FG-RLHF
AlpacaFarm
Ours
Init value from reward
x
X
X
V
EOS trick
x
X
Reward normalization
X
Reward whitening
/
J
V
X
Advantage whitening
<
V
<
Adaptive KL controller
X
X
X
KL clamping Multiple rollouts per prompt
X
X
X
X
Table 10: Variations in implementation of the PPO algorithm. Compared with serveral open-source
PPO implementations: Quark [33], Rainier [30], Crystal [31], FG-RLHF [58], and AlpacaFarm [14].
thus the resulting logprobs and values are also obtained from a slightly stale version of the models. In experiments, this severely destabilized training, and closer investigation shows that the value model loss did not converge fast enough to supply policy training with reliable advantage signals. As remedy, made the forward passes be carried out the minibatch level that these logprobs and values are obtained from the most current models. Note that this slightly deviates from most PPO implementations.
In Table 11, we compare the training time and performance of our default PPO setup and the larger prompt batch size setup. Increasing the prompt batch size by 8x reduces the training time by 5x (60 hours + 12 hours), while also decreasing the overall performance by 0.6% (61.7% - 61.1%). To remedy the performance loss, experimented with generating multiple (up 4) rollouts for each prompt, which increased the effective batch size for gradient updates while keeping the total number of gradient steps fixed. When using rollout multiplier of 4, most performance loss can be recovered, while the training speedup is also less dramatic.
Our conclusion from this set of experiments is that, it is important for the forward passes to be performed online, and to get the extra mile offered by PPO, the rollouts should also be generated fully online. Deviating from this may speed up training, but at some performance cost. Since
21

---

B
9
# training ex.
grad update bsz
# grad updates
Training Time
Avg. Perf.
64
64
1x
1x
1x
60 h
61.7
512
64
1
1x
1x
1x
12 h
61.1
512
64
2x
2x
1x
16 h
61.1
256
4x
1x
32 h
61.4
Table 1I: Performance of PPO under bigger prompt batch size. The first uses the same experiment setup as the PPO model trained on UltraFeedback (FG), as in Table 2. Hyperparameter notations are same as Table B = prompt batch size, b minibatch size for rollouts per prompt, = gradient accumulation. Number of training examples, gradient update batch size, and total number gradient updates are relative to the first row. We keep the total number of gradient updates fixed, train all models for 1 epoch. Increasing the prompt batch size can speed up PPO training at some performance cost, and most performance and (which effectively increases the gradient update batch size).
passes,
can be recovered by increasing
GSM
BBH
Codex-Eval+
MBPP+
AEval
AlEval 2
IFEval
XSTest
Model
Tulu 13B (SFT)
46.0
49.5
19.3
38.1
78.9
5.0
43.4
85.3
13B UltraF RM
60.5
50.0
29.3
39.9
91.6
20.5
47.1
84.6
13B Mix RM
60.5
54.3
29.3
39.9
92.7
22.3
50.3
86.2
70B UltraF RM
67.0
59.9
37.2
41.4
92.6
22.4
51.0
86.3
70B Mix RM
67.5
59.8
35.4
38.5
90.9
19.0
46.6
85.6
Table 12: Full results from Best-of-N evaluation summarized in Table 3.
didn't get better results with the large prompt batch size setting, did all other experiments with the default prompt batch size of 64.
H Reward Model Evaluation Details
Best-of-N Details For best-of-N, we sample 16 responses from TüLU 2 13B with a temperature of 0.7 for each evaluation task we examine. We then pass these responses (along with the prompt used for generation) into the given reward model, and use the top-scoring response as the final output.
Given list of these top-scoring outputs, we then pass these the of the given evaluation setup. We examine only a subset of evaluations, focussing on the evaluations that rely on long-form model generations (as we are most interested in the reward model's ability judge these outputs during PPO training). In particular, we look at GSM8k, BBH, Codex-Eval MBPP+, A AlpacaEvval 1 and 2, IFEval, XStest. For Codex-Eval+ and MBPP+, we use Pass @ instead and just evaluate the top-scoring example (unlike other tables in which we use Pass @ 10). When reporting the average score in Table 3, we first calculate an average score per evaluation task category (following the same evaluation categories as in App. D), and then report the average over categories.
We report the full best-of-N results across each evaluation in Table 12.
Chat
Chat Hard
Safety
Reasoning
Prior Sets
Score
13B UltraFeedback RM
74.3
49.3
52.2
65.0
67.9
61.0
13B Mix RM
97.2
61.2
85.9
78.1
73.7
79.8
70B UltraFeedback RM
96.4
60.5
63.7
74.8
71.4
73.6
70B Mix RM
94.4
52.2
83.3
65.9
73.5
73.9
Table 13: Full results from RewardBench evaluation for results shown in Table Score is weighted average of subsets with prior sets given weight 0.5 and all other sets given weight following Lambert et al. [26].
22

---

0.575
RM
30
RO4 7O8
62
0.525
AlpacaEval 2
Overall Avg.
GSM
61
0.500
24
60-
0.450
0.425
KSA 176
22
59
BM 130 RM 740
R04 7001
0.025
0.030
0,035
0.040
0.045
0.000
0.025
0.030 0.025 0.040 0,045
0.050
0.025 0.030 0.035
0.040 0.045 0.050
KL Penalty Confficient
KL Penalty Coefficient B
KLL Penalty Coefficient
Figure 4: Performance of models trained with 13B and 70B UltraF. RMs with varying KL coefficient values. (a) GSM Accuracy, (b) AlpacaEval winrate, (c) Overall performance across entire evaluation suite. Overall best performance occurs different KL coefficient values for different reward models. However, AlpacaEval 2 performance grows with reduced KL coefficient values.
RewardBench Details We report the subset scores of reward models RewardBench in Table 13.
I KL Penalty Coefficient Exploration
As in Figure 4, observe that the optimal KL penalty coefficient (3 in Eq. 2) value changes with the reward model used. While using the smaller 13B reward model results in large drops in overall performance as shrinks, the larger 70B RM is more robust and actually achieves higher performance at smaller 3 than the 13B UltraF. RMM This is in line with prior work suggesting that robust lower
larger reward models less prone overoptimization [17, 10), and such (which encourages more optimization against the RM). This highlights an additional potential benefit of larger RMs: they may be casier to tune due to their increased robustness choices of B.
We also note that some evaluations do not suffer as much from reduced 3. For example, AlpacaEval 2 performance (Fig. 4c) actually is highest the lowest 3 shown. This is likely due the match between A AlpacaEval and PPO training: as the reward models are trained on GPT-4 preferences, more closely optimizing against them (via lower 3) also improves the rate which the model generations are preferred by GPT-4. However, it also comes with significant reductions in other evaluations (as in the drop in GSM performance in Fig 4), which is undesirable when training a generalist multi-task model. This is in line with prior work observing that there appears to be an 'alignment tax' (i.e., reduced scores on various capabilities including reasoning and math) when performing learning from preference feedback [37, 4], and that this tax can be partially controlled through tuning 3 (c.g.. Fig. 34 in Ouyang et [37]).
J Prompt Domain Identification Details
In order to find additional unlabelled prompts, we mine prompts from UltraFeedback [11], Wild- Chat [62], and LMSYS-IM [63]. We only consider the prompts and throw away the accompanying model responses. We categorise unlabelled prompts by prompting TčLU 2 70B. We first prompt TÜLU 2 70B tag the prompts with various categories using the prompt shown in Figure We then sample Code and Math prompts by picking prompts that include 'Coding' or 'Math' tags and as few other tags as possible. We remove all prompts with 'unclear' and 'multilingual" tags as we wish to focus on clear queries made in English. The authors examined small portion of the mined prompts by hand and found this resulted in solid collection of high-quality prompts for math and code respectively.
23

---

Instruction
Please label the task tags for the user query.
## User Query
(Sinstruction}
## Tagging the user input
### Task Tags
all _a__aass_tags =
"Coding", Users seek help with writing, reviewing, or fixing code programming. "Math", # Queries related to mathematical concepts, problems, and calculations. "Asking for Advice", Users seek recommendations, suggestions, or advice on various topics.
"Brainstorming", Involves generating ideas, creative thinking, or exploring possibilities.
"Classification", Queries require categorizing or organizing information into groups classes.
"Closed Question Answering", Users ask questions that require a specific answer short response.
"Creative Writing", Users seek assistance with crafting stories, poems, other creative texts.
"Extraction", Involves extracting specific information or details from larger body of text.
"Inhabiting Character/Persona", # Users engage in scenarios requiring the model to adopt a character or persona.
"Open Question Answering", # Users ask questions that require detailed or elaborate responses.
"Rewriting", # Users ask for help in rephrasing, summarizing, or rewriting text. "Summarization", Involves condensing information, text, or content into shorter form.
"Multilingual", Queries involving non-English natural languages. "Unclear", # Queries that are ambiguous, vague, or unclear.
]
## Output Format
Note that you only select the most relevant task types.
Please use the multilingual tag if the query is in language other than English.
Add the unclear tag if there is obvious question to answer in the prompt.
Now, please output your tags below in a json format by filling in the placeholders in 0
{
"tags": [" [tag 1]", " "[tag 2]", ... ]
}
Do not add any additional characters.
Figure 5: Prompt used for classifying unlabelled prompts.
24

---

80
MMLU
GSM
BBH
Evaluation Pert.
60
HumanEval+
MBPP
TruthfulQA
AlpacaEval 1
AlpacaEval 2
IFEval
XSTest
ToxiGen
Cat. Avg.
1500 Train Steps
Figure 6: Performance of all evaluations over PPO training steps when training using the 70B UltraFeedback RM and UltraFeddbcck prompts for epochs. Grey dashed lines indicate epoch boundaries.
Alpacalival 2
JFevll
60
60-
00
10
Train
Figure 7: Performance (left) AlpacaEval 2, (middle) IFEval, (right) GSM8k PPO training steps when training using the 70B UltraFeedback RM and UltraFeedback prompts for 3 epochs. Grey dashed lines indicate epoch boundaries.
K
Compute Details
train all models on 256 or 128-size TPUv3 pods. PPO training with a 13B reward model and 13B takes 3 days 256-size pod, with the reward model training taking roughly 10 hours to train. DPO training on UltraFeedback (roughly 60,000 samples) for a 13B model takes 9 hours. We note that conducted additional smaller-sizz size runs that do not report results from in this paper as part of our research to further explore dataset and hyperparameter choices.
policy the Ulrraeeedback
L Model Performance over Training
investigate how the performance of our best model (PPO training TÜLU 2 13B with the 70B UltraFeedback RM and UltraFeedback prompts), and continue training beyond the epoch result reported the main text of the paper. We show these evaluations in Figure 6. We find that how performance changes over training quite distinct between each evaluation: while some evaluations
25

---

continuously improve (e.g., AlpacaEval 2), others improve and then degrade (IFEval, GSM8K), or remain relatively unchanged over training (MMLU). We show the AlpacaEval IFEval, and GSM8k performances individually in Figure 7. This highlights the need to measure broad variety of benchmarks: while just examining AlpacaEval 2 would suggest that training for epochs (or longer) is best, examining IFEval we would find that the model after 500 steps (0.5 epochs) is best, and GSM8k peaks at the I epoch mark. As such, we also broadly caution against evaluating model performance only on llmaasjudde benchmarks such vlp,, as we observe that improved AlpacaEval performance may come at the cost other desirable properties.
26

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims
made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: See Appendix A. Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification: This is empirical paper without theoretical results. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
27

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Best efforts have been made to make sure the results are reproducible, and we will provide code and model artifacts for all models discussed in the paper after the review period. All base datasets and models used in this paper are publicly available.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We provide links to our codebase used, with instructions for how to run the experiments described in this work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ Sublic/guides (CodeeSumisssioooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
28

---

reproducible, they including code, unless this is central to the contribution (e.g., for new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. cc/ ctce/pullccgguuu /uddeed/odddduuuiiii wwwssssssnsssliiieeeeeooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included many details possible about hyperparameter choices, data preparation, etc. Data details are in Appendix B, and hyperparameter details in Appendices F and G. We describe evaluation settings in detail in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail
that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report results from single runs due the computational cost of experiments, and make sure word claims accordingly.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative стror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
29

---

Answer: [Yes]
Justification: We provide details Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http::///neuueee http//nuripppppee httpc//ccccc..ttcccccceeeeeleeee
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid-
eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed Appendix Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deploymeent of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification: Our work primarily focusses discussing and explaining results. We will apply appropriate safeguards on artifacts when we publicly release them.
Guidelines:
The answer NA means that the paper poses such risks.
30

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Dataset details (including citations and licenses) are provided Appendix B. We provide model and code details in Appendix C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We provide links to released assets and code in our abstract, and the assets themselves provide details about the hyperparameters, datasets, etc. used to produce them, and further documentation about the released assets. Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
31

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "qNpFkMVypS", "reviewer": "Reviewer_mnCL", "summary": "This paper disentangles core components of current learning from preference feedback algorithms in alignment, conducts comprehensive experiments on the individual effect of each component, and provides a recipe of learning from preference feedback based on experiment results.", "questions": "Here are some questions that I'd like to discuss with the authors to enhance my understanding of this work:\n1. What is the data ratio of Mix and UltraF in the reward model experiment?\n2. What is the codebase used in this paper?\n3. On the right side of Table 3, the GSM scores indicate that the 70B UltraF.RM with UltraF.prompts performs best, whereas Figure 3 shows a different result. Is this discrepancy due to different experimental settings?", "limitations": "none", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper aims to understand PPO and DPO from a practical perspective by conducting comprehensive experiments on core components of the RLHF pipeline. The results enhance the understanding of RLHF and provide valuable reference data for the community.\n2. This paper is well-written and clearly presented.", "weaknesses": "1. The experiment testing the influence of training dataset size for the reward model is not sound enough. The dataset is a mixture of different high-quality datasets, which may affect the reward model's performance on different tasks, making it difficult to attribute changes in performance solely to the dataset size. Besides, the conclusion that \"both increasing the reward model dataset (‘Mix’) and reward model size (from 13B to 70B) improve performance\" is not evident, as the dataset size shows little influence in the 70B model.\n2. The limited number of models tested in the paper may restrict the impact of this work, as results could vary among different models, as the authors have mentioned in their \"Limitations\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and feedback, and for noting that our results are a valuable reference for the community and enhance understanding of RLHF. We address your feedback and questions below:\n\n**Concerns**:\n1. **Testing the influence of the training dataset size on reward model & downstream performance.**\n\nOur focus in these experiments is to see if improvements to the reward model (either through incorporating different data or increasing the RM size) lead to improvements in the downstream policy trained through PPO. In this sense, we are interested to see that both types of changes did not yield consistent large changes in downstream performance in Table 3, despite the fact that we observe improvements in RewardBench and Best-of-N settings, which more directly test RM performance.\nWe will clarify that \"both increasing the reward model dataset (‘Mix’) and reward model size (from 13B to 70B) improve performance\" means that taking either option may improve performance, as indeed we agree that we do not see evidence that a larger dataset together with a larger RM further improves performance.\n\n2. **Limited number of models tested.**\n\nPlease see our general response, point 1. We provide additional results using Llama 3 and think further extending our observations to other models would be interesting future work.\n\n**Questions**:\n1. **What is the data ratio of Mix and UltraF?**\n\nWe provide exact sizes of the subsets and overall sizes of Mix and UltraF in Table 6 in the appendix. Mix contains roughly 260k samples from Stack Exchange, HH-RLHF, HelpSteer, PRM800k, and Nectar. UltraF contains roughly 60k samples. UltraFeedback itself is made up of prompts and completions (from varied models) from FalseQA, Evol Instruct, TruthfulQA, Flan V2, ShareGPT, and UltraChat (see Table 7 in Appendix E or the UltraFeedback paper itself [1] for more details).\nWe add HelpSteer, Nectar, and HH-RLHF as they are the next best-performing datasets after UltraFeedback, and downsample HH-RLHF to avoid it making up most of the mixture. We then add Stack Exchange and PRM800k data to further diversify the data mixture, adding in more code and math-related data, which we empirically confirm aids reward model performance (see Table 3).\n\n\n2. **What is the codebase used in this paper?**\n\nWe extend EasyLM (https://github.com/young-geng/EasyLM) with our own DPO and PPO implementations to make the training work on our infrastructure, similar to Tulu 2. We compare our implementation details to InstructGPT, Rainier/Crystal, Finegrained-RLHF, AlpacaFarm, and Quark in Table 10 in Appendix F.2, and referenced the hyperparameters used in these approaches (especially InstructGPT, AlpacaFarm, and the Llama-2 paper) when choosing hyperparameters for our own experiments. We will release a link to the code after the anonymity period.\n\n\n3. **Table 3 vs Figure 3 unmatched GSM scores**\n\nYes, the different numbers reflect different experimental settings. In Figure 3, we limit training to 20k prompts due to the small number of GSM8k training prompts, and also to ensure that we are able to pick more relevant prompts for the mined set - we will clarify this in the figure caption.\n\n[1] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oTEsyTt8jO", "reviewer": "Reviewer_WxZd", "summary": "This work concentrates on methods for LLM learning from preference feedback and conducts a lot of experiments to identify key aspects of the preference-based methods. The work gives an ordering for the  importance of the core aspects: preference data quality, algorithm choice, reward model quality, and finally targeted policy training prompt.", "questions": "Here I have some detailed questions and I would be thankful if authors can give some further explanations.\n1. On page 5 line 153, the quality of pairwise judgements is mentioned but there is not much explanation for this. Can we give a relatively quantitative method to judge the quality of pairs?\n2. On page 5 line 161, it is mentioned that PPO outperforms DPO. Since there is a lot of DPO related research recently, I'm willing to see some more discussion about PPO and DPO. Is the offline property of DPO the main reason?\n3. On page 6 table 3, it seems quite strange that 70B Mix RM has a lower RewardBench Score than 13B Mix RM. Is there any reason for this result?\n4. On page 7 line 201, it is mentioned that there are no improvements in overall downstream performance. I noticed that the PPO is trained on UltraF prompts. Since the two Mix RMs are trained on more data, I might expect the two models can achieve better performance if the PPO is trained on the Mix prompts. In Fig.3, we observed that Mix RMs gain higher acc on GSM train. Also, I noticed that in table 4, Mix RMs trained on the code-and-math-mixed prompts have poor performance. Can we get some principles for dataset selection for RM?\n5. On page 7 table 4, I found that the set of UF prompts and the set of Mixed prompts have the size. Since more code and math prompts are used in the Mixed prompts, some UF prompts are not used. Could this lead to the degradation of the Avg. Across all Evals? What if we use all UF prompts and some further code and math prompts to train a PPO?\n6. On page 18 table 7, the UF dataset is split into Strong, Middle and Weak sets. Are the four datasets (UF all, Strong, Middle and Weak) downsampled to the same size for comparision?\n7. On page 19 equation (5), is there any typo with the \\pi_{ref}？Does it refer to the SFT policy?\n8. The work shows that preference data is most important. I noticed that effect of dataset size is not well discussed. Is there any observation about this? Can more data provide more improvements?\n9. I am curious about the training process during these experiments. For example, how the training reward changes, how the KL divergence between PPO policy and SFT policy changes, how the entropy of the policy changes, and how the length of the responses changes during the whole training processes. I would be grateful if authors could share some interesting observations during training.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Since the pipeline of preference-based methods has many details, it is hard to see works like this one to have done plenty of ablation experiments. This work brings a lot of observations for this community and can help researchers to have more understanding on different parts of preference-based methods.\n\n  This work gives an order for the importance of four key elements of preference learning. This is a relatively systematical investigation for reference-based learning.\n\n  This work also gives a recipe for learning from preferences and this may guide better performance for LLM.", "weaknesses": "Although a lot of observations are provided in this work, it seems that authors don't provide a deeper and more systematic understanding of the whole process of preference learning. Indeed, this is not an easy task, but I think it would be really helpful if we could draw some deeper insights from all these observations.\n\n  Since there are many  elements involved in the preference training process, it is not easy to control variates. I found that some results might be improved with more ablations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and questions. We address your concerns and questions below and hope this provides further insights into our work.\n\n**Concerns**:\n1. **Results could be improved with more ablations.**\n\nPlease see point 1 of our general response, where we additionally provide new results with a new base model (llama 3).\n\n2. **It would be helpful to draw deeper insights.**\n\nOur aim in this work is primarily to explore and test the performance of popular RLHF methods on existing datasets, and is primarily empirical. We hope that our work serves as a useful starting point for further, more focused studies into RLHF (for example, investigating further why using much larger RMs does not lead to much larger gains). We also note that most reviewers agree our work does contain observations and results that are of interest to the broader community. \n\n**Questions**:\n\n1. **The quality of pairwise judgements.**\n\nQuality here is primarily referring to the use of overall vs fine-grained judgements when deciding the chosen/rejected pairs in UltraFeedback, which has been shown to lead to improved reward model performance [1, table 2], even when using the same chosen/rejected pairs (and we find similarly improves DPO performance). We focus on the effect of data on downstream performance here, noting that it appears improving the chosen/rejected judgements can improve downstream performance.\n\n2. **Why is PPO better?**\n\nWe believe the offline vs online aspects of DPO and PPO are quite important for performance (as noted at the end of section 2.1), and there is concurrent work also suggesting this [2,3]. Additionally, we hypothesize that the RM being finetuned on preference data without any regularization terms (e.g. the beta in DPO) may allow it to more closely fit human preferences, which may result in a stronger policy downstream.\n\n3. **Why does 70B mix rm have a lower RewardBench score than the 13B model?**\n\nRewardBench consists of outputs from varied models across varied prompts, and so may be somewhat out-of-domain compared to best-of-N (BoN) evaluations for the purpose of identifying good RMs for PPO. In contrast, our BoN evaluation indicates that both 70B RMs outperform the 13B models, suggesting that larger models are better at identifying superior outputs from the same initial policy used in PPO, which more directly relates to how the RM is used during PPO training. Examining BoN and RewardBench scores in more detail in Tables 12 and 13, we see that 70B mix RM outperforms the 13B mix RM on math (GSM), while degrading in instruction following (AEval 1 & 2, IFEval), which matches with the worse performance on the chat subsets of RewardBench. We find it surprising the 70B mix RM underperforms in the reasoning subset, considering the improved GSM performance, potentially due to differences in evaluation distributions (RewardBench reasoning subset consists mostly of coding questions).\n\n4. **Can we get some principles for dataset selection for RM?**\n\nFigure 3’s best results are found when using the GSM8k train set as prompts, which is somewhat different to the typical zero-shot setting used for modern LMs, where we assume our models will be tested on tasks not explicitly selected for during training. We did try training our models with Mix RMs and mixture prompts for longer but still did not observe improvements in performance (see answer to Q5). Overall, the fact that RewardBench and BoN does not translate neatly to downstream performance from PPO makes determining good methods for dataset selection for RMs difficult. It would be interesting to further explore how to improve RM dataset selection with an eye towards downstream performance in future work.\n \n5. **What if we use all UF prompts and some further code and math prompts to train?**\n\nWe find that further training the model to use all the UltraFeedback prompts on top of additional math and code prompts does not result in consistent improved performance at the 13B scale, suggesting that using the full UF prompt with the mix prompts set does not provide consistent additional gains:\n\n| RM Type | avg perf with 60k prompts | avg perf with *all* UF + Mix prompts |\n|-|-|-|\n| 13B UltraF. RM |  61.9 | 61.3 |\n| 13B Mix RM | 60.9 | 61.4 |\n\n6. **Are the four datasets (UF all, Strong, Middle and Weak) downsampled to the same size?**\n\nYes (see Appendix E line 668).\n\n7. **Does page 19 equation (5) \\pi_{ref} refer to the SFT policy?**\n\nPi_ref does refer to the SFT policy, we will clarify this in our updated paper.\n\n8. **Can more data provide more improvements?**\n\nWe first note that as seen in Table 1, datasets that are simply larger are not necessarily more performant - SHP-2 and StackExchange are by far the largest datasets we test, but underperform much smaller datasets. Similarly, Nectar is 3x larger than UltraFeedback but performs slightly worse for DPO. However, we do also observe some evidence that increasing dataset size (or at least, increasing up to 60k samples) helps: training on all UltraFeedback data outperforms training on the {weak,middle,strong} subsets (all of which consist of 10k datapoints) - see Table 7 in the Appendix.\n\n9. **I am curious about the training process during these experiments.**\n\nWe recorded logs of reward, KL divergences, average response lengths, and more for all our experiments. We observed that average model output length tended to increase over training, as observed in prior work. Interestingly, we also observed that performance on different specific evaluations behaved differently over training, and provide further details in the additional rebuttal material PDF - please see the PDF attached to the general response.\n\n\n[1] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML.\n\n[2] Xu et al (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study.\n\n[3] Tajwar, et al  (2024). Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ynS8NjBdM", "reviewer": "Reviewer_wF89", "summary": "The work summarizes the area of learning from preferences for optimizing language models. Specifically, the analyze four aspects: preference data, learning algorithm, reward  model, and policy training prompts. They empirically  answer questions on the downstream  improvement by improvements in each of these axis. Overall, they observe that the largest improvement arises from higher quality preference data, followed by choice of learning algorithm (PPO over DPO) with relatively smaller gains on reward model capabilities followed by training prompts.", "questions": "How do these insights scale to other paradigms of language models such as in the  multimodal regime?", "limitations": "Yes, Section A in the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* Clarity of writing: The work is well written, concise and clear to follow.\n* Significance:  We gain practical insights from this work, including the use of synthetic datasets and large reward models, which can be valuable for practitioners in the field.\n* Quality: Empirical results are well documented, methods are sound\n* Originality: While this offers new insights, the work extends existing methodologies  rather than introducing novel concepts.", "weaknesses": "* Novelty: While the work helps answers some excellent questions, the overall scope of the work seems limited. It would be interesting if this could be expanded to a more comprehensive study, see the question below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and noting that our work is well written and clear to follow, and that the insights in our work are useful for practitioners in the field. We address concerns and questions below:\n\n**Concerns**:\n\n1. **Novelty**\n\nWhile we do not propose entirely new methods ourselves, the novelty of this work is more on the comprehensive setup for comparing current resources for RLHF with LMs, and the findings supported by a large number of experimental results. To our knowledge, there is little prior work comparing and examining RLHF methods in detail with extensive empirical backing. Additionally, various findings in our work (e.g., the hardship in transferring reward model performance to policy model, the sensitivity to the prompt distribution in PPO) are the first instance of these observations/results being discussed publicly to the best of our knowledge. We believe such analysis should have novelty in itself and are valuable given the current status of LLM research, and believe that multiple reviewers do point out that our insights have practical use for researchers in the field.\n\n**Questions**:\n\n1. **Extending to multimodal models?**\n\nIn our work, we decided to focus on understanding popular methods and datasets across a range of evaluations, working on popular text-only models. Multimodal RLHF approaches are still very new compared to the language model space, and there are not as many available instruction tuning or preference datasets to explore. Nonetheless, we believe that exploring how our findings extend to the multimodal regime would be an interesting direction for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f03Z5hZXb8", "reviewer": "Reviewer_P8L3", "summary": "The paper identifies four core components in training from preference feedback (RLHF): labeled preference data, learning algorithm, reward model, and unlabeled training prompts; and conducts a study on the effect of each component separately to disentangle their contributions to performance.\nThe authors experiment with variations of each component and make several claims:\n1. There is a clear ordering of the impact of each component on performance: preference data quality > algorithm choice > reward model quality > targeted policy prompts.\n2. Synthetic data with (aggregated) per-aspect preferences works best.\n3. PPO outperforms DPO\n4. Increasing the reward model size or its training dataset size improves its performance on reward benchmarks, but has marginal effects on downstream tasks overall.\n5. targeted online training prompts can improve performance in domain-specific settings, but not overall performance.\n\nHence, they suggest a recipe with the best of each component.", "questions": "How did the authors select the preference datasets to compare DPO vs PPO in Table 2? One would expect to have them compared on all datasets as in Table 1 or to have a justification for the datasets that have been dropped/kept.", "limitations": "Although in the appendix, the authors adequately state the limitations of their work and its broader impacts.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The paper is well presented: the claims are clear, the experiments backing them are easy to identify, and the narrative is easy to follow.\n\nThe description of the experimental setup is comprehensive and very detailed, helping the applicability of the paper.\n\nThe authors experiment with a comprehensive collection of preference datasets and benchmarks.\n\nThe authors make several interesting observations from their experiments such as the effects f the preference datasets used showing mostly on dimensions such as truthfulness and instruction following, or the improvement of reward modeling showing in rewards benchmarks but not translating to downstream tasks.", "weaknesses": "I find that the claims made in the paper often deviate from the experimental support provided, they are either overstated compared to the scope or the experiments, or inaccurately reflect the scope of the experiments. Specifically,\n\n**Algorithm choice:**\n\nAlthough comparing components such as prompt datasets is possible when keeping all else the same (algo, hyperparameters, etc), the claim \"PPO outperforms DPO\" cannot be made without carefully considering hyperparameters.\nThe authors have a single hyperparameter configuration for each model, and although they may have been validated in other settings, they have not been optimized (within a fixed but same budget) for each algorithm on the author's setting.\nThe DPO hyperparameters were originally used by Zephyr-Beta, and taken without tuning for TULU 2 [Appendix F.1, 22].\nFor PPO, apart from the number of epochs that have been cut to 1, it's not clear how the authors obtained the rest of the hyperparameters.\n\nFurthermore, from Table 2, \"PPO outperforms DPO\" _on average_ on _pre-selected_ datasets. When looking at individual benchmarks DPO models outperform PPO models in quite a number of tasks, and Table 2 only includes a subset of datasets which has not been motivated in the paper.\n\nTherefore I believe that the claim is misleading.\n\n**Preference dataset**:\n\nOnly Table 1 with DPO is used to formulate the conclusions on the best preference dataset. For the claim to hold in the scope of the paper (considering its limitations with a single pre-trained model) the preference datasets have to be compared with at least multiple algorithms (and ideally hyperparameters).\nTable 2 provides some results for PPO but on all datasets, so it fails to complement Table 1.\n\nThe authors do not provide insights about why synthetic data performs best. It is appreciated to observe such difference in performance, but the claim would have more value and impact with some analysis or intuition as to why it would hold.\n\nThe claim \"Preference-based learning has the strongest effect on instruction following and truthfulness performance.\" may only be valid in when the preference data itself mainly includes preferences based on instruction following and truthfulness, which is the case in the datasets considered in the paper. However, it may be very valid, but inefficient, to train factuality using preferences.\nThe scope of such a claim should be carefully explicited.\n\n**Reward model**:\n\nAlthough the paper aims to disentangle the effects of each component in RLHF, the experiments it conducts on some components tend to depend on the results from the previous components.\nE.g. the reward model experiments are only carried out on the preference datasets providing the best performance, giving a biased view on the effects of the reward model and does not allow to formulate an ordering on the performance contribution of each component independently.\n\nAlso, the average improvement of 0.6 from the 13B to the 70B UltraF. reward model has been qualified as marginal while the improvement of 0.7 from DPO to PPO has been qualified as significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and noting that our work is clearly written, with an easy to follow narrative, detailed experimental setup, and multiple interesting observations. We clarify key points of our experimental setup and choices (e.g. datasets) below, which hopefully address your key concerns. We will add these additional details and clarifications to our updated paper.\n\n**Concerns**:\n\n1. **Algorithm choice & Hyperparameter choice**\n\nIn particular, for DPO, we tested values of {0.1, 0.01, 0.001} for beta {5e-6, 5e-7, 5e-8} for LR for both HH-RLHF and UltraFeedback. For PPO, we experimented with KL penalty coefficients of {0.01, 0.025, 0.0325, 0.05}, as well as varying the batch size (including taking more samples per gradient step, and taking multiple steps over the same generations). We discuss some of the details around our hyperparameter choices in Appendices F and G, and will explicitly add the details given above. For PPO hyperparameters we did not search extensively over (clip threshold, discounting factors, learning rates, adam betas), we borrow primarily from InstructGPT [2] and Llama-2 [3], and open-source repositories (Rainier [4], Finegrained-RLHF [5], AlpacaFarm [6]), which are common reference points for RLHF work. We provide additional notes on hyperparameter choice and additional observations using PPO in Appendices F and G. As such, we believe we made a reasonable attempt to tune both DPO and PPO hyperparameters independently, although we note that further performance gains could be made by tuning on a per-dataset basis.\n\n2. **Dataset choice in Table 2**\n\nWe apologize for not explaining the dataset choice in Table 2: we explicitly chose the top-performing dataset of each source from Table 1 (StackExchange from Web, HH-RLHF from human, Ultrafeedback from synthetic). We also include the second-best performing dataset overall (Nectar) and an additional human-annotated dataset from the popular evaluation platform (Chatbot Arena; we chose 2023 over 2024 to reduce computational cost). We will add these details to our updated manuscript.\n\n3. **Why does synthetic data perform well?**\n\nWe hypothesize that the synthetic data we test performs well for a few reasons: (1) annotation quality is generally high due to the use of GPT-4 as a judge [1] and GPT4 labels may be hypothetically more consistent, (2) the prompts chosen cover a wide range of topics useful for downstream evaluations (math, reasoning, chat, coding, etc.), (3) there is some alignment between synthetic preferences and our evaluation setting - for example, Chatbot Arena data is human-sourced, but shows a clear degradation in safety which hurts its overall performance, and using AlpacaEval in our evaluation means we have a slight bias to GPT-generated data (although this is not the case for the many other evaluations we examine). We believe that further exploring how well synthetically generated data compares to expert-level human annotations, and exploring differences between the two, is an interesting avenue for future work.\n\n4. **Preference-based learning’s effect on factuality.**\n\nWe agree and will adjust the wording to make it clearer that our observations are based on the datasets we consider in our introduction, conclusion, and section 3. We agree that a carefully-made factuality dataset may indeed improve performance (even if it is non-optimal compared to SFT-based approaches).\n\n5. **Identifying the performance contribution independent of other components.**\n\nWe agree that there may be ways in which the contribution of a different component varies based on the setting (for example, some datasets not tested may scale better than those tested for the reward model experiments). Due to computational constraints, we did not test every single possible combination of {rm size, prompt, dataset, hyperparameter choice}, but focussed on running ablations using the best choices from the previous steps (e.g., focussing on UltraFeedback for PPO ablations, and using the best datasets found for DPO for comparing to PPO).\n\n6. **Marginal vs non-marginal improvement**\n\nWe agree that both changes give improvements of similar size, although we believe that the PPO claim is slightly stronger due to being a result of tests across multiple datasets (in fact, the p-value for the observed difference is .04 under a two-tailed paired t-test), while the larger RM size increase in performance was only from one experiment. As such, we have less confidence in the claim that the larger RM helps, although the boost on GSM8k appears large. We will adjust the wording in each section accordingly to better match this (explicitly noting the statistical significance of the DPO & PPO result, and noting that the jump in performance from using the larger RM is of similar scale, but is from a single run and driven largely by GSM8k improvements).\n\nAdditionally, we were somewhat surprised that a 10x increase in RM size (and almost 2x increase in best-of-N improvements, from 5.8 to 10.3) did not yield similarly large changes in the overall performance, and this coloured our reporting. We will adjust our wording accordingly (small changes in light of large BoN & compute increases).\n\n**Questions**:\n\n1. **How did the authors select the preference datasets to compare DPO vs PPO in Table 2?**\n\nPlease see our response to concern 2 above.\n\n---\n[1] Zheng et al (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS.\n\n[2] Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.\n\n[3] Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv.\n\n[4] Liu et al. (2022). Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering. EMNLP.\n\n[5] Wu et al. (2023). Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. NeurIPS.\n\n[6] Dubois et al (2023). AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their comments and feedback. We are happy that reviewers have noted our results and findings are of interest to the community (mnCL, WxZd, wF89), and enhances understanding of RLHF (mnCL), with comprehensive experiments/datasets (P8L3, mnCL). Additionally, we are happy that reviewers noted that we describe our experimental setup comprehensively (P8L3, wF89), and that the paper overall is clearly written and presented (mnCL, wF89, P8L3).

We address some common concerns below:

1. **We base our work on one pretrained model + More ablations & experiment combinations could be useful in further validating our hypotheses** (WxZd, mnCL, P8L3).

First, our end goal is to develop a recipe for strong open model performance, using publicly-available resources (hence our choice of Tulu 2, a state-of-the-art open LM with available SFT checkpoints). Focussing on 1 model allows us to explore other factors (such as dataset choice, RM choice, algorithm choice, etc) more cleanly and without larger computational costs. We aim to be comprehensive and varied in testing RLHF datasets (testing 12 varied datasets) and ablations of PPO itself (testing 4 different RMs each with 4 different prompt sets - GSM8k, mined prompts, mixed, and UltraFeedback, alongside the PPO experiments across varied datasets). Additionally, we note that prior work exploring RLHF methods often similarly focus on one model [1,2,3,4, inter alia]. We agree that further exploring how our results apply to other base models would be interesting, although it would require a large computational budget due to the number of ablations to run. Relatedly, our computational limitations meant we could not run every possible ablation (for example, we could not do many PPO training runs with a 70B RM, and so had to be selective in what to run there), and we note that reviewers generally agreed that the findings in our work are of interest to researchers in the field. However, we do additionally report results applying our recipe to Llama 3 8B, first finetuning Llama 3 on Tulu 2 and then performing DPO/PPO training with UltraFeedback and the hyperparameters given in our work (we use an 8B and not a 70B RM due to computational limits):

| Model | factuality | reason. | coding | inst. foll. | safety | truthf. | avg |
|- | - | - | - | - |- |- | - |
| Llama 3 + Tulu 2 SFT | 58.0 | 58.6 | 56.4 | 42.6 | 92.8 | 59.2 | 61.3 |
| +DPO	| 59.4 | 56.2 | 55.6 | 50.4 | 91.7 | 71.4 | 64.1 |
|+PPO	| 59.5 | 57.0 | 55.9 |56.0 | 91.4 | 69.6 | **64.9** |

Similarly to our prior results, we find better performance using PPO with UltraFeedback, and both approaches provide improvements over the SFT base model. Additionally, these models outperform our llama 2-based models on average. We believe further extending our work to other models, or to multimodal models (as pointed out by reviewer wF89). Finally, we additionally provide a PDF with this general response detailing an investigation looking at the performance of a 13B Tulu 2 model trained using PPO and a 70B UltraF. RM with UltraF. prompts over the course of training, showing how different evaluations are affected by our PPO training.

2. **Ensuring sound experimental setup and matching claims** (P8L3).

We made best-effort attempts to tune both PPO and DPO and compare them in a fair manner, ensuring that we had solid hyperparameters for both without doing a grid search for every experiment (important as PPO experiments took ~ 3 days to run on our compute). We performed much initial tuning of both PPO and DPO in initial experiments on HH-RLHF and UltraFeedback, and provide details in our response to reviewer P8L3 (concern 1).

Additionally, we clarify that our claims are made in the context of our empirical work, and will make this context more explicit in the text. Our intention is to explore the performance of two popular RLHF methods using public data (within our computational budget), and so our claims hold strictly only for the datasets and models tested, although we did our best to cover as many datasets as feasible with our computational budget (13 for DPO and 5 for PPO), as well as further PPO settings (4 different reward models up to 70B in size). We will clarify our claims where reviewers have pointed out they may be unclear.

---
[1] Sun et al. (2024). SALMON: Self-Alignment with Instructable Reward Models. ICLR.

[2] Shen et al. (2024). The Trickle-down Impact of Reward (In-)consistency on RLHF. ICLR.

[3] Wu et al. (2023). Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. NeurIPS.

[4] Cui et al. (2024). UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML.

Meta-review: The primary concerns raised by the reviewers related to the limitations in the breadth of the experiments and the depth of analysis. However, these concerns were largely addressed by the additional experiments during the rebuttal period.  The authors have argued that the constraints were due to computational limitations, which were managed effectively to extract meaningful insights from the experiments conducted. Moreover, the empirical results obtained are still highly valuable. The observations made in this paper are practical and provide a good foundation for further research. We recommend the paper for acceptance.
Decision: Accept (poster)