RLHF Workflow: From Reward Modeling to Online RLHF
Authors: Hanze Dong
Wei Xiong
Bo Pang
Haoxiang Wang
Han Zhao
Yingbo Zhou
Nan Jiang
Doyen Sahoo
Caiming Xiong
Tong Zhang
Venue: Accepted by TMLR ()

Abstract: We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature. However, existing open-source RLHF projects are still largely confined to the offline learning setting. In this technical report, we aim to fill in this gap and provide a detailed recipe that is easy to reproduce for online iterative RLHF. In particular, since online human feedback is usually infeasible for open-source communities with limited resources, we start by constructing preference models using a diverse set of open-source datasets and use the constructed proxy preference model to approximate human feedback. Then, we discuss the theoretical insights and algorithmic principles behind online iterative RLHF, followed by a detailed practical implementation. Our trained LLM  achieves impressive performance on LLM chatbot benchmarks, including AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning (SFT) and iterative RLHF can obtain state-of-the-art performance with fully open-source datasets. Further, we have made our models, curated datasets, and comprehensive step-by-step code guidebooks publicly available.

Published in Transactions on Machine Learning Research (09/2024)
RLHF Workflow: From Reward Modeling to Online RLHF
A Comprehensive Practical Alignment Recipe of Iterative Preference Learning
Hanze Dong 1+
Wei Xiong2
Bo Pang"
Haoxiang Wang"*
Han Zhaoa
Yingbo Zhou-
Nan Jiang
Doyen Sahoo!
Caiming Xiong"!
Tong Zhang?1
Salesforce Al Research
University of Mlinois Urbona- Catlmaa-haamarggn
Reviewed on OpenReviow:
Abstract
We present the workflow of Online Iterative Reinforcement Learning from Human Feed- back (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by large margin in the recent large language model (LLM) literature. How- ever, existing open-source RLHF projects are still largely confined to the offiine learning setting. this technical report, we aim to fill in this gap and provide detailed recipe that easy to reproduce for online iterative RLHF, particular, since online human feed- back is usually infeasible for open-source communties with limited resources, we start by constructing preference models using a diverse of open-source datasets and use the con- structed proxy preference model to approximate human feedback. Then, discuss the theoretical insights and algorithmic principles behind ouline ilerative RLHF, followed by detailed practical implementation. Our trained LLM achieves impressive performance on LLM chatbot benchmarks, including AlpacaEval-22, Arena-Hard, and MT-Bench, as well as other academic benchmarks such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning (SFT) and iterative RLHF can obtain state-of-the-art performance with fully open-source datasets. Further, we have made our models, curated datasets, and comprehensive step-by-step code guidebooks publicly available.
1 Introduction
Reinforcement Learning from Human Feedback (RLHF) (Christiano al. 2017 Ziegler et al., 2019) has become key technique for integrating human preference signals into machine learning methods. Tn particular, RLHF has become standard component in the post-training pipe line of foundation Large Language Models (LLMs), which serves to align the outputs of these models with human values such as helpfulness, harmlessness, and honesty (Ouyang et al.) 2022 Bai et al.] 2022a).. Notable examples include the revolutionary closod-source ChatGPT (OpenAl, 2023). Claude (Anthropic 2023 and Gemini (Taam
let al. 2023), as well as the powerful open-source models like Zephyr (Tunstall et al. 2023 Starling (Zhu let 2023), and LLaMA-3 (Meta) 2024). In particular, since the introduction of ChatGPT, RLHF has attracted significant interest in diverse set of communities. However, compared to the supervised fine- tuning that is rather well studied with many great open-source projects like Open-Hermes (Teknium 2023 and Vicuna [Zheng el al. 2023). RLHF remains relatively under-explored within the open-source communily.
"The first four authors core contributors to this project listed in random order. The full authorship contribution state- ments are provided in Appendix A Email: dong, b.pang, yingho. zhou, zhou, dsahoo, exiong) Sxiong)eealesfors www.esooree.eeccooooooooooooooe. con, (wx13, hwang264, hanzhao, nanjiang, fozhang)@illinag edu. Prroeec leads.

---

Published in Transactions on Machine Learning Research (09/2024)
To facilitate discussion, we build upon the standard RLHF workflow (Quyang et al. 2022 Bai et al, 2022b; Touvron et. al. 2023). We characterize an LLM by policy which takes a prompt a E X and produces a response a E A from the distribution T(-1z). We denote the initial model of RLHFF as no, which is fine-tuned on some iestruction-follwiint data after the pre-training stage. We assume that we have a prompt set that sampled from some unknown but fixed distribution x r~ dg-
The central principle of RLHFF to learn from relative feedback, rather than an absolute reward signal, is common in traditional RL literature. This approach is preferred because human raters often struggle to provide accurate absolute ratings; instead, they find it easier to choose between two options, indicating which response they prefer (Christiano et al. 2017). This idea can further date back to the study of ducling bandit (Joachims et al.) 2007; Yue et al. 2012) in the context of online decision making. Specifically, we assume that we have access to a Preference Oracle, that is a proxy of a real-world human rater. Mathematically, we have the following formal definition.
Definition 1 (Preference Oracle). There exists preference oracle P:: : A [0, 1], and we can query it to receive the preference signal:
~ Ber > a² a²|z a' ,d)),
where Ber(l) is Bernoulli distribution with parameter and meons a² is preferred to a², and y 0 means that a² is preferred.
To further simplify the problem, it is commonly assumed that the preference signal can be modeled using the reward-based Bradley- Bradley-Terry model, a well-known approach in preference learning (Bradley & Terry 1952:
Ouyang et al. 2022 Bai et al. 2022a Touvron et al., 2023
Definition 2 (Bradley-Terry Model). There crists ground-truth reward function r* and the preference
model satisfies:
(x,(r"(z,a') P(a' a² "lr,,a',")) exp(r*(z,a))) + (x,(r"(x,
(1)
where -= 1/(1 exp(-2)) is the sigmoid function.
Although the BT model may not fully capture the complex human preference, tends out to be a useful approximation connect the learning objective of RLHFF with reward maximization and has achieved tremendous success in making ChatGPT [Ouyang et al. 2022) and Claude (Bai et al. 2022a Mcanwhile, in response to the imperfect nature of BT model, the goal in this RLHF formulation is to optimize the following KL-regularized target:
To(a|z)]
J(x) = Exnd, (a
r" + oog 1ogg [[aaz))
(x. (r. Er [a)
- (Taa) (2)
where >>>00 0 the KL. penalty coefficient. This formulation is widely studied in practice (Ziegler et al. 2019
Wu et al. 2021 Ouyang et al., 2022 Rafailov et al. 2023 Liu et al. 2023a Xiong et al. 2023 and admits
the following intractable fosed-form solution (Zhang
2023
x <'(a2z) = ((x) To(a|z) exp (_r (x,
(3)
where Z(x) = Eo'es Ta(a'[z) exp (2, (z) is the normalization constant.
In the subsequent subsections, we first describe the existing approaches, and discuss their challenges, which should serve as the motivation for our project.
1.1 Previous RLHF Algorithms and Their Challenges
Generally, previous RLHF methods can be largely divided into two categories: (1) deep RL-based approach
using Proximal Policy Optimization (PPO) (Schulman al. 2017 Christiano et al. 2017 Ziegler et. al.
2

---

Published in Transactions on Machine Learning Research (09/2024)
2019
and (2) (offline) direct preference learning (e.g., DPO) approaches (Zhao et al. 2023 Rafailov et al,
2023
Azar et al. 2023 Tang et al. 2024).
DRL-based framework. The DRL-based framework consists of two stages. In the first stage, a reward model is trained. Specifically, given a preference dataset Dor = {(x, a", ,.)}, where a" is a response preferred over at given the instruction or prompt 2. The log-likelihood function of the BT model can be expressed as:
EDn (0)
log (o(ra(z,a") ( ro(u,a')))
(4)
(s,0 Po)
We can compute the maximum likelihood estimator (MLE) TMLE based on Dotr by maximizing the &D.. (0). In the second stage, DRL methods like PPO can be applied optimize against the following reward:
f(x, 0) = FMLE(E, a) - v) log <<aa²)
xo(a|ar)
This approach has been employed by ChatGPT (Ouyang et al.
2022 and Claude (Bai et al. 2022a) and 2023 However, it is known that even in
has contributed to the aligument of LLaMA-2 [Touvron et al.
the best case, tuning the DRL method to its best performance requires extensive efforts in hyper-paramett
selection and code-level optimization [Choshen et al.) 2019 Engstrom et al. 2020). This becomes even more challenging in the context of LLMs, as fine-tuning LLMs is computationally expensive and searching the complicated hyper-parameters configuration is generally infeasible. Additionally, the PPO algorithm requires loading multiple LLMs simultaneously, including the actor (policy), critic (value network), reward model, and reference model (for KL cstimation), which places significant pressure on GPU memory, especially for resource-consttaiieee open-source projects.
Direct preference learning. In view of the above issues of PPO, there is an innovative line of work that directly learns from human preference datasets without explicitly constructing a reward function (Zhao et al. 2023: Rafailov et al. 2023 Azar et al. 2023). Among these methods, the direct preference optimization (DPO) algorithm is particularly popular. It leverages Equation 3 to formulate reward as a function of policy and directly optimizes the following loss function using the preference dataset Dotti
logo o (1log (a(a"l>) (a(a=[>) 17 log To(a' ]-
To(a To(aa|=))
Doaa (0,
Σ
(5)
(s..",ol)EDor
In the ideal case where there is no optimization error, the minimizer of Equation 5 is the same as the two-staged DRL framework (Rafailov et al. 2023 Azar et al. 2023). Meanwhile, direct preference learning algorithms are generally easier to tune and require fewer computational resources compared to DRL methods. Considering these factors, in our project, we focus on direct preference learning algorithms while leaving the study of the DRL-based framework for future research.
The open-source project Zephyr (Tunstall et al. 2023) serves as a milestone to popularize the DPO algorithm, where the authors provide a comprehensive guide for training LLMs through vanilla DPO and distillation from the teacher model ChatGPT. Following the Zephyr framework, many open-source models have been fino-tuned using vanilla DPO and their qualitics are largely improved compared to their SFT counterparts. Impressively, on various LLM benchmarks" leaderboards, such as those reported by (Dubois el. al. 2023 Zheng et al. 2023). models fine-tuned using DPO predominantly exhibit superior alignment and effectiveness.
While the vanilla offline direct preference lcarning algorithms are useful in some case studies, they also face certain challenges. Specifically, they are considered offline because they learn from an offiin preference dataset collected prior to the alignment process. We can formulate this data collection process as:
I I do, a' ~ Tb.a² ~ y Ber > a² ,,)",,',a²))
(6)
Here, (= 7 represent two behavior policies, often taken as To, other soureed models or proprietary models. The term "offline learning" implics that we cannot further query the preference oracle during the training process. However, the finite dataset Dor fails cover the entire prompt-response space and the resulting policy model often performs poorly when faced with out-of-distribution data. (Burns et al. 2023
3

---

Published in Transactions on Machine Learning Research (09/2024)
Preference Data
Prompt Collection
X
a
a²
x
y
Update
Policy Model
Prompt
Responses
Preference
Parameters
y
(LLM being aligned)
Training
Generate
responses
Output
Preference
Reward Model
Reward
a
a
Model
Reward Modeling
Online Iterative RLHF
Figure 1: A simplified illustration of reward modeling and online iterative RLHF.
In partieular, along the way of the RLHF training, the average density ratio T)(a exp(25) reported in Figure 13 of Bai et al. (2022a). Therefore, the distribution shift between policies usually very large, and it unlikely that we can learn the optimal policy solely from a pre-collected dataset.
1.2 Online Iterative RLHF
In contrast, the Claude project (Bai et al. 2022a and LLaMA-2 project Touvroo el al. 2023 have demon- strated that online iterative RLHF can significantly improve model performance. The process of online iterative RLHF, formally formulated in Xiong al. (2023 be summarized follows. Given the pre-collected preference dataset D = Dot (il applicable, otherwise empty), for each iteration / E [T]:
we first update the policy pair (m),,?) based on the historical data collected so far;
we collect m tuples as Dj: sample a random prompt by 24,s ~ da, collect two responses by (a/,5 af,))
(m), mf",, and query the preference signal Ve.s ~ P;
update t DUD,
The effectiveness of online data can be intuitively understood continuously collecting new online data to strategically explore the prompt-response space and mitigating the out-of-distribution (OOD) issue. We also refer readers to Xiong et al. (2023) for a detailed theoretical explanation for online iterative RLHE. The hybrid formulation presented here mainly for generality and motivated by the LLaMA-2 project Touvron et al. 2023). This formulation also loosely related to some of the previous RI. literature like Xie
et al. 2021);; Song et al. (2022).
We notice that the results presented in LLaMA-2 and Claude are based the deep RL method, PPO, and the data, models, and training details are not fully accessible to the open-source community. Morcover, compared to the vanilla oflline DPO, its online iterative version still largely under-explored in the literature. Xiong et al. (2023) made the first step towards understanding the advantage of online exploration DPO training from thcoretical perspective, and the main purpose of this work to provide a detailed guidance make the online iterative RLHF pipeline more accessible to the opun-source community that others easily reproduce.
4

---

Published in Transactions on Machine Learning Research (09/2024)
1.3 Human Feedback Approximation
Ideally, the online preference signal is sampled from a representative group of human labelers. However, human feedback is extremely expensive in practice, which the open-source community usually cannot afford. In the literature, there is a line of work showing that training & proxy preference model, and using the preference model to give proxy labels in a semi-supervised manner improve the model performance (Dong
2023 Yuan et al. 2023: Liu et al. 2023a Hoang Tran 2024). We conjecture that this because the reward model (discriminator) usually generalizes better than the policy (generator).
et al.
In particular, Hoang Tran 20024) shows that if the preference model (reward model) is trained on diverse set of preference datasets, the Pair-RM (Jiang et al. 2023) with only 0.4B parameters
provide iterative preference learning with meaningful signals so that the resulting mode achieves an improssive AlpacaEval-2 length-control win rale of 26.4%. Motivated by this line of work, we first train a proxy preference (reward) model based on the diverse open-source preference datasets in Section 2 and then use the resulting model to provide preference signals for the subsoquent itcrative RLHF.
1.4
Related Work
We have presented many related works in the area of RLHF in the previous subsections. For completeness, we also include a more comprehensive literature review in this subsection.
RLHF and RLHF algorithms. The idea of learning from relative feedback could date back to the study of dueling bandit. (Joachims et al.) 2007; Yue el al. 2012) and the current RLHF framework was first popularized by Christiano et al. (2017). which served to direct the attention of the deep RL community to the baaeed feedback. Then, these techniquos were further introduced to finc-tune LLMs for the summarization task (Ziegler et al. 2019 Stiennon et al. 2020 The dominant RLHF framework used for the modern LLM alignment was first thoroughly developed in Instruct-GPT (ChatGPT) (Ouyang et al. 2022), Claude (Bai et al., (022a), and LLaMA-2 (Touvron et al. 2023) projects. These works typically involve constructing a reward model based on the MLE of the Bradlcy-Terry model, and then using the PPO algorithm to optimize the reward signals with KL, regularization. One notable exception is that the
LLaMA-2 uses a mixture of rejection sampling fine-tuning (Dong et al.
2023 Wang et al. 2024) and PPO in their RLHF pipcline. We refer the intcrested readers to Bai et al.
[202aa) Touvron et al. (2023) for
detailed description. However, the use of PPO in RLHF has limitations. It is known to be unstable (Choshen)
ct al. 2019), sensitive to implementation (Engstrom et al. (2020). and resource-intensive (Yuan et al. 2023)
Despite some efforts to improve PPO in the context of RLHF (Li et al. 2023] Chan et al. 2024 Chang ett al.) 2024 Zhong et al.) 2024), reproducing the successful results achieved with PPO is challenging for the open-source community due to these limitations as requires extensive efforts and resources that the open-source communitics usually cannot afford. In recognition of these issues of PPO, a line of work studies the (offline) direct preference learning algorithms, including Slic (Zhao et al. 2023) DPO (Rafailov et al.,
2023 IPO Azar et al. 2023), KTO (Ethayarajh et al. 2024). ARM (Pang et al. 2024), and GPO (Tang) ct al. 2024 These algorithms skip the reward modeling step, and optimize a designed loss target on the offline preference dataset directly (hence the name). It is widely observed that the direct preference learning algorithms are much more stable than the PPO, and achieve impressive performance evaluated by standard
benchmarks (Tunstall et al. 2023 Dubois et al. 2023 Zheng et al. 2023
Despite the advances offered by these direct preference learning algorithms, their implementation is typically offline and off-policy. This means they operate on preference datasets that were previously collected by other models often powerful, proprietary teacher models like ChatGPPTbbee the training process begins
(Tunstall et al. 2023 Cui et al. 2023).
RLHF benefits from online (iterative) learning. Roughly speaking, online iterative learning means that we will deploy the intermediate models and query human feedback for the responses of these models. Intuitively, this strategy can help to mitigate the OOD issue of the learned reward model (Gao el al., 2023). and its advantages have been reported in Ouyang et al. (2022); Touvron et al. (2023) for the PPO- based framework. Even when the additional feedback is derived from proxy reward constructed from the same
5

---

Published in Transactions on Machine Learning Research (09/2024)
X
BT Model
1
0r
a'
Pola a = dr, 12)
x
BT Model
a²
o: Sigmoid Function
X
Preference Model
a
Z0
Po(a > a² x a²) = o(zg)
0
a²
Figure 2: Illustration of the Bradley-Terry (BT) model and preference model.
offline dataset (similar to semi-supervised learning), iterative rejection sampling fine-tuning (RAFT) Dong ct al. 2023 and DPO based on samples from the target distribution estimator have been shown to outperform
the original offline counterparts (Pang et al., 2024 Liu
al.
2023a Furthermore, recent works Xiong
ct al. 2023 Xu al., 20236 Hoang Tran 2024 Yuan al.) 2024b Swamy al. 2024 Chen et al. 2024b
Ye al. 2024 Guo et al., 2024 Rosset
2024 Tajwar al., 2024 Calandriello et al. 2024 Wu et al.
2024) have demonstrated that online iterative variants of direct preference learning algorithms significantly outperform their offline counterparts. In particular, we refer the interested readers to Guo et al. (2024) for the extensive experimental results with different offline base algorithms.
2 Reward Modeling as Human Feedback Approximation
We present the details of preference model construction in this section, where we study both the reward modeling as MLE of the BT model and the general preference model. We consider two versions of the training set: MIX1: HH-RLHF + SHP + UltraFeedback + Summarization, which is the mixture dataset used to train the current state-of-the-art open-source model. (Stiennon et al. 2020), and MIX2: all the open-source datasets we collect, with details provided in Table
2.1 Tradley-Terry Reward Model and Preference Model
Bradley-Terry model construction. We follow the previous works (Oyang et al. 2022 Bai et al. 2022a initialize the reward model using the SFT mode We replace the last layer with a lincar head predict scalar score suitable for preference learning. The reward model is trained using the negative log-likelihood loss function, enabling maximum likelihood estimation (MLE). This loss function is defined
(R) = "BB.aa,a''iDooo. l an") ro(xxaa)))
where a is the preferred response over a'. We train the LLaMA-3-8BB-bar reward model for one epoch with a global batch size of 512. The learning rate is set Ir 2 10-6, and cosine learning rate schedule with a warm-up ratio of 0.03 employed.
2 For preference ceceenncerrewarr modeling, we meta-llama// Meta- Laaita--aaaa------ 3-8B-Instrued since only this checkpoint available in the early stage this project.

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Comparison of the test accuracy between the Bradley-Terry (BT) reward model and the preference
model. We evaluate the model using the Reward-Beni Bench [Lambert ct al. 2024).
Base Model
Type
Data Mixture
Chat
Chat Hard
Safety
Reasoning
LLaMA-3-8B-it
Prompting BT
93.6
44.3
71.3
73.5
LLaMA-2-13BB
mix1
96.4
55.5
55.0
62.4
LLaMA-3-8B-H
BT
mix2
99.4
65.1
87.8
86,4
LLaMA-3-SBitt
Preference
mix2
98.3
65.8
89.7
94.7
Preference model construction. A (pairwise) preference model takes prompl. and two responses a',a² as the input and predicts the probability of P(a' > a² a²|zz, a', a²) (Jiang et al., 2023). We follow Zhao next-token predictor for preference et al. 20233: Liu et al. (2023a) to leverage the LLM's capability modeling. Specifically, for given preference pair (z, a', a², where A indicates that the first response is preferred, the pair is formatted as an instruction-follloinng task:
instruction = [CONTEXT] {x} |RESPONSE A] {a²} JRESPONSE BJ {a²), and label A.
If the second response is preferred, we replace the label A with B. Then, we simply treat the preference modeling as an instruction-folloiing task to fine-tune the model on these instruction-label pairs. To mitigate position bias (the preference model may prefer the response that is given the position of RESPONSE A), the order of the responses randomized during data formatting. During inference, if we denote the probability of decoding A as PA and the probability of decoding as pB, then P(a a a²|x,, al a²²) is taken packed PA/(px Pg)- We train the LLLMA-3-8B-basce preference model for one epoch. The samples into blocks with length 3072 and global batch size of 128 is used. The learning rate is set to Ir = 5 x 10-6, and a cosine learning rate schedule with a warm-up up ratio of 0.03 is employed. We mention in passing that it is possible to include detailed rubrics in the data format to further improve the preference dataset, which we leave for future work (Qin et al. 2023
2.2 Evaluation Result
We evaluate the models using the RewardBench (Lambert et al., 2024), a benchmark designed to assess reward model capabilitics in four categorics: Chat, Chat-Hard, Safety, and Reasoning. The main evaluation results are Table is evident that without explicit training, the prompting approach inferior to both the BT model and the preference model across all metrics. Meanwhile, the preference model outperforms the BT model in reasoning tasks related to coding and math. We also notice that with more data, especially data specified in coding, math, and safety, the reward model trained by mix2 achieves higher accuracy in safety and reasoning compared with early versions of attempts. In particular, a reference model in Table [ where we can observe that the extra data related to safety and reasoning (as well as a stronger base model) largely contribute to the superior performance of our reward model.
use the Ultra-RMM-33B
Length bias in reward modeling. It known that the LLMs aligned by RLHF usually give longer responses (Xiong et al. 2023 Yuan et al., 2024b b), where the length bias also exists in the reward models, likely inffuenced by the preference data used. To better understand this bias, randomly sample 2K prompts from the prompt set and use the SFT model to generate 8 responses per prompt. Then, compute the lengths and rewards of the responses and plot the heatmaps of the Pearson corrolation cocfficient between them in Figure 3 Clearly, both of the two reward models are biased loward the longer responses to some degree. In comparison, UltraRM-13B demonstrates a stronger bias, as we observe that the mean coefficient of the UltraRM-13B (left) 0.19, while it is 0.06 for our BT reward (right). This may partially result from the use of additional Capybara, Oren,rea, and Ultralnteract, whose preferred responses are shorter than the rejected responses. We will return to the ablation study of the impacts of the reward models in Section
7

---

Published in Transactions on Machine Learning Research (09/2024)
1400
14
Mean response length
1200
12
Mean response length
14
1.00
1000
800
800
600
800
400
200
-0.75 -0.25
-1.00 -0.50 -0.25
Reward-length Pearson coefficient
Reward-length Pearson coefficient
Figure 3: The heatmap of the Pearson correlation coefficients between reward and response length. For each prompt, we use the SFT model generate 16 responses and compute the coefficient. We also group the prompts by the mean responses (y-axis). The left figure is the UltraRM-13B and the right is BT reward based mix2. We observe that the heatmaps for both UltraRM-13B and our BT reward model show tendency toward the positive side. This indicates that the reward models exhibit preference for longer responses.
3 Iterative Policy Optimization
We develop the main algorithms for the online iterative RLHF in this section, with both theoretical insights and implementation details. In particular, the algorithms are in direct preference learning style for stable and efficient training.
3.1 Supervised Fine- Tuning
The base model used in this project is LLaMA-3-8B. To ensure the reproducibility and openness of the project, we perform SFT by ourselves obtain the initial policy To. We collect a set of high-quality instruc- tion datasets for SFT, such as ShareGPT (Chiang et al.
2023). SlimOrea (Lian et al. 2023b). MathInstruct (sce the Appendix for full list.. The training is
Yue et al., 2023 and Evol-Instruct (Xu al.) 2023a
5, A cosine scheduler is employed, and the global batch size is set to 32 with warm-up ratio of 0.03. To accelerate training, we follow Diao et al. (023
carried out for
epoch with learning rate of 2 10
Tunstall et al. (2023) to pack the samples and use block size of 8192.
3.2 Iterative Direct Preference Learning: Theoretical Insights and Algorithmic Principles
From a theoretical perspective, one of the most critical metrics in the reinforcement learning process is the uncertainty estimator Т. For example, considering the lincar case, r= (0, ő(x ((x,a)), a)) such that E Rd, ||@] < B. For any two policies m7, T²,, we define the information gain as Ir m/. m/) = 3||E, o(x at) - a ll), (9 is constant coefficient), which is the projection of the new feature difference to the historical feature covariance matrix Σ, XI Σ(=1 ao [(0(x,a) o(I, a² (o(x))
o(x,a²))
We present the main algorithmic framework in Algorithm for the online iterative RLHF, and summarize the key principles and algorithmic ideas as follows.
Hybrid batch learning. We formulate a slightly more general framework to combine an initial offline dataset with online data collected during training, hence the name hybrid lcarning, similar to the recipe of Claude (Bai et al. 2022a) and LLaMA-2 Touvron al. 2023). We also use large batch size m for sparse updates.
8

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm 1 Theoretical Online Iterative RLHF with Enhancer
Input: offline dataset Desf (can be empty); batch size >> 0, and preference model PP, the number of iteration
T.
for t=I,...,T do
3: Exploitation with the main agent: denote the MLE TMLE (no need to explicitly compute the reward function
if use DPO) and compute the best guess we have so
argmax Rowdy (
a) (
mell
4: Exploration with the enhancer: given the policy, assume we have the uncertainty quantifier with respect to m.
as 12" (A, Ne, x and compute the policy Tे = argmax 1 (A, me, x²) where
Ile {n' E Il: Er (T(((())))))) (1z))
(A,
me
(8)
enhancer move away..
Information we can get.
5: Collect D, (2x,, at, oz, y1 yi) by 2, du, at ~ at (+12,), of m²(()x),) and y ~ Ber (P(at > af
6: end for
7: Output: the best policy in (wte)) by validation set.
Non-symmetric structure to balance exploitation and eaploration. The framework also features a non-symmetric structure by involving main agent and an enhancer.
Main agent aims to learn σ². Specifically, for each iteration, the first agent, referred to as the main agent, always takes the optimal policy under the ML.E FMLE of the historical data, which can be viewed as a fully exploitation of the information we collected so far;
Enhancer aims to assist the main agent's learning. Since the main agent solely exploits the historical data, it effective only when we can continuously obtain new information about the alignment problem from the newly collected online data or the oflline data Doff has provided enough coverage (which unlikely to hold in practice as we discuss in Section I.1]. The enhancer, therefore, explores in the direction where there is more uncertainty relative to the main agent's policy mf (measured by I (A, m), x))), while maintaining moderate KL divergence with m.
We have the following theoretical guarantees when strategic exploration methods applied.
Theorem 1 (Informal Theorem 2 in (Xiong et al. 2023 For any precision parometer > 0, with batch size m = =Ō(4) and other suitable choices of hupeer-prrumeters then, with high probability, after at most T T= (dd)) iterations, we can find a to E [T' so that -(())) m [DKL(T (-I2a.)llTT,
S c, where J(T) = Eg (aam((1a)[r"(6 (I. a)] - (x(((((() To [x)]] is the KL-regularized value. Here Ô hides some log factors and de is complesity measure of the RLHF problem. particulur, if the reward function can be embedded into a d-dimensional space that is linear in the feature map of @ X A Rdd we have
de d. (Zhong et al.) 2022 Liu et al. 20236
3.3 Practical Implementation Details
We now shift our focus from the theoretical insight the practical implementation. We provide an illustra- tion of our implementation in Fig. 4
The MLE policy. Since the main agent only exploits the data, we can run DPO on the historical data to approximate the optimal policy under the TMLe: TALEE We remark that while we use DPO here due to its simplicity, the Algorithm [ can be implemented by combining it with any oracle algorithms (e.g., PPO and InfoNCA Chen et al. 2024a) that are approximations the KL-regularized optimization problem.
Exploration policy. The primary challenge lies in the choice of enhancer policy for exploration. Recall that our goal is to find enhancer policy that maximizes the relative uncertainty to the main agent from the

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm 2 Practical Version of Online Iterative RLHF with BT Reward Model
Input: offline dataset Dotf (can he empty); batch size 0, rejection sampling parameter and reward model
r; the number of iteration T.
2: for t=I,...,T" do
3: Compute Ti by the DPO algorithm with Dor U Dia-1 using the SFT policy * reference model.
4: Sample a batch of prompls {ai]== from do. For each prompt, we sample n//2 responses using me with temper- ature 1.0 and n/2 responses using with temperalure 0.7.
5: For each prompt Ii, we rank them using r and take the best response and the worst one to construct a preference pair into D.. Eventually, we collect preference pairs.
6: end for
7: Output: the best policy in (rtr) by validation set.
confidence set defined in Equation Unfortunately, the uncertainty estimator does not have an analytical form except the linear case. But the main insight we can derive here is to maximize the policy difference with oH; while maintaining a moderate KT. divergence. This motivates us to use model variants of xt- We discuss some popular heuristic implementations here
Adjusting Temperature and Training Steps. In the project of Claude (Bai et al. 2022a), the authors choose to use the models with different training steps as (m] m,). For instance, if we run PPO for 2 epoch in total, we may take m/ as the model saved at the end of the first epoch and take π² as the one saved at the end of the second epoch. Additionally, the LLaMA-2 project (Touvron et al. 2023) adjusts the sampling temperature of के to induce xς. These modifications introduce diversity in the models and facilitate exploration.
Rejection Sampling is another popular ensemble- basede exploration approach (Nakano al. 2021 Dong et al. 2023 Gulcehre et al. 2023). In the context of LLMs, it is typically restricted to the best-of-n sampling. Specifically, we sample n independent responses by T/ for each prompt, and then use a wveference/rewarr function to rank the responses and take the one with the highest reward as the final output. In other words, we take oं as the best-of-n variant of op. In this way, the xद enlarges the margins between oi and provides exploration. Mcanwhile, in this case, the KI divergence between the two policies is upper bounded by log n-1 and is usually far better than this conservative estimation (Beirami et al. 2024
After the first submission of this work, there is a line of works proposing to use biased DPO loss in the online iterative framework to encourage exploration (Xic et al. 2024 Zhang ct al. 2024 Cen lt al. 2024), whose idea originates from the theoretical RL study (Xiong 2023 Liu et al. 2023b) Specifically, they add a SFT loss term, also known as the "feel-good" term, into the loss function so that the algorithm favors the model that is more optimistic. We refer the interested readers to these works for more detailed algorithm description and empirical results. We also integrate the option of adding such bias term in the revision of our public code.
In our experiments, we use the DPO to approximate the computational oracle and implement DPO with the open-source package T We run DPO with the reference model Zo (the SFT model) on the historical data for 2 epochs to get the MLE policy = MLE, We use cosine learning rate scheduler with peak learning rate of 5e-7 and 0.03 warm-up ratio. We use global batch size of 128 and use KL coefficient of 1 0.1. To accelerate training, we do not restart from at each iteration as in Bai et al. (2022a); Xiong et al. (2023) but use the last-iteration model as the initial checkpoint and use To as the reference model. In this way, the data used for training is the same as that of Bai et al.] (2022a) and Xiong et al. (2023) but is of a different order. We do not see performance regression with this choice, and it saves us for half of the training time.
To facilitate exploration, we combine the temperature tuning with the rejection sampling strategy with = 8. Instead of fixing m/ nn ,LE (like the center of confidence set) and optimizing the oद solely to be the best-of-8 variant of n. M,LLE, we take m/ and Ti as the best-of-8 policy and worst-of-8 policy induced by m
MLE
10

---

Published in Transactions on Machine Learning Research (09/2024)
Our Workflow of Iterative Direct Preference Learning Historical Data
a²
a²
SFT g
Add to
New Data
Prompt
Responses
Preference
Best-of-1 Response
DPO
a
best
Policy Model MLE
Sampling
Reward Model
Rating
a
x
Prompt
(Initialized from #;-1
a
worst
y
n Responses per Prompt
Worst-of-11 Response
Illustration of our implementation of iterative direct preference learning. In iteration = 1, the historical dataset is emply, and the resulting policy model TILL is the same as its initialization, To, which is the SFT model checkpoint. After that, the historical dataset grows with preference data collected from previous iterations.
Figure 4:
In other words, take the best response and the worst response as ranked by the reward model to get. preference pair. In this case, we jointly optimize the two policies to maximize their difference (measured by the uncertainty); which tends to be more efficient in practice and enjoys the same theoretical guarantee
stated in Theorem I This choice is similar to Hoang Tran (2024); Pace et al. (0224); Yuan et al. (2024b): Xu et al. (2024). We also drop the pair where मे and oे give the same response, which implies that the uncertainty in this direction is already small. For this round of experiments, we still use the reward function trained as the MLE of the BT reward model to rank the responses for the following reasons. First, to rank n responses, the complexity of using the reward model linear in n, while it far more complicated with observe significant length bias in
the pairwise preference model. Second, during the early experiments, the iterative RLHFF, Therefore, would like to explore the strategy to mitigate the length bias, and it is relatively easier to penalize the reward value with the length of the response. Finally, the BT reward model is comparable with the preference model except for the reasoning task and it may be already satisfactory for our goal. We leave a more comprehensive comparison between the BT reward model and preference model for future study.
Prompt set, and data generation. We collect prompts from UltraFeedback CCii al. 2023 HelpSteer (Wang al.] 2023). OpenOrca (Lian et al. 2023a), UltraInteract (Yuan et al.) 2024a). Capybara (Daniele & Suphavadeeprasi, 2023) and DIBT-10K and prepare the full prompt set.In our experiments,
use subset of 60K prompts and iterate for three iterations, so 20K prompts are used to generate 20K x 16 responses per iteration. To accelerate data generation, we use VLLM (Kwon et al. 2023] for inference. We set the max generation length as 2048, and use a sampling temperature of 1.0/0.7 without any top-k/top-p strategy. To offer more intuitive comprehension of our prompts collection, we provide visualization plots in the Appendix (Figure 5
11

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Evaluation results and comparison between the resulting models and existing models. means that the model is based the mixturo-of-cxperts architecture. We report the length-control win rate of AlpacaEval-2 as recommended by the authors. RS is short for rejection sampling (Dong et al. 2023 and X means that the value is unavailable. Only underline results are better than our 8B model.
Model
Size
Method
LC AlpacaEval-2
MT-Bench
Mhat-Arena-Haan
Gemma-7B-it
7B
SIT
10.4
6.38
7.5
Zephyr-7B-ben
7B
Vanilla DPO
13.1
7.34
Mistral-7B-v0..2-tt
7B
SFT
17.1
7.51
12.6
Open-Chat-0106
7B
SFT
15.6
7.8
Starling-BB-beta
7B
PPO
25.8
8.12
23.0
LLaMA-3-8B-it
8B
RS+DPO+PPO
22.9
8.16
20.6
Ours (SFT baseline)
8B
SFT
10.2
7.69
5.6
Ours (DPO baseline)
8B
Vanilla DPO
22.5
8.17
22.4
Ours (Iterative RLHF)
8B
Iterative DPO
31.3
8.46
29.1
Vicuma-33b-v/
33B
SFT
17.6
7.12
Yi-34B-Chat
34B
SFT
27.2
X
23.1
Mixtral-8x7B-it
45K
SFT
23.7
8,30
23.4
Tulu-2-DPO-70B
70B
Vanilla DPO
21.2
7.89
15.0
LLaMA-3-70B-it
70B
DSO
34.4
8.95
41.1
Mixtral--x22B-it
141B*
SIT
30.9
8.66
36.4
GPPT-3.5-turbo-II.
19.3
8.35
18.9
GPT-3.5-turbo-6613
22.7
8.39
21.8
GPT-4-0613
30.2
9.18
37.9
Claude-3-Opp
40.5
9.00
60.4
GPT-4 Turbo (04/09)
55.0
X
82.6
4
Evaluation of the Model
4.1
Benchmarks
We evaluate the models by standard benchmarks, including AlpacaEval-2, MT-Bench, and Chat-Arena--ard. Details are provided in the Appendix.
We also measure the ability of the resulting models using academic benchmark, including GSM-8K (Cobbe et al. 2021), MMLU (Hendrycks et al.) 2020), HurnaF (Chen et. al.) 2021), TruthfulQA (Lin et al. 2021), ARC (Clark et al. 2018), and MBPP (Austin et al. 2021). These benchmarks evaluate the models' ability in coding, reasoning, and general knowledge. particular, is known that RLHF alignment can introduce performance degeneration reasoning, calibration (providing accurate confidence estimates), and truthfulness capabilities (generating accurate and factual responses), which is also referred to the alignment tax in the literature (Ouyang et al.) 2022 Bai et al. 2022a OpenAl 2023). Therefore, evaluating our model on these benchmarks is crucial to understanding the impact of iterative RLHF on these specific aspects.
4.2 Main Results
Online iterative RLHF significantly improves conversation quality. We evaluate our model's con- versation abilitics using AlpacaEval-2, MT-Bench, and Chat-Arena-Hard, (results in Table 2 Compared to other open-source models with less than 10B parameters, our model outperforms them on the conversation -orruuttioo-ooooiiing
and instruction- following benchmarks with a significant margin. Notably, our model trained with iterative DPO consistently outperforms that of vamilla offiine DPQ (DPO baseline). This demonstrates the advantage of online iterative RLHF. Moreover, our model outperforms the which are aligned by DPO PPO and are much larger than our base model. These results show that the
and GPT-3.5-turb-- 1106,
12

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Evaluation results of the resulting model on academic benchmarks and comparison with other
open-accoss LLMs.
Model
Size
Method
GSM-8K
MMLU
HvmanEval
TruthfulQA
ARC
MBPP
LLaMA-3-8B-i
8B
RS RS-DPO+PPO
79.6
66.0
61.6
43.9
59.5
61.1
Ours (SPT bascline)
SFT
74.2
64.7
65.2
53.4
62.3
Ours (DPO haseline)
813
Vanilla DPO
79.8
64.5
63.4
61.8
65.2
60.3
Ours (Iterative RLHF)
Iterative DPO
80.7
65.3
64.6
60.4
64.3
60.8
online ilerative RLHF can effectively adjust the style of the model responses, thus improving the conversation quality.
Academic Task. As RLHF can impact model's reasoning and calibration abilities, typically in negative
way (Bai et al. 2022a Ouyang et al. 2022 OpenAT 2023). we compare our model's performance academic benchmarks (Table with the SFT chockpoint and other basclines. We don't 't observe significant performance regression compared to the SFT baseline. Interestingly, our iteratively DPO-aligned model even outperforms the SFT model in GSM-SK, MMLU, TruthfulQA, and ARC benchmarks. We believe that these increased capacities of the model are injected in the pre-training stage and SFT stage, and iterative DPO helps it leverage them more effectively. This is because the 60K alignment data used in the iterative RLHF are orders of magnitude less than those used in the previous two stages.
Remark 1. The RLHlF-oligned models based on some initial checkpoints and more epochs can approach or even outperform the state-of-thar ofte--of-theart closed-source models like GPT-4 and Claude on benchmarks. However, we remark that we should be more careful in interpreting these results because the test sets of the benchmarks are finite and may not be representative enough to capture the complicnted real-world scenarios. Moreover, the increased possibility of small models overfitting the benchmarks may lead to benchmark hacking, which means that the real capacity of model with high score is still limited. In porticular, while it is possible to get even higher results the benchmark (e.g., 44.84 in LCC AlpacaEval-2 and 35.7 in Chen--Arenahhart but the performance on academic benchmarks drops significantly); we presented our current model by human evaluation on some rundomly chosen Lest prompls. We also found that GPT-based evaluation highly depends on the configuration. Our model obtains 37.2 LC win-rate (45.4 win-rate) with config, which has better agreement with human evaluation.
Ablation study on filtering data with length penalty. We observed that the aligned model's response length was significantly longer than the SFT baseline (potentially due to reward model bias shown in Figure 3) To address this, we conducted an ablation study by incorporating a length penalty into the reward function:
F(x, a) f(x, a) Alal,
(9)
where |a| is the number of characters of the response. We compare the model trained with this penalty the vanilla version and report the results in Table As expected, the length penalty effectively mitigated the length bias, leading to shorter responses. In particular, the model trained with length penalty achieves superior length--onnrol AlpacaEval-2 win rate, as well as better results on some academic benchmarks. This demonstrates the advantage of mitigating length bias and motivates us to study the verbosity issue in reward modeling further. Finally, we notice that the model trained with length penalty is worse in the Chat- Arena-Hard benchmark. This may suggest that we also need a length-control version for this benchmark provide a more reasonable evaluation.
Table 4: Ablation study on the impact of reward models and length penalty in the online iterative RLHF The response length is averaged over the responses to the Ahen--rena-Haan Benchmark.
RMdM
Len. Pen.
LC Alp.
Arcua-H.
Len.
GSM-SK
MMLU
HumenEval
Truthfui
ARC
MBPP
Qurs
31.3
29.1
656
80.7
65.3
64.6
62.2
64.3
60.8
Oure-concise
0.001
38.1
22.1
382
78.8
65.5
06.5
60.4
65.1
62.4
UltraRM-13B
20.7
21.3
745
78.9
64.9
63.7
59.9
63.6
60.8
13

---

Published in Transactions on Machine Learning Research (09/2024)
On the impact of reward model. We investigate the effects of the reward (preference) model used in the online iterative RLHF. Our model's performance is compared to a model trained with UltraRM-13B, and the ablation study results are summarized in Table 4 We observe that the model trained with UltraRM-13B has longer responses than ours, which is consistent with its stronger bias, as shown in Figure 3 Considering the alignment lax; the accuracy on the academic benchmarks drops more than our models. One important reason is that the Ult3aRM-13BB does not have a good reasoning ability (sce Table D so it may not provide appropriate preference signals for reasoning-relate conversions. For instance, the model may favor some responses with many comments in the coding task, which tend to be very helpful but are indeed useless when evaluated by humans. Notably, the model trained with UltraRM-13B achieves a higher Chat-Arena-Haut win rate than our concise version, which also supports the verbosity bias of the Arena-Hard benchmark. During the training process, we also observe that the model trained with UltraRM-13B achieves a lower training loss, which may suggest that the signals of UltraRM-13B are more consistent and easy to learn. In contrast, the convergence under our reward model is slower due to the complex preference signal.
5 End Note and Future Direction
In this technical report, we study the workflow of the online iterative RLHF, which leverages on-policy sampling and external preference signals from a proxy preference model trained on a diverse set of open-source preference datasots. The resulting model demonstrates impressive performance on standard benchmarks, and the report provides detailed instructions for reproducing the results, including data, code, models, and hyper-parameter choices.
There are still many potential directions to explore. First, as we can see in Table + the iterative RLHF heavily relies on the quality of the preference signal. In this project, we use a proxy scalar reward model trained on a diverse set of open-source datasets to approximate human feedback. Tt would be interesting to see whether we can design a more effective strategy to model different types of preference signals, like a multi-head reward (Wang et al., 2024) and classification-based activation strategy (Touvron et al.) 2023) Second, while the rejection sampling seems to be good heuristic exploration strategy, it is still interesting to see whether can design more effective ways for exploration. Finally, most of the models after RLHF tend to reply the prompts with much longer responses. Such length bias is further amplified in the iterative RLHF framework. We presented a preliminary study on this issue by leveraging an additional length penalty in reward for data filtering. It would be interesting to see whether we can further mitigate this issue by additional algorithmic designs or post-training techniques.
We hope the results of this project can advance the direction of online iterative RLHF and contribute to the training of stronger and larger open-source LLMss
References
Anthropic. Introducing claude. 2023. URL http:///wwaaaatttoopccc//oooooaa
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint ar xrxi:22008.077 2021.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. general theoretical paradigm to understand learning from human preferences. arXiv preprint arXio:2310. 12036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv: 05866, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint ar XrX::22212.000 2022b.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models. orXiv preprint arXiv:2308. 0.5.3088.111,, 15812, 2023.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda. Theertha Suresh. Theoretical guarantees the best-of-n alignment policy. arXie preprint arXiv:2411.014 018899, 2024.
Ralph Allan Bradley and Milton E Terry, Rank analysis of incomplete block designs: 1. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-lo-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXix:2312. 09990, 2023.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human aligmment of large language models through online preference optimisation. arXiv preprint arXie:2403. 08635, 2024.
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivizer preference optimization: A unified approach to online and offline rlhf. Xiv preprint ar arx::2405. 19320, 2024.
Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint 202.00002, 2024.
Jonathan D Chang, Wenhao Shan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf. Xrxiv preprint arXiv:2204.08449 2024.
Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402 05369, 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethca Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, Dave Cumminns, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantamu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Πya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint 2xXx:2401.003333 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL httpss:/1mmss.orgg 2023-03-30-vicuna//
Leshemn Choshen, Lior Fax, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint 1rXiv:1907.01152 2019.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ail reasoning challenge. w:Xiu:1803.0555M00 2018.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXic:210. 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Luigi Daniele and Suphavadeeprasit. Amplify-instruc:: Synthetically generated diverse multi-turn conversa-
tions for effecient Ilm training. arxiv preprint arXiv:coming soon), 2023. URL https://huggiiggace.
Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. ai preprint or 2rvii:2306. 12420, 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang, RAFT: Reward ranked finetuning for generative foundation model aligmment. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:
//openreview pepparreviee..et/foouumdddddaaad net:/orrumi.dmmm75507000
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feodback. arXiv preprint Xrx::2230 14987, 2023.
Logan Engstrom, Andrew Tyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint ar 2iv:2000. 1 12729, 2020.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding datasct difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1988-6008. PMLR, 17-23 Jul 2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint Xiv:2402.01306 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMI.R, 2023.
Caglar Gulcchre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Wocrts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self- ceff-training (rest) for language modeling. Xi preprint arXiv:230800800 2023.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model aligmment from online ai feedback. arXiv preprint Xix::2400 792, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXie preprint arXiv:2009.0330 2020.
Braden Hancock Hoang Tran, Chris Glaze. Shookel-miistral-pairm------
https://huggingfant
http:///soooeee-sssaaa-------- Shorzee--Misttral-PPiPP----
2024. URL
Jiaming Ji, Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of IIm via human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Anaual Meeting of the Association for Computational Linguistics (ACL 2023), 2023.
Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS), 25(2):7-os, 2007.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM STGOPS 29th. Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. Xiv preprint orXiv:2433. 13787, 2024.
Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv e-prints, pp. arXiv-2310, 2023.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. httiep///ttts////nggggagaeeeeeee OpenOrca 2023a.
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023b. URL
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods.
arXiv preprint xrx::21009. 09907998, 2021.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309. 06657, 2023a.
Zhihan Liu, Miao Lu, Wei Xiong, Han Zhong, Hao Hu, Shenao Zhang, Sirui Zheng, Zhuoran Yang, and Zhaoran Wang. Maximize to explore: One objective function fusing estimation, planning, and exploration. In Thirth-seventh Conference on Neural Information Processing Systems, 2023b.
Meta. Introducing meta Ilama 3: The most capable openly available Ilm to date. Meta AI Blog, 2024.
https: //ai meii..mta.com cwwmooo///a//a wwt.lcoocc/o.c.eccc......oooooot meta- -latta--lama311
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the po- tential of slms in grade school math, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXie preprint arXiv:2112. 12.332212 2021.
Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training a reproducible long context text embedder, 2024.
OpenAL. Gpt-4 technical report. ArXiv, abs/ fts//333..87774 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Aduances in Newral Information Processing Systems, 35:27730 27744, 2022.
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of-n: Synthetic preference goneration for improved reward modeling. arXio preprint ar xxXi::2411.1206, 2024.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Bo Pang, Caiming Xiong, and Yingbo Zhou. Arm: Alignment with residual energy-based model. In Pro- ceeilings of the 2024 Conference of the North Americun Chapler of the Associalion for Compulational Linguistics: Human Language Technologies, June 2024.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfong Gao. Instruction tuning with gpt-4. arXiv preprint ar orXio:2304.034 032277, 2023.
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXix:2306. 17563, 2023.
Rafael Rafailov, Archit Sharma, Fric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305.1820 2023.
Corby Rosset, Ching-An An Cheng, Arindam Mitra, Michael Santacroce, Ahuned Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self celf-improve with general preferences. arXiv preprint arXiv:2404.0871 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. Xixiv preprint Xrwi 007406447 2017.
Yuda Song, Yifei Zhou, Ayush Sekhari, Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXie preprint arXio:2210.06716 2022.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NewrIPS, 2020.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint Xft:24000040566 2024.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint ar X4m:22404.1 14367, 2024.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXim:2402.057/4 2024.
Gemini Team, Rohan Anil, Scbastian Borgcaud, Yonghui Wu, fean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models, arXiv preprint XXX2:22122.18005, 2023.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist 11m assistants, 2023. URL https: //igggingaai co co//aatttt//eeettttm//////ereerr
Tekniuml. Gpteacher, 2023. URL
GitHub repository.
Li Tianle, Chiang Wei-Lin, Dunlap Evan, Frick nad Lisa, Zhu Banghua, Gonzalez Joseph E., and lon Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL https:
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Naznoen Rajani, Kashif Rasul, Youncs Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, al. Zephyr: Direct distillation of Im alignment. Xiv preprint 2rX 16944, 2023.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi- objective rewards. arXiv preprint arXic:2402.18571, 2024.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Helpsteer: Multi- attribute helpfulness dataset for steerlm, 2023.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arviv preprint Xrv:223312 02120, 2023.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human leedback. orXiv preprinl Xiv:2209.1086. 2021.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. play preference optimization for language model alignment. arXiv preprint arXiv:2405. 00675, 2024.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample- efficient oflline and online reinforcement. learning. Advances in newral information processing systems, 34: 11895-27407, 2021.
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit "*-approximation for sample-effciient rlhf. arXiv preprint orXiv:2405.20046 2024.
Wei Xiong. sufficient condition of sample-efficient reinforcement learning with general function approxi- mation. The Hong Kong University of Science and Technology, 2023.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arxiv preprinl arXiv:2304.12 12244, 2023a.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint 2xXv:2312.16682. 2023b.
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. Chatglm-math: math: Improving math problem-solving in large language models with a self-critique pipcline. arXiv preprint arXiv:2404. 02890, 2024.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint Xxx::4422.07814 2024.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruob- ing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. Xr preprinl arXiv:2401.10020 2024b.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint orXiv:2304.05900 2023.
Xiang Yuc, Ge Zhang Xingwei Qu, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2300.05653. 2023.
19

---

Published in Transactions on Machine Learning Research (09/2024)
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Compuler and System Sciences, 78(5):1538-155 1556, 2012.
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-explorigg language models: Active preference elicitation for online alignment. orXiv preprint arXiv:2405. 19392, 2024.
Tong Zhang. Mathematical analysis of machine learning algorithms. Cambridge University Press, 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint orXiv:2305.10255 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprinl ar Xrx:22300 05668, 2023.
Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arX1:2211.01192, 2022.
Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arxiv preprinl (xXi::2404.19922 2024.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7h, 7b: Improving Ilm helpfulness & harmlessness with rlaif, 2023.
Danicl M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint wxXiv:190900850. 2019.
20

---

Published in Transactions on Machine Learning Research (09/2024)
A Authorship and Credit Attribution
All authors provided valuable contributions to this project, each bringing unique expertise and insights that were crucial for its success.
HD first demonstrated that iterative DPO algorithm can achieve state-of-the-art performance; wrote a development version code for SFT, iterative RLHF; contributed to the training of SFT model and BT- RM; conducted extensive experiments on training and ayper-parameter tuning of iterative RLHF; delivered the released BT reward model; provide preference dataset for final BT-RM and some initial versions of the prompt data; conducted the RMM evaluation and GPT-based evaluation of the generative models; contributed to paper writing; contributed to the public version of ilerative RLHF code.
WX wrote the codes for the Bradley Terry reward model and conducted most of the experiments for both reward and preference model training; delivered the released pairwise preference model; contributed to the preference dataset search and hyper-parameter tuning; initiated and organized the online iterative RLHF project; wrote the initial code for the online iterative DPO and prepared its public version on GitHub; contributed to the evaluation of the reward and preference models; assisted in the collection and the cleaning of the preference dataset; wrote the paper.
BP conducted most of the final SFT and RLHF experiments and delivered the released SFT and RLHF model; independently wrote development version code for SFT and iterative RLHF;; developed the SFT recipes (data, typer-parameter, model selection); conducted extensive experiments on the training and hyper-parameter tuning of SFT, offline and iterative RLHF; conducted the GPT-based evaluation and all the academic benchmarks; contributed to paper writing.
HW initiated the training code of the pairwise preference model, and conducted experiments in the training of the Bradley Terry reward model and pairwise preference model; collected, filtered, and deduplicated the prompt set; contributed to the preference dataset collection and data cleaning; contributed to the evaluation and analysis of the reward and preference models; made substantial writing contributions to the reward modeling section and created illustrative figures for the algorithmic frameworks and dataset visualization.
HZ, YZ, NJ, DS, CX, TZ supported and advised the works of the junior authors, provided computational resources, and suggested experiments and writings.
B
Additional Experimental Details
B.1 Preference Datasets
Following the Pair-RM (Jiang al. 2023) and LLaMA-2 (Touvron et al. 2023), we use mixture of open- source datasets the training set. Here is brief introduction to the datasets:
HH-RLHF Bai et al. 2022a is a pairwise preference dataset where each sample is accompanied by a conversation history and two alternative responses written by an early Claude model with 52B parameters. The preferences of the responses are annotated by humans.
SHP [Ethayarajh et al. 2022 is sourced from Reddit and includes examples from 18 subreddits, such as askacademia, askbaking, askengineers, and changemyview. Each example is a Reddit post with a question/insti /instruction and a pair of top-level comments. One comment is preferred by more Reddit users than the other. All preferences and responses are provided by humans. Following (Cij al. 2023), only samples with score ratio are used, and at most pairs are taken for each prompt.
HelpSteer (Wang et al. 2023 This open-source dalaset (aag et al. 2023 contains prompts, responses, and five human--nnotatee annotated attributes (helpfulness, correctness, coherence, complexity, and verbosity) ranging from 0 to 4. The prompts are generated using a mixture of templato-generatol and human-gen generatee methods, while responses are generated by an in-house LLM. The authors generate up to responses per prompt, and we can construct pairwise comparisons based on them.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Dataset
#Prompts
Prompt Len.
Preferred Len.
Rejected Len.
Completion
Annotator
#Pairs
HH-RLHF
115092
160.4
82.2
73.6
LLM
Human
115396
SHP
31003
186.2
173.6
88.8
Human
Human
93301
HelpSteer
8592
530
116.4
89.3
LLM
Human
37131
6975
21.5
70.4
74.6
LLM
Human
26874
UltraFeedback
63591
161.5
279.5
211.1
LLM
GPT-4
340025
U Interact
76086
507.4
396.6
416.7
LLM
GPT-4
161927
Codel Itra Feed back
9938
172.8
427.6
400.6
LLM
GPT-3.5
50156
Argilaa-Mate
2352
36.5
276.5
265.3
LLM
GPT-4
2418
OpenOrca
6791
153.3
165.4
260.5
LLM
GPT-4
6926
Capybara
14740
634.5
348.4
401.9
L.M
GPT-4
14811
Table 5: summarization of open-source preference datasets. "Prompt Len." represents the average prompt length in terms of tokens, and "Preforred/ /Rejertdd Len." stands for the average length of preferred or rejected responses. All of these lengths are averaged over all pairs and we use the tokenizer of LLaMA- 3-8B. "Completion" marks the data source of the text completions for prompts. We apply pre-processing tochniques to all these datasets and delete the noisy samples from the original dataset. For the dataset whose prompt is with multiple responses, we include all the possible comparisons except those with the same gcore/ranking to compute the total number of comparison pairs.
PKU-SafeRLHP (Ji 2024). This dataset (Ji al. 2024) consists of 30k+ expert comparison data. Each sample includes two responses to question and two preference signals for helpfulness and safely, respectively. The responses are generated by open-source chatbots, and the preference signals are merged through the results of 14 harm category multi-class classficiation.
Feedbaack (Cui et al., 2023) consists of 64k prompts from diverse resources (including Ul- traChat, ShareGPT, Instruct TruthfulQA, FalseQA, and FLAN) and the authors generate 4 responses per prompt using 4 different LLMs sampled from diverse set of statc-of-the-art open- source LLMs. The preference is from GPT-4 based on a fine fine-grained annotation instruction, which contains different aspects, namely instruction-followingg truthfulness, honesty and helpfulness. The dataset collection strategy of UltraFeedbbcc has also influenced many subsequent works.
Codeelttraeeddback is generated similarly with the Ultrafeedback but focuses on the coding task. The annotation is from GPT-3.5.
UltraIntcract Yuan et al. 2024a) is preference dataset designed for complex reasoning tasks. The authors collect a prelerence Tree for each instruction, with the instruction being the rool and each action node. A trajectory is root-to-leaf path consisting of a sequence of actions. Paired correct and incorrect nodes or trajectories are used for preference learning.
Distilabel-Capybare is a preference dataset of multi-turn dialogues whose prompts are taken from Daniele & Suphavadeeprasit (2023), where the responses are generated by open-source LLMs and preferences are generated by GPT-4.
Distilabel-Orea is collected similarly with Capybara but with the prompts from Lian al. (2023a).
The training of LLMs is highly lata-deeendentt To ensure high-quality training data, we conduct filtering process on the open-source datasets we use. This process removes low-quality and meaningless samples. Additionally, conversations with empty rounds incorrect labels (implied by the other features of the dataset) are eliminated. Furthermore, in datasels where absolute scores are available, pairwise comparisons with small margins are excluded as these preference signals tend to be noisy (Bansal et al.) 2023). This process roughly delctes 10% of the data. We summarize the statistics of the open-source datascts that are used for the training in Table 5 and prepare them, as well as our data filtering script, on the huggingface.
www.aaatl---o--o-a-aaeaaaaaaa-oo pap-ia pairs
22

---

Atlas with the Published in Transactions on Machine Learning Research (09/2024)
Table 6: A summarization of the benchmarks we use in this project. We list the metric and number of shots (indicating zero-shot learning in-context learning) used for LLM evaluation each datasct.
Benchmark EC-AlpacaEval-2 MT-Bench Chat-Arena-Hard
GSM-SK
MMLU
HumanEval TruthfulQA
ARC
MBPP
Metric
win rate
win rale
Num. of Shots
0
0
5
25
0
We consider two versions of the training set:
Mix1: HH-RLHF SHP UltraFoodback Summarization (Stiennon al. 2020
MIx2: all the datasets in Table 5
The Mix1 dataset is similar to the construction of UltraFeedback (Cui al. 2023 with additional
summarization dataset. In comparison, the MIx2 consists of more reasoning preference pairs (math and code) and safety data. We also consider three different approaches to model the preference signals, including prompting in the LLM-as-a-judee manner (Zheng et al. 2023), reward modeling as the MLE of the BT reward model, and the preference model.
B.2 Benchmark Details
AlpacaEval-2 (Dubois et al. 2023
This benchmark focuses on single-turn conversations and consists of 805 test prompts covering various topics. The models are compared head-to-head with GPT-4- Preview (11/06) to compute the win rate. The same GPT-4 model is used as the judge. To mitigate the length bias of GPT-4, length-control variant of the benchmark is also proposed.
MT-Bench Zheng et al. 2023 This benchmark a multi-turn benchmark and includes 160 test prompts from different areas. The model should first answer an initial question, and then pre- defined follow-up question. The model's responses are then rated by the GPT-4 model with scale from 1-10, and the final score is computed the average score of two turns.
Chat-Arena-Har (Tianle et al. 2024h: This benchmark consists of 500 test prompts from the live data in Chatbot Arena, a crowd-sourced platform for LLM evaluations. The prompts evaluate the model's ability in specificity, domain knowledge, complexity, problem-solving, creativity, technical accuracy, and real-world application. In addition to the agreement to human preference, compared with AlpacaLival-2 and MT-Bench, Chat-Arena- Hrrd further enjoys a clear separability among dif- ferent models.
We also summarize the benchmarks we use in this project in Table 6
B.3 Other details
Prompt Visualization. We provide the visualization generated
Nomic
nomic-anbed-text-v11. text embedding model (Nussbaum et al. 2024
SFT Data List. We collect open-sourced instruction-fiettunii data for SET model training. The following data is included: ShareGPT (Chiang et al. 2023), Evol-Instruct (Xu et al.) 2023a SlimOrca (Lian (Wei et al. 2023). GPT4-LLM
et al. 2023b), MathInstruct (Yue et al. 2023),
PPng et al. 2023 OrcaMath MMitra et al. 2024h.
GPTeacher [Teknium] 2023), Ulraanteract Yuuan et al.
2024a).
Offline Vanilla DPO. We use Nectar dataset for Offline DPO. We run 1 epoch with batch size 128. learning rate 50-7, and cosine decay scheduler.
httpss//atlas. nomic.ai/
23

---

Published in Transactions on Machine Learning Research (09/2024)
Visualization of Prompts
Hate Speeen
Hate Speech
Transiation
Translation
Fantasy Story
Fantosy Story
Hypothesis
Hypothesis
Disease Treatment
Disease Treatment
Fibonacci
Fibonacc
Math Simplify
Math Simplify
Secestty C1)
Petwris
Uhrozeedook
Pytoo Proguaming
Hepdlear
Eqaadora Coksdation
OpenOrca
Math
Geometrin
Sequeence
Colored by Datasets
Colored by Topics
Figure 5: Visualization of our prompt collection via Nomic Atlas. The left figure is colored by the data sources of prompts, and the right figure is colored by topics, which are auto-generated by the custom topic model of Nomic Atlas.
Parameter
Value
n b_baach_size, size_per pere_ice_eevice device
n grwggenntdttttc accummuatontnnnii
optim
adamw torch cosine
Ir lr_scheduler_tpp
num train araii_eepochs
beta
0.1
Table 7: Training parameters
Additional Plots.
We also have some additional training plots Figure 6 7 and have visualized our per-
formance as Figure
Hyperparameters.
The hyperparameters are listed in Table 7
Figure The training record of preference modeling. From the left to right, we present the records of training loss, gradient norm, and the learning rate, respectively.
24

---

Published in Transactions on Machine Learning Research (09/2024)
Unerinng Eee
Traingg Stap
Thanking
Tasning Skc
Figure 7: The training record of reward modeling. From the left to right, present the records of training loss, gradient norm, and the learning rate, respectively.
MTGenchh
AlpaciEval 2.0LC
Vin Mutia
8.0
Ieraton
Beration
Deexiton
Baration2
AlpacuEval 2.0
Chat Arena Hard
SFT
Itordion
Berttion
Natatio 3
SFT
Eoration 1
tertion22
Iteotion3 3
Desation
Figure 8: Model performance with respect to RLHF iterations.
25

---

Published in Transactions on Machine Learning Research (09/2024)
Comprehensive Evaluation Results on Academic and Practical Benchmarks
80
Model
LLaMA-3-8B-4
70
Ours (SFT baseline)
(DPO baseline)
60
(lterative RLHF)
Score (%)
GSM-BK
MMLU
HumanEval
TruthfulQA
ARC
MBPP
LC AlpacaEvval-2
MT-Bench (x5)
Chat-Arena-Hard
Dataset and Metrics
Model Evaluation on Academic and Practical Benchmarks
Model
LLaMA-3-8B-I
70
(Iterative
Score (%)
50
30
20
10
GSM-8K
MMLU
HumanEval
TruthfulQA
ARC
MBPP
LC AlpacaEval-2
MT Bench (x5)
Chat-Arena-Hard
Dataset and Metrics
Figure 9: Evaluation of our models and LLaMA-3 A-1---------s--
26

---

Published in Transactions on Machine Learning Research (09/2024)
C
Case Studies
To showcase the significant improvements achieved through online RLHF, we conducted qualitative analysis of the generated responses. We observed that, after applying online RLHF, the responses became more detailed and better formatted, often utilizing bullet points, highlights, bold text, and enumeration. This enhanced clarity and structure were key factors contributing to the improved win-rate observed our experiments. Below, we present several examples that illustrate these improvements.
Prompt
What Gremolata?
Offline RLHF Response
Gremolata is flavorful, aromatic mixture used as garnish seasoning for variety of Italian dishos, particularly osso buco, a dish made from braised veal shanks. It is also used other dishes like fish, pasta, or vegetables. Gremolata is typically made from a combination of finely chopped fresh parsley, garlic, and lemon zest. Sometimes, basil or mint can be added as well. The mixture usually prepared by finely chopping mincing the ingredients and then mixing them together. Gremolata adds bright, fresh, and citrusy flavor to the dishes it is used in. It is often sprinkled on top of the dish just before serving, allowing the flavors to meld and enhance the overall taste.
Online RLHF Response
Gremolata is a flavorful, aromatic mixture used as a garnish or seasoning in Ttalian cuisine, partic- ularly in dishes such as Osso Buco, risotto, and other meat or fish dishes. It is often served on top of the dish to add burst of fresh flavors.
The basic ingredients of Gremolata are:
Fresh parsley: This is the main component, providing the fresh, herbaceous flavor and color to the mixture. Flat-leaf parsley is often preferred for its robust flavor.
1.
2. Garlic: Finely chopped or minced raw garlic gives sharp, pungent taste to the Gremolata.
3. Lemon zest: The zest from a lemon adds a bright, citrusy flavor and a slight tang to the mixture. In some variations, you might also find:
Lemon juice: This adds a little extra acidily and freshness the mix. Fresh basil or mint: These herbs can be used instead of, or in addition to, parsley, depending on the dish and personal preference.
To prepare Gremolata, simply mix the ingredients together in small bowl. The proportions can vary depending on personal taste, but general guideline is 1 part lemon zest, 2 parts chopped parsley, and 12 part chopped garlic. Adjust to your liking. Gremolata is typically added just before serving to maintain its freshness and bright flavors. It pairs well with rich and savory dishes, as helps to brighten and lighten the overall taste.
Figure 10: Case Study Example 1.
27

---

Published in Transactions on Machine Learning Research (09/2024)
Prompt
What are the best, exercises for beginners?
Offine RLHF Response
The best exercises for beginners are those that simple, safe, and effective in improving overall fitness. Some of the top beginner exercises include:
Squats: Helps strengthen the lower body, particularly the glutes, quadriceps, and hamstrings. 2. Push-ups: Improves upper body strength, targeting the chest, triceps, and shoulders. 3. Planks: Engages the core muscles and strengthens the lower back, arms, and glutes. Lunges: Strengthens the quadriceps, hamstrings, and glutes, while improving balance and flexibility.
Standing Rows: great exercise for the upper back, biceps, and shoulders. Bicep curls: Targets the biceps, promoting arm strength and definition. 7. Tricep extensions: Strengthens the triceps, an important musele for overall upper body strength. 8. Walking or jogging: A simple and effective way to improve cardiovascular fitness and burn calories. 9. Bodyweight squats and lunges: These are great for beginners as they only require bodyweight and help build strength and balance. 10. Stretching and yoga: Incorporating flexibiliity exercises like yoga or simple stretches can help prevent injury and promote overall mobility.
Remember to start with low weights or bodyweight exercises, gradually increasing the intensity, and always focus on proper form to avoid injury. It's also a good idea to start with a warm-up and cool-down, and consider working with personal trainer or coach to learn the correct techniques.
Online RLHF Response
The best exercises for beginners are those that are simple, safe, and effective in improving overall fitness, strength, and flexibility. Here some top recommendations for starting your fitness journey:
"*Bodyweight Squats""; A fundamental excrcise that targets the lower body, including the glutes, hamstrings, and quadriceps. Stand with foet shoulder-width apart, lower your hips as if sitting back into chair, then push back up the starting position. Aim for 2-3 sets of 10-15 reps.
**Push-ups**. A classic upper hody strengthener that targets the chest, triceps, and shoulders. Start on your hands and knees, then lower your body until your chest nearly touches the ground, then push back up. Modify by starting on your knees if needed. Aim 2-3 sets of 8-12 reps.
3. **Plank**; A core-strengthening exercise that also works the shoulders, back, and glutes. Start in a push-up position, then lower your forearms to the ground, keeping your hody straight line from head to heels. Hold for 30 weconds to 1 minute, then rest and repeat for 2-3 sets.
lower body exercise that targets the glutes, hamstrings, and quadriceps. Stand with feet. together, then step forward with one foot, lowering your hips until both knees are bent at a 90-degree angle. Push back to the starting position and repeat on the other side. Aim for 2-3 sets of 8-12 reps per leg.
4. **Lngge**
5. **Bent-over Row**: A back and shoulder strengthener that targets the latissimus dorsi, rhomboids, and traps. Hold dumbbell in each hand, bend forward at the hips with straight back, then row the weights up to your sides, squeezing your shoulder blades together. Lower the weights and repeat for 2-3 sets of 8-12 reps.
6. **alkiing or * A great way to improve cardiovascular fitness and burn calories. Start with brisk walk, gradually increasing speed and duration you become more comfortable. Aim for at least 30 minutes of moderateeintennsity exercise, such brisk walking, days week.
7. **Yoga Pilatos** These low-impact exercises excellent for improving flexibility, balance, and core strength. Start with beginner classes videos, focusing on proper form and breathing. Aim for loast 2-3 sessions per week. **aaff Raises**; lower body exercise that targets the calf muscles. Stand on the edge of step or platform, then raise your heels as high as possible, then lower back down. Aim for 2-3 sets of 10-15 reps.
Remember warm up before starting any exercise, and always listen your body. If exercise causes pain or discomfort, stop and consult a healthcare professional or fitness instructor. As you progress, you can gradually increase the intensity, duration, and variety of your workouts to continue challenging your body and achieving your fitness goals.
Figure 11: Case Study Example 2.
28

---

Published in Transactions on Machine Learning Research (09/2024)
Prompt
"What lype of soil is suitable for cactus?"
Offline RLHF Response
Cactus plants thrive in well-draining soil that is porous and allows excess water to escape quickly. The idcal soil for cactus should have a pll level between 6.0 and 7.5, which is slightly acidic to neutral. The soil should also be low in mutrients, as cactus plants generally prefer a nutrient-poor environment.
The following soil types are suitable for cactus plants:
1. Cpclus-specific soil mix: This is a commercial soil mix designed specifically for cactus plants. It typically contains a blend of sand, coconut coir, perlite, and peat moss or sphagnum moss. This mix provides excellent drainage and aeration while retaining moisture.
2. Desert soil mix: A desert soil mix can also be suitahle for cactus plants. It usually consists of blend of sand, perlite, andor vermiculite, This mix is well-draining and allows water to escape quickly.
Sandy soil: A sandy soil with high percentage of sand suitable for cactus plants. It provides excellent drainage and allows water escape quickly.
Gravel and cactus mix: Layering a cactus mix top of layer of small rocks gravel also be good option. This mimics the natural environment of cactus plants the desert and allows water to drain away quickly. It's essential to avoid using heavy, clay-based soils, as they can retain too much water, leading root rot and other issues for cactus plants. Additionally, it's essential to ensure that the soil is well-acrated, as cactus plants require good air circulation around their roots.
Online RLHF Response
Cactus plants thrive in well-draining soil that allows excess water to pass through and does not hold moisture for long periods. The ideal soil for cactus plants is a mix that mimics their natural desert habitat. Here are some characteristics of suitable cactus soil:
1. *CCoarse Texture**: The soil should have coarse texture, allowing water to drain quickly. This is usually achieved by using materials like sand, perlite, pumice. These materials help prevent waterlogged roots and root rot, which are common issues for cacti.
2. **Lommy Soil**; A loamy soil that is combination of sand, silt, and clay is also suitable. It retains some moisture while still allowing for proper drainage. Adding some coarse materials like perlite pumice the loamy soil can enhance drainage further.
**Low Nutrient Content**; Cacti have shallow root systems and do not require lot of nutrients. Therefore, soil with low nutrient content, such a cpccus-specific potting mix, is ideal. Don't regular garden soil, may have many nutrients and lead excessive growth and weak
** Acidic Slightly Alkaline**; Cactus plants prefer slightly acidic slightly alkaline soil, with pH range 6.0 to 7.5. This range is generally neutral and does not require additional adjustments.
**Dry and * The soil should be dry on the surface to prevent moisture buildup, but still aerated enough to allow oxygen to reach the roots. This promotes healthy root growth and prevents issues like root rot.
To create a suitable cactus soil mix, you can use a combination of the following ingredients:
2 parts cactus or succulent potting mix (which usually contains peat, perlite, and coconut coir)
1 part coarse sand (such as sharp sand or builder's is sand)
1 part perlite or pumice (to improve drainage)
Mix these ingredients thoroughly and use the resulting soil for your cactus plants. You can also purchase commercial cactus and succulent soil mixes from a nursery or garden center, which are usually well-suited for these types of plants.
Remember to always allow the soil to dry out completely between waterings, cactus plants adapted to survive in arid conditions. Overwatering is common cause of cactus issues, be cantious when watering your cacti.
Figure 12: Case Study Example 3.
29


Reviews:
[{"review_id": "8O1zMlvO3V", "reviewer": "Reviewer_YMeA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n1. The iterative online methods is a very natural idea and the paper provides a relatively principled framework for this idea. \n2. The reward modeling part is very significant for any online RLHF method and the paper does a decent effort investigating the rewad modeling aspect, e.g., different reward model contruction and length control.\n3. The experiment result is comprehensive and demonstrates the benefit of the iterative online method.\n\n## Weakness\n\n1. The preference model construction of the reward model seems strange. I might have missed something, but if we follow this construction, doesn't it mean that it is possible $\\hat P(a^1 \\prec a^2 \\mid x,a^1,a^2) + \\hat P(a^2 \\prec a^2 \\mid x,a^1,a^2) < 1$? Does it not matter in practice or with fair amount of training we can ensure the support of the token after \"label =\" is only on \"A\" and \"B\" with good generalization?\n2. As the paper also mentions, the exploration scheme in line 4 of Alg. 1 is generally computationally intractable. I also wonder why we need such a relatively complicated exploration strategy. As [1] proves, using forward KL as exploration bonus seems to suffice even in general function approximation setting?\n3. It is hard to tell if the current suggested exploration strategies are actually performing exploration -- there seems no active exploration in these methods. For example, labeling from samples of different checkpoint does not lead to exploration, it is more like reweighting/filtering on the current distributions?\n4. It will be beneficial to contextualize the results and contributions by comparing and contrasting with hybrid learning in general RL, e.g., [2,3] by comparing the settings and desiderata. \n\n## Minor\n1. Honestly, I did not get why the current figure 3 imply that the reward model is biased towards longer reponses. Maybe more explaination could be helpful. \n2. what do mix1 and mix2 mean in table 1?\n3. In the main text, it says fig.3 is generated with 8 reponses per prompt, but the caption says 16.  \n\n[1] Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF\n\n[2] Policy finetuning: Bridging sample-efficient offline and online reinforcement learning\n\n[3] Hybrid rl: Using both offline and online data can make rl efficient", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to provide a recipe/workflow for online iterative RLHF methods and reward modeling (which aims to replace ground truth reward function). \n\nThe paper first provides two ways of reward model modeling: one through the BT model construction (a more classical approach), and the second one trhough preference model construction (a more recent approach). The paper compares the two constructions over the Reward-Bench benchmark. \n\nThe paper then introduce the online iterative RLHF recipe. They start from the theoretical framework from Xiong el al., 2023, and then provide practical algorithms to replace the part with computational infeasibility. Specially, the paper provides how to combine the offline data and the way to collect and label online data for the interative RLHF procedure. Finally, the experiment result across a comprehensive benchmarks demonstrate the effectiveness of the iterative methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. adding the discussion with [1,2,3] (however, I am not sure if the submission is earlier or [1], please disregard if the submission is earlier).\n2. explaining the connection between the current data collection method and exploration.\n3. fix the minor issues.\n4. adding hyperparameter table and algorithm box for the practical version.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2gRH98fbvK", "reviewer": "Reviewer_nVba", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Understanding this paper requires extensive prior knowledge, and its writing makes it difficult to read. \n> For example, the core of this paper revolves around RLHF research, especially since the author only discusses RLHF in the first paragraph: as a key technique for integrating human preference signals into machine learning methods, particularly in aligning Large Language Models (LLMs) with human values and preferences.\n\n2. What are \"human preference\" and \"values\"? From a technical perspective, how are these two reflected? These are all prior knowledge requirements. Additionally, the author begins discussing issues with RLHF in the second paragraph without defining RLHF, which is confusing.\n\n4. Some crucial concepts and references in the field of alignment are not discussed by the author, such as:\n\n- The challenges faced by RLHF. e.g. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback\n- The 3H criteria of large language models. e.g. A General Language Assistant as a Laboratory for Alignment\n- The definition of alignment. e.g. AI Alignment: A Comprehensive Survey", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. During reading, some confusing points arise, for example:\n\"The dominant RLHF framework used for the LLM alignment was first popularized in Christiano et al. (2017).\"\nChristiano et al. (2017) did not conduct LLM alignment?\n\n1. The author elaborates on LLMs alignment but does not define what alignment is throughout the text?\nIn section 2.2, the author mentions \"Length bias in reward modeling,\" which is an interesting point regarding RM's length bias, but the author does not define this issue or explain why their proposed method can alleviate it.\n\n1. In this technical report, we aim to fill in this gap and provide a detailed recipe that is easy to reproduce for online iterative RLHF. What does this \"recipe\" specifically refer to?\ndiscuss the theoretical insights and algorithmic principles behind online iterative RLHF; The text contains eight numbered formulas, the first six of which are common sense. What specific insights are being referred to?\n\nRegarding **reproducibility**, the experimental parameters provided by the author are insufficient to support reproducing the algorithm in this paper.\n\nHere are some of my points of confusion, and I am very willing to communicate further with the author.", "broader_impact_concerns": "see above.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xjKyBl6bQy", "reviewer": "Reviewer_7Xe9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: This work gives a simple but effective online RLHF framework. It only makes use of public available datasets with no extra human labels, and yet achieves impressive performance. \n\nWeakness: All components in this work are not novel. Experiment results align with that of existing works, and there is no new insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an online RLHF workflow using open source data and models. Specifically, they propose to first train a proxy reward model using a mixture of public available datasets. Then, they perform iterative Direct Preference Learning to update the model. They obtain a model that has impressive performance in conversation tasks, while retain the model's ability in academic task. They also explore incorporating length penalty in reward modeling and show that it gives superior performance in length-control benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. (critical) For such an online approach, it is important to see how the performance of the model changes w.r.t the number of iterations. I don't see such a plot in the current version.\n2. (critical) Some case study should be included to compare offline and online RLHF approach. For example, what types of prompts are better answered by the online trained model compared to the offline one.\n3. (good to have) The preference model overall has better performance according to Table 1, and it would be interesting to see some initial RL experiment results using preference model.", "broader_impact_concerns": "RLHF system learns from human preference, and it is possible that existing stereotypes are reinforced during the training of such system.\n\nThe long-term consequences of deploying RLHF systems at scale are not fully understood. These systems have the potential to influence societal behaviors and decision-making processes in profound ways.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper gives a detailed workflow/recipe for online iterative RLHF methods, inspired by a theoretical framework. It also includes two approaches to reward modeling, one based on the Bradley-Terry model, the other based on preference model. The workflow comes with comprehensive experiments, and demonstrates benefits of iterative online methods. The topic is important and relevant to a fast-growing community. While some of the key findings may not be novel, the detailed recipe is a useful contribution to the community. The initial submission raised questions about (minor) technical issues and related work. The revised version addressed these concerns satisfactorily. That said, I encourage the authors to polish the writing thoroughly (cf, comments by reviewer nVba). A few more detailed comments:

* The paper makes a useful distinction of preference learning and deep RL, as two approaches to RLHF, and focuses on the former. Is it more accurate to change the framing (eg, title) from RLHF to direct preferencd learning?
* The paper refers to Yue et al (2012) as the origin of using relative feedback. In fact, the idea can be traced back even further to the learning-to-rank literature: https://doi.org/10.1145/1229179.1229181
* In eqn (8), where is \Gamm_t^m defined?
* Page 9: ‘feed-good’ → ‘feel-good’?
Decision: Accept (poster)