REBEL: Reinforcement Learning via Regressing Relative Rewards
Authors: Zhaolin Gao
Jonathan Daniel Chang
Wenhao Zhan
Owen Oertell
Gokul Swamy
Kianté Brantley
Thorsten Joachims
J. Andrew Bagnell
Jason D. Lee
Wen Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Reinforcement Learning from Human Feedback
Primary area: reinforcement_learning
TL;DR: We present REBEL, a new reinforcement learning algorithm that simplifies policy optimization to regressing relative rewards, offering strong theoretical guarantees and empirical performances.

Abstract: While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) applications, including the fine-tuning of generative models. Unfortunately, PPO requires multiple heuristics to enable stable convergence (e.g. value networks, clipping), and is notorious for its sensitivity to the precise implementation of these components. In response, we take a step back and ask what a *minimalist* RL algorithm for the era of generative models would look like. We propose REBEL, an algorithm that cleanly reduces the problem of policy optimization to regressing the *relative reward* between two completions to a prompt in terms of the policy, enabling strikingly lightweight implementation. In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL, which allows us to match the strongest known theoretical guarantees in terms of convergence and sample complexity in the RL literature. REBEL can also cleanly incorporate offline data and be extended to handle the intransitive preferences we frequently see in practice. Empirically, we find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO, all while being simpler to implement and more computationally efficient than PPO. When fine-tuning Llama-3-8B-Instruct, REBEL achieves strong performance in AlpacaEval 2.0, MT-Bench, and Open LLM Leaderboard. Implementation of REBEL can be found at <https://github.com/ZhaolinGao/REBEL>, and models trained by REBEL can be found at <https://huggingface.co/Cornell-AGI>.

REBEL: Reinforcement Learning via Regressing Relative Rewards
Zhaolin Gao', Jonathan D. Chang?; Wenhao Zhan Owen Oertell', Gokul Swamy', Kianté Brantley", Thorsten Joachims', J. Andrew Bagnell Jason D. Lee", Wen Sun 1 Cornell University, 2 Databricks Mosaic Research, Princeton University, Carnegie Mellon University, Harvard University, 6 Aurora Innovation
Abstract
While originally developed for continuous control problems, Proximal Policy Op- timization (PPO) has emerged as the work-horse of variety of reinforcement learning (RL) applications. including the fine-tuning of generative models. Un- fortunately, PPO requires multiple heuristics to enable stable convergence (e.g. value networks, clipping). and is notorious for its sensitivity to the precise imple- mentation of these components. In response, we take step back and ask what a minimalist RL algorithm for the era generative models would look like. We propose REBEL, algorithm that cleanly reduces the problem of policy optimiza- tion to regressing the relative reward between two completions to a prompt in terms of the policy, enabling strikingly lightweight implementation. In theory, prove that fundamental RL algorithms like Natural Policy Gradient can seen match the strongest known theoretical
variants of REBEL, which allows guarantees in terms of convergence and sample complexity in the RL literature. REBEL can also cleanly incorporate offline data and be extended handle the intransitive preferences we frequently see in practice. Empirically, we find that REBEL provides unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO, all while being simpler implement and more computationally efficient than PPO. When fine-tuning Mumma-3-8B-Instemee REBEL achieves strong performance in AlpacaEval 2.0, MT- Bench, and Open LLM Leaderboard. Implementation of REBEL can be found https: //ti////tttbbb www//haolinaaoo be found https: htttt:::///uggigaaae.... http:e////ngiigghec....oooooo...
and models trained by REBEL can
Introduction
The generality of the reinforcement learning (RL) paradigm is striking: from continuous control prob- lems (Kalashnikov et al., 2018) 8) to, more recently, the fine-tuning of generative models (Stiennon et al., 2022; Ouyang et al., 2022). RL has enabled concrete progress across variety of decision -aaiinn tasks. Specifically, when comes to fine-tuning generative models, Proximal Policy Optimization (PPO, Schulman et al. (2017)) has emerged as the de-facto RL algorithm of choice, from language models (LLMss) (Ziegler et al., 2020; Stiennon et al., 2022; Ouyang et al., 2022; Touvron et al., 2023) to generative image models (Black et al., 2023; Fan et al., 2024; Oertell al., 2024).
*Work done at Cornell
{zg292, ojo2, tj36, 8corneel edu. j. changodatabriccks. com, kibrantley harvard. edu, {wenhao zhan, jaaemmaee edu, (gswamy, bam.aeae111121ddeeeeer cmu, edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Image Generation
Language Modeling
RLHF
REBEL
2 O exp(r
Alpacatval Win Roks 50
reinforcement
regression
learning
22,6
9
Figure 1: We present REBEL: a simple and scalable RL algorithm that performs policy optimization via iteratively regressing the difference in rewards in lerms of the policy, allowing us eliminate much of the complexity (e.g. value functions, clipping) of algorithms like PPO (Schulman et al., 2017). We apply REBEL to problems in both image generation and language modeling and find that despite its conceptual and impleeentattion-leee simplicity, REBEL is able to match or sometimes outperform the performance of PPO while out-performing purely offline techniques like DPO (Rafailov et al., 2023). REBEL also achieves strong performance on common benchmarks such as AlpacaEval when fine-tuning a Llama-3-8B model.
If take step back however, it is odd that we are using algorithm designed for optimizing two-layer networks for continuous control tasks from scratch, even though we are now fine-tuning generative models with billions of parameters. In the continuous control setting, the randomly initial- ized neural networks and the possible stochasticity in the dynamics necessitate variance reduction through learned value function a baseline (Schulman et al., 2015), while clipping updates is important to limit distribution shift from iteration to iteration (Kakade and Langford, 2002). This means that when applied to generative model fine-tuning, we need store four models memory simultaneously (the policy, the reference policy. the critic, and the reward model), each with billions of parameters. Furthermore, we often add a KL regularization making explicit clipping unnecessary nor advisable, as pointed out by Ahmadian et al. (2024). Even outside of the generative modeling context, PPO notorious for the wide range of performances measured, with differences being attributed to seemingly inconsequential implementation details (Henderson et al., 2019; Engstrom et al., 2020). This begs the question: are there simpler algorithms that better scale to modern RL applicattion?
the base model for fine-tuning,
Our answer is REBEL: algorithm that reduces the problem of RL to solving sequence of squared loss regression problems on iteratively collected datasets. Each regression problem directly uses the policy predict the difference in rewards. This allows to eliminate the complexity of using value functions, avoids heuristics like clipping, and scales easily to problems in both language modeling and image generation. Our key insight that regressor that can predict the difference in rewards between trajectories in dataset implicitly captures an improved policy.
Rather than being mere heuristic, REBEL comes with strong guarantees in theory and can be seen as a strict generalization of classical techniques (e.g., NPG) in reinforcement learning. Furthermore, REBEL cleanly incorporates offline datasets when available, intransitive preferences (Swamy et al., 2024), empirically out-performs techniques like PPO and DPO (Rafailov et al., 2023) in language generation, and has faster convergence with similar asymptotic performance in image generation. When fine-tuning a Llama-3-8B model, REBEL also demonstrates very competitive performance on AlpacaEval 2.0, MT-bench, Open LLM Leaderboard, and Arena Hard simultaneously. We begin by formalizing the preference fine-tuning setup before deriving our core algorithmic technique.
be extended robustly handle
2 REBEL: REgression to RElative REward Based RL
We consider the contextual bandit formulation (Langford and Zhang, 2007) of RL which has been used to formalize the generation process of models like LLMs (Rafailov et al., 2023; Ramamurthy al., 2022; Chang et al., 2023) and Diffusion Models (Black al., 2023; Fan et al., 2024; Oertell al., 2024) due to the determinism of the transitions. More explicitly, in the deterministic transition

---

Algorithm/ REgression to RElative REward Based RI. (REBEL)
1: Input: Reward r, policy class II = { 0 E )}. base distribution μ, learning rate
2: Initialize policy Tee
3: for ? = 0 to T - do
4: /asa distribution a can either be an offline dataset or me
5: Collect dataset D. = far y, s/} where P. me(-)))))) v ~ p(-1z)
6: Solve square loss regression problem:
(
(1
To(U)Z) To. (U)0))
In <o(v(z)) ne u/(x)))) ) (r(2,, (0(0))((())))))))))))))))))))- r(x,,0)))
oc+1 argmin
Σ (e.w)")).
BE
7: end for
setting, explicit states are not required as they are isomorphic to the sequence of actions. Furthermore, the entire sequence of actions can be considered single "arm" in bandit problem with action space that scales exponentially in size with the horizon of the problem.
We denote by (a,, y) (prompt, response) pair, where X' is the prompt and y Y is the response (e.g. a sequence of tokens, or in general sequence of actions). We assume access to a reward function r(z,y) y) from which we can query for reward signals (the exact form of does not need to be known). Querying r at (x. y) will return a scalar r(z, y). measuring the quality of the prompt completion. Such reward function could be pre-defined metric (e.g., Rouge score against human responses) or a learned model from an offline human demonstration or preference data (e.g. the RLHF paradigm (Christiano et al., 2017; Ziegler et al., 2020)), as focus on in our experiments.
Denote by т X + AYY policy (e.g- an LLM) that maps from a prompt x to a distribution over the response space Y/ We use to denote the distribution over prompts (i.e. initial states contexts) I and Th (g((z)) to denote a policy with parameter 0. At times, we interchangeably use Te and me. when it is clear from the context. We emphasize that while we focus on the bandit formulation for notational simplicity, the algorithms proposed here can be applied to any deterministic MDP where æ is the initial state and the trajectory y consists of the sequence of actions.
At each iteration of all algorithms, our goal will be solve the following KL-constrained RL problem:
T argmax Ee. y) E. KL. (-e])))(((e((()))))
#Ell
Intuitively, this can be thought of asking for the optimizer to fine-tune the policy t++1 according to while staying close in terms action distribution to some baseline policy Te.
2.1 Deriving REBEL: REgression RElative REward Based RL
From Ziebart et al. (2008), we know that there exists a closed-form solution to the above minimum relative entropy problem (Eq. 2, Grünwald and Dawid (2004)):
Vz, y: (z)()) = Te (x(l(x)) exp(1r(x, v)) Z(x) = Σ n. cxp(nr(x,))
(3)
Z Z(x)
As observed by Rafailov al. (2023), we can invert Eq. 3 and write the reward in terms of the policy:
Vz, r(x. v) = In(Z(x)) + In (T+1()l=)) Te(g(z))
(4)
As soon X' and y become large, we can longer guarantee the above expression holds ex- actly at all (x, y) and therefore need to turn our attention to choosing a policy such that Eq. 4 is approximately true. We propose using simple square loss objective between the two sides of Eq. 4 to measure the goodness of a policy, i.c. reducing RL to regression problem:
(r(z,)) [I (nx))))) + In (v+2()))) ))
Unfortunately, this loss function includes the par- tition function Z(x), which can be challenging approximate over large input/ output domains.
<e(gl=)
3

---

However, observe that Z only depends on and not y. Thus, if we have access to paired samples, ie. (z. y) and (z, y²). , we can instead regress the difference in rewards eliminate this term:
(r(2,,) ( r(x,,')) 1f (In
(T In T((y)x)
<<(3'( (z) (=
(5)
Of course, we need evaluate this loss function
distribution of samples. In particular,
propose using an on-policy dataset D. = {z, y'3 with I p.y ~ TT(-1z), ~ u(-lzx) where u some base distribution. The base distribution μ can either be fixed offline dataset (e.g. the instruction fine-tuning dataset) or itself. Thus, the choice of base distribution μ determines whether REBEL is hybrid or fully online. Putting all together, we arrive our core REBEL objective in Eq. 1. Critically, observe that if we were able to perfectly solve this regression problem, we would indeed recover the optimal solution the KL-constrained RL problem we outlined in Eq. 2.
3 Understanding REBEL an Adaptive Policy Gradient
In this section, we interpret REBEL as an adaptive policy gradient method to illuminate the relationship to past techniques. We start by introducing algorithms such as Mirror Descent, NPG, and PPO, followed by illustrating why REBEL addresses the limitations of these past algorithms. For concision, postpone an in-depth discussion of related work to Appendix A.
3.1 Adaptive Gradient Algorithms for Policy Optimization
Mirror Descent. If X' and y are small discrete spaces, we can use the closed-form expression for the minimum relative entropy problem (Eq. 3). This is equivalent to the classic Mirror Descent (MD) algorithm with KL as the Bregman divergence. Both NPG and PPO approximations of MD.
Natural Policy Gradient. When y and X' are large, we use parameterized policy denoted as me with parameter 9. Natural Policy Gradient (NPG, Kakade (2001)) approximates the KL in Equation 2 via its second-order Taylor expansion, whose Hessian is known as the Fisher Information Matrix (FIM, Bagnell and Schneider (2003)), F,, i.e. F. E, Byyaw (-Iz) [V In To, (3)2:) V In mo, (yl=:) "].
Thus, E, (Te( (Te(- Te. (- (z)) = (0 -0,) -,)TT F2 - 01). The NPG update can be formulated as:
0.11 0. + FFF (Ex,sraa, (-1x) no, no, (v)x))((z,)))
(6)
where F is pseudo-inverse of Ft. As mentioned above, this update procedure can be understood as performing gradient updates in the local geometry induced by the Fisher information matrix, which ensures that we are taking small steps in policy space rather than in parameter space. NPG, unfortunately, does not scale to modern settings due to need of inverting the FIM each iteration.
Proximal Policy Optimization. Proximal Policy Optimization (PPO, Schulman et al. (2017)) takes more direct route than NPG and uses clipped updates
[o(U)z)) TT. (ylx)) -1:11 c.1 r(z, v).
0.41 argmax En.yy-oo, (-Ix)
c,
0
While the clipping operator can set the gradient to be zero at samples (x. y) where Th.. (y) ()) is much larger or smaller than ne, (y)z), it cannot actually guarantee T8.,. staying close to IT, phenomenon empirically observed in prior work (Hsu et al., 2020). Furthermore, hard clipping is not adaptive - treats all (x., y) equally and clips whenever the ratio is outside of fixed range. In contrast, constraining the KI. divergence to the prior policy allows one to vary the ratio π (v)==)/m ( at different (r, y), as long as the total KL divergence across the state space is small. Lastly, clipping reduces the effective size of batch of training examples and thus wastes training samples.
3.2 Connections between REBEL and MD / NPG
Exact REBEL is Mirror Descent. First, to build intuition, interpret algorithm's 's behavior under the assumption that the least square regression optimization returns the exact Bayes Optimal solution (i.e., our learned predictor achieves zero prediction error everywhere):
1 ((1n (1n
T8.4 (2)x)
In To..: (1/1z)) )
Yx, v.v':
To. (v' (x) r(x. - -((,
(8)
17
no, (y/x)

---

Conditioned on Eq. 8 being true, a few lines of algebraic manipulation reveal that there must exist a function c(x) which is independent of y, such that Va, In #0 (M/2)
1 () r(z, v) + c(x)). Taking an exp on both sides and re-arrange terms, we get Va, y #0 (y/x) x Te, (y x)) exp (,, g)). In other words, under the strong assumption that least square regression returns a point- wise accurate estimator (i.e., Eq. 8), we see the REBEL recovers the exact MD update, which gives (a) a fast 1/T convergence rate (Shani et al., 2020; Agarwal et al., 2021), (b) conservativity, i.c. max, KL (-[=)]] (-1z)) bounded as long as maxx,y [ y) | bounded, and (c) monotonic policy improvement via the NPG standard analysis (Agarwal et al., 2021).
NPG is Approximate REBEL with Gauss-Newton Updates. We provide another interpretation of REBEL by showing that NPG (Eq. 6) can be understood as a special case of REBEL where the least square problem in Eq. 1 is approximately solved via single iteration of the Gauss-Newton algorithm. We start by approximating our predictor 7) In To (v(a))/ (y (z) by its first order Taylor expansion
at 0g: (In (o(y)ar) (z) In To, (v]=)) ! Vo In no, (ylx))(()))) (0 - 0t), where x indicates that we ignore
higher order terms in the expansion. Define ō == 0 - 0g and replace 1 (In no(y)(>) - In Te, ( (vlar) by its first order approximation in Eq. 1. Then, we h
7
2 (Vo In no, (v|x) Vo In To (1'lz)) 8 (r(x,v) r(,, (/))
min E pu (12)) m)
Further simplifying notation, we denote the uniform mixture of Te and i as Tmin (-1x² (1x(+1z) + p(12))//2 and the Fisher information matrix F, averaged under said mixture as Solving the above least squares problem to obtain a minimum norm solution, we have the following result.
F. Ex-pp.--mmmr(lll Vo In no, (ul=r) (Vo o In no, (1]2z))
Claim 1. The minimun norm minimizer 8* of the least squares problem in Eq. 9 recovers advantage-basee NPG update: 8 == nFF (Ba-n.y= (1a) Vo In TT. (y]z) (z, v)l) where F.! is pseudo-inverse of Fy, and the advantage is defined as AT (x, y) == r(z, y) (=) y).
The proof of this claim is deferred Appendix B.
The implicit variance reduction effect of REBEL We show that regressing to relative rewards has variance reduction effect by extending the previous derivation on REBEL with Gauss-Newton update to the setting of finite data D = {In: Un. I Denote the unbiased estimate of the
Fisher information matrix as F. N Σ following claim.
Ve To (ynl (Vo In me, (val*,))) ] and have the
Claim 2. The minimum norm minimizer 5* in Eq. 9 under finite setting has the form 8* " 2N En (V In no, (r()))))))n (aen, - where F is pseudo-inverse of Ft.
v(h) + V To (w (r (r vh)) r(Ins Un))
The proof of this claim is deferred to Appendix C. Looking at the gradient formulation V In Te, (ym [2n) (r(zn, Bn) - r(zn, v/a) in 8", we see that r(z,, V')) serves as a baseline for vari- ance reduction. Interestingly, this gradient formulation is similar to RLOO (REINFORCE with leave-one-out) (Kool et al., 2019). However, different from RLOO, we pre-condition this variance reduced policy gradient formulation via the Fisher information matrix, leading to better performance.
A REBEL With Cause. Our algorithm REBEL addresses the limitations of NPG (scalability) and PPO (lack of conservativity or adaptivity) from above. First, unlike NPG, it does not rely on the Fisher Information Matrix at all and can easily scale to modern LLM and image generation applications, yet can be interpreted as a generalization of NPG. Second, in contrast to PPO, it doesn't 't have unjustified heuristics and thus enjoys strong convergence and regret guarantees just like NPG. Building on Swamy et al. (2024), we also show how to extend REBEL to preference-bate based settings without assuming transitivity in Appendix D.
4 Theoretical Analysis
In the previous section, we interpret REBEL as exact MD and show its convergence by assuming that least square regression always returns a predictor that is accurate everywhere. While such
5

---

an explanation is simple and has also been used in prior work (Calandriello et al., 2024; Rosset al., 2024). point-wise distt-iiittbbutionn generalization is an extremely strong condition and is significantly beyond what standard supervised learning method can promise. In this section, we substantially relax this condition via a reduction- analysis: As long as we can solve the regression problems well in an in-distribution manner, REBEL can compete against any policy covered by the training data distributions. Formally, we assume the following generalization condition holds on the regressors we find.
Assumption 1 (Regression generalization bounds). Over T iterations, assume that for all t, we have the following for some c:
(2 17
[1
In T8441 (u|æ) In (8,,((1(2)) (f(2,v) ((c.,)))(xx.)))) sc, To. (1/lz))
TT. TT. (y|z)
Detailed justifications for this assumption are provided in Appendix E.
Data Coverage. Recall that the base distribution p. can be some behavior policy, which in RLHF can be a human labeler, a supervised fine-tuned policy (SFT), or just the current learned policy (i.e., on-policy). Given a test policy æ, we denote by Cp-pa the concentrability coefficient, i.e.
Cp-v
max 2,91 (1)0))
(10)
We say i covers i if Op-1z < +oo. Our goal is to bound the regret between our learned policies and an arbitrary comparator n²" (e.g. the optimal policy if it covered by μ4) using and the concentrability coefficient defined in Eq. 10. The following theorem formally states the regret bound of our algorithm.
Theorem 1. Under Assumption 1, after T many iterations, with proper learning rate n, among the learned policies T TI..... NT, there must exist policy ft, such that:
Vn" (le) v) - -B S o (VT+ Vittveere)
The above theorem shows reduction from RL to supervised learning as long as supervised learning works (i.e.. is small), then REBEL can compete against any policy π that is covered by the base data distribution p. In the regret bound, the 1/VT comes from Mirror Descent style update, and Cp-tz captures the cost of distribution shift: we train our regressors under distribution Te and μe, but we want the learned regressor to predict well under x². Similar to the NPG analysis from Agarwal et al. (2021), we now have a slower convergence rate 1/VT, due to the fact that we have approximation error from learning. Such an agnostic regret bound- - being able to compete against any policy that is covered by training distributions - is the strongest type of agnostic learning results known in the RL literature, matching the best of what has appeared in prior policy optimization work including PSDP (Bagnell et al., 2003), CPI (Kakade and Langford, 2002), NPG (Agarwal et al., 2021), and PC-PG (Agarwal et al., 2020). While in this work we use the simplest and most intuitive definition of coverage - the density ratio based definition in Eq. 10 - extension to more general ones such as transfer error (Agarwal et al., 2020, 2021) concentrability coefficients that incorporate the function class (c.g., Song et al. (2023)) is straightforward. We defer the proof of the above theorem and the detailed constants that we omitted in the O notation Appendix F. We include an extension of the above analysis to the general preference setting in Appendix G.
Remark 1 (Discussion on the size of the response space y and other design choices of the sampling distributions). In REBEL, when sampling pair (y. V'), ) we in default sample y y~ TT, i.e., we make sure at least one of them is an on on-policy sample. This is to make sure that the training distribution at iteration covers T. which plays essential role in avoiding polynomial dependency on the size of the action space ()]? On the other hand, as long as we have some off off-policy distribution ve that covers ne for all t, we can use it to sample y and pay an additional concentrability coefficient
"The sample complexity of the Q-NPG algorithm presented from Agarwal et al. (2019) has polynomial dependence on the size of the action space since it samples actions uniform randomly in order to cover both m and n." REBEL leverages that can reset from the same context æ, and thus directly draw two samples per and x² simultaneously.
context one from and one from μ.

---

Model size
Algorithm
Winrate (1)
RM Score (1)
[L/a (1)
Model size
Algorithm
Winrate (1)
SFT
24.9 (1-2.3))
-0.51 (=0.5))
SFT
45.2 (=2.49)
DPO
42.7 (::1.9
0.10 (20.02)
29.6 (40063)
DPO
68.4 (=2.01)
1.4B
Iterative DPO
47.2 (+:1.34)
1.73 (#0.05)
29.7 (+:057)
REINFORCE
70.7"
PPO
51.2 (2:1-42
1.74 (#0.00)
29.3 (1:061)
6.9B
PPO
77.6
REBEL
55.1 (+1.335)
1.84 (+-004)
32.6 (+0.559
RIOO (E: = 2)
74.2"
SFT
28.2 (1:2.31)
-0.38 (=000))
RLOO 4)
77.9"
DPO
53.7 (1:1.63
2.40 (#0.02)
64.3 (+1)25)
REBEL
(=:1.74
2,8B
Iterative DPO
63.1 (1:1,4)
2.37 (40003)
28.1 (40.51)
directly obtained from Ahmudian (2024)
PPO
67.4 (1:1.30)
2.37 (+0.03)
27.2 (+0.55)
directly obtained from Hunng et (2024)
REBEL
70.2 (1:1.32)
2.44 (1:0.02
29.0 (1:0.60)
Table 1: Results on TL;DR Summarization. Results are averaged over three seed and the standard deviations across seeds are in parentheses. The best-performing method for each size and metric is highlighted in bold and the second best is underlined. REBEL outperforms all baselines on winrate.
y. Tee (o.l/l///((yle in the final bound. In experiments, we test the combination of the best-of-N of nt as the base distribution μ and the worst-of-N of my as the vs. Setting μ to be the best-of-N of me makes μ cover higher quality comparator policies. Selecting Vt as the worst-of-N of Th still ensures coverage to n. while at the same time increasing the reward gap r(x, y) rxx, 3/), which we find is helpful experimentally.
Experiments
Our implementation of REBEL closely follows the psuedocode in Algorithm 1. In each iteration, REBEL collects dataset D, = {I, y. y'), where x p. y Tr(-1z), y p(-lzr). Subsequently, REBEL optimizes the least squares regression problem in Eq. 1 through gradient descent with Adam (Loshchilov and Hutter, 2017). We choose μ = ne such that both y and y' are generated by the current policy. We empirically assess REBEL'S 's performance on both natural language generation and text-guided image generation. Additional experiment details are in Appendix H.
5.1
Summarization
Task. We use the TL.DR dataset (Stiennon et al., 2020) where x is a forum post from Reddit and is a summary generated by the policy. The dataset comprises human reference summaries and preference data. We compare REBEL with baseline RL algorithms, REINFORCE (Williams, 1992) and its multi-sample extension, REINFORCE Leave-One-Our (RLOO) (Kool et al., 2019), PPO (Schulman et al., 2017), Direct Preference Optimization (DPO) (Rafailov et al., 2023), and Iterative DPO (Guo al., 2024). Our implementation of Iterative DPO replaces our square regression objective with the DPO objective where the binary preference labels are obtained based on the reward difference. The implementation detail of the baseline methods is provided in Appendix H.1.3. Following prior work (Stiennon et al., 2020; Rafailov et al., 2023; Ahmadian et al., 2024), we train DPO on the preference dataset, while conducting online RI. (RLOO, PPO, Iterative DPO, REBEL) on the human reference dataset. We include results with three different model sizes: 1.4B, 2.8B, and 6.9B based on the pre-trained models from Pythia (Biderman et al., 2023). Each model is trained from a supervised fine-tuned (SFT) model using reward model (RM) of the same size.
Evaluation. We evaluate each method by its balance between reward model score and KL-divergence with the SFT policy, testing the effectiveness of the algorithm in optimizing the regularized RL objective. To evaluate the quality of the generation, we compute the winrate (Rafailov et al., 2023) against human references using GPT4 (OpenAI, 2023). The winrate is computed from randomly sampled subset (10%) of the test set with 600 samples. We report the average results over three seeds.
Quality Analysis. Table presents a comparison between REBEL and baseline methods. Notably, REBEL outperforms all the baselines on RM score with 1.4B and 2.8B parameters with slightly larger KL than PPO. In addition, REBEL achieves the highest winrate under GPT4 when evaluated against human references, indicating the benefit of regressing the relative rewards. An ablation analysis on parameter is in Appendix and the trade-off between the reward model score and KL-divergence is discussed in Appendix K.
7

---

Runtime & Memory Analysis. We an-
alyze the runtime and peak memory us- age for 2.8B models with REINFORCE,
Generation Policy Update
Poak Paak Memory Usage (G8)
40-
100
35
RLOO, PPO, DPO, Iterative DPO, and REBEL. The runtime includes both the gen- eration time and the time required for pol- icy updates. Both runtime and peak mem- ory usage are measured on A6000 GPUs using the same hyperparameters detailed in Appendix H.1.5 for batch of 512 prompts. The measurements are averaged over 100 batches. Methods are ascendingly ordered by winrate. To the right of the dashed line, PPO, RLOO (k: = 4), and REBEL have the highest winrates, which are comparable among them.
80-
30
Time (4)
25
20
15
10
DPO
REIK- FORCE
RaDo
DPO
PP0
(x-4)
REBEL
DPO
REIN
FORCE
RLDO
(K-31
Figure 2: Plot of runtime and memory usage. Base- lines the left-hand side of the dashed line have lower winrates. Methods on the right-hand side of the dashed line have similar winrates to REBEL.
While DPO and REINFORCE are more time and memory-efficient, their performance does not match up to REBEL, as shown in Table 1. RLOO (I: 2) and Iterative DPO have similar runtime and memory usage REBEL since we μ Tr, making REBEL also generate twice per prompt. However, both methods have worse performance than REBEL. Compared to PPO and RLÔO (k = 4), REBEL demonstrates shorter runtimes and lower peak memory usage. PPO is slow and requires more memory since it needs to update two networks (the policy network and the value network). RIOO (k 4) requires four generations per prompt which makes it slow and less memory efficient. In summary, compared to the two baselines (PPO and RLOO (k- = 4)) that achieve similar winrates as REBEL, REBEL more compulationally tractable and simpler implement.
5.2 General Chat
Task. We consider general chat scenario where x is prompt from the user and is a response. We adapt the setting from Zhu al. (2023), using OpenChat-3.5 (Wang et al., 2024) the base model, Starling-RM-7B-alpae
(Zhu et al., 2023) as the reward model, and the Nectar dataset (Zhu et al., 2023). This setup enables direct comparison between REBEL and APA (Zhu al., 2023) which is used to train Starling-LM-7B-aappp
Evaluation. Following previous works, Bench (Zheng et al., 2023), and Open LLM Leaderboard (Beeching et al., 2023) as metrics. AlpacaE- val 2.0 uses prompts from AlpacaFarm (Dubois et al., 2024) compare model responses against reference response generated by GPT-4-TTurbo. We report the winrate over the reference responses. MT-Bench consists of 80 open-ended questions various topics. Answers are scored directly by GPT-4. Open LLM Leaderboard consists of MMLU (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), Are (Clark al., 2018), Winogrande (Sakaguchi al., 2019), TruthfulQA (Lin et al., 2022), and HellaSwag (Zellers et al., 2019). The prompts of the tasks consist of zero or few-shot samples.
AlpacaEval 2.0 (Dubois et al., 2024), MT-
Quality Analysis. The results between models trained with REBEL and baseline methods shown in Table 2. For MT-Bench and AlpacaEval 2.0, under the same setup, REBEL outperforms APA (Zhu et al., 2023) on both metrics, demonstrating the effectiveness of REBEL under chat setting and its superior performance over APA. For the metrics on Open L.M Leaderboard, REBEL is able to enhance the performance of GSM8K and HellaSwag and maintain the overall average the base models. Similar values on MMLU as base models indicate that we preserve the basic capability of the pre-trained model during the RL fine-tuning process. We include a breakdown of MT-Bench in Appendix M.
5.2.1 Ablation: batch size and data sampling distributions
Task. In the previous section, we sample y and y' from Te (-12)) and we use small batch size with [De] = 32. In this section, we investigate the alternative sampling distribution described in Remark 1. Specifically,, at each iteration, we generate 5 responses from n. for each prompt in the entire dataset (i.e., [D,] is the size of the entire dataset), rank them based on the reward model, and set y to be the best of the five responses, and y' to the worst of the five responses. We perform iterations for this setup with
3-BB In-sturutt (Meta, 2024) as the base model, ArmoRM-Llamai- -Llama3-
8

---

Method
MT-Bench
AlpacaEval 2.0
MMLU
GSMBK (5-shot)
Are (25 (25-shot)
Winogrande (5-shol)
TrathfulQA (0-shot)
HellaSwag (10-shot)
LC Win Rate
Win Rate
(5-shot)
Base
7,69
12.2
11.7
63.6
68.5
64.9
80.6
47.3
84.7
APA
7,43
14.7
14.2
63.4
68.0
64.9
81.1
47.3
84.8
REBEL
8.06
17.3
12.8
63.7
68.8
64.3
80.4
48.2
85.0
Table 2: Results General Chat. The best-performing method for each metric is highlighted in bold. Note that the APA result is directly obtained by evaluating the Starling-LM- LL-M---lpphht model.
Method
MT-Bench
AlpacaFival 2.0
MMLU (Sishot)
GSMSK (S-shot)
Are (25
Winogrande (5:shot)
TruthfuQQA (0-shoi)
HellaSwag (10-shot)
All
LC Win Rute
Win Rate
Base DPO
8.10
22.9
22.6
65.8
75.3
62.0
75.5
51.7
78.7
22.3
8.11
44.9
41.6
66.1
74.6
61.3
75.5
51.8
78.9
34.0
REBEL
8.13
48.3
41.8
66.3
75.8
61.7
75.9
28.7
34.5
REGEL (iter 2)
8.07
50.0
48.5
65.9
75.4
61.3
75.5
50.3
78.6
30.4
REBEL. (iter 3)
8.01
49.7
48.
66.0
75.7
61.1
75.7
49.8
78.8
30.0
Table Ablation Results. In this table, REBEL uses larger batch size (the entire dataset) with the
best-of-N and worst-of-N (N 5) of ne as the sampling distributions for generating pairs y. v/. The best-performing method for each metric highlighted in bold. Note that DPO trained on the online data generated by the base model and labeled by the RMM.
8 (Wang et al.) as the reward model, and the UltraFedback dataset (Cui et al., 2023). We compare REBEL with DPO which is also trained for one epoch on the entire dataset with best-of-5 as Yw and worst-of-5 as y sampled from To. In other words, the training data used for the first iteration of REBEL is the same the one we use for DPO We follow the same evaluation methods as the previous section and include Arena Hard (AH) (Li et al., 2024) in our analysis.
Quality Analysis. Results in Table 3 show that REBEL can significantly improve the base model's s performance, especially on AlpacaEval 2.0 and Arena Hard. Compared to DPO, the model trained by REBEL with I iteration is better in almost all datasets, demonstrating the benefit of using the fine-grained reward gap in policy optimization over just the zero-one labels. In this large batch setting, we find that more iterations in general do not help performance. We conjecture that this is the issue of overfitting to the training dataset. A more diverse and larger dataset can potentially address this issue.
5.3 Image Generation
Task. We also consider the setting of image generation, where, given a consistency model (Song et al., 2023) and target reward function, we seek to train the consistency model to output images that garner a higher reward. We use 45 common animals as generation prompts similar to Black et al. (2023); Oertell et al. (2024) and the latent consistency model (Luo et al., 2023) distillation of the Dreamshaper v7 model, a finetune of stable diffusion (Rombach et al., 2021). We compare REBEL to a clipped. policy gradient objective (Black et al., 2023; Fan al., 2024; Oertell et al., 2024) with the aim to optimize aesthetic quality to obtain a high reward from the LAION aesthetic score predictor (Schuhmann, 2022). This base- line does not use critics or GAE for advantage estimates. However, the clipping objective is clearly motivated by PPO, and thus, we simply name this baseline as PPO.
9.0
LAION Aesthetic Score
REBEL
8.5
PPO
8.0
7.5
Evaluation. We evaluate the reward under the LAION aesthetic reward model for an equal number of reward queries/samples generated and equal number of gradi- ent updates. The aesthetic predictor is trained to predict human-labeled scores of images Images that tend to have the highest reward are artwork. Following Agarwal et al. (2021), we report inter-quartile
7.0
6.5
6.0
scale of to 10. means (IQM) with 95% confidence intervals (CIs) across three seeds for both REBEL and PPO. The CIs were cal- culated with percentile bootstrap with stratified sampling over three random seeds.
2000D
40000
60000
Reward Querles
Figure 3: Learning curves as a func- tion of reward queries to the LAION aesthetic predictor. The colored areas represent 95% Cls.
'Directly training DPO on the original Ultrafeedback preference dataset does not provide strong performance under AlpacaEval (e.g.. the LC-winrate is around 28%, see Song et al. (2024)). So for a fair comparison to REBEL, train DPO on the online data generated by mo and labeled by the reward model.

---

PPO
REBEL 8.29
8.00
6.17
6.85
7,08
Figure 4: Generated images using PPO and REBEL during intermediate checkpoint. At the same number of epochs, REBEL observes higher reward under the reward model. This can further seen by the more diverse background of images generated from REBEL with less training time.
Quality Analysis. Figure 3 shows REBEL optimizes the consistency model faster during the beginning of training and eventually achieves performance similar to that of PPO. For our experiments, we tuned both batch size and learning rate for algorithms, testing batch sizes of [4, 8, 16] per GPU and learning rates [1e - 4, 4, Ge - 4, le - 3]. The main difference implementation between PPO and REBEL is the replacement of the clipped PPO objective with regression objective. To maximize LAION -predicted aesthetic quality, both REBEL and PPO transform a model that produces plain images into that produces artistic drawings. We found across multiple seeds that REBEL produced lush backgrounds when compared to PPO's generations. Please see Appendix 1.3 for more examples of generated images.
Conclusion and Future Work
We propose REBEL, an RL algorithm that reduces the problem of RL to solving a sequence of relative reward regression problems on iteratively collected datasets. contrast to policy gradient approaches that require additional networks and heuristics like clipping to ensure optimization stability, it suffices for REBEL to merely drive down error on least squares problem, making strikingly simple to implement and scale. In theory, REBEL matches the best guarantees agnostic setting, while practice, REBEL is able to match and sometimes outperform methods that are far more complex to implement expensive run across both language modeling and guided image generation tasks.
have for RL algorithms in the
There are several open questions raised by our work. The first is whether using loss function other than square loss (e.g. log loss or cross-entropy) could lead to better performance in practice (Farebrother et al., 2024) tighter bounds (e.g. first-order gap-dependent) in theory (Foster and Krishnamurthy, 2021; Wang et al., 2023, 2024). The second is whether, the general (i.e. non- utility-basedl preference setting, the coverage condition assumed in our analysis is necessary we conjecture it is. Relatedly, would be interesting to explore whether using preference (rather than reward) models to provide supervision for REBEL replicates the performance improvements reported by Swamy et (2024); Munos et al. (2023); Calandriello et al. (2024). Third, while we focus primarily the bandit setting in the preceding sections, it would be interesting consider the more general RL setting and explore how offline datasets can be used to improve the efficiency of policy optimization via techniques like resets (Bagnell et al., 2003; Ross and Bagnell, 2014 4; Swamy et al., 2023; Chang et al., 2023, 2024; Ren et al., 2024; Dice et al., 2024).
10

---

Acknowledgements
ZG and JDC are supported by Linkedln under the LankedIn-Corma Grant. GKS supported by his family and friends. KB is supported by NSF under grant No. 2127309 to the Computing Research Association for the CIFellows Project. JDL acknowledges support of the NSF CCF 2002272, NSF IIS 2107304, and NSF CAREER Award 21 144994. WS acknowledges funding from NSF IIS-2154711 NSF CAREER 2339395, DARPA LANCER: LeArning Network CybERagents, and Cornell Infosys Collaboration.
References
Kalashnikov, D.; Irpan, A.; Pastor, P:; Ibarz, J.; Herzog, A.; Jang, E.; Quillen, D.; Holly, E.; Kalakrishnan, M.; Vanhoucke, V:; others Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on robot learning. 2018; pp 651-673.
Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D. M.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; Christiano, P. Learning summarize from human feedback. 2022.
Ouyang, et al. Training language models to follow instructions with human feedback. 2022.
Schulman, J.; Wolski, F:; Dhariwal, P.; Radford, A.; Klimov, O. Proximal Policy Optimization Algorithms. 2017.
Ziegler, D. M.; Stiemnon, N.; Wu, J., Brown, T. B.; Radford, A.; Amodei, D.; Christiano, P.; Irving, G. Fine-Tuning Language Models from Human Preferences. 2020.
Touvron, H. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023.
Black, K.; Janner, M.; Du, Y.; Kostrikov, I.; Levine, S. Training diffusion models with reinforcement learning, arXiv preprint S.MKv::230..33300 2023,
Fan, Y.; Watkins, O.; Du, Y.; Liu, H.; Ryu, M.; Boutilier, C.; Abbeel, P.; Ghavamzadeh, M.; Lee, K.;
Lee, K. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems 2024, 36.
Oertell, O.; Chang, J. D.; Zhang, Y.; Brantley, K.; Sun, W. RL for Consistency Models: Faster Reward Guided Text-to-Image Generation. arXiv preprint 4rXhv:2404.03673 2024,
Schulman, J.; Moritz, P.; Levine, S.; Jordan, M.; Abbeel, P. High-dimensional continuous control
using generalized advantage estimation. arXiv preprint 1rXiiv:500.025 2015,
Kakade, S.; Langford, J. Approximately optimal approximate reinforcement learning. Proceedings of the Nineteenth International Conference on Machine Learning. 2002; pp 267-274.
Ahmadian, A.; Cremer, C.; Gallé, M.; Fadace, M.; Kreutzer, J.; Pictquin, O.; Üstün, A.; Hooker, Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMss. 2024.
Henderson, P.; Islam, R.; Bachman, P.; Pineau, J.; Precup, D.; Meger, D. Deep Reinforcement Learning that Matters. 2019.
Engstrom, L.; Ilyas, A.; Santurkar, S.; Tsipras, D.; Janoos, F:; Rudolph, L.; Madry, A. Implementation Matters Deep Policy Gradients: A Case Study on PPO and TRPO. 2020.
Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; Finn, C. Direct Preference
Optimization: Your Language Model is Secretly Reward Model. 2023.
Swamy, G.; Dann, C.; Kidambi, R.; Wu, Z. S.; Agarwal, A. A Minimaximalist Approach to Rein- forcement Learning from Human Feedback. arXiv preprint arXiv:2401.0400 04056 2024,
Langford, J.: Zhang, T. The spoch-greedyy algorithm for multi-armed bandits with side information.
Advances in neural information processing systems 2007, 20.

---

Ramamurthy, R.; Ammanabrolu, P.; Brantley, K.; Hessel, J.; Sifa, R.; Bauckhage, C.; Hajishirzi, H.; Choi, Y. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines. and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01244 2022.
Chang, J. D.; Brantley, K.; Ramamurthy, R.; Misra, D.; Sun, W. Learning to Generate Better Than Your LLM. 2023.
Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; Amodei, D. Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems. 2017.
Ziehart, B. D.; Maas, A. L.; Bagnell, J. A.; Dey, A. K.; others Maximum entropy inverse reinforcement learning. Aaai. 2008; pp 1433-1438.
Grünwald, P. D.; Dawid, A. P. Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. 2004.
Kakade, S. M. Natural Policy Gradient. Advances in Neural Information Processing Systems. 2001.
Bagnell, J. A.; Schneider, J. Covariant policy search. Proceedings of the 18th international joint conference on Artificial intelligence. 2003; pp 1019-1024.
Hsu, C. C.-Y:; Mendler-Dünner, C.; Hardt, M. Revisiting Design Choices in Proximal Policy Opti- mization. 2020.
Shani, L.; Efroni, Y. Mannor, S. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. Proceedings of the AAAI Conference on Artificial Intelligence. 2020: pp 5668-5675.
Agarwal, A.; Kakade, S. M.; Lee, J. D.; Mahajan, G. On the theory policy gradient methods:
Optimality, approximation, and distribution shift. Journal of Machine Learning Research 2021, 22, 1-76.
Kool, W.; van Hoof, H.; Welling, M. Buy 4 REINFORCE Samples, Get a Baseline for Free! DeepRLStracPPeedee ICLR. 2019.
Calandriello, D.; Guo, D.; Munos, R.; Rowland, M.; Tang, Y.; Pires, B. A.; Richemond, P. H.; Lan, C. L.; Valko, M.; Liu, T.; others Human Alignment of Large Language Models through Online Preference Optimisation. arXiv preprint veXiv:2403.0863 2024.
Rosset, C.: Cheng, C.-A.; Mitra, A.; Santacroce, M.; Awadallah, A.; Xie, T. Direct Nash Opti- mization: Teaching Language Models to Self-Improve with General Preferences. arXiv preprint arXiv:2404.03715 2024,
Bagnell, J.; Kakade, S. M.; Schneider, J.; Ng, A. Policy search by dynamic programming. Advances in neural information processing systems 2003, 16.
Agarwal, A.; Henaff, M.; Kakade, S.; Sun, W. Pe-pg: Policy cover directed exploration for provable policy gradient learning, Advances in neural information processing systems 2020, 33, 13399- 13412.
Song, Y.; Zhou, Y.; Sekhari, A.; Bagnell, J. A.; Krishnamurthy, A.; Sun, W. Hybrid RL: Using Both Offline and Online Data Can Make RI. Efficient. 2023.
Agarwal, A.; Jiang. N.; Kakade, S. M.; Sun, W. Reinforcement learning: Theory and algorithms. 2019.
Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. arXiv preprint 17X1v 2017,
Huang, S.; Noukhovitch, M.; Hosseini, A.; Rasul, K.; Wang, W.; Tunstall, I. The N+ Implementation Details of RLHF with PPO: A Case Study TL:DR Summarization. 2024.
12

---

Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems 2020, 33, 3008-3021.
Williams, R. Simple Statistical Gradient Gradient-Followingg Algorithms for Connectionist Reinforcement Learning. Mach. Learn. 1992. 8, 229-256.
Guo, S.; Zhang, B.; Liu, T:; Liu, T.; Khalman, M.; Llinares, F.; Rame, A.; Mesnard, T.; Zhao, Y.; Piot, B.; others Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792 2024,
Biderman, S.; Schoelkopf, H.; Anthony, Q G.; Bradley, H.; O' Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, S.; Raff, E.; others Pythia: A suite for analyzing large language models across training and scaling. International Conference on Machine Learning. 2023; pp 2397-2430.
OpenAl Gpt-4 technical report. 2023.
Zhu, B.; Frick, E.; Wu, T.; Zhu, H.; Jiao, Starling-7B: Improving LLM Helpfulness & Harmlessness with RLAIF. 2023.
Wang, G.; Cheng, S.; Zhan, X.; Li, X.; Song, S.; Liu, OpenChat: Advancing Open-source Language Models with Mixed-Qualis Data. 2024.
Dubois, Y.; Galambosi, B.; Liang, P.; Hashimoto, B. Length-Controlled Way to Debias Automatic Evaluators. 2024.
Controlled AlpacaEval: A Simple
Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y:; Lin, Z.; Li, Z.; Li, D.;
Xing, E. P.; Zhang, H.; Gonzalez, J. E.; Stoica, I. Judging LLM-as-a-Judge with MTT-Bench and Chatbot Arena. 2023.
Beeching, E.; Fourrier, C.; Habib, N.; Han, S.; Lambert, N.; Rajani, N.; Sanseviero, O.; Tunstall, L.; Wolf, T. Open LLM Leaderboard. cettps:/huugiing open 11m_ lalllllmmmeeleeeeeee leaderboard, 2023.
co/ apwccss/uugingg////ee
Dubois, Y.; Li, X.; Taori, R.; Zhang, T.; Gulrajani, I,; Ba, J.; Guestrin, C.; Liang, P.; Hashimoto, T. B.
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. 2024.
Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; Steinhardt, J. Measuring
Massive Multitask Language Understanding. 2021.
Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.;
Hilton, J.; Nakano, R.; Hesse, C.; Schulman, J. Training Verifiers to Solve Math Word Problems. 2021.
Clark, P.: Cowhey, I,; Etzioni, O.; Khot, T.: Sabharwal, A.; Schoenick, C.; Tafjord, O. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. 2018.
Sakaguchi, K.; Bras, L.; Bhagavatula, C.; Choi, WINOGRANDE: An Adversarial Winograd Schema Challenge at Scale. 2019.
Lin, S.; Hilton, J.; Evans, O. TruthfulQA: Measuring How Models Mimic Human Falsehoods. 2022.
Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; Choi, HellaSwag: Can a Machine Really Finish Your Sentence? 2019.
Zhu, B.; Sharma, H.; Frujeri, F. V.; Dong, S.; Zhu, C.: Jordan, M. I, Jiao, J. Fine-Tuning Language Models with Indante Policy Alignment. 2023.
Meta Introducing Meta Llama 3: The most capable openly available LLM to date. 2024: https: //ai meta. hoombllg/////ooooooooooooo/ooooo htt///mtaalllm 11aaa11//m-3//...
Wang, H.; Xiong, W.; Xie, T.; Zhao, H.; Zhang, T. Interpretable Preferences via Multi-Ohjective Reward Modeling and Mfxture-of-Experre arXiv preprint 12.444206.18245
13

---

Cui, G.; Yuan, L.; Ding, N.; Yao, G.; Zhu, W.; Ni, Y.; Xie, G.; Liu, Z.; Sun, M. UltraFeedback: Boosting Language Models with High-quality Feedback. 2023.
Song, Y: Swamy, G.; Singh, A.; Bagnell, J. A.; Sun, W. Understanding Preference Fine-Tuning Through the Lens of Coverage. arXiv preprint arXiv:2406 240.00016.14 2024,
Li, T:; Chiang, W.-L.:; Frick, E.; Dunlap, L.; Wu, T:: Zhu, B.; Gonzalez, J. E.; Stoica, I. From Crowdsourced Data to High- Q Benchmarks: Arena-Hard and BenchBuilder Pipeline. 2024; https://arxiv: org/ aaaaaba///// abs/2406. 11939.
Song, Y.; Dhariwal, P.; Chen, M.; Sutskever, I. Consistency models. arXiv preprint urXiv:2303.01469 2023,
Luo, S.; Tan, Y.; Huang, L.; Li, J.; Zhao, H. Latent Consistency Models: Synthesizing High- Resolution Images with Few-Step Inference. 2023.
Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. 2021.
Schuhmano, C. Laion aesthetics. htttps:/laaie aeriion.i//bogoe. htlp:///tta///aaaa////////////aa httpaa////laaaooasssttttttttt 2022.
Agarwal, R.; Schwarzer, M.; Castro, P. S.; Courville, A. C.; Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances neural information processing systems
2021, 34, 29304 29320.
Farebrother, J.; Orbay, J.; Vuong, Q.; Taīga, A. A.; Chebotar, Y.; Xiao, T.; Irpan, A.; Levine, S.;
Castro, P. S.; Faust, A.; Kumar, A.; Agarwal, R. Stop Regressing: Training Value Functions via Classification for Scalable Deep RL. 2024.
Foster, D. J.; Krishnamurthy, A. Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discrimination. 2021.
Wang, K.; Zhou, K.; Wu, R.; Kallus, N.; Sun, W. The benefits of being distributional: Small-loss
bounds for reinforcement learning. Advances in Neural Information Processing Systems 2023, 36.
Wang, K.; Oertell, O.; Agarwal, A.; Kallus, N.; Sun, W. More Benefits Being Distributional: Second-Order Bounds for Reinforcement Learning. arXiv preprint 4rX /98 2024,
Munos, R.; Valko, M.; Calandriello, D.; Azar, M. G.; Rowland, M.; Guo. Z. D.; Tang, Y.; Geist, M.; Mesnard, T.; Michi, A.; others Nash learning from human feedback. arXiv preprint arXiv:2312.00m 00886 2023,
Ross, S.; Bagnell, J. A. Reinforcement and Imitation Learning via Interactive No-Regret Learning. ArXiv 2014, abs/1406.5575
Swamy, G.; Choudhury, S.; Bagnell, J. A.; Wu, Z. Inverse Reinforcement Learning without Reinforcement Learning, ArXiv 2023, abs/2303. 14623.
Chang. J. D.; Shan, W.; Oertell, O.; Brantley, K.; Misra, D.; Lee, J. D.; Sun, W. Dataset Reset Policy Optimization for RLHF arXiv preprint arXiv:2404.08495 2024,
Ren, Swamy, G.; Wu, S.; Bagnell, J. A.; Choudhury, Hybrid Inverse Reinforcement Learning. arXiv preprint arXiv:2402.0884. 2024,
Dice, N. E.; Swamy, G.; Choudhury, S.: Sun, W. Efficient Inverse Reinforcement Learning without Compounding Errors. ICML 2024 Workshop on Models of Human Feedback for AI Alignment. 2024.
Nemirovskij. A. S.; Yudin, D. B. Problem complexity and method efficiency in optimization. 1983.
Konda, V.; Tsitsiklis, Actor-Critic Algorithms. Advances in Neural Information Processing Systems. 1999.
Richter, L.; Boustati, A.; Nüsken, N.; Ruiz, F.; Akyildiz, O. D. VarGrad: a low-variance gradient
estimator for variational inference, Advances in Neural Information Processing Systems 2020, 33, 13392.
14

---

Zhu, B.; Jordan, M.; Jiao, J. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. International Conference on Machine Learning. 2023; pp 43037-43067.
Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; Moritz, P. Trust region policy optimization. International conference machine learning. 2015; 1889-1897.
Peters. J.; Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. Proceedings of the 24th international conference Machine learning. 2007: pp 745-750.
Peng, X. B.; Kumar, A.; Zhang, G.; Levine, S. Advantage- -eighet Regression: Simple and Scalable Off-Policy Reinforcement Learning. 2019.
Zhou, Z:; Liu, J.; Yang, C.: Shao, J.; Liu, Y:; Yue, X.; Ouyang, W.; Qiao, Y. Beyond One-Preference- Fits-All Alignment: Multi-Objectio Direct Preference Optimization. 2023.
Jacq, A.; Geist, M.; Paiva, A.; Pictquin, O. Learning from a Learner. Proceedings of the 36th
International Conference on Machine Learning. 2019; pp 2990-2999.
Watson, J.; Huang, S. H.; Heess, N. Coherent Soft Imitation Learning. 2023.
Anthropic Introducing the next generation of Claude. 2024; atcpp::/www.anthhroicc. mattrropi..com/nessss 3-maaily.
Nakano, R. et al. WebGPT: Browser-assisted question-answering with human feedback. 2022.
Lee, K.; Liu, H.; Ryu, M.; Watkins, O.; Du, Y.; Boutilier, C.; Abbeel, P.; Ghavamzadeh, M.; Gu, S. S.
Aligning Text-to-Image Models using Human Feedback. 2023.
Azar, M. G.; Rowland, M.; Piot, B.; Guo, D.; Calandriello, D.; Valko, M.; Munos, R. A General Theoretical Paradigm Understand Learning from Human Preferences. 2023.
Ethayarajh, K.; Xu, W.; Kiela, D. Better, Cheaper. Faster LLM Alignment with KTO. 2023; https: //contextual. ai/better ai/better-cheeper fwster-sslaaee 11m al weliinneent--m with-kto/.
Lambert, N.; Pyatkin, V.; Morrison, J.; Miranda, L.; Lin, B. Y.; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y.; Smith, N. A.; Hajishirzi, H. RewardBench: Evaluating Reward Models for Language Modeling. 2024.
Tajwar, F.; Singh, A.; Sharma, A.; Rafailov, R.; Schneider, J.; Xie, T.; Ermon, S.; Finn, C.; Kumar, A. Preference Fine- Muni-Tuning of LLMs Should Leverage Suboptimal, On On-Policy Data. 2024.
Ross, S.; Gordon, G.; Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. Proceedings the fourteenth international conference on artificial intelligence and statistics. 2011; pp 627-635.
Swamy, G.; Choudhury, S.; Bagnell, J. A.; Wu, S. Of moments and matching: A game- theeettccreetc framework for closing the imitation gap. International Conference on Machine Learning. 2021; pp 10022-10032
Xiong, W.; Dong, H.; Ye, C.; Wang. Z.; Zhong, H.; Ji, H.; Jiang, N.; Zhang, T. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint. 2024.
Mao, X.; Li, F.-L;; Xu, H.; Zhang, W.; Luu, A. T. Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration. arXiv preprint arXiv:2402. 16030 2024,
Munos, R. Error bounds for approximate policy iteration. ICML. 2003; pp 560-567.
Wu, Y.: Sun, Z.; Yuan, H.; Ji, K.; Yang, Y.; Gu, Q. Self-Play Preference Optimization for Language Model Alignment. arXiv preprini arXiv:2405.00675 2024,
Ball, P. J.; Smith, L.; Kostrikov, I.; Levine, S. Efficient Online Reinforcement Learning with Offline Data, 2023.
Zhou, Y.: Sekhari, A.; Song, Y.; Sun, W. Offline data enhanced on-policy policy gradient with provable guarantees. arXiv preprint arXiv:2311.08384 2023,
15

---

May, K. O. Intransitivity, utility, and the aggregation of preference patterns, Econometrica: Journal of the Econometric Society 1954, 1-13.
Tversky, A. Intransitivity of preferences. Psychological review 1969, 76, 31.
Gardner, M. Mathematical games. 1970; httpss//ww..ccinniiicceeeeeeeeee http:://.eecaaaaaaaaaa.aoa..o.aa com/article/ mathematical gww.g-amm---220--
Dudík, M.; Hofmann, K.; Schapire, R. E.; Slivkins, A.; Zoghi, M. Contextual dueling bandits.
Conference on Learning Theory. 2015; pp 563-587.
Ye, C.; Xiong, W.; Zhang. Y.; Jiang, N.; Zhang, T. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint 4rXiv:2402.07314 2024.
Kreweras, G. Aggregation of preference orderings. Mathematics and Social Sciences I: Proceedings of the seminars of Menthon-Saine Saint-Bernardd, France (1-27 July 1960) and of Gösing, Austria (3-27 July 1962). 1965; pp 73-79.
Fishburn, P. C. Probabilistic social choice based on simple voting comparisons. The Review of Economic Studies 1984, 51, 683-692.
Kramer, G. H. On Class of Equilibrium Conditions for Majority Rule. Econometrica 1973, 41, 285-97.
Simpson, P. B. On Defining Areas of Voter Choice: Professor Tullock on Stable Voting. The Quarterly Journal of Economics 1969, 83, 478-490.
Yue, Y.; Broder, J.; Kleinberg, R.; Joachims, T. The k-armed dueling bandits problem. Journal of Computer and System Sciences 2012, 78, 1538-1556.
Wang, Y.; Liu, Q.; Jin, C. Is RLHF More Difficult than Standard RL? A Theoretical Perspective. Teirty- Conference on Neural Information Processing Systems. 2023.
Cui, Q.; Du, S. S. When are Offline Two-Player Zero-Sum Markov Games Solvable? Advances in Neural Information Processing Systems 2022, 35, 25779-25791.
Zhong, H.; Xiong. W.: Tan, J.; Wang. L.; Zhang, T.; Wang. Z.; Yang. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. International Conference on Machine Learning. 2022: pp 271 17-27142.
Cui, Q: Du, S. Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonus. Advances in Neural Information Processing Systems. 2022; pp 1 11739-11751.
Xiong, W.; Zhong, H.; Shi, C.; Shen, C.; Wang, L.; Zhang, T. Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markoy Game. The Eleventh International Conference Learning Representations, 2023.
Hu, E. J.: Shen, Y.; Wallis, Allen-Zhu, Z.; Li, Y.: Wang, S.; Wang, L.; Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations. 2022.
Huang, S.; Noukhovitch, M.; Hosseini, A.; Rasul, K.; Wang, W.: Tunstall, L. The N+ Implemen- tation Details of RLHF with PPO: A Case Study on TL; DR Summarization. arXiv preprint arXiv:2403.17031 2024,
16

---

A Detailed Discussion of Related Work
Policy Gradients. Policy gradient (PG) methods (Nemirovskij and Yudin, 1983; Williams, 1992; Konda and Tsitsiklis, 1999; Kakade, 2001; Schulman et al., 2017) are prominent class of RL algo- rithms due to their direct, gradient- based policy optimization, robustness to model misspecification (Agarwal et al., 2020), and scalability modern Al applications from fine-tuning LLMs (Stiennon et al., 2022) to optimizing text-to-image generators (Oertell et al., 2024).
Broadly speaking, we can taxonomize PG methods into two families. The first family is based on REINFORCE (Williams, 1992) and often includes variance reduction techniques (Kool et al., 2019; Richter et al., 2020; Zhu et al., 2023). While prior work by Ahmadian et al. (2024) has shown that REINFORCE based approaches can outperform more complex RL algorithms like PPO on LLM fine-tuning tasks like TL;DR, we find that a properly optimized version of PPO still out-performs REINFOCCE baseline. The second family is adaptive PG techniques that precondition the policy gradient (usually with the inverse of the Fisher Information Matrix) to ensure it is covariant to re-parameterizations of the policy, which include NPG (Kakade, 2001; Bagnell and Schneider, 2003) and its practical approximations like TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017). Intuitively, the preconditioning ensures that we make small changes in terms of action distributions, rather than in terms of the actual policy parameters, leading to faster and more stable convergence. Unfortunately, computing and then inverting the Fisher Information Matrix is computationally intensive and therefore we often resort to approximations in practice, as done in TRPO. However, these approximations are still difficult to apply to large-scale generative models, necessitating even coarser approximations like PPO. In contrast, REBEL does not need any such approximations to be implemented at scale, giving us much closer connection between theory and practice.
Reward Regression. The heart of REBEL is a novel reduction from RL to iterative squared loss regression. While using regression to fit either the reward (Peters and Schaal, 2007) or the value (Peng et al., 2019) targets which are then used to extract a policy have previously been explored, our method instead takes a page from DPO (Rafailov et al., 2023; Zhou et al., 2023) and inverse RL methods (Jacq et al., 2019; Watson et al., 2023) to implicitly parameterize the reward regressor in terms of the policy. This collapses the two-stage procedure of prior methods into a single step.
Preference Fine- (PFT) of Generative Models. RL has attracted renewed interest due to its central role in "aligning" language models- - i.c., adapting their distribution of prompt completions towards the set of responses preferred by human raters.
One family of techniques for PFT, often referred to as Reinforcement Learning from Human Feedback (RLHF) involves first fitting a reward model (i.e. classifier) to the human preference data and then using this model to provide reward values to a downstream RL algorithm (often PPO) (Christiano et al., 2017; Ziegler et al., 2020). LLMs fine-tuned by this procedure include GPT-N (OpenAI, 2023), Claude-N (Anthropic, 2024), and Llama-N (Meta, 2024). Similar approaches have proved beneficial for tasks like summarization (Stiennon et al., 2022), question answering (Nakano et al., 2022), text-to-image generation (Lee et al., 2023), and instruction following (Ouyang et al., 2022).
Another family of techniques for PFT essentially treats the problem as supervised learning and uses a variety of ranking loss functions. It includes DPO (Rafailov et al., 2023), IPO (Azar et al., 2023), and KTO (Ethayarajh et al., 2023). These techniques are simpler to implement as they remove components like an explicit reward model, value network, and on-policy training from the standard RLHF setup. However, recent work finds their performance to be lesser than that of on-policy methods (Lambert et al., 2024; Tajwar et al., 2024), which agrees with our findings. This is perhaps caused by their lack of interaction during training, leading to the well-known covariate shift/compounding error issue (Ross et al., 2011; Swamy et al., 2021) and the associated lower levels of performance.
The third family of PFT techniques combines elements from the previous two: it involves running an offline algorithm iteratively, collecting on-policy preference feedback from either a supervisor model such as GPT4 (Rosset et al., 2024; Xiong et al., 2024; Guo et al., 2024) or from preference model fit on human data (Calandriello et al., 2024). All of these approaches can be considered instantiations of the general SPO reduction proposed by Swamy et al. (2024), which itself can be thought of as a preference-be baaee variant of DAgger (Ross et al., 201 Recent work by Tajwar et al. (2024) confirms the empirical strength of these techniques which leverage additional online data. Our approach fits best into this family of techniques we also iteratively update our model by solving a sequence of supervised learning problems over on-policy datasets. However, REBEL comes with several key
17

---

differentiating factors from the prior work. Online versions of DPO or IPO (Xiong et al., 2024; Tajwar al., 2024; Guo et al., 2024; Calandriello et al., 2024; Munos al., 2023) essentially use a reward / preference model to generate binary win-loss labels while REBEL actually uses the output of the reward model as a regression target, taking advantage of this more nuanced feedback, In contrast to Rosset et al. (2024), algorithmically. REBEL does not use any online preference feedback from GPT4 nor does it require to generate large number of responses per prompt, both of which are extremely expensive as reported by Rosset et al. (2024). Theoretically, we are able to prove policy performance bounds under a much weaker coverage condition. Unlike Mao et al. (2024) that regularize to the initial policy To during updates, we perform conservative updates by regularizing T(+1 to πe. When doing the former, it is difficult to prove convergence or monotonic improvement as the current policy can just bounce around ball centered at To. well-known issue in the theory of approximate policy iteration (Kakade and Langford, 2002; Munos, 2003). In contrast, by incorporating the prior policy's probabilities into regression problem, we able prove stronger guarantees for REBEL. When applying REBEL to the general preference setting, our algorithm shares some similarities with the concurrent work of Wu al. (2024), which also leverages the DPO reparameterization trick cleanly implement an Online Mirror Descent approach to computing minimax winner via self-play. The key difference that REBEL uses paired responses, while the algorithm of Wu al. (2024) does not. Using paired responses, REBEL is able to cancel out the partition function and therefore does not need resort to heuristic approximations of it (specifically, assuming the partition function is always equal to constant). Furthermore, we can run REBEL with datasets consisting of mixture of on-policy and off- off-policy data with strong guarantees, enabling hybrid training. as previously explored in the RL (Song et al., 2023; Ball et al., 2023; Zhou et al., 2023) and inverse RL. (Ren et al., 2024) literature.
18

---

B Proof of Claim 1
We prove claim 1 in this section. We start from deriving the Fisher information matrix.
Fe = Ex W (o To, (y/x) Vo Te (Vo In To, (y)z) - Ve In (v
172
2 Eg Egggormmi
Va Voln Vo To. (y/x))
172
where the last equality uses the fact that cross terms from completing the square are zero. Now recall Eq. which an ordinarly least square regression problem. The minimum norm solution of the least square regression problem is:
(n/2) (Ba.rraaa (V V Vn (v)]=)) (r(z, v) r(x,81))) (n/2) (Ex.yoo. (ol En,y' Vo ne, (2/12 (x)r 1/)l
-B Vo
=m1/2)Bb Fi! (En.swa Vo In E (ynn(1))
+ Ezger Vo
v) ((
[n (au)) Vo In
(I,
where again the fact that Bynao, (122 Vo In To, (v)=)g(a)) 0 for any function g(x),, and define Advantage A v) r(z, (a,,
19

---

C Proof of Claim
We prove claim 2 in this section. We start by approximating our predictor 1 In (z) To. (1)az) 7 by its first order Taylor expansion at 0g: (In (o(v) - In no, (v/=²) ~ , Vo In na, (u|zr) T (0 Oz).
where x indicates that we ignore higher order terms in the expansion. Setting := 0 Og and replace (In (n(o))))))))))))) - In ne. (y(z)) by its first order approximation in Eq. 1, we arrive at:
min
Σ
2 (Volnt (zla) Ve (n 8 (r(a,v) - ra.v1)))
(1I)
under finite setting.
Following the previous derivation, we have the unbiased estimate of Fisher information matrix under the finite setting as:
F.
2 2
Σ
Vo No, (unlzn) In To, To, (yalan))
Since Eq. 11 is ordinarly least square regression problem. The minimum norm solution of the least square regression problem is:
5 (
(Vo In me, (Unlzn)) Vo In To, (oh (r( Mne) r (ar v'h))
= - 2N
Σ n
Vin v Wo, (vml Un) - r( v/))) + V In To, (w 2nd - r(Ens Vn)))
20

---

D Extending REBEL to General Preferences
In the above discussion, we assume we are given access ground reward function. However, in the generative model fine-tuning applications of RL, we often need to learn from human preferences, rather than rewards. This shift introduces a complication: not all preferences can be rationalized by an underlying utility function. In particular, intransitive preferences which are well well-known to result from aggregation of different sub- populatiins or users evaluating different pairs of items on the basis of different features (May, 1954; Tversky, 1969; Gardner, 1970) cannot be accurately captured by single reward model. To see this, note that if we have a > b, b > C, and > a, it is impossible to have a reward model that simultaneously sets f(a) > -(bb, f(b) > f(c), and r(c) > f(a). As we increase the space of possible choices that of all possible prompt completions, the probability of such intransitivities sharply increases (Dudík et al., 2015), as reflected in the high levels of annotator disagreement in LLM fine-tuning datasets (Touvron et al., 2023). Thus, rather than assuming access to reward model, in such settings, we assume access to preference model (Munos et al., 2023; Swamy et al., 2024; Rosset et al., 2024; Ye et al., 2024).
D.1 Came-Theoretic Perspective on Learning from Preferences
More specifically, for any tuple (z. y, y'), we assume we have access to P(y > 3 (r): the probability that y is preferred y'. We then define our preference model I as
1( v/) - 2- Ply > v'(z) 1.
(12)
Observe that (a,, y, y E [-1, is skew-symmetric, i.e., ((r, y, v) (x,, + [2,, y .))
for all x N. y' E y. If the learner can only receive binary feedback o E {0, 1} indicating the preference between y and y', we assume o sampled from a Bernuulii distribution with mean P(y > y' [z), where o = 1 means that y is preferred over y' and 0 otherwise.
Given access such a preference model, solution concept the preference aggregation problem with deep roots in the social choice theory literature (Kreweras, 1965; Fishburn, 1984; Kramer, 1973; Simpson, 1969) and the dueling bandit literature (Yue et al., 2012; Dudík et al., 2015) is that of minimax winner (MW) TMw: the Nash Equilibrium strategy of the symmetric two-player zero-sum game with l payoff function. In particular, due to the skew-symmetric property of I, Swamy et al. (2024) proved that there exists a policy TMW such that
max (1tp::pppp/a(//)).//// a)) [xx y, v')] min
[(x y, v/)]
This implies that (TMW: TMW) is Nash Equilibrium (Wang et al., 2023; Munos et al., 2023; Swamy et al., 2024; Ye et al., 2024). As is standard in game solving, our objective to obtain an c-approximate MW स measured by the duality gap (DG):
DG i
max (ao (Z,
min E Egwp.y (11)) (( u. v.'') S e.
In the following discussion, will use 1(x, y, T) to denote Ey (m) (II v/) and 1(TT, x') to
denote E (. (1ap:///aa y, v²) for notational convenience.
D.2 Self-Play Preference Optimization (SPO) with REBEL as Base Learner
We can straightforwarddy extend REBEL to the general preference setting via an instantiation of the Self-Play Preference Optimization (SPO) reduction of Swamy et al. (2024). In short, Swamy et al. (2024) prove that rather than performing adversarial training, we are able to perform simple and stable self-play procedure while retaining strong theoretical guarantees. Practically, this corresponds to sampling at leas two completions from the current policy, querying learned preference supervisor model on each pair, and using the winrate for each completion as its reward. We will now describe how we can adapt REBEL to this mode of feedback.
Assuming that we can query the preference oracle 1(x, y. v') at will, we can modify the least square objective Eq. (1) to
4.11 == argmin
Σ
(1 17
In no, (y|xr)
To (y(l))
In To(y'1o))
((x,v.v") 1(x, V. -''))
2
To. (v'(z))
2. o ED.
21

---

where P.y Tr(-)=),"" T((-)x),y/ p(-12)) When the exact value of 1(x, y. y²) is unavailable but only binary preference feedback 0y {0, 1 sampling from Berouulli with mean (x, V, v²) is available, we can just replace I(ar, y. 1/ I( v v²) by Oz.y' Oy'.v" It is easy to see that the Bayes optimal of the above least square regression problem is equal to:
By (- M. =²) By"" v/ x²) ((,, y, me) 11,, Te).
Swamy (2024) define an iteration- deaetion--eendeent reward re(ar, y) == Ey (w) Mx, y, y") (x, y, )T). Thus, the above regression problem can be understood setting where the reward function changes at each iteration t. Swamy et al. (2024) shows that running the exact MD (Eq. 3) with this iteration depeedend reward function r; leads to fast convergence an approximate Minimax Winner, a property that we will use to provide the regret bound of REBEL in the general preference setting while accounting for nonzero mean squared error.
extension of REBEL the
22

---

E Justification for Assumption
Intuitively, this assumption is saying that there a function in our class of regressors that is able to accurately fit the difference of rewards. Recall that our class of regressors is isomorphic to our policy class. Therefore, as long as our class of policies expressive, we would expect this assumption to hold with small c. For all domains we consider, our policy class is a flexible set of generative models (e.g. Transformer-ban assumption holds in practice see Figure 6 in Appendix L. for empirical evidence of this point and Example for more discussion.
LLMs or diffusion models). Thus, we believe reasonable to believe this
More formally, the above assumption bounds the standard in-distribution generalization error (v.s. the point-wise guarantee in Eq. 8) of well-defined supervised learning problem: least squares regression. The generalization error captures the possible errors from the learning process for 0.+1 and it could depend on the complexity of the policy class and the number of samples used in the dataset D, For instance, when the the function In In x' induced by the log- de--dffffeence of two policies (TT, x') are rich enough (e.g., policies are deep neural networks) to capture the reward difference, then in this assumption converges to zero as we increase the number of training data. Note that while e can be small, does not imply that the learned predictor will have small prediction error in point-wise manner- it almost certainly will not.
Example 1. One simple example is when Ty((²) & exp(0) (0p00To(zu0 v)) for some features 6(I, y)- - In( (a((T(///)//a (z)/m<(///==) = (0 (0)+)) (δ(x, y) 6(x,y') which means that regression problem in Eq. I is classic linear regression problem. When the reward r(x, y) is also linear in feature Φ(x. y). then Eq. I is a well-specified linear regression problem, and e typically scales in the rate of O [d/ with d being the dimension of feature 6.
In this case, (n(r(yl))/T<(UU (t
We can extend the above example to the case where e.g,, RBF kernel or even Neural Tangent Kernel, which allows to capture the case where is softmax wide neural network with the least square regression problem solved by gradient flow. The error again scales poly(d/ (Pl). where d is the effective dimension of the corresponding kernel.
the feature corresponding to some kernel,
23

---

F Proof of Theorem 1
In this section, we provide the proof of theorem 1. For notation simplicity, throughout the proof, we denote ny for . and define f( y) == 1 In ((t2())))
The following lemma shows that the learned functiin fa can predict reward r well under both ne and j up terms that are y-independent.
Lemma 1. Consider any t E [T]. Define ((x,)) y) = ft v) r(x, Define (x) = ( (a, y) and (x) [x) Under assumption 1, for all t, we have the following:
E (a. (Je(z, y) - r(r,y) - (x) Se
(13)
En a) (fe(r. r(r, Aa(r))² Sc,
(14)
E. ( Sc.
(15)
Proof. From assumption 1, we have:
Emm y1) An, (2) (())) (((
- r(x,
(x) - An(x))²
Br mn (F(( A (x) r(2,01))) Rx v1)
(z) - r(),
-2E wa
AR. ((.))) (f( 1/2)
A1(x) - r(x, (2))
+ +2E ~TT (ft( - (z) -=2, (A (z) (x)
-2E 2E ~ (2) - Ax(x) -r(x, A. (x)) + Ex (A (x) - 22
En ~m (z)())))) A.. (x) r(z. yn ))² En,yz~ (I, 82) (z) ((.
+ (x)
se.
In the above, we first complete the square, and then we only keep terms that are not necessarily zero. Since all the remaining three terms are non-negative, this concludes the proof.
By the definition of ft, we have A(x. y) = In Nesn r(x. y). Taking exp on both sides, we get:
n. (xle)) y) + A(z, v) (x))))
Vr,y: Te+1 (u/x) = (u(=)) exp (m](((, v) + A (ar
exp( exp(-nA,(2))
Denote g ge(x,)) v) == r(x, v) + A(x,y) (x),), and the advantage A, = ge v)
By' (1a) (.. We can rewrite the above update rule as:
Yr,y: (x)(( xx कe (vlx) (x,
(16)
In other words, the algorithm can be understood as rumning MD the sequence of A, for t 1. The following lemma is the standard MD regret lemma.
Lemma 2. Assume maxx,y, |Ae(a,y)] S A E IR+, and mo{-12) uniform over Y Then with 7 = (l((DD/(AAT)) for the sequence of policies computed by REBEL, we have:
T-1
Vn, x
Σ
(e) Ar(,,)) S 2A/II(DD))T.
Proof. For completeness, (e(y)r) exp(nA (x) where (z(x) is the normalization constant, taking log on both sides, and add Ey-+ we have:
provide the proof here. Start with (f/1(y)z)
- [L(((((>)]] (-Ix)) (L(T( + (2) By (-)=)() In Zz()).
Rearrange terms, we get:
-K(((())))l + KL(T(1 Ey ( (n,((,))) v) + In (x)
24

---

For In (x)), using the condition that S 1/A, we have nA, v) S 1, which allows us to use the inequality exp(x) + x + x² for any x S 1, which lead the following inequality:
In (x) In (By-a((1)) exp(n)((2; v)))
- In Σ T<(y)x) (1 (1+ +,,, - o A((,
In + 0 -/ 2 A
where the last inequality uses In(1 + x) S x. and we used the fact that E. Eynn (a)) Ar v) = 0 due to the definition of advantage A,. Thus, we have:
< (-I=) + (L (-<r)(((>)]] T (+1x) S (B 2
Sum over all iterations and do the telescoping sum, we get:
T Σ
E y Ar & KL(T(()))| TnA? S (n((D))/n==Ta
With v (I(T)))(44T)) we conclude the proof.
With the above, now we are ready to conclude the proof of the main theorem.
Proof of Theorem 1. Consider comparator policy T². We start with the performance difference between σt and the uniform mixture policy मे Σ TT/T:
T
T-1
((a)) v) - (1)) = T
E Byy (12)) (A v)),
T
t=
where we define the real advantage (,( == r(x, (ee v). Continue, we have:
T
E Bgy (1x)) (x, 3))
T
1=
T-1
T-1
E B ( x) (Ae(z,
+
E
By (112) (A**(z,v) A(xx, v)
T
T
t=
(P)
< 2A
EgE y (11z) (At
(A (I, Ac(x.u)2
T
T
where the last inequality uses Lemma At(x,)))²
We now just need bound Eg (ATh
nt (y(2x) Ey (12) (1e (A.((()) ax(x,)))== E,E p(ylz)
(A*((x))) Ac(x,
S
(I, Ar y))²
where the last inequality uses the definition of concentrability coefficient C.., We now bound E (2 (1 v) Axx 2) Recall the definiton of A, from Lemma
E, E (A (z. y) Axxx, v))²
= Ee y) - Ey (a),) (x, v) + Eyyna y/1)²
22Ex.g-n(1z)) (r(z. y) (x. y))² + B. (r(x,)) - (x.
Recall the ge(x,, v) = r(x, A(x (x) and from Lemma we can see that
Eeggn( (1 (1 2 = E (A (zx (x)))) s c.
25

---

For ByByyrmoti (-))) (r(a, 3²) 90 y²))² we have:
E, Ey (r(2, v/) gi vx) = (xByy-m((1a)) (A 1/) - Ax()))
By (k) (z, x. + A., (a) - (z))
1 Ey (12) (A 3/) -., (z)) 2E, (A... (z) (z)) S 4c,
where the last inequality uses Lemma again. This step relies on the fact that one of the samples is always on-policy, i.e., from T..
Combine things together, we can conclude that:
3) (1a) (I,
Ac(x, v))2 -
Finally, for the regret, conclude:
In |
V 10c 2 In |n
(x (A (x, v)) 2A1
Σ
v Cx. 10e.
T
T
26

---

G Extension of analysis to General Preferences
Extending the above analysis to the general preference case straightforward except that requires stronger coverage condition. This is because we want to find Nash Equilibrium, which requires comparison between the learned policy against all the other policies. Results from the Markov Game literature (Cui and Du, 2022; Zhong et al., 2022; Cui and Du, 2022; Xiong et al., 2023) and Cui and Du (2022) have shown that the standard single policy coverage condition used single-player optimization is provably not sufficient. In particular, they propose using notion of unilateral concentrability for efficient learning, which can be defined as
TMw(y)z))<("(o) ply (
Cuni.u max T.
in the general preference setting. Notably, the above unilateral concentrability coefficient Cuni.p m(yl=] since Cy. S Cuni.p < Ch- Therefore in the following
is equivalent to Cp. == n, p(y/#]
discussion, we will use C₁ as the coverage condition. In addition, we also assume the generalization error of the regression problem is small,
Assumption 2 (Regression generalization bounds for general preference). Over T iterations, assume that for all t, we have:
(1 77
In T8 (u/æx)
Te. (2/(( (x)) ne. (1.(z)))) m0)1 (1(x, y. mT) 1x,, y' s..e mi))) S
In
TT, (y|z)
for some
Under the above coverage condition and generalization bound, we can show that REBEL is able to learn an approximate Minimax Winner:
Theorem 2. With assumption after T many iterations, with a proper learning rate 1) the policy क Unif( (T Meree IT satisfies that:
DG( so CV
Here the O-notation hides problem-dependers constants that are independent of e, Cu,
Note that the coverage condition here is much stronger than the single policy coverage condition in the RL setting. We conjecture that this is the cost one has to pay by moving to the more general preference setting and leaving the investigation of the necessarily coverage condition for future work.
G.1 Proof of Theorem
Recall that re(a,,) 10, y, m.). Let us define (' v) = Se(a, - re(a,,)) v), A me
(x)
Eyszi(la) A (xx y) and (z) = Ey y). Then following the same arguments in
Lemma 1, we have
- r)( E Exxp.r~t((1 -r((,, AI, (x)) Se.
(17)
E Exnpyyva(( (J - () ((()) Se,
(18)
Egnp [AL, (x) ( Se.
(19)
With slight abuse of the notation, We also use 9r and A₁ (z, y) to denote (r, y) + +1 and ge ge y) Ex (x, s/) Then following the same arguments in Lemma 2,
(x, -) A (x.
Vn, Σ
(1) s (DDIIT
(20)
27

---

Note that we have
T
max((T,
R)
max
TT)
x
T
T E B
1
max
Exmp. Ex---sp- = max
[A (x. v),.
x T
T
where A == re(a,, 8) - E g [2) v)l. The last step is due to the skew symmetry of I, iee, (eywa,(()))) (1 (x, y)] = 1(x, T,, Tx) = 0. Then by following the same arguments in the proof of have for any policy n,
Theorem I, with (19)01))19)22))
T (espuvaatl)) (A, s $2AV In y
V10C
This implies that
In D
In | 22AV
max (ax max 2 T
+ V100
VIOC,e.
T
Note that due to the skew symmetry of I, we have
min ( (T, π)
min (e [1x v.2/)] max (1))) 1(xx, ((x,1,
In | max (a (1,,) = mmx - 1-22-2244 -2
/10Cμe.
T
Therefore we have
In |
DG( S 4A
+ 22 100,
T
28

---

H
Additional Experiment Details
H.1 Summarization
H.1.1
Dataset Details
We present dataset details in Table 4. Dataset available at https://github. com/ openai/
summarize- www.-rfooo--eeeee
Table 4: Dataset split, prompts, and maximum generation length for TL,DR summarization
Dataset
Train/Val/Test
Prompt
Generation Length
Human Reference
117K/6445K6655K
"TL;DR:"
53
Preference
92.9K/83.8K/
"TL:DR:"
53
H.1.2 Model Details
For SFT models, we train Pythia 1.4B (Biderman et al., 2023)² model for epoch over the dataset with human references as labels, and use the existing fine-tuned 2.8B* and 6.9B" models. For reward models, we train Pythia 1.4B parameter model for epoch over the preference dataset and use the existing reward models with 2.8B² and 6.9B* parameters. For both REBEL and bascline methods using 1.4B and 2.8B parameters, we trained the policy and/or the critic using low-rank adapters (LoRA) (Hu et al., 2022) on top of our SFT and/or reward model respectively. For the 6.9B models, we perform full-parameter training. The 1.4B and 2.8B models are trained on A6000 GPUs for one day and two days respectively. The 6.9B model is train on 8 H100 GPUs for two days.
H.1.3 Baseline Implementation Details
For supervised fine-tuning (SFT), reward modeling training, PPO, and DPO, we follow the implemen- tation at https: //github. cww com/vwxyzjn/ //sn////iz//eeeeeeee wwamariie__foonffeedbbeeee frrrom_feedbacc_eeettteee details (Huang et al., 2024). For iterative dpo, we implement as follows:
Algorithm 2 Iterative DPO
1: Input: Reward r, policy class = {ma}, parameter 3
2: Initialize policy T0.-
3: for = 0 to T do
4: Collect dataset D, {x, y, y'} where p.y ne(-)x), of ne(-(x)
5: Solve square loss regression problem:
0.41 = argmin
Σ
In
(33l (T(()))
To (5((z))) .1nn
)sga (r(z, - r(x,
To, (3(z)) no, (v'lz)
(21)
end for
where sgn is a sign function. Our implementation of iterative DPO is similar to REBEL where, at each iteration, we update with respect to Te, The major difference that REBEL regresses toward the differences in rewards while iterative DPO only utilizes the pairwise preference signal from the rewards.
'HuggingFace Model Card: BleutherA//pyttiie-1.66 1.4-hii---1---bduppee sft tldr
'uugginFaace Model Card: wwaygju/EleutherAAl 2yttha-2.8bb-eddue
"HuggingFace Model Card: vwxyzjn/EleutherAI 6atp:lhherrrr_pprhhoop.e. sft tld
" 'HuggingFace Model Card:
htt.a:l-A___aaaaa222----ddaaapa pewarrd rewarddwddd__.... tldr
$ FauggingFace Model Card: wwwzgio/EleutherrALLytiiieeeeee deb-heeduee.bdddddueeededee reward tldr
29

---

H.1.4 Reward Details
To ensure that xo remains close to no, we apply an additional KL penalty to the reward:
r(z,y) = (M v) 7l1 na, (y/=) In mo, (y]xr)
(22)
where (M y) is score from the reward model given prompt æ and response y. Furthermore, to ensure that the generations terminate within the maximum generation length, we penalize any generation that exceeds this length by setting r(z, y) to a small fixed constant, l.
For TL;DR summarization, we set 7 = 0.05 and r -1.
H.1.5
Hyperparameter Details
Parameter setting for TL;DR summarization
Setting
Parameters
SFT & RM
batch size:
schedule:
cosine decay
learning rate: 3c-6
train epochs:
PPO
batch size: 512
discount factor:
learning rate: 3e-6
gae A: 0.95
schedule: linear decay
clip ratio: 0.2
train epochs:
value function coeff: 0.1
num epochs: 4
kl coefficient:
DPO
batch size: 64
train epochs:
learning rate: 3e-6
B: 0.05
schedule: linear decay
RLOO
batch size: 512
epochs: kl coefficient: 0.05 K: 2 or
learning rate: 3e-6
schedule: linear decay
REINFORCE
batch size: 512
train epochs:
learning rate: 3e-6
kl coefficient: 0.05
schedule: linear decay
Iterative DPO
batch size: 512
num epochs: 4 B: 0.05
learning rate: 3e-6
schedule: lincar decay
kl cocfficient: 0.05
train epochs:
REBEL
batch size: 512
num epochs: yj: 1.0 kl cocfficient: 0.05
lcarning rate: 36-6
schedule: lincar decay
train epochs:
LoRA Adapter
1024
dropout: 0.0
Config
a: 2048
bias: False
Generation
sampling: true top k: 0.0
min length: 53
max new tokens: 53
top p: 1.0
temperature: 0.1
30

---

H.1.6
Winrate Details
We are using gpt-4-0613 checkpoint for winrate computations. Below we show the prompt for winrate evaluation and an example evaluation from GPT4.
Prompt for Winrate
Which of the following summaries does better joh of summarizing the most important points in the given forum post, without including unimportant irrelevant details? Judge based on accuracy, coverage, and coherence.
### Post: ((post))
#H# Summary A: (summarya]]
### Summary B: (ssummaryb)]
### Instructions:
FIRST provide one-sentence comparison two summaries, explaining which you prefer why. SECOND), on a new line, state only "A" or "B" to indicate your choice. Your response should use format: Comparison: cone-sentence comparison and explanation Preferred: <"A" or "B">
Example Evaluation from GPT4
Prompt
SUBREDDIT: r/AskReddit
TITLE: How you get someone out your head?
POST: Hi,
I'm and have been with my girlfriend for 5 years now. We recently moved together. We've always loved each other intensely.
Problem, recently started have feelings for an other person (a friend). This person has a boyfriend now 3 years, and has absolutely no ideas. Those feelings were so strong, it was hard to hide them. After 2 months of me heing distant and really sad, my girlfriend forced me to say what was bothering me. I'm not a good liar, and now she knows.
We decided to give us week alone, I went to my parents.
Now, I'm completely lost. would like for those feelings to go away, to leave me alone. But can't.
keep on thinking about this person, and hate that.
What do do? It's been 3 months now, and I'm just desperate.
TL.DR:
Reference (Summary A)
long relationship: fell love with an other person: admitted would like to disappear, though it doesn't.
REBEL
I recently started to have feelings for other person (a friend). We decided to give week alone, keep thinking about that person, and hate it. What do I do?
Generation
(Summary B)
Evaluation from GPT4
Comparison:
Summary A is tou brief and rather disjointed, while Summary B more accurately conveys the emotional conflict portrayed in the forum post in a coherent manner. Preferred: B
31

---

H.2
General Chat
H.2.1
Dataset Details
We present dataset details in Table 5.
Table 5: Dataset details for General Chat
Dataset
Size
Prompt Length
Generation Length
Nectar
183k
1024
1024
UltraFeedback
64k
1024
1024
H.2.2 Model Details
For OpenChat-3.5", we only train the last four layers and keep other layers frozen. For Meta-Llama- 3-8B-Instruce perform full-parameter training. For Starling-RMy -0----M------- RM-7B-alpha" 1 and ArmoRM- Llama3-8B-v0l we directly use the reward scores without any normalizations. We filter out prompts that are longer than 1, 024 tokens (2.3%) to fit the input length. OpenChat-3.5 is trained for four days, and
is train for one day on 8 H100 GPUs.
H.2.3 Reward Details
To ensure that Ty remains close to Top apply additional KL penalty to the reward:
r(x,y) RM (x, y) (In no, (ulæ²) - In To, (u(z))
(23)
where RM (zt y) is score from the reward model given prompt æ and response Furthermore, ensure that the generations terminate within the maximum generation length, we penalize any generation that exceeds this length by setting r(x, y) to a small fixed constant, r.
For the general chat experiments, we set I -4
H.2.4 Hyperparameter Details
Parameter setting for General Chat
Setting
Parameters
Base model: OpenChat-3.5
batch size: 32
Reward Model: Beatling-MM7B--appp
learning rate: 1e-7
Dataset: Nectar
schedule: linear decay
train epochs:
num epochs:
fj: 1.0
y: 0.05
-4
Base model: Meta-Llama----3 Reward Model: ArmoRM
mini-batch size: 128
learning rate: 3e-7
Datasct: UltraFeedback
schedule: cosine decay
warm ratio: 0.
train epochs:
iteration: 3
le6 (iter 1), le4 (iter 2), le2 (iter 3) ?: 0
"HuggingFace Dataset Card: berkeley nest/eerr
FagggingFace Datasct Card: Dpenbmb/UlraFeechacck
"HuggingFace Model Card: openchar/openchat_.55
12 FuggingFace Model Card: Bott:llamm//MccaeLaaa3-------- Instrucc
13 Hugging Model Card: berkeloynnes/Sarringg----------
"HuggingFace Model Card: RLHFFowAAmoRM-Jaama -..
32

---

H.3
Image Generation
H.3.1
Dataset Details
Generation prompts: cat, dog, horse, monkey, rabbit, zebra, spider, bird, sheep, deer, cow, goat, lion, tiger, bear, raccoon, fox, wolf, lizard, beetle, ant, butterfly, fish, shark, whale, dolphin, squirell mouse, rat, snake, turtle, frog, chicken, duck, goose, bee, pig, turkey, fly, llama, camel, bat, gorilla, hedgehog, kangaroo.
H.3.2 Model Details
We use the latent consistency model (Luo al., 2023) distillation of the Dreamshaper v7 model for our experiments. Experiments are conducted on 4 A6000 GPUs with each run requiring 10 hours.
H.3.3
Hyperparameter Details
Parameter setting for Consistency Models
Setting
Parameters
PPO
advantage clip maximum: 10
batches per epoch: 10
clip range: lc-4
learning rate: le-4
gradient accumulation steps: 8
max gradient norm: 5
number of cpochs: 100
horizon:
number of sample inner epochs: 2 sample batch size (per GPU);
rolling statisties buffer size: 32
rolling statisties min count: 16
train batch size (per GPU):
LoRA rank: 8
Lora
REBEL
advantage clip maximum: 10 batches per epoch:
learning rate: 3e-4
fy:
gradient accumulation steps: max gradient norm: 5
number of epochs: 100
horizon:
number of sample inner epochs: 1
sample batch size (per GPU): 8
rolling statisties buffer size: 32
rolling statisties min count: 16
train batch size (per GPU):
LoRA rank: 8
Lora
H'ugingffcee model card: SimianLuo/LCM SimianLuoLLCCMDeeanhaaaaa Dreamshaper
33

---

Example Generations
L.1 Summarization Generations from 2.8B model
Prompt
SUBREDDIT: r/relationhhip
TITLE: Me [18 M] with my college friend [19 F] about month, I wanna hang out with her but I don't 't have car.
POST: This is the link for earlier thread I posted about her
She basically doesn't 't have any friends and will spend all her spring break at home playing video games. wanna hang out with her, but both of us don't have car. live with my grandma 10 minutes away from her house. Would it be weird to ask her to spend a day with playing video games and eating pizza?
Another thing could is her to go see movie with me, but I would just meet there since I'll going with a bike. Do you guys think it's okay to do that? also have experience with girls. You can say this is the first girl in my life.
TL:DR:
Generation
My college friend doesn't 't have any friends and spends all her spring break home playing video games. wamna hang out with her but neither of have Do you think it's okay to her hang out?
Prompt
SUBREDDIT: r/AskReddit
TITLE: Reddit, what kind of effect do you think that crime shows/movies might have in reducing the occurrence of thought-out, "intelligent" crime?
POST: had this thought few days ago when watching episode of NCIS and they did the usual "create super high resolution image from a very low resolution image." I got to wondering how much of is real and how much of is fake (or, conversely, how much might under--xagggeated) Based on these types of shows/movics (bank robbery, conning, etc.) where the bad guy nearly 100% of the time gets caught, only an idiot would think it would be good idea to do any of this stuff, and from my [limited] knowledge of the general criminal, I' 'm under the impression that they mainly uneducated people in dire situations (or esychopaths/mentall the heist type of crime not assault or murder) are driven extreme lengths make the "asyc""quickk" dollar. So I wondered, is possible that these shows/movies that show the criminal, no matter how smart and creative, almost always getting caught could actually be (subconsciously or consciously) reducing the amount of more well thought through crime throughout the world?
unstable.. but I' thinking mostly with regards
TL:DR:
Generation
had this thought recently watching NCIS episode where criminals almost always get caught, is it possible that these crime shows/movies that show the criminal almost always getting caught could actually be (subconsciously or consciously) reducing the crime?
34

---

1.2
Chat Generations from REBEL-LIama-3
Prompt
Hi, I'd like to play ice hockey. Can you explain how the game works?
Generation
Hockey enthusiast! I'd be happy to explain the hasics of ice hockey. Here's comprehen- sive overview to get you started:
Objective:
The objective of ice hockey is to score more goals than the opposing team by using your stick to hit puck (a rubber disc) into the opponent's goal.
Gameplay:
Game Structure: A standard hockey game consists three periods, each lasting 20 minutes.
2. Teams: Each team consists of six players on the icc at time:
Forwards (3): Players who score goals and create scoring opportunities.
Defensemen (2); Players who defend their goal and prevent the opposing team
from scoring.
Goalie (1): The player responsible for preventing the puck from entering the goal.
Gameplay Flow: The game starts with face-off, where two players from opposing teams try to gain control the puck after it's dropped onto the ice by the referee. Players can move the puck using their sticks, skates, or by carrying
A goal is scored when a player hits the puck into the opponent's 's goal using their stick, skate, or body.
Scoring:
5. Penalties: If player breaks the rules, they can be sent the penalty box for set amount of time (2-5 minutes). During this time, their must play shorthanded (with one fewer player on the ice).
Power Play: If opposing team scores during penalty, the penalized team gets a power play, where they have a man advantage on the ice.
7. Icing: If a player shoots the puck from behind their own blue line and it crosses the opponent's goal line without being touched, it's considered icing, and the opposing team is awarded face-off in their own end zone.
Offside: player cannot enter the opponent's zone before the puck, they'll Il be called offside.
Common Penalties:
Hooking: Using the stick hook opponent's leg or body.
Holding: Grabbing onto another player's jersey or equipment.
Tripping: Using a stick, leg, or body to trip an opponent.
Interference: Interfering with opponent who doesn't 'I have possession of the puck.
Equipment:
Stick: A long, flat stick used to pass, shoot, and control the puck.
Skates: Specialized ice skates designed for speed, agility, and stopping.
Pads: Protective gear worn by players, including shoulder pads, elbow pads, gloves, shin guards, and hockey pants.
Helmet: A protective helmet worn by all players to prevent head injuries.
Goalie Gear: Specialized equipment worn by goalies, including large pad covering their body, gloves, and mask.
Basic Strategies:
Forechecking: Applying pressure to the opponent's defensemen to gain posses- sion of the puck.
Backchecking: Pressuring the opponent's forwards prevent them from scoring.
Breakouts: Quickly moving the puck from your own end zone to the neutral zone to create scoring opportunitics.
This is solid foundation to get you started. As you play more, you'll 'lll learn about the game's nuances strategies. Have fun playing ice hockey!
35

---

Prompt
Why do people give Reddit Gold to admins?
Generation
On Reddit, Reddit Gold (also known Goldj virtual currency that allows show appreciation for content creators, moderators, other users who contribute positively the community. When someone gives Reddit Gold to an admin (or moderator), it's often a way to express gratitude for their efforts in maintaining the community's quality, resolving issues, and keeping the discussion civil.
Here Appreciation moderation: Admins work tirelessly ensure that discussions remain on-topic, respectful, and follow community guidelines. Giving Gold acknowledges their dedication and hard work in maintaining a positive eavironment.
reasons why people might give Reddit Gold admins:
2. Recognition of expertise: Admins often possess in-depth knowledge about specific topics or communities, Giving Gold shows appreciation for their expertise and willingness to share it with others.
Support for community building: Admins play crucial role in fostering engagement, encouraging participation, creating opportunities for users to connect with each other. Gold gifts demonstrate support for their efforts in building a strong community.
Rewarding good behavior: When admins demonstrate excellent behavior, such as resolving confficts fairly providing helpful guidance, users might give Gold as way recognize and reward their good hehavior.
5. Boosting morale: Giving Gold can be morale booster for admins, acknowledging their value the community and motivating them to continue their excellent work.
It's worth noting that Reddit Gold is not required or expected, and admins don't receive any direct benefits from receiving Gold. However, the gesture of appreciation can go long way in fostering positive and supportive community.
36

---

I.3 Image Generations
Example Generations of REBEL
37

---

38

---

J Ablation Analysis
Winrate (1)
RM Score (1)
[L(<||)Tee) (1)
0.3
55.5%
1.37
10.4
0.7
59.9%
1.60
14.2
1.0
70.2%
2.44
29.0
2.0
62.5%
1.76
16.9
Table 6: REBEL ablation of the key hyperparameter on summarization task and 2.8B model. The best-performing v) for each metric is highlighted in bold.
Just like DPO, tuning REBEL is much more straightforward than PPO since the only hyperparameter REBEL introduced is y/ We investigate how sensitive REBEL to learning rate in the loss. The results of ablation summarization task and 2.8B model is shown in Table 6 with the same setting detailed in Appendix H.1.5 except for 1j. REBEL achieves the best performance when = 1, while increasing or decreasing 7) leads decreased performance. Our result here indicates that 7) is an important hyperparameter that requires tuning for achieving good performance. Setting to 1.0 is good starting point since, for all of our experiments from language modeling image generation, 1 achieves the best results.
K
Trade-off between Reward Model Score and KL-divergence
35
REBEL
PPO
30
40
KL (rz]| (rerl
00
25
20-
1.6
1.8
2.0
2.2
2.4
2.6 2.0
2.1
2.2
2.3
2.4
2.5
2.6
27
Score (+)
RM Score
(+)
Figure 5: Plot of Reward vs KI--Divergence for 2.8B REBEL and PPO for summarization. We evaluate the models across the entire test set every 100 steps for 2,000 steps. Left: each point represents the average reward score and KL-divergence for specific time step; the eclipse represents the confidence interval with standard deviations. Right: we divide the KL distribution at the 2,000-step into 10 bins with equal size and average the corresponding RM scores in each bin.
The trade-off between the reward model score and KL-divergence is shown in Figure 5. We evaluate the 2.8B REBEL and PPO every 400 gradient updates during training for 8,000 updates on summa- rization. The sample complexity of each update is held constant across both algorithms for fair comparison. For the left plot, each point represents the average divergence and score over the entire test set, and the eclipse represents the confidence interval with 2 standard deviations. As observed previously. PPO exhibits lower divergence, whereas REBEL shows higher divergence but capable of achieving larger RM scores. Notably, towards the end of the training (going to the right part of the left plot), REBEL and PPO have similar KL and RM scores. For the right plot in Figure 5, we analyze single checkpoint for each algorithm at the end of training. For each algorithm, we group every generation from the test set by its KI. distribution into 10 equally sized bins and calculate the average of the corresponding RM score for each bin. We can see that REBEL achieves higher RMM scores for generations with small divergence while requiring larger divergence for generations with the highest scores.
39

---

L Regression Loss During Training
MSE During Training: 6.9B Pythia Policy
14-
12
Mean Squared Error (MSE)
10
8-
6
2
250
500
750
1000
1250
1500
1750
2000
Iterations REBEL
Figure 6: REBEL's reward difference prediction error throughout training of our 6.9B parameter policy on the summarization task. The reward used for this task is unbounded with the range of values of the human labels in the validation set being 1-.81, 7.31]. We plot both the smoothed values with moving average and the loss vales at each iteration.
Figure shows the observed loss of Eq. that we observed when finetuning the 6.9B Pythia model summarization. We see that REBEL minimizes the loss throughout training maintaining relatively low mean squared error given that our observed rewards were mostly between [-10, 10j. Note that learned reward model, however, unbounded.
M Breakdown of MT-BBench
model
Staring-U-- Uk-78-aiph
Writing
OpanChut-3.5
Humentties
Reasoning
Coding
Figure 7: Breakdown of MT-Bench results eight dimensions.
Figure shows the breakdown MT-Bench results. REBEL SEBELL-OpenChat-33 outperforms both APA (SartinggLMM-7B-alphha and base (OpenChat-3.55) models on six out of eight dimensions including writing, roleplay, math, extraction, STEM, and humanities.
40

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction (Section 1) accurately present the primary claims of the paper, aligning well with the detailed contributions and scope described in the main body.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of exact Bayes Optimal solution assumption in Section 3.2 is discussed in Section 4.
Justification and limitation of Assumption are discussed in Appendix E.
The regret bound of REBEL is discussed in Section 4. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
41

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All of the assumptions and claims are clearly stated and cross-referenced.
Proof of Claim is provided Appendix B.
Proof of Claim 2 is provided in Appendix C.
Justification of Assumption is provided Appendix E.
Proof of Theorem is provided in Appendix F.
Proof of Theorem 2 is provided in Appendix G.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detail the datasets, models, hyperparameters, and evaluation metrics in Section and Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
42

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide an anonymized version of data and code as supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
aublic/gudess wwt.o/eeuuisssiooooooo...oeooeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurlPS code and data submission guidelines (https: httpl/ic/ggiieseee guw Cooeeuumissiooooooooooo
//nips "nips.cc/pull wwtpccccppppcccctccccccccccccccc
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail the datasets (splits), model pipelines, and complete sets of hyperpa- rameters in Section and Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
43

---

Justification: Our results for summarization and image generation are averages of three random seeds. Specifically we report the standard deviation across seeds for summarization and IQM with 95% confidence intervals for our image generation results. For general chat experiments, we train once to obtain the final result. This consistent with how the community reports general chatbot evaluations on AlpacaEval 2.0 and Chatbot Arena / MTBench. We follow other works in reporting our checkpoint's results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Resources for experiments are discussed Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: h/tp::////euuiee http//nurip...e/ cc cc/public//Ethicicccceeeeeee httpbllc/Ethisccudeel......
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
44

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper introduces an algorithm that could potentially be applied to any reinforcement learning task.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
45

---

Datasets and models we used in experiments are cited in Section 5 and the model/dataset cards are detailed in Appendix H.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code used in the paper is well documented with instructions.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
46

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
47


Reviews:
[{"review_id": "g1x1J8JQyO", "reviewer": "Reviewer_dEG5", "summary": "The authors present REBEL, a method for solving contextual bandit problems (such as the alignment of language models) via regressing relative rewards. They first derive their objective by demonstrating that the use of paired responses means that you can get rid of the partition function, which is impossible to estimate. \n\nThey then connect their method to previous methods in RL including detailing, but not . They demonstrate that under strong assumptions REBEL is equivalent to mirror descent, and that under assumptions of coverage by the reference policy, that REBEL produces returns close to an optimal policy. \n\nFinally the authors run experiments on summarisation, general chat and image alignment, demonstrating their method compares favourably to other methods.", "questions": "* Do the authors have any idea why REBEL seems to have a slightly higher KL than the other methods?\n* Although in image alignment REBEL seems to do similarly to PPO, it also has higher variance. Do you know why that might be?\n* Are the results for the 6.8B model significant? It seems as though REBEL produces very similar performance to e.g. PPO. For the smaller models the separation seems larger, is there a reason why the separation in performance between REBEL and other methods is bigger for smaller models?\n* What are the error bars in Table 1? Is that standard deviation?", "limitations": "The authors discuss the limitations throughout their work at relevant stages.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of using relative rewards to remove the partition function is a nice and simple idea\n* The theoretical connections of their method to prior methods grounds their work nicely in existing RL approaches. \n* The empirical results seem to demonstrate their method is competitive or better than other approaches. \n* REBEL compares favourably in terms of runtime and memory usage with other, similarly performing methods. \n\nOverall the theoretical and empirical examinations of their method seems very thorough.", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging review and comments. We respond to your individual questions below.\n\n> Do the authors have any idea why REBEL seems to have a slightly higher KL than the other methods?\n\nThe KL divergence is generally close across methods. For the TL;DR experiments, following previous work [1], we apply an addition KL penalty to the reward: $r(x, y) = RM(x, y) - \\gamma (\\ln \\pi_{\\theta_{t}} (y|x) - \\ln \\pi_{\\theta_{0}} (y|x))$. PPO incorporates a clipping mechanism on top of this regularization, which allows it to control the KL-divergence more strictly. This difference in approach can lead to REBEL exhibiting a slightly higher KL-divergence compared to PPO.\n\n> Although in image alignment REBEL seems to do similarly to PPO, it also has higher variance. Do you know why that might be?\n\nTo fairly compare REBEL and PPO, we tuned all hyperparameters, including parameters related to reward queries for PPO. Afterward, to ensure that REBEL was compared fairly to PPO, we kept those hyperparameters related to reward queries constant so that REBEL and PPO had the same number of reward queries per update. We believe that if we were to modify the reward queries per update, REBEL would show a lower variance than our current runs.\n\n> Are the results for the 6.8B model significant? It seems as though REBEL produces very similar performance to e.g. PPO. For the smaller models the separation seems larger, is there a reason why the separation in performance between REBEL and other methods is bigger for smaller models?\n\nThe discrepancy in our training setup could be the reason for the smaller separation in the larger model versus the smaller models. Specifically, we performed hyperparameter tuning using LoRA [3] on the 1.4B and 2.8B models. We then directly applied these hyperparameters to the larger 6.9B model for full-parameter training without additional tuning (detailed in Appendix H.1.2). The results for the baselines that we compared to (REINFORCE, PPO, RLOO) for the 6.9B models are obtained from [1,2], which might have directly tuned the 6.9B model. These differences contribute to the decreasing performance gains as model size increases.\n\n> What are the error bars in Table 1? Is that standard deviation?\n\nThe error bars are standard deviations. Results are averaged over three seeds and the standard deviations across seeds are in parentheses.\n\n[1] Huang S, Noukhovitch M, Hosseini A, Rasul K, Wang W, Tunstall L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL; DR Summarization.\n\n[2] Ahmadian A, Cremer C, Gallé M, Fadaee M, Kreutzer J, Üstün A, Hooker S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.\n\n[3] Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. Lora: Low-rank adaptation of large language models. arXiv", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Em20XXbwf", "reviewer": "Reviewer_HFsB", "summary": "This work presents REBEL, a minimalist reinforcement learning algorithm that does policy optimization by solving a sequence of regression problems using relative rewards as targets. Theoretical analysis shows that Natural Policy Gradient (NPG) is a variant of REBEL, and thus theoretical guarantees for NPG can be applied to REBEL.  Experimental results  show that REBEL matches or outperforms existing baselines, most notably PPO and RLOO, on multiple tasks.", "questions": "- The reward model becomes increasingly off-distribution as the policy is updated. Although it is standard practice to keep reward models fixed even with iterative methods, prior works generally use it to generate preference labels between pairs of outputs. Since this work uses the difference of scores as the regression target, the off-distribution reward scores might have a greater impact here. Concisely, how significant a problem is reward model over-optimization [1] for REBEL?\n-  It would be interesting to see and understand the differences between reward-weighted regression baseline (RWR) and REBEL as they have some close connections. \n-  Is there an optimal choice of $\\mu$ ? What are the intuitive differences between using the  $\\mu= \\pi_{ref}$  and $\\mu= \\pi_{t}$ ? As the policy improves, samples $y,y’ \\sim \\pi_{t}$ are in the high reward region, and it can be difficult to separate them since these might be off-distribution for the reward model. Given these constraints of the reward model, there might be better choices of $\\mu$ that allow for better prediction of score differences. It would be interesting to see an ablation study on this, or a well-reasoned answer that explains the tradeoffs between different choices of $\\mu$. \n- Why are datasets not aggregated? Instead, only the most recently collected dataset is used for training. \n\n[1] : Gao, L., Schulman, J., & Hilton, J. (2023, July). Scaling laws for reward model overoptimization. In International Conference on Machine Learning (pp. 10835-10866). PMLR.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-organized and technically sound. The general flow of the paper is smooth and proposed methods are explained adequately. The paper has an appropriate number of citations and properly details existing work in the related work section. \n- The method is simple to implement and has little engineering overhead. Given the minimalist implementation, the results are impressive, surpassing even PPO, which typically requires significant engineering.", "weaknesses": "- There are no significant weaknesses in this work, barring some clarifying details. \n- I believe that at least a brief section on related work should be included in the main paper, the in-depth one can be deferred to the appendix. In terms of space, I personally do not think Section 2.2 adds much value to the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review of our paper. We respond to your individual questions below.\n\n> I believe that at least a brief section on related work should be included in the main paper, the in-depth one can be deferred to the appendix. In terms of space, I personally do not think Section 2.2 adds much value to the main paper.\n\nThank you for your suggestion. We will include a brief section on related work in the main paper and have a detailed discussion in the appendix.\n\n> How significant a problem is reward model over-optimization [1] for REBEL?\n\nIn the context of RLHF for LLMs, we choose reward models that are highly ranked on the Reward Bench [2]. These reward models are trained using extensive datasets that include responses generated from a variety of different policies [3]. This diversity makes it challenging to over-optimize a reward model, even for methods such as REBEL. In addition, following previous work [4], we apply an addition KL penalty to the reward, $r(x, y) = RM(x, y) - \\gamma (\\ln \\pi_{\\theta_{t}}(y|x) - \\ln \\pi_{\\theta_{0}} (y|x))$, to prevent over-optimization for both REBEL and the baseline methods.\n\nIn the image generation setting, following previous work [5, 6], no KL regularization is used. We observe that both PPO and REBEL over-optimize the reward model towards the end of the training. They tend to ignore the input prompt and generate the same image that maximizes the reward model score. For fair comparison to [6], we also did not include KL.\n\n> It would be interesting to see and understand the differences between reward-weighted regression baseline (RWR) and REBEL as they have some close connections.\n\nRWR can be understood as reward-weighted imitation learning. Prior work on RL for diffusion models shows that RWR is not as effective as RL (PPO in that case) [5].  REBEL, on the other hand, is a full RL algorithm from its connection to NPG where it generalizes NPG.\n\n> Is there an optimal choice of $\\mu$? What are the intuitive differences between using the $\\mu=\\pi_{ref}$ and $\\mu=\\pi_{t}$?.\n\nSetting $\\mu$ to $\\pi_{ref}$ or another distribution can encourage exploration, especially when the policy is still learning and the quality of the generations is not yet optimal. This approach can help in discovering diverse and potentially higher-reward samples that the current policy might not generate.\n\nIn our experiments, we found that setting $\\mu=\\pi_t$ yielded better results. We attribute this to the lower quality of the offline dataset. In TL;DR summarization and Ultrafeedback, our trained policies can actually generate better responses than the ones in the datasets. This is shown in Table 1, where the 2.8B and 6.9B models can easily reach high winrates compared to offline human demonstrations. In this case, setting $\\mu$ to $\\pi_{ref}$ or the offline data does not provide significant benefits. Therefore, we use $\\pi_t$ as $\\mu$ in all our experiments.\n\n> Why are datasets not aggregated? Instead, only the most recently collected dataset is used for training.\n\nOur approach can be understood as online RL, e.g. PPO, where only on-policy data is used for update. We could aggregate the datasets which makes the approach more off-policy. While we have not tested this in our experiments, it could be an interesting direction to explore if being off-policy would lead to better sample efficiency.\n\nIn addition, previous RLHF works [4, 8, 9] also use batches collected by the current policy (i.e. online batch), with each batch containing a new set of prompts. To ensure a fair comparison with previous methods, we primarily focus on the on-policy approach.\n\n[1] Gao, L., Schulman, J., & Hilton, J. (2023, July). Scaling laws for reward model overoptimization. In International Conference on Machine Learning (pp. 10835-10866). PMLR.\n\n[2] Lambert N, Pyatkin V, Morrison J, Miranda LJ, Lin BY, Chandu K, Dziri N, Kumar S, Zick T, Choi Y, Smith NA. Rewardbench: Evaluating reward models for language modeling. arXiv\n\n[3] Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., Zhang, T. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint. 2024. arXiv\n\n[4] Huang S, Noukhovitch M, Hosseini A, Rasul K, Wang W, Tunstall L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL; DR Summarization.\n\n[5] Black K, Janner M, Du Y, Kostrikov I, Levine S. Training diffusion models with reinforcement learning. arXiv\n\n[6] Oertell O, Chang JD, Zhang Y, Brantley K, Sun W. Rl for consistency models: Faster reward guided text-to-image generation. arXiv\n\n[7[ Korbak T, Shi K, Chen A, Bhalerao RV, Buckley C, Phang J, Bowman SR, Perez E. Pretraining language models with human preferences. In International Conference on Machine Learning 2023 Jul 3 (pp. 17506-17533). PMLR\n\n[8] Stiennon N, Ouyang L, Wu J, Ziegler D, Lowe R, Voss C, Radford A, Amodei D, Christiano PF. Learning to summarize with human feedback. Advances in Neural Information Processing Systems. 2020;33:3008-21.\n\n[9] Ahmadian A, Cremer C, Gallé M, Fadaee M, Kreutzer J, Üstün A, Hooker S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2NT1MRXOr", "reviewer": "Reviewer_ihUH", "summary": "This paper proposes the REBEL algorithm that reduces policy optimization to iteratively solving squared loss regression problems on the difference in rewards between trajectories, based on DPO's analysis. The paper transforms the resulting equation for r(x, y) presented in DPO to a regression loss function, and avoids the intractable calculation of Z(x) by calculating the loss based on a pair of samples from the same input prompt x, i.e., (x, y) and (x, y'). One of the goals for REBEL is to serve as a simple and lightweight RL algorithm that eliminates the need for complex components like value functions and clipping heuristics used in PPO. The authors provide a theoretical analysis showing that Natural Policy Gradient can be seen as a special case of REBEL under some assumptions. The authors conduct two kinds of empirical analysis including language modeling and image generation tasks to demonstrate the performance of REBEL.", "questions": "1. Sample efficiency and convergence guarantees:\n- The authors claim that REBEL matches the strongest known theoretical guarantees in terms of convergence and sample complexity. However, the experiments only compare performance at a specific epoch without demonstrating improved sample efficiency. Can the authors provide experimental results that support their claim of improved sample efficiency compared to other algorithms?\n- It would be helpful to see convergence plots that show the performance of REBEL and baseline algorithms over the course of training, rather than just at a selected epoch. This would provide a clearer picture of the sample efficiency and convergence properties of REBEL.\n2. Relationship between regressor performance and dataset quality:\n- The authors state that a regressor that can predict the difference in rewards between trajectories implicitly captures an improved policy. Is the performance of this regressor dependent on the quality of the dataset used for training? How does the quality of the dataset affect the regressor's ability to capture an improved policy?\n- Can the authors provide insights or experiments that explore the relationship between dataset quality and the effectiveness of REBEL?\n3. Applicability to deterministic MDPs:\n- The authors mention that REBEL can be applied to any deterministic MDP where the initial state is x and the trajectory y consists of a sequence of actions. Is there any experimental or theoretical support for this claim?\n- It would strengthen the paper if the authors could provide empirical results or theoretical analysis that demonstrates the effectiveness of REBEL in deterministic MDPs beyond the bandit formulation.\n4. Choice of base distribution \\mu:\n- The authors state that the choice of base distribution \\mu determines whether REBEL is hybrid or fully online. Can they provide experimental results that compare different forms of \\mu across various types of tasks? What are the practical guidelines for choosing \\mu in real-world applications?\n- Insights into the impact of different choices of \\mu on the performance and behavior of REBEL would be valuable for practitioners looking to apply this algorithm.\n5. Stochastic MDPs and the need for critic-based variance reduction:\n- The authors leave the experimental validation of REBEL in stochastic MDPs for future work but mention that trajectory-level rewards can be high-variance, potentially requiring critic-based variance reduction. In what practical situations would the transition dynamics be stochastic? If critic-based variance reduction is needed, how does this align with the introductory section's criticism of PPO's complexity?\n- The lack of experimental support for REBEL's performance in stochastic MDPs is a significant limitation. Can the authors provide any preliminary results or theoretical insights that support their claims about REBEL's applicability to stochastic environments?\n6. Performance comparison with baselines:\n- In the experiments conducted by the authors, DPO performs significantly worse than PPO, especially in Table 1, where DPO is inferior in every case. Can the authors provide an explanation for this discrepancy? Is it due to differences in implementation or hyperparameter settings?\n- In Figure 3, the comparison between PPO and REBEL is made at an intermediate checkpoint where REBEL observes a higher reward under the reward model. Is it possible that PPO has already overfit at this selected epoch? How was this specific epoch number chosen for REBEL? What would the comparison look like if the best-performing epoch for each algorithm were considered? Additionally, why is the comparison limited to only PPO? It would be informative to include other state-of-the-art RL algorithms in the comparison to better understand the relative performance of REBEL.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality:\n    - This paper presents a new angle by transforming the analysis of the reward function presented in the DPO paper into a reward regression loss, leading to the proposed REBEL algorithm. \n    - The authors make connections between REBEL and existing RL methods like NPG considering some assumptions, showing that these algorithms can be seen as special cases or approximations of REBEL under certain conditions. \n\n- Quality:\n    - The paper provides a thorough theoretical analysis comparing REBEL with existing RL approaches. \n\n- Clarity: \n    - The paper is well-written and easy to understand, with a clear logical flow from motivation to theoretical analysis to empirical validation. The authors do an good job of explaining the intuition behind REBEL and highlighting its connections to prior work.\n\n- Significance:\n    - The paper tackles the important problem of developing simpler and more efficient RL algorithms that can scale to large-scale generative model fine-tuning.", "weaknesses": "1. Insufficient experimental validation and limited baseline comparisons:\n- While the paper presents empirical results on language modeling and image generation tasks, the experimental validation of REBEL could be more comprehensive. The authors should consider including a wider range of benchmarks and datasets to demonstrate the generality and robustness of their approach.\n- The comparison with baseline algorithms like PPO and DPO is somewhat limited. The authors should provide more details on the hyperparameter settings and training procedures for the baselines to ensure a fair comparison. Moreover, the poor performance of DPO compared to PPO in the experiments raises questions about the implementation or hyperparameter choices.\n- The authors claim that REBEL matches the strongest known theoretical guarantees in terms of convergence and sample complexity. However, the experiments only compare performance at a specific epoch without demonstrating improved sample efficiency. Convergence plots showing the performance of REBEL and baselines over the course of training would provide a clearer picture of the sample efficiency and convergence properties.\n\n2. Lack of support for certain claims and limited exploration of key aspects:\n- The paper makes several claims regarding the advantages of REBEL, such as its ability to handle intransitive preferences, incorporate offline datasets, and apply to deterministic MDPs. However, there is a lack of corresponding experimental evidence or theoretical analysis to substantiate these claims.\n- The relationship between the regressor's performance and the quality of the dataset used for training is not explored in depth. Insights or experiments that investigate how dataset quality and diversity affect the regressor's ability to capture an improved policy would strengthen the paper.\n- The choice of base distribution \\mu is mentioned as a determining factor for whether REBEL is hybrid or fully online. However, the paper does not provide experimental results comparing different forms of \\mu across various tasks or practical guidelines for choosing \\mu in real-world applications.\n\n3. Inconsistencies and potential conflicts with previous statements:\n- The authors mention that critic-based variance reduction might be necessary for high-variance trajectory-level rewards in stochastic MDPs, which seems to contradict the criticism of PPO's complexity in the introductory section. The lack of experimental support for REBEL's performance in stochastic MDPs is a significant limitation, and the authors should provide preliminary results or theoretical insights to support their claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We address each of your points below.\n\n> Insufficient experimental validation and limited baseline comparisons\n\n> Performance comparison with baselines\n\nOur experimental section is comprehensive compared to previous works on RLHF [1, 2], incorporating a general chat dataset and an image generation task to evaluate the robustness and generality of REBEL. We compared REBEL with several algorithms, including DPO, PPO, iterative DPO, REINFORCE, RLOO, and APA. Detailed hyperparameter settings are provided in Appendix H.1.4, H.2.3, H.3.3.\n\nThe lower performance of DPO compared to PPO is not due to implementation or hyperparameter choices. On TL;DR, our DPO results are better than the ones reported in [3]. Results for PPO, REINFORCE, and RLOO are directly obtained from prior papers [2, 4] which are exclusively focusing on these algorithms on TL;DR. For general chat, we directly report winrates from the released starling-alpha model by APA's authors. Thus, we believe that our comparison to baselines is fair.\n\nIn Figure 3, the comparison at an intermediate checkpoint is intended to highlight the sample efficiency of REBEL. There is no indication that PPO has overfitted at this epoch, as the image quality of PPO continues to improve afterward.\n\n> Sample efficiency and convergence guarantees\n\nFor the image generation task, as illustrated in Figure 4, REBEL converges faster during the initial training phase and eventually achieves performance comparable to that of PPO. In addition, we plot the reward vs. step for TL;DR dataset in the pdf of the global rebuttal. The plot demonstrates REBEL's faster convergence compared to iterative DPO and PPO.\n\n> Lack of support for certain claims and limited exploration of key aspects\n\n> Applicability to deterministic MDPs\n\n> Choice of base distribution \\mu\n\nSince our focus is not on the preference model itself, we did not conduct experiments specifically targeting preference models. However, we provided a method to extend REBEL to preference models in Appendix D, with further analysis provided in Appendix G.\n\nIn our experiments, we found that setting $\\mu=\\pi_t$ yielded better results. We attribute this to the lower quality of the offline dataset. In TL;DR summarization and Ultrafeedback, our trained policies can generate better responses than the ones in the datasets. This is shown in Table 1, where the 2.8B and 6.9B models can easily reach high winrates compared to offline human demonstrations. In this case, setting $\\mu$ to $\\pi_{ref}$ or the offline data does not help significantly. Therefore, we use $\\pi_t$ as $\\mu$ in all experiments.\n\nSince transitions are deterministic, we could formulate this as a bandit problem where $x$ is the initial state and $y$ is the trajectory consisting of a sequence of actions [1, 2, 10]. Applying token generations in LLMs and image generation to deterministic MDPs has been proposed in prior RLHF works [5, 6, 7, 8]. We adapt this bandit setup to make a fair comparison to the previous works.\n\n> Inconsistencies and potential conflicts with previous statements\n\n> Stochastic MDPs and the need for critic-based variance reduction\n\nLack of experimental support in stochastic MDPs is not a significant limitation as deterministic MDPs have wide applications in LLMs and image generations, which are high-impact areas. We emphasize that all prior RLHF works [1, 2, 4, 6] all focus on deterministic MDP.\n\nThere is also no conflict in our criticism of PPO's complexity. Our paper focuses on deterministic transitions; in this context, many prior work have argued that PPO and critic-based variance reduction methods are not necessary [2, 8, 9]. We agree that in highly stochastic settings, critics might still be necessary, which is indeed stated in the paper.\n\n> Relationship between regressor performance and dataset quality:\n\nA high-quality dataset, i.e. a dataset that has diverse coverage, would increase the generalization ability of the learned regressor. E.g., if the training distribution is diverse, then the learned regressor can achieve better performance under the comparator policy $\\pi^*$ distributions, which, as our theory indicates, will ensure convergence to $\\pi^*$. Our work focuses on online RL and studying the dataset quality is not the main focus of the paper.\n\n[1] Rafailov R, Sharma A, Mitchell E, Manning CD, Ermon S, Finn C. Direct preference optimization: Your language model is secretly a reward model.\n\n[2] Ahmadian A, Cremer C, Gallé M, Fadaee M, Kreutzer J, Üstün A, Hooker S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.\n\n[3] Rafailov R, Chittepu Y, Park R, Sikchi H, Hejna J, Knox B, Finn C, Niekum S. Scaling laws for reward model overoptimization in direct alignment algorithms.\n\n[4] Huang S, Noukhovitch M, Hosseini A, Rasul K, Wang W, Tunstall L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL; DR Summarization.\n\n[5] Ramamurthy R, Ammanabrolu P, Brantley K, Hessel J, Sifa R, Bauckhage C, Hajishirzi H, Choi Y. Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.\n\n[6] Stiennon N, Ouyang L, Wu J, Ziegler D, Lowe R, Voss C, Radford A, Amodei D, Christiano PF. Learning to summarize with human feedback.\n\n[7] Chang JD, Shan W, Oertell O, Brantley K, Misra D, Lee JD, Sun W. Dataset reset policy optimization for rlhf.\n\n[8] Black K, Janner M, Du Y, Kostrikov I, Levine S. Training diffusion models with reinforcement learning.\n\n[9] Oertell O, Chang JD, Zhang Y, Brantley K, Sun W. Rl for consistency models: Faster reward guided text-to-image generation.\n\n[10] Wu T, Zhu B, Zhang R, Wen Z, Ramchandran K, Jiao J. Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7UbyEuYlg", "reviewer": "Reviewer_zZ2j", "summary": "This paper reduces the complex policy optimization procedure of alignment to a simple regression objective, using the relation between optimal policy and reward. The paper conduct detailed theoretical analysis in revealing the relation between the proposed algorithm *REBEL* and *NPG/MD*. Comprehensive experiments in both text and image generation exhibit the effectiveness of *REBEL*.", "questions": "none", "limitations": "none", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies simplified version of policy optimization in RLHF (compared to PPO), which is a research topic of interest.\n2. The theoretical analysis of *REBEL* is detailed and insightful.\n3. The presentation of this paper is logically clear and has good readability.\n4. The experiments in this paper are comprehensive, and the experimental results are well presented.", "weaknesses": "1. The statement \"REBEL ... be extended to handle intransistive preferences ....\" in the abstract is not adequately presented in the main content of the paper. As the major influence brought by intransistive preferences is the degradation of reward score accuracy, which is not addressed by this paper.\n2. I would suggest the authors to summarize the limitations of the proposed method in a separate \"Limitations\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review of our paper.\n\n> The statement \"REBEL ... be extended to handle intransitive preferences ....\" in the abstract is not adequately presented in the main content of the paper. As the major influence brought by intransistive preferences is the degradation of reward score accuracy, which is not addressed by this paper.\n\nWe appreciate your observation and agree with your assessment. Since our focus is not on the preference model itself, we did not conduct experiments specifically targeting preference models. However, we provided a method to extend REBEL to preference models. The extension of REBEL to address intransitive preferences is discussed in detail in Appendix D, with further analysis provided in Appendix G. How to pre-train a good preference model without any degradation of reward score accuracy is beyond the scope of our paper.\n\n> I would suggest the authors to summarize the limitations of the proposed method in a separate \"Limitations\" section.\n\nThank you for the suggestion. We will include a dedicated \"Limitations\" section in the next version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and insightful comments, which have significantly improved our paper. We are pleased that the reviewers appreciated our algorithm's simplicity, the detailed theoretical connections to prior methods, and the thorough empirical results. We summarize the main suggestions below and address each reviewer's comments individually in each reviewer’s rebuttal.

* Reviewer dEG5 raises questions regarding the experimental details and empirical results. We address these concerns in our detailed response to the reviewer.

* Reviewer HFsB provides interesting suggestions for exploring data aggregation and different choices of $\mu$. We would certainly explore this direction in future investigations into the REBEL algorithm.

* Reviewer ihUH expresses concerns about the empirical evidence supporting our claims. In response to this, we conduct an additional convergence experiment. We plot the reward vs. step for TL;DR dataset in the attached pdf. The plot demonstrates REBEL's faster convergence and higher rewards compared to iterative DPO and PPO.

* Reviewer zZ2j suggests including a separate “Limitation” section. We will certainly include this in the next version of the paper.

Meta-review: The reviewers all agreed that this paper has clear presentation, strong theoretical contributions and good empirical evaluations. The simplicity of the approach can potentially make it very useful for practitioners, once tested at larger scales. This is a clear accept.
Decision: Accept (poster)