Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024)

Abstract: AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. PROXIMAL POLICY OPTIMIZATION (PPO) has been positioned by recent literature as the canonical method for the RL part of RLHF. However, it involves both high computational cost and sensitive hyperparameter tuning. We posit that most of the motivational principles that led to the development of PPO are less of a practical concern in RLHF and advocate for a less computationally expensive method that preserves and even increases performance. We revisit the formulation of alignment from human preferences in the context of RL. Keeping simplicity as a guiding principle, we show that many components of PPO are unnecessary in an RLHF context and that far simpler REINFORCE-style optimization variants outperform both PPO and newly proposed “RLfree” methods such as DPO and RAFT. Our work suggests that careful adaptation to LLMs alignment characteristics enables benefiting from online RL optimization at low cost.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267 August 11-16, 2024 ©2024 Association for Computational Linguistics
AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. PROXIMAL POLICY OPTIMIZATION (PPO) has been positioned by recent literature as the canonical method for the RL part of RLHF. However, it involves both high computational cost and sensitive hyperparameter tuning. We posit that most of the motivational principles that led to the development of PPO are less of a practical concern in RLHF and advocate for a less computationally expensive method that preserves and even increases performance. We revisit the formulation of alignment from human preferences in the context of RL. Keeping simplicity as a guiding principle, we show that many components of PPO are unnecessary in an RLHF context and that far simpler REINFORCE-style optimization variants outperform both PPO and newly proposed “RLfree” methods such as DPO and RAFT. Our work suggests that careful adaptation to LLMs alignment characteristics enables benefiting from online RL optimization at low cost.

heading: 1 Introduction
text: State-of-art Large Language Models (LLMs) are typically pre-trained on tremendous amounts of text (Brown et al., 2020; OpenAI et al., 2023; Anil et al., 2023; Touvron et al., 2023a,b; Üstün et al., 2024) spanning trillions of tokens. These training corpora often contain many complex preferences, relations, and intentions that may not all be desirable for an LLM to exhibit. A question of great interest to both the research and wider practitioner community is how to align these models to human preferences?
Despite being the focus of considerable research effort (Khalifa et al., 2021; Ouyang et al., 2022; Bai et al., 2022b; Lee et al., 2023; Tunstall et al.,
2023), there is a lack of consensus regarding the optimal approach to achieve this goal. Reinforcement Learning from Human Feedback (RLHF), one of the most widely regarded alignment approaches, directly borrows from traditional RL literature and uses techniques such as PROXIMAL POLICY OPTIMIZATION (PPO) to maximize the reward score produced by a reward model that is typically trained as a binary classifier on pairs of completions labeled by human annotators. While PPO has become a canonical approach cemented in popularity through its usage in the seminal literature on RLHF (Stiennon et al., 2020; Nakano et al., 2022; Bai et al., 2022b), getting PPO to work in practice is non-trivial for non-RL specialists and comes with known issues:
1. Computational Cost: PPO typically requires loading up to 4 models simultaneously: the generator, the reference (for KL estimation), the critic, and the reward model, where the training of the generative and critic models are interleaved (Schulman et al., 2017). This challenge is further exacerbated by the size of modern LLMs, ranging in the billions of parameters (OpenAI et al., 2023; Stiennon et al., 2020; Touvron et al., 2023a).
2. Optimization challenges: the unstable and sensitive nature of online RL optimization, and the relative algorithmic complexity of PPO requires niche expertise to tune it well (Engstrom et al., 2020).
Recent works propose “RL-free” methods such as DPO (Rafailov et al., 2023), IPO (Azar et al., 2023) or iterative fine-tuning approaches to LLM preference training (Yuan et al., 2023; Zhao et al., 2023; Dong et al., 2023). However, these works fail to question whether a simpler solution within an RL paradigm exists. Instead, all these approaches attempt to answer this question by stripping all
12248
RL components from RLHF and the difficulties that come with it (Rafailov et al., 2023; Zhao et al., 2023; Yuan et al., 2023; Liu et al., 2023; Dong et al., 2023; Azar et al., 2023; Choi et al., 2024). Iterative fine-tuning techniques rely solely on a powerful reward model to identify a subset of samples to train on, while DPO and IPO avoid both reinforcement learning and training a separate reward model by directly learning from human feedback.
In contrast to these approaches, we remain in the RL paradigm, but instead return to basics. The core question we seek to explore in this work is can we avoid the computational and optimization complexity of PPO while preserving performance?. We isolate several key differences between traditional Deep-RL settings which originally motivated PPO and typical human-preference learning settings for LLMs. We note that PPO, as an approach, emphasizes stability across iterations, aiming to train an effective policy with the premise of small, stable updates. PPO was designed for a regime where off-policy gradient updates are large enough to introduce instability. This regime dominates traditional Deep-RL benchmarks (Engstrom et al., 2020; Schulman et al., 2017). However, in this work, we posit that the setting of RLHF, which involves fine-tuning a pre-trained LLM, is lacking in these characteristics.
In contrast to traditional Deep-RL settings, the initialization of the policy, in the form of a pretrained and supervised fine-tuned (SFT) model, is far from a random parameterization. While the conceivable search space is enormous, due to the pretraining and SFT stages, only a far smaller subset of tokens is likely to be generated as the probability mass is concentrated on these few tokens. Thus, while traditional Deep-RL settings require strong regularization to reduce the high variance of the gradient estimators; we observe empirically this is less of a practical concern in RLHF and motivate a less computationally expensive method that preserves robustness (Wu et al., 2018; Kreutzer et al., 2021).
Furthermore, we revisit how learning from human preferences is formulated in the context of RL where generating each token is modeled as an action, and each partial sequence, starting with the prompt, is seen as a state. In practice, this modeling assumption for PPO method is often voided. We argue and show that the modeling of partial sequences is unnecessary in this setting where rewards are only attributed to full generations, with
no true rewards for any intermediary tokens in the generation. Thus, it is more appropriate and efficient to model the entire generation as a single action with the initial state determined by the prompt.
Given these observations, while keeping simplicity as a guiding principle, we explore the use of the REINFORCE estimator (Williams, 1992) and its multi-sample extension REINFORCE LEAVEONE-OUT (RLOO) (Kool et al., 2019) to optimize the sequence-level objective. We break apart PPO and show that the most basic policy gradient algorithm, Vanilla Policy Gradient REINFORCE consistently outperforms PPO. PPO is unnecessarily complicated for a pre-trained LLM environment. Unlike PPO, we can use REINFORCE to directly optimize the full trajectory (sequence) return coupled with unbiased baselines, whereas actor-critic algorithms (Sutton et al., 1999), such as PPO, bootstrap off intermediary state value-functions to reduce variance at the cost of introducing bias into the estimator.
We arrive at consistent results across models including Llama (Touvron et al., 2023a), Pythia (Biderman et al., 2023) and datasets such as the Anthropic Helpful & Harmless (Bai et al., 2022a) and TL;DR Summarize (Stiennon et al., 2020):
1. PPO is not the right tool for doing RL in RLHF. We break apart PPO and show that the most “basic” policy gradient algorithm, Vanilla Policy Gradient REINFORCE (Sutton and Barto, 2020), is consistently outperforming PPO by 3.2% to 20.3% in terms of win-rate, across all dataset and base model pairing.
2. RLOO outperforms key baselines. Built on top of REINFORCE, RLOO enables using multiple online samples, and we empirically show it consistently outperforms baselines such as PPO, DPO (Rafailov et al., 2023) as well as RAFT (Dong et al., 2023) across all datasets and models. We show that RLOO makes better use of online samples than RAFT while presenting a higher robustness to noise and degree of KL penalty.
3. Modeling partial completions is not necessary. We effectively demonstrate that modeling partial sequences is an unnecessary undertaking for LLM preference training. Instead, modeling the full generations preserves performance while reducing complexity in the RL
stage and significantly accelerating learning.
4. RLOO is relatively robust to noise and KL penalty sensitivity. We also accompany our results with a multi-dimensional analysis concerning language fluency, diversity, and robustness to noise. We showcase RLOO robustness to noise and degree of KL penalty compared to RAFT.

heading: 2 Background
text: The original RLHF pipeline for LLMs proposed in (Ziegler et al., 2020) consists of three stages:
(1) SFT Stage: A pre-trained LM is instructiontuned using a dataset consisting of a given instruction prompt, and (typically) a human-written completion. The LM/policy is trained with a crossentropy loss over the completion only. Often, the SFT model, denoted as πsft is used to initialize both the reward model and the RLHF policy.
LRM = − log σ(rϕ(x, y+)− rϕ(x, y−)) (1)
where σ denotes the logistic function.
(3) RL Stage: In this stage, the reward model is used to provide online feedback in the optimization of the policy with the following objective:
max πθ
Ex∼D,y∼πθ(.|x)[rϕ(x, y)− βpKL], (2)
with pKL = DKLπθ(.|x)||πref(.|x) (3)
where β is meant to control the distance from the initial policy, πref during the optimization of rθ(x, y) as proposed in (Stiennon et al., 2022). The KL-penalty pKL is crucial as penalty-free optimization of the reward model leads to degradation in
the coherence of the model. Optimizing this objective is equivalent to maximizing the following KL-shaped reward in expectation:
R(x, y) = rϕ(x, y)− β log πθ(y|x) πref(y|x)
(4)
While reinforcement learning approaches share the components above, techniques differ in the formulation of the reward. To understand these differences, we introduce PPO and distinct alternatives such as REINFORCE and REINFORCE LeaveOne-Out in the following sections.

heading: 2.1 PPO
text: When using PPO in the RL stage, the initial state is determined by the prompt, each generated token is modeled as an action, and partial sequences are seen as states, with a discount factor (γ ∈ [0, 1]) of 1 used. In this framework, only generating the <EOS> token carries a reward as output by the reward model which is combined with KL penalty, while for all other tokens in the vocabulary, only the KL component is non-zero:
R(x, y) = ∑T
t=1 Rt(x, yt) (5)
where yt denotes the t-th token of y, T the number of tokens in the trajectory, and Ri the correspondingly shaped reward.
In practice, the following token-level clipped objective is used in PPO:
min ( f(yt|st)Âλ(yt, st), (6)
clip1+ϵ1−ϵ(f(yt|st))Âλ(yt, st) )
with f(yt|st) = πθ(yt|st) πold(yt|st) ,
where st = {y<t, x} represents the state i.e. context at generation step t that is composed of the history of generated tokens y<t and the given prompt x, πold is an older policy (not the same as πref ), and Â(yt, st) is the estimated advantage function, and ϵ is the clipping ratio. The advantage function Â(yt, st) is estimated using Generalized Advantage Estimation (GAE) (Schulman et al., 2018), of generating token (action) yt, at partial completion (state) at token t− 1 of the generation.

heading: 2.2 REINFORCE
text: Given that in LLM applications, r(x, y) is only obtained at the end of the full sequence, it may be more appropriate to model the entire generation as a single action, as opposed to each token. Although it has not been explored in the context of LLM alignment, modeling the full completion as a single action, as in the bandit formulation, allows using the REINFORCE estimator (Kreutzer et al., 2017; Nguyen et al., 2017a; Williams, 1992). This allows for back-propagating through the discrete action (generation) space, and directly optimize the KLshaped reward objective for the entire sequence.
Ex∼D,y∼πθ(.|x)[R(y, x)∇θ log πθ(y|x)] (7)
To improve learning, one can reduce the variance of the estimator in Eq. 7, while keeping it unbiased, by subtracting a baseline b that has high covariance with the stochastic gradient estimate of Eq. 7 (Williams, 1992; Mnih and Gregor, 2014):
Ex∼D,y∼πθ(.|x)[(R(y, x)− b)∇θ log πθ(y|x)] (8)
With a strong parameter-free choice for the baseline being the moving average of all rewards throughout training (Williams, 1992):
bMA = 1
S ∑ s R(xs, ys) (9)
Where S is the number of training steps, and (xs, ys) is the prompt-completion pair at the step s.

heading: 2.3 REINFORCE Leave-One-Out (RLOO)
text: The baseline in Eq. 9 is simple to implement and computationally cheap. However, it can be improved upon if we have access to multiple online samples, that can be used for further unbiased variance reduction: (1) The rewards for each sample can serve all other samples as a baseline. (2) Policy updates can be done on an average of gradient estimates for each sample, resulting in a
variance-reduced multi-sample Monte-Carlo (MC) estimate. This is the intuition behind the REINFORCE Leave-One-Out (RLOO) estimator, proposed by (Kool et al., 2019)1:
1
k
k∑
i=1
[R(y(i), x)− 1 k − 1 ∑
j ̸=i R(y(j), x)]
∇ log π(y(i)|x) for y(1), ..., y(k) i.i.d∼ πθ(.|x) RLOOk considers each y(i) individually and uses the remaining k − 1 samples to create an unbiased estimate of the expected return for the prompt, akin to a parameter-free value-function, but estimated at each training step. This is a much more effective baseline (as our experiments show) than bMA since it’s created on-the-fly for each sample and at each training step, but comes at a cost of increased sampling time during training.

heading: 2.4 Alternatives to RL in Preference Training
text: Iterative Fine-tuning Iterative fine-tuning methods use the trained reward model to rank completions of online or offline sampled prompts, and then iteratively fine-tune the policy on a selected subset (Gulcehre et al., 2023; Dong et al., 2023). We benchmark Reward rAnked FineTuning (RAFT; Dong et al., 2023), a simple cross-entropy loss is used on the best-ranked completion out of k online samples, based on R(x, y) or r(x, y). We note that RAFT does not make full use of all samples because it only optimizes using filtered top-ranked samples. In contrast, RLOO fully leverages constructing a baseline and a multi-sample MC estimate for the policy gradient.
Direct Preference Optimization (DPO) Unlike other methods, DPO (Rafailov et al., 2023) skips the reward modeling stage in the traditional RLHF pipeline and uses preference pairs to directly optimize the policy with the following loss:
− log σ(β log πθ(y+|x) πref(y+|x) − β log πθ(y−|x) πref(y−|x) )

heading: 3 From PPO to REINFORCE
text: We scrutinize individual components of PPO that we consider not an ideal fit for RLHF. We explain the theoretical origin, motivate with the practical conditions of LLM RLHF, and provide empirical support from preliminary experiments.
1In collaboration with the Huggingface team, we provide an open-source implementation of RLOO in the TRL (von Werra et al., 2020) library: https://github.com/ huggingface/trl

heading: 3.1 Revisiting the need for low-variance estimators
text: Actor-critic algorithms, such as PPO, were motivated in formulation by the high variance observed in traditional RL settings. PPO leverages lowervariance estimators of the total trajectory return to improve learning. These estimators are constructed by bootstrapping off a state-value function (Sutton et al., 1999; Schulman et al., 2018; Sutton and Barto, 2020). While bootstrapping reduces variance, the trade-off is the introduction of bias which risks optimizing for biased rewards.
In contrast, REINFORCE uses an unbiased Monte-Carlo estimator of the trajectory return that can have high variance in theory, especially if it is approximated with only a single sample, which is not frequently preferred in traditional Deep-RL environments. Recent work has offered a plethora of evidence that REINFORCE suffers from high variance and fails in the presence of large action spaces like NLP (Ranzato et al., 2016; Bahdanau et al., 2017; Ding and Soricut, 2017; Ammanabrolu and Hausknecht, 2020; Ammanabrolu et al., 2022; Martin et al., 2022; Korbak et al., 2022). However, we note that these findings were based on scenarios with poor conditioning when training from random or weak initialization as opposed to warm-starting it from a strong pre-trained model.
Here, we question whether this empirical evidence holds for RLHF. We posit that this is not a practical concern in fine-tuning LLMs due to the extremely strong initialization of the policy (a pre-trained LLM). In this setting, strong initialization coupled with prompt conditioning leads to the concentration of probability mass on a few tokens at each generation step, even though the number of possible actions is in theory enormous (refer to Appendix B for further discussion on the effect of conditioning). The optimization landscape is far less likely to present problems like destructively large and high-variance gradient updates. Thus, attempting to reduce variance further at the cost of introducing bias is not worth it.
Empirical support To validate this hypothesis, we vary the weight placed upon variance minimization and bias introduction. In the formulation of PPO in Sect. 2.1, the advantage estimator GAE (Schulman et al., 2018) is relied upon to trade-off bias and variance when estimating the true advantage function in PPO.
GAE introduces a hyper-parameter λ ∈ [0, 1] in
the true advantage function, which balances bias and variance of the constructed estimator. The closer λ is to 1, the higher the observed variance. The optimal choice of where to set λ = 0 depends on the environment. In a highly stochastic environment, minimizing variance at the cost of bias is a worthy trade-off. However, given a stable environment where variance is already low, the introduction of bias is needless.
At the extreme of 1 which imposes minimal bias at the trade-off of variance, the advantage term reduces to return the estimator used in Vanilla Policy Gradient (PG) REINFORCE, which directly builds on the REINFORCE estimator, by optimizing the trajectory returns starting from each token in the generation
∑T i=t γT−i−1Rt(x, yt)− bψ(st) (10)
where bψ(st) is a learned baseline state st, akin to how a value network is learned in the traditional RL setting, using a standard MLE loss 1 2( ∑T i=t γ T−i−1Ri(x, yi)−bψ(st))2. Note that the key distinguishing factor between Vanilla PG and REINFORCE as referred to in this work, is that Vanilla PG uses the REINFORCE estimator on the trajectory return starting from the context formed by the prompt and a partial completion, whereas the REINFORCE estimator described in Section 2.2 is applied to the the full trajectory return. We will return to this distinction in the results Section 5.1 when we evaluate whether evaluating partial completions is necessary in RLHF.
In Figure 1, we present the results of evaluating the reward for PPO given GAE with different value of λ. Two variants impose minimal bias but invite high variance ((Âλ=1.00 (Vanilla PG introduced above), and Âλ=0.95) and two variants which over-index on minimizing variance at the cost of bias (Âλ=0.0,and Âλ=0.5). Figure 1 plots the reward and observe that the most extreme variant Vanilla PG (unbiased Aλ=1.0) performs the best given it presents no bias at the risk of high variance. We observe a monotonically decreasing reward with decreasing λ. This supports our hypothesis that reducing variance at the cost of bias in an RHLF setting needlessly introduces bias given the stable default properties of the environment.

heading: 3.2 Clipping is Rarely Necessary in RLHF
text: Next, we turn to the clipping-ratio ϵ (see Eq. 6), which is used to prevent large policy updates when
πθ πold deviates far from 1, i.e., to prevent updates that are too far off from the current policy (Schulman et al., 2017).
In Figure 1, we compare reward curves for independent PPO training with and without clipping. Note that we also turn off clipping for the value network, for these set of experiments, as it has been observed to have a noticeable impact on learning in traditional Deep-RL environments (Engstrom et al., 2020). The removal of these components does not impact learning meaningfully. We empirically found in our RLHF setting that the loss is actually clipped on average < 5% of the time per batch, throughout training across all dataset and base-model pairings, which indicates that the learning regime is close to being “on-policy”, with policies varying slowly from one iteration to the other.
To further validate this, we completely turn off clipping followed by removing the ratio πθπold , while λ = 1, which reduces the PPO loss to that of Vanilla PG. If anything the removal of clipping gives a slight boost in performance, validating our hypothesis that large off-policy updates in our optimization regime are rare and do not have catastrophic effects on learning as they do in traditional Deep-RL.

heading: 3.3 Modeling Partial Completions is Not Necessary
text: As described in Sect. 2, PPO models each token as an action whereas REINFORCE models the entire generation as a single action, as opposed to each token. In practice, in LLM RLHF a r(x, y) is only attributed to the <EOS> token, where for all other tokens, only log π(yt|st)πref(yt|st) composes Rt(x, y), which is not meaningful.
From a pure RL point of view, the environment dynamics are fully deterministic (PD({y<t+1,x}|st, yt) = 1), meaning that our environment (context) changes deterministically based on the new token/action predicted. Hence, the problem can be reduced to a bandit problem, where the Markov Decision Process (MDP) consists of only the initial state as determined by the prompt, and the terminal state, which is always reached after the generation (Kreutzer et al., 2017; Nguyen et al., 2017b). Note that modeling the entire generation as a single action is done explicitly by REINFORCE but is also done implicitly with iterative fine-tuning methods which generate the entire completion before filtering using a reward model.
In the results section 5.1 we will explicitly compare REINFORCE and RLOO which both model the full trajectory return to PPO and Vanilla PG which both model the partial completion. We ask is modeling the entire generation as a single action sufficient to achieve similar or better performance in RLHF?

heading: 4 Experimental Setup

heading: 4.1 Training Details
text: Datasets We report results on the TL;DR Summarize (Stiennon et al., 2020) and Anthropic Helpful and Harmless Dialogue (Bai et al., 2022a) datasets. The trainig split of TL;DR Summarize 2 dataset contains 116k human-written instructions and 93k human-annotated preference pairs. The preprocessed Anthropic-HH3 dataset contains 112k many training preference pairs.
Models For both datasets, we use Pythia-6.9B (Biderman et al., 2023) as the pretrained basemodel. To ablate the effect of the pre-trained model quality on the human preference optimization, we also experiment with Llama-7B coupled with the Anthropic-HH dataset. To ensure a fair comparison across all methods, we use a context length of 512 tokens during both supervised fine-tuning and the reward model training. We initialize both the reward model and policy with the corresponding SFT checkpoint, unless noted otherwise.
Experimental Details For the TL;DR Summarize dataset, we use the dedicated SFT split. Since the original Antrophic-HH dataset does not include a separate SFT split, we use prompts and the preferred responses from the binary comparisons during the SFT stage similar to prior work (Yuan et al., 2023; Dong et al., 2023; Rafailov et al., 2023). In the preference training stage, we use the same prompts as in the SFT stage to generate completions. Further details on the experimental setup and hyper-parameters are given in Appendix D

heading: 4.2 Evaluation
text: Optimization Quality For all online methods (all methods except DPO), to measure how well they optimize the intrinsic objective , we report average rewards (using the training RM) on 1000 samples from the test set. To measure how well
2https://github.com/openai/ summarize-from-feedback
3https://huggingface.co/datasets/Dahoas/ full-hh-rlhf
each method optimizes the extrinsic objective of aligning the models to human preference, we report simulated win rates in accordance with Alpacafarm framework (Dubois et al., 2024) where we use GPT4 as a proxy for human evaluation.
Alignment Tax RLHF fine-tuning is often associated with a drop in diversity and language fluency which is referred to as alignment tax (Askell et al., 2021; Kirk et al., 2024). Hence, we also report metrics which serve as proxies for fluency and diversity similar to (Dong et al., 2023). To measure fluency, we report perplexity measured using the preferred completions from the test set, similar to (Dong et al., 2023). Finally, we measure the diversity coupled with length by using the average n-gram diversity Li et al. (2016).

heading: 5 Results and Discussion

heading: 5.1 Reward Optimization
text: The objective of RLOO, REINFORCE with baseline, RAFT, PPO, and vanilla PG is to maximize the reward score, hence we compare the success
of optimization for each method. Fig. 2 shows the test reward scores plotted during the training for all methods. On each dataset, we use the same reward model for all the methods, thus their test reward scores are directly comparable.
Modeling Partial Completions vs Full Generations In Fig. 2, we observe that methods that do not model partial completions such as REINFORCE with baseline, RLOO, and RAFT consistently outperform Vanilla PG and PPO where each token is modeled as an action (i.e. partial completions). In addition to their higher performance in reward optimization, these methods are more efficient than Vanilla PG and PPO since they do not require training a learned baseline and a value network as in Vanilla PG or PPO respectively. This suggests that modeling partial sequences is unnecessary.
Sampling Efficiency Given the same sampling budget, k, RLOO consistently outperforms RAFT throughout training in terms of reward scores. Noticeably, even with a smaller sampling budget RLOOk=2, RLOO either closely matches or outperforms RAFTk=4, across all datasets and models. In this setting, RLOO uses only half the onlinesample budget given the same step count.
This confirms that RLOO leads to better optimization by using all the samples generated, unlike RAFT where only the top-ranked sample is used for fine-tuning. Showcasing the same finding, Fig. 5 in Appendix A plots the rewards with respect to the number of samples generated during training regardless of the k value.

heading: 5.2 Simulated Win Rates
text: Table 1 presents the win-rates against the original completions in TL;DR Summarize and AntrophicHH for each method. Here, we also include DPO.
Modeling partial completions is not necessary Recall that the key distinguishing factor between Vanilla PG and REINFORCE as referred to in this work, is that while Vanilla PG treats each token as an action, REINFORCE operates on the entire generation. As seen in Table 1, REINFORCE with baseline is on par with Vanilla PG on both TL;DR (70.7 vs 70.4) and HH (37.9 vs 36.4) datasets when using Pythia-based models. Moreover, REINFORCE with baseline outperforms Vanilla PG in HH dataset with a Llama-based model, achieving a higher win-rate (55.3 vs 52.3).
This confirms the effectiveness of only modeling the entire generation and not partial completions, even without using multiple samples during RLHF.
Win-rates are inline with test reward scores RLOO with k = 4 achieves the highest win-rates, outperforming PPO by 10.3, 14.5, and 32.1 for TL;DR, HH (Pythia) and HH (Llama), respectively. As the only exception, RLOO achieves the highest win-rate with k = 2 in HH dataset.
RLOO is more sample efficient than RAFT Comparing RLOO with RAFT, under the same sampling budget k, RLOO consistently outperforms RAFT in all datasets and models. When averaged across the three dataset and model pairings, RLOO achieves win-rates of 61.3 and 61.9 for k = 2 and k = 4, respectively, while RAFT scores 56.1 and 59.5, respectively. Notably, RLOO exhibits the highest increase in win-rate compared to RAFT, up to 9.9 as in the HH dataset with k = 2 and Pythia-based models (second column in Table 1).

heading: 5.2.1 Alignment Tax
text: Table 2 shows various intrinsic evaluation metrics including perplexity and diversity scores of Llamabased models in Antrophic-HH datasets.
Length of Generations Noticeably, the DPO trained model tends to be over-verbose (the longest generation length of 104 tokens on average) while the PPO trained model leads to short generations (the shortest on average with 16 tokens). We provide example responses in Appendix F.
Perplexity and Diversity As seen in Table 2, perplexity (PPL) scores are relatively close amongst RLOO, RAFT, and REINFORCE with baseline where all three methods achieve significantly lower perplexity than PPO and Vanilla PG.
In terms of diversity, Diversity-1 scores are similar across RLOO, RAFT, REINFORCE with baseline and Vanilla PG. Diversity-2 scores tend to slightly decrease for the methods with higher reward optimization (Askell et al., 2021). This is unsurprising given the significant difference in their generation length compared to other methods. Overall, RLOO and REINFORCE with a baseline maintain fluency and diversity in generations in
comparison with other methods while achieving higher reward scores and win-rates.
Reward variance Lower reward variance is desirable for applications such as safety and harmlessness where there is a high risk associated with generating a low-reward sample. Results in Table 2 show that RLOO for the same k value leads to slightly lower reward variance amongst the generations, compared to RAFT which is the most competitive method to RLOO in terms of reward optimization. Finally, Vanilla PG leads to the highest reward variance. REINFORCE with baseline, however, empirically results in 27% less variance, even though it either outperforms or is on par with Vanilla PG in terms of reward optimization and win-rates.

heading: 5.2.2 Robustness
text: As previously noted, a major drawback of RAFT is that it only optimizes on the highest-ranked sample and discards the rest of the online samples. Thus, the factors that can lead to inaccurate ranking of the best completion, can also impede learning significantly. We demonstrate this fragility by showing the effects of 1) high β for the KL-term and 2) inserted reward noise on RAFT in comparison to RLOO.
Mismatch from KL-penalty In Figure 3, we show the evolution of the KL distance and test reward curve r(x, y) throughout training for RLOO and RAFT, using k = 2, on the HH dataset using Pythia-based models. We vary the KL regularization, using β = {0.25, 0.5, 1}. Here, a larger KL penalty in R(x, y) (higher β) potentially increases mismatches between rankings of the k online samples. However, the choice of β often depends on multiple factors such as the distribution of the data and output logits of the base model, which may not allow for a low β value even with early-stopping.
We find that RAFT is more sensitive to higher KL regularization. In a low-regularized regime (β = {0.1}), RLOO and RAFT converge to equal KL distances from the reference policy while RLOO achieves a higher reward. However increased regularization with β = {0.25, 0.5, 1.0}, not only RAFT is worse at optimizing the reward, but also deviates more from the reference policy.
Mismatch from Reward Noise The reward model is itself a noisy proxy of reward signal due to the inherently noisy nature of human preference (Nguyen et al., 2017b; Kreutzer et al., 2018). Inspired by literature in Bayesian deep learning on
modeling Aleatoric uncertainty (Kendall and Gal, 2017; Collier et al., 2021), to simulate the effect of such noise in different degrees, for each prompt, we add noise to the rewards. Concretely, we add noise ϵ to the output logits of the binary classifier: rσ(x, y) = r(x, y) + ϵ where ϵ ∼ N (0, σ2).
Figure 4 shows the drop in reward at different levels for noise σ = {1.0, 3.0, 5.0}. As expected, the unaltered training reward decreases for both RLOO and RAFT. However, the drop is far more pronounced for RAFT with σ = {3.0, 5.0} This is due to the addition of reward noise, impacting the relative rankings hence the training reward. In contrast, RLOO presents a relatively robust reward optimization under a noisy reward signal.

heading: 6 Conclusion
text: At a high level, this work posits that the RLHF setting of fine-tuning an LLM has a strong initialization of the policy, which coupled with further conditioning on a prompt, alleviates historical concerns with high variance and large action spaces. We support this position with empirical results, showing that techniques such as REINFORCE & RLOO rarely used in traditional RL settings due to high variance outperform PPO. We also show the improved robustness of RLOO relative to iterative fine-tuning methods like RAFT — achieving a best of both worlds sweet spot.

heading: 7 Limitations
text: As one of the limitations of our work, we do not study the reward model (RM) over-optimization, which refers to the problem when optimizing the
trajectory of the proxy reward, the model diverges from the “gold” reward objective (Gao et al., 2022). This aspect has not been studied yet also for iterative fine-tuning methods such as RAFT and deserves a dedicated study. We leave this to future work.
Another limitation is the exploration of RLOO baseline in a single token action framework, where partial sequences are modeled. In this work, we show that modeling partial sequences are unnecessary undertaking in the RLHF context. However, understanding how RLOO performs in such modeling formulation can potentially be useful in other contexts.
Finally, we limit our work to rewards from human preferences and we did not explore the other rewards such as ROUGE, BLEU, or other metrics used in the NLP.

heading: 8 Acknowledgement
text: We would like to thank Ivan Zhang, Phil Blunsom, Florian Strub, Max Bartolo, Bharat Venkitesh, Roger Grosse, and Keiran Paster for helpful discussions. We would like to especially thank Matthieu Geist for multiple fruitful discussions on the framing of this work and feedback on the final manuscript. We would also like to thank our colleagues in Cohere and Cohere For AI for their continued support throughout the under-taking of this project.

heading: A Additional Results
text: Below we present additional experimental results on the RLOO estimator.
A.1 RLOO vs RAFT Sample Efficiency
Total Online SamplesFigure 5: RLOO and RAFT Rewards are plotted at various points through-out training based on the total # of samples seen (regardless of whether k = 2 or k = 4). RLOO is more efficient than RAFT in-terms of online sample use, regardless of k and dataset.

heading: B Effective Conditioning
text: To evaluate the hypothesis that probability mass is heavily concentrated and conditioning significantly narrows the likely generation space, we empirically study the characteristics of the output distributions and each generation step. We use the Llama SFT model used for the HH experiments in Results section.
Probability Mass Concentration Figure 6 (Right) plots the total probability mass concentrated in the top {1, 16, 32, 64} tokens. There’s a notable jump in the total probability mass after the first token is generated, which points to the effectiveness of conditioning from the first token and prompt. From that point on, a significant (∼ 60%) portion of probability mass is put on only the single most probable token at each step, with more than ∼ 90% of the total mass being concentrated on the top 16 tokens, with diminishing increases that point for the top 32 & 64 tokens. This empirical evidence directly supports our reoccurring claim that even though the feasible search (action) space at each step is enormous, in practice due to the conditioning from the SFT model and the prompt, most of the probability mass is only distributed amongst a fraction of the possible tokens.
Low Entropy Figure 6 (Left) plots the Normalized Entropy Ĥ(X) = H(X)Hmax(X) , where Hmax(X) is the entropy of the uniform distribution under the vocabulary-size. Similar to the jump in probability mass in Figure 6 right, as expected, the biggest the drop in entropy occurs right after the first token is generated and only slightly rises up to the end of the generation and is consistently low. This is further supporting evidence that the generation space is heavily skewed and naturally suggests there to be low variance in the probability of the generations, due to the entropy in the generative process being consistently low. This further motivates the single action modelling formulation as it suggests that the first conditioning in the generation is the most impactful.

heading: C RLOO’s Connection to Contrastive Loss
text: Multiple other works in iterative fine-tuning (Zhao et al., 2023; Yuan et al., 2023),utilize a contrastive-style loss by up weighing the log-probabilities of positive samples and down-weighing the probabilities of negative samples, as determined by the reward model
Lk=2c = − log π(y+|x) + log π(y−|x) (11)
We also have the corresponding loss with k = 2 to Equation 2.3 as:
Lk=2RLOO = (R(y+, x)−R(y−, x))
2 (− log π(y+|x) + log π(y−|x)) (12)
It’s clear that RLOOk=2 loss is exactly the contrastive loss but weighted by the difference between the absolute cores (the 1k factor is merged into the learning-rate).

heading: D Training Details
text: Below are additional details on training and data preprocessing. Data-preprocessing For each dataset, we filter out prompts that exceed a pre-specified length to minimize the occurrence of generations not containing the EOS token. We filter prompts longer than 448 and 348 tokens, for the TL;DR and HH datasets, respectively. SFT Training For the TL;DR Summarize dataset, we use the dedicated SFT split. For Antrophic-HH, since the original dataset does not include a separate SFT split, we use prompts and the preferred responses from the binary comparisons during the SFT stage. This is consistent with prior work (Yuan et al., 2023; Dong et al., 2023; Rafailov et al., 2023).
In terms of training hyperparameters, for the Pythia models, similar to previous work (Touvron et al., 2023b; Bai et al., 2022a), we train for 2 epochs with an initial learning rate of 2e-5 in both summarization and dialogue tasks. For the Antrophic-HH dataset, since we don’t have an SFT set, we use the preferred responses from the binary comparisons which make up the HH dataset. This is consistent with prior work (Yuan et al., 2023; Dong et al., 2023; Rafailov et al., 2023). For the summarize dataset, we use the dedicated SFT set indicated by the initial dataset. For the Llama models, we found that 1 epoch for the SFT stage is sufficient. RM Training In the RM stage, we train RM for 1 epoch with an initial learning rate of 1e-5.
For both RM and SFT training, we use a cosine decay learning schedule (Loshchilov and Hutter, 2016) and a 0.03 warm-up ratio. Preference Training For TL;DR Summarize dataset where we only experiment with Pythia models, we train each variant for 600 steps with a rollout batch-size of 512. We use a β value of 0.03.
For Anthropic-HH, we train Pythia models for 393 steps with the same batch-size configuration as for TL;DR summarize. As for Llama models, we follow the setup in (Dong et al., 2023) and use 2048 rollout and step batch size over 2 epochs. We use β = 0.10 for all Anthropic-HH experiments unless otherwise noted. For both datasets, we use the same prompts as in the SFT stage to roll out online generations. Across both datasets and all the models, we use a constant learning rate of 1e-6 with a linear warm-up duration of 3 % of total steps. Learning-rates were chosen after a sweep of {1×10−6, 1×10−5, 2×10−5} for RAFT and RLOO, and {1× 10−6, 1× 10−5} for PPO and Vanilla PG. For all algorithms, we take 2 gradient steps for each batch.

heading: E GPT-4 Evaluation Prompts
text: TL;DR Summarize: Which of the following summaries does a better job of summarizing the most important points in the given forum post, without including unimportant or irrelevant details? A good summary is both precise and concise.
Post: {instruction} Summary (A): {output_1} Summary (B): {output_2} FIRST provide a one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, on a new line, state only "Summary (A)" or " Summary (B)" to indicate your choice. Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <"Summary (A)" or "Summary (B)">
Anthropic-HH: For the following query to a chatbot assistant, which response is more helpful? Query: instruction
Response (A): {output_1} Response (B): {output_2} FIRST provide a one-sentence comparison of the two responses and explain which you feel is more helpful. SECOND, on a new line, state only "Response (A)" or "Response (B)" to indicate which response is more helpful. If they are equally good or bad, state "Neither". Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <"Response (A)" or "Response
ı(B)" or "Neither">

heading: F Example Responses
text: F.1 TL;DR Summarize (Pythia)
PROMPT: Below is a reddit POST and the corresponding SUBREDDIT and TITLE. Write a both precise and concise summary of the contents of the post. SUBREDDIT:r/loseit TITLE:Caffeine addiction. I have a horrible caffeine addiction, and I don’t like sacrificing any of my daily calories for coffee. I used to drink 5-6 Diet Dr. Peppers a day, but I have switched to almost exclusively drinking only water most days. I do have a Diet Dr. Pepper occasionally, and I know that technically it’s okay to probably have one a day, but I honestly just feel better all around when I drink only water. But if I don’t have some caffeine in the morning, I will be writhing in the floor in pain with a skull-pounding headache by mid-day. I know. Ridiculous, right? So, I went to the vitamin aisle the other day in search of a supplement with a little caffeine and the only thing I found was One A Day Adult Vitacraves Gummies with Energy Support. Here’s the website with [nutritional info] One serving is 20 calories and has 90 mg of caffeine. There’s also some vitamin B6, B12, and folic acid in there that supposedly helps turn food into energy. I have no idea if that’s true. I’ve been taking them a couple days, and so far I do feel more energized and I haven’t had any headaches. The issue is that one bottle is around $9 and there are only 60 in a bottle. You have to take 4 (4!) a day, which is very intriguing considering the brand name and all. So it’ll cost me around $20 a month. Which is admittedly cheaper than what I used to spend on Diet Dr. Peppers, but still. I was just curious if anyone else had a better option for caffeine in a supplement or vitamin?,
RLOO (K=4): "I have a horrible caffeine addiction. Went to the vitamin aisle and only found One A Day Adult Vitacraves Gummies with Energy Support. Cost $9/bottle and contains 90 mg of caffeine. Anyone else have a better option for caffeine in a supplement or vitamin?" RLOO (K=2): "I have a horrible caffeine addiction. I switched to exclusively drinking only water most days. I went to the vitamin aisle and only found one brand of supplement with caffeine. It costs $9 a bottle. Anyone else have a better option for caffeine in a supplement or vitamin?" REINFORCE W/ B.: "I have a horrible caffeine addiction. I don’t like sacrificing any of my daily calories for coffee. I went to the vitamin aisle and only found a supplement with 90 mg of caffeine. Is there a better option for caffeine in a supplement or vitamin?", RAFT (K=4): "I have a horrible caffeine addiction. Went to the vitamin aisle and only found One A Day Adult Vitacraves Gummies with Energy Support. Cost $9 a bottle and have 90 mg of caffeine. Anyone else have a better option for caffeine in a supplement or vitamin?" RAFT (K=2) "I have a horrible caffeine addiction. I switched to only drinking only water most days. Found One A Day Adult Vitacraves Gummies with Energy Support. Cost $9 a bottle. Anyone else have a better option for caffeine in a supplement or vitamin?" PPO: "I have a horrible caffeine addiction. Went to the vitamin aisle in search of a supplement with caffeine but only found One A Day Adult Vitacraves Gummies with Energy Support. Cost around $20 a month. Any better options? " VANILLA PG: "I have a horrible caffeine addiction and was looking for a supplement or vitamin with caffeine. Only found One A Day Adult Vitacraves Gummies with Energy Support which costs $9 per bottle. Any better options for caffeine in a supplement or vitamin?", DPO: "I have a horrible caffeine addiction. I like only drinking water most days, but occasionally I want some caffeine. I found some caffeine in a supplement in the vitamin aisle, but it’s expensive and I was just wondering if anyone else has a better option."
F.2 HH (Llama)
PROMPT:: "Human: Why does it hurt when I stretch too hard?Assistant:"
RLOO (K=4): "Stretching too hard can cause muscle soreness or injury. This is due to a combination of muscle strain and inflammation. It’s important to stretch gradually, and to listen to your body, so you don’t overdo it." RLOO (K=2): "Stretching too hard can cause muscle pain or muscle soreness. This is usually temporary, and will go away as your muscles get used to the stretch. It’s important to start with a light stretch, and gradually increase the stretch as your muscles get used to it." REINFORCE W/ B.: "When you stretch too hard, you can cause minor muscle strain. It2̆019s usually not a big deal, but it can cause some discomfort. It2̆019s also a sign that you might be overdoing it, so it can be a warning sign." RAFT (K=4): "Stretching too hard can cause muscle strain. This is especially true for muscles that aren2̆019t used to stretching, or for muscles that are already fatigued. It2̆019s also possible to injure a muscle if you stretch it too far, but this is less common." RAFT (K=2): "When you stretch too hard, you can cause a minor muscle tear or strain." PPO: "Stretching too hard can cause muscle soreness or injury." VANILLA PG: "Stretching too hard can cause muscle soreness and pain. This is because you are overstretching the muscle, which can cause damage to the muscle fibers. It is best to stretch gradually and gently, to avoid injury." DPO : "Stretching can be uncomfortable if you do it too aggressively, because it can cause minor muscle soreness or even minor injury. When you stretch, you are putting stress on your muscles and connective tissue, and this stress can sometimes cause minor damage or injury. If you stretch too aggressively or for too long, you can cause minor muscle soreness or even minor injury. This can cause pain or discomfort during or after the stretching. It can also cause minor damage to your connective tissue, which can cause pain or discomfort after the stretch."

heading: Future Work
text: We leave this to future work. Another limitation is the exploration of RLOO baseline in a single token action framework, where partial sequences are modeled. In this work, we show that modeling partial sequences are unnecessary undertaking in the RLHF context. However, understanding how RLOO performs in such modeling formulation can potentially be useful in other contexts. Finally, we limit our work to rewards from human preferences and we did not explore the other rewards such as ROUGE, BLEU, or other metrics used in the NLP