Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
Authors: Michael Noukhovitch
Shengyi Huang
Sophie Xhonneux
Arian Hosseini
Rishabh Agarwal
Aaron Courville
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning from human feedback
efficient llm finetuning
off-policy RL
Primary area: foundation or frontier models, including LLMs
TL;DR: Asynchronous generation and learning can make RLHF 2x faster to train, while maintaining performance

Abstract: The dominant paradigm for RLHF is *online* and *on-policy* RL: synchronously generating from the large language model (LLM) policy, labelling with a reward model, and learning using feedback on the LLM's own outputs. While performant, this paradigm is computationally inefficient. Inspired by classical deep RL literature, we propose separating generation and learning in RLHF. This enables asynchronous generation of new samples while simultaneously training  on old samples, leading to faster training and more compute-optimal scaling. However, asynchronous training relies on an underexplored regime, online but *off-policy* RLHF: learning on samples from previous iterations of our model which give a worse training signal. We tackle the fundamental challenge in this regime: how much off-policyness can we tolerate for asynchronous training to speed up learning but maintain performance? Among several RLHF algorithms we test, online DPO is found to be most robust to off-policy data, and robustness increases with the scale of the policy model. We study further compute optimizations for asynchronous RLHF but find that they come at a performance cost, giving rise to a trade-off. We verify the scalability of asynchronous RLHF by training a general-purpose chatbot from LLaMA 3.1 8B on an instruction-following task $\sim$40\% faster than a synchronous run while matching final performance. Finally, we extend our results to math and reasoning to demonstrate asynchronous RL can finetune Rho 1B on GSM8k $\sim$70\% faster while matching synchronous accuracy.

Published as a conference paper at ICLR 2025
ASYNCHRONOUS RLHF: FASTER AND MORE EFFICIENT PFLICYY RL FOR LANGUAGE MODELS
Michael Noukhovitch - 3 dh
Shengyi Chengyy Huang
Sophie Xhonneux Eh dh Arian Hosseini @ dh
Rishabh Agarwal
Aaron Courville a
Mila Quebec AI Institute Université de Montréal
Allen Institute for AI
Google Deepmind
Canada CIFAR AI Chair
ABSTRACT
The dominant paradigm for RLHF is online and on-policy RL: synchronously generating from the large language model (LLM) policy, labelling with reward model, and learning using feedback on the LLM's own outputs. While perfor- mant, this paradigm is computationally inefficient. Inspired by classical deep RL literature, we propose separating generation and learning in RLHF, This enables asynchronous generation of new samples while simultaneously training on old samples, leading to faster training and more compute-optimal scaling. However, asynchronous training relies on an underexplored regime, online but off-policy RLHF: learning samples from previous iterations of our model which give a worse training signal. We tackle the fundamental challenge in this regime: how much off-policyness can we tolerate for asynchronous training to speed up learn- ing but maintain performance? Among several RLHF algorithms we test, on- line DPO is found to be most robust to off- off-policy data, and robustness increases with the scale of the policy model. We study further compute optimizations for asynchronous RLHF but find that they come to a trade-off. We verify the scalability of asynchronous RI.FF by training a general-purpose chatbot from LLaMA 3.1 8B on 400% faster than synchronous run while matching final performance. Finally, we extend our results math and reasoning demonstrate asynchronous RL finetune Rho IB on GSM8K 70% faster while matching synchronous accuracy.
performance cost, giving rise
ffllooii task
INTRODUCTION
Reinforcement learning (RI.) is critical for training AT assistants based on large language mod- els (LLMs) to ensure they follow instructions (OpenAT, 2022), are helpful and harmless (Bai et al., 2022a), and are factually accurate (Roit et al., 2023). As LLMs have increased in size and capability, the scale and complexity of RL finetuning for LLMs has also substantially increased. State-of-the- art LLMs are often finetuned for weeks (Llama Team, 2024; Google Deepmind, 2024), presumably with large amounts of compute resources.
Yet the dominant paradigm for RL finetuning of LLMs, online on-policy RL (Ouyang et al., 2022), computationally inefficient. Online RL methods generate batch of responses from the model, get feedback on this batch (e.g. from a reward model), and update on-policy with feedback on exactly this model's responses, before generating the next batch. Recent offline methods efficiently learn directly from a fixed dataset of responses and feedback (Rafailov et al., 2023) but they underperform online methods (Xu al., 2024). Since feedback on model's own generations is crucial to good performance (Tang et al., 2024a), we propose generating responses online but learning off policy on previous iterations' feedback. By running both processes asynchronously and leveraging new efficient generation libraries (Kwon et al., 2023), we can greatly reduce compute time.
"mi chael neckhoww www.tthnttttteeemcccmommmmooottt real. ca, code at git www.//mckhoo////am..o... r1hf

---

Published as a conference paper at ICLR 2025
0.950
Method
Asyne
Gold Win-Rate
0.925
Syne
0.900
Scale
2.8b
0.875
lb
410m
0.850
0.825
100
150
200
250
300
Compute Time (minutes)
Figure 1: Asynchronous off-policy RLHF is more computationally efficient, and matches the
win-rate of synchronous on-policy RLHF on TLDR across model scales. On 4x A A100 GPUs, results in training 2.8B Pythia model 25% faster and improvements in speed increase with scale.
This work focuses on RL finetuning with human feedback (RLHF) and makes a first step into ef- ficient, asynchronous RLHE. We demonstrate strong results and find insights the widely-used RLHF benchmark, TLDR summarization (Stiennon et al., 2020)
We propose asynchronous RLHF and demonstrate that it requires off-policy learning, an underexplored direction for RLHF research. Moreover, show that RLHF performance generally degrades with more off off-policyness.
2. We evaluate many popular RLHF losses and find that Online DPO is most robust to off- policy data and robustness improves with the size of the policy model.
3. We scale model sizes and show that asynchronous RLHF training speed scales better than synchronous RLHF. We achieve the same performance as synchronous state-of-the-an methods 25% faster with 2.8B models (Figure 1).
4. We demonstrate ways to further optimize compute efficiency in gencration-constrainne and coiitng-constraaned scenarios. In our setup, we improve further and achieve nearly the same performance 250% faster with 2.8B models.
We then scale up and train general purpose chatbot by finetuning LLaMA 3.1 8B on high-quality dataset human-wrre -rrtten demonstrations, No Robots (Rajani et al., 2023)
5. At scale, asynchronous RLHF trains 40% faster than synchronous approach and achieves equal performance as measured by GPT-4.
Finally, we demonstrate our results extend to general RL for math and reasoning by finetuning Rho 1B (Lin al., 2024) on Grade School Math problems (Cobbe al., 2021)
6. On math, asynchronous RL trains 68% faster than synchronous while matching exceed- ing satte-of-theaat finetuning numbers (Kazemnejad et al., 2024)
2 BACKGROUND
2.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
RLHF is a method to align models with hard-to-quantiffy human preferences using human or syn- thetic feedback (Christiano al., 2017; Bai et al., 2022b). In the standard setup for LLMs (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022), we first gather a dataset of prompts x and two responses y.y' (e.g. from model) and have humans judge which response is better and which is worse. Next, we learn reward model r (z, y) on the dataset to approximate human judge- ment of responses. Finally, train our model by learning online: iteratively generating responses
2

---

Published as a conference paper at ICLR 2025
to prompts, labelling responses with the reward model, and using RL to optimize the reward. As LLMs are initialized from pretrained weights, RLHF seeks to optimize the reward while maintaining pretrained model abilities. We add Kullback-Lieber divergence (KL) loss to the objective to keep the model Hy close to the initial model Timit in order to reduce reward model overoptimization (Gao al., 2022) and alignment tax (Askell et al., 2021).
max (y)) (r(2, v) 3o
The standard method for this approach is Proximal Policy Optimization (PPO; Schulman et al., 2015) which uses actor-critic framework to optimize the objective. REINFORCE Leave-One- Out (RLOO; Ahmadian et al., 2024) simplifies PPO by reducing to the REINFORCE estimator (Williams, 1992) and empirically estimating baseline using multiple samples instead of using value network. Recently Guo et al. (2024); Calandriello et al. (2024) find competitive performance with Online DPO on the RLHF objective. They sample two online continuations, rank them as better (y+) and worse (y-) with the reward model, and optimize the objective of direct preference optimization (DPO; Rafailov et al., 2023).
Blog To To (z)) Blog To (y py-yyla) (x)
max By 18- (z) log Xg
Tinit (1n4(1//4))
Tinit (y- (x)
2.2 ASYNCHRONOUS DEEP RL.
Prior work in deep reinforcement learning (DRL) has focused mostly multi-step environments that run on CPU (Bellemare et al., 2013; Tassa et al., 2018; 8; Lillicrap et al., 2019). These algorithms typically on-policy, meaning the training data comes from rolling out the latest policy. This makes the training synchronous: the learner updates can only occur after policy rollouts, which is slow and can under-utilize hardware resources such GPUs. To improve throughput and scalabil- ity, methods were proposed to parallelize the actor's and learner's computation (Mnih et al., 2016; Espeholt et al., 2018; Berner al., 2019). Learners and actors can run faster independently but this introduces off-policy data, that is, the rollout data comes from slightly outdated policies. Despite the benefits of asynchronous DRL, to our knowledge, published RLHF works are always synchronous and asynchronous RLHF is severely under-explored.
2.3 EFFICIENT LLM TRAINING AND GENERATION
As LLMs have become a more mature technology, a significant effort has focused on improving the efficiency and speed of L.M training and inference. Although some techniques be leveraged for both (e.g. FlashAttention (Dao et al., 2022)), the problem of efficient training and generation are quite separate and require different methods (Liu et al., 2024). Efficient LLM training involves sharding large models, reducing optimizer states, pipeline batching, and speeding up backpropoga- tion (Rasley et al., 2020; Rajbhandari al., 2020). Efficient LLM generation focuses custom ker- nels, effective management of the KV cache, continuous batching (Kwon et al., 2023), and specula- tive decoding (Cai et al., 2024). As methods have advanced, the backends have diverged and current state-of-the-arr libraries for LLM training are separate from L.M inference.
3 ASYNCHRONOUS OFF-POLICY RLHF
The dominant paradigm for RLHF is fully on- line, on-policy RL: synchronously generate samples then train on these samples using a reward signal (Figure 2, top). To do so, we either (1) use the training library models for both training and inefficient generation, or (2) have generation and training GPUs alternate with some GPUs being idle while the others are working." The second option is clearly inefficient. However, the first option
On-policy RLHF is Computationally Inefficient
'A naive approach is to include both training and generation representations of model on each GPU but given ever larger LLMs, this isn't feasible memory-wise. A more advanced approach can interleave training and generation backends (Mei et al., 2024; Shen et al., 2024) to utilize both tools. But this incurs overhead from either slow switching between backends complex manual conversion the two. It also comes at the cost of reduced available memory since the latest inference tools build/optimize execution graphs that must stay
3

---

Published as a conference paper at ICLR 2025
Slow Generation (x(x) +
3/t
((41(0) + y1+1
((+2(2) + M1++2
GPU 1
GPU 2
On-Policy Training
V V2r v0)
V (ac, Vo (12, (612)
Fast Generation (z())) -+ (x)) -> M4+1 (x+2(x)) + 3/++2
GPU
GPU
Off-Policy Training
Vos r ge)
Beng (2,g041) Vesss (a, (2, Mey2)
Figure 2: Synchronous vs Asynchronous RLHF. Top: The current RLHF paradigm synchronously generates and then trains, leveraging the same GPUs for both. This means using slow training libraries for LLM generation. Bottom: We propose Cleanba-style (Huang et al., 2023) asynchronous RLHF, separating generation and training to different GPUs. This allows leveraging LLM inference libraries e.g. vllm (Kwon et al., 2023), greatly reduce generation time. Training time increases because we are learning on only one GPU but the overall runtime for three updates is lower. The caveat is that asynchronous learning requires off-policy training: learning on data created by our model at a previous timestep e.g. 0.+1 is updated using data generated by 0,
does not take into account the divergence between efficient LLM training and generation strate- gies, as discussed $2.3. Although training libraries can be used for inference, they are woefully outmatched. For example, let's compare the most popular library for training, Hugging Face trans- formers (Wolf et al., 2020), with a popular library for inference, vllm (Kwon et al., 2023). We find that vllm is 12x faster than transformers at generating 1024 batches of modest 128 tokens with 7B model. Empirically, this gap increases superlinearly with model size. Overall, neither option for synchronous on-policy training attractive.
3.1 OFF-POLICY RLHF
To optimize compute efficiency, it is crucial to separate generation and training on separate GPUs, so each may take full advantage of their optimizations. The clear solution is to use both generation and training GPUs simultancously and asynchronously. As shown in Figure 2, this requires training on samples that were already generated by our model at a previous iteration, also known as off- off-policy RL. See Appendix D for pseudocode. First, we investigate how off-policy learning affects RLHF methods and then we apply our learnings to optimize compute efficiency for asynchronous RLHF.
Empirical Setup We experiment on the widely-used RLHF benchmark. TLDR Summarization (Stiennon et al., 2020), which provides an SFT dataset of Reddit posts with summaries (Völske et al., 2017) and a feedback dataset of paired summaries where one is rated higher by humans. We follow Gao et al. (2022); Tang et al. (2024a) create controlled TLDR setup where can accurately measure improvements on preferences as well as reward model overoptimization. We relabel the feedback dataset using well-trained 6.7B "gold" reward model from Huang et al. (2024) that acts ground truth labeller for our task. Following Huang et al. (2024), we finetune Pythia 410m (Biderman et al., 2023) on the SFT dataset to produce SFT policies and, from the SFT checkpoint, train reward model on the relabelled dataset. Finally, we train an RLHF policy from the SFT checkpoint using the fixed reward model. We run methods with a mini-batch size of 512 for 256 steps, so approximately 130,000 samples or "episodes" are seen over the course of training.
Evaluation At inference time, we evaluate success by the win rate, according our gold model, of generated summaries over the human-written summaries in the SFT dataset. To evaluate align- ment tax, we measure how far our RLHF policy has drifted from its SFT initialization using an
in GPU memory. Fundamentally, we can do much better optimization and leverage more existing tools for training and inference they put on separate GPUs. See Appendix C for discussion
4

---

Published as a conference paper at ICLR 2025
05
311
Cold
Kd.
04
0.4
12
16
03
1100
114
RL (irrpprplsi
Figure 3: Trade-off' between Win-Rate and KL in Off-Policy PPO. PPO performance decreases as learning becomes more ooflppl Win-rate is highest when learning is fully on-policy (generate then train on N generated by the same old policy. This increases off-policyness and reduces win-rate. Left: Gold win-rate over training Middle: KL (perplexity) over training, higher further from initial model Right: Gold win-rate KL
mini-batchess). As we increase N, our model must take more steps on data
approximation of the Kullback-Lieher divergance (KL.), we measure the SFT model's perplexity on the RLHF policy's summaries.
3.2 OFF-POLICY WIN-RATE AND KI.
To evaluate robustness to off-policy data, we modify the on-policy RLHF setup to incorporate vary- ing levels of off off-policyness. Whereas the on-policy setup generates one mini-batch, labels with reward model, and updates, we propose to generate N mini-batches. Each iteration therefore con- sists of N mini-batch updates. The first update is fully on-policy as the model has not changed from generation time. But after each mini-batch update and gradient step, the model moves further away from the policy that generated the data. By increasing N, we can increase the level of off-policyness of the updates. This setting correspond to iterative RLHF approaches that generate and label batches of data, e.g. LLaMA 3.1 (Llama Team, 2024).
First, show the performance of the standard online baseline, PPO, as learning becomes more off- policy. We vary N from (on-policy) to 64 (very off-policy) and plot the gold win-rate and KI. over training in Figure 3 (left and middle). We corroborate prior work (Tang et al., 2024a; Tajwar al., 2024) and find that very off-policy data (and therefore offline data) is worse than on-policy. We extend those results and also find that on-policyness proportional learning success for RLHF, with logarithmic dropoff such that N = and N = are quite similar.
To accurately compare methods, we plot win-rate and KL against each other in a pareto curve (Noukhovitch et al., 2023) in Figure 3 (right). We find all values of N conform to the same general curve. For PPO, off-policyness did change the pareto frontier, the fundamental tradeoff of win- rate vs KL of our method. However, off off-policyness seems to slow down how training progresses along the frontier. This is in line with previous results from deep RL where data staleness reduces training speed (OpenAl et al., 2019).
3.3 ROBUSTNESS OF RLHF LOSSES TO OIT-POLICYNESS
Next, we investigate which RLHF loss is most robust to off off-policyness, potentially allowing more asynchronous training. We compare current popular methods, namely PPO, RLOO?, and Online DPO across range of off-policyness (N 1, 4, 8, 16) in Figure 4 (left). Although PPO is best on-policy RL (N 1), its performance is greatly reduced when moving to off-policy learning, as is RLOO's. Online DPO is clearly the most robust to off off-policyness. It is able to achieve higher win-rate at lower KL for slightly off oollcy learning (N 4) and is the only method achieve any reasonably amount of learning in highly off off-policy scenarios (N 64).
Both PPO and RLOO only sample completion per prompt whereas Online DPO samples 2. To disentangle this effect, we also run simple Best-of-2 baseline (Gao et al., 2022) that samples
2To compare the strongest possible methods, create modification to RLOO that is robust to off- policyness, see Appendix
5

---

Published as a conference paper at ICLR 2025
088
Onlon DPO
0.85
PPO
0.54
RLOO
X
Cold Wa-Raaee
Best-o6-2
0.50
082
Mint-batchos
Wite
0.75
0.50
2
0778
4
0.88
0.00
130
111
1.22
1.13
111
1,05
1100
1.22
1,100
1175
1230
1.23
RL (perpledit)
(perplexity)
Figure 4: Robustness of RLHF Losses to Off-Policyye Off-Policyness. Online DPO is more robust to off- policyness than PPO, RLOO (Left) or Best-of-2 SFT (Right). Performance is shown across levels of off- off-policyness increasing off- polleeyeless,, Online DPO retains much more performance than other methods, as evi- denced by off- off-policy points still being clustered close optimal performance.
mediated by number of mini-batches N E (1, 4, 16). With higher N
o
Polky Soale
410m
0.9
lb
Gold Win Rale
Gold Win-Rate
0.7
0.6
RM Scale
0.7
410m
35
Lb
0.6
2.8
1.10
L12
114
1.16
LO8 1.10
L12
1.14
1.16
KL (perplexity)
KL (perplexity)
Figure 5: Scaling Model Size with Off-Policy RLHF. Plotting the final win-rate vs KL for N -> 64 mini-batches, covering spectrum of on-policy to off off-policy RI. Scaling policy size (left) improves off- off-policy robustness as seen by tighter clustering of points. But scaling reward model size (right) does not, even though it reduces overoptimization, achieving reward with smaller KL.
completions and does supervised finetuning on the completion with the higher reward. We find that Best-of-2 also does not retain performance (Figure 4, right), implying that Online DPO's robustness may be due to the contrastive nature of the loss.
3.4 SCALING MODEL SIZE WITH OFF-POLICY RLHF
We scale our setup to Pythia model sizes 410m, 1b, and 2.8b to investigate how scaling affect off- policy RLHF with Online DPO. For clarity, we now plot the off-policy pareto curve by taking the final win-rate and KL at each of N E (1,2.4,8, 32, 64). We compare separately scaling the policy and the reward model.
Scaling Policy. First, we scale the policy size with a 410m, 1B and 2.8B model while keeping 410m reward model and show results in Figure (left). As policy size increases, more points on the off- off-policy pareto frontier
clustered towards the best-performing point. For example, 410m has two points (N = 16, 32) far from the optimal area and a wide spread, whereas 2.8b's worst point (N 64) is still quite close to optimal. This means scaling policy size increases robustness: more off-policy runs can approach the best possible win-rate and KL tradeoff.
Scaling Reward Model. Next, scale the reward model across 410m, 1b, and 2.8b while keeping a 410m policy and show results Figure (right). Following Gao et al. (2022), increasing reward model size allows achieving the same win-rate lower KI., redueing overoptimization. Though

---

Published as a conference paper at ICLR 2025
Asyne Generation
Asyne Generation
GPU 1
GPU
Off-Policy Training
Off- Pif-Policy Training
Figure 6: Asynchronous RLHF can Braining-Bounde (left) or Generation- (right). In practice, generation and training speeds differ so challenge of asynchronous learning is how best balance usage and leverage idle compute time to further improve training.
points are clustering in terms of KL, they are not clustering in terms of gold win-rate. More off- policy points do not achieve relatively better performance, evidenced by the 410m reward model achieving the highest win-rate for the most off-policy point (N = 64). Therefore, we observe that only policy scale, not reward model scale, that increases robustness to off-policy learning.
3.5 SCALING ASYNCHRONOUS OFF-POLICY RLHF
We apply our learnings to an actual asynchronous RLHF setup. Our results suggest we should aim to be on-policy possible so adapt the simplest, most on-policy asynchronous RL frame- work, Cleanba (Huang et al., 2023). At time step t, we generate completions for prompts with current model, Y/t (2r), and train on completions generated by our model one timestep back, max9 r(r, 1 + KKI,, as shown in Figure 2. We run both methods on A 1100 GPUs. For synchronous RLHH, formers. For asynchronous RLHF, we reserve one GPU for generation using the vllm library, and the rest for Online DPO training using Hugging Face transformers. We train the same three scales of model 410m, 1B, and 2.8B and set the policy and reward size to be the same.
all 4 GPUs for both generation and training with Hugging Face trans-
Across scales, we find that our one-step off aff-poolicy, asynchronous RLHF matches the final win-rate vs KL performance of fully on-policy, synchronous RLHF. In terms of compute, we plot the final gold win-rate against the clock time necessary to reach it in Figure 1. Our method more efficient at every model size and due to vlim, improvements scale such that at 2.8B, our run is 25% faster.
OPTIMIZING ASYNCHRONOUS RLHF
Although we have found a significant speedup, the naive asynchronous method is under-utilizing compute. Our model of asynchronous learning requires training and generation take approxi- mately similar amounts of time, which is not always reasonable assumption. If the speed of train- ing or generation is mismatched, some our GPU time will be spent idling, as shown in Figure 6. We propose a solution to take advantage of idling time in each scenario.
4.1 GENERATION- BOUND RLHF
Generation and obtaining reward signal can be fundamentally slower than inference. In the classic RLHF setup, generation is autoregressive and scales linearly with the length of the response to generate, whereas reward model inference can be constant. Recent work shows that reward may require human labelling (Llama Team, 2024), output chain-of- reasoning (Zhang et al., 2024; Ankner et al., 2024), executing external tools such Learn verifiers (Google Deepmind, 2024). In this scenario, we have extra training compute cycles and ask the question, "is it useful to train more on existing data?". Following previous work with PPO (Ouyang et al., 2022), we experiment with taking multiple updates on the same batch of generated data i.e. "ppo epochs" (Schulman et al., 2015). In our asynchronous TLDR setup, we generate N mini-batches and perform T 1,2,3 updates per mini-batel
We plot results across different scales in Figure/ 7 (left). At 410m and 1B scales, models achieve higher win-rate for the same number of generated samples, showing that multiple updates make training more sample efficient. This means that extra training time can used to increase win- rate. But measuring the final points on the pareto frontier in Figure 7 (right), we find that increasing
7

---

Published as a conference paper at ICLR 2025
Mini batch Updates
0.0c
0.950
Cold Wm-Rate
Cold Win-Rare
11500
0.90
Soale
0.815
22.50
0.00
0.825
110m
0.86
09800
400030
60,000
S9000)
120CX
1.10
111
1.2
13
114
1.15
Epaade
KL (perpexiit)
Interation-Bouna RLHF. We can leverage extra training GPU cycles to do multiple updates on the same generated mini-batch ("ppo epochs"). Left: At 410m and 1B scales, more updates per batch increases the win-rate achieved at any given episode, making training more data efficient. Right: Across scales, more updates change the pareto frontier and cause models to achieve the same win-rate at higher KI..
Figure 7: Optimizing
updates per mini-batch also increases drift in terms of KL. Therefore, in generation- scenarios, multiple updates may increase the win-rate with the same compute-ting time but incurs higher KL.
4.2 SHANINGG-BUNNE RLHF
The other option is if training is slower than generation. In our 2.8B experiments above, training on GPUs takes twice the time of generating on GPU, so our generation GPU idling for half the time. We believe that we can sample more continuations to improve Online DPO training. Inspired by the findings of Pace al. (2024) for reward model training, we propose to generate K samples instead of 2 at each timestep and apply the DPO objective only the highest and lowest rewarded completions. this way, our generation and reward model inference takes K/2 times longer while training remains the same. For TLDR, we experiment with K 4 and find the margin of reward between our highest and lowest samples is approximately 2x larger than our standard = 2 setup. We believe this can provide proceeds much faster. So we reduce the learning rate 2x and also train for half the number of steps.
clear gradient for training and, indeed, find that training
We plot the win-rate against compute time across three scales in Figure (left). We find that can achieve the same gold win-rate in just over half the time. As we were training-bound, increasing the number generations, while keeping training samples fixed, did not significantly increase our per-step training time. And K 4 asynchronous training allows us to reduce training steps by half, training 2.5x faster than synchronous. The caveat is that achieving this win-rate comes cost of higher KL as shown in Figure 8 (right). Though difference in KL decreases with scale, we still find a visible difference 2.8B. Similar to generation- -oound, optimizing training-boume RLHF can improve speed but at the cost of KI..
SCALING ASYNCHRONOUS RLHF
5.1 GENERAL-PURPONS CHATBOT
Next, we verify our findings at a larger scale by training an helpful instruction mnstruction-following chatbot with RLHE First, we create and label preference dataset. We finetune LLaMA 3.1 (Llama Team, 2024) on dataset of 10,000 human-written demonstrations for instructions, No Robots (Rajani et al., 2023) to create our model, to get a total 4 including the human reference the dataset. We create pairs (4 choose 2) of completions per prompt and GPT-4o judge (Zheng et al., 2023) create synthetic preference dataset. We train a reward model on this dataset from the LLaMA 3.1 SFT checkpoint.
SFT checkpoint. Then, sample another 3 completion per prompt from
We best- rest-performing algorithm, Online DPO, and train 8 H100s sync on on-policy and async off- eef-policy for 100,000 episodes. For each sample, we generate a completion of up to 1024 tokens per prompt but, since model is larger and we generate more tokens, generation using the huggingface
8

---

Published as a conference paper at ICLR 2025
0.95
0.96
Gold Wat-Bate
00312
Cold Win-Bate
0.92
0.90
Sevie 2.80
0.000
0.36
0.81
088
0.82
0.82
1300
Compute Time (mibnutes]
KL (paplesit)
Figure 8: Optimizing Training- Training-Bound Bound RLHF We can leverage extra generation GPU cycles to sample K completions per prompt instead of 2. Left: Sampling K = 4 improves the gradient such that we can train for half the number of steps and, across scales, achieve the same final win-rate at a fraction of the compute time. Right: The trade-off is that increasing K causes models to drift more in terms of KL order to achieve the same win-rate.
Model SFT
Win Rate t Compute Time +
Average Response Sequence Length
31.80%
198.40
Sync Online DPO
57.20%
230
286.21
Async Online DPO
57.20%
142
290.55
Table 1: Asyne RLHF Works at Scale for Chatbots. Asyne is effective at training a general- purpose chatbot with LLaMA 3.1 8B. runs 38% faster than sync while matching KL and final GPT4-o win rate against the human- responses on the No Robots test set (Rajani et al., 2023)
transformers library is too slow to feasible. So for both sync and async, we reserve one GPU for generation with vllm and the remaining seven for training. Synchronous on-policy learning idles the generation GPU while training and vice versa, whereas asynchronous trains off-policy as previously. We plot the reward and KL over training in Figure 9 in Appendix A.2 and find that asyne generally achieves the same reward and KL sync while being 38% faster. We evaluate the final models with GPT-4o a judge (Zheng et al., 2023), comparing their completions human-written responses on the No Robots test set. In Table 1, we find async achieves the exact same win-rate as sync, 57.2% while running 38% faster. Overall, we confirm that asynchronous RLHF is equally performant while being faster, even more large scale. We note that our async runtime could be even further improved and discuss major considerations in section A.2.
5.2 MATH AND REASONING
We now demonstrate that asyne can work generally for RL with language models using the well- known benchmark of grade-school level math word problems, GSMSk (Cobbe et al., 2021). The setup generally follow Kazemnejad et al. (2024), using Rho-IB (Lin et al., 2024), a state-of-the-ar LLM trained on natural language and math corpora, further finetuned on the ground truth reasoning and answers in the training dataset (Havrilla et al., 2024). To train, a reasoning trace and final answer if the answer string exactly matches
sampled for each math question, and the reward ground truth, and ( otherwise (Singh et al., 2023). We run for approximately 128k prompts and, following Kazemnejad et al. (2024), sample completions per prompt to therefore train for a total of 512k episodes. Final models evaluated using the pass@) @ metric the test dataset by greedy sampling I completion for each question and reporting the percentage of correct answers. Final KL measured using the perplexity of the base model on the generated completions.
We compare a strong, existing PPO baseline (Kazemnejad et al., 2024) against sync and async Online DPO in 4 GPU setup above and report the final evaluations in Table 2. Our sync Online DPO baseline slightly improves over the existing sync PPO baseline. Asyne, once again, matches sync performance while being more compute efficient, running 68% faster. This large improvement

---

Published as a conference paper at ICLR 2025
Model
Pass @
on Test Set +
PPL
Compute Time (Minutes) +
SFT
40.3%
Sync PPO*
50.3%
864*
Sync Online DPO
52.2% %
1.0916
218
Asyne Online DPO
52.6 %
1.0922
129
Table 2: Asyne RL works at Scale for Math and Reasoning on GSM8K. Training Rho IB on GSM8K, Sync Online DPO outperforms the strong, existing Syne PPO baseline. Furthermore, Asyne Online DPO is 68% faster and achieves the same performance (pass @ 1) and KI. (PPL. as Sync. *Syne PPO results from Kazemnejad al. (2024) used comparable IxA100 GPUs
demonstrates that asyne RL is perhaps even more suited to reasoning which eschews reward model and makes efficiency purely about optimizing LLM generation and training.
6 RELATED WORK
The most popular attempts making RLHF more efficient comes in the form of recent offline methods i.e. direct preference optimization (Rafailov et al., 2023, DPO) and followups (Tang et al., 2024b; Rafailov et al., 2024). By directly optimizing a policy using the feedback dataset, their method avoids costly online generation and is much more compute-efficient But recent works have shown that it is worse than online methods at achieving high reward (Xu et al., 2024) exactly because it eschews online generations (Tang et al., 2024a). Online and, specifically, on-policy data generated by the the model being trained key to achieving high reward while maintain pretrained model capabilities (Tajwar et al., 2024; Tang et al., 2024b; Agarwal al., 2023).
Our investigation therefore focuses on optimizing online RLHF methods but not exactly on-policy data. RLHF with off-policy data, generated from previous versions of our model, has been scarcely attempted as no previous methods have focused on asynchronous learning. Munos et al. (2023) provides theoretical arguments for learning from generations by an exponential moving average of the model, however, in practice, Calandriello et al. (2024) finds this to be equal or worse than learning on-policy. Though Tang et al. (2024a) focus on online vs offline methods, one additional experiment in their appendix bears similarities to our N mini-batches setup. Their results imply that more off-policy data decreases online RLHF performance. We greatly extend this direction and investigate which methods perform best off-policy well as how performance is affected by scale.
This work demonstrates a novel approach to efficiency for RLHF and proposes practical ways to tackle it. Complementary to work, Mei et al. (2024); Shen et al. (2024) focus on the engineering challenges efficient, synchronous RLHF and propose clever distributed training techniques to account for generation, reward model inference, and training. Hu et al. (2024) provide another engineering solution that leverages vllm to improve generation speed. Our proposed asynchronous RLHF may remove some of the engineering challenges of synchronous RLHF (e.g. by separating generation and learning), which can make future engineering approaches even more efficient.
CONCLUSION
This work makes first step towards asynchronous RLHF, demonstrating how it can improve effi- ciency while maintaining performance. We demonstrate that an off-policy regime does not have to impact performance and the possibility of further performance/speed tradeoffs. While synchronous RLHF libraries are currently well-optimized and likely outperform our setup, we believe we have proven the viability of asynchronous learning and encourage the community to investigate and opti- mize this new paradigm. Previously in deep RL, as environments became more complex and model sizes increased, asynchronous learning became the dominant paradigm (Mnih et al., 2016; Berner et al., 2019). In RLHF, model sizes are increasing and recent works have proposed more complex multi-turn environment setups (Shani al., 2024; Kumar et al., 2024). As such, it seems likely that asynchronous RLHF will become a computational necessity and we believe it is important to turn RLHF research towards this new paradigm and with the challenges it presents.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We note model training details in Appendix A. Our experiments are based on existing open-source codebases and code used in the paper is open-sourced on github https: / github.... com/ mnoukhov/ / assnn p_rhhih. All baseline model checkpoints and training datasets are released on HuggingFace Hub, see github repo for details. To extend this work, one-step async RLHF has been integrated into the open-instion library and notably used for Tulu 3 (Lambert et al., 2025).
ACKNOWLEDGMENTS
MN thanks Samuel Lavoie for many helpful discussions, Amirhossein Kazmnejad and Milad Agha- johari for great help in GSM8k experiments, MN and AC thank members of Sony Research, Fa- bien Cardinaux, Lukas Mauch, Stefan Uhlich, James MacGlashan, Bac Nguyen Cong, and Ghouthi Boukli Hacene, for their constant feedback and ideas. MN is funded by Sony, est aussi soutenu par la bourse Fonds de recherche du Québec Nature et Technologies. MN is grateful Mila and ServiceNow for resources used in experiments and grateful to Google for cloud credits provided through a credit award.
AUTHOR CONTRIBUTIONS
MN led the project, proposed the idea, wrote the code and ran TLDR experiments, and helped write the paper. SH wrote the code and ran large-scale experiments, helped write code for TLDR, proposed Cleanba and wrote key code for asynchronous training, gave feedback in meetings, and helped write the paper. SX wrote code for HH-RLHF that did not end in the final paper, helped run TLDRR experiments, gave feedhack in meetings, and helped write the paper. AH wrote some initial code for best-of-n, gave feedback in meetings, and helped edit the paper. RA advised throughout the project, proposed experiments, and helped write the paper. AC was the main advisor, proposed research directions and experiments, and helped edit the paper.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. Generalized Knowledge Distillation for Auto-regressive Language Models, October 2023. URL http://t//at /trttp://arxv..... org/ abbs/////////// 220.//3022..0 3664 9. arXiv:2306. 13649 [cs].
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to Basies: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs,, February 2024. URL
24C2. 142..44440 arXiv:2402. 14740 [cs].
7achary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprint arXiv:2408. 1791, 2024.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, 7ac Hatfield-Dodds, Danny Her- nandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A General Language Assistant as a Laboratory for Alignment, December 2021. URL http:://arxa org/ abs/2bl2 00861. arXiv:2112.00861 [cs].
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022a. URL http: / arxii wwwp:v.ooor/aaaooeee abs 2204 05862. arXiv:2204.0586 [cs].
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols-
11

---

Published as a conference paper at ICLR 2025
son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li. Eli Tran- Johnson, Ethan Perez, Jamie Ketr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI Feedback, 2022b.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learn- ing Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelli- gence Research, 47:253-25 June 2013. ISSN 1076-9757. doi: 10.1613/jair.3912. URL 4708. arXiv: [cs].
http:///rxiin httt:::/axxiv.or/abbb//////
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dkebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0668/ 2019.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Lan- guage Models Across Training and Scaling, May 2023. URL 2304.00 01373. arXiv:2304.01373 [cs].
www.iiii.rg.oooeeeoooooooooooorr abs/
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, January 2024. URL https:/ htapps///axiiiia
aatpaaaxiiv...////.a//aaaa org 2ttp://bs//22...11. 10774v3.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human Alignment of Large Language Models through Online Preference Optimisation, March 2024. 2Xiv:2403.08635 [cs, stat].
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL https: /ttppaaaaassa////////e nips.co..oooooo ca/ paper/2017/ hha.///ahh/a//a heeech//5eeccee /ttp///22222add533:1111111
html.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. URL http:
//arxiv.org/abll 2110. 41 68. arXiv:2110.144 10.14168 [cs].
org/ aabs///////// 22072 08691. arXiv:2307.0866 006699 [cs]. Tri Dao. FlashAttention-2:: Faster Attention with Better Parallelism and Work Partitioning, July 2023. URL http::/t//aaee
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, May 2022. URL https://arxiv. org/abs/2200 2205. 14135v2.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable Distributed Deep-RL. with Importance Weighted Learner Architectures. In ICML. arXiv, June 2018. doi: 48550/arxiv.. 11.500arXiv..122.01555 1802.01561. URL httttttpp///iiii http://iwiiiiiraa////aaaaae/e//e 01561. arXiv: 006.01561 [cs].
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Chris Cremer, Arash Ah- madian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, and Matthieu Geist. Contrastive Policy Gradient: Aligning LLMs sequence-level scores in supervised-friendly fashion. In EMNLP. arXiv, June 2024. doi: 104485500arXxw.2466.1113 //arxiv.org/aas/////. 19185. arXiv:2406.19185
URL http:
12

---

Published as a conference paper at ICLR 2025
Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization, October 2022. URL httpp://axxx..../// htb.///2..107611 arXiv:2210.10760 [cs, stat].
AlphaProof and AlphaGeometry Team Google Deepmind.
achieves silver-
medal standard solving International Mathematical Olympiad problems, Septem- ber 2024.
URL https://deepmm: /ttp//////////ss///////ooog ai-solves- waw-ooo-rollmeeeee at
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable, 2022. URL https:/ http//gitttbbbb com wtt.ccmm///////e hunm/hhgggngface///// accelerate.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Di- rect Language Model Alignment from Online AI Feedback, February 2024. URL http:
h//arxiv.org//aaaee. /2wp/////.r/b////2.....2222 04792. a:Xiv2440..44792 [cs].
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yo Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching Large Language Models to Reason with Reinforcement Learning, March 2024. URL http:
//arxiv. org/ htte//bb/244000
www.////.... 240..2433...2 04 642. arXiv:2403.04642 [cs].
Jian Hu, Xibin Wu, Weixun Wang. Xianyu, Dehao Zhang, and Cao. OpenRLHF: An Easy- to-use, Scalable and High-performance RLHF Framework, July 2024. URL http://axxxx org/abs/2405 11143. arXiv:2405.1 [es].
Shengyi Huang, Anssi Kanervisto, Antonin Rafin, Weixun Wang, Santiago Ontañón, and Rousslan Fermand Julien Dossa. A2C is special case of PPO, May 2022. URL 2205. 09123. M.Xi:22205009123
Shengyi Huang, Jiayi Weng, Rujikorn Charakorn, Min Lin, Zhongwen Xu, and Santiago Ontañón. Cleanba: A Reproducible and Efficient Distributed Reinforcement Leaming Platform, September 2023. URL http:://araet org/ als 2210. 60036. arXiw:2310.00036 [cs].
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N+ Implementation Details of RLHF with PPO: A Case Study on TL.DR Summa- rization, March 2024. URL http: /tttpp//axxiv...// wawxiv.xiiivv///vv///axx//aaavvv org/ abs///b///aa 2403. 17031. arXiv:2403.17031 [es].
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah Smith, Iz Beltagy, and Hannanch Hajishirzi. Camels in changing climate: Enhancing Im adaptation with tulu 2, 2023.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback, 2024.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Unlocking RL Potential For LLM Reason- ing Through Refined Credit Assignment, October 2024. URL 2410. 01 1767. arXiv:2410.01679.
org/ abs//
Terry Kong. NeMo-Alignee-PRR#320: Upgrading TRTLLM to v13 by terrykong Pull Request #320 NVIDIA/NeMo-Alignee, November 2024. URL NeMo Aligner/pull/322
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su. John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint 4409 129/7, 2024.
13

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. arXiv, September 2023. doi: 00.488500an2Xiv2209..666 180...
URL http:
//arxiv.org abs/ 2309. 06180. arXiv:2309.0618 [cs].
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu. Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, Victoria Graf. Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tulu Pushing Frontiers Open Language Model Post- cost-Training, February 2025. URL http:// /ttp///axivvvv/ orgoorggaa
abs abs 2411 11124. arXiv:2411 11111.15.. 15124 [cs].
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Can- wen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, and Thomas Wolf. Datasets: A Commu- http://axxvvv...o nity Library for Natural Language Processing, September 2021. URL abs / 2109.2199. 02846. arXiv:2 S.4iv221990088.0 [cs].
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019.
URL https:tp//// /ttxxaxvvvvv////e// org/ aas/a/ss11159 02971.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not All Tokens Are What You Need, May 2024. URL http: / awwiv.ivvvv..... org/ abs /2044. 07965. arXiv:2404.07965 [cs].
Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, et al. Understanding lmss: A comprehensive overview from training to inference. arXiv preprint arXiv:2401.022 020088 2024.
Meta Llama Team. The Llama Herd of Models, August 2024.
Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. ReaLHF: Optimized RLHF Training for Large Language Models through Parameter Reallocation, June 2024. URL
attps::/aaee //tpss////xxvv./// org/ alss
14088v1.
mglss and nbro. Answer to "Why is the log probability replaced with the importance sampling in the loss function?", J July 2019. URL https: http://///aassttteetttaaee www.ssttccacceeececc.o.. a/ 13216.
Volodymyr Mnih, Adrià Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In /CML. arXiv, June 2016. doi: 00.48550/axii.1602..7777 URL http: [cs].
org/abs 602 717.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash Learning from Human Feedback. December 2023. URL http://arrk
org/abss/3122 00886. arXiv:2312.0088 [cs, stat].
Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron Courville. Language Model Align- ment with Elastic Reset. NeurIPS, May 2023.
NVIDIA. PyTriton: Framework facilitating NVIDIA Triton Inference Server us- age in Python environments., November 2024a. URL https: /tt::://gtttttt com/ hetoo-iiffeennc--eveeer//eeeee
original-date: 2022-11-08TT1::0:555..
NVIDIA. TenSorRT-LLM, November 2024b. URL https: btttps://githhane github. www/NVIIIII
TensorRT-I.IM orieinal-date: 0023-08-16T17:14.7777
14

---

Published as a conference paper at ICLR 2025
OpenAl. ChatGPT: Optimizing Language Models for Dialogue, November 2022. URL hotps: http://cww...at..oo//aaaaaaaaaaa https: fettps://cpenaiie http://aaa.aaaaaaaa..c.ccccccaaa
/ htwebbacchee.. htwp//aac...oogg//ceeee wtt.::gggggeessseoeeee.i.... wwt.://ttttttttccctceeeeeaaeeeee
OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with Large Scale Deep Reinforcement Learning, December 2019. URL http: http://axxvvvv./o/ee abs 1912. 0668C. arXiv:1912.0665
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacoh Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, March 2022. URL http: http/aaavvvv/..////e//eeeeea.av. org/ abs / 2203. C2155. arXiv:2203.021: [cs].
Alizée Pace, Jonathan Mallinson, Eric Malmi, Scbastian Krause, and Aliaksei Severyn. West-of-N: Synthetic Preference Generation for Improved Reward Modeling, January 2024. URL http:
www.ii.or//ro.ooo..... abs 2401. 12086. 6. arXiv:2401. 12086 [cs].
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-PPeformance Deep Learning Library, December 2019. URL http:// pttp://arxvv.ooi org. abs/ / 1912..111112 01703. arXiv:1912.01703 [cs, stat].
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model Secretly Reward Model, May 2023. URL
18290. urXiv:2305.88290 [cs].
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From Sr5 to SQ Your Language Model is Secretly Q-Function, April 2024. / http://www.rr/a/////...... arXiv:2404. 12358 [cs].
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. Hugging Face No Robots, 2023, URL https: htss:///hgginggace..e co/
datasets Bangeets/Huginggg /t/g//ugggngg/ www.nnggaaceeee... no_ootsss..
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20, pp. 1-16, Atlanta, Georgia, November 2020. IEEE Press. ISBN 778-1-72819-998.
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. DeepSpeed: System Op- timizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pp. 3505-3506 New York, NY, USA, August 2020. Association for Computing Machinery. ISBN 978- -4993-9-4-99
doi: 01145/3394446634477700 URL attps://dou orooorrgg...... 1144 3113333944488.. 4..144//3344....47000
Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Léonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin, and Idan Szpektor. Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback. In ACL. arXiv, May 2023. URL http:://aaiiii orggabs//3/rr abs/2306 001 86. arXiv:2306.0M 86 [cs].
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust Region Policy Optimization. In ICML. arXiv, 2015. doi: w08550/arii..5522.055... URL http: //arxiv..o wwwpxiiwrrg/ab////5.. 05477. arXiv: [cs].
15

---

Published as a conference paper at ICLR 2025
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Pol- icy Optimization Algorithms, August 2017. httttt////axxi arXiv:1707.06347 [es].
http:://axxvvv..g..//es htllgg/bb/1777/// 06347..
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, Avinatan Hassidim, Yossi Matias, and Rémi Munos, Multi-turn Reinforcement Learning from Preference Human Feedback, May 2024. URL http:/ /ttpvv.riivvvvva//iiv/.......avl org/ abs 20abs/2405 2405. 14 455.. arXiv:2405. 14655 [cs].
Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng. Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksii Kuchaiev. NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment. In CoLMM arXiv, September 2024. doi: 0.448500an2iw.2455500445 URL http: http://axxvv..o http//awww..rr.//////aaaaaaa 01481. arXiv:2405.01481.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro, Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Par- allelism, March 2020. URL http: http//axxvv..rr/. wwt.b/////aaaoooooooo.oo..osss 1999. 08053. arXiv: 190.08599..
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Pe- ter J. Liu, James Harrison, Jachoon Lec, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond Hu- man Data: Scaling Self- Training for Problem Sollvig with Language Models, December 2023. URL www.thtp://.. /ttp///axvvvvvvv orgo abs/ 2312 2. 6585. arXiv:231 13.06688 [cs].
Nisan Stienmon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize from human feedback. In NeurIPS. arXiv, 2020. URL http:://///// http:://xxivv..ooo.
0020090013220 01325. arXiv:2009.0133 [cs].
Richard Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018. URL http::tt///. / http:://nnnoopm//ee wwwpooplpptele wwtpommppeeeeeeeseee net wwttbbok//teeeeeeeeooooooooeeeet book 2nd. html.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ermon, Chelsea Finn, and Aviral Kumar. Preference Fine- TinneTTning of LLMs Should Leverage Suboptimal, On-Policy Data, April 2024. URL http: arwiv..iivvvveevvv/a...aaaaavvvv wwwp:/ivvorg/////oo www.orggg///////eeee 14367. arXiv:2404.1 14367 [es].
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms, May 2024a. URI.htpp: //aaxxiiia wwwpiii.orgg/bba. 2405 08448. 48884:2405.08448 [cs] version: 1.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Row- land, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. General- ized Preference Optimization: A Unified Approach to Offline Alignment, February 2024b. URL httrgg/bb//200 httt:://arxia org/ abs /2402 05749. arXiv:2402.05749 [cs].
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap, and Martin Ried- miller. DeepMind Control Suite, January 2018. URL ttttp://arxin org/ 00690. urXiv:1801.0069 [cs].
Leandro Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lambert. TRL: Transformer Reinforcement Learning, 2023. URL https: : http:p:///thtttit /github... com/ 1 vwerra/trl.
16

---

Published as a conference paper at ICLR 2025
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to Learn Automatic Summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the Workshop New Frontiers in Summarization, pp. 59- 63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: N0.18653/11/117-45000 /tt. www.nnhwwoooooooooooooooooo www.orgg////00001111
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannanch Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023.
Ronald J Williams. Simple statistical gradient- algorithms for connectionist reinforcement learning. Machine Learning, pp. 28, 1992.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguisties. doi: 10. 20.18633/1/222000 116655//1/2000..mmpmmdeeeee//// eenne-demos...... URL https: http::ss///alaaaahlloloooo
org/ 2020. emnlp-demos. 6.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO Superior to PPO for LLM Alignment? Comprehensive Study, April 2024. URL http: / aarriviivviii v...v..o org/ abs 11.04.17711 10X4v:2404.10719 [es].
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari.. Xiaoxia Wu, Am- mar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin. Masahiro Tanaka, Shuai Che, Shuai- wen Leon Song, and Yuxiong He. DeepSpeed Easy, Fast and Affordable RLHF Training of ChatGPT-like Models All Scales, August 2023. URL http:://axxii/ org/ abs abs/2308.
01320. arXiv:2308.013: 01320.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mchran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Sto- ica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurlPS. arXiv, October 2023. doi: 10.485500axXi..2300556663 URL http:://a www.wwwwwwwwa.../../..//a.....ae arXiv:2306.05685 [cs].
05685.
Daniel M. Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, 2019.
URL http: /ac::///arre arxiv. org/ abs/ 1909. 08593. arXiv: 1909.08993 [cs, stat].
17

---

Published as a conference paper at ICLR 2025
A EXPERIMENT DETAILS
A.1 TLDR SUMMARIZATION
Experiments on TLDR Summarization are trained using the Hugging Face trl library(von Werra et al., 2023) which leverages Pytorch (Paszke et al., 2019), Accelerate (Gugger et al., 2022), and Datasets (Lhoest et al., 2021). The base models used are the "dedupep" versions of Pythia 410m, IB, and 2.8B. We follow Huang al. (2024) for all dataset preprocessing and supervised finetuning hyperparameters. We relabel the dataset with Huang et al. (2024) 6.7B reward model by getting the score for each pair of completions and assigning the completion with the higher score as the "chosen" completion yt. the other being the "rejected" completion y-- We show the baseline results after supervised finetuning, before RLHF training in Table 3.
Model
Win Rate
KL (Perplexity)
SFT 410m
25.36%
1.075
SFT 1B
26.82%
1.071
SFT 2.8B
35.16%
1.068
Table 3: The win-rate and perplexity of models after supervised finetuning, before RLHF training
For RLHF training, follow the hyperparameters and suggestions of Huang al. (2024) with slight modifications. For PPO, see hyperparameters in Table 4.
Hyperparameter
Value
Learning Rate
x 10
Learning Rate Schedule
Linear
Generation Temperature
0.7
Batch Size (effective)
512
Max Token Length
1,024
Max Prompt Token Length
512
Response Length
128
Number of PPO Epochs
Total Episodes
131,072
KL penalty coefficient
0.05
Penalty Reward Value for Completions
Without EOS Token
-1.
Table 4: PPO Training Hyperparameters
We use the same hyperparameters for all methods with the following method-specific modifications
RLOO sets k
Online DPO sets 0.1
Best-of-2 sets learning rate to 10 as it tends overfit quickly
A.2 No ROBOTS FNSTRICTIONNFFOLOOWING
Hyperparameters Large-scale experiments were trained with Open Instruct (Wang et al., 2023; Ivison et al., 2023; 2024)². We finetune LLaMA 3.1 (Llama Team, 2024) on a dataset of 10,000 human-written demonstrations for instructions, No Robots (Rajani al., 2023) create our SFT checkpoint. The SFT hyperparameters are in Table 5.
Given this SFT checkpoint, we generate a synthetic preference dataset using GPT4-o. First, we gen- crate 3 demonstrations with temperature 0.7 per prompt from the SFT model, totaling 4 generations
hhttps:// http:////gttthu..ooa/.aooaa lenaa /opan-ipppe-aaaeeeee instrct
18

---

Published as a conference paper at ICLR 2025
Hyperparameter
Value
Model
Meta-Llama-331-88
Max Sequence Length
4,096
Batch Size (effective)
128
Learning Rate
5.0 10
Learning Rate Schedule
Linear
Learning Rate Warmup Ratio
0.03
Learning Rate Weight Decay
0.0
Number of Epochs
2
Table 5: No Robot SFT Model Training Hyperparameters
per prompt when counting the reference completion in the dataset. We create 6 pairs (4 choose 2) of completions per prompt and use GPT-4o judge (Zheng al., 2023) to create synthetic preference dataset. We train reward model on this dataset from the LLaMA 3.1 SFT checkpoint, using hyperparameters from Table
Hyperparameter
Value
Model
The Trained No Robot SFT Checkpoint
Learning Rate
3 10
Learning Rate Schedule
Linear
Batch Size (effective)
256
Max Sequence Length
1,024
Number of Epochs
1
Table 6: Reward Modeling Hyperparameters
Given the SFT model and reward model, we then train Online DPO on 8 H100s synchronously on-policy and asynchronously off-policy for 100,000 episodes. For each sample, generate completion of up to 1024 tokens per prompt, an appropriate length for the task. Since our model larger and generate more tokens, generation using the huggingface transformers library is considerably slower than vllm (i.e., 20x slower in preliminary testing), and infeasible. So for both sync and async, reserve GPU for generation with vllm and the remaining seven for training. Synchronous on-policy learning idles the generation GPU while training and vice versa, whereas asynchronous trains off-policy previously. Table has the hyperparameters.
Hyperparameter
Value
Model
The Trained No Robot SFT Checkpoint The Trained RM Checkpoint
Reward Model
Learning Rate
x 10-7
Learning Rate Schedule
Linear
Generation Temperature
0.7
Batch Size (effective)
256
Max Token Length
1,024
Max Prompt Token Length
512
Number of Epochs
1
Total Episodes
100,000
Beta (DPO coefficient)
0.03
Response Length
1,024
Penalty Reward Value for Completions
Without an EOS Token
-0..0
Table 7: Online DPO Training Hyperparameters
19

---

Published as a conference paper at ICLR 2025
2.5
Acy
Asyne Syne
2.0
Slyme
Reward Madel Store
20
1.5
KL Divergence
10
03
10
-1.5
n
30
100
150
2010
0
50
100
150
200
Relative Time (Minutes)
Relative Time (Minntes)
Figure 9: Asynchronous RLHF works at scale for General-Purpose Chatbot. Comparing syn- chronous and asynchronous online DPO for training an 8B general- purpose chatbot. Asynchronous learning achieves the same reward model score at lower KL and 38% faster.
ceneral-purpose
For an additional evaluation, also generate completions on the trained online DPO checkpoints and compare these completions with human-wiin written completions using GPT4-o judge. The win rate and average length generated responses for all models are in Table 8. The async online DPO checkpoint actually obtains exactly the same win rate as the syne online DPO checkpoints. This is perhaps less surprising since both models have very similar KL. and scores the end of the training, as indicated in Figure 9,
Model
Win Rate
Average Response Sequence Length
SFT
31.80%
198.40
Async Online DPO
57.20%
290.55
Sync Online DPO
57.20%
286.21
Human
N/A
179.726
Table 8: The trained models' GPT4-o win rate against the humtn-written written responses the test split of the No Robots dataset (Rajani et al., 2023)
Practical Considerations for Asynchronous Runtime Interestingly, our asynchronous speedup could be even faster. For the synchronous experiments, vllm generation takes 21 seconds and train- ing takes 33 seconds. We have 233 steps of training, so it takes roughly (21 + 33) seconds 233 Ro 209 minutes. In an ideal setup, we expect asynchronous RLHF train at the speed of the slower process, training i.c. 33 seconds * 233 z 128 minutes. roughly 63% faster than the synchronous training time. In practice, though, we find asynchronous training to take 151 minutes: 26 seconds for generation and 39 seconds for training. We note two possible reasons for the slowdown:
1. Global interpreter lock (GIL): With Python, only one thread can execute at any given time and we run a threads for each of generation and training. This issue mitigated when we call torch operations, which can run in parallel internally. However, GIL does occur additional blocking for our generation and learning.
2. Communication between training and generation: The generation process must pass generated completions to training and the training process must pass updated model pa- rameters generation. The latter can be expensive and passing policy parameters is synchronous GPU call which can slow down training.
Although these issues are outweighed by our improvements, solving them may important moti- vation for future work. For example, the latter issue can be mitigated by reducing the frequency of synchronization between generation and learning. One potential solution is generating more mini- batches of data and learning more off-policy in $3.2.
20

---

Published as a conference paper at ICLR 2025
2.3
Asyne
Asyme
Slyre
Syne
Reward Madel Store
25
15
KL. Divergenee
20
05
10
5
-1.5
0
100
200
300
400
500
100
200
310
400
500
Relative Time (Minates)
Relative Time (Minules)
Figure 10: Asynchronous RLHF also works at scale with PPO. Comparing syne and asyne PPO for training an 8B general-purpose chatbot. Asyne achieves the same reward model score a similar KL and 38% faster.
Model
Win Rate f
Average Response Length
Compute Time (Minutes) +
SFT
3 31.8%
198.40
Sync Online DPO
57.2%
286.21
213.04
Async Online DPO
57.2%
290.55
154.03
Sync PPO
53.0%
220.35
507.33
Async PPO
52.6%
229.42
446.08
Table Asyne PPO also matches Syne PPO while being faster for General-PP Peerrsse Chatbots: Trained model GPT4-o win rate against the human-writted responses the No Robots test set (Ra- jani et al., 2023), average length of the generated responses, and compute time to train on 8xH100 GPUs. Just with Online DPO, Async PPO closely matches the performance of Sync PPO while being faster to train. Though GPT-4o judges Online DPO to be most performant, PPO models gen- erate notably shorter responses.
PPO We aim to verify that asynerhonous RLHF will work with other methods scale well. We therefore run the same setup as $5 with PPO, instead of Online DPO. All hyperparameters are the same Online DPO, see Table 7, except decrease the KL coefficient value did not perform well for PPO. We plot the training curves Figure 10. As previously, we find that asynchronous learning nearly exactly matches the performance of synchronous learning. while being faster. We note strange spike in KL for both runs, perhaps due to instability of PPO. We evaluate the performance the final models using GPT-4o win-rate in Table and find that asyn- chronous PPO nearly exactly matches the performance of synchronous PPO. Overall asynchronous learning is shown to be effective for PPO as well as Online DPO.
= 0.01 the original
Although PPO achieves a similar reward model score to Online DPO, it performed worse when evaluated by GPT-4o. This likely due to the instability PPO's optimization and difficulty in finding the best possible hyperparameters. PPO is also more than 2x slower than Online DPO as requires maintaining a value network in memory which reduces batch size and also training the value network which takes time.
A.3 GSM8K
Hyperparameters We mainly use the hyperparameters Kazemnejad et al. (2024) but modify them slightly, as shown in Table 10. Kazemnejad et al. (2024) only experiment with PPO and RLOO (as well as a variant of PPO) where they sample 8 completions per prompt. We found 4 completions to have the same performance as 8 for RLOO
sample and train on 4 completions. For Online DPO, we sample 4 completions per prompt then choose the best and worst as our DPO pair, as in $4.2. This means that sampling takes the same amount of time RLOO, but training is faster
21

---

Published as a conference paper at ICLR 2025
Hyperparameter
Value
Model
Rho- IB SFT on GSM8k
Learning Rate
10
Learning Rate Schedule
Constant
Generation Temperature
0.7
Max Prompt Token Length
512
Response Length
512
Number of PPO Epochs
Batch Size (effective)
252
Number of Completions per Prompt
Total Prompts Seen
129024
Total Episodes
516096
Beta (DPO and KL coefficient)
0.05
Table 10: Online DPO and RLOO Training Hyperparameters for GSM8k
0.9
0.8
1.8
Thin Score
Tain Seore
0.7
0.7
0.6
Sync Online DPO
Syne Online DFO Asyue Ouline DPO
0.5
Syne RLGO
044
100
230
300
400
0
50
100
150
200
Compute Time (Minutes)
Compute Time (Minutes)
Figure 11: Online DPO is performant on GSM8K, Async Online DPO is performant and faster. Left: Sync Online DPO matches the general train performance of sync RLOO as measured by the train score over compute. Both methods are run for 512k episodes but Online DPO trains on only the top/bottom completions, so runs faster. Right: Asyne Online DPO is 68% faster than Sync for GSM8k training and reaches nearly identical train score.
since throw out 2 samples, leading to speed improvements in Figure 11 left. Preliminary experiments taking the best and worst 2 for RLOO yielded worse results.
Due the length of outputting reasoning steps, GSM8k requires generating 512 tokens for the output. This makes generation with HuggingFace transformers infeasible*. For our 4 GPU exper- iments, we therefore synchronously generate on one GPU with vLLM and train on the other three with transformers, as in $5, alternating training and generation". We run on 4xL40s GPUs.
Online DPO outperforms RLOO, PPO Our base model achieves 40.3% pass@) I on the GSM8k test set. We run RLOO, and Online DPO and use existing numbers from a well-tuned PPO baseline from Kazemnejad et al. (2024). We plot RLOO vs Online DPO train score (percentage of correct answers per batch) in Figure 11 (left) and the final results in Table 11. We find that Online DPO outperforms RLOO and achieves 52.6% final pass @1 after 512k episodes. In comparison, Kazem- nejad et al. (2024)'s well-tuned PPO achieves 50.1 1% after 650k episodes. We also note that our synchronous Online DPO takes & 3.5 hours to run 40l 48Gb GPUs whereas Kazemnejad et al. (2024) synchronous PPO takes 14.4 hours on larger 4xA 80Gb GPUs with comparable
'Generating a hatch 1024 examples with transformers takes 60 seconds all available optimizations like Flash-Attention 2 (Dao, 2023). contrast, vL..M takes only Rs 11.5 seconds running on single 80GB A100 This corresponds the synchronous RL.FF paradigm used by Hu al. (2024)
80GB A100 GPUs with
22

---

Published as a conference paper at ICLR 2025
Model
Pass @ on Test Set f
PPI. +
Compute Time (Minutes) +
SFT
40.3%
Syne PPO*
50.3%
864
Sync RLOO
50.0%
1.0778
385
Sync Online DPO
52.2%
1.0916
218
Table 11: Online DPO is the most performant method for GSM8k. Final models' pass I on the GSM8K set. a heuristic measure of KL, and compute time train 4xL40s GPUs. Online DPO improves over RLOO and a well-tuned PPO baseline while Asynchronous Online DPO achieves the same results 68% faster. *Syne PPO scores and times are from Kazemnejad al. (2024) trained with comparable 4xA100 GPUs
Fast Generation
(x(x) + Yt
((+1(2) + 9t-1
((+2(a) - yt++2
GPU
GPU
On-Policy Training
V Vottz (3x, M1)
V Varz2 (2t, Mh-1)
Fast Generation
(1(x)= + D1
(.+1(2) 7 Met1 (.22(x) - y1+2
APU
GPU 2
Off-Poliey Training
Veu ((x, v.)
Par, Bo41)
Figure 12: Comparing Simple Synchronous Training to Asynchronous Training. Top: The sim- ple but effective approach efficient synchronous training, e.g. implemented by Hu et (2024), separates training and generation onto different GPUs and leverages state-of-the-ar generation library like vLLM to generate and state-of-the-art training library like Deepspeed for training. or- der train synchronous, you idle generation while training and vice-versa. Bottom: Asynchronous RLHF speeds up training by training off- polic on previous steps' generations and therefore removes idling time.
speed while also leveraging vLLM for generation and deepspeed for training. This demonstrates speed and effectiveness of our synchronous baseline. Online DPO also required essentially no hyperparameter tuning to achieve reasonable performance, opposed to PPO and RLOO. We also note that Kazemnejad et al. (2024)'s proposed method VinePPO, an advanced version PPO that relies more samples, outperforms Online DPO with Pass @ of 53.4% but it requires much more compute time ( 68 hours). We do not claim that Online DPO state-of-the-art for GSM8k but note it is strong baseline.
Asynchronous Speedup Analysis Here, explain how we achieve the speedup in our GSM8k experiments. We visually demonstrate the synchronous and asynchronous paradigms in Figure 12. As noted above in the details above, this synchronous paradigm is necessary HuggingFace trans- formers is too slow for generation so we must leverage vLLLM. We also note this synchronous paradigm used in an existing competitive library, OpenRLHF (Hu et al., 2024).
In our GSM8k experiments, training takes up GPUs and generation takes 1. In our synchronous setup with Online DPO, generation takes on average 12.2 seconds, getting the reward (evaluating the answer) takes 0.10, and the training step takes 12.8 seconds. This adds up to 25.1 seconds whereas the average actual step time is 25.5 seconds, showing that synchronous training adds an overhead of 0.4 seconds. Asynchronous training runs generation and training the same time but at the cost of increased overhead. Since we are training-bouma we would expect the average step time to be 12.9 seconds but actual step time is 15.1 seconds. Although save lot of time by running training and generation asynchronously. we lose some speed due to 2.2 seconds overhead, for reasons outlined in section A.2.
23

---

Published as a conference paper at ICLR 2025
B OFF-POLICY RLOO
We wish to use formulation of RLOO (Ahmadian et al., 2024) that robust to off-policy data. Flet-Berliac et al. (2024) argue that the formulation is already robust to off-policy data. But both empirically and theoretically, we find this isn't 't the case. Below, we argue for our off off-policy RLOO formulation, which we call Proximal RLOO.
RLOO (Ahmadian et al., 2024) with k = 2 samples 2 completions for each prompt from the model y1, 1/2 TT(- then updates the loss objective
1 [log mo (v1lz) (R(y), a) R(y2, x)) log wo (V2(1)) (R(yz. x)) R(y1, x))]
I(0)= 2
For simplicity, we will focus on the gradient of just one sample yn and write the baselined reward as an advantage A(yn (æ) = R(), R(y2,
LRLOO(() = loog I (31 (3)1)))((11)))
We can see that RLOO is just REINFORCE with a baseline and the gradient of the loss is quite standard
VgLRLOO(() Vo log to (()
Contrastive Policy Gradient (CoPG; Flet-Berliac et al., 2024) proposes an RLHF algorithm that is argued to be robust to off-po data and has connections to RLOO. our online, off-policy setup, samples are taken from previous policy Told. Here, CoPG can be seen modification of RLOO with & = 2 divided by the log-probability of the sample under the policy that generated it, Vold-
To(v1 Tota(11 (x)
(CoPG(() -log
Alva (x)
As shown in Flet-Berliag al. (2024), this has the exact same gradient vanilla RLOO
Tee (v (x)
VoLcoPa(0) = Va log
A(v (æ:)
notd (y1 (x:)
Ve log no (v1 xz) A(yl
This is argued mean that RLOO is already good objective for off-policy data but given that there is no reference to wold, we don't see how this can be the case.
Instead, we leverage an off- policy RLOO that follows the framework and suggestions of Proximal Policy Optimization (PPO; Schulman et al., 2017). Specifically, our loss uses an importance sam- pling ratio (Sutton & Barto, 2018):
To( (x: Á(yn (x)
L(0)
Told (V1 (x)
24

---

Published as a conference paper at ICLR 2025
0.8
RI4X0 - CaPG
RLOO Ohis
Gold Win-Rete
Minii Iust
0.4
15
00
KL (perpledit)
Figure 13: Our Proximal RLOO outperforms CoPG-style RLOO for online, off-policy learning
This ratio is still present in the gradient, which we derive with the log-probability trick (Huang et al., 2022; mglss& & nbro, 2019):
V&L(0)- V (o(vi
A(y1 (x)
(B xz)
Ta (1/1
(311 (z)
A(y (z)
(y1 (x)
Tald xx)
(o (a:)
V VoTo(y1 (z)
A(y1(z)
Told (V1 (x)
T0(11
log No (y1 A (x)
Mold
This demonstrates our loss gives the RLOO gradient with an importance sampling ratio between our current policy and the policy that generated the data Vold-
We also add PPO's clipping of the importance sampling ratio (here renamed re) to within of 1, for stability.
Lfiial = min (ro(er (ll +c)((11x)))
where ro(un)= = no ( (x)
(11
We call this method, Proximal RLOO, in reference to PPO. We compare the two methods in terms of off-policy robustness using our setup 3.3. As shown in Figure 13, CoPG performance drops to 0 as data becomes more off-policy (N = 16). In contrast, our PPO-style RLOO remains robust.
C WHY EFFICIENT SYNCHRONOUS RLHF IS NOT FEASIBLE
C.I TRAINING LIBRARIES ARE INEFFICIENT FOR GENERATION
Whereas asynchronous learning can fully leverage state-of-the the-hhh generation libraries, naive ap- proach to synchronous learning will generate using the training library (von Werra et al., 2023). We demonstate the necessity of efficient generation libraries by comparing the most popular open- source training library HuggingFace Transformers (Wolf al., 2020) and popular generation li- brary vLLM (Kwon et al., 2023) Figure 14. It is clear that generating with a training library is infeasible at larger scales.
More advanced approaches may attempt to integrate both efficient training and generation into a single backend, e.g. Deepspeed-Chat's Hybrid Engine (Yao et al., 2023). But specific generation libraries, like vLLM, are known to be "substantially better" and lead to large performance gains (Hu al., 2024).
25

---

Published as a conference paper at ICLR 2025
400
HF transformers
Generation Time (seconds)
vLLM
300
200
100
410M IB
2.8B Num Parameters
6.7B
Figure 14: vLLM is much faster than HF transformers Comparing the time to generate 128 to- kens from a batch 512 examples of prompt length 512 tokens each. Scaling model sizes from Pythia 410m 6.7B, we that vLLM is not just faster at each model scale, the difference is exponential. becomes infeasible to generate from large models using a training library like Hug- gingFace transformers
C.2 INTEGRATING GENERATION INTO SYNCHRONOUS RLHF TRAINING IS DIFFICULT
Since generation libraries are so much more efficient, an intelligent approach to synchronous RLHF must integrate the generation libraries into itself. For a best-case scenario, consider the arguable state-of-the-art synchronous RLHF library, NeMo-Aligner (Shen et al., 2024).
For NeMo-Aligners PPO, it combines efficient training backend, Megatron-LM (Shoeybi al., 2020), with an efficient generation backend, TensorRT-LLE (NVIDIA, 2024b). In order to leverage both of these, Shen al. (2024) implement clever but complex system convert training models to the generation backend on the fly. Although this feat is done impressively quickly, it still comes with downsides
Reduced Available Memory, Slower Training Building the TensorRT-LLM engine is expensive, so better to build it once and keep GPU memory. Therefore the training run has less available memory to use. So training must be done with gradient checkpointing to reduce memory usage backprop, this makes training slower.
Dynamic Model Resharding, More Overhead in Generation Training done using pipeline parallism to reduce memory but comes the increased cost of overhead communication. In contrast, inference could leverage tensor parallelism must therefore be re-sharded (which takes time) before converting from training to inference or suffer increased communication overhead. In both cases, there is increased overhead for generation.
reduce overhead. If there is enough space, models
C.3 MAINTAINING GENERATION IN SYNCHRONOUS RLHF VERY DIFFICULT
Despite these hurdles, NeMo-Aligne is quite performant for now. The issue is that there are con- tinual updates to both the training backend, Megatron-LM, and the generation backend, TensorRT- LLM. As case study, we look how NeMo-Aligner is maintained its underlying libraries change.
NeMo-Ale AlignAl was originally built with TensorRT-LLM version 0.11 as its generation backend. By the time of its release on September 8, 2024 TensorRT-LLM had already upgraded to version 0.12 and included new, necessary features like support for the SOTA open-source model LLaMA 3.1 (Llama Team, 2024).
The maintainers of NlMonAligner began working to integrate TensorRT-LLM 0.12 into their library (Kong, 2024) but they were working it, TensorRT-LLMM 0.13 released. They quickly
26

---

Published as a conference paper at ICLR 2025
adapted the PR and after one and half months of work, they integrated TensorRT-LLM 0.13 into NeMgeAligner. The same week, TensorRT-LLM released 0.14.
Each version of the library brought important speed and feature developments such LLaMA 3.1 support (0.12), KV cache reuse for LoRA (0.13), and fast logits copying (0.14) as well updating the underlying TensorRT library and fixing important bugs. Despite NeMo-Aligns and TensorRT- LLM both being developed by NVIDIA, it was still infeasible for the Memsor team to quickly integrate updates the generation library.
Generation libraries are generally built as stand-alone libraries (Kwon et al., 2023). Synchronous RLHF must integrate new developments and manually work around any new paradigms, breaking changes, and force those libraries to cooperate in their training paradigm. This makes it infeasible keep up with the latest developments. contrast, asynchronous RLHF can use those libraries as stand-alone processes that run parallel to training and integrating new updates is mostly frictionless.
C.4 SYNCHRONOUS RLHF IS ALREADY PARTIALLY ASYNCHRONOUS
Although state-of-the-art synchronous RLHF uses the same GPUs for generating and training the policy, it may still leverages asynchronous reward critic models. NeMo-Aligner's (Shen et al., 2024) PPO training has to leverage four models
PPO policy (for training and generation)
reference policy (for KL divergence loss)
PPO critic (to compute value estimates)
reward model (to provide reward for completions)
Using Py'rriton (NVIDIA, 2024a), the policy and reference policy are on one set of GPUs, but the critic and reward model are actually placed completely separate set of GPUs. The two servers (policy and critic/reward model) run and communicate asynchronously to permit pipelining (Shen al., 2024).
This pipeline can suffer from the same resource allocation issues as noted in $4 so Shen et al. (2024) suggest reserving compute allocation sizes such that [reward model inference critic inference] R [policy generation + reference policy inference] and [critic train] < [policy train + policy inference initialization].
Therefore, synchronous training libraries may already be partially set up to handle asynchronous training. fully asynchronous NeMo-Aligner would have create third PyTriton server with just the policy for generation and perhaps add another restriction to the compute allocation sizes, relatively minimal change.
D ASYNCHRONOUS ALGORITHM
Algorithm Cleanba-style (Huang et al., 2023) Asynchronous RLHF Initialize: base model reward model R, dataset D, RLHF Loss L (e.g. PPO, Online DPO)
Generate first batch of completions yo Te, Te, (zo)
for batch of prompts x; E do
send previous prompts 2;-1 and completions ØK-1 TRAIN send current parameters 0; and new prompts o; to GENERATE asynchronously TRAIN and GENERATE below
procedure OFF-POLICY TRAIN(I;-1: 1
procedure GENERATE(2,, 8,)
reward samples r;-1 R( 1, 11)) loss 1 <- 11
update generation model 0 <- 0g
generate new samples ~ To, (x:))
off-policy update 0,41 V
27


Reviews:
[{"review_id": "fwxr09O9JM", "reviewer": "Reviewer_dsGt", "summary": "The paper propose a asynchronous approach for accelerating the training of RLHF when multiple GPUs are available. Specifically, they propose to do generation and training simultaneously by letting policy learn on off-policy data. Experiments on TLDR dataset shows that the algorithm can acclerate the training by about 40%. Additionally, the paper provides some empirical results on the effect of how the off-policy data may affect the policy learning.", "questions": "Please refer to my weakness part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The asynchronous implementation enables model to do generation and training simultaneously and reduces the time for waiting reward signal. Experiments in finetuning Llama 3.1 shows a 40% acceleration in terms of reward convergence.", "weaknesses": "* The study on the effect of using off-policy data for updating is not very convincing. Based on the protocol in the paper (line 200-207), the policy seems to be trained on the data generated by the *same* outdated policy, which may be the key that results in performance drop. Specifically, the data generated by the same policy may be homogeneous (think about an extreme case where the policy generates $N$ same data pairs). In this case, the model only needs to learn a small update to adapt on these data, resulting a small KL divergence. A better experimental design is to train model using data generated by *different model* (that is $N$ steps behind) for each update.\n\n* Given different computational facility and tasks, it is not trivial to design an optimal resource allocation scheme for generation and training, which may require heavy manual tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fqcld2PGBy", "reviewer": "Reviewer_6pCS", "summary": "This paper discusses the possibility to achieve faster and more efficient RLHF training of LLMs by exploiting online but off-policy data. The authors propose to optimize compute efficiency by separating the generation and training tasks in RLHF on different sets of GPUs and executing them asynchronously. However, this requires training on off-policy samples. This paper further explores properties of asynchronous off-policy RLHF on TLDR Summarization benchmark. The results show that the online DPO method is the most robust to off-policy data, and robustness increases with the scale of the policy model. Moreover, some optimization methods targeting generation-bound and training-bound scenarios are also proposed to further increase the efficiency of asynchronous RLHF. Finally, the authors train a LLaMA 3.1 8B model on an instruction following task with online DPO, showing that the asynchronous method is 40% faster than its synchronous counterpart.", "questions": "Please respond to the questions in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-organized and the ideas could be easily understood.\n2. The asynchronous training in RLHF of LLMs is an relatively unexplored topic compared to synchronous methods. This paper has made a first attempt to research in this area.\n3. The experiments on the robustness of different RLHF algorithms to off-policy data is abundant. It could be inspiring to researches in related topics.", "weaknesses": "1. For asynchronous RLHF, it is unclear to me where the computation efficiency increase comes from. Compared to synchronous RLHF, does asynchronous RLHF reduce communication overheads or GPU idle time? Or does it decrease the overall computation required to achieve the same goal? It would be much more easier for readers to understand the performance gain if the authors could provide a breakdown analysis in the experiments. \n2. From the first paragraph in Section 3, it seems that a major obstacle that prevents the synchronous RLHF to be efficient is that there are no efficient implementation of generation in existing training libraries. From my perspective, this is only an issue that could be solved by engineering efforts. If this issue is solved, what is the advantage of asynchronous RLHF? Alternatively, could you explain why it is infeasible to make generation efficient in the training libraries? \n3. The authors have also mentioned in the footnote that there are a more advanced approach to interleave training and generation [1]. However, besides the paged memory in vLLM (which will reserve a large portion of GPU memory for KV cache), there are also many optimizations that could be applied to the generation (such as CUDAGraph, speculative decoding, continuous batching, etc.) in RLHF without harming the performance of training tasks, making it much more faster than the huggingface transformers implementation. If these optimization methods are applied, what is the advantage of asynchronous RLHF compared to the approach in [1]?\n4. In section 5, the experiment result only shows the asynchronous version of online DPO compared to the synchronized version of online DPO. However, it is still unclear how the asynchronized online DPO performs compared to other synchronized algorithms with the same amount of computational resources. Could you provide cases in which the asynchronized DPO outperforms the synchronized version of another RLHF algorithm (e.g. PPO)? This would make the advantage of asynchronous RLHF much more convincing.\n5. The benchmark used in this paper is TLDR Summarization, which is relatively simple compared to problems such as coding and math.  As an extension to problem 4, will the asynchronized online DPO perform better than the synchronized PPO in these harder problems? \n\n[1] Mei, Z., Fu, W., Li, K., Wang, G., Zhang, H., & Wu, Y. (2024). ReaLHF: Optimized RLHF Training for Large Language Models through Parameter Reallocation. https://arxiv.org/abs/2406.14088", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XiFea5aWBx", "reviewer": "Reviewer_7CQN", "summary": "This paper explores the possibility of asynchronous RLHF training for LLMs. Specifically, the generation and training steps, which constitute the majority of RLHF's processing time, are conducted asynchronously on separate GPUs. This approach naturally introduces staleness or so-called \"off-policyness,\" meaning the training samples are generated by earlier versions of the LLM rather than the most recent one.\n\nThe authors find that off-policyness can significantly impact the performance of PPO across a wide range of model sizes. In contrast, DPO demonstrates greater resilience to off-policyness. Accordingly, the authors have built a system that employs online asynchronous DPO for RLHF, which can achieve 20% faster training than the synchronous approach in wall-clock time while delivering comparable or superior performance.\n\nCurrently, I don't find any critical flaws in this paper and would recommend for acceptance.", "questions": "+ How can we determine when the system is generation-bound or training-bound? Since generation length may vary during training, the system could be generation-bound at the start but training-bound later. The optimizations in Section 4 require selecting the paradigm in advance, which may not be feasible.\n\n+ It was previously observed that off-policyness negatively affects learning performance in the OpenAI Five paper [1]. A dedicated section explaining this relationship could add clarity.\n\n[1] Dota 2 with Large Scale Deep Reinforcement Learning, Figure 5, https://arxiv.org/pdf/1912.06680", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper presents a novel perspective on off-policyness in RLHF. I especially appreciate the experiments comparing various RLHF loss functions and scaling behaviors related to off-policyness in Section 3. These experiments not only enrich this paper but also validate concurrent works on applying DPO to off-policy data and advocating for on-policy PPO training.\n\n* The paper is well-organized and easy to follow.", "weaknesses": "While I don't have significant concerns that would prevent a positive recommendation, it would be helpful if the authors could clarify the following points to avoid potential misunderstandings.\n\n## Motivation\n\nThe motivation presented in the first paragraph of Section 3 suggests that training libraries are slow for generation, and using separate libraries for training and generation could lead to GPU underutilization. This appears to be more of an implementation issue than a scientific one, which may not fully justify the exploration of off-policy RLHF. If RLHF systems can indeed achieve sufficient speed, investigating off-policy algorithms may not be necessary. For similar reasons, I have concerns regarding the experiments in Section 3.5.\n\n## Experiments\n\nThe title \"large-scale asynchronous RLHF\" seems overstated. LLaMA-8b is not generally considered large-scale. A more accurate title could be \"Asynchronous RLHF with LLaMA-8B.\"\n\nFigure 9 should also include a comparison with online PPO and in more benchmark tasks. It's good to use TL;DR for analysis in Sec.3, but more diverse evaluation is required for the final experiment. The authors can consider benchmarks like SafeRLHF or MATH.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iD0FOIN2Ln", "reviewer": "Reviewer_rLXT", "summary": "This paper studies incorporating off-policy asynchronous training to speed up RLHF. The main idea is to use separate GPUs for generation and training, which can leverage the acceleration techniques developed for both training and inference. Experiment results show that DPO is more robust under off-policy data, and the proposed method can achieve 40% speed up in training LLaMA 3.1 8B without sacrificing final performance.", "questions": "Nothing necessary stands out.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper studies an important problem: how to speed up RLHF training. The proposed method is easy to implement and the experiment results are quite promising. I think it is quite meaningful to study such off-policy asynchronous training setting, as fully online training is quite expensive and leads to low utilization of GPUs. This paper gives a thorough study of such setting with meaning conclusions.", "weaknesses": "1. There are many other RLHF algorithms except PPO and DPO, such as IPO, KTO, etc. It would be great to see experiments on these algorithms.\n2. It would be better to perform experiments on various types of dataset to further verify the results of this paper, as different types of dataset might have distinct properties. For example, it would be interesting to see the results on reasoning dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)