PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
Authors: Alexandre Piché
Ehsan Kamalloo
Rafael Pardinas
Xiaoyin Chen
Dzmitry Bahdanau
Venue: Accepted by TMLR ()

Abstract: Reinforcement Learning (RL) is increasingly utilized to enhance the reasoning capabilities of Large Language Models (LLMs). However, effectively scaling these RL methods presents significant challenges, primarily due to the difficulty in maintaining high AI accelerator utilization without generating stale, off-policy data that harms common RL algorithms. This paper introduces PipelineRL, an approach designed to achieve a superior trade-off between hardware efficiency and data on-policyness for LLM training. PipelineRL employs concurrent asynchronous data generation and model training, distinguished by the novel in-flight weight updates. This mechanism allows the LLM generation engine to receive updated model weights with minimal interruption during the generation of token sequences, thereby maximizing both the accelerator utilization and the freshness of training data. Experiments conducted on long-form reasoning tasks using 128 H100 GPUs demonstrate that PipelineRL achieves approximately $\sim 2x$ faster learning compared to conventional RL baselines while maintaining highly on-policy training data. A scalable and modular open-source implementation of PipelineRL is also released as a key contribution.

Published in Transactions on Machine Learning Research (03/2026)
Pip Faster On-policy Reinforcement Learning for Long Sequence Generation
Alexandre Piché ServiceNow AT Research'
alezendrrppcche@@mall.om
Ehsan Kamalloo ServiceNow AT Research
chsan. waw.eaaalloooeeeereeeeo........
Rafael Pardinas ServiceNow AT Research
rafael, taanedpeeeeaaas@eeesccccooo....
Xiaoyin Chen
siaoyin.cheen@mmir queeee quebec
Mila, Unéversité Montréal
Dzmitry Bahdanau
ServiceNow AI Research"
Mila, McGill Universily
Canada CIFAR Al Chair
Reviewed on OpenReview: https: //tp://oppppe openreview. net/ forum? id= A35ak140yp
Abstract
Reinforcement Learning (RL) is increasingly utilized to enhance the reasoning capabilities of Targe Language Models (LLMs). However, effectively scaling these RL methods presents significant challenges, primarily due to the difficulty in maintaining high Al accelerator utilization without generating stale, off-policy data that harms common RL algorithms. This paper introduces PipelineRI., tween hardware efficiency and data on-policyness for LLM training. PipelineRL employs concurrent asynchronous data generation and model training, distinguished by the novel in-flight weight updotes. This mechanism allows the LLM generation engine to receive up- dated model weights with minimal interruption during the generation of token sequences, thereby maximizing both the accelerator utilization and the freshness training data. Ex- periments conducted on long-form reasoning tasks using 128 H100 GPUs demonstrate that PipelineRL achieves approximately 2r faster learning compared conventional RL base- lines while maintaining highly on-policy training data. A scalable and modular open-source implementation of PipelineRL is also released as key contribution.
approach designed to achieve a superior trade-off be-
Introduction
Reinforcement Learning (RL) has recently bocome a popular tool to enhance the reasoning and agentic capabilities of Large Language Models (LLMs) (Guo et al. 2025 Wei et al. 2025 While RL expands the range of training signals one can use to enhance LLMs, this advanced learning paradigm comes with extra challenges, including being particularly hard to effectively scale arises from the fact that AI accelerators (like GPUs and TPUs) deliver high throughput only when generating sequences at large batch size. Hence, naively adding more accelerators to an on-policy RL setup brings increasingly diminishing learning speed improvements because the per-accelerator throughput decreases, while the overall generation latency reaches a plateau. The common workaround of generating training data
compute. The scaling difficulty
* Currently affliited with another institution.

---

Published in Transactions on Machine Learning Research (03/2026)
inference batch size decreases
constant batch size inference
Optim step
Optim step
GPUs
GPUs
weights through time
Optim
Optim
Optim
Optim
step
step
step
step
GPU idle
Time
Time
a) Conventional RL.
b) Pipeline RL with inflight weight updates.
Figure 1: a) Conventional RL alternates between using all the GPUs for generation and then training, b) PipelineRL runs generation and training concurrently, always using the freshest model weights for generations thanks to the in-flight weight updates.
for multiple optimizer steps results in & lag between the currently trained policy and the behavior policy that generates the training data. The lagging off-policy data is known to harm the commonly used effective RI. algorithms (Noukhovitch et al. 2024). including, REINFORCE, (Williams) 1992), PPO (Schulman al. 2017) and GRPO (Shao with on-policy near on-policy data, with the behavior and current policy being very close.
2024 Guo al. 2025). because these algorithms were designed be trained
In this paper, we present the PipelineRI. approach to RL for LLMs that achieves better trade-off between hardware utilization and on-policy learning. Like prior work on efficient RL (Espeholt et al.) 2018 2019) PipelineRL. features concurrent. asynchronous data generation and training. PipelineRL, adapts prior asy- chronous RL ideas to long-sqquene generation with LLMs by introducing in-flight weight updates. As shown in Figure D during an in-flight weight update the LLM generation engine only bricfly pauses receive the model weights via high-speed inter-accelerator network, and then proceeds to continue the generation of in in-progress token sequences. In-flight updates eliminate the wasteful waits for the last sequence to finish, ensure high accelerator utilization at a constant generation batch size, and maximize the policy adherence of the recently generated tokens.
Our experiments on RL training for long-form reasoning show that on DGX-H100 nodes, PipelineRI. learns 2x faster than the comparable conventional RL baseline. We also observe that PipelineRL training data stays highly on-policy, and that models trained by PipelineRL perform comparably similarly trained models from the literature. Lastly, key contribution of this work is scalable and modular PipelineRT. implementation that relcase open-source software.
2 Background
2.1
Reinforcement Learning for Large Language Models
Reinforcement learning (RL) is commonly used to train Large Language Models (LLM) to respect human preferences (Ouyang et al. 2022) for the LLM's outputs or to perform long-form reasoming to solve prob- lems (Guo et al.) 2025). One can view LLM's weights as parameterizing a multi-stop policy that assigns probabilities the next loken yi given the prompt æ and the previously generated tokens yez:
mlu a vey)))
(1)
Recent works have shown that variations of basic policy gradient algorithms such as REINFORCE (Willamms
1992) are as effective for training LLMs as more sophisticated alternatives (Ahmadian al. 2024 Roux
al. 2025 Given set of prompts 31,... Eny REINFORCE, maximizes the expected return J(m of the
2

---

Published in Transactions on Machine Learning Research (03/2026)
policy by following an estimate (T) of the policy gradient VJ(T)):
J(T)
E Bywst R(z
(2)
m
VJ(T)
(1g)) V Vlog 2,) v)]
(3)
m
j=1
VJ(T) = m
(R(z),8)) vol U).s:)) V log x (Uza 1 Ms,cs),
(4)
where R(z,.y) v) is the reward and (o(2,, Ve.se) is a value function learned by minimizing (R(z,, us (1(3,,055)))
In most practical RL sctups, the current policy æ will often differ from the behavior policy μ that generates yks due to the weights lagging, quantization or implementation difference between the inference and training softwares. This difference usually handled by either trust region constraint (Schulman et al. 2017) using Importance Sampling (IS). In practice, the importance sampling weights are truncated to reduce the variance of the estimator (Munos et al. 2016 Espeholt et al. 2018):
T min
[Ml xz) plyk xz))
Jis(TT)
(R(ar. U;) vo( Vi.s:)) V (Uja [yy (j.<x)
(5)
The Effective Sample Size (ESS) (Kong 1992 is commonly used to quantify the quality of importance sampling estimators in RL (Schlegel et al., 2019 Fakoor et al.) 2020). When using off off-policy RL, ESS measures how many samples from the current policy æ would yield equivalent performance to weighted samples from the behavior policy μ.. The (normalized) ESS is defined as:
ESS
Σ
N
(6)
where wi are importance weights for sample of size N. This metric effectively ranges between 0 and 1 when normalized, with values closer to indicating more efficient sampling, e-g. the ESS of on-policy data is exactly 1. Small ESS will result in high variance REINFORCE gradient estimate and might destabilize the learning process.
2.2 Conventional RL
Most RL implementations alternate between generating sequences and training the policy on the generated data. We refer to this approach as Conventional RL and describe it in detail in Algorithm [ When training involves doing G > 1 optimizer steps, the current policy æ gets ahead of the behavior policy j that was used to generate the data. We adopt the term log to refer to the number of optimizer steps between and æ.
2.3 Efficient Sequence Generation with LLMs
Transformer models generate sequences one token at a time, left-to-right. To make this process efficient, advanced generation (inference) engines such as vLLM and SGLang process a batch sequences at a time, while carefully managing their past keys and values a paged structure called KV cache (Kwon et al. 2023b). All modern generation engines support adding new generation requests in-flight to the ones progress without stopping the generation process. Based accelerator specifications, generation engines should achieve the maximum generation throughput at very large batch sizes of several thousand sequences In practice, at very large batch sizes, the per-sequence latency can become prohibitively high, KV cache may grow too large to fit. in accelerator memory, or the request queue management overheads can dominate.
httpss:/docs... nvidia oraea//ceecccteeeaeeetaoteeteeee
3

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm Conventional RL
Require: Current policy и.
Require: Optimizer state apt_state.
Require: Number of optimizer steps per RI. step G.
Require: Training batch size B.
while True do
generation
RL step starts
1
Initialize behavior policy μ
sequences + generale BG sequences from μ
batches t split sequences in G batches of size B
// training
lag 0
b lag between μ and æ
for batch in batches do
T, opt state + optimizer step(a, opt. state, batch)
lag lag
end for end while
RL step ends
338
per Second
Wall-clock (5)
500
Throughput (x10"
tokes)
6900
Tim Rnte
400
3000
300
200
100
256
512
1024
2048
100 200
256
generation starts
Number sequences per
(a) Throughput vs batch size.
(b) Inference batch size vs time.
(c) Time vs Throughput.
Figure 2: Analysis of generation times and throughput. We perform all measurements using vLLM engine serving Qwen 2.5 7B model on H100 GPU. (a) Short prompt generation throughput increases up batch size 256. (b) Generation batch size gradually decreases to suboptimal values as the engine finishes soquences (c) Generation time reaches platcan and throughput decreasos the number of soquences per GPU goes down. We report the average of 5 runs and 95% CI.
3 The learning speed ceiling of Conventional RL
Reinforcement learning for LLMs can be slow when the LLM is trained to generate long sequences of tokens, e.g., long-form reasoning to solve mathematical problems, because each generalion can take up to several minutes. Here we explain why it is challenging to effectively scale up long sequence RL, ie. to effectively use a larger number of accelerators N to make average reward R(t) at time t grow faster. As a mathematical function, one can view R(t) as a composition of the functions R(S) and S(1), where is the number of samples the RL learner will have processed by time A faster RL learner will have a higher learning speed AR which we can express as the product of learning effectiveness and learning throughput as follows: At
AR
AR AS
AS AL
AL
speed
effoctiveness
throughput
The Conventional RL algorithm from Algorithm 1 has the highest A3 when it is fully on-policy, i.e., when one performs only one optimizer step per each RL step. Yet the throughput A: can be low because the accelerators will be working at most batch size B samples at time. Increasing the number of accelerators N will yield diminishing returns in increasing AS because the throughput of each accelerator will decrease when the number of samples per accelerator
AR AS in the pure on-policy case
A7
N goes below the optimal range
8
4

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 2 PipelineRL: Actor and Trainer Processes
Require: Current policy weights и.
Require:
Generation batch size H.
Require:
Training sequence queue Qtrain-
Actor Process:
function ACTOR
sequences in progress Sprog <-
while True do
Sfin: Sprag + pop finished sequences from Sprog
5:
Qtrain-puth put(ppun yti))
>> Send finished seqs to the trainer
6:
if len(Sprog < H) then
7:
add H - (Spoo)) prompts to Sprog end if
if Trainer requests weight update then f srrelvewweiht__ugee
In-flight check for new weights
10:
11: end if
12: Sprog < generate next tokens with f
13:
end while
14:
end function
Trainer Process:
15:
function TRAINER(T, opt state)
16: batch + [
17:
while True do
18:
request actor weight update(T)
b In-flight weight update
19:
batch + get B sequences from Qtrain т, opt__sateee_tttt state + optimizer_step(T, opt state, batch)
20:
21:
end while
22:
end function
(Figure 2ch. For example, see Figure 2a for inference throughput for a 7B Qwen model on a single H100 GPU. One can see that the throughput. increases almost linearly up to the generation batch size of 128. Hence, e.g, using 2NV GPUs to generate 32 samples will not be much faster than using N GPUs to generate 64. Furthermore, the LLM finishes the shorter generations, there will be fewer longer generations still in progress, Figure 2b for illustration. Hence, to make good use of the hardware, one should use each accelerator to generate many times more sequences than the optimal batch size.
Commonly, to increase the throughput, most practitioners perform multiple G> optimizer steps per RL step, which entails generating BG rollouts at each generation stage. This way, one can often achieve a higher throughput At
N bocomes too small. is, however, known from the literature that going too off-policy by using high value of G will eventually decrease the learning effectiveness AR (Noukhovitch et al. 2024). Clearly, AS
AS by incrcasing N up to point when BG
points, the rollouts from the old policy become too stale and longer useful as the source of learning signal for the current policy, Hence, given fixed optimizer batch size B, one scales Conventional RL by increasing G and N until the product AR AS longer improves, and the hard ceiling of AR for the given number of accelerators N is achieved.
A$ At no
4 Pushing the learning speed ceiling with PipelineRL
The Pipeline RL method differs from Conventional RL in two aspects: (1) running training and generation in parallel asynchronously, and (2) updating the generation weights after every optimizer step fnghttth i.c. without stopping the sequence generation. Algorithm g provides an abstracted formal description of PipelineRL in terms of two concurrent Actor and Trainer processes that communicate via a sample queue and high-handwidth weight. transfer network.
5

---

Published in Transactions on Machine Learning Research (03/2026)
Optim step
Token lag
a) Conventional RL G
b) PipelineRL N accelerators
c) PipelineRL. 2N accelerators
Figure 3: For Conventional RL, the token lag increases with the number of optimizer steps. In PipelineRL with N accelerators, the token lag varies throughout the sequence, where earlier tokens have higher lag. The lag structure in each batch is the same. Doubling the PipelineRL accelerators, everything else constant, double the lag of early tokens.
The penetiveness-througnne trade-off for PipelineRL is the opposite of that of Conventional RL. Namely, adding more accelerators to a PipelineRI. setup leads to a linear increase of AS but may eventually harm Ar
AR In Figure 3 we illustrate how PipelineRL produces mized-policy sequences in which earlier tokens AS
are more off-policy than the recent ones. Doubling N will double the lag of the earliest tokens as well as the average lag in the PipelineRL batch. Notably, the ofl-policysess profile is different for PipelineRL and its conventional counterpart. Taking the average token lag as a proxy for off-policyness, in PipelineRL all batches are equally off-policy, whereas for Conventional RT later batches become progressively more off- policy. This difference makes hard to analytically reason about the At improvement that PipelineRL can bring over the baseline, because AR 25 can only be estimated empirically by running RL experiments. supplementary material, we present our simulation how, for the same maximum lag Imaz PipelineRL can learn 1.5x faster than Conventional RI. The empirical gains can be even larger, depending on how frequently one can make weight updates without hurting the learning effectiveness AR AS
Configuring PipelineRL vs Conventional RL For fixed batch size B and a mumber of accelerators N, one can configure Conventional RI. by choosing the number of optimizer steps G, trading off the learning effectiveness for the throughput. The PipelineRL configuration can likewise be mostly reduced to a single parameter, namely the number training accelerators T out of N available ones. Setting higher will almost linearly decrease the time trrain that is needed for the trainer to process B sequences and perform optimizer step. T effectively determines the optimal generation batch size H to be used at all N - T accelerators. Using a lower H leads to lower maximum generation latency (gen, which consequently reduces the maximum lag Imas = (ftgen/teress/ Hence, it makes sense to use the smallest H that suffices to produce enough training data. Consoquently, the maximum lag Imaz for PipelineRL grows with the number of training accelerators T, as higher T requires a higher H and leads to a lower ttrain and a higher tyen- On the contrary, the sample throughput of PipelineRI, grows with T up to point when N T accelerators cannot generate enough data for the over-powered trainer. We recommend avoiding extreme configurations with T too high (very high lag G) and T too low (bad hardware utilization, can just as well scale down the compute).
Architecture and Implementation Details Our PipelineRL implementation concurrentyy runs many distributed vLI.M generation engines and DeepSpeed training workers in thrce stage pipeline that we de- scribe in Figure ! The middle Preprocessor stage that we omitted from Algorithm 2 for simplicity, computes reference model log-probabilities often used in Reinforcement Learning from Human Feedback (Ouyang al.

---

Published in Transactions on Machine Learning Research (03/2026)
POST /request wweguesttweighttteeeooeo update
Weight update
inifiniband Streaming
FOST pit.pprocessggoop
HTTP
/data/0/o
Inference
Inference weights
Ref weights Ref LLMO
Ref LUM!
trainer
Artor LiMO
Actir LLM1
Weights
Stream
POST vidchapoamppctoon
Process
actor
/actor/O/0
preprocess
/data/O/T
Figure 4: The three pipeline stages of PipelineRI implementation: actor, preprocessor and trainer. Earlier stages stream the data to the latter ones using Redis as the st reamingg broker.
0.00
10
0.48
0.40
0.40
Samples (1e6)
Rewerd
Reward
0.55
0.05
0.8
0.50
0.30
0.75
0.25
Pl
PipelineRl ConFiL
0.2
PipeiinRR
0.20
ComFR G-8
0.00
G-8
8.5
ConFRL D-52
Com/L G-32
DonvRL G-42
200 Time (minute)
Samples (168)
Time (minutes)
(a) Reward vs time.
(b) Reward vs samples.
(c) Samples vs time.
Figure 5: (a) PipelineRL attains the same average reward faster than the conventional RL baselines. (b) PipelineRL achieves the same sample efficiency faster than the conventional RL G 32 basclinc.
and G 16. (c) PipelineRT, generates samples 2x
2022 The PipelineRL architecture is highly modular any generation software that supports the three HTTP APT endpoints that PipelineRI requires can be casily integrated in the future. The three APIs are the popular /vt/chat/completionn for generation, 'init_procees___ _gropp for creating the weight transfer process group, and trequest_weight,... weight_update update for inttin the in-flight weight update. Key optimizations in PipelineRL include online sequence packing for fast training and using ring buffers to minimize the lag when earlier pipeline stages rum faster than the later ones, c.g. when the trainer makes chockpoint.
5 Experiments
AR For the experimental validation of PipelineRL's high learning effectiveness as and throughput A7 we have chosen the challenging task of training a base (i.e. not instruction-tunedt model to perform long-form reasoning solve mathematical problems. We find this task be a great testbed for PipelineRI, because the policy undergoes rapid changes over the course of training. particular, the length of generated sequences grows dramatically (Guo et al. 2025). making it essential to stay on-policy for effective learning.
AS

---

Published in Transactions on Machine Learning Research (03/2026)
PipelineRL
ConvRL
ConvRL G=16
ConvRL G=32
00
Max:
80
10
0.8
0.2
0.8
0.8
0.2
0.6
0.0
0.8 Samples
Samples
wd
Samples
Samples
146
(a) Max lag.
PipelineRL
ConvRL G=8
ConvRL G=16
ConvRL G=32
1.000
Effecive Sample Size (ESS)
0.85
0.900
0.0
0.2
0.4
0.8
0.8
1.0
0.0
02
0.4
0.6
0.8
1.0
0.0
02
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Samples
Samples
Samples
Samples
(b) Effective sample size.
PipelineRL. attains higher max lag that every conventional RL method, but obscrved in Figure ib the Effective Sample Size is similar to G=8. This indicates that while the max lag is quite high, PipelineRL stays mostly on-policy as measured by the ESS.
Figure 6: In Figure 6a
Experimental setup. For each experiment, we train the Qwen 2.5 base model (Yang et al. 2024) with 7B parameters on 17K math problems from the OpenRcasoner Zero dataset (Hu et al. 2025 for 1024 optimizer steps with the batch size 1024 and maximum sequence length of 32k. We use Adam optimizer (Kingma 2014) with the learning rate 1e-6. We run the PipelineRL experiments on 16 DGX-H100 nodes, using 48 GPUs for generation at batch size H 64 and 80 GPUs for training. We tweak PipelineRL to simulate Conventional RL by accumulating and shuflling buller of BG samples at the Preprocessor stage before the G optimizer steps of each RT. step start. To estimate the Conventional RL throughput, we use 2 nodes for generation at batch size H = 64 and nodes for training, and then add correction for training on 8x fewer GPUs than what an efficient Conventional RL implementation with a quick inneeration-traiiin transition could use. To estimate the inference throughput on 128 GPUS instead of 16 GPUs, we submit. 128 batches of 16x 1024xX and lake the maximum completion time. We give reward to completed sequence with the correct 128
16
answer and 0 otherwise. We also give a soft penalty to the model when it produces between 30k and 32k tokens. We train every model with importance weighted REINFORCE, as described in Section and clamp the importance weights to For our experiment we use vLLM (Kwon al. 2023a to generate trajectories and use DeepSpeed (Rasley et al. 2020) through accelerate to train the model. We did not recompute the KV cache after each il-flight updates as it did significantly improved on-policyness, see Figure 7
PipelineRL learns faster due to higher throughput. We compare the learning speed of PipelineRL to that of Conventional RL with G = 32 optimizer steps, that was the maximum G for which Conventional RL training was stable. PipelineRL achieves the same reward values approximately ~ 2r faster than this baseline (Figure 5a) due to ~ 2x faster sample throughput (Figure 5ch. The main cause of the throughput increase is that GPU utilization for = 32 experiment on 128 GPUs is relatively low for each GPU when it has to generate just divergence, Figure 10
(024/128-22 256 sequences (soe Figure pb Further increasing G results in
AR PipelineRL learns effectively. To better measure learning effectiveness As of PipelineRL, we also run Conventional RT experiments with G 8, G 16, and G 32 optimizer steps. Notably, the R(S) curves

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Success rate of models trained with PipelineRL compared to results in the literature.
Method
Math 500
AIME24
# samples (-10%)
training data
Qwen 2.5 base 7b
31.6
3.3
SimpleRL Zero
78.2
20.0
0.82
Math Level 3-5
Zeng et al. 2025
OpenReasoner Zero
82.0
20.0
8.2
OpenReasoner
Hu al. 2025)
PipelineRL (batch size 1024)
81
17.5
2.0
OpenReasoner OpenReasoner
PipelincRI. (batch size 4096)
84.6
19.8
6.2
1--3
1e-3
1.5
1.0
2
0.5
0.0
30
10
20
30
Lag (g)
Lag (g)
Lag (g)
a) Starting checkpoint C=0
b) Starting checkpoint C=100
o) Starting checkpoint C=190
Conventional
PipelineRL with KV cache recomputed
PipelineRL
Figure 7: For three different starting checkpoints, PipelineRL with and without KV cache recomputation stay more on-policy than Conventional RI. as measured by the KL. divergence.
are indistinguishable for all compared methods up a point where G 32 is slower and unstable, likely because of going too far off-policy. This result validates that PipelineRL.s Iss signature in-flight weight updates do no harm to the sequence generation process.
PipelineRL matches comparable results on reasoning tasks. Table compares the test performance of PipelineRL to similar experiments that start training from the same Qwen 2.5 7B model. In this exper- iment we used batch size 4096 because found it leads to higher performance. On the math reasoning benchmarks MATH500 [Hendrycks et al., 2021) and AIME2024 (Li et al. 2024). PipelineRL matches exceeds the success rate of Open Reasoner Zero and SimpleRL. Zero.
PipelineRL stays more on-policy. To gain a better understanding of which training methods stay more on-policy, we plot the evolution of the max lag and the ESS on-policyness measure throughout the training. Figure shows that PipelineRL obtains a higher max lag than the conventional RL baselines. Notably some tokens have lag of more than 50k samples. However Figure GD shows that, in terms of ESS, PipelineRI, maintains similar on-policyness as G = 8. We further observe that the ESS of G = 16 and in particular
G = 32 drops throughout training.
5.1 Impact of in-flight weight updates on on-policyness
In this section, we compare the sampling distribution of in-flight weight updates to 1) conventional RL with different max lag and 2) in-flight weight update with KV cache recomputation. For this experiment, we save set of consecutive checkpoints C, one after every optimizer step. To replicate the in-flight weight update, start from checkpoint and update the weights of the behavior policy every L tokens with the
Rmax
9

---

Published in Transactions on Machine Learning Research (03/2026)
subsequent checkpoint, where L is the maximum sequence length and Imax is the maximum lag. Specifically, the PipelineRI. behavior policy is defined as:
14 == (C( PC+. (Ctgg)---..ttt
Fid1y-. Ftg-1:t9)
(8)
I. Lokens for lag g > 1 since the first weight update takes longer than the next updates due to the bubble at the beginning of training, see Figure [b). b). We also use & to stress that the KV cache for the previous tokens is stale as it was computed under previous model weights. We then compute the KT. between the mixed behavior policy PC:C+g and the on-policy behavior policy JAC+9- We also report the KL with the mixed behavior policy with updated KV cache which we denote as Pipelinefil with KV cache recomputed. To replicate conventional RL, we sample N sequences from the behavior policy μ == = PC and compute the KL with on- on-policy behavior policy μC+9 for different lag k.
where 4 = 21, and ig = 1 Smaax
Smax
In this experiment,
fine-tune Qwen 2.5 basc 7B on the OpenReasoner Zero (Hu et al. 2025 data for 222 optimizer steps. We consider three stages in training to measure KI-ddvergeece:: starting at checkpoint 0, 100, and 190. The maximum lag Jmax is set to 32 and the maximum sequence length L is 2048. As presented in Figure 7 the distribution of mixed-policy sequences closcly aligns with that of fully on-policy sequences across all three stages in the training. In contrast, off-policy sequences exhibit consistently higher divergences as lag increases. Also, using stale KV-cache for mixed policy sequences introduces only slightly higher divergence compared to recomputing the cache. This supports our design choice in Pipeline-Ro RL to opt for the more efficient approach of retaining the KV cache.
6 Related work
Asynchronous and tightthroughput RL has been extensively studied. IMPALA (Espeholt al. 2018 decoupled acting from learning to maximize GPU utilization. Like PipelineRL, IMPALA used truncated importance weights to estimate the value function from off-policy samples. Furthermore, IMPALA kept the policy weights constant. for the length of an episode. SeedRL (Espeholt et al. 2019) proposed to update the model's parameters during an episode, resulting in trajectories where different actions were sampled by different policies. OpenAl Five (OpenAI et al.) 2019) was trained using asynchronous PPO to achiove superhuman performance on Dota 2. These previous works were focused on RL for video games and learns from partial rollouts. PipelineRT. only learns from completed rollouts. Closer to our work, (Noukhovitch et al. 2024) explores asynchronous RL for LLMs. In their approach, data generation for the next G optimizer steps is synchronized with training the previous G optimizer steps, leading to higher off off-policyness than Conventional RL, unlike PipelineRI. The same study shows that offline methods such as DPO [Rafailov 2023 can better tolerate off-policyness.
ct al.
There exist several other scalable open-source RL implementations. veRL (Sheng et al. 2024 implements
Conventional RL efficiently by using a sophisticated hybrid generation-trainnng engine that supports quick transitions between training and generation on the same GPUs We believe vcRL's 's throughput would be
similar to our Conventional RL baseline. Without the hybrid engine, in OpenRLHF (Hu et. al. 2024) training GPUs idle during generation and vice-versa. Concurrently, Magistral(Mistrall et al 2025 introduced in-flight weight updates.
also
7 Conclusion and Discussion
We have shown how in-flight weight updates help PipelineRL break the learning speed ceiling of the con- ventional two-stage RL approach. We believe that for long sequence generation, in particular, this speedup would be very difficult to attain with another asynchronous RL approach, as synchronous waits for genera- tion to finish would hurt the throughput and/or learning effectiveness. The stale KV-cache risk that in-flight updates introduce can be mitigated by recomputing the KV cache after each update, which can be done fast at high GPU utilization, but will still lower the throughput.
We believe PipelineRL may be particular useful for training LLMs to excel at agentic behaviors that involve multiple LIM generations interspersed with environment. interactions. Another promising direction for
10

---

Published in Transactions on Machine Learning Research (03/2026)
future work is to study when the recent low lag tokens in PipelineRL are helpful, and the contrary, where PipelineRL's constantly high lag of early tokens in long sequences hurts. Furthermore, there might exist better RL algorithms when combined with PipelineRL than traditional algorithms such as REINFORCE with truncated importance sampling.
Limitations PipelineRL will only bring a limited throughput increase over Conventional RL the LLM is asked to generate the exact same number of tokens for the same prompt. In this unlikely scenario, Conventional RL will be likewise capable of maintaining a constant generation batch size. The PipelineRL's stable average token lag and the low lag of recent tokens in each batch may, however, still affect the lcarning effectiveness. The PipelineRL throughput advantages will likewise decrease in setups with scarce or extensive compute resources, In the former case, each GPU will get enough generation tasks for the GPU utilization to be high. In the latter, the learning speed will be bounded not by the hardware utilization but by the best possible generation latency and by the environment feedback delay.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arXiv preprint orXiw:2002.147. 14740, 2024.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Lasse Espcholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. SEED RL: Scalable and efficient deep-RL with accelerated central inference. arXiv preprinl arXiv:1910.0650 2019.
Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P30: Policy-on policy-off policy optimization. In Uncertainty in artificial intelligence, pp. 1017-1027. PMLR, 2020.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeck-R1. Incentivizing reasoning capability in LLMs via reinforcement learning. Xiv preprint arXiv:2501.129/ 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2108. 03874, 2021.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. OpenRLHF: An Easy-to- use, Scalable and Tigh-performance RLHF Framework, November 2024. URL http://arxiv. org abs/ 2405. 11143 arXiv:2405.1 111444 [cs].
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open- Reasoner-Zero: An open source approach to scaling up reinforcement learning the base model. arXiv preprint arX:t22503.24900 2025.
Diederik P Kingma. Adam: method for stochastic optimization. arXio preprint orXiv:1412.600 2014.
Augustine Kong. A note on importance sampling using standardized weights. Universily of Chicago, Depl. of Statistics, Tech. Rep, 348:14, 1992.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ton Stoica. Efficientt memory management for large language model serving with pagedattention, 2023a. URL http::://arxiv.or//aab/////// 06180
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with Ateenttttntion, September 2023b. URL
arXiv:2309.06144 [os).
11

---

Published in Transactions on Machine Learning Research (03/2026)
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. NuminaMath: The largest public dataset in AT4Maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Mistral-AL, ;, Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaune Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Léonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Sagar Vaze, Teven Le Scao, Yihan Wang, Adam Yang, Alexander H. Liu, Alexandre Sablayrolles, Amélie Héliou, Amélie Mar- tin, Andy Ehrenberg, Anmol Agarwal, Antoine Roux, Arthur Darcet, Arthur Mensch, Baptiste Bout, Baptiste Rozière, Bandouin De Monicault, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Clémence Lanfranchi, Darius Dabert, Devon Mizelle, Diego de las Casas, Elliot Chane-Sane, Emilien Fugier, Emma Bou Hanna, Gauthier Delerce, Gauthier Guinet, Georgii Novikov, Guillaune Martin, Hi- manshu Jaju, Jan Ludziejewski, Amar, Josselin Somerville Roberts, Julien Denize, Karan Saxena, Kush Jain, Lingxiao Zhao, Louis Mar- tin, Luyu Gao, Lélio Renard Lavaud, Marie Pellat, Mathilde Guillaumin, Mathis Felardos, Maximilian Augustin, Mickaël Seznee, Nikhil Raghuraman, Olivier Duchemne, Patricia Wang, Patrick von Platen, Pa- tryk Saffer, Paul Jacob, Paul Wambergue, Paula Kurylowiez, Pavankumar Reddy Muddireddy, Philomène Chagniot, Pierre Stock, Pravesh Agrawal, Romain Sauvestre, Rémi Delacourt, Sanchit Gandhi, Sandeop Subramanian, Shashwat Dalal, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Thibault Schueller, Thibaut Lavril, Thomas Robert, Thomas Wang, Témothée Lacroix, Va- leriia Nemychnikova, Victor Paltz, Virgile Richard, Wen-Ding Li, William Marshall, Xuanyu Zhang, and Yunhao Tang. Magistral, 2025. URL https://arxiv org/ abs 10910
Chabran, Jean-Malo Delignon, Joachim Studnia, Jonas
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off off-policy rein- forcement learning. Advances in neural information processing systems, 29, 2016.
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous RLHF: Faster and more efficient off-policy RL for language models. arXiv preprint arXiv:24 18252, 2024.
OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Cather- ine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang, Dota 2 with large scale deep reinforcement learning, 2019. URL battps://arzne //tp///axvvv/.////////////.aa. org/ awtpaabs....0/ 06680
Long Ouyang, Jelfrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information. Processing Systems, (1:53728-5374)) 2023.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD, pp. 3505 3506, 2020.
URL https:///oi. org/10 100.111///34446.3406666
Nicolas Le Roux, Mare G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Fric Thibodeau-Laufer, Sándor Toth, and Sam Work. Tapered off-policy REINFORCE: Stable and efficient reinforcement learning for LLMs. arXiv preprint 2xki::2503.12866, 2025.
Matthew Schlegel, Wesley Chung, Daniel Graves, Jian Qian, and Martha White. Importance resampling for off-policy prediction. Advances in Newral Information Processing Systems, 32, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. ar Xiv preprint ar Xive Xix::107000000000 707.06347, 2017.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeckMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXim:2402. (000:240,003300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv preprint arXiv:2409. 19256, 2024.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida Wang. SWE-RL: Advancing LLM reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18444, 2025.
Ronald J Williams. Simple statist.ica gradient-following algorithms for connectionist reinforcement learning.
Machine learning, 8:229- 256, 1992.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. ar preprint arXiv:242. 12.15115 2024.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL -oo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint 11Xi::250.11889, 2025.
A Analyical estimate of PipelineRL speedup for fixed max lag
In this additional section we estimate how much faster PipelineRL can be compared to Conventional RL for the same value of maximum token lag Imaz. We will be using the following notation, mostly the same as in the main text:
N the number of accelerators
S BG is the number of sequences that are processed in each Conventional RL step
is the maximum and L is the average sequence length for the current policy
K SL is total number of tokens that Conventional RL processes each optimizer step
We will additionally use U(h) to refer to the accelerator's 's maximum flops utilization when running typical Transformer kernels at batch size h.
A.1 Units
To compare throughputs of diffcrent RL approaches useful to adopt time and throughput units that don't depend on the particular GPU model and the LLM size. To this end we introduce a time unit called flash:
Fgen M
(9)
where Fgen is the number of FLOPss required for one token forward pass for the chosen LLM, and M is the maximum theoretical FLOPs throughput for the given GPU. The meaning of flash is the theoretically smallest amortized time that a token generation can take. Thus generating K tokens will take at least K flashes, though at more typical generation utilization of ~ 0.1 rate it will take 10K flashes. For very long sequences F gen can vary significantly duc to attention FLOPs becoming a large part of total FLOPs, but for simplicity here we will abstract away from this detail.
Having introduced flash f as the unit, we will measure the system throughput in tokens per flash.
Let be the amortized training time per token. will be similar at scale for PipelineRL and Conventional RL, because both approaches benefit from sequence packing.
13

---

Published in Transactions on Machine Learning Research (03/2026)
GPU utilization as function of the batch size h
0.8
0.7
0.6
0.5
U(h)
0.4
0.3
0.2
0.1
0.0
200
400
600
800
1000
Figure 8: H100 utilization at batch size h as the ratio of maximum theoretical bf16 FLOPS throughput. We use (4096, h) (h, 16384) matrix multiplications for the measurement. For every we consider padding up to + 64 to increase the speed, because empirically we observed large utilization bumps when divisible by a higher power of (up to 128).
A.2 Conventional RL throughput
We can express Conventional RL throughput as follows:
K
Teonv
Ceane + ttran con't
(10)
where tgen and foom are times that generation and training take respectively. Let's look at these terms closer:
k()/NF U(K(I/N)
Conn
(11)
KT
ttrain
(12)
cone
N
where h(1) is the number of sequences still in progress after 1 steps of decoding, and U(h) is the GPU utilization at batch size h. To understand Equation [11 recall that generating k tokens by definitions takes k flashes under perfect GPU utilization and kkU(k) at the utilization Uk)).
We can rewrite this in terms tokens flash throughputs:
T Teonu
(13)
Toons
K
y g-ge comm
(14)
RUYN
U(h()/N)
.trrin conn
(15)
14

---

Published in Transactions on Machine Learning Research (03/2026)
RL throughput vs max lag mmax 128 GPUs and batch size 128
Conventional RL N=128
Pipeline RL N=128
20
RL throughput (tokens/flash)
15
1:57x faster
10
5
50
100
150
200
250
Lag gmax
Figure Pipeline RL and Conventional RL throughputs as the function of the maximum lag Jmar for a setup with N 128 GPUs and batch size B N.
At low batch size per GPU at step I, hx(() = h(1)/N, the ratio (x(O)U((ww(() will only decrease very # slowly as a function of N, bocause for modern GPUs U(x) nearly constant for small æ. This is the formal explanation for Conventional RL's decreasing efficiency as N grows.
The maximum token lag in the setup we described above S-1.
A.3 PipelineRL throughput
For PipelineRL the system throughput is determined by the slowest pipeline stage. Using the concepts introduced above, the throughput of PipclineRL can be estimated follows:
ppipeline = min pipelsne T train
gen
(16)
pipeline)
gen pipeline
U(H)T N -
(17)
train pipeline T
(18)
To understand the maximum lag of Pipeline RL consider the fact. that the generation GPUs will produce HIL tokens during the time it takes to generate the longest possible sequence of length L. On average there
will be HIL sequences in these tokens. Thus, in the worst case when an optimizer step happened just before L
the longest sequence generation started, long sequence will be used for training Imar HIL optimizer T.B
steps later than its generation started.
To build same-lag equivalent for conventional RL system, needs to maximize Tpipeline (H,1) while keeping 1111 < S 1. We found this problem difficult to solve analytically, and performed a straight- forward search of all (H, configurations for our investigations below.
15

---

Published in Transactions on Machine Learning Research (03/2026)
A.4 A PipelineRL speedup case study
To compute the exact throughput boost that PipelineRL. brings it is necessary to make assumplions about the sequence length distribution and the hardware that is used for the experiments. For the case-study below, we assume uniform length distribution from to the max length and H100 as the GPU. We visualizo the GPU utilization table (h) in Figure S The reader can see that (h) grows almost linearly up to h ~ 200, which makes possible compress the generation on fewer GPUs at higher utilization. For setup with N 128 GPUs and training batch B 128 considered all possible (I, (I) configurations of PipelineRL and plotted their throughput 1.57x faster for Imaz 133. This lag value can to be too high for many practical setups, but with higher batch size of c.g. B = 2048 the same number of sequences to be generated by cach GPU will correspond to practical 16x lower lag ymaz ~ 8.
function of the lag Imaz- Figure shows that PipelineRL can be up to
The mechanics of how PipelineRL achieved the improvement are as follows:
gen pipeline
16.9, p prain pipeliie
17.08, Ppipeline 16.9, H = 192, T= 44
cone 18.3, I cone
,.train
26.02, cconv 10.7
Clearly, the root cause of RipelineRL's speedup is that the 44 generation GPUs produce 16.9 tokens per flash, that is more efficient than having 128 GPUs produce 18.3 tokens per flash in the Conventional RL case.
B Additional Results
ConvRL G-64
1.0
0.5
0.8
Success Rate
ESS
0.4
0.1
0.2
0.0
0.0
Com/LL G+64
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Samples
te6
Samples
1e6
(a) G=64 reward.
(b) G=64 Effective Sample Size.
Figure 10: G=64 diverges.
16


Reviews:
[{"review_id": "fXRXZIaqJA", "reviewer": "Reviewer_snwq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces PipelineRL, a novel training pipeline designed to speed up on-policy reinforcement learning for LLMs, especially in long sequence generation tasks. Specifically, the key idea is to concurrently generate data and train the model in an asynchronous manner, enabled by a novel \"in-flight\" weight update mechanism. That is, the generation process only pauses briefly to receive updated model weights and then continues generating tokens, which maximizes GPU utilization while keeping the training data nearly on-policy. In addition, the authors also provide empirical demonstration of substantial speedup. Particularly, on a mathematical reasoning benchmark using a 7B LLM and deployed over 128 GPUs, the method achieves roughly 2 $\\times$ faster learning to reach comparable performance relative to a standard two-stage RL baseline. Moreover, the authors show that despite asynchronous updates, the data remains sufficiently on-policy and final model performance on benchmarks remains competitive.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide  empirical evidence to support their main claims. Regarding the efficiency, PipelineRL consistently learns faster than the baseline. For example, on a long-form mathematical reasoning benchmark, PipelineRL reaches a given reward level roughly twice as fast as a standard synchronous RL setup, which is shown in their Figure 5.\nAdditionally, regarding the on-policyness and training stability of PipelineRL, they measure metrics like the effective sample size and KL divergence to quantify how \"on-policy\" the data remains. The results show that despite the asynchronous updates, which introduce some lag in using slightly older weights for part of a sequence, PipelineRL's data distribution stays nearly as on-policy as a tightly synchronous RL with minimal lag, which is supported by Figure 6.", "audience_yn": "Yes", "audience_explanation": "There is a growing interest in training LLMs with RL (e.g., RLHF and other goal-driven fine-tuning of LLMs), as well as a general interest in techniques for scaling up machine learning algorithms efficiently. This work sits at the intersection of those areas by addressing how to make on-policy RL feasible and faster on cutting-edge hardware for very long sequence generation. Researchers and practitioners working on large-scale RL, distributed training, or LLM optimization will likely be interested in this work.", "requested_changes": "1.  The experiments of this paper focus on a single task domain (math reasoning task) and a specific model architecture and scale. Including another type of long-sequence generation (e.g., long-form text generation, dialogue, or code generation) and other choices of model would strengthen generality claims.\n\n2. The paper introduces the risk of stale key-value caches when weights are updated during sequence generation and notes that recomputing the cache can mitigate this at some cost. It would help to add a bit more detail on how this was handled in practice. For example, did the authors find it necessary to recompute the cache periodically, or did training work well without any recomputation? Was there any observable impact of stale caches on the generated tokens or rewards? \n\n3. Minor point: Caption in Figure 7: \"checkpoint\" ---> \"checkpoints\"", "broader_impact_concerns": "The work does not raise any obvious ethical or societal concerns. It is focused on a training methodology to improve efficiency, with no new application or data that might be sensitive.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7P0RoTjS2v", "reviewer": "Reviewer_oihW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the significant challenge of effectively scaling RL methods used to enhance the reasoning capabilities of LLMs. The core contribution is PipelineRL, an approach designed to achieve a superior trade-off between hardware efficiency (accelerator utilization) and the freshness of training data (on-policy).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The contribution is largely about system design rather than novelty. This type of asynchronous RL design has been explored in Deep RL (Distributed Asynchronous Actor-Critic A3C [1]) by DeepMind around 2019. However, verifying and validating that the same principle works, and releasing the system design blueprint **publicly** is a good contribution, and can help speed up the effort in the open-source community.\n\nAlso, it's worth noting that RL advancements are often closely tied to advances in system design. Even though the paper focuses on system design, it remains of great interest to the RL community.\n\n[1] Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., Van Hasselt, H., & Silver, D. (2018). Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933.", "audience_yn": "Yes", "audience_explanation": "Scaling up RL training for LLM is an important topic, and many RL people would be interested in knowing the findings.", "requested_changes": "Some of my questions are about making this paper more scientific instead of pure engineering:\n- Not enough analysis/results were around the trade-off between synchronization frequency (in-flight weights update) and final policy performance -- how should we set these hyperparameters? If I missed such an analysis, please refer to the sections in the paper or plots.\n- Figure 3 is very interesting in highlighting the difference between Pipeline RL scaling and synchronized RL scaling. However, there isn't any empirical exploration on how the scaling happens. Can you comment on how Pipeline RL can scale up?\n- It's a little unclear what a good future direction should be. You don't have to do them, but any ideas on what other researchers can explore?", "broader_impact_concerns": "There isn't a significant safety risk beyond the risk of doing RL/AI research and how it might engender our society. No such statement is necessary for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tjb7PpK7s1", "reviewer": "Reviewer_pNXA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary of Contributions\n\nThis paper investigates the limitation of conventional RL when scaling up, due to insufficient hardware utilization. The authors present PipelineRL, which optimizes the generation-training pipeline from two aspects: (1) raining and generation run in parallel and asynchronously (2) update generation weights in-flight. Through Eq.(7), they analyze both conventional RL and PipelineRL from effectiveness and throughput. Experiments verify the advantage of the proposed method.\n\n## Strength\n\nThe paper is easy to follow. From the intuition level, the analysis in Section 3 and 4 make sense to me. The experiments verify the effectiveness of PipelineRL.\n\n## Weakness\n\n1. PipelineRL seems not a quite novel idea. At least to my knowledge, such a framework (or similar ones) has been widely applied in industry. There are also a few similar techniques in published literature. For example, \"partial rollout\" seems a very relevant technique. In partial rollout, once a training batch is collected, the unfinished queries are paused and only resumed after the weights update is finished.\n\n2. The analysis through effectiveness and throughput is interesting, but I expect it to be more rigorous. Throughout the paper, I did not find a formal statement to quantify $\\Delta R/\\Delta S$ or $\\Delta S/ \\Delta t$ (maybe by B,G,N, etc). To explain the advantage of PipelineRL, it would be better to formally show it has both higher effectiveness and throughput than conventional RL.\nFig3-(b) is just a schematic illustration, and it is totally unclear how well the curve and points in the figure aligns with the practice. It can cause some confusion.\n\n3. By Fig5-(b), it seems that Pipeline RL does not improve or decrease the sample efficiency, which seems a bit counter-intuitive. Pipeline RL and conventional RL should have different level of \"off-policiness\", which is the main factor to determine the sample efficiency. Although I'm not sure which one of them are more off-policy, there should be some divergence in the curve. It would be great if the authors can explain it.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Fig 3-(b) is not plotted with real data but just a schematic illustration. I think it is not very rigorous.\n\nComparing different methods from the perspective of effectivenss and throughput is interesting, but there is no formal quantification for those quantities. This makes their arguments not very convincing.", "audience_yn": "Yes", "audience_explanation": "The focus of the paper is to optimize the RL training process in LLM. LLM post-training is a popular topic, not only for people working on LLM but also for people more on the RL side.\n\nThough, I feel the contribution of this paper is a bit off-topic to the machine learning community, because the main improvement of the proposed PipelineRL, compared with conventional RL, is more on the implementation side rather than innovation on the algorithmic level.", "requested_changes": "My proposed adjustments are based on my concerns in weakness.\n\n1. I encourage the authors to do a deeper survey on existing relevant techniques and compare with them in related works. For example, I think partial rollout is relevant, but you can correct me if I'm wrong.\n\n2. Provide a formal quantification for $\\Delta R/\\Delta S$ or $\\Delta S/ \\Delta t$. Or formally prove and state the proposed PipelineRL has higher effectiveness or throughput than conventional RL (at least in some cases). Remove Fig 3-(b) or replace it with figure plotted with real data.\n\n3. Add explanation for why curves in Fig 5-(b) do not diverge. \n\n1 & 2 are critical to securing my recommendation, and 3 is not very important.", "broader_impact_concerns": "I do not have concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)