OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
Authors: Kaizhuo Yan
YingJie Yu
Yifan Yu
Haizhong Zheng
Fan Lai
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning from Human Feedback
Training Efficiency
Primary area: reinforcement learning

Abstract: Proximal Policy Optimization (PPO)-based reinforcement learning from human feedback (RLHF) is a widely adopted paradigm for aligning large language models (LLMs) with human preferences. However, its training pipeline suffers from substantial inefficiencies due to sequential multi-model dependencies (e.g., reward model depends on actor outputs) and long-tail response lengths, where a few long responses straggle the stage completion. We present OPPO, a novel, lightweight, and model-agnostic PPO-based RLHF framework that improves training efficiency by overlapping pipeline execution. OPPO introduces two novel techniques: (1) Intra-step overlap, which streams upstream model outputs (e.g., actor model) in right-sized chunks, enabling the downstream model (e.g., reward) to begin prefill while the upstream continues decoding; and (2) Inter-step overlap, which adaptively overcommits a few prompts and defers long generations to future steps, mitigating tail latency without discarding partial work. OPPO integrates easily with existing PPO implementations with a lightweight wrapper. Extensive evaluations show that OPPO accelerates PPO-based RLHF training by $1.8\times$--$2.8\times$ and improves GPU utilization by $1.4\times$--$2.1\times$ without compromising training convergence.

Published as a conference paper at ICLR 2026
OPPO: ACCELERATING PPO-BASED RLHF VIA PIPELINE OVERLAP
Kaizhuo Yan Yingjie Yu" Yifan Yu! Haizhong Zheng² Fan Lai' 'University of Illinois Urbana- Champagnn (kaizhuc2, yyu69, yifanyu4, fanlai]@i11in edu ha i 1ilzhhozzandrere cmu. edu
Carmegic Mellon University
+Equal contribution
ABSTRACT
Proximal Policy Optimization (PPO)- reinforeement learning from human feedback (RLHF) is a widely adopted paradigm for aligning large language models (LLMs) with human preferences. However, its training pipeline suffers from substantial inefficiencies due to sequential multi-model dependencies (e.g.. reward model depends on actor outputs) and long-tail response lengths, where a few long responses straggle the stage completion. We present OPPO, a novel, lightweight, and -ades-gg PPO-based RLHF framework that improves training efficiency by overlapping pipeline execution. OPPO introduces two novel techniques: (1) Intra-step overlap, which streams upstream model outputs (e.g.. actor model) in right-sized chunks, enabling the downstream model (e.g., reward) to begin prefill while the upstream continues decoding; and (2) Inter-step overlap, which adaptively overcommits a few prompts and defers long generations to future steps, mitigating tail latency without discarding partial work. OPPO integrates easily with existing PPO implementations with lightweight wrapper. Extensive evaluations show that OPPO accelerates PPO-based RLHF training by 1.8 x 2x288x and improves GPU utilization by 1.4x x without compromising training convergence.
INTRODUCTION
Reinforcement Learning from Human Feedback (RIHF) has become cornerstone for aligning large language models (L.LMs) with human preferences. Among RLHF methods, Proximal Policy Optimization (PPO) (Schulman et al., 2017) has been the de facto standard due to its training stability and flexibility across diverse reward models and objectives. Following InstructGPT (Ouyang et al.) 2022). PPO remains the standard for online alignment in both research and industry. Recent work shows it outperforms offline methods like DPO on reasoning tasks (Xu et al. 20244, and it supports massive-scale training in modern tool chains (Shen et al.) (024). A standard PPO-based RLHF pipeline involves four models: an actor (policy), a critic (value function), a reference policy (for KI. regularization), and reward model trained human-labeled preferences. Each training step consists of three sequential stages: (I) Generation: the actor generates responses to prompts: (2) Scoring: responses are evaluated by the critic, reference, and reward models; and (3) Training: the actor and critic models are updated using advantage estimates and gradients.
Despite its effectiveness, PPO-based RLHF faces significant training inefficiencies rooted in its multi-model dependencies. Running and coordinating four LLMs imposes substantial resource requirements, and each stage constrained by its slowest component. For example, the actor model's generation suffers from severe long-tail latency: few long responses can delay downstream stages, such as the reward and value models, leading idle resources and poor training throughput of the pipeline. As LLMs grow larger and context lengths increase, these bottlenecks worsen, (Grattafiori ct al. 2024) making PPO-based RLHF increasingly costly to train (.2.)
Recent advances tackle PPO-based RLHF inefficiencies from both algorithmic and system perspec- tives. On the algorithmic side, methods such as Direct Preference Optimization (DPO) (Rafalloy et al. 2024) and Group-Relae Relative Policy Optimization (GRPO) (Shao et al.) (024) remove compo- nents like the value or reward model. However, these approaches often suffer from instability due sparse rewards, requiring many rollouts to capture intrinsic advantages, and face task-specific

---

Published as a conference paper at ICLR 2026
Prompt
Gen Stage Decoding
Scoring Stage Prefilling
Streaming Gen
Pause Streaming Gen
Global setting: Batch size -4
step
step k +
==
Intra-step Overlap
Inter-stop Overlap
(a) Existing paradigm.
(b) OPPO paradigm.
(a) In the existing paradigm, the scoring stage begins only after the response is fully generated. In contrast, (b) the OPPO paradigm interleaves scoring with generation without altering the final responses (intra-step overlap), and carries unfinished overcommitted sequences into the next iteration (inter-step overlap). A batch size of 4 and an overcommitment degree of 2 in illustrations.
Figure 1:
reward design challenges (Feng et
2024 Fisch al., 2025 Chen et 2024 Asynchronous
RLHF (Noukhovitch et al. (025) reduces pipeline dependencies but introduces staleness, which can harm convergence ($2.2) In contrast, system-level approaches, such as RLHFuse (Zhong et 2025), AReal (Fu et al. 2025), and Verl (Sheng et al. (025), improve throughput via fine-graines parallelism, for example by collocating models to reduce communication overhead or dynamically scaling GPU resources to match workload demands.
In this paper, we explore a complementary opportunity to accelerate PPO-based RLHF: maximizing execution overlap in the training pipeline. We introduce two novel insights: (1) Intra-step Overlap, which streams tokens from the actor to downstream models, enabling generation and scoring stages to overlap the execution without altering the generated response; and (2) Inter-step overlap, which selective overcommits a few prompts per batch and prioritizes faster completions, deferring stragglers to future iterations hide tail latency without wasting partial generation.
Realizing both overlaps introduces non-trivial challenges. First, overlapping generation and scoring hide the prefilling latency of downstream models during the decoding execution of the actor model, but also increases resource contention, risking slowing generation due to concurrent executions. Second. excessive overcommitment inflates batch sizes, deferring too many responses per iteration. This not only raises per-batch latency but introduces staleness, ultimately harming convergence.
Contributions. this paper, we present OPPO, lightweight PPO-based RLHF training framework that improves training efficiency via pipeline overlap, minimizing idle time without compromising convergence. OPPO novelly addresses the aforementioned challenges:
Intra-step Overlap: While the actor generates responses, OPPO streams newly generated tokens to downstream models (e.g., reward model) in adaptive chunks. This enables incremental prefilling and overlaps the generation and scoring stages. Chunk sizes are automatically adjusted online, based on the leftover resources, to balance overlap against resource contention, preserving algorithm correctness and stability ($3.1).
Inter-step Overlap: To mitigate loig-tail latency, OPPO adaptively overcommits few prompts generations are deferred and resumed in future iterations, preserving partial work and maintaining batch size. It adapts the overcommitment level online, trading small statistical deviations (e.g., reward differences) for large throughput gains ($3.2).
per step.
Generalized and lightweight: Our evaluations show that OPPO achieves 1.8x-2.83 speedup and improves GPU utilization by 1.4x-2.1x for PPO-based RLHF with only lightweight wrapper, and generalizes to other paradigms such as DPO with similar benefits ($4)
BACKGROUND AND MOTIVATION
next outline the PPO-based RLHF framework
2. then highlight key inefficiencies in existing
training designs that motivate work ( 2.2
2

---

Published as a conference paper at ICLR 2026
100
A40
66"
Wann-up Mid-phasc
Staleness
GPU Uttlizaiio
A100
H200
Probubilit Density
5c-
Converged
Staleness 5
Staleness 10
4e"
60
3e-
40
2e-
-1-
20
le"
Reward
Training
6000
-2
0
100
200
300
Generation
Scoring
Dutput Length (Atokens)
Training Time Steps
(a) GPU util. varies across stages.
(b) Heterogeneous rollout lengths.
(c) Asynchrony hurts quality.
Figure 2: PPO-based RLHF faces (a) varying resource demands across pipeline stages, and (b)
heterogeneous rollout lengths, both of which can produce stragglers that prolong step execution. (c) Existing approaches for asynchronous training risk harming convergence.
2.1 BACKGROUND: PPO-BASED RLHF
Figure Ia depicts single step of a standard PPO-based RLHF pipeline. Given a batch of prompts, the actor model generates output sequences. These then scored by the reward model, producing scalar rewards that reflect alignment with human preferences. reference model, typically a frozen copy of the base pretrained model, computes KL divergence penalty that regularizes the update, discouraging the new policy from drifting too far from the original distribution.
The value model estimates the expected return of each sequence and computes its advantage AG:
8, = (V
A
Σ 2=
where r, the reward at step 1, (st) the estimated value of state 8t, 7 the discount factor, and is the generalized advantage estimation (GAE) parameter. In every step, actor model is updated by optimizing the clipped surrogate objective:
(0,) )- Et D8,-1 min At, clip (r,,),,, e, At)]
Tay (as 31)
where rı (0,)
_ (as St.)
These four models (actor, reward, reference, and value) form tightly coupled intra-step pipeline spanning generation, scoring, and training stages. PPO-based RLHF typically runs hundreds such iterative steps, creating inter-step pipeline across updates.
2.2 TRAINING INEFFICIENCY OF PPO-BASED RLHF
Unlike pre-training single model, PPO-based RLHF pipelines introduce two types of execution de- pendencies that limit hardware utilization and training speed: intra-step and inter-step dependencies.
Inefficiency due to Intra-step Dependency. Each model in the RLHF pipeline (e.g., actor, reward, and value models) exhibits distinct computational characteristics. commparsss GPU utilization across three GPU types (A40, A100, H200). Response generation in the actor model is memory- intensive due to autoregressive per-token decoding, resulting in low GPU utilization (440%), whereas scoring and training stages relatively compute-intensive (e.g., due long-context prefilling in scoring). This heterogeneous resource utilization highlights how mismatched compute demands across stages create idle GPU time, motivating designs to scavenge unused resources.
compares
The inefficiency is further amplified by the long-tailed tailed distribution of response lengths (Figure (b). While most sequences are short, subset of responses significantly longer. Since stage completion
3

---

Published as a conference paper at ICLR 2026
depends the last sequence (rollout), these heterogeneous lengths introduce tail stragglers. Worse, the length distribution evolves across stages (e.g., at the warm-up stage and converged stage), making optimizations such as dynamically resizing GPU allocations challenging.
Inefficiency due to Inter-step Dependency. Each step involves updating model weights, A natural strategy to improve throughput is tolerate training staleness, where the reward model evaluates actor outputs from previous steps instead of synchronizing in real time, as in AReal (Fu et al. (225)) However, as shown in Figure 2c] asynchronous training (e.g.. with staleness 5) can not only slow step-to-reward convergence but also lowers the post-training model quality, emphasizing the need for careful inter-step synchronization RLHF pipelines.
OPPO: OVERLAPPING PPO-BASED RLHF TRAINING PIPELINES
To address both intra- and inter-step inefficiencies, we introduce OPPO, an Overlapped PPO-based RLHF training paradigm. As illustrated in Figure [ OPPO overlaps the training stages reduce idle time and improve resource efficiency, tackling two key sources of step latency: sequential stage dependencies within step, and the long-tailed distribution of output lengths. At its core two complementary techniques: (1) intra-step overlap, which overlaps reward scoring with actor generation within single step, and (2) inter-step overlap, which selectively overcommits a few prompts and carries unfinished prompts into the next step to mitigate tail-induced stalling.
3.1 OVERLAPPING INTRA-STEP TRAINING PIPELINE
Sequential dependencies across pipeline stages and the long- distribution of response lengths often block downstream execution in PPO-based RLHE. For example, the reward model cannot begin the scoring of a sequence (rollout) until the actor completes generation for that sequence, leading to idle resources and underutilized GPUs. At the same time, heterogeneous resource utilization across models presents a new opportunity: while the upstream actor continues memory-intensive decoding, downstream operators (e.g., reward model) can start the (sub)prefilling of partial outputs in a streaming manner.
By dividing actor generation into chunks and streaming them to the reward model, OPPO overlaps the actor decoding stage with the reward prefilling stage, hiding latency and reducing execution bubbles. This design naturally benefits setups where models are placed separate GPUs, but also improves efficiency when models are colocated, due to their mismatched compute demands (Figure To realize intra-step overlap, OPPO partitions actor outputs into right-sized chunks and streams each chunk to the reward model as it is generated. Scoring proceeds progressively within each PPO step: while the actor decodes the k-th chunk, the reward model concurrently processes the prefilling of (k - 1)-th chunk. At the end of the step. the reward model completes prefilling for the final chunk and computes the score based on the entire sequence, whose previous chunks have been processed.
Importantly, this streaming does not alter the response generation Vt, the policy log prg-rrbbabilttes, or the critic/value terms used in computing the advantage A(y.). Formally, letting y; denote the full response and y ... (Ti) its prefixes with y/ (T:) yi. the streamed gradient estimator is
B
T
Bar (0) =
1
(i.1) in
A(y:) V log (p xz),
B
where (1.t) marks the final prefix. Because each sample follows exactly the same prefix, the inner (0) = (f) point-wise. Thus, intra-step streaming does not change the PPO sum collapses, and (
update, preserving both expectation and variance of the gradient estimator.
Dynamic Control on Intra-step Overlap. However, streaming introduces a tradeoff in chunk size. As shown in Figure 7b large chunks (e.g., 3K tokens) result in low overlap, reducing the benefits of intra-step streaming and reverting to baseline sequential execution. Conversely, small chunks (e.g., 10 tokens) can cause severe resouree contention, especially when models are colocated, due to frequent GPU context switching to execute different models. OPPO addresses this by exploiting two key insights: (1) the tradeoff between chunk size and overlap efficiency monotonic and predictable,
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 OPPO Training with Intra-step and Inter-step Overlap
Require: Batch size B, initial A, chunk size C, window size W, bounds Amin Amax
1: Initialize Buffer +- capoacta = + A); reward_scores +
2: for each training iteration do
Stage I: Fill buffer to capacity
3: while |Buffer] < B + A do
4: Buffer.add fromm (dtasett))t
5: end while
> Stage 2: Generation with intra-step overlap
6: finished
7: while [finished] < B do active Beffer.get_uuiiinn geL_uinnissed) if active = 0 then break
8:
9:
11:
end if
parallel do
13:
chunks +- Actor.generate caapttttt_uhunaiiieeeeeee size C)
14:
Reward.reward, httprdrrewar_iiinmmeeaaaaaleo (ncremental(activv))
b Finished -innishedd-preeill--ccoddd else else-tprefill.
15:
Siplate_stttssaaeee
chunks)
16:
end while
> Stage 3: PPO update with inter-step overlap
17:
ppo_batchs batch finished|: B] reward scores.appenddppo_battee
ppo
18:
19:
PPO.step(ppoobatte bbtch)
20:
Buffer.eemove(ppoo_balee bpttbp
Unfinished sequences remain for next iteration > Dynamiic A update
if [rwardd_ssooe scores] 2 2W then
22:
d 6 -eean(reward_aag. sereeaaddsscrss(-__ scores[-W :]) mean(rewardd sscores-211 -W]]
23:
Achange f- 1--max11, [4/4]
24:
A < - sign d) change Amins Amax)
25:
Buffer.set catpssstttccccty(BB_e A)
26:
reward scores = reward screisccorssssss
27:
end if
28:
end for
and (2) PPO training runs for many steps, allowing ample opportunities for exploration. Therefore, OPPO periodically (e.g., every 50 training steps) applies a few candidate chunk sizes (e.g., 128, 256, 512) across different steps and selects the best-performing configuration for subsequent windows.
3.2 OVERLAPPING INTER-STEP TRAINING PIPELINE
While intra-step overlap improves efficiency within a single PPO step, it does not fully address tail latency caused by the heterogeneous response lengths within batch. Here, the response must complete generation before reward scoring and subsequent policy updates. Due to the long-tailed distribution of generation lengths, few slow prompts can delay the entire step. This motivates inter-step design that allows overlapping across PPO steps without hurting convergence.
OPPO addresses this challenge by overcommitting few additional prompts per batch mitigate long-tail stragglers. Specifically, if the original batch size is B, OPPO executes B + A prompts per step. The key insight is that sequence generation is typically not computation-bouna so adding few extra prompts has minimal impact per-batch execution time while substantially reducing the effect long-tail sequences. During each step, the first completed prompts are used for PPO updates, while unfinished A sequences are deferred to the next step. This mechanism ensures that long sequences are not starved, finishing subsequent steps, and partial work (generation) preserved across steps.
The overall procedure, combining intra- and inter-step overlap, is summarized Algorithm I where the buffer holds up to B + A sequences, and generation proceeds in parallel with intra-step streaming. The threshold A, which controlling the number of unfinished sequences carried over to the next step, introduces tradeoff between efficiency and convergence. A small A reduces overlap and may leave GPUs idle due to tail sequences, while a large increases overlap but risks inflating per-step latency and introducing staleness in the PPO update.
5

---

Published as a conference paper at ICLR 2026
Dynamic Control Inter-step Overlap. OPPO automatically adjusts A based training
dynamics. Let R denote the average reward in step t, and consider a sliding window of w steps. Define the slope of improvement over the window as 8t A is then updated according to
1 to Ei=l-w1 ( R3-1). The threshold
min max: D + ôine)
st > 0,
A++1
max max (mins A. ōdse) if St S 0,
where oin and daz are fixed momentum (e.g., I), and Amin and maxx are bounds on the buffer size. As training starts to converge and 8; - O, A, naturally decays toward Amin (often zero), preventing overcomittement to ensure convergence while effectively mitigating tail-induced delays across steps.
4
EVALUATIONS
4.1 EXPERIMENTAL SETUP
experiments are conducted on high-end NVIDIA GPUs with different configurations. Stack- Exchange-Paire Paired with Qwen2.5-7B-Bl Inn-trrrttttt is on 8 X220 (141GB) GPUs, while GSM8K with Qwen2.5-7B runs on 4x (96GB) GPUs. Stack Brock--xchange-Pee Paired with Qwen2.5-3B --3-Ittenn and OpenCoder-SFT with Qwent..-33BIInttruct are executed on AxA10 (80GB) GPUs.
Models & Datasets. We follow state-of-the-an PPO settings using the Transformer Reinforcement Learning (TRL) library (von Werra et al.) 2020a). For actor models, we experiment with Qwen2.5- 7B, Ow ---tttntt and Qwen2.5 --re2.5--BB Instruct, each augmented with value head for PPO optimization. The reward model either a Qwen2.5-7B or a rule-based evaluator (for math tasks). We evaluate three popular tasks widely used in RLHF research (detailed evaluation setup in AppendixAA.Ij.
Free-form generation: Black-Ecchange-Paee Paired (von Werra et al. 2020b), which contains QA pairs with preference labels.
Math reasoning: GSMSK (Cobbe et al., 2021). which consists of grade-school math word problems. We convert it into preference format by ranking paired outputs by correctness and reasoning clarity.
Code generation: OpenCoder-SFT (Stage 2) (Huang et al. 2024 which contains large-scale programming tasks across multiple languages.
Baselines. We follow the standard distributed PPO setting. Based the memory and computation resource requirements of each model, we allocate seven GPUs to the generation and training stages, and one GPU the scoring stage (i.e., reward model). We compare OPPO against TRL's PPO (yon) Werra et al. 2020a), the state-of-the-art and widely adopted framework in PPO. It is worth noting that OPPO complementary existing PPO frameworks and be integrated into them. Unless otherwise specified, we use a training batch size of 112.
Metrics. We evaluate both efficiency and quality. Efficiency is measured by training speed, includ- ing time-to-reward and step-to-reward. Quality is measured by the final achieved reward. All results are averaged over five independent runs.
4.2 END-TO-END PERFORMANCE COMPARISON
We start by evaluating OPPO's end-to-end efficiency and quality performance.
OPPO achieves substantial PPO training speedup. Figure 3 shows that OPPO consistently accelerates PPO training by 1.8x-2.8x across all tasks. On Stack-Exchange with Qwen2.5-7B- Instruct, OPPO reaches a reward of 4.17 2,300 minutes versus 4,300 minutes for the baseline, yielding 1.9x speedup. With 1ree2.5-3B-IInstruee on the same dataset, OPPO achieves a reward of 5.12 in 5,200 minutes compared to 13,000 minutes, corresponding to a 2.5x improvement. These gains stem from two sources: (7) intra-step overlap, which hides reward prefilling latency during

---

Published as a conference paper at ICLR 2026
Qwwen2.5-7B-han Bennn.5s7Blluxiuuu LRY Inter
Qwanm 5-3B-Inxtruct
Qwee225511
Qwen 5-38--333800 (artrruc
1.0 Ioter
fatter
0.00
2.5
Rowards
0.75
1.5
0.20
1HL
-0.5
1000
2000
3000
4000
3000
6000
9000
12000
275
530
025
1100
450
900
1350
1800
Times (minutiax)
Figure 3: OPPO improves PPO-baser RLHF training efficiency by 1.8x over TRL across datasets, enabled by overlapping actor generation with reward scoring and early stopping.
Owenn2 5- 7B Instruct
Qwen2 5 S.ntttst
Qwen2 5-7B
Qwen2. 5-222-55----ntttd
Stack Euthange Parndd
Staci Sachasge Duired
GSMMK
3.5
Opencidda 53r (Suge
newards
22
0.75
1.5-
0.70
0.5-
0.65
-0.5
200
4.0
900
12000
100
120
Steps
Figure 4: OPPO achieves efficiency gains without affecting training quality.
actor decoding, and (ii) inter-step overlap with dynamic A, which mitigates tail stragglers that would otherwise block shorter generations. OPPO achieves 2.4x x and 2.8x speedup on OpenCoder-SFT (Stage 2) with Qwen2.5-3B- Inssuuttt and on GSM8K with Qwen2.5-7B, respectively.
OPPO preserves training convergence. Despite substantial wall-clock speedups, Figure shows that OPPO does not sacrifice training convergence. On Stack-Exchange, OPPO and the baseline follow nearly identical trajectories on both Qwen2.5-78 77B-Sttrtttt and Qwen2.5-3BB 1--B-IInstrue training, such as reaching a reward of ~-20 by step 150, then plateauing at <..1 by step 600 and 55.12 by step 1,000, respectively, On GSM8K with Qwen2.5-7B. both methods exhibit the same characteristic learning phases: an initial accuracy of 0.70, a dip to 0.66 around steps 25-50 as the model unlearns initial biases, and steady improvement to 0.82 by step 200. Finally, OpenCoder-SFF (Stage 2) with Swen2.5-3B-Innstuue tasks, the near-identical step-to-reward eurves confirm that OPPO achieves near-optimal balance between execution efficiency and convergence quality.
both methods converge to a plateau around 2.4 by step 80. Across all
OPPO largely boosts hardware resource utilization. Figure 5 shows that OPPO substantially improves GPU utilization. On the Baack-Exchanee-Piie Paired dataset with the Qwen2.5-7B-Insttenn model, utilization increases from 50.6% to 71.0%. 1.4x improvement. With the Qwen2.5-3B- Instruct model on the same dataset, utilization rises from 38.7% to 73.6%, a 1.9x improvement. On GSM8K with the Qwen2.5-7B model, OPPO boosts utilization from 45.7% to 67.3%, 1.5x improvement. On OpenCoder-SFT (Stage 2) with the Qwen2.5-3BB-Intteee improves from 35.7% to 74.1% %, corresponding 2. increase. Note that utilization does not reach 100% because of unavoidable parallelism bubbles, memory stalls, and communication overheads.
model, GPU utilization
OPPO improves performance in multi-node settings. Table shows that OPPO achieves 4.49 reduction end-to-end step latency than TRL on Blnck-Exchange-Pure Paireed with the Qwen2.5-7B --B-nn-ttttttt model across two nodes (each 4x A A100-40GB)
OPPO delivers improvements over different model parallelism plans. The distinct system-level benefits of OPPO are evaluated by comparing it against tatte-of-the-ar frameworks, including VeRL (configured with data parallelism (DP), sequence parallelism (SP), and fully async w/ SP) and AReaL. Table 4 shows OPPO achieves the lowest latency (99,84s), outperforming VeRL w/ DP by 1.26x and surpassing highly optimized systems such as AReaL and VeRL variants. These results suggest that OPPO targets latency source distinct from sequence-level optimizations. While frameworks like VeRL and AReaL process responses only after full generation and leave the reward model idle, OPPO 's intra-step overlap streams intermediate chunks to utilize this time. Consequently, OPPO
7

---

Published as a conference paper at ICLR 2026
Qwwen2.5 2rtnntrrtt
Qwenn2.5 BiBtttttt
Qwin2.
Owwn2
GPU Uttazation 05
96.8. 196
93.7 96.2
93.5
36,0
74.1
50.6
38.7
50-
45.7
35.7
Actor
Training
Actor
Training
Actor
Trainng
Actor
Training
Figure 5: OPPO improves GPU utilization in the inference stage by 1.4x-2.xx, enabling more efficient compute by overlapping actor generation with reward scoring.
TRL
OPPO
Mean latency (s)
498.30
111.08
Speed up
1,00x
4.49x
Table 1: OPPO achieves lower end-to-end step latency than TRL by 4.5x in multi-node settings.
addresses bottleneck orthogonal to sequence parallelism, making a complementary optimization composable with existing strategies.
4.3 ABLATION STUDIES
Performance Breakdown by Design Components. We ablate OPPO into two variants to isolate the impact of each design choice: (I) OPPO w/o Intra, which disables intra-step overlap (i.e., streaming upstream decoding to the reward model), and (2) OPPO w/o Inter, which disables inter-step overlap (i.e., batch overcommitment with dynamic A).
reports their performance on -eee-Exehnne-PPee baseline requires 4,200 minutes to reach a reward 4.17. Adding only intra-step overlap reduces this to 3,500 minutes (1.2x speedup), as streaming hides about 17% of scoring latency during generation. However, the gain is bounded by stragglers from the longest sequences in each batch. Applying only inter-step overlap reduces training time further to 2,700 minutes (1.6x speedup). For Qwen2.5-3B In-trutuut,, the TRL baseline requires 13,000 minutes to reach a reward of 5. Intra-step overlap reduces this 10,000 minutes (1.3x speedup), while inter-step overlap achieves 6,300 minutes (2.06x speedup). Again, all configurations converge to similar final rewards, confirming that intra- and inter-step overlaps address orthogonal bottlenecks while preserving training quality.
For Qwen2.5-7B -nsttttttB- the TRL
Robustness and Staleness. As detailed in Algorithm I the A controller adapts to windowed reward trend, updating A only through bounded, gradual steps. This design prevents abrupt jumps and effectively filters short-term oscillations. The request-deferral distribution in Table confirms the stability of this approach: the vast majority of requests are processed immediately, and nearly all deferred requests are delayed by only single step. This indicates neither perpetual deferral of difficult prompts nor excessive staleness that would affect rewards.
Effectiveness of Inter-step Adaptation. Figure 7a com- pares OPPO with fixed and dynamic A. With fixed A 4, training converges more slowly since fewer long-tail generations are stopped early, limiting overlap benefits. Fixed A accelerates convergence by skip- ping more long-tail tail generations, but its static threshold cannot adapt well across all phases of training. In contrast, dynamic A
300
250
Rowards
Stap Spred
L.50
100
OPPO OPPO w/ diees &
OPPO died 5
15-
1050
1400
100
500
3 1.00
2000
3000
(a) Inter-step adaptation (A).
(b) Impact chunk size.
Figure 7: Ablation studies on efficiency: (a) fixed vs. dynamic A. and (b) chunk size effect on step speed.
8

---

Published as a conference paper at ICLR 2026
Owen2.5-7B-Instte
Qwen2.553B-Innse I-nnsnsuttt
Stack--ochwng--Phiiee
Stack-Etrhongi- Pairee
5-
Rewards
3-
4-
2
1-
OPPO
TRLL
2-
1-
OPPO w/o Intra
OPPO wio Inter
0-
1000
2000
3000
4000
-10
3000
6000
9000
12000
Times (minutes)
Figure 6: Performance breakdown showing the impact of OPPO's s intra- and inter-step overlaps. Both
optimizations drive the 1.8x speedup without harming convergence quality.
Deferred steps
2
Avg. deferred steps
Share of requests
78.48%
20.20%
0.23%
1.05%
0.24
Distribution of requests deferral shows most requests are not deferred, and nearly all others Table 2:
are delayed by only a single step.
consistently achieves the best performance by adapting the threshold over time, leading to faster convergence and more stable rewards. These results underscore that static choices of A create efficiency-stabiliy tradcoffs, whereas dynamic A eliminates this tension by adjusting to the evolving distribution rollout lengths throughout training.
Impact of Streaming Degrees. Figure 7b shows the effect of chunk size step speed across different model sizes. Small chunks (100 tokens) incur high scheduling and GPU context-switch overhead, reducing throughput despite high overlap. Moderate chunks (500 tokens) strike the best balance, yielding the fastest step speeds by maximizing overlap while avoiding overhead. Large chunks (1000-3000 tokens) reduce overlap opportunities and push execution closer to sequential mode, causing step times to rise again. These results highlight that throughput is highly sensitive to chunk size, and the optimal setting depends on model scale and workload.
OPPO Preserves Final Accuracy. Table 3 shows that for the 3B model, OPPO consistently outperforms the TRL baseline across all benchmarks, with gains ranging from 0.07 to 0.92 percentage points (mean: 0.48 pp). For the 7B model, the differences are minimal (-0.24 to +0.25 pp; mean: +0.02 pp): OPPO achieves higher accuracy on ARC-Challenge, HellaSwag, and GSM8K, while showing slight declines ARC-Easy and TruthfulQA-MC2. These fluctuations fall within the expected statistical variance of RLHF training. Overall, the comparable performance across both model scales confirms that OPPO's pipeline-overlap strategy accelerates training without sacrificing model quality.
Applicability beyond PPO. OPPO's benefits extend any online preference-optimizzaie method involving variable-length on-policy generations (e.g., DPO or GRPO). These methods can adopt the same scheduling logic: generate B+A items, update on the first B completions, and carry unfinished long generations forward to the next iteration. This strategy reduces tail latency without altering the optimization objective or the distribution of responses used for updates. As shown in Figure 3 OPPO achieves 2.35x faster convergence in rule-based PPO setting on GSM8K (without reward model), confirming that the inter-step overlap mechanism remains effective even in ntn-standard or simplified RLHF pipelines.
RELATED WORK
PPO-based RLHF Efficiency. Hydra-PPO (Santacroce et al. 2023) reduces memory and latency by combining LoRA with parameter sharing across actor, critic, and reward models. Offline PPO methods (Hu et al. 2023 Noukhovitch et al. 2025) improve stability and efficiency by training from fixed preference datasets, thereby avoiding costly online rollouts. Data-centric approaches such as LIMO (Ye et al. 2025) and S1 (Muennighoff al. 2025) demonstrate that small, curated datasets

---

Published as a conference paper at ICLR 2026
Qwen2.5-3B Model
Qwen2.5-7B Model
Tasks
TRL
OPPO
Change
TRL
OPPO
Change
ARC-Challenge
48.89
49.57
+0.68
55.55
55.80
+0.25
ARC-Easy
74.54
75.08
+0,54
81.57
81.36
-0.21
HcllaSwag
75.01
75.19
+0.18
80.70
80.79
+0.09
TruthfulQA MC2
59.07
59.99
40.92
64.27
64.03
-0.24
GSM8K
63.46
63.53
+0.07
82.56
82.79
+0.23
Average
64.19
64.67
+0.48
72.93
72.95
+0.02
Table Evaluation results on core tasks (0-shot) and math tasks (5-shot). We report accuracy (%) for TRL-trained models and OPPO-trained models, along with the absolute change.
VeRL
VeRL w/ DP+SP
AReaL. OPPO
Mean latency
125.36
120.47
109.92
99.84
Table 4: OPPO achieves the lowest per-step latency under identical hardware and rollout settings, suggesting system-level benefits beyond VeRL (DP, DP+SP, Fully Asyne w/ SP) and AReaL.
prioritizes samples using impact- baaed scoring, while ADARFT (Shi et al., 2025) adopts a lightweight curriculum that adjusts difficulty through reward signals. These methods primarily optimize data or optimization strategy, whereas our work focuses on improving system-level efficiency by restructuring PPO's 's execution pipeline.
yield competitive performance. LIMR (ii al.
2025)
Model Training Efficiency. System-level techniques seek accelerate RLHF training by rethink- ing the execution stack. TRL (von Werra et al.) 2020a) provides scalable multi-node training with
parameter-efficient fine-tuning. OpenRLHF (Hu et al., (025b) integrates vLLM (Kwon et al. 2023 with Ray (Liaw et al. 2018) to accelerate generation and scheduling. HybridFlow (Sheng et al. 2025 improves throughput by combining single- and multi-controller paradigms, while RLHFuse Zhong al. 2025) boosts GPU utilization through stage fusion and micro-batee scheduling. Our approach complements these efforts by compounding PPO's ' disaggregated stages with intra- and inter-step overlap, further improving utilization and throughput.
RLHF Optimizations. Another active direction reduces algorithmic complexity or improves robustness. Critic-free algorithms- such as GRPO (Shao al. 2024). ReMax (Li et al.) (0024), RLOO (Ahmadian et al.) (202)) and REINFORCE++ (Hu et al. [025a -remoee the value network, estimating advantages directly from normalized rewards over multiple rollouts. RL-free methods including DPO (Rafailov et al. 2024] and EXO (Ji et al., 2024) bypass reinforcement learning entirely, while robustness-focuse methods like RLP (Lang et al. (024) and BSPO (Dai et al.) 2025 mitigate reward misalignment. Other efforts, such as LoCo-RLHF (Lee et al. 2024). address preference heterogeneity. Our method is orthogonal to these algorithmic improvements, as it preserves PPO semantics while accelerating its execution.
CONCLUSION
We introduce OPPO, lightweight framework for efficient PPO-based RLHF training by maximizing execution overlap. OPPO introduces a new dimension of efficiency- sfficiency-iitra-steee intra-step overlap, which streams actor tokens downstream models for incremental prefilling, and inter-step overlap, which strategically defers stragglers to future steps. Both overlaps convert idle time into useful work. Our extensive evaluations across free-form generation, math reasoning, and code generation tasks. show that OPPO accelerates PPO training by to 2.8 raises GPU utilization by over 2. x, and generalizes to alternative paradigms such DPO.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace. Julia Kreutzer, Olivier Pietquin, Ahmet Ustün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in 1ims, 2024. URL URL.attp:://axxx..crr///bba//
14 4400
Zhuotong Chen, Fang Liu, Jennifer Zhu, Wanyu Du, and Yanjun Qi. Towards improved preference optimization pipeline: from data generation to budget-controlled regularization, 2024. URL
https:://ariiii org/ abs 221/2411110 /tt.:////21111.111111.. 05875
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL httphhttts://attt www.ariii........oooommmmo org/ aabs/ab///// 803.803.00 05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL inttps://arxiss
abs/ 2110.1111000111 10..11166 14168
Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, and Gang Pan. Mitigating reward over- optimization in rlhf via behavior-supported regularization, 2025. URL htttps://rxxv..org
abs/2503. 18130
Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective, 2024. URL htttps:///axiii
04626
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation, 2025.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang. Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025. URL
abs/2505.24293
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024. URL htttp:://eenod.. www.eaoddwwoggee.orrroooeeee rww.rrscorrrs//e 212008622662
Aaron Grattafiori Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshor,, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere and Bethany Biron. The llama 3 herd of models, 2024. URL pettps//aarae www. abs abtp://.//2222212/
Jian Hu, Li Tao, June Yang. and Chandler Zhou. Aligning language models with offline learning from human feedback, 2023. URL httttttt////xiiiii www.riiii.ii.o.oeoooooooooo httpg//bb//30...22 12050
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rhhf algorithm with robustness to both prompt and reward models, 2025a. URL httpt::/aaxii...ttt orb/bs//
2501. 03262
Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang. Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, and Yiming Liu. Openrlhf: An tase-to-use scalable and performafom rlhf framework. 2025b. URL htthts:: //arxivv brg/abb/2405... 11143
11

---

Published as a conference paper at ICLR 2026
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models, 2024, URL BtLLttss:///axiv..om
abs /twpb///1111111.. 04 04905
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang. and Minlie Huang. Towards efficient exact optimization of language model alignment, 2024. WtR.httss://ar...
org/abs/2402 2402. 00856
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL /wtp///aavvvvvvvva//aa org/ abs///b///// 2309. 06180
Hao Lang, Fei Huang, and Yongbin Li. Fine-tuning language models with reward learning on policy, 2024. URL
abs
Seong Jin Lee, Will Wei Sun, and Yufeng Liu. Low-rank contextual reinforcement learning from heterogeneous human feedback, 2024. URL
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling, 2025. URL
http:/orgggabbbbboe httpsss/arxxvvoorrr////////
httpss///arxi
11886
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: simple, effective, and efficient reinforcement learning method for aligning large language models, 2024.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training, 2018. URL https
/arxiv. /aravv.org/ab//1877.0333
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human 07958
falsehoods, 2022. URL
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL http:/ss///aaxv..o/////////////e
19393
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models, 2025.
2410. 18252
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022. URL
awarg///b//// org/ abs 2203. 02155
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct prefereecc optimizatioon:: Your language model is secretly reward model, 2024. URL
ht
18290
Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient rlhf: Reducing the 00754
memory usage of ppo, 2023, URL http://:///aa.....a./a//aaa
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https::/arxiv.o:0/// wwtp.//w.gg.bbb...ooooo 063477
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li. Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. /tttps://axxi....
2402. 03300
12

---

Published as a conference paper at ICLR 2026
Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong. Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksii Kuchaiev. Nemo-aligner: Scalable toolkit for efficient model alignment, 2024. URL https:
//arxiv.org/ abs http://bbb22240.000 211.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. In Proceed- ings of the Twentieth European Conference on Computer Systems, EuroSys '25, 1297. ACM, March 2025. doi: 10.1 1411144669911136666
URL http://dx doi http://v......../.0111111111 wtt./...g/./.....
3689031 3696075
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. via adaptive curriculum learning, 2025. URL https:
finetuning 05520
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https::/gitthui http://ww....ooigggggaa////////e
2020a.
Leandro von Werra et al. Stack-exchange-pairee dataset.
co/
datasets Bitiaaetts/wweerr///ttteeee exchangecheeee wacchange-paaredd paired 2020b. Accessed: 2025-09-23.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao
Yu, and Yi Wu. Is dpo superior ppo for Ilm alignment? comprehensive study, 2024. URL
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijic Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025, itttss::t///arii http://axxxv...//a/././/
abs abwpg//ab///02222 03387
Rowan Zellers, Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL. htt.0///...aae//aa...oooooo////. 00100000///01115007777
Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, and Xin Jin. Optimizing rlhf training for large language models
with stage fusion, 2025. http:::::: /tt////aiiii www./rgg//////ooooeo... /2409 13221
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 MODEL EVALUATION PROTOCOL
We conduct comprehensive evaluations assess the impact of OPPO model quality across different model scales. Our evaluation protocol compares models trained by TRL. with standard PPO-based RLHF against models trained with our proposed overlapping optimization techniques. We evaluate two model configurations: Qwen2.5-7h and Qwen2.5-3H each fine-tuned on the Stack- Exchange- Paired dataset for comparable training steps.
A.1.1 BENCHMARK SUITE
To ensure comprehensive assessment of model capabilities, employ the Language Model Evaluation Harness (Gao et al. 2024), a standardized framework for evaluating language models across diverse tasks. Our evaluation suite comprises six core benchmarks that assess different aspects of model performance:
Reasoning and Common Sense Tasks:
HellaSwag Zellers al. 2019): Evaluates commonsense reasoning through sentence completion, requiring models to select plausible continuations of everyday scenarios.
ARC (AI2 Reasoning Challenge) (Clark al. 2018): Comprises two Lubsets-ARR-EEE and ARC-Challenge- Challenge-assessinn scientific reasoning through grade-school science questions of varying difficulty.
Truthfulness and Mathematical Reasoning:
TruthfulQA-MC2 (Lin al. 2022): Measures the model's tendency generate truthful responses through multiple-choice questions designed to elicit common misconceptions.
GSM8K (Cobbe et al. 2021: Evaluates mathematical reasoning through grade school math word problems requiring multi-step solutions.
A.1.2 EVALUATION METRICS
For each benchmark, we report multiple metrics capture nuanced performance differences:
Standard Accuracy (acc): Raw accuracy scores computed directly from model predictions.
Normalized Accuracy pacc_norm): Leength-normalized accuracy accounting for varying response lengths, particularly relevant for multiple-choice tasks.
Exact Match Scores: For GSM8K, we report both strict-match scores (requiring exact numerical answers) and flexible-extract scores (allowing for minor formatting variations).
A.1.3 EVALUATION PIPELINE
Our evaluation pipeline follows systematic approach to ensure reproducible and reliable results: Stage 1: Environment Configuration. Each evaluation begins with proper environment initialization, including CUDA device allocation and verification of GPU availability. We employ float16 precision for all evaluations to maintain consistency with training configurations while optimizing memory utilization.
Stage 2: Batch Processing. Models are evaluated using adaptive batch sizing based on available GPU memory. For 7B models, utilize batch size of 4, while 3B models support batch size of 8, maximizing throughput without encountering out-of-memory errors. All evaluations employ greedy decoding ensure deterministic and reproducible results.
Stage 3: Task-Specific Evaluation. Each benchmark task is evaluated independently to isolate performance characteristics. The evaluation hamness automatically handles task-specific preprocessing, including few-shot prompt construction where applicable. For reasoning asks (ARC, HellaSwag), employ 25-shot, 10-shot, and 5-shot evaluations, respectively, following established protocols.
14

---

Published as a conference paper at ICLR 2026
TruthfulQA-MC2 uses O-shot evaluation to assess inherent model knowledge without exemplar influence.
A.1.4 STATISTICAL CONSIDERATIONS
To ensure statistical validity of our comparisons, we maintain consistent evaluation conditions across all model pairs:
Fixed random seeds for reproducible prompt sampling
Identical prompt formulations and few-shot examples
Consistent tokenization and preprocessing pipelines
Synchronized evaluation checkpoints (e.g., 800 steps for 7B models, 1200 steps for 3B models)
Evaluations are conducted on NVIDIA A100 40GB GPUs, with a complete evaluation of a single model requiring approximately 2-3 hours depending model size. The evaluation pipeline is designed to be portable and reproducible, with automated dependency management and environment configuration scripts to facilitate replication across different computational environments.
A.2 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language model (ChatGPT) only as a writing assistant tool to check grammar, improve readability, and polish sentence clarity.
A.3 REPRODUCIBILITY STATEMENT
To ensure reproducibility of our results, provide comprehensive implementation details throughout the paper and supplementary materials. The complete OPPO algorithm specified in Algorithm [ with detailed descriptions of the intra-step and inter-step overlap mechanisms in Sections 3.1 and 3.2. Experimental configurations, including model architectures and training settings for both 3B and 7B models, are detailed in Section 4.1 and Appendix A. We utilize publicly available datasets with standard preprocessing procedures described Section 4. The dynamic control parameters for overcommitment degree adaptation are fully specified in Sections 3.1 and 3.2, including bounds, momentum values, and window sizes. Our implementation requires minimal modifications to existing PPO codebases, with the specific integration points outlined in Section 3. All experiments were conducted on NVIDIA A A100 and H200 GPUs. We will release our implementation code upon publication to facilitate reproduction and adoption of OPPO existing RLHF pipelines.
15


Reviews:
[{"review_id": "fjw0G3Dtz6", "reviewer": "Reviewer_bNZ4", "summary": "This paper introduces a new pipeline methodology for PPO. The methodology is illustrated for single-node training and results in significantly faster training times.", "questions": "Do you have any results in a multinode setting?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The impacts on the speed of post training are quite significant, and the implementation appears to be easy to add to existing code bases. The paper is well-written with detailed experiments and clear plots. I also really appreciate the annotated algorithm.", "weaknesses": "> Among RLHF methods, Proximal Policy Optimization (PPO) (Schulman et al., 2017) has been the de facto standard due to its training stability and flexibility across diverse reward models and objectives. PPO underpins the training of state-of-the art LLMs such as GPT-4 (OpenAI et al., 2024), Gemma (Team et al., 2024), and LLaMA (Touvron et al., 2023; Grattafiori et al., 2024). \n\nThe quoted passage is wrong. As far as I am aware there is no public information on the RLHF strategies used in GPT-4 or Gemma 3, and while LLaMA 2 (Touvron et al., 2023) used PPO, LLaMA 3 and 4  (Grattafiori et al., 2024; Meta, 2025) use DPO. I do not believe that PPO is the current standard anymore, with both DPO and GRPO being significantly more popular. Qwen 2 used DPO, Qwen 2.5 used both DPO and GRPO, and Qwen 3 used just GRPO. GRPO is used by DeepSeek in all their models. DPO is used by Tulu, Hermes3, and several other models as well. I am unaware of any major model trained in 2024 or 2025 that discloses that it uses PPO. None of this means that improving PPO's efficiency isn't useful, but this paper heavily oversells how widely used PPO is and uses that as a major motivator.\n\nMore importantly, this paper appears to exclusively examine single-node training. While that's not inherently problematic it is a strong limitation to the impact of the methodology since it would be inapplicable for large models. In the absense of any discussion or results of multinode training, I'm going to assume that the method doesn't work in a multinode settting. I strongly recommend including this as a limitation in the discussion of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XswQzZ3UTS", "reviewer": "Reviewer_Hdb3", "summary": "This paper identifies and addresses two major sources of inefficiency in standard PPO-based RLHF training pipelines: (1) sequential dependencies between the actor, critic, and reward models, where downstream models must wait for the upstream actor to fully complete generation , and (2) long-tail latency, where a few long \"straggler\" responses in a batch delay the completion of the entire step .\n\nThe authors propose OPPO, a lightweight systems optimization framework that introduces two novel overlap techniques:\n\nIntra-step Overlap: The actor model streams its generated tokens in adaptive chunks to the downstream critic and reward models. This allows the downstream models to begin their computationally expensive \"prefill\" stage concurrently while the actor is still decoding, effectively hiding the scoring latency .\n\nInter-step Overlap: The pipeline \"overcommits\" by starting with a batch of B + Δ prompts. It sends the first B responses that complete to the training stage, while the Δ unfinished straggler responses are deferred and resumed in the next iteration. This mitigates tail latency without discarding partial work .\n\nEmpirically, OPPO is shown to accelerate end-to-end PPO training by 1.8×–2.8× and improve GPU utilization by 1.4×–2.1×, all while achieving identical step-to-reward convergence and final model quality", "questions": "Could you elaborate on the \"few lines of code change\" claim? This appears to require a complex streaming and scheduling data structure (Algorithm 1) . What is the practical engineering lift to integrate OPPO into an existing TRL-based pipeline?\n\nThe dynamic control for Δ is based on the reward slope st. How sensitive is this heuristic to a noisy reward signal? If the reward oscillates (common in RL), would this cause Δ to become unstable and harm performance?\n\nThe paper mentions OPPO \"generalizes to other paradigms such as DPO\". How would this work? The DPO pipeline does not have the same multi-stage, multi-model dependency as PPO. What exactly would be overlapped?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Massive Efficiency Gains: The headline result of a 1.8×–2.8× speedup on a notoriously slow and expensive pipeline is a major strength. This is a huge practical win.\n\nPreservation of Convergence: The paper's strongest evidence is that this speedup comes \"for free.\" Figure 4 and Table 1 show that the step-to-reward convergence and final model quality are nearly identical to the baseline. This proves it's a true systems optimization, not an algorithmic trade-off.\n\nOrthogonal Solution: The method attacks two different, orthogonal bottlenecks: pipeline bubbles (via intra-step overlap) and straggler latency (via inter-step overlap). The ablation in Figure 6 clearly shows that both components are necessary for the full speedup.\n\nSmart Adaptive Control: The heuristic for dynamically controlling the overcommitment level Δ is very clever. It adaptively becomes more aggressive when training is improving and more conservative (decaying Δ) as training converges, balancing speed with stability.", "weaknesses": "\"Lightweight\" Claim: The paper claims the method is \"lightweight\" and requires \"a few lines of code change\". This seems like a significant overstatement. Implementing a streaming data buffer, inter-model communication for chunks, and the dynamic Δ scheduler (Algorithm 1)  is a non-trivial systems engineering task that deeply modifies the training loop's data flow.\n\nHeuristic-Driven Controls: The method's dynamic controls are entirely heuristic. The chunk size is tuned by \"periodically... appl[ying] a few candidate chunk sizes\" , and the Δ level is based on the reward slope. While these heuristics are smart and work well, they are not theoretically derived and may require careful tuning (e.g., the window size W, bounds Δmin/Δmax).\n\nUnanalyzed Staleness: The inter-step overlap introduces a form of staleness. While the policy used for generation is consistent (the work is just deferred), the prompts that are deferred are from an older data distribution. The paper shows that high staleness is bad (Fig 2c)  but doesn't analyze if this prompt-distribution staleness has any subtle, negative effects on the converged model, especially if Δ is large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wd9QiJGjUT", "reviewer": "Reviewer_ty1D", "summary": "The paper focuses on improving the training efficiency of PPO algorithms. The authors propose two overlapping techniques:\n- Intra-step overlap: allows the downstream model (e.g., reward model) to begin prefill while the upstream model continues decoding.\n- Inter-step overlap: adaptively overcommits a few prompts and defers longer generations to future steps.\nExperiments demonstrate that the proposed methods significantly accelerate PPO-based RLHF training.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow.\n\n- Training efficiency of PPO algorithms is an important problem. The proposed two overlapping techniques are intuitive and well motivated.\n\n- Experiments across different tasks are conducted to demonstrate the acceleration achieved by the proposed algorithm.", "weaknesses": "- For intra-step overlap, in reasoning scenarios, difficult problems often require generating long responses. With this technique, is there a risk that updates for difficult problems will always be deferred, while the algorithm converges quickly on easier problems?\n\n- Could intra-step overlap affect the distribution being learned, as discussed in the first question?\n\n- There are other techniques, such as fine-grained parallelism adopted in VERL for accelerating PPO, but these are not included in the experiments. Could the authors elaborate on the reason for this omission?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "99HvxX1ZCm", "reviewer": "Reviewer_FKvv", "summary": "This paper examines training system efficiency in PPO-based RLHF algorithms, revealing significant GPU underutilization and latency bottlenecks in current synchronous approaches. To overcome these limitations, the authors introduce a dual-strategy solution: First, intra-step overlap enables pipelined execution by chunking upstream model outputs (e.g., actor model) and streaming them to downstream models (e.g., reward model), allowing prefill operations to commence before upstream decoding completes. Second, inter-step overlap employs adaptive prompt overcommitment, strategically postponing lengthy generations to subsequent steps to reduce tail latency while preserving computational progress. Experimental results on the TRL framework demonstrate 1.8×–2.8× efficiency gains over baseline implementations.", "questions": "- Regarding the intra-step overlap strategy, could this be implemented entirely at the sequence level? Specifically, would it be feasible to use an asynchronous approach where the generation batch is split into mini-batches, with each completed sequence immediately forwarded to the reward model for scoring upon generation completion?\n- Could the authors provide a detailed comparison with recent advances in Verl, AReaL, RoLL, and OpenRLHF? These represent more modern, industry-grade systems with their own optimization strategies. A comparison would help clarify the unique contributions of this work relative to the current state-of-the-art, as TRL may not reflect the most competitive baseline.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Provides valuable empirical analysis of inefficiencies in practical PPO-based training systems.\n- Introduces effective actor generation and reward scoring overlap strategies to improve pipeline efficiency.\n- Demonstrates promising empirical results on small models, achieving reduced running time without performance degradation.", "weaknesses": "- The paper lacks systematic analysis of system efficiency. A more principled approach to analyzing the dimensions for reducing time complexity and examining related hyperparameters would strengthen the contribution.\n\n- The improvements are demonstrated on small-scale models with limited training data. It remains unclear whether these advantages extend to large-scale models or architectures beyond those tested in the paper. A theoretical framework and analysis would make the claims more convincing.\n\n- The paper lacks comparison with existing systems such as Verl, OpenRLHF, AReaL, and RoLL. These systems have explored numerous resource management and load-balancing strategies for addressing challenges like long-tail generation.\n\n- The improvements are restricted to PPO algorithms in the RLHF setting, which limits the paper's impact. A significant portion of modern alignment training uses GRPO and RLVR approaches, where no reward model is involved, and the proposed overlap strategies may not be applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)