FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning
Authors: Yizhou Zhang
Ning Lv
Teng Wang
Jisheng Dang
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
group relative policy optimization
speculative decoding
acceleration
Primary area: foundation or frontier models, including LLMs

Abstract: Group relative policy optimization (GRPO) has demonstrated significant potential in improving the reasoning capabilities of large language models (LLMs) via reinforcement learning. However, its practical deployment is impeded by an excessively slow training process, primarily attributed to the computationally intensive autoregressive generation of multiple responses per query, which makes the generation phase the primary performance bottleneck. Although speculative decoding presents a promising direction for acceleration, its direct application in GRPO achieves limited speedup under high-concurrency training conditions. To overcome this limitation, we propose a concurrency-aware speculative decoding framework that dynamically adjusts the drafting and verification strategy according to real-time concurrency levels, thereby maximizing the acceleration of the generation process. Furthermore, to address performance degradation arising from distributional drift between the evolving target model and the fixed draft model during training, we introduce an online draft learning mechanism that enables the draft model to continuously adapt using feedback signals from the target model. Experimental results across multiple mathematical reasoning datasets and models demonstrate that the proposed method achieves end-to-end speedups of 2.35x to 2.72x, significantly surpassing baseline approaches in efficiency. The code is available at https://github.com/yedaotian9/FastGRPO.

Published as a conference paper at ICLR 2026
FASTGRPO: ACCELERATING POLICY OPTIMIZATION VIA CONCURRENCY-AWARE SPECULATIVE DECODING AND ONLINE DRAFT LEARNING
Yizhou Zhang Ning Lv
Teng Wang
Jisheng Dangl.31
Lanzhou University
2 The University of Hong Kong
3 National University of Singapore
ABSTRACT
Group relative policy optimization (GRPO) has demonstrated significant poten- tial in improving the reasoning capabilities of large language models (LLMs) via reinforcement learning. However, its practical deployment is impeded by an ex- cessively slow training process, primarily attributed to the computationally in- tensive autoregressive generation of multiple responses per query, which makes the generation phase the primary performance bottleneck. Although speculative decoding presents promising direction for acceleration, its direct application in GRPO achieves limited speedup under high-concurrenccy training conditions. speculative decod-
To overcome this limitation, we propose a coneurrency-awee ing framework that dynamically adjusts the drafting and verification strategy ac- cording to real-time concurrency levels, thereby maximizing the acceleration of the generation process. Furthermore, to address performance degradation aris- ing from distributional drift between the evolving target model and the fixed draft model during training, we introduce online draft learning mechanism that en- ables the draft model to continuously adapt using fcedback signals from the target model. Experimental results across multiple mathematical reasoning datasets and models demonstrate that the proposed method achieves end-to-end speedups of 2.35x to 2.72x, significantly surpassing baseline approaches in efficiency. The code is available at htnps///ghubboomm/taaaat////ssss
INTRODUCTION
Group relative policy optimization (GRPO) has recently emerged as promising framework for enhancing the reasoning capabilities of large language models (LLMs) through reinforcement learn- ing (Guo et al. 2025). In each training iteration, the LLM generates a group responses to a given query. These responses are subsequently evaluated using predefined rule-based reward function, and the resulting rewards are standardized prior model updates via policy optimization (Shao et al. 2024). This approach exploits group-level feedback signals to guide the model toward more accurate and coherent reasoning behaviors.
However, compared to conventional supervised fine-tuning (SFT) (Ouyang et al.) 2022). ). the GRPO training paradigm suffers from significantly lower throughput, hindering its adoption and limiting empirical exploration. This bottleneck arises primarily from the autoregressive inference required for response generation, which dominates the training pipeline. As shown in Figure (a), the gener- ation phase (i.e., response sampling) accounts for 91 % to 98% of total training time across multiple mathematical reasoning datasets, making it the primary target for performance optimization.
To this end, we propose integrating speculative decoding, compute-effiiien and accuracy- preserving technique originally introduced by (Leviathan et al. (003) for accelerating autoregressive inference, into the GRPO framework. Speculative decoding employs smaller, faster "draft model"
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2026
Curtentt Model
Generation Time
Other Time
Render
Cavositrr FM
14.4%
Daraa AM
11 91.2%
242
Kaiia /M
Owena.5-72 5-70
$0.0%4 694
4%
Spedis Rati
Llamao 1:00
90.8%
00.00
LINN MARI S/W
(b)
Figure (a) Proportion of generation time during GRPO training across multiple models and datasets, showing that generation dominates the overall time cost. (b) Speedup ratios on various models and datasets, demonstrating the significant and broad acceleration achieved by our approach.
to generate candidate token sequences in advance, which are then efficiently verified by the larger target model. This method capitalizes on the observation that autoregressive decoding is typically memory-bound rather than compute-bound, thereby enabling increased computational parallelism and improved hardware utilization. By reducing the number of direct forward passes through the target model, speculative decoding offers the potential for substantial acceleration of the generation phase in GRPO.
However, direct integration of speculative decoding into the GRPO generation phase yields signifi- cantly lower speedup compared to results reported in prior work (Weng et al. [2025] Li et al. 2025b), and in certain cases even leads performance degradation. This discrepancy arises because spec- ulative decoding is predominantly evaluated in low-concurrency. low-latency settings (e.g.. edge device deployment), whereas the generation phase in GRPO operates under high-coneurrency and conditions. Although speculative decoding alleviates memory bandwidth pressure, it introduces additional computational overhead. nigh-concurrency scenarios, this trade-off may shift the system from a eeenory-bound to compute-bound regime, thereby negating the expected performance gains.
To address this challenge, we analyze the responses generated during each step of the GRPO train- ing process and observe significant variability in sequence lengths across batches. This variability leads to a substantial reduction in effective concurrency during the generation phase, with effective concurrency declining from a high initial batch size to nearly one as sequences terminate at different times due to uneven completion. Motivated by this observation, we propose concurrency-aware speculative decoding strategy specifically designed for the generation phase in GRPO. Our method dynamically adjusts its configuration (i.e., the draft tree size and the number of verification tokens) based on real-time concurrency levels. Consequently, it achieves moderate speedup during the early, high-concurrency stage and delivers progressively greater acceleration concurrency decreases, thereby minimizing the overall latency of the generation phase.
Furthermore, we identify a critical challenge inherent in the GRPO training framework: the target model undergoes continuous parameter updates, resulting in an increasing divergence between the target model and the fixed draft model. This distributional shift inevitably degrades model align- ment, leading to declining token acceptance rates and diminishing acceleration gains over time. To mitigate this issue, we propose an online draft learning strategy within the GRPO loop. Our approach updates the draft model using online feedback signals derived from the evolving target model. This continual adaptation significantly enhances the draft model's representational fidelity, thereby increasing the average length of accepted speculative tokens and sustaining higher acceler- ation ratios throughout training, rather than allowing them to deteriorate.
and other models across three mathematical reasoning datasets of varying difficulty: GSM8K (Cobbe et al. 2021). SimpleRL-Abel-Level33
We conduct experiments on Srren2..-7B-Inttreee
Llama3. 10mma3.1-8B
(Zeng et al. 2025), and DAPO-Math-- 17K (Yu et al.) 2025). As illus- trated in Figure (b). our proposed method substantially improves inference acceleration, achieving end-to-end speedups of 2.35x to 2.72x across different models and datasets.
2

---

Published as a conference paper at ICLR 2026
Qenat 5-1 oo X Oree 5-TB-battaaoooo
Llas Ai (iblled SemesS/tttuurrr (Pac))
SuN Dea
Mor Sute
Geoeration 3o Ttaiiing Time Rati
25-
2.0
15-
Sperdap
10-
1.5
52
Suquence Zanglh
Figure 2: Ratio of generation time to parame- ter update time for different generation length in Different Models and Two Speculative Decoding GRPO framework. The model is Qwen2.5-7B- Instruct and other setups are in Section 5.1
Figure 3: Speedup Ratio vs. Batch Size under
Strategies. The model is Qwen2.5-7B-Inste and other setups are in Section 5.1
2 PRELIMINARIES
Speculative Decoding. Speculative decoding (Leviathan et al. 2023 is an inference acceleration technique for autoregressive language models that leverages a fast draft model to propose candidate token sequences, which are then verified in parallel by more capable but slower target model. The method is motivated by two key observations: (1) not all generation steps require the full capacity of the target model; some tokens
be predicted accurately by smaller models (Hinton et al., 2013 Jaszczur et al. 2021: Hubara et al., 2018 So et al. 2022]] Shazeer 2019); and (2) in modern hardware, inference often limited by memory bandwidth rather than compute (Vaswani et al., 2017 Park et al.] 2025), leaving room for additional parallel computation. The process operates two phases. First, the draft model autoregressively generates several candidate tokens. Then, the target model performs a single forward pass over the input prefix concatenated with these speculative tokens, using an appropriate attention mask compute logits for all positions parallel. Drafted tokens are only accepted if they match the target model's 's output. By employing strict acceptance criteria, the final generated sequences are guaranteed to be identical to those produced by the target model standalone, ensuring that the acceleration of the rollout process does not compromise the performance of GRPO.
Draft Model Architectures. We adopt the draft model architecture from EAGLE (L al. 2024b)
enhanced by Feature Sampling and Partial Alignment Distillation (Gui et al. 2024). This approach trains standalone draft model to autoregressively generate hidden states, which then decoded into tokens using the LM head the target LLM. The draft model takes the previously sampled tokens as input stabilize generation, enabling effective feature-level modeling while leveraging the target model's output distribution.
Verification Strategies. We adopt the adaptive token tree verification strategy introduced in EAGLE-2 (Li et al.] (024a). In this approach, the draft model first expands the candidate token tree over a number of drafting steps. The candidate tokens are then reranked based on their pre- dicted confidences. During verification, the target model evaluates all branches in parallel using tree-structured attention, and accepts the longest path whose generated tokens match the draft to- kens under strict token-level comparison.
3 OBSERVATIONS IN GRPO TRAINING
This section empirically investigates the computational dynamics of GRPO training, identifying the generation bottleneck and intra-batch length variance. All experiments follow the setup in 5.2
Inference Bottleneck. The primary time-consuming bottleneck in GRPO training is the genera- tion phase (i.e., rollout sampling), which far exceeds the cost of parameter updates, as shown in Figure (a). This disparity stems from two main factors. First, the generation cost scales linearly with the output sequence length. As shown in Figure 2 the ratio of generation to update time in- creases from approximately 6x to over 20x the model matures. This aligns with prior findings
3

---

training progressively inflates the inference Published as a conference paper at ICLR 2026
Algorithm 1 FastGRPO Framework.
Require: Policy model Ti (target model), draft model kd. prompts D, group size G, hyperparameter Cpesk
Ensure: Updated We and Wd
1: for iteration 1 / do Sample batch D, while not all sequences Sgenetated end with EOS
[qu)m= C, me prefill Da; set Beur
4: // Concureecc--wwaee Speculative Decoding
5: Obtain hyperparameters Idrast, Kdraft, Nverity from Eq.
Eq.
Eq..
based on Beur and Opeak
6:
for f 1 to Ldeatt do
Generate children for Karaft paths and update candidate tokens Toundi Il draft expansion
8:
Select Nowrify tokens from Teandi based on their confidences // druft reranking
9:
Verify selected tokens in parallel // verification phase Extract accepted tokens and corresponding hidden states; update Sgenerated
10:
Update Beur and Seenratedd if any sequence ends with EOS
Online Draft Model Updating
13: Draft Training phase: Update Fa using Sgenerated and corresponding hidden states
14:
Policy Training phase: Compute rewards for Sgenerated and update
that more capable policy models tend to generate longer, more complex outputs (Liu et al. 2025 Wu et al. 2025 Shrivastava et al. 2025). Consequently, the growing sequence length throughout Second, the prevalence of low-reward-variant rollouts reduces data efficiency and increases the inference cost. GRPO requires response groups with non-zero reward variance derive learnable gradient. However, models often produce homo- geneous outputs] where all responses are uniformly correct or incorrect, resulting in eero-variance groups that must be discarded.
High Intra-Batch Variance in Response Lengths. The group-based relative advantage inher- ently encourages length variation during response sampling. Our analysis reveals significant length variation in responses generated within single GRPO batch. Across various models and datasets, the maximum sequence length is typically 3 to times the minimum, and the range often exceeds the mean length. This heterogeneity is more pronounced on challenging datasets and correlates with intrinsic model properties like output diversity. The high intra-batch variance characteristic leads to a notable transition from high to low concurrency during the generation phase of GRPO, rather than sustaining high concurrency throughout.
FASTGRPO
Overview. FastGRPO integrates two mechanisms: concurrency-aware speculative decoding and online draft learning. During GRPO generation phase, celerate batched rollouts, and the decoding hyperparameters are adaptively tuned to the current concurrency (i.e., batch size). In the parameter update phase, alternately freeze model and train the other: the draft model is trained on the target model's previously generated hidden states match its output distribution, while the target model follows standard GRPO (reward scoring and aaicc-grradeent updates). The full procedure is summarized in Algorithm
speculative decoding to
4.1 CONCURRENCY-WWARR SPECULATIVE DECODING FOR GRPO TRAINING
Motivation. Speculative decoding is primarily designed as an acceleration technique for low- concurrency inference scenarios. In principle, its performance gain arises from the verification phase: by enabling the target model to validate multiple candidate tokens in a single forward pass, reduces the frequency of parameter transfers from GPU memory on-chip SRAM, thereby alleviat- ing memory bandwidth pressure. However, this benefit comes at the cost of increased computational load. Specifically, additional FLOPs are incurred during draft generation and the verification of in- valid speculative tokens.
For instance, moderately difficult dataset SimpleRI filtered out. This problem intensifies on datasets that are either simple complex for the model.
about 20% of generated groups
4

---

Published as a conference paper at ICLR 2026
We study the impact of batch size speculative decoding speedup. As shown in Figure 3 using a configuration optimized for small batches (B = 1) leads to significant performance degradation at larger batch sizes, with speedup dropping below 1.0x at higher concurrency, indicating growing computational overhead. In contrast, dynamically adjusting key speculative decoding parameters (e.g.. draft tree size and number of verified tokens) as batch size increases effectively mitigates this degradation. The speedup remains robust under high concurrency, demonstrating that adaptive tuning is essential for maintaining efficiency. These results highlight the need align speculative decoding hyperparameters with system concurrency to balance compute and memory demands.
Our earlier analysis reveals that the generation phase in GRPO does not operate under static con- currency: instead, it evolves dynamically from high to low concurrency as sequences complete at different times due variable lengths. Therefore, method that adapts speculative decoding param- eters in real time according to the current effective batch size is both practical and effective. Such adaptivity enables sustained acceleration throughout the generation phase, minimizing the overall latency of the GRPO training pipeline.
Analysis of Operational Intensity. We begin with a theoretical analysis of operational inten- sity (Williams al. 2009; Yang et al. 2020 2021). defined as the ratio of computation (in FLOPs) to memory traffic (in bytes). This metric is central to understanding the performance charac- teristies of low- versus high-coneurrency inference scenarios. For general matrix multiplication (GEMM) (Kurzak et al. 2010;) Xu et al. 2022). the operational intensity scales as:
2BD,n Dont
2B
2
IGEMM = (BDio + Din Dou + BDuut)
B Dout
1
8 D.n.
where the input matrices have dimensions B x Din and Dm x Dout. respectively, and is the bytes per element (e.g.. for BF16 FP16). The numerator represents the total number of arithmetic operations, while the denominator accounts for the memory traffic, i.e., the total storage size of the input and output matrices. The final approximation holds when < min Din Dout); J In the Transformer architecture, the time cost is predominantly dominated by series of GEMM opera- tions. Therefore, IGEMM can serve as a reasonable approximation of the operational intensity of Transformer models.
Theoretically, the peak operational intensity of modern accelerators is determined by their hardware specifications, specifically by the ratio between peak FLOPS and memory bandwidth. In practice, however, analytically determining this peak value is challenging due to implementation-spedde factors such kernel optimizations, memory tiling strategies, and SRAM capacity limitations in matrix computations. These factors introduce non-ideal hardware behaviors that are difficult to model precisely. Instead, we approximate this theoretical limit through empirical profiling. We define a hardware- and seetitectuur-eppnddee reaches its practical peak operational intensity for given model and specific GPU. In other words, Cpeak represents the concurrency level at which the system transitions from being memory-bound to compute- bound. (See Appendix B for the detailed measurement methodology.)
constant, Cpeak- as the batch size B at which IGEMM
Concurrency-aware Strategy. We propose adaptive speculative decoding strategy that dynam- ically adjusts the number of verified tokens per sequence (denoted as Nwerify) and the draft tree size based on the level of concurrency. Our implementation follows EAGLE-2 (Li et al.) (022a)), where the draft tree structure is governed by two key hyperparameters: Käraft, the number of candidate tokens proposed by the draft model per forward pass, and Ldraft, the number of sequential drafting steps.
To make IGEMM approximately equal to Cpeak and thereby fully utilize both computational and memory bandwidth resources, we propose setting:
Cpeak B
N Weriify
As previously discussed, IGEMM is proportional to the batch size B. and Cpeak represents the op- timal batch size at which the GPU achieves peak hardware efficiency under given configuration. In the verification stage of speculative decoding, the effective batch size for the GEMM operations is determined by the total number of tokens verified per batch, given by the product Nverity x B.
5

---

Published as a conference paper at ICLR 2026
By maintaining Nverify Cpeak/ B, this effective batch size remains approximately constant Cpeak. ensuring that the system operates at the compute-mentn balance point. This maximizes hardware utilization and achieves the optimal speedup for speculative decoding under varying levels of concurrency.
The draft tree size should scale with Nverity. longer verification sequences can only yield benefits the draft model produces sufficiently diverse candidates. Therefore, we propose both Kraaft and Liraft should be positively correlated with Nverify-
To ensure that the number of tokens generated by the draft model is both sufficient and effective, we propose this configuration for Kdraft:
Kdraft = min (Nverity 1, K Kaax
Setting Kdrait = Nverity - 1 is typically sufficient for two reasons. First, the draft model faces similar compute and memory constraints as the target model; keeping Karaft S Nvurify ensures the drafting phase remains compute- memory balanced. Second, one of the Nverify tokens is the root of the draft tree, which generated by the target model the prior step and does not require speculative expansion. Thus, only Nverity 1 slots are available for draft-generater tokens. The upper bound K draft prevents inefficiency from excessive branching. The total number of candidates, 1+ Kdraf (Ldraft 1). K2 draft often far exceeds Nverify. large Karaft leads to many unverified candidates and wasted computation. It also increases tree construction overhead with diminishing gains in acceptance rate. Hence, K draft balances speculation coverage with efficiency.
For Idraft we propose a design that is positively correlated with log, (Nveritt) by the exponential growth of the draft tree's s candidate space with depth: deeper trees enable combi- natorial expansion of candidate sequences, allowing longer expected acceptance length under larger verification budgets. Experimentally, we refine this principle into a practical instantiation:
)- This is motivated
Ldraft min (1182)
<Nverify
)]
draft
where a is hyperparameter that encodes the approximation quality of the draft model. A stronger draft model (i.e., one with higher predictive accuracy) corresponds to a smaller a, enabling deeper speculation Nverity increases. Conversely, weaker models require larger prevent over- speculation and the generation of invalid branches.
This design accounts for the autoregressive nature of the draft model: prediction fidelity degrades with depth due to error accumulation, causing the likelihood of valid continuations to decay expo- nentially toward Icaf nodes. As result, excessive depth yields diminishing returns and may even incur negative gains from increased overhead in the drafting phase. The upper bound L further ensures robustness and computational efficiency across diverse workloads.
4.2 DRAFT MODEL. LEARNING WITH ONLINE TARGET FREDBACK
In the GRPO training framework, the target model undergoes continuous policy updates, re-
4.3
Avarage Accopted Langh
sulting in time-varying output token distribu-
3.5
tion. When fixed draft model is employed throughout the training process, its alignment with the target model progressively deterio- rates, leading to a reduction in the average ac- ceptance length during speculative decoding. This degradation directly translates into a de- clining speedup ratio, illustrated in Figure 4
web Online Draft Leaning w! Oniis Drai Loaming
3.0
2.5
Epoch
Figure 4: Average accepted length versus train- ing steps in GRPO, with and without online draft learning. The model is Qwen2.5-7B-Insse the dataset is SimpleRL--bellLLwelBtte
To mitigate this misalignment, we introduce an online draft learning strategy: at each GRPO iteration, the draft model is updated online us- ing responses generated by the current target model. This enables the draft model to adapt dynamically to the evolving target policy, thereby preserving and enhancing its predictive consistency over time. As demonstrated in Figure 4 this
and

---

Published as a conference paper at ICLR 2026
approach yields steadily increasing acceptance rate, in contrast to the performance decay observed with a static draft model.
A key benefit of online draft learning arises from the quality and policy relevance of the supervi- sion signal. Unlike conventional pre-training on static, offline datasets, the draft model learns from on-policy generations produced by the target model, which are inherently aligned with its current reasoning dynamics. This facilitates more accurate modeling of the target's output distribution. Moreover, data that cannot be used train the target model due to non-zero reward can still serve as useful supervision signals for the draft model, thereby reducing data wastage present in the tradi- tional GRPO framework.
Furthermore, the computational overhead of online draft learning is negligible, contributing only approximately 2% to 3% of the total training cost across various models and datasets. This efficiency is attributed to the fact that, in standard draft model training, the primary computational burden lies in executing forward passes on the target model to obtain supervisory signals (i.e., logits and hidden states). In our GRPO pipeline, these signals are naturally generated during the generation phase and can be cached without additional computation. Consequently, the draft model trained using supervision that is effectively "free," rendering the online draft learning approach both highly effective and computationally practical.
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Models and Datasets. We evaluate our method on Llama3. 1-BB --8B-Instruct (Dubey et al. 2024 Dieiecl-RR--issiil--e--------
(Guo et al., 2025), and multiple models from the Qwen Yang al. 2024) series. Experiments are conducted on three mathematical reasoming datasets of increasing dif- ficulty: GSM8K (Cobbe et al. 202T)
2025) (mid-level reasoning). and (grade-school math), SimpleRL. Abel-ee (Zeng al., (Yu et al. (2025) (advanced and diverse prob- lems). This setup enables evaluation of our method's s effectiveness and generalization across model families and complexity levels.
Pre-training of Draft Model. Following prior work (Li et al. 2024b, Gui et al., 2024). we use ShareGPT-68K (Aeala) 2023) as the pre-training dataset for the draft model. The draft model is trained for 10 epochs with a batch size of 16. The learning rate is set to le-4, with the AdamW
optimizer (Loshchilov & Hutter 2017).
Metrics. We use speedup ratio (Li et al. 2024b), defined as the ratio of baseline wall-clock time to method's time. We report: (1) Generation speedup ratio (Gen SR): speedup during the generation phase, reflecting gains from speculative decoding; (2) End-to-end speedup ratio (E2E SR): overall speedup across the full GRPO pipeline.
5.2 THE SPEEDUP OF GRPO PROCESS
Online Draft Learning. During the generation phase, the target model produces responses using speculative decoding. The prompt batch size per GPU is set to 4, and we sample 8 responses for each prompt. The target model optimized with a learning rate of le-6, while the draft model is trained with a learning rate of 5e-5. The entire training process runs for 10 epochs using the Adam W optimizer. All experiments are conducted using GPU H800 SXM accelerators.
Comparison. As shown in Table [ we use standard autoregressive decoding as the baseline, with a speedup ratio of 1.00x. We also compare against three established speculative decoding methods: EAGLE-2 (Li et al.) 2024a). HASS (Zhang et al.) 2024) and EAGLE-3 (Li et al. 2025b). The results demonstrate that our method significantly outperforms the unmodified speculative decoding baselines. Within the GRPO framework, our approach achieves substantial end-to-end speedup of 2.35x to 2.72x. Notably, all experiments utilize speculative decoding with strict acceptance criteria, ensuring that the post-training model performance remains consistent with the vanilla baseline.

---

Published as a conference paper at ICLR 2026
Table 1: Acceleration comparison in terms of Generation Speedup Ratio (Gen SR) and End-to- End Speedup Ratio (E2E SR) across different models on three mathematical reasoning datasets of increasing difficulty: GSM8K, SimpleRL-Abel-Leveerr
Q2.5-7B- and Qwen2.5-7B-Instee
I, L3.1-8--,, DS-RI-Qwen-T7H Q2.5- Q2.5-Math-7B-- represent 1llmma.11-8B-Istmee 7B-Instruct, respectively.
Qwen2.5-Math-7B, and Qwen2.5-Man
GSM8K
SinpleeL--Aee--LeeeBBt
DAPO-Math-17E
Average
Model
Method
Gen SR
E2E SR
Gen
E2E SR
Gen SR
E2E
Gen SR
E2E SR
EAGLE-2
1.17
1.16
1.12
1.11
1.10
1.09
1.13
1.12
Q2.5-7B-1
HASS
1.21
1.20
1.15
1.13
1.11
1.10
1.16
1.14
EAGLE33
1.28
1.20
1.13
FastGRPO
2.66
2.43
2.91
2.52
2.78
2.53
2.78
2.49
EAGLE-2
1.25
1.22
1.18
1.17
1.14
1.13
1.19
1.17
L3.1-8B-1
HASS
1.29
1.26
1.23
1.22
1.17
1.16
1.23
1.21
EAGLE-3
1.35
FastRRPO
3.04
2.51
2.92
2.69
2.83
2.67
2.93
2.62
DS-RI-Qwen-7B
FastGRPO
2.69
2.41
2.81
2.56
2.73
2.60
2.74
2.52
02.5-Muth-7B
FistGRPO
2.75
2.53
2.96
2.72
2.84
2.66
2.85
2.64
Q2.5-Math 7B-1
FastGRPOO
5.3 ABLATION STUDY
Online Draft Learning. We conduct experiments to evaluate the effectiveness of online draft learning in accelerating the GRPO training process, employing the Qwen2.5-77B-Instruue model. We compare Gen SR and E2E SR with and without online draft learning, under identical configurations detailed in Section 5.2 As shown in Table 3 the proposed online draft learning strategy signifi- cantly enhances computational efficiency, increasing the generation speedup ratio by approximately 0.7x 0.9x across various experimental settings. This improvement underscores the strong corre- lation between the draft model's 's predictive accuracy and the policy alignment of its training data. Furthermore, we observe that the training process may induce a degree of overfitting in the draft model to the current target policy. However, in the context of speculative decoding within GRPO, such overfitting is not detrimental; rather, it enhances prediction accuracy and contributes positively to the overall acceleration.
Moreover, we evaluate the performance of the draft model before and after online draft learn- ing using SimpleRL-Abel-LevelBBa training dataset. 'he online draft learning pro- cedure follows the setup detailed in Section 5.2 Experiments conducted with batch size of and sampling temperature of O. Following established evaluation protocols (Weng et al. 2025), we assess the draft model across three distinct task categories: multi-ture dialogue, code generation. and mathematical reasoning. The cor- responding benchmark datasets are MTT-bench (Zheng al.] (2023), HumanEyal (Chen) 2021). and GSM8K (Cobbe et al. (202T,, respectively. The following metrics are used to evaluate acceleration performance: (I) Speedup Ratio (SR): The measured end-to-end speedup relative to vanilla autore- gressive decoding. (2) Average Acceptance Length (T): The average number tokens accepted by the target model per verification step.
Table 2: SR and average acceptance length (7) on three benchmarks. The target model Qwen2.5- IB-Instr
as the
MI-bench
HumanEval
GSM8K
Average
State
SR
SR
SR
SR
Before
1.71
3.34
2.01
3.87
2.11
3,82
1.94
3.68
After
1.72
3,37
2.05
3.93
2.45
4.55
2.07
3.95
As shown in Table 2 the draft model after online draft learning maintains performance compa- rable to its pre-online-trainine counterpart general ceneral-purpose tasks, indicating no degradation in generalization capability. Notably, it achieves substantial gains in-domain, where the speedup ra- tio increases by 0.34. These results indicate that online draft learning enhances the draft model's alignment with the target domain without compromising its versatility.
Concurrency-aware Speculative Decoding. We conduct experiments to evaluate the effective- ness of our proposed coneurrency-awar speculative decoding. The model is - and other experimental setups are identical to those described Section 5.22 We compare our method against the following strategies: (1) Vanilla autoregressive generation with early termina-
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study on Gen SR and E2E SR across three datasets under various configurations.
GSM8K
HinpleRL-AAee-Leveele
DAPO-Math-T7
Average
Method
E2E
Gen SR
E2E SR
Gen SR
E2E
E2E SR
vanilla
1.00
1.00
1.00
1,00
1.00
1.00
1.00)
1.00
FastGRPP)
2.66
2.43
2.91
2.52
2.78
2.53
2.78
2.49
FastGRPO w/o online draft learning
1.78
2.16
2.01
1.89
1.83
1.74
vanilla w/ early termination
1.22
1.21
1.68
1.61
1.62
1.58
1.51
1.47
FastGRPO concurrency-awenn
2.27
2.14
2.30
2.44
2.19
2.43
tion for completed sequences (denoted as vanilla w/ early termination); (2) Speculative decoding without concurrency-awar strategy and early termination (denoted as FastGRPO w/o concurrency-
aware).
As shown in Table 3 the vanilla w/ early termination strategy provides moderate speedups, achiev- ing generation speedup ratio values in the range 1.2x-1.7x. However, vanilla autoregressive gen- eration cannot effectively exploit the computational headroom that emerges high-batch inputs re- duce concurrency due to early sequence termination. contrast, speculative decoding inherently benefits from such underutilized capacity by enabling parallel generation. Our concurrency aawae speculative decoding approach further enhances this advantage through dynamic adjustment of de- coding parameters, allowing more efficient resource utilization. The results demonstrate that method consistently outperforms other strategies across all evaluated tasks, yielding significantly higher speedup ratios and validating the efficacy of the proposed concurrency-awarc mechanism.
The Need for Draft Model Pretraining. We conduct an experimental analysis to evaluate the effectiveness of pretraining the draft model in accelerating the GRPO training pipeline. We design a controlled ablation scenario: the draft model is not pretrained and is trained solely through online draft learning with the target model. Speculative decoding is enabled after 128 training steps. As shown in Figure 5 although the initial token acceptance length is relatively low, it increases rapidly and converges to level comparable to that of the pretrained draft model within just 1-2 epochs.
These results suggest that, when the sole ob- jective is to accelerate GRPO training, pretrain- ing the draft model may be unnecessary in most cases. The online draft learning process alone is
Average Accopted Length
3,8
sufficient to quickly develop a competent draft policy that enables effective speculation, reduc- ing the need for additional pretraining over- head.
wes pretrained draft model pretrained croft modsl
Epoch
Transfer Experiments on GRPO Variants.
To evaluate the generality of method, applied our acceleration framework to two
Figure 5: Average accepted length versus train- ing steps in GRPO, with and without pretrained draft model. The model is Qwenn.5-7B-Innst and the dataset is SinpleRL--AeelLeeelBoo
GRPO variants: DAPO (Yu et al.) 2025) and GPG (Chu et al., 2025). The experiments are conducted using the Dwve2.55-7BInnstuul model, with the same datasets and evaluation
our approach achieves significant speedups. These results indicate that our method is effective not only under standard GRPO but also variant frameworks, achieving over 2x end-to-end speedup ratio consistently. This demon- strates the broad applicability and robustness of our acceleration framework.
metrics those used in Section [. As shown in Table 3
Table 4: Gen SR and E2E SR achieved by our acceleration framework on GRPO variants across multiple mathematical reasoning datasets.
GSMSK
HinpleRL-Ahhe-Levvllee
DAPO-Math-17K
Average
Method
Gen SR
E2E SR
Gen SR
E2B SR
Gen SR
E2E SR
Gen SR
EZE SR
DAPO
2.60
2.32
2.87
2.31
2.74
2.39
2.74
2.34
GPG
2.62
2.48
2.93
2.66
2.71
2.57
2.75
2.57

---

Published as a conference paper at ICLR 2026
6 RELATED WORK
Speculative Decoding. Speculative decoding accelerates LLM inference by using a draft model propose token sequences, which are then efficiently verified by the target model (Leviathan) ctt al. 2023). To improve draft-target alignment, recent methods such as Eagle (Li et al. 2024b), GliDe (Du al., 2024), and Clover-2 (Xiao al.) 2024 have explored reusing target model fea- tures or KV caches. Other efforts, including HASS (Zhang et al., (202) and Eagle-3 (Li et al.) 2025b), attempt to resolve the discrepancy between training and inference dynamics. Furthermore, SpecExec (Svirschevski al., 2024) demonstrates the significant potential of speculative decoding for deploying models exceeding 50B parameters on consumer- grade GPUs, while APD (Israel al, 2025) explores the decoding of diffusion large language models by adopting parallel verification mechanism similar to speculative decoding. However, the most effective training data for draft model is the output generated by the target model itself, yet the collection of such data is typically resource-intensive Our framework addresses this by effectively reusing the target model's 's real-time generation during the RL process for joint optimization, which has not been fully explored in prior work.
Reinforcement Learning Acceleration. Reinforcement learning has become key paradigm for enhancing reasoning and task-solving capabilities in large language models (LLMs) (Comanici al. 2025: Yang et al.] 2025). Methods such as GRPO (Shao et al.} 2024). DAPO (Yu et al,
2025
GSPO (Zheng et al., 2025a) and others (Tan et al., [025 Fan et al. 2025 Chen et al.
20256 a Zhang et al.) 2025b Chu et al. 2025 Mrouch et al., 2025 Zhao et al.) 2025 Robeyns
& Aitchissonn 2025 Zhang et al. 2025a Yue et al. (025). enable stable policy optimization through
clipping mechanisms. However, RL trainig remains highly inefficient due to long, autoregressive rollout phases that dominate end-to-end latency (Sheng al.
2025 Zhong et al. 2025). GPU un- derutilization stems from both slow token generation and load imbalance caused by variable-length rollouts (Fu al. 2025 Team et al. 2025). To mitigate these issues, recent systems such as Pipelin- eRL (Piché et al., 2025) StreamRL (Zhong et al. 2025) and AReal (Fu al. 2025) decouple rollout and training via asynchronous strategies. Kimi K2 (Team et al. 2025) introduces comprehensive RL infrastructure, optimizing across computing clusters, colocated architectures, and efficient en- gine switching. Furthermore, CPPO (Lin et al., 2025) and GRESO Zheng et al., (2025b) facilitate end-to-end speedups through different clipping strategies, while TreePO (Li et al., (225a) utilizes tree-structured sampling to improve efficiency. Despite these system-level advancements, the core bottleneck which is inefficient autoregressive rollout generation remains largely unaddressed.
7 CONCLUSION
In this paper, propose concurrency-aware speculative decoding framework with online draft learning to accelerate GRPO. Our method dynamically adjusts decoding parameters based on con- currency levels, maximizing efficiency during high-variance generation, while online joint learning maintains strong alignment between draft and target models amid parameter updates. This dual strategy sustains high token acceptance and computational efficiency throughout training. Experi- ments across multiple models and mathematical reasoning benchmarks show consistent end-to-end speedups of 2.35x to 2.72x, outperforming baseline approaches.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Acala.
Sharegpt v4.3 unfiltered cleaned split, 2023. URL https://ugginffaee co/
datasets/Aealal haaaa//aaaaarPPPP Vicuna Lttpr_aiiaaaaaaaaaaaaaaee
wnt.lteere//bloo//aa////aaaaeree _V.
htw.tteeeed_ceeeee________e_ satpeddd__p...._.e json
Mark Chen. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,
2021.
Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization. arXiv preprint arXiv:2505. /2346, 2025a
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hao Wang, Haiyu Wu, Huayu Li, Aristeidis Sotiras, Yalin Wang, and Abolfazl Razi. Dra-grpo: Exploring diversity-aware reward adjustment for rl-zero-like training of large language models. arXiv preprint arXiv:2505.0965.. 2025b.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.025446 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14163 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with cape: A low-hassle method accelerate speculative decoding. arXiv preprint arXiv:2402.020022 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Lishui Fan, Yu Zhang, Mouxiang Chen, and Zhongxin Liu. Posterior-grpo: Rewarding reasoning processes in code generation. arXiv preprint arXiv:2508.05177 2025.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint wxxiv25505.24298 2025.
Lujun Gui, Bin Xiao, Lei Su, and Weipeng Chen. Boosting lossless speculative decoding via feature sampling and partial alignment distillation. arXiv preprint arXiv:2408. /5562, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint
02031, 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. journal of machine learning research, 18(187):1-30, 2018.
Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion Ilms via adaptive parallel decoding. arXiv preprint arXiv:2506.004/ 2025.
Sebastian Jaszezur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojeiech Gajewski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers, Advances in Neural Information Processing Systems, 34:9895-9907, 2021.

---

Published as a conference paper at ICLR 2026
Jakub Kurzak, David A Bader, and Jack Dongarra. Scientific computing with multicore and accel- erators. CRC Press, 2010.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023,
Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, et al. Treepo: Bridging the gap of policy optimiza- tion and efficacy and inference efficiency with heuristic tree-based modeling. arXiv preprint arXiv:2508. 17445, 2025a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16852 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401. 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceler- ation of large language models via training-ting time test. arXiv preprint arXiv:2503.018 2025b.
Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-basee reasoning models. arXiv preprint urXiv:2503.23342 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078. 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Youssef Mroueh, Nicolas Dupuis, Brian Belgodere, Apoorva Nitsure, Mattia Rigotti, Kristjan Gree- newald, Jiri Navratil, Jerret Ross, and Jesus Rios. Revisiting group relative policy optimization: Insights into on-policy and off-policy training. arXiv preprint arXiv:2505.222577 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-277443 2022.
Sihyeong Park, Sungryeol Jeon, Chaelyn Lee, Scokhun Jeon, Byung-Soo Kim, and Jemin Lee. A survey on inference engines for large language models: Perspectives on optimization and effi- ciency. arXiv preprint drXiv:2505.01658 2025.
Alexandre Piché, Ehsan Kamalloo, Rafael Pardinas, Xiaoyin Chen, and Dzmitry Bahdanau. Pipelin- erl: Faster on-policy reinforcement learning for long sequence generation. arXiv preprint arXiv:2509. 19128, 2025.
Maxime Robeyns and Laurence Aitchison. Improving llm-generated code quality with grpo. arXiv preprint arXiv:2506.022 02211, 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Noam Shazeer. Fast transformer decoding: One write-head all you need. arXiv preprint
arXiv:1911.02150, 2019.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning. arXiv preprint arXiv:2508.09776, 2025.
12

---

Published as a conference paper at ICLR 2026
David So, Wojeiech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoe Le. Searching for efficient transformers for language modeling. Advances in neural information processing systems, 34:6010-660 6022. 2021.
Ruslan Svirschevski, Avner May. Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive IIm inference consumer de- vices. Advances in Neural Information Processing Systems, 37:16342-16366 2024.
Hongze Tan, Jianfei Pan, Jinghao Lin, Tao Chen, Zhihang Zheng, Zhihao Tang, and Haihua Yang- Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint D.Xiv:2508.04349 2025.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Yepeng Weng, Dianwen Mei. Huishi Qiu, Xujie Chen, Li Liu, Jiang Tian, and Zhongchao Shi. Coral: Learning consistent representations across multi-step training with lighter speculative drafter. arXiv preprint arXiv:2502. 16880, 2025.
Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual perfor- mance model for multicore architectures. Communications of the ACM, 52(4):65-76, 2009.
Yuhao Wu, Yushi Bai, Zhiqiang Hu, Roy Ka-Wei Lee, and Juanzi Li. Longwriter-zero: Mastering ultra-long text generation via reinforcement learning. arXiv preprint arXiv:2506.188411 2025.
Bin Xiao, Lujun Gui, Lei Su, and Weipeng Chen. Clover-2: Accurate inference for regressive lightweight speculative decoding. arXiv preprint arXiv:2408.00264 2024.
Le Xu, Hong An, Junshi Chen, and Pengfei Zhang. high-performane matrix multiplication on the new generation shenwei processor. In 2022 IEEE 24th Ius Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Inl Conf on Smart City; 8th Int Conf Dependability in Sensor, Cloud & Big Data Systems & Application pp. 894-903, IEEE, 2022.
CMPCC/DS//SarCCiyDepedddd
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint DrXi1:2505.09388 2025.
Charlene Yang, Thorsten Kurth, and Samuel Williams. Hierarchical roofline analysis for gpus: Ac- celerating performance optimization for the nersc-9 perlmutter system. Concurrency and Com- putation: Practice and Experience, 32(20):e5547, 2020.
Charlene Yang, Yunsong Wang, Thorsten Kurth, Steven Farrell, and Samuel Williams. Hierarchical roofline performance analysis for deep learning applications. In Intelligent Computing: Proceed- ings of the 2021 Computing Conference, Volume 2, pp. 473-491. Springer, 2021.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
13

---

Published as a conference paper at ICLR 2026
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du et al. Vapo: Efficient and reliable reinforeement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized represttttoss for speculative sampling. arXiv preprint 2XXi::2408./5666 2024.
Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng- Critique-grpo- Advancing IIm reasoning with natural language and numerical feedback. arXiv preprint arXiv:2506.031 03166, 2025a.
Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Edge-grpo: Entropy-driven grpo with guided error correction for advantage diversity. arXiv preprint arXiv:2507.21845 2025b.
Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lx, Shao- han Huang, Lei Cui, Qixiang Ye, al. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yugiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507. 18071, 2025a.
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai. Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for I1m reasoning via selective rollouts. arXiv preprint arXiv:2506.022177 2025b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a- alm-as-a-judge with mt- het-bench and chatbot arena. Advances in neural information processing systems. 36:46595 46623, 2023.
Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, al. Streamrl: Scalable, heterogeneous, and elastic rl for 1lms with disaggregated stream generation. arXiv preprint arXiv:2504.1590 15930, 2025.
14

---

Published as a conference paper at ICLR 2026
A THE OPERATIONAL INTENSITY OF ACCELERATORS
As mentioned in the main text, Ipeak
Peak FLOPS
Below we calculate the Ieaa for several
Memory Bandwidth
modern GPUs.
Table 5: The peak of several modern GPUs.
GPU
Peak BF16 Tensor Core (TFLOPS)
Memory Bandwidth (TB/s)
Ipeak
A 100 40GB PCIe x 16
312
1,555
200.6
A100 40GB SXM
312
1.555
200.6
A 100 80GB PCIe 16
312
1.935
161.2
100 80GB SXM
312
1,555
153.0
H100 SXM
1979
3.35
590.7
H100 PCIe
1513
3.026
500
H100 NVL
3958
7.8
507.4
H800 SXM
1979
3.35
590.7
H800 PCIe
1513
756.5
H200 SXM
1979
4.8
412.3
B100
3500
437.5
B200
2250
562.5
is obvious that /peak > IGEMM- This growing disparity reflects fundamental imbalance mod- emn GPU architecture design: while computational throughput has scaled rapidly through specialized co-design, memory bandwidth constrained by physical lim-
tensor units and algorithm-hardwe its-has not kept pace. As result, the peak operational intensity (Tpeak) increasingly diverges from the arithmetic intensity required by canonical operations such as GEMM (General Matrix Multipli- cation), indicating that most workloads will be memory-boum under naive execution.
This imbalance is not accidental, but rather a consequence of deepening hardware constraints at the physical layer. The bandwidth of off-chip memory interfaces, even with advanced HBM3e stacks, is approaching the limits imposed by signal integrity, power delivery, and thermal density in 2.51D3D packaging. As NVIDIA's architecture team has noted: "We are nearing the electrical and thermodynamic boundaries of how fast we can move data across a package. Future gains must come from smarter data reuse, on-chip caching, and sparsity-aware execution, not just wider buses." (NVIDIA, GTC 2024). This explains the architectural trajectory toward higher Lpeak: to sustain performance under bandwidth saturation, GPUs must extract more computation per byte transferred, effectively shifting the burden from raw bandwidth scaling to maximizing data locality and reuse.
Moreover, the introduction of structured sparsity (e.g., 2:4 sparsity in Ampere and beyond) and narrow-precision formats (FP8, INT4) enables GPUs like the H100, H200, and B100/B200 to oper- ate efficiently in regimes where /peak > 500, despite memory bandwidth improvements being sub- linear relative to FL.OPs growth. For instance, while the H200 doubles HBM bandwidth compared to the H100 (from 3.35 TB/s to 4.8 TB/s), its BF16 FLOPS remain unchanged at 1979 TFLOPS, result- ing in lower 4peak deliberate design choice to alleviate the memory bottleneck for large-model inference.
In this context, the sustained high arithmetic intensity of modern accelerators is not sign of bound- less scalability, but rather an adaptive response to the hard limits of interconnect physics. Fu- ture GPU generations will continue exhibit high Tpeak not because memory bandwidth can scale indefinitely, but because they witt-w the constraints of energy, area, and signal propaga- mion-maximize computational return every joule and every byte moved. This paradigm shift un- derscores the necessity of efficiency, not just peak performance, defines the frontier of acceleration.
co-design in the post-Dennard scaling era, where
B THE MEASUREMENT METHODOLOGY OF VERIFICATION
The theoretical compute-to -hapuuteto-bandwidth ba-ddwddddhhh ratio Ipaak is difficult to derive analytically due hardware- and implementation-spenn factors such as kernel operation, memory tiling, and SRAM constraints. Instead, empirically estimate the effective Cpeak-
15

---

Published as a conference paper at ICLR 2026
Algorithm 2 Empirical Measurement of Cpea
Require: Target model model, max batch size Bmax, warm-up iterations W, repetition count N Ensure: Estimated Prak
1: for B to Bmax do
2: Create imput: input_ids E 2Bx1
W do 4: Perform warm-up forward pass: outputs = model ids)
for w
5: Synchronize CUDA
6: for 1 to N do
7: Perform timing operation: oulpuls = model(inpuul.dde))
8: Synchronize CUDA
9: Append cur _tr_ to times
10: Plot times B
11: Detect knee/elhow point in the curve
12: return Blnee Cpeak
To measure this quantity, we perform the following experiment on given target model and accel- erator (e.g.. GPU): We evaluate the forward pass latency of the target model across varying batch sizes B. while keeping the input sequence length fixed at 1. Specifically, for each B E [1, Bmax], we feed the model batch of B identical single-token inputs (typically the EOS token), and measure the average inference time per forward pass after warm-up and multiple repetitions.
As increases, the operational intensity of the attention and feed-forward layers rises, leading to improved hardware utilization. However, beyond certain point-when the compute demand satu- rates available resources-the latency begins to grow obviously. This manifests as a clear knee point in the latency vs. B curve (as shown in Figure 6 Algorithm summarizes the measurement procedure.
It is worth noting that, since our draft model also adopts Transformer architecture with hidden and intermediate layer dimensions identical to those of the target model, the Cpeak of the draft model is the same that of the target model. Therefore, the draft model differs in architecture or scale, its own Cd must measured empirically. In such cases, the optimal Kdran. should satisfy Kärafe S Cdran/ B, where B is the current batch size, to ensure efficient parallel verification.
separate measurement is required. However, if
C RELATED DATASETS
This section briefly introduces the six datasets used in our study, serving as supplementary informa- tion for the main text.
ShareGPT-68K. ShareGPT-68K (Aeala 2023) dialogue dataset collected from user-shared conversations the ShareGPT platform. This cleaned split retains multi-turn human-Al interac- tions while removing low-quality or malformed entries, such as incomplete messages or non-text content. It provides diverse and realistic conversational samples, making it suitable for training or evaluating open-domain dialogue systems.
GSM8K. GSM8K (Cobbe et al. 2222 1) (Grade School Math 8K) is benchmark dataset consisting of 8,500 grade-school-les math word problems, each accompanied by human- written, step-by- step solution. The problems require multi-step arithmetic reasoning and are designed evaluate model's ability to perform reliable and interpretable mathematical reasoning in natural language.
SinpleRL--Ael-LevelBBee (Zeng al. 2025 is curated subset of the MATH dataset, containing problems at difficulty levels 3 to 5. It is specifically designed for studying the impact of data difficulty on zero-shot reinforcement learning and model generalization. This dataset has been used training and evaluating high perforrrfoeeecc models such as Mistral-
SinpleRL-Abee-Levetttee
Small-241 and various Qwen series models.
16

---

Published as a conference paper at ICLR 2026
0.015
0.018
0.017
0.015
0.015
0.014
Batch Size
Figure 6: Forward pass latency vs. batch size B for Qwen2.5-7B-Instruee on a GPU H800 SXM.
DAPO-Math-17K. DAPO-Math-17K Yu et al. 2025) is a large-scale mathematical reasoning dataset comprising approximately 17,000 challenging problems across diverse domains, including algebra, calculus, and discrete mathematics. The dataset emphasizes complex reasoning and is used assess the robustness and generalization of models in advanced mathematical problem solving. particularly under reinforcement learning settings.
MT-Bench. MT-Bench (Zheng et al. 2023) multi-turn dialogue evaluation benchmark that assesses the quality of language models conversational settings. It features a diverse set of multi- turn scenarios covering creative writing, reasoning, and role-playing. Responses are evaluated based on coherence, consistency, and instruction following, often using LLM-as judee protocols, making it a widely adopted metric for conversational capability analysis.
Hvma HumanEval (Chen] 2021 code generation benchmark consisting of 164 hand- written programming problems in Python. Each problem includes function signature, docstring, and test cases. Model performance evaluated by executing the generated code and measuring pass @1 accuracy-iie -ic., whether the solution passes all test cases. It is standard benchmark for assessing functional correctness code synthesis tasks.
D OTHERS
Average Acceptance Length (r). The average number of tokens accepted by the target model verification step. In dynamic batch settings, 7 is computed the total number of accepted tokens divided by the total number of sequence-level verification steps. The latter defined as the cumulative sum of the number of active (i.e., unfinished) sequences across all verification iterations. Specifically, the batch size at iteration is B,, and one verification pass is executed per active sequence, the total verification count increases by B..
17


Reviews:
[{"review_id": "IEXMsJiOvl", "reviewer": "Reviewer_1RsX", "summary": "This paper propose to use speculative decoing in the RL training process, especially to acccelerate the rollout process of GRPO. It then identifies two issues: 1) speculative decoding may achieve limited speedup when the batch size is large; 2) the accept length decline in the mid and later stage of RL training if we donot update the draft module. The authors propose to adjust draft strategies to build the draft path/tree based on the concurrency and add the draft module updates in the actor update process.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper address a key issue in RL training especially GRPO-like algorithm, and attempts to use speculative decoding to accelerate the major bottleneck-the rollout stage. Specifically,\n\n1. The paper propose a concurrency-aware way to adjust the N_draft, draft branching and draft depth, to balance the compute and memory, achieving the sweet spot and avoiding compute/memory bound.\n\n2. It add an online distillation for the draft module that uses KL divergence to update draft and actor accordingly.\n\n3. Good ablation study. It includes results w/ and w/o concurrency adjustment, online learning, and the need for draft pretraining.", "weaknesses": "1. Performance experiments missing. It is unclear whether using speculative decoding can leads to reduced accuracy in final performance. Even speculative decoding align with the original sample probability, but the difference between rollout/training may worsen and other issue may appear.\n\n2. The accept rate doesn't seem to improce significantly w/ and w/o online training shown in Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jerFk8ZaRb", "reviewer": "Reviewer_1Sgp", "summary": "This paper targets the severe inference-time bottleneck in Group Relative Policy Optimization (GRPO) for large language models (LLMs), where autoregressive generation dominates training time. The authors introduce FastGRPO, a concurrency-aware speculative decoding framework with online draft model learning specifically tailored for high-concurrency RL training. By dynamically tuning drafting/verification parameters based on concurrency and continuously adapting the draft model to the evolving target, the method achieves substantial end-to-end speedups (2.35x–2.72x) over standard and state-of-the-art baselines on various LLMs and mathematical reasoning datasets.", "questions": "The concurrency scaling formulas rely on empirically determined hardware constants (e.g., $C_{\\text{peak}}$ via the “knee” in Figure 6). How robust are these to fluctuating system loads, mixed-precision, and multi-GPU setups? Could you provide stricter sensitivity analysis or calibration guides?\n\nWould FastGRPO’s approach generalize to autoregressive tasks with less structural output (e.g., open-ended dialogue, story generation) or with non-standard reward distributions? Are there tasks where adaptation of the draft model actually hurts acceptance (e.g., under creative instruction-following), and how is that detected or mitigated?\n\nCan the authors report statistical variances or confidence intervals for the speedup/acceptance results in Tables 1, 3, and 4? Is the acceleration robust across multiple random seeds and different training restarts?\n\nFor the “draft model pretraining may be unnecessary” claim (Section 5.3 and Figure 5), could you clarify if there are scenarios (e.g., pathological/outlier domains) where failing to pretrain leads to persistent misalignment or poor acceptance rates?\n\nAre there any practical trade-offs or user-visible impacts on final model quality (e.g., accuracy, diversity, reward coverage) due to the continuous adaptation of the draft model vs. a static approach? Are there empirical results measuring possible “overspecialization” or drift toward the target policy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work pinpoints practical, under-discussed causes of inefficiency in LLM RL training—especially the interplay between sequence length variance, dynamic concurrency, and memory/compute bottlenecks. This is well exemplified in Figure 1, where the generation phase is shown to consume up to 98% of GRPO training time, emphasizing the criticality of accelerating this stage.\n\nThe concurrency-aware speculative decoding scheme is well-motivated and nontrivial, adapting speculative tree depth and verification in response to real-time batch concurrency (see Algorithm 1 and the design in Section 4.1). The operational intensity formulation (Equation in Section 4.1) is technically thoughtful, and the methodology is justified theoretically and empirically.\n\nThe online draft learning mechanism shows astute recognition of distributional drift, which is convincingly quantified in Figure 4 (not provided in this snippet, but referenced), and ablation results (Table 3) indicate a robust solution.\n\nThe experiments are broad and carefully chosen. The acceleration is confirmed across multiple models (Qwen2.5-7B-Instruct, Llama3.1-8B-Instruct, DeepSeek-R1-Distill-Qwen-7B, etc.) and three mathematical datasets of varying complexity.", "weaknesses": "Lack of direct comparison with very recent parallel or hardware-aware decoding approaches: Methods such as SpecExec (Svirschevski et al., 2024) and TurboAttention (Kang et al., 2025) are not discussed or compared, though they address related efficiency targets under high concurrency. This omission reduces clarity on FastGRPO’s relative standing among recent innovations.\n\nEmpirical results lack coverage beyond math/reasoning tasks: While Section 5.3 briefly covers a few general-purpose benchmarks (e.g., MT-Bench, HumanEval), the primary evaluations remain dominated by mathematical datasets. The method’s generality for non-mathematical generative RL tasks (e.g., dialogue, instruction following) is only partially tested and reported in a superficial manner.\n\nThe adaptivity logic for tuning $N_{\\text{verify}}$, $K_{\\text{draft}}$, and $L_{\\text{draft}}$ is somewhat heuristic and under-explored for edge cases: Although the operational intensity framework is principled, the parameters depend on empirically-set hardware constants and logarithmic scaling (see Section 4.1 and Appendix B), whose practical calibration details are not fully fleshed out for heterogeneous or less conventional hardware platforms.\n\nMathematical clarity, though good, could be improved: For example, the formula for $K_{\\text{draft}}$ in Section 4.1 is not explicitly linked to actual tree coverage efficiency, and the effect of $\\alpha$ is described more narratively than analytically. The stepwise impacts of speculative tree over-expansion (e.g., wasted computation) are mostly argued intuitively.\n\nExposition of experimental protocols needs crisper detail: For instance, Table 1 and Table 3 report impressive speedup, but the precise wall-clock measurement setup (system load, parallel jobs, handling of inter-job contention) is only mentioned for some cases and not consistently. The “draft learning is free” claim would be more credible with breakdowns of GPU utilization and timeline plots. Similarly, Appendix B details the measurement of $C_{\\text{peak}}$, but system variance or sensitivity to other workloads is not discussed.\n\nLimited discussion of failure or suboptimal regimes: The work reports major speedups, but does not adequately address situations where speculative decoding might degrade (e.g., for models with unstable or highly creative output distributions, overfitting of the draft, or pathological reward landscapes).\n\nAblation details and datasets could be better visualized: Table 3 is comprehensive, but for a work whose core is acceleration, per-epoch training time curves, acceptance lengths, or variance plots (vs. batch size or sequence diversity) would add depth. Figures focus mainly on speedup ratios or batch size, avoiding more granular runtime or alignment dynamics.\n\nSome related methods and baselines are not explicitly cited or compared. Notable omissions include adaptive parallel decoding (Israel et al., 2025) and task-specific alternatives for cache-aware masking or split-parallelism (Federici et al., 2025; Polisetty et al., 2025).\n\n\n\nNo explicit statistical reporting of significance/testing: Performance gains are substantial but confidence intervals, run-to-run variance, or significance tests are absent, which would be crucial in justifying claims of “consistent” or “robust” improvement.\n\nSpecific references to figures/tables as required:\n\nFigure 1 illustrates the overwhelming dominance of generation time across datasets and models (91–98%), which strongly motivates the acceleration focus of FastGRPO.\nFigure 3: Clearly demonstrates that naive speculative decoding fails to provide speedup at high batch sizes (with curves “below 1.0x”), while the proposed adaptive tuning (“Optimal”) yields robust, sustained acceleration as concurrency grows—a direct validation of the core claim.\nTable 1: Quantifies the end-to-end and generation-phase speedup over state-of-the-art speculative decoding methods (EAGLE-2/3, HASS), confirming FastGRPO’s efficacy in both overall and granular metrics across a set of LLMs and datasets.\nTable 3: Ablates online draft learning and concurrency-aware tuning, showing granular Gen SR/E2E SR ratios for different setups and revealing that both components are crucial for the full speedup.\nAppendix Figure 6: (noted in methodology) empirically motivates the critical batch size $C_{\\text{peak}}$ and provides a hardware-specific context for the adaptive scaling formulas introduced in Section 4.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T7em7bZb29", "reviewer": "Reviewer_Hu9Z", "summary": "This paper introduces a method to incorporate speculative decoding into GRPO to accelerate the rollout process in reinforcement learning. The authors argue that directly applying speculative decoding is inefficient because the original algorithm does not handle high concurrency. To address this, they propose FastGRPO, which improves concurrency efficiency, and further suggest updating the draft model online. Experiments show clear speedup gains, and the paper is generally well written with comprehensive coverage of datasets and models.", "questions": "See weaknesses.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a clear and important bottleneck in RL training. The motivation is sound, the method is technically reasonable, and the presentation is clear. The acceleration results are convincing, and the experimental setup is thorough and well-organized.", "weaknesses": "The main concern is that the paper does not analyze how the proposed approach affects RL performance. Both the method and the experiments focus solely on acceleration metrics, without considering the impact on learning quality. Rollout is a sampling process that benefits from high temperature to maintain data diversity, which is crucial for effective policy learning. It is unclear whether the proposed acceleration changes the diversity or quality of the collected trajectories. The authors should include analyses or experiments evaluating these aspects, such as reward comparisons or diversity statistics.\n\nIn addition, the experiments do not compare with other state-of-the-art RL acceleration methods, such as PipelineRL or similar parallel rollout techniques, which limits the strength of the empirical validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)