Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
Authors: Ziyan Wang
Zheng Wang
Xingwei Qu
Qi Cheng
Jie Fu
Shengpu Tang
Minjia Zhang
Xiaoming Huo
Venue: ICLR 2026 Poster (2026)
Keywords: LLM reasoning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Slow-Fast Policy Optimization (SFPO), a reposition-before-update method that stabilizes on-policy optimization for LLM reasoning.

Abstract: Reinforcement learning (RL) has become central to enhancing reasoning in large language models (LLMs). Yet on-policy algorithms such as Group Relative Policy Optimization (GRPO) often suffer in early training: noisy gradients from low-quality rollouts lead to unstable updates and inefficient exploration. We introduce Slow-Fast Policy Optimization (SFPO), a simple yet efficient mechanism to address the above limitations via decomposing each iteration into three stages: a short fast trajectory of inner steps on the same batch, a reposition step to control off-policy drift, and a final slow correction. This reposition-before-update design preserves the objective and rollout process unchanged, making SFPO plug-compatible with existing policy-gradient pipelines. Extensive experiments demonstrate that SFPO consistently improves stability, reduces number of rollouts, and accelerates convergence of reasoning RL training. Specifically, it outperforms GRPO by up to 2.80 points on math reasoning benchmarks. It also achieves up to 4.93$\times$ fewer rollouts and a 4.19$\times$ reduction in wall-clock time to match GRPO’s best accuracy.

Published as a conference paper at ICLR 2026
SLOW-FAST POLICY OPTIMIZATION: REPOSITION-
BEFORE-UPDATE FOR LLM REASONING
Ziyan Wang of
Zheng Wang?1 2.
Xingwei Qu Qi Cheng
Jie Fu²
Shengpu Tang"
Minjia Zhang² Xiaoming Huo
Georgia Institute of Technology University of Illinois Urbana-Chane Uihana-Champaign 3 University of Manchester (wzy, huo Sggatech.edu
NVIDIA 5 "Iddependent .Emory University
(zhengw10, minjiaz] @uuicc.uuuuu... edu
t Equal contribution
ABSTRACT
Reinforcement learning (RL) has become central to enbancing reasoning in large language models (LLMss. Yet on-policy algorithms such as Group Relative Policy Optimization (GRPO) often suffer in early training: noisy gradients from low- quality rollouts lead to unstable updates and inefficient exploration. We introduce Slow-Fast Policy Optimization (SFPO), simple yet efficient framework to address the above limitations via decomposing each step into three stages: a short fast trajec- tory of inner steps the same batch, reposition mechanism control off- off-pollcy drift, and a final slow correction. This reposition-before upddte design preserves the objective and rollout process unchanged, making SFPÓ plug-compatible with existing policy-gradient pipelines. Extensive experiments demonstrate that SFPO consistently improves stability, reduces number of rollouts, and accelerates con- vergence of reasoning RL training. Specifically, it outperforms GRPO by up to 2.80 points in average on math reasoning benchmarks. also achieves up to 4.93 fewer rollouts and an up to 4.19x reduction in wall-clock time to match GRPO's best accuracy.
INTRODUCTION
Large language models (LLMs) have recently achieved remarkable progress on complex multi-step reasoning tasks, especially mathematics problem solving and scientific question answering (OpenAI et al., 2024; DeepSeek-Al et al., 2025b). A central driver of these advances has been reinforcement learning (RL), which fine-tunes LLMs using reward signals tied to semantic correctness and solution quality. For example, Matp (Shao et al., 2024) demonstrates that Group Relative Policy Optimization (GRPO) can substantially improve open-source LMs on mathematical reasoning, while DeepSeek-RI (Guo et al., 2025) shows that RL signals alone can induce emergent reasoning behaviors across various domains.
Despite these successes, GRPO inherits structural inefficiencies that make training fragile in LLM reasoning. During the early stages of training, when rollouts are particularly weak or uninformative, stochastic rewards induce high- high-variance gradients that destabilize updates (Yu al., 2025a; Zheng et al., 2025; Dai et al., 2025; Shen et al., 2025). Compounding this, common practice in GRPO implementations is to use single optimization step per rollout batch, and such a one-shot update tends to produce noisier, less reliable step update direction, especially at the early training stages, thereby underutilizing each batch and discarding information that could be further exploited (Schulman et al., 2017; Engstrom et al., 2020). Although the original GRPO paper (Shao et al., 2024) explored reusing rollout data, our findings suggest that simply applying this off-policy update to the policy model can degrade performance training proceeds, and how to use it effectively to build strong reasoning abilities in LLMs with satisfying sample efficiency remains an open question.
Project website is available at https::///// h/sll//swwwww// hw//////wwaaatpo.... wit...i.oo..

---

Published as a conference paper at ICLR 2026
Rollout Generation
Stage Fast Trajectory
Stage Reposition
Stage Slow Correction
41 q
4
qu
Policy Model Teg
Policy Model 7 1D
Policy Model gu
ge 6t afor ont)
Policy Gradient Update x 1
Pollcy Gradient Update K
Policy Model JT.
Pollcy Model f..
Figure 1: Pipeline of SFPO iteration Starting from the current policy Tæ..o, first generate rollouts for training. Stage I (Fast Trajectory): apply K successive gradient updates on the same batch obtain 0=.K Stage II (Reposition): interpolate between 0z. and the starting point 0%.0 to form O=.K controlling off-policy drift. Stage III (Slow Correction): perform one additional update on ge.K yielding Tg++1,0 for the next iteration.
In this work, introduce Slow-Fast Policy Optimization (SFPO), simple yet effective enhance- ment to on-policy volicy-gradien methods (e.g., GRPO) that stabilizes noisy updates and improves sample efficiency while leaving the base loss, rollout collection, and KL/clip regularization unchanged. Specifically, each training step is restructured into three coordinated stages: (i) fast-multiple inner updates on the same batch stabilize the search direction; (ii) reposition intoonooaitttoooiiiinn back toward the starting point to control off- off-policy drift; and (iii) slow an extra gradient correction to align with local curvature. This reposition-befort before-update design transforms noisy one-shot updates into structured trajectories, yielding more stable optimization, higher sample efficiency, and faster convergence. We empirically validate the efficacy of SFPO through extensive experiments across five models and six reasoning benchmarks. The evaluation shows that SFPO outperforms GRPO by up to 2.80 points in average on math reasoning benchmarks. It also uses up to 4.93x fewer rollouts while reducing wall-clock time by up to 4.19x x reach GRPO's best accuracy. We summarize our contributions as follows:
We propose SFPO, a plug-compatible update rule that reuses rollout batches via a fast-reposition-sllow decomposition, with theoretical insights for each stage.
SFPO introduces no changes to the underlying objective, rollout generation, or regularization, enabling drop-in integration into existing LLM reasoning pipelines.
Through extensive experiments on math reasoning benchmarks, we show that SFPO consis- tently improves stability, sample efficiency, and convergence speed over GRPO.
PRELIMINARIES
Group Relative Policy Optimization (GRPO) (Shao et al., 2024) a policy gradient method that has been widely applied in large-scale LLM training pipelines. GRPO dispenses with an explicit value function and instead normalizes rewards across a group of responses to the same prompt. Given an input q, the policy Te generates candidate sequences 1. with corresponding rewards (rx) G The normalized advantage for each candidate is defined as
mean (fr:) A,
sr.((r))
This group-based normalization can be interpreted as a form of reward shaping: by emphasizing relative differences among candidate sequences for the same input, GRPO strengthens the reliability of the gradient signal. Instead of embedding a KL penalty inside the reward, GRPO directly regularizes the policy by including explicit KI. term between the learned policy and reference policy. The
2

---

Published as a conference paper at ICLR 2026
training objective is
JGRPO P0.
6
02
(in(
- -, ,, +)) - 3D [Tel
(2)
1=1
where (0) Teaa (1, is the token-level importance ratio between the new and old policies, é is the clipping range that prevents overly aggressive updates, 8 controls the KL regularization strength, and Tef is fixed reference policy. Here, q denotes the input prompt, generated sequence, and Oiee its 1-th token.
Limitations of GRPO. Despite its popularity, GRPO inherits structural drawbacks from the underlying on-policy update rule. At each iteration, the parameters are updated via single stochastic gradient step estimated from batch of rollouts. The randomness of rewards leads to high- variance gradient estimates, making updates unstable. Group normalization partially dampens this effect but remains sensitive fluctuations within each batch. Moreover, restricting each batch to a single update discards potentially useful gradient information across inner steps, leading to inefficient data use and limited variance reduction. These drawbacks underscore the need for an update mechanism that can stabilize gradient directions while making more effective use of available samples.
3 METHOD
To address the instability and inefficiency of one-shot policy updates, we propose Slow-Fast Policy Optimization (SFPO), a simple yet general update mechanism that reuses rollouts more effectively while remaining comp-otlaeibee with standard on-policy policy gradient algorithms. Each iteration consists of three stages: a fast trajectory of multiple inner updates on the same batch, (ii) reposition step that interpolates back toward the on-policy point to control drift, and (iii) slow correction via an extra-gradient update. As illustrated in Fig. I and Alg. I, this fast-reposition design transforms noisy one-shot updates into well-structured update trajectories, yielding more stable optimization and higher sample efficiency without additional rollouts.
Notation. To align with standard gradient-based optimization notation, we define L(0) (()) that maximizing the objective J is equivalent to minimizing the loss L.
3.1 STAGE I: FAST TRAJECTORY
In standard on-policy policy-gradient methods such as GRPO, each outer iteration is updated by single stochastic gradient:
g-+1 = -
(3)
where (oL( is estimated from batch of rollouts. Such one-shot updates suffer from high variance and often drive the policy in unstable directions, especially during early training. SEPO mitigates this by performing multiple inner updates on the same batch rollouts.
Formally, starting from parameters g², at the beginning of iteration 8, we execute a short fast trajectory of K inner updates:
g+ = ga.k - V
k K-
(4)
This produces a sequence 8=.0 + Hr. + ... - grK where each step refines the gradient direction using the same rollout data.
Intuition. Unlike one-shot updates that rely on a single noisy gradient, the displacement
K-1
00.0
VoL(Or))
captures the cumulative effect of K sequential corrections. Even some individual steps are perturbed by noise, their composition tends to damp idiosyncratic fluctuations and align with the underlying
3

---

Published as a conference paper at ICLR 2026
gradient direction. Geometrically, this can be viewed as integrating the local gradient field along short trajectory in parameter space rather than trusting a single noisy vector at 6%.0. As a result, the update direction at the end of Stage I is typically more stable and less sensitive to randomness in any single gradient estimate.
Compact Mathematical Intuition. Let 0₂ == 0².h and consider the K inner steps 0k+1 0g- (((()) starting at 0g 0=.0. In small neighborhood, linearizing the gradient field, VL(0) = Do + Ho(O fo) with Do V and Ho = V²L((0), yields the closed-form displacement
0K 0o - (1- 11 90.
(6)
where Hy denotes the (pseudo)invvrrse on the range Hu (the X case is understood by continuity). Spectrally, along an eigen-direction with curvature 2 O, the scalar gain is (1 - (1 - .)))*)/ /A: small À it behaves like Kn (steadily accumulating progress in gentle directions), while for larger À saturates below 1 (damping stiff-directinn oscillations). Thus the fast trajectory acts as a curvature- aware low-pass filter that stabilizes the endpoint direction relative to one-shot step. We use this as local intuition under sufficiently small step size (e.g. < 1/|] Hullz) and neighborhoods where positive-cureee curvature directions dominate; in practice KL/clip regularization and Stage Il reposition mitigate adverse effects of negative curvature.
/ for
3.2 STAGE II: REPOSITION
While the fast trajectory of Stage improves stability, it also changes the nature of the update from on-policy to off-policy. Since all inner steps 0s.1 g=.K reuse the same rollouts generated at ge.0, the endpoint 8 no longer corresponds to the distribution that produced those samples. This distribution mismatch is a fundamental drawback of off-policy learning, as it biases gradient estimates and can destabilize training.
Inspired by Lookahead Optimization (Zhang et al., 2019), SFPO introduces a reposition step that interpolates the fast trajectory back toward its starting point:
@s.K 0000 a( 0*."),
10.
Here a regulates the degree of off- off-policy drift: smaller values keep the update close the original on-policy iterate, while larger values rely more on the fast trajectory at the risk of greater mismatch.
Intuition. The interpolation is equivalent to solving linearized proximal subproblem around 6*.0;
K
VoL(O)))) -800) + 2 - 00.00
min
(8)
whose unique solution coincides with gs.k for X ag Thus, acts implicit trust-region radius: smaller a implies larger proximal weight X, enforcing stronger contraction toward the on-policy point."
3.3
STAGE III: SLOW CORRECTION
After repositioning, SFPO applies more (slow) correction step at the interpolated point:
0++1 O..K Vo
This yields predictor-correet structure: Stage produces a stabilized fast trajectory, Stage II tempers off-policy drift via reposition, and Stage III applies slow correction aligned with the local curvature the update point.
"A heuristic alternative is obtained one replaces Σg VoL(B**) by the averaged gradient and uses the approximation 0*.K go.0 -n 7. leading À x 1/(onK).
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 SFPO: unified fast-reposition-sllw update.
Require: Initial policy Tgo.o, dataset D. hyperparameterss S, K, oo: 7, w, T, loss (())
Initialize rolling buffer H +- 0, stats (1, σ)) + (0, 1), trigger index s* <- +00
for s = 0, I.....,..
1 do
Generate rollouts with the current policy Tg..o prompts from D. a - 0g [[ 8 < s*1 // Set a for this iteration from past trigger (non- )
if 0 then
0s.K + 00,0 // Skip fast trajectory & reposition
else
for kt = 0, 1, b....... K do gs gerk - r)VoL(*) Stage I: Fast Trajectory
end for
8a. t 80.0 + aag 00.0)
// Stage II: Reposition
end if
0=+10 + j=.K -V // Stage III: Slow Correction
Compute entropy Hςi update rolling buffer H. (keep last w ones) and (Ma 0g).
Z, -,- for numerical stability)
o,te
if +00 and [Z,] 2 r then
S- +
will set a = 0 for all future s' 2 S*
end if end for return final policy Tgs.0
Theoretical intuition.
Under I-smoothness and sufficiently small y,. the descent lemma implies B[C(O++])] S C(0*0) -all/)))nn/V///// cnl|V[((*:0)] + L
F(K, F(,,LL
(10)
where F(K, a) represents the combined effect of Stage I (fast trajectory length K) and Stage (reposition factor a).. Intuitively. F(K,a) a) reflects a balance between exploiting more gradient information and controlling distributional drift: increasing leverages the same rollout data across multiple steps but also amplifies off-policy mismatch, while larger interpolates more aggressively toward the fast trajectory at the risk of greater instability. practice, since increasing K raises both wall-clock cost and F(K, a), we adopt small K with moderately large while the slow correction in Stage III mitigates overshooting and preserves progress along the stabilized trajectory direction.
3.4 SCHEDULING &
nonzero is essential to exploit the stabilized fast trajectory when K >> O: if O, the reposition collapses to fa.0 and the fast trajectory is discarded, eliminating the benefit of Stage I. However, the same aggressiveness that helps early can be counter-productee is large, a nonzero accelerates progress by moving along the stabilized fast direction; but as we approach minimum, the signal weakens while curvature and stochastic noise dominate, so large a amplifies drift and instability. This motivates an adaptive schedule for a.
near a minimizer. When [|
Why adapt a rather than K? K controls both stabilization and wall-clock runtime: increasing K reduces oscillations but incurs proportional compute cost, making mid-training changes impractical. By contrast, a is a soft trust parameter: decides how much of the stabilized fast trajectory is exploited, without changing runtime or discarding the already-computed K steps. Thus is the natural lever to adapt dynamically.
Adaptive rule in practice. In our implementation, a is scheduled online at each iteration. After generating rollouts with the current policy, we compute the policy entropy H, and maintain a rolling buffer of the past entropy values (window size w). Let μ4, and o, denote the mean and standard deviation within this buffer. We define the one-sided z-score
Z, H.-
(11)
/. 2 7 for a threshold r, we mark the current step s* and set a = for all s 2 s*. Otherwise we keep Ogg- Intuitively, sharp entropy fluctuations signal that the policy is close to local optimum
5

---

Published as a conference paper at ICLR 2026
where noise dominates, interpolation should be disabled. This setropy-triggered schedule exploits the fast trajectory in the high-signal early phase, while reverting to pure on on-policy updates near convergence for stability."
3.5 UNIFIED SFPO UPDATE
Collecting the three stages, the unified SFPO update for each step is:
K- VoL(Os
0++1 0000 1
VoL(O*.))
(12)
slow correction
fast trajectory & reposition
As shown in Alg. 1, SFPO unifies the three stages into a single update rule and serves as plug- compatible drop-in replacement for on-policy policy-gradient methods such as GRPO. Our exper- iments show that this structural change consistently improves stability and sample efficiency on diverse math reasoning benchmarks, with minimal engineering overhead.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Models. We conduct the reasoning rl training with our proposed SFPO on wide range of models: Qwen2.5-Math-I1 (Yang et al., 2024). DeepSeek-R1- -i------iD--ii--ee 1.5B (DeepSeek-Al 2025a), Qwen3-4B-BB -ase (Yang et al., 2025), @wwe2.5-Math..
al., (Yang et al., 2024), and DeepSeek- we set the context
RI-Distill-Qwen-77 (DeepSeek-Al) et al., 2025a). For Sven2.5-Math-1.5B/t length in the training process as 4096 the same as their maximum context length; while for Qwen3- BB-Base and DeepSeek-RR Boocck-R--Dssillll D--isttill-Qwen--- 1.5B/7B we set the context length as 8192 for the better accuracy-efficieng tradeoff.
Reasoning RL Training. We conduct the reasoning RL training on two different training datasets to test the effectiveness of SFPO different scales of dataset. The first is the combination of DAPO training dataset (Yu et al., 2025a) and Math training dataset (Hendrycks et al., 2021), which is total of approximate 24K data. The second one is the Skywork-OR1 Math RL training dataset (He et al., 2025b) with 105k data. The training batch size is set to 256, and the number of responses for each question is set to 8 by default. The total trianing step is set 400 by default. All the training experiments are done based on verl (Sheng et al., 2025) with a single 8-GPU node.
Baseline and Evaluation. We compare SFPO with vanilla GRPO and the base model without rl training on six commonly used mathematical reasoning benchmarks with variant difficulty: Math500 (Hendrycks et al., 2021), AIME24 (Art of Problem Solving, 2024a), AIME25 (MAA), AMC (Art of Problem Solving, 2024b). MinervaMath (Lewkowycz et al., 2022), Olympiad Bench (He al., 2024). Each benchmark evaluated multiple times with rollout temperature being 1, and we report the average Pass@1 accuracy by default.
4.2
MAIN RESULTS
4.2.1
MATH REASONING BENCHMARKS
As illustrated in Table our proposed SFPO consistently outperforms vanilla GRPO across all base models and benchmarks. Specifically, for small-scale models such as Qwen2.5-Math-11. and DS-distilled--wenn-l...
SFPO demonstrates superior performance enhancements on math reasoning benchmarks, raising the average accuracy from 38.35 to 40. 19 with an absolute gain of +1.84, and from 47.73 50.53 with gain of +2.80, respectively. The improvements are particularly pronounced on challenging tasks such as AIME24 and AIME25, where DS DS--dstilled--weeree 1aal--een-15555 1,5B achieves an absolute gain of +7.5 on AIME25. The larger models also exhibit similar performance
AAthhough decaying schedule on could be a formal solution, we find little empirical difference, as shown in Fig. 7 of Sec. 5. Hence, for simplicity and efficiency, downgrade SFPO GRPO once the iteration reaches 8"

---

Published as a conference paper at ICLR 2026
Table 1: Performance on math reasoning benchmarks with DAPO and Math training dataset.
Model
Method
Math-500
AIME24
AIME25
AMC
Minerva
Olympiad
Avg
Base
55.55
9.17
37.65
17.74
28.45
25.73
Qwen2.5-Math-155B
GRPO
77.15
16.67
11.67
53.31
31.89
39.42
38.35
SFPO
78.35
20.00
15.00
56.02
32.07
39.72
40.19
Base
71.65
21.67
9.17
53.61
27.02
38.54
36,94
Qwen2.5-Math-7B
GRPO
82.50
34.20
20,83
71.08
36.76
44.80
48.36
SFPO
82.30
35.00
20.83
74.49
36.94
45.59
49.19
Base
73.80
16.67
20.00
47.59
28.86
36.94
37.31
DS-distilled--wee--l..
GRPO
84.65
30.00
23.33
66.86
31.71
49.85
47.73
SFPO
86.10
32.50
30.83
70.28
32.81
50.67
50.53
83.60
61.75
41.73
48.89
48.21
DS-distilled-Qwen--B
GRPO
50.00
43.65
61.24
60.47
SFPO
54.17
44.49
65.73
Base
45.25
2:50
0.83
20.48
15.99
20.66
17.62
Qwen3-41B-Ban
GRPO
83,35
16.67
17.50
59.03
38,78
48.59
43.99
SFPO
84.30
21.67
20.83
57.23
40.81
48.67
45,59
gains. For 1rren.5--Math-77 SFPO raises the average accuracy from 48.36 to 49.19 with absolute gain of +0.83. For DS DS-distilled-QU Sotilledd-we--7B- 7B. SFPO boosts the average accuracy from 60.47 absolute gain of +2.57. For Dwen3-4B-Ban model, SFPO improves
to 63.04, corresponding average accuracy from 43.99 to 45.59, an absolute gain of +1.60, highlighting its robustness across various models. Moreover, Table demonstrates that SFPO can consistently achieve better training performance compared to GRPO for the larger training dataset Skywork-orl, proving the robustness of SFPO across different scales and distributions of training datasets.
4.2.2 TRAINING DYNAMICS.
We compare the training dynamics between SIPO and GRPO to better understand the differences in optimization behavior as shown in Fig. 2 and Fig. 3.
Table Performance on AIME24/25 with
Skywork-orl training dataset.
Model
Method
AIME24
AIME25
clearly spot that SFPO consistently outperforms GRPO across all base models throughout the training process. Not only does SFPO achieve faster convergence in the early stages, but it also sustains higher global accuracy by the end of training. For example, Qwen3-4B- Base model achieves sharper rise and stabilizes higher accuracy within only 150 training steps, while vanilla GRPO cannot surpass this accuracy even after 400 steps.
Validation. From Fig. 2,
Base
20.40
17.90
DS-Qwen-1.5B
GRPO
32.92
25.83
STPO
34.17
27.50
Basc
25.42
22,92
DS-Qwen-7B
GRPO
42.50
29.20
SIPO
43.75
30.00
Response Length. Moreover, distinct training behaviors between SFPO and GRPO for DeepSeek- RI-Distill-Qwen--7 are shown Fig. including the response length, entropy loss, and the reward throughout the training process.
Specifically, GRPO gradually collapses overly short responses while SFPO quickly converges to stable range of around 2700 tokens with better accuracy. highlighting SFPO's ability to regulate response length more effectively and avoid overthinking with verbose responses (Liu al., 2025).
Entropy. From Fig. 3(b), we can observe that SFPO makes model's entropy loss lower compared to GRPO. Typically, lower entropy means weak exploration ability for reasoning models; however, the entropy reduction under SFPO mainly reflects the model's ability to eliminate unproductive search paths early, rather than suppressing exploration altogether as evidenced by its sustained accuracy gains. In fact, the model still explores sufficiently broad set of reasoning trajectories; therefore. the lower entropy observed under SEPO should be viewed as a sign of more efficient exploration rather than sign of limited exploration.
7

---

Published as a conference paper at ICLR 2026
Qwen2.5-Math-
DS-RI-Distill-- Qwen-1.58
Qwen3-4B-Base
DS-RI-Disttll-Qwemm
Accuracy
0.36
0.38
Aver Pverage Accuracy
0.50
0.48
Average Accuracy
0.45
Average Accuracy
0.60
0.62
0.48
0.58
0.34
0.32
0.44
0.35
240 Stops
0.56
Steps
0.30
0.42
0.28
0.40
0.52
0.26
0.38
0.25
0.50
Di--5
0.48
100
200
300
400
0
100
200
300
400
Training Step
Training Step
Training Step
Training Step
(a)
(b)
(c)
(d)
Figure 2: Average validation accuracy of different base models throughout learning process.
Mean Response Length
Entropy Loss
Reward
0.85
SFPQ
SFPO ERPO
Response Length
4500
GRPO
Reward Score
4000
Entropy
0..4
3500
0.3
3000
0.2
2500
XPRO ORRO
100
200
300
400
0
100
200
300
400
100
200
300
400
Training Step
Training Step
Training Step
(a)
(b)
(c)
Figure 3: Comparison of training behaviors in terms of response length, entropy loss, and reward.
Reward Score. SFPO also achieves higher and more stable reward throughout the training process compared GRPO, indicating stronger alignment with the reward function and robust convergence. This is further reflected in its superior accuracy and well-controledd response length.
4.2.3 EFFICIENCY ANALYSIS.
GAPO
SFPO (Curs)
We evaluate the efficiency gains of SFPO over GRPO by comparing the total number of rollouts and
wall-clock time required to reach the same benchmark accuracy. The results in Fig. 4 illustrate that SFPO consis- tently outperforms GRPO in both rollout ef- ficiency and training time across all model scales. To be specific, SFPO requires 3.21x, 3.50x, and 4.93x fewer rollouts than GRPO for DS-Qwen-1.5BB Qwen3-4B-Base, and DS-Qwen-7B, respectively, to reach the same best accuracy. This advantage directly trans- lates into reduced training time, where SFPO
G0Tx] sonojoo
paatiin Tione Aamonn
2.50x
2.62x
Train
(a)
(b)
Figure 4: Comparison between GRPO and SFPO. (a)
Number of rollouts required achieve the same best
accuracy as GRPO. (b) Corresponding training time.
achieves 2.62x, 2.65x, and 4. 19x speedups over GRPO the same models, significantly lowering the training cost. Note that SFPO does not introduce extra GPU memory overhead as it does not need to store the heavy optimizer status. The detailed profiling results for GPU memory usage in the training process can be found in Appendix B. These significant efficiency gains align with expectations, since the primary bottleneck in the training process lies in rollout generation, which accounts for more than 70% of the overall inference time (He al., 2025a). By substantially reducing the number of rollouts required and harnessing the reposition mechanism. SFPO alleviates this bottleneck and achieves faster training.
5 ANALYSIS AND ABLATION STUDY
Impact of a and K. As discussed in Sec. 3, the hyperparameters and K jointly determine the stability of SFPO. To better understand their impacts, we study two settings: small K-3 and large K=7, under varying a, shown in Fig. When K is small, SFPO remains stable across different values and consistently outperforms GRPO. This aligns with our theoretical intuition that performing multiple inner updates on the same rollout batch, followed by reposition, effectively reduces gradient
8

---

Published as a conference paper at ICLR 2026
GRPO
0202
2-04
0-06
1:0
K=3
K=
Impact K
Impact of Stage III
Average Accuracy
Average Accuracy
0.50
Average Accuracy
0.50
Average Accuracy
0.48
0.48
0.48
0.40
0.46
0.46
0.46
0.44
0.42
0.42
0.42
0.40
0.40
0.38
0.38
200
50
100
150
20c
100
200
600
50
100
150
200
Training Step
Training Step (b)
Training Step
Training Step
(a)
(c)
(d)
Figure 5: Average training accuracy of different settings throughout the training process. (a): Small
k=3 with varying values of a. (b): Large k=7 with varying values of a. (c): Varying values of k with fixed = 0.8. (d): Impact of the existence of stage III.
Average Accuracy
GRPO
SFPO w/ EC
SFPO w/O EC
GAPO
SFPO w/ Horr Stop
SFPO w/ Gradual Decline
Average Accuracy
0.60
2.00
Entropy Loss
0.50
1.50
0.58
0.40
0.55
0.55
1.00
0.30
0.50
0.53
0.50
0.53
0.50
0.20
0.10
0.00
100
200
100
200
50 100 150 200 Training Step
Training Step
Training Step
Training Step
Figure 6: Comparison between SFPO w/ and
Figure 7: Comparison between SFPO with dif- ferent a decay strategies. The blue dashed line indicates the stop step identified by z-score.
w/o entropy control (EC). The blue dashed line indicates the stop step identified by z-score.
noise and produces more reliable update direction. However, when K is large, the fast weights drift substantially from the original parameters. A large a then amplifies this mismatch by pulling the slow weights too aggressively toward unstable fast trajectories, injecting noise and causing performance drops. As shown in Fig. 5(b), smaller a mitigates this drift and restores stability, confirming the interaction between K and a.
Against Off-Policy Overfitting, When = 1 (no interpolation), performance initially rises but steadily declines training progresses. Without interpolation, the model rapidly adapts by overfitting to small batches of rollouts, yielding short-term gains but injecting growing noise into gradient updates, which causes instability and long-term degradation. This effect is amplified when K is large, as shown in Fig. 5(b), where the fast weights drift substantially and the slow weights fully adopt these noisy trajectories, leading to sharp performance drops. In contrast, for small interpolation keeps the update closer to the on-policy region, mitigating distributional drift and stabilizing training. However, if a is overly small and K small, the algorithm under- utilizes the stabilized fast trajectory, slightly slowing early improvements. While for large K, smaller is preferable, where interpolation effectively counteracts the severe off-policy drift induced by many inner steps. Overall, interpolation serves as simple, effective regularizer against the potential off-policy overfitting introduced by Stage I.
Interpolation
The Importance of Stage III: Slow Correction. As shown in Fig. 5(d), incorporating slow cor- rection consistently improves stability and accuracy over GRPO. Without this stage, the reposition stage leaves the iterate at an interpolated point that may deviate from the true descent direction. Intuitively, slow correction provides a curvature correct optimization trajectory. Beyond its immediate benefit, slow correction also provides a natural interface for future extensions: As discussed in Sec. 7, one could, for instance, perform an additional rollout on small held-out meta-test batch after repositioning, enabling curriculum or meta-learning variants of SFPO without altering its overall structure.
adjustment that realigns updates with the
Necessity of Entropy Control. We evaluate the effect of adaptive entropy control (EC) on SFPO using the DS-Qwen-7B model. As shown in Fig, 6, removing EC causes noticeable accuracy drop after roughly 100 steps. This degradation coincides with a rapid divergence of entropy loss, suggesting that the policy becomes unstable and overfits noisy rollouts. These results highlight entropy control as key factor for maintaining the stability and reliability of SFPO. For simplicity, implement entropy control through lightweight scheduling of a. As shown in Alg. 1, once

---

Published as a conference paper at ICLR 2026
predefined eatropy-triggee is activated, a is set to 0, reverting SFPO standard GRPO for subsequent steps. This simple mechanism effectively stabilizes training while retaining the efficiency gains.
Strategies for Decaying a. We further investigate alternative strategies for reducing the value of once the stop step is identified by the z-score criterion. Specifically, we compare two representative approaches: (i) our default method, where is directly set zero after the trigger step s*, and (ii) a more gradual linear decay schedule, where a decreases to zero over several subsequent steps. As illustrated in Fig. both strategies yield similar accuracy and stability curves, suggesting that the decay schedule itself has negligible influence on the overall performance of SFPO. This result implies that once the satropy-tiigge condition is met, the role of a becomes marginal, and any further adjustment has limited benefit. Consequently, for simplicity and wall-clock efficiency, we adopt the direct reset rule of setting a after 8*. The formal procedure is summarized in Alg. 1, and additional details can be found in Sec. 3.4.
RELATED WORKS
RL for LLM Reasoning, OpenAl 01 (OpenAl) introduced a paradigm shift in LLM reasoning by extending the reasoning horizon before final responses. DeepSeek-RI (Guo al., 2025) further advanced this line by open-sourcing both its training algorithm, the value-model-free GRPO (Shao et al., 2024), and model weights, achieving performance comparable to O1. Subsequent work has focused on stabilizing and simplifying GRPO: DAPO (Yu et al., 2025b) identifies entropy collapse as a key challenge and proposes effective remedies, while Dr. GRPO (Liu et al., 2025) removes normalization terms without sacrificing performance. In contrast, SIPO is orthogonal to both families: it leaves the objective unchanged yet restructures the update itself into a fast-reposition-slow decomposition, improving variance reduction, stability, and sample efficiency in LLM reasoning.
Data Efficiency for LLM Reasoning. Despite focusing on designing novel training pipelines, a complementary line of work (Ivison et al., 2025; Xia et al., 2024; Muennighoff et al., 2025; Ye al., 2025) improves the efficiency of LLM training through data filtering. One direction focuses on pruning data for supervised fine-tuning (Xia et al., 2024; Chen et al., 2023; Ivison et al., 2022). Another direction targets reinforcement learning, where studies (Muldrew et al., 2024; Liu al., 2024; Das et al., 2024; Li et al., 2025; Fatemi et al., 2025; Wang et al., 2025) show that GRPO requires only a small subset of the training data to improve reasoning ability. However, these methods largely optimize which data is used rather than how updates are performed. SFPO is orthogonal: it assumes no change to the data pipeline but restructures the policy update itself. By converting one-shot updates into a fast reposiiinn slow trajectory, SFPO reduces variance and stabilizes learning.
Off-policy GRPO. Original iterative GRPO (Shao et al., 2024) tries to update policy model for multiple times with the same batch of rollout data. Mroueh (2025) provided theoretical analysis for this off- off-policy policy gradient operation under verifiable rewards, reformulating GRPO as a KL-regularized contrastive loss and proving its iterative dynamics amplify success rates relative to the reference policy. Mroueh al. (2025) examined both on- and off-policy variants, deriving reward improvement conditions, introducing clipped surrogate objectives to stabilize training, and empirically showing off-policy GRPO can match or exceed on-policy performance. Unlike these modifications, SFPO preserves the on-policy GRPO update but restructures one-step optimization into a fast-reposition-sloo framework to tackle sample inefficiency and stabilize updates.
7 CONCLUSION
We proposed Slow-Fast Policy Optimization (SFPO), a simple three-stage update mechanism that stabilizes early training by combining fast trajectory, reposition step, and a slow correction. SFPO directly addresses the instability of one-shot GRPO updates, achieving higher reasoning accuracy and fewer generated tokens without added wall-clock cost. Beyond these gains, its modular structure naturally opens avenues for curriculum or meta-learning extensions, and our nauropy-rriggere a schedule suggests richer adaptive rules. We believe SFPO offers not only effective plug-in for current policy optimization, but also a foundation for scalable and aata-aware optimization strategies in the L.M reasoning area.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Huo is partially sponsored by a subcontract of NSF grant 2229876, the Russell Chandler IIII Professorship at Georgia Institute of Technology, an NIH-sponsored Georgia Clinical & Translational Science Alliance, and the Georgia Department of Transportation.
REPRODUCIBILITY STATEMENT
All results reported in Sec. 4 are fully reproducible. We will release the code and experiment scripts upon acceptance.
REFERENCES
Art of Problem Solving. Aime problems and solutions. httppps//aattffrrrollmmss/lt 2024a. Accessed: 2025-04-20.
com/ bickiiiiiex..iee
Art of Problem Solving. Ame problems and solutions. htttps:/artofproblle. www.ofrrrobemmoooiingg.
com/ wwkiiiiii////// wntiinidxxxi//m/ http//iinx..pppppp..ceppe 2025-04-22
2024b. Problems and Solutions, 2024b. Accessed:
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. arXiv preprint urXiv:2307.08700 2023.
Muzhi Dai, Shixuan Liu, and Qingyi Si. Stable reinforcement learning for efficient reasoning. arXiv preprint arXiv:2505. 18086, 2025. URL https: //axxiv orgoorg//aaoooo abss/////////// 2505 . 18086.
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint arXiv:2402.10500 2024.
DeepSeek-AL, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li. Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu. Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei. Yang Zhang, Yanhong Xu. Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in Ims via reinforcement learning, 2025a.
URL https: / /arxiv org/ abs/
/bs//5011 2501 12948.
11

---

Published as a conference paper at ICLR 2026
DeepSeek-AL, Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang
Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang. Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun. Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng. Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu. Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng. Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang. Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseck-v3 technical report, 2025b. URL htthtt:///// http//xxivvvvv. org/ abs / 22222412222 19437.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005. 12729, 2020.
Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05120 2025.
D. Guo et al. Deepseck-r:: Incentivizing reasoning capability in Imms through rl. Nature, 2025. URL https: //oii/ooi/////////// wwgg/o.......o.ooooooooo 1038 0115////41186622222 025-0005----000 0942222. To appear / in press.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. OlympiadBench: challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. /4008, 2024. URL https: / rxxiv. org/ abs 2402. 14008.
Jingkai He, Tianjian Li, Erhu Feng, Dong Du, Qian Liu, Tao Liu, Yubin Xia, and Haibo Chen. History rhymes: Accelerating Ilm reinforcement learning with rhymerl, 2025a. URL https:
//arxiv... www.wv.w...................ooeee abs/ 2500 18588.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner I technical report, 2025b. URL https: 22312.
//
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: / wttpvviivvv.o/ org/ alss 221... 221...1333........
Hamish Ivison, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Data-efficient finetuning using cross-task nearest neighbors. arXiv preprint arXiv:22/2.00/ 2022.
Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint arXiv:2503.0/887, 2025.
12

---

Published as a conference paper at ICLR 2026
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo. and et al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems (NeurlPS), volume 35, pp. 3843-3857, 2022.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling, arXiv preprini arXiv:2502.11886, 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Zijun Liu, Boqun Kou, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. Enabling weak Ilms to judge response reliability via meta ranking. arXiv preprint 2rX1v:200../214,, 2024.
MAA. American invitational mathematics examination (aime). https: /maa.or//aoaaaaa./////////
math-competite www--ooopeiiiinioo swantiiiiinnnniie...... Mathematics Competition Series; n.d.a.
Youssef Mroueh. Reinforcement learning with verifiable rewards: Gipo's effective loss, dynamics, and success amplification. arXiv preprint arXiv:2503. 06639, 2025.
Youssef Mrouch, Nicolas Dupuis, Brian Belgodere, Apoorva Nitsure, Mattia Rigotti, Kristjan Greenewald, Jiri Navratil, Jerret Ross, and Jesus Rios. Revisiting group relative policy optimization: Insights into on-policy and off poolcy training. arXiv preprint arXiv:2505.22257, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Perey Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 2025.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. arXiv preprint arXiv:2402.08/// 14, 2024.
OpenAl. Learning to reason with 1lms: Openai ol preview. URL
com/
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel -Shaadet-Shappro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage. Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim. Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko. Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély,
13

---

Published as a conference paper at ICLR 2026
Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' OeKeefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Πya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang. Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https: /tt///aaivvvv// org/ abs / 2303. 08774.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint 1707006777 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
Si Shen, Peijun Shen, Wenhua Zhao, and Danhao Zhu. Mitigating think-answer mismatch in llm reasoning through noise-aware advantage reweighting. arXiv preprint arXiv:5508.05928, 2025.
URL https::t//// httpps:///axiivi /wwxxvvvvvvv./////////..///aaavl org/ abs/abas....///////////// 05928.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceed- ings of the Twentieth European Conference Computer Systems, EuroSys '25, pp. 1297. ACM, March 2025. doi: 10. 10.1145/368903133666077 URL http:// //tp///xxx///// doi www.oooog....1.. 3689031 3696075.
Yiping Wang. Qing Yang. Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuchai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2057, 2025.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint 24XXi:2402.04333 2024.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-mmg model via self self-improvement, 2024. URL https: : / /rxivv. org/ abs / 22099 12122.
technical report: Toward mathematical expert
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL hts / /aaiivvvvvv org/ abs/ 2505.55555 0 09388.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijic Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387 2025.
14

---

Published as a conference paper at ICLR 2026
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng. Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025a. URL ptttpp://axxiv.or/ abs/2503 14476.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint vrXiv:2503.14476 2025b.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey Hinton. Lookahead optimizer: k steps forward, step back. Advances in neural information processing systems, 32, 2019.
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for Il reasoning via selective rollouts. arXiv preprint Xiv:22506002177, 2025.
15

---

Published as a conference paper at ICLR 2026
A LLM USAGE
Large language models (LLMs) were used solely to improve the writing of this paper, including grammar, clarity, and readability. They were not used for generating ideas, designing experiments, conducting analyses, producing scientific content. All research contributions, technical claims, and conclusions are entirely the work of the authors.
B GPU MEMORY PROFILING RESULTS
SFPO
GRPO
100%
100%
97%
97%
94%
94%
91%
88%
88%
85%
(b)
Figure 8:
Comparison of GPU memory consumption during one RL training step between SFPO and GRPO for Deepseek-RR--isiill--eeeee
model.
From Fig. 8, consumption during one training step. This aligns with our expectation: SFPO does not need to store the heavy optimizer states and parameters but only need to store copy of the model weight, which does not introduce significant overhead, especially when the model is sharded across GPUs.
clearly observe that SFPO and GRPO demonstrate similar GPU memory
C ADDITIONAL RESULTS
C.1
IMPACT OF INTERPOLATION SCHEME IN STAGE
GRPO
SFPO w/ Stage
SFPO Stage
K=3
K=7
0.50
0.50
Average Accuracy
0.48
Average Accuracy
0.46
0.46
0.44
0.44
0.42
0.40
0.40
0.38
0.38
100
200
300
400
50
150
200
Training Step
Training Step
Figure 9:
Ablation on interpolation scheme in Stage II with PeepSeek-RR--Diiill--eeee
To make the significance of stage II much clearer, we further conduct an explicit comparison between SFPO with and without the interpolation scheme in stage using and the results are demonstrated in the Fig. 9. Specifically,
When K = 3, SFPO without Stage II is slightly better at the very beginning of training because the drift accumulated over only three fast updates remains relatively small. However, training progresses, its performance begins to degrade due to the growing off-policy mismatch. Since K is still relatively close to the vanilla GRPO, the degradation the later stage becomes comparable standard GRPO.
16

---

Published as a conference paper at ICLR 2026
When K the difference becomes substantially II initially gains some accuracy, but the lack of reposition step causes severe off-policy drift as training continues. With K 7, this drift grows large enough that the method moves far away from the on-policy region used by standard GRPO, eventually causing the training to collapse. In contrast, SFPO with Stage II remains stable throughout training and consistently achieves the best overall performance.
pronounced. SFPO without Stage
These results confirm that the interpolation-base reposition step in Stage II is crucial for stabilizing multi-step updates and turning the additional inner updates into consistent gains.
C.2 SFPO TRAINING ON CODING TASKS
DS-RI-DistilllQwwen-
K=3
K=7
22.00
19.50
Average Accuracy
SFPO
Average Accuracy
19.00
GRPO
Average Accuracy
22.00
GAPO
0.8
21.00
08
20.00
18.50
20.00
19.00
18.00
19.00
17.50
18.00
17.00
17.00
17.00
16.50
100
150
200
50
100
150
200
Training Step (a)
Training Step (b)
Training Step (c)
Figure 10: (a): Validation accuracy of LiveCodeBench on CrepSee-RI--isstll--ee 1.5B between
SFPO and GRPO throughout the learning process. (b): Ablation of when K 3 for coding tasks.
(c): Ablation of a when = 7 for coding tasks.
We additionally trained DeepSeek-a SeepSeek-RR-Distilll R1-Distill-Qwen 1.5B on the Skywork-OR1 Code RL training dataset (He et al., 2025b) with 14. 1k data. The training hyperparameters mirror those used for math tasks, with K = 3 and = 0.8 by default. We then evaluate the resulting checkpoints on LiveCodeBench with 32K response length and report Pass (Avg 4). The results are illustrated in Fig. 10(a). As shown, within the 400 training steps, GRPO reaches maximum Pass I (Avg 4) of 18.91 at 400 steps, whereas SFPO attains a higher peak performance of 21.77 at 350 steps. This consistent improvement the coding benchmark provides additional evidence that the efficacy of SFPO extends beyond purely mathematical reasoning tasks.
We also conduct additional ablation study investigate the impact of a and K RL training for coding tasks. Specifically, we consider two scenarios: (i) small inner-update budget K = 3 with varying a 0.2, 0.8, 1.0, and (ii) larger budget K with the same set of values. As shown in Fig. 10(b) and Fig. 10(c), when K = 3, the default value = 0.8 achieves the best accuracy through out the training process, which is consistent with the findings math tasks. While for K = 7, smaller = 0.2 is preferred to mitigate stronger distribution drift and restore stability, again aligning with the trends observed in math RL training.
C.3 INTERPOLATION IN STAGE-II AGAINST KL PENALTY
We conducted additional experiments comparing the KL-penalty baseline with the interpolation scheme used in our SFPO method. Specifically, we use Qwen3-4B- as the policy model. The training dataset and all other settings are identical to those in Section 4.1 of the submitted manuscript. We then compare interpolation against the KI.-penalty baseline over mathematical reasoning benchmarks under two inner update settings, with K 3, 0.8 and K 7, 0.2, respectively.
The results are shown in the Fig. 11. We can observe that for k=3, interpolation consistently outperforms the KL-penalty variant throughout training: for example, at step 200 interpolation more pronounced
reaches 45.44, compared to 40.64 with KL penalty. The gap becomes when k=7: the interpolation scheme remains stable and continues to improve up to 46.14 at step 200, whereas the KL-penalty variant becomes unstable and collapses after step 80, yielding
17

---

Published as a conference paper at ICLR 2026
K=3
Average Accuracy
Average Accuracy
0.40
0.35
0.30
0.30
0.20
penalty
KL penalty
SFPO
0.10
SFRO
0.00
50
100
150
200
Training Step
Training Step
Figure 11: Comparison of using interpolation scheme in stage II (SFPO) and using KL divergence penalty in stage II on Mathematical reasoning benchmarks for Qwen3-4B-Bar -B-BBsc through the RL training process.
scores. These results empirically demonstrate our interpolation- Stage II provides both better performance and substantially improved stability compared to KL-constrained update.
C.4 APPLYING SFPO ON DAPO
DS-R1 DS-RI-DistilllQQenn-.88
Qwen3-4B-Base
Average Accuracy
0.48
Average Accuracy
0.45
0.46
0.40
120 Steps
0.44
225 Steps
0.35
0.42
0.40
DAPO
0.25
DAPO
0.38
0.20
100
200
300
400
100
200
300
400
Training Step
Training Step
Figure 12: Comparison of DAPO and SFPO math benchmarks. Validation average accuracy versus training step for -eepSSee--I--istil---eeee--
(left) and Qwen3-4B-Base (right).
To directly examine whether the performance gains of SFPO come primarily from handling zero- advantage batches, we compare DAPO against DAPO with SFPO applied using DeepSeck-RI- Distill-Qwen-l.55 and Qwen3-4B-Ban benchmarks throughout the training process is reported in Fig. 12. From these two curves, we observe that for both models, SFPO not only achieves higher accuracy than DAPO over the entire training trajectory, but also reaches the target performance substantially earlier: roughly 225 steps faster SS-RR-Disstill-weer 1.5B and 120 steps faster on Qwen3 Brren3-4-BBace B-Base. We further summarize the best accuracy achieved during training in the Table 3, where for each method select the checkpoint with the highest average score across all math benchmarks. On DS-RI-Distill-Qwee -...----een...B.. SFPO attains average accuracy of 50.56, while DAPO reaches 49.30; Qwen3-4B 4eee3-4B-Base- SFPO achieves 47.87 compared to 46.48 for DAPO, with consistent improvements across most individual benchmarks.
the policy models. The validation accuracy on the math
Since DAPO already employs dynamic sampling to mitigate the zero-advantage issue, these results indicate that SFPO continues to provide clear benefits even in setting where this particular source of variance is explicitly addressed. This suggests that SFPO's gains do primarily stem from handling aero-advantage batches. Instead, SFPO operates at higher-level optimization layer that is independent of the internal mechanies GRPO or DAPO; it is therefore orthogonal these methods and can be seamlessly integrated with GRPO, DAPO, or any GRPO-family variant.
C.5 COMPUTE COST AND WALL-CLOCK EFFICIENCY OF SFPO VS GRPO
In SFPO, each iteration performs fast policy updates on the same batch, so the backward-pac compute for the policy update scales approximately linearly with K. Previous works have shown that under GRPO, about 70% the per-step time is spent on rollout generation, 20% on the policy update, and x 10% other overhead. Increasing K from I to multiplies only the 20%
18

---

Published as a conference paper at ICLR 2026
Table 3: Performance math reasoning benchmarks for DAPO and applying SFPO top of DAPO.
Model
Method
Math-500
AIME24
AIME25
AMC
Minerva
Olympiad
Avg
DAPO
85.40
32.50
24.17
68.07
33.64
52.00
49.30
DS-distilled--wee-1.55
SEPO
86.25
33.33
26.67
70.18
34.38
52.52
50.56
DAPO
84,50
22.50
20.83
59.64
39.98
51.41
46.48
Qwen3-4B-Bas
STPO
85.55
25.83
23.33
58.13
41.27
53.12
47.87
"policy-update" portion, so the worst-case theoretical overhead is
Tsro
1 (K 1) 0.2 = 1.4.
(13)
TaRPO
In practice, measured average per-step wall-clock times on DeepSee--R--Distil---ee 1wen-1.5BB are
300 s (~~ 240 for rollout generation and ~ 40 for the actor update) for GRPO and ~ 410s
(~ 240 for rollout generation and 150s for the actor update) for SFPO (with K and 0.8). i.e., SFPO steps are only R 1.37 more expensive than GRPO steps. Moreover, our ablations on and K over both math and coding benchmarks show that this relatively small K 3 with moderately large = 0.8 is already sufficient; larger K does not yield further gains and is therefore unnecessary in practice.
We compare SFPO and GRPO DeepSeck-RI-Distinn
Qwen-1.5B in terms of validation accuracy versus wall- clock training time in Fig. 13. Across essentially the entire range from about to 33 hours, the SFPO curve stays above the GRPO curve, indicating that for any fixed time budget SFPO yields better policies. Quantitatively, SFPO typically about 1-3.6 accuracy points higher than GRPO at matched wall-clock times, despite having slightly higher per-step compute. Conversely, if fix accuracy targets, SFPO reaches the same accuracy level in 3-6.6x less wall- clock time, showing that its additional inner updates trans- late into substantially improved overall compute efficiency.
Average Accuracy
0.48
0.46
0.44
0.42
0.40
GRPO
0.38
SFPO
30 Wall-clock Time (hours)
15
In summary, while Stage of SFPO does ineur additional gradient updates and thus increases per-iteration compute as grows, we deliberately operate in a regime with a small K where the per-step overhead is modest. Under fair comparisons at matched wall-clock time and at fixed accuracy targets, SFPO is substantially more compute-efficit than GRPO, indicating that the extra Stage-I cost is more than compensated by improved sample and optimization efficiency.
Figure 13: Wall-clock efficiency of SFPO vs. GRPO DeepSeek-RI- Distill-Qwen-I55B.
D DISCUSSION
D.I NOVELTY
Three-stage training pipeline. In the standard GRPO pipeline, the model first generates rollouts and then performs single update using the GRPO objective. However, this one-shot update makes early training highly cample-inefficieer and sensitive to noisy rollouts, increasing both training cost and instability. In contrast, our three-stage pipeline (fast -> reposition - slow) reaches the best GRPO performance many steps earlier than vanilla GRPO. This results substantial reduction in rollout usage and overall training budget, while preserving the original GRPO objective and rollout process.
plug-in, higher-level optimization mechanism. Our SFPO does not require substantial modifica- tions GRPO because it operates at higher level of the optimization pipeline. Unlike GRPO variants that introduce different twists to advantage computation, SFPO is orthogonal to these methods and be seamlessly combined with them without changing their training objective, KL regularization, or rollout procedure. This makes SFPO an casy-to-use, plug-and and-play mechanism applicable to any GR familyy method.
19

---

Published as a conference paper at ICLR 2026
D.2 NAMING OF SFPO
When naming our SFPO method, the terms "fast" in Stage I and "slow" in Stage III are simply naming choices. our terminology, "slow" and "fast" refer to the frequency of updates, not the computational speed. Specifically, Stage performs multiple fast, high-frequency updates (K inner steps), whereas Stage III performs only a single correction update. Thus, Stage is called fast due to its update frequency, while Stage III is called slow because it occurs only once per iteration.
D.3 MORE ENTROPY ANALYSIS
In general, lower entropy can sometimes indicate premature convergence an over-deterministic policy. In our setting, however, the observed entropy reduction in SFPO is better interpreted as more efficient exploration, rather than collapsing exploration.
In vanilla GRPO, each update depends on single-step, high- variance advantage estimate, which makes the early update direction unstable. As a result, GRPO often keeps higher entropy to compen- sate for this noise.
In contrast, SFPO aggregates multiple gradient steps (Stage 1), producing a much more stable and accurate update direction. Given this reduced variance, the policy can more confidently up-weight promising actions and down-weight clearly suboptimal ones, so it does not need to maintain as much randomness keep improving. Importantly, SFPO still retains GRPO's entropy bonus, so it does not suppress exploration intentionally.
Overall, the lower entropy observe corresponds to more efficient exploration, not reduced explo- ration. The performance improvements and stable diversity metrics strongly support this interpreta- tion.
D.4 INFORMAL DEFINITION AND ANALYSIS OF F(K, a) IN SEC. 3.3.
We adopt local quadratic-
model around 0=.0; (()) ~ ((--0)TT(0000 0-) with ž g(0) = (Z())= = H(0 (²), and eg = 80.0 - 0" Stage uses stochastic gradients Jk = (8 + Ex
where [E[] 0, El 1| o7. and Cov(ER, Ee) has correlation coefficients Pik-0 [0. 1]. The
Stage III query at Øs.K has noise & with B[E] 0, E E < o3.
Deterministic fast trajectory. In the quadratic model, Stage I obeys ex+1 (I - nH)c,, hence ек = (I - HHHHH eo and
Ak
def ga.K
=ek -c -c- - nH) H* gu. g0.0
9o
g(000))
(14)
Stage II repositions to Hz.K 0z.0 Ak. A standard Fisher/ Gauss-Newton proxy yields the drift magnitude
Drift(K, a.) a² |A a² A = a² 90 a²9o HT 1 (I -))))// 90. (15)
Spectrally, for an eigenpair (ux, À > 0),
(K, a² (1-(I-nA)*)* (go, ux)
(16)
which increases monotonically with K and is quadratic in a, and saturates as K + cc.
Stochastic components.
The fast trajectory accumulates noise Σk=1 & with
K-1 B a
K-1
Eull
< a² oz S(K,
S(K, = K
Σ A=1
(K A)PA E [K, K3], (17)
A=o
and the slow query contributes 2 < o².
20

---

Published as a conference paper at ICLR 2026
Informal definition.
We informally define the F(K, a.) as
F(K,a) R a² 30 H (T
) ) + ooo) .))
(18)
slow noise
oft-policy drifl (bias)
where x means up multiplicative constants depending only on (H,
Dependence on K and a. For fixed a, both drift and S(K, p) are non-decreasing in (monotone under < 1) 1/Amax), so F(K, a) increases with K. For fixed K, F(K, ai) grows quadratically in a. In the small-step regime (nAK < 1), Driftx (K, a) z a²(nK)² ux)², showing that excessively large K makes F(K, ox) dominate the descent term.
Implication for Stage III. The descent-lemma bound, E[[( < L(0*.0) cnl [| F(K,L-((K,a)) a)), treats F(K, a) as the residual that the one-step slow update must compensate. Thus, choosing small K and moderate or keeps F(K, a) manageable while retaining the stability benefits of the fast trajectory.
21


Reviews:
[{"review_id": "kUdOhTchhh", "reviewer": "Reviewer_UnHR", "summary": "The paper proposes Slow–Fast Policy Optimization (SFPO), it's a three-stage modification to on-policy GRPO-style training for reasoning LLMs: (i) a fast trajectory of $K$ inner gradient steps on the same rollout batch, (ii) a reposition interpolation back toward the start point controlled by $\\alpha$, and (iii) a slow correction extra step. See Alg. 1 (p. 3). Authors also introduce an entropy-triggered schedule that sets $\\alpha \\leftarrow 0$ after a $z$-score threshold on policy entropy (Eq. 11), which effectively reverts to GRPO near convergence.", "questions": "See weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Simple, drop-in recipe that practitioners can try with minimal code churn (Alg. 1).\n- Large rollout/time reductions to reach GRPO’s best accuracy (Fig. 4), aligning with the observation that rollouts dominate wall-clock.", "weaknesses": "1. GRPO-K: $K$ inner updates on the same batch without reposition or slow correction (a direct control for “just do more steps per batch”).  \n\n2. Lookahead-GRPO applies only the Stage II interpolation around GRPO.  \n\n3. Extra-grad-GRPO applies only the Stage III predictor–corrector step.  \n\n4. Clarify whether per-iteration compute (backprop) increases due to $K$; if so, compare at matched FLOPs (or matched wall-clock) and at fixed accuracy targets, not “GRPO’s best accuracy”.\n\n5. The theoretical section is insufficient. Make Eq. (10) precise (define $c$ and $F(K,\\alpha)$; assumptions on clipping/KL; conditions handling negative curvature), or clearly mark it as heuristic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OE02Y0tpFx", "reviewer": "Reviewer_JUEZ", "summary": "This paper argues that the direction of token-level updates, measured by the signed log-probability difference between base and RLVR models, is more informative than magnitude-based metrics for understanding RLVR's effect on LLM reasoning. The authors propose and validate two methods, test-time extrapolation and training-time advantage reweighting, that exploit the directional insight to improve reasoning performance.", "questions": "How does the proposed direction-based extrapolation perform in non-mathematical reasoning tasks?\n\nThe token-replacement experiment convincingly shows that \\Delta\\log p identifies critical tokens. However, does this intervention sometimes harm performance? Are there cases were replacing a base model token with the RLVR model's choice leads to a wrong answer, and what characterizes those tokens?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel and intuitive directional metric that effectively captures sparse, reasoning-critical updates, supported by rigorous token-replacement experiments and gradient analysis.\n\n The proposed methods are simple yet effective, demonstrating consistent gains across multiple models and benchmarks without requiring additional training data.", "weaknesses": "The test-time extrapolation method requires access to both the base and RLVR models, which may limit its practicality in settings where only the fine-tuned model is available.\n\n    The paper focuses primarily on mathematical reasoning benchmarks (e.g., AIME, AMC); it remains unclear whether the findings generalize to other reasoning domains or more diverse tasks.\n\n    The theoretical justification (Theorem 4.1) relies on a simplified tabular softmax bandit setting, which may not fully reflect the complexity of modern LLM training dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrG8MTAtEC", "reviewer": "Reviewer_5eGS", "summary": "This paper introduces Slow-Fast Policy Optimization (SFPO), a three-stage update mechanism designed to improve the stability and sample efficiency of on-policy reinforcement learning for LLM reasoning. SFPO replaces the standard single gradient step with a fast trajectory of multiple inner updates, a reposition step to control off-policy drift, and a final slow correction step. Extensive experiments demonstrate that SFPO significantly outperforms the GRPO baseline in accuracy, convergence speed, and wall-clock time on various math reasoning benchmarks.", "questions": "1.  Could you clarify the nomenclature for Stage I, the 'Fast Trajectory'? The term 'fast' is somewhat ambiguous. Does it refer to the speed of convergence, the magnitude of the parameter change, or some other property?\n2.  A crucial baseline seems to be missing from the ablation studies. To isolate the benefits of Stage II (Reposition) and Stage III (Slow Correction), have you considered a baseline that only uses Stage I? This would be equivalent to applying GRPO with $K$ inner updates on the same batch, which would more clearly demonstrate the contribution of the reposition and slow correction steps.\n3.  Stage II is designed to control off-policy drift via interpolation. A more common approach in policy gradient methods is to use a KL divergence penalty to constrain the policy update. Could you comment on the design choice of interpolation over a KL penalty? Have you compared the performance of your reposition step against a KL-constrained update in this multi-step update setting?\n4.  The paper claims that SFPO addresses the high-variance gradient issue in GRPO. A significant source of variance in GRPO can come from batches with uniform rewards (e.g., all successes or all failures), which result in zero advantage for all samples. Does SFPO's performance gain stem primarily from better handling these 'zero-advantage' batches? To verify this, it might be necessary to conduct experiments with an algorithm like DAPO, which is explicitly designed to use *dynamic sampling* to mitigate this issue, to see if SFPO provides similar benefits in a setting where this specific source of variance is already addressed.\n5.  Could you provide a derivation or at least a reference for the descent guarantee in Equation (10)? A more formal explanation of the assumptions and the term $F(K, \\alpha) $ would strengthen the paper's theoretical grounding.\n6.  Would you be able to correct the numerical errors in the text describing the results in Table 1 (lines 322-323)? The discrepancies between the reported gains and the values in the table are substantial.\n7. The interpretation of lower entropy in SFPO as \"more efficient exploration\" (lines 374-375) is intriguing. Could you elaborate on this? Is it possible that the method simply encourages faster convergence to a more deterministic policy, which might be beneficial for exploitation but could also be interpreted as a sign of reduced exploration?\n\nI look forward to the author's response. I am willing to reconsider my score if the questions above, particularly those concerning the experimental details, numerical correctness, and key design choices, are adequately addressed.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The proposed method addresses a significant and well-recognized problem: the instability and sample inefficiency of on-policy algorithms like GRPO in the context of LLM reasoning. The proposed three-stage solution is simple, intuitive, and clearly motivated as a way to reduce update variance and make better use of collected data.\n2.  A major strength of SFPO is its \"plug-and-play\" compatibility with existing policy gradient pipelines. By modifying only the parameter update rule while leaving the loss function, rollout generation, and regularization terms of GRPO intact, the method offers a practical, low-effort path to improving existing training setups.\n3.  The paper is supported by a comprehensive set of experiments that demonstrate consistent and substantial gains over a strong GRPO baseline across multiple models and benchmarks. The reported improvements in sample efficiency (up to $4.93\\times$ fewer rollouts) and wall-clock time (up to $4.19\\times $ reduction) are highly significant. Furthermore, the ablation studies effectively justify the inclusion of each component of SFPO, particularly the adaptive entropy control and the slow correction stage.", "weaknesses": "1.  The reporting of experimental results contains significant numerical errors and omissions. In Section 4.2.1, the claimed performance gains for Qwen2.5-Math-7B (+1.80) and DS-distilled-Qwen-7B (+0.8) do not match the values in Table 1, which show gains of +0.83 and +2.57, respectively. \n2.  The reference list is not properly curated. For example, the citations \"Yu et al., 2025a\" and \"Yu et al., 2025b\" both point to the exact same arXiv preprint, which is misleading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HlwKCxbKsK", "reviewer": "Reviewer_GcTt", "summary": "This paper introduces Slow-Fast Policy Optimization (SFPO), a plug-and-play method to stablize and accelerate the learning of on-policy RL algorithms such as GRPO. Specifically, SFPO has three stages for each training step: Fast Trajectory, Reposition, and Slow Correction, which mitigate the noisy single-step rollout updates and ensures a more stable optimization process. Empirical studies in mathmatical benchmarks demonstrates the consistent improvement of SFPO compared with GRPO.", "questions": "1. What does \"Slow\" refer to in stage 3? Does this stage require significantly more time to update the gradient?\n\n2. Could you elaborate on the time consumption in Stage 1? Since gradient updates are computationally intensive, this stage might become slower as K increases. Additionally, in agentic scenarios involving multi-turn interactive trajectories, performing multiple policy gradient updates may significantly reduce optimization efficiency.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is well-structured and thoughtfully designed. The Fast Trajectory stage stabilizes the optimization process, while the Reposition and Slow Correction stages help to better align the optimization with the on-policy objective. The motivation behind each stage is clearly articulated, making the overall approach sound and convincing.\n\n2. The empirical results are thorough and well-presented. The paper evaluates SFPO across 5 models, 3 methods, and 6 benchmarks, clearly demonstrating substantial improvements in both efficacy and efficiency. Furthermore, comprehensive analyses and ablation studies are provided, strengthening the validity of SFPO’s performance claims.\n\n3. The paper is well-organized, featuring clear and informative figures, tables, and algorithm illustrations. Additionally, Section 3 provides intuitive explanations that greatly aid in understanding the underlying mechanism of the proposed SFPO.", "weaknesses": "1. The empirical evaluation is somewhat limited in scope, as all experiments are conducted only on mathematical benchmarks, without including domains such as code or logic reasoning. Additionally, GRPO is the sole baseline used; incorporating additional baselines like DAPO and GSPO would further strengthen the evidence for SFPO’s effectiveness.\n\n2. The ablation studies indicate that SFPO is sensitive to the hyperparameters $\\alpha$ and $K$, which may hinder its direct applicability to domains beyond mathematical reasoning. Users may need to carefully tune these hyperparameters to achieve optimal performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)