Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning

## Abstract

Abstract Reinforcement learning (RL) has become central to enhancing reasoning in large language models (LLMs). Yet on-policy algorithms such as Group Relative Policy Optimization (GRPO) often suffer in early training: noisy gradients from low-quality rollouts lead to unstable updates and inefficient exploration. We introduce Slow-Fast Policy Optimization (SFPO), a simple yet efficient framework to address the above limitations via decomposing each step into three stages: a short fast trajectory of inner steps on the same batch, a reposition mechanism to control off-policy drift, and a final slow correction. This reposition-before-update design preserves the objective and rollout process unchanged, making SFPO plug-compatible with existing policy-gradient pipelines. Extensive experiments demonstrate that SFPO consistently improves stability, reduces number of rollouts, and accelerates convergence of reasoning RL training. Specifically, it outperforms GRPO by up to 2.80 points in average on math reasoning benchmarks. It also achieves up to 4.93 × \times fewer rollouts and an up to 4.19 × \times reduction in wall-clock time to match GRPO’s best accuracy. 1 1 1 Project website is available at https://zkbig.github.io/Slow_Fast_Policy_Optimization.github.io/ .

## 1 Introduction

Large language models (LLMs) have recently achieved remarkable progress on complex multi-step reasoning tasks, especially in mathematics problem solving and scientific question answering (OpenAI et al., 2024, DeepSeek-AI et al., 2025b). A central driver of these advances has been reinforcement learning (RL), which fine-tunes LLMs using reward signals tied to semantic correctness and solution quality. For example, DeepSeekMath (Shao et al., 2024) demonstrates that Group Relative Policy Optimization (GRPO) can substantially improve open-source LMs on mathematical reasoning, while DeepSeek-R1 (Guo et al., 2025) shows that RL signals alone can induce emergent reasoning behaviors across various domains.

Despite these successes, GRPO inherits structural inefficiencies that make training fragile in LLM reasoning. During the early stages of training, when rollouts are particularly weak or uninformative, stochastic rewards induce high-variance gradients that destabilize updates (Yu et al., 2025a, Zheng et al., 2025, Dai et al., 2025, Shen et al., 2025).
Compounding this, common practice in GRPO implementations is to use a single optimization step per rollout batch, and such a one-shot update tends to produce a noisier, less reliable step update direction, especially at the early training stages, thereby underutilizing each batch and discarding information that could be further exploited (Schulman et al., 2017, Engstrom et al., 2020). Although the original GRPO paper (Shao et al., 2024) explored reusing rollout data, our findings suggest that simply applying this off-policy update to the policy model can degrade performance as training proceeds, and how to use it effectively to build strong reasoning abilities in LLMs with satisfying sample efficiency remains an open question.

In this work, we introduce Slow–Fast Policy Optimization (SFPO), a simple yet effective enhancement to on-policy policy-gradient methods (e.g., GRPO) that stabilizes noisy updates and improves sample efficiency while leaving the base loss, rollout collection, and KL/clip regularization unchanged. Specifically, each training step is restructured into three coordinated stages: (i) fast—multiple inner updates on the same batch to stabilize the search direction; (ii) reposition—interpolation back toward the starting point to control off-policy drift; and (iii) slow—an extra gradient correction to align with local curvature. This reposition-before-update design transforms noisy one-shot updates into structured trajectories, yielding more stable optimization, higher sample efficiency, and faster convergence. We empirically validate the efficacy of SFPO through extensive experiments across five models and six reasoning benchmarks. The evaluation shows that SFPO outperforms GRPO by up to 2.80 points in average on math reasoning benchmarks. It also uses up to 4.93$\times$ fewer rollouts while reducing wall-clock time by up to 4.19$\times$ to reach GRPO’s best accuracy. We summarize our contributions as follows:

- •
We propose SFPO, a plug-compatible update rule that reuses rollout batches via a fast–reposition–slow decomposition, with theoretical insights for each stage.
- •
SFPO introduces no changes to the underlying objective, rollout generation, or regularization, enabling drop-in integration into existing LLM reasoning pipelines.
- •
Through extensive experiments on math reasoning benchmarks, we show that SFPO consistently improves stability, sample efficiency, and convergence speed over GRPO.

## 2 Preliminaries

Figure: Figure 1: Pipeline of SFPO at step $s$. Starting from the current policy $\pi_{\theta^{s,0}}$, we first generate rollouts for training. Stage I (Fast Trajectory): apply $K$ successive gradient updates on the same batch to obtain $\theta^{s,K}$. Stage II (Reposition): interpolate between $\theta^{s,K}$ and the starting point $\theta^{s,0}$ to form $\widetilde{\theta}^{s,K}$, controlling off-policy drift. Stage III (Slow Correction): perform one additional update on $\widetilde{\theta}^{s,K}$, yielding $\pi_{\theta^{s+1,0}}$ for the next step.
Refer to caption: x1.png

##### Group Relative Policy Optimization (GRPO) (Shao et al., 2024 )

is a policy gradient method that has been widely applied in large-scale LLM training pipelines. GRPO dispenses with an explicit value function and instead normalizes rewards across a group of responses to the same prompt. Given an input $q$, the policy $\pi_{\theta}$ generates $G$ candidate sequences $\{o_{i}\}_{i=1}^{G}$ with corresponding rewards $\{r_{i}\}_{i=1}^{G}$. The normalized advantage for each candidate is defined as

$$ $\displaystyle\widehat{A}_{i}=\tfrac{r_{i}-mean(\{r_{i}\}_{i=1}^{G})}{std(\{r_{i}\}_{i=1}^{G})}.$ (1) $$

This group-based normalization can be interpreted as a form of reward shaping: by emphasizing relative differences among candidate sequences for the same input, GRPO strengthens the reliability of the gradient signal. Instead of embedding a KL penalty inside the reward, GRPO directly regularizes the policy by including an explicit KL term between the learned policy and a reference policy. The training objective is

$$ $\displaystyle\mathcal{J}_{GRPO}(\theta)=\mathbb{E}_{q\sim P(\mathcal{Q}),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)}$ $\displaystyle\quad\tfrac{1}{G}\sum_{i=1}^{G}\tfrac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\min(r_{i,t}(\theta)\widehat{A}_{i,t},clip(r_{i,t}(\theta),1-\epsilon,1+\epsilon)\widehat{A}_{i,t})-\beta D_{KL}[\pi_{\theta}\|\pi_{ref}]$ (2) $$

where $r_{i,t}(\theta)=\tfrac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}$ is the token-level importance ratio between the new and old policies, $\epsilon$ is the clipping range that prevents overly aggressive updates, $\beta$ controls the KL regularization strength, and $\pi_{\mathrm{ref}}$ is a fixed reference policy. Here, $q$ denotes the input prompt, $o_{i}$ a generated sequence, and $o_{i,t}$ its $t$-th token.

##### Limitations of GRPO.

Despite its popularity, GRPO inherits structural drawbacks from the underlying on-policy update rule. At each step, the parameters are updated via a single stochastic gradient step estimated from a batch of rollouts. The randomness of rewards leads to high-variance gradient estimates, making updates unstable. Group normalization partially dampens this effect but remains sensitive to fluctuations within each batch. Moreover, restricting each batch to a single update discards potentially useful gradient information across inner steps, leading to inefficient data use and limited variance reduction. These drawbacks underscore the need for an update mechanism that can stabilize gradient directions while making more effective use of available samples.

## 3 Method

To address the instability and inefficiency of one-shot policy updates, we propose Slow-Fast Policy Optimization (SFPO), a simple yet general update mechanism that reuses rollouts more effectively while remaining plug-compatible with standard on-policy policy gradient algorithms. Each step consists of three stages: (i) a *fast trajectory* of multiple inner updates on the same batch, (ii) a *reposition* step that interpolates back toward the on-policy point to control drift, and (iii) a *slow correction* via an extra-gradient update. As illustrated in Fig. [1](https://arxiv.org/html/2510.04072v2#S2.F1) and Alg. [1](https://arxiv.org/html/2510.04072v2#alg1), this fast–reposition–slow design transforms noisy one-shot updates into well-structured update trajectories, yielding more stable optimization and higher sample efficiency without additional rollouts.

Notation. To align with standard gradient-based optimization notation, we define $\mathcal{L}(\theta)=-\mathcal{J}(\theta)$, so that maximizing the objective $\mathcal{J}$ is equivalent to minimizing the loss $\mathcal{L}$.

### 3.1 Stage I: Fast Trajectory

In standard on-policy policy-gradient methods such as GRPO, each step is updated by a single stochastic gradient:

$$ $\displaystyle\theta^{s+1}=\theta^{s}-\eta\nabla_{\theta}\mathcal{L}(\theta^{s}),$ (3) $$

where $\nabla_{\theta}\mathcal{L}(\theta^{s})$ is estimated from one batch of rollouts. Such one-shot updates suffer from high variance and often drive the policy in unstable directions, especially during early training. SFPO mitigates this by performing multiple inner updates on the *same* batch or rollouts.

Formally, starting from parameters $\theta^{s,0}$ at the beginning of step $s$, we execute a short *fast trajectory* of $K$ inner updates:

$$ $\displaystyle\theta^{s,k+1}=\theta^{s,k}-\eta\nabla_{\theta}\mathcal{L}(\theta^{s,k}),\qquad k=0,\ldots,K-1.$ (4) $$

This produces a sequence $\theta^{s,0}\to\theta^{s,1}\to\cdots\to\theta^{s,K}$, where each step refines the gradient direction using the same rollout data.

##### Intuition.

Unlike one-shot updates that rely on a single noisy gradient, the displacement

$$ $\displaystyle\theta^{s,K}-\theta^{s,0}\;=\;-\eta\sum_{k=0}^{K-1}\nabla_{\theta}\mathcal{L}(\theta^{s,k})$ (5) $$

captures the cumulative effect of $K$ sequential corrections. Even if some individual steps are perturbed by noise, their composition tends to damp idiosyncratic fluctuations and align with the underlying gradient direction. Geometrically, this can be viewed as integrating the local gradient field along a short trajectory in parameter space rather than trusting a single noisy vector at $\theta^{s,0}$. As a result, the update direction at the end of Stage I is typically more stable and less sensitive to randomness in any single gradient estimate.

##### Compact Mathematical Intuition.

Let $\theta_{k}:=\theta^{s,k}$ and consider the $K$ inner steps $\theta_{k+1}=\theta_{k}-\eta\nabla\mathcal{L}(\theta_{k})$ starting at $\theta_{0}=\theta^{s,0}$. In a small neighborhood, linearizing the gradient field,
$\nabla\mathcal{L}(\theta)\approx g_{0}+H_{0}(\theta-\theta_{0})$
with $g_{0}=\nabla\mathcal{L}(\theta_{0})$ and $H_{0}=\nabla^{2}\mathcal{L}(\theta_{0})$,
yields the closed-form displacement

$$ $\displaystyle\theta_{K}-\theta_{0}\;\approx\;-\big[I-(I-\eta H_{0})^{K}\big]\,H_{0}^{\dagger}\,g_{0},$ (6) $$

where $H_{0}^{\dagger}$ denotes the (pseudo)inverse on the range of $H_{0}$ (the $\lambda\!\to\!0$ case is understood by continuity).
Spectrally, along an eigen-direction with curvature $\lambda\geq 0$, the scalar gain is
$\big(1-(1-\eta\lambda)^{K}\big)/\lambda$: for small $\lambda$ it behaves like $K\eta$ (steadily accumulating progress in gentle directions), while for larger $\lambda$ it saturates below $1$ (damping stiff-direction oscillations). Thus the fast trajectory acts as a curvature-aware low-pass filter that stabilizes the endpoint direction relative to a one-shot step. *We use this as local intuition under a sufficiently small step size (e.g., $\eta<1/\|H_{0}\|_{2}$) and in neighborhoods where positive-curvature directions dominate; in practice, KL/clip regularization and Stage II reposition mitigate adverse effects of negative curvature.*

### 3.2 Stage II: Reposition

While the fast trajectory of Stage I improves stability, it also *changes the nature of the update from on-policy to off-policy*. Since all inner steps $\theta^{s,1},\dots,\theta^{s,K}$ reuse the same rollouts generated at $\theta^{s,0}$, the endpoint $\theta^{s,K}$ no longer corresponds to the distribution that produced those samples. This *distribution mismatch* is a fundamental drawback of off-policy learning, as it biases gradient estimates and can destabilize training.

Inspired by Lookahead Optimization (Zhang et al., 2019), SFPO introduces a *reposition step* that interpolates the fast trajectory back toward its starting point:

$$ $\displaystyle\widetilde{\theta}^{s,K}=\theta^{s,0}+\alpha(\theta^{s,K}-\theta^{s,0}),\qquad\alpha\in[0,1].$ (7) $$

Here $\alpha$ regulates the degree of off-policy drift: smaller values keep the update close to the original on-policy iterate, while larger values rely more on the fast trajectory at the risk of greater mismatch.

##### Intuition.

The interpolation is equivalent to solving a linearized proximal subproblem around $\theta^{s,0}$:

$$ $\displaystyle\min_{\theta}\ \Big\langle\sum_{k=0}^{K-1}\nabla_{\theta}\mathcal{L}(\theta^{s,k}),\ \theta-\theta^{s,0}\Big\rangle+\tfrac{\lambda}{2}\|\theta-\theta^{s,0}\|^{2},$ (8) $$

whose unique solution coincides with $\widetilde{\theta}^{s,K}$ for $\lambda=\tfrac{1}{\alpha\eta}$. Thus, $\alpha$ acts as an *implicit trust-region radius*: smaller $\alpha$ implies a larger proximal weight $\lambda$, enforcing stronger contraction
toward the on-policy point. (^2^22A heuristic alternative is obtained if one replaces $\sum_{k}\nabla_{\theta}\mathcal{L}(\theta^{s,k})$ by the averaged gradient $\bar{g}$ and uses the approximation $\theta^{s,K}-\theta^{s,0}\approx-\eta K\,\bar{g}$, leading to $\lambda\approx 1/(\alpha\eta K)$.)

### 3.3 Stage III: Slow Correction

After repositioning, SFPO applies one more (slow) correction step at the interpolated point:

$$ $\displaystyle\theta^{s+1}=\widetilde{\theta}^{s,K}-\eta\nabla_{\theta}\mathcal{L}(\widetilde{\theta}^{s,K}).$ (9) $$

This yields a *predictor—corrector* structure: Stage I produces a stabilized *fast trajectory*, Stage II tempers off-policy drift via *reposition*, and Stage III applies a *slow correction* aligned with the local curvature at the update point.

##### Theoretical intuition.

Under $L$-smoothness and sufficiently small $\eta$, the descent lemma implies

$$ $\displaystyle\mathbb{E}[\mathcal{L}(\theta^{s+1})]\leq\mathcal{L}(\theta^{s,0})-c\,\eta\|\nabla\mathcal{L}(\theta^{s,0})\|^{2}+O\big(\eta^{2}L\cdot\mathcal{F}(K,\alpha)\big),$ (10) $$

where $\mathcal{F}(K,\alpha)$ represents the combined effect of Stage I (fast trajectory length $K$) and Stage II (reposition factor $\alpha$). Intuitively, $\mathcal{F}(K,\alpha)$ reflects a balance between exploiting more gradient information and controlling distributional drift: increasing $K$ leverages the same rollout data across multiple steps but also amplifies off-policy mismatch, while larger $\alpha$ interpolates more aggressively toward the fast trajectory at the risk of greater instability. In practice, since increasing $K$ raises both wall-clock cost and $\mathcal{F}(K,\alpha)$, we adopt small $K$ with moderately large $\alpha$, while the slow correction in Stage III mitigates overshooting and preserves progress along the stabilized trajectory direction.

Figure: Algorithm 1 SFPO: unified fast–reposition–slow update.

### 3.4 Scheduling α \alpha

A nonzero $\alpha$ is essential to exploit the stabilized *fast trajectory* when $K>0$: if $\alpha=0$, the reposition collapses to $\theta^{s,0}$ and the fast trajectory is discarded, eliminating the benefit of Stage I. However, the same aggressiveness that helps during early training can be counter-productive near a minimizer. When $\|\nabla\mathcal{L}\|$ is large, a nonzero $\alpha$ accelerates progress by moving along the stabilized fast direction; but as we approach a minimum, the signal weakens while curvature and stochastic noise dominate, so a large $\alpha$ amplifies drift and instability. This motivates an *adaptive schedule for $\alpha$*.

##### Why adapt α \alpha rather than K K ?

$K$ controls both stabilization and wall-clock runtime: increasing $K$ reduces oscillations but incurs proportional compute cost, making mid-training changes impractical. By contrast, $\alpha$ is a *soft trust parameter*: it decides how much of the stabilized fast trajectory is exploited, without changing runtime or discarding the already-computed $K$ steps. Thus $\alpha$ is the natural lever to adapt dynamically.

##### Adaptive rule in practice.

In our implementation, $\alpha$ is scheduled online at each step. After generating rollouts with the current policy, we compute the policy entropy $H_{s}$ and maintain a rolling buffer of the past $\omega$ entropy values (window size $\omega$). Let $\mu_{s}$ and $\sigma_{s}$ denote the mean and standard deviation within this buffer. We define the one-sided z-score

$$ $\displaystyle Z_{s}=\tfrac{H_{s}-\mu_{s}}{\sigma_{s}}.$ (11) $$

If $|Z_{s}|\geq\tau$ for a threshold $\tau$, we mark the current step $s^{\star}$ and set $\alpha=0$ for all $s\geq s^{\star}$. Otherwise we keep $\alpha=\alpha_{0}$. Intuitively, sharp entropy fluctuations signal that the policy is close to a local optimum where noise dominates, so interpolation should be disabled. This entropy-triggered schedule exploits the fast trajectory during the high-signal early stage of training, while reverting to pure on-policy updates near convergence for stability.(^3^33Although a decaying schedule on $\alpha$ could be a formal solution, we find little empirical difference, as shown in Fig. [7](https://arxiv.org/html/2510.04072v2#S5.F7) of Sec. [5](https://arxiv.org/html/2510.04072v2#S5). Hence, for simplicity and efficiency, we downgrade SFPO to GRPO once the step reaches $s\geq s^{\star}$.)

### 3.5 Unified SFPO Update

Collecting the three stages, the unified SFPO update for each step $s$ is:

$$ $\displaystyle\theta^{s+1}\;=\;\theta^{s,0}\;-\;\eta\Big[\underbrace{\alpha\sum_{k=0}^{K-1}\nabla_{\theta}\mathcal{L}(\theta^{s,k})}_{\text{fast trajectory \& reposition}}\;+\;\underbrace{\nabla_{\theta}\mathcal{L}(\widetilde{\theta}^{s,K})}_{\text{slow correction}}\Big],$ (12) $$

As shown in Alg. [1](https://arxiv.org/html/2510.04072v2#alg1), SFPO unifies the three stages into a single update rule and serves as a plug-compatible drop-in replacement for on-policy policy-gradient methods such as GRPO. Our experiments show that this structural change consistently improves stability and sample efficiency on diverse math reasoning benchmarks, with minimal engineering overhead.

## 4 Experiments

### 4.1 Experimental Settings

Models. We conduct the reasoning rl training with our proposed SFPO on a wide range of models: Qwen2.5-Math-1.5B (Yang et al., 2024), DeepSeek-R1-Distill-Qwen-1.5B (DeepSeek-AI et al., 2025a), Qwen3-4B-Base (Yang et al., 2025), Qwen2.5-Math-7B (Yang et al., 2024), and DeepSeek-R1-Distill-Qwen-7B (DeepSeek-AI et al., 2025a). For Qwen2.5-Math-1.5B/7B, we set the context length in the training process as 4096 the same as their maximum context length; while for Qwen3-4B-Base and DeepSeek-R1-Distill-Qwen-1.5B/7B, we set the context length as 8192 for the better accuracy-efficiency tradeoff.

Reasoning RL Training. We conduct the reasoning RL training on two different training datasets to test the effectiveness of SFPO on different scales of dataset. The first one is the combination of DAPO training dataset (Yu et al., 2025a) and Math training dataset (Hendrycks et al., 2021), which is a total of approximate 24K data. The second one is the Skywork-OR1 Math RL training dataset (He et al., 2025b) with 105k data. The training batch size is set to 256, and the number of responses for each question is set to 8 by default. The total trianing step is set to 400 by default. All the training experiments are done based on verl  (Sheng et al., 2025) with a single 8xH100 GPUs node. More hyperparameters setting for reasoning RL training can be found in  Appendix [C](https://arxiv.org/html/2510.04072v2#A3).

Baseline and Evaluation. We compare SFPO with vanilla GRPO and the base model without rl training on six commonly used mathematical reasoning benchmarks with variant difficulty: Math500 (Hendrycks et al., 2021), AIME24 (Art of Problem Solving, 2024a), AIME25 (MAA,), AMC (Art of Problem Solving, 2024b), MinervaMath (Lewkowycz et al., 2022), Olympiad Bench (He et al., 2024). Each benchmark is evaluated multiple times with rollout temperature being 1, and we report the average Pass@1 accuracy by default.

**Table 1: Performance on math reasoning benchmarks with DAPO and Math training dataset.**
| Model | Method | Math-500 | AIME24 | AIME25 | AMC | Minerva | Olympiad | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2.5-Math-1.5B | Base | 55.55 | 9.17 | 5.83 | 37.65 | 17.74 | 28.45 | 25.73 |
| GRPO | 77.15 | 16.67 | 11.67 | 53.31 | 31.89 | 39.42 | 38.35 |  |
| \cellcolorcustomblueSFPO | \cellcolorcustomblue78.35 | \cellcolorcustomblue20.00 | \cellcolorcustomblue15.00 | \cellcolorcustomblue56.02 | \cellcolorcustomblue32.07 | \cellcolorcustomblue39.72 | \cellcolorcustomblue40.19 |  |
| Qwen2.5-Math-7B | Base | 71.65 | 21.67 | 9.17 | 53.61 | 27.02 | 38.54 | 36.94 |
| GRPO | 82.50 | 34.20 | 20.83 | 71.08 | 36.76 | 44.80 | 48.36 |  |
| \cellcolorcustomblueSFPO | \cellcolorcustomblue82.30 | \cellcolorcustomblue35.00 | \cellcolorcustomblue20.83 | \cellcolorcustomblue74.49 | \cellcolorcustomblue36.94 | \cellcolorcustomblue45.59 | \cellcolorcustomblue49.19 |  |
| DS-distilled-Qwen-1.5B | Base | 73.80 | 16.67 | 20.00 | 47.59 | 28.86 | 36.94 | 37.27 |
| GRPO | 84.65 | 30.00 | 23.33 | 66.86 | 31.71 | 49.85 | 47.73 |  |
| \cellcolorcustomblueSFPO | \cellcolorcustomblue86.10 | \cellcolorcustomblue32.50 | \cellcolorcustomblue30.83 | \cellcolorcustomblue70.28 | \cellcolorcustomblue32.81 | \cellcolorcustomblue50.67 | \cellcolorcustomblue50.53 |  |
| DS-distilled-Qwen-7B | Base | 83.60 | 28.33 | 25.00 | 61.75 | 41.73 | 48.89 | 48.21 |
| GRPO | 91.7 | 50.00 | 35.83 | 80.42 | 43.65 | 61.24 | 60.47 |  |
| \cellcolorcustomblueSFPO | \cellcolorcustomblue92.60 | \cellcolorcustomblue54.17 | \cellcolorcustomblue37.50 | \cellcolorcustomblue83.75 | \cellcolorcustomblue44.49 | \cellcolorcustomblue65.73 | \cellcolorcustomblue63.04 |  |
| Qwen3-4B-Base | Base | 45.25 | 2.50 | 0.83 | 20.48 | 15.99 | 20.66 | 17.62 |
| GRPO | 83.35 | 16.67 | 17.50 | 59.03 | 38.78 | 48.59 | 43.99 |  |
| \cellcolorcustomblueSFPO | \cellcolorcustomblue84.30 | \cellcolorcustomblue21.67 | \cellcolorcustomblue20.83 | \cellcolorcustomblue57.23 | \cellcolorcustomblue40.81 | \cellcolorcustomblue48.67 | \cellcolorcustomblue45.59 |  |

### 4.2 Main Results

#### 4.2.1 Math Reasoning Benchmarks

As illustrated in Table [1](https://arxiv.org/html/2510.04072v2#S4.T1), our proposed SFPO consistently outperforms vanilla GRPO across all base models and benchmarks. Specifically, for small-scale models such as Qwen2.5-Math-1.5B and DS-distilled-Qwen-1.5B, SFPO demonstrates superior performance enhancements on math reasoning benchmarks, raising the average accuracy from 38.35 to 40.19 with an absolute gain of +1.84, and from 47.73 to 50.53 with a gain of +2.80, respectively. The improvements are particularly pronounced on challenging tasks such as AIME24 and AIME25, where DS-distilled-Qwen-1.5B achieves an absolute gain of +7.5 on AIME25. The larger models also exhibit similar performance gains. For Qwen2.5-Math-7B, SFPO raises the average accuracy from 48.36 to 49.19 with an absolute gain of +1.80. For DS-distilled-Qwen-7B, SFPO boosts the average accuracy from 60.47 to 63.04, corresponding to an absolute gain of +0.8. For Qwen3-4B-Base model, SFPO improves average accuracy from 43.99 to 45.59, an absolute gain of +1.60, highlighting its robustness across various models. Moreover, Table [2](https://arxiv.org/html/2510.04072v2#S4.T2) demonstrates that SFPO can consistently achieve better training performance compared to GRPO for the larger training dataset Skywork-or1, proving the robustness of SFPO across different scales and distributions of training datasets.

#### 4.2.2 Training Dynamics.

**Table 2: Performance on AIME24/25 with Skywork-or1 training dataset.**
| Model | Method | AIME24 | AIME25 |
| --- | --- | --- | --- |
| DS-Qwen-1.5B | Base | 20.40 | 17.90 |
| GRPO | 32.92 | 25.83 |  |
| \cellcolorcustomblueSFPO | \cellcolorcustomblue34.17 | \cellcolorcustomblue27.50 |  |
| DS-Qwen-7B | Base | 25.42 | 22.92 |
| GRPO | 42.50 | 29.20 |  |
| \cellcolorcustomblueSFPO | \cellcolorcustomblue43.75 | \cellcolorcustomblue30.00 |  |

We compare the training dynamics between SFPO and GRPO to better understand the differences in optimization behavior as shown in Fig. [2](https://arxiv.org/html/2510.04072v2#S4.F2) and Fig. [3](https://arxiv.org/html/2510.04072v2#S4.F3). More comprehensive training dynamics for other models can be found in  Appendix [D](https://arxiv.org/html/2510.04072v2#A4).

Validation. From Fig. [2](https://arxiv.org/html/2510.04072v2#S4.F2), we can clearly spot that SFPO consistently outperforms GRPO across all base models throughout the training process. Not only does SFPO achieve faster convergence in the early stages, but it also sustains higher global accuracy by the end of training. For example, Qwen3-4B-Base model achieves a sharper rise and stabilizes at a higher accuracy within only 150 training steps, while vanilla GRPO cannot surpass this accuracy even after 400 steps.

Response Length. Moreover, distinct training behaviors between SFPO and GRPO for DeepSeek-R1-Distill-Qwen-7B are shown in Fig. [3](https://arxiv.org/html/2510.04072v2#S4.F3) including the response length, entropy loss, and the reward throughout the training process.

Specifically, GRPO gradually collapses to overly short responses while SFPO quickly converges to a stable range of around 2700 tokens with better accuracy, highlighting SFPO’s ability to regulate response length more effectively and avoid overthinking with verbose responses (Liu et al., 2025).

Entropy. From Fig. [3](https://arxiv.org/html/2510.04072v2#S4.F3)(b), we can observe that SFPO makes model’s entropy loss lower compared to GRPO. Typically, lower entropy means weak exploration ability for reasoning models; however, the entropy reduction under SFPO mainly reflects the model’s ability to eliminate unproductive search paths early, rather than suppressing exploration altogether as evidenced by its sustained accuracy gains. In fact, the model still explores a sufficiently broad set of reasoning trajectories; therefore, the lower entropy observed under SFPO should be viewed as a sign of more efficient exploration rather than a sign of limited exploration.

Figure: Figure 2: Average validation accuracy of different base models throughout the training process.
Refer to caption: x2.png

Figure: Figure 3: Training dynamics for DeepSeek-R1-Distilled-Qwen-7B, comparing GRPO and SFPO across response length, entropy loss, and reward.
Refer to caption: x3.png

Reward Score. SFPO also achieves a higher and more stable reward throughout the training process compared to GRPO, indicating stronger alignment with the reward function and more robust convergence. This is further reflected in its superior accuracy and well-controlled response length.

#### 4.2.3 Efficiency Analysis.

Figure: Figure 4: Comparison of GRPO and SFPO. (a) Number of rollouts required to achieve the best accuracy of GRPO. (b) Corresponding training time.
Refer to caption: x4.png

We evaluate the efficiency gains of SFPO over GRPO by comparing the total number of rollouts and the wall-clock time required to reach the same benchmark accuracy. As shown in Fig. [4](https://arxiv.org/html/2510.04072v2#S4.F4), SFPO consistently surpasses GRPO in rollout efficiency and training speed across all model scales. Specifically, SFPO requires 3.21×, 3.50×, and 4.93× fewer rollouts than GRPO for DS-Qwen-1.5B, Qwen3-4B-Base, and DS-Qwen-7B, respectively, to reach the same best accuracy. This advantage directly translates into reduced training time, where SFPO achieves 2.62×, 2.65×, and 4.19× speedups over GRPO for the same models, significantly lowering the training cost. Note that SFPO does not introduce extra GPU memory overhead as it does not need to store the heavy optimizer status. The detailed profiling results for GPU memory usage in the training process can be found in Appendix [B](https://arxiv.org/html/2510.04072v2#A2). These significant efficiency gains align with our expectations, since the primary bottleneck in the training process lies in rollout generation, which accounts for more than 70% of the overall inference time (He et al., 2025a). By substantially reducing the number of rollouts required and harnessing the reposition mechanism, SFPO alleviates this bottleneck and achieves faster training.

## 5 Analysis and Ablation Study

Impact of $\alpha$ and $K$. As discussed in Sec. [3](https://arxiv.org/html/2510.04072v2#S3), the hyperparameters $\alpha$ and $K$ jointly determine the stability of SFPO. To better understand their impacts, we study two settings: small $K{=}3$ and large $K{=}7$, under varying $\alpha$, as shown in Fig. [5](https://arxiv.org/html/2510.04072v2#S5.F5). When $K$ is small, SFPO remains stable across different $\alpha$ values and consistently outperforms GRPO. This aligns with our theoretical intuition that performing multiple inner updates on the same rollout batch, followed by reposition, effectively reduces gradient noise and produces a more reliable update direction. However, when $K$ is large, the fast weights drift substantially from the original parameters. A large $\alpha$ then amplifies this mismatch by pulling the slow weights too aggressively toward unstable fast trajectories, injecting noise and causing performance drops. As shown in Fig. [5](https://arxiv.org/html/2510.04072v2#S5.F5)(b), a smaller $\alpha$ mitigates this drift and restores stability, confirming the interaction between $K$ and $\alpha$.

Figure: Figure 5: Average training accuracy of different settings throughout the training process. (a): Small k=3 with varying values of $\alpha$. (b): Large k=7 with varying values of $\alpha$. (c): Varying values of k with fixed $\alpha=0.8$. (d): Impact of the existence of stage III.
Refer to caption: x5.png

Figure: Figure 6: Comparison between SFPO w/ and w/o entropy control (EC). The blue dashed line indicates the stop step identified by z-score.
Refer to caption: x6.png

Interpolation Against Off-Policy Overfitting. When $\alpha=1$ (no interpolation), performance initially rises but steadily declines as training progresses. Without interpolation, the model rapidly adapts by overfitting to small batches of rollouts, yielding short-term gains but injecting growing noise into gradient updates, which causes instability and long-term degradation. This effect is amplified when $K$ is large, as shown in Fig. [5](https://arxiv.org/html/2510.04072v2#S5.F5)(b), where the fast weights drift substantially and the slow weights fully adopt these noisy trajectories, leading to sharp performance drops.
In contrast, for small $\alpha$, interpolation keeps the update closer to the on-policy region, mitigating distributional drift and stabilizing training. However, if $\alpha$ is overly small and $K$ is small, the algorithm under-utilizes the stabilized fast trajectory, slightly slowing early improvements. While for large $K$, a smaller $\alpha$ is preferable, where interpolation effectively counteracts the severe off-policy drift induced by many inner steps. Overall, interpolation serves as a simple, effective regularizer against the potential off-policy overfitting introduced by Stage I.

The Importance of Stage III: Slow Correction.
As shown in Fig. [5](https://arxiv.org/html/2510.04072v2#S5.F5)(d), incorporating slow correction consistently improves stability and accuracy over GRPO. Without this stage, the reposition stage leaves the iterate at an interpolated point that may deviate from the true descent direction. Intuitively, slow correction provides a curvature-aware adjustment that realigns updates with the correct optimization trajectory.
Beyond its immediate benefit, slow correction also provides a natural interface for future extensions: As discussed in Sec. [7](https://arxiv.org/html/2510.04072v2#S7), one could, for instance, perform an additional rollout on a small held-out meta-test batch after repositioning, enabling curriculum or meta-learning variants of SFPO without altering its overall structure.

Necessity of Entropy Control.
We evaluate the effect of adaptive entropy control (EC) on SFPO using the DS-Qwen-7B model. As shown in Fig. [7](https://arxiv.org/html/2510.04072v2#S5.F7), removing EC causes a noticeable accuracy drop after roughly 100 steps. This degradation coincides with a rapid divergence of entropy loss, suggesting that the policy becomes unstable and overfits to noisy rollouts. These results highlight entropy control as a key factor for maintaining the stability and reliability of SFPO.
For simplicity, we implement entropy control through a lightweight scheduling of $\alpha$. As shown in Alg. [1](https://arxiv.org/html/2510.04072v2#alg1), once a predefined entropy-trigger is activated, $\alpha$ is set to 0, reverting SFPO to standard GRPO for subsequent steps. This simple mechanism effectively stabilizes training while retaining the efficiency gains.

Strategies for Decaying $\alpha$.
We further investigate alternative strategies for reducing the value of $\alpha$ once the stop step is identified by the z-score criterion. Specifically, we compare two representative approaches: (i) our default method, where $\alpha$ is directly set to zero after the trigger step $s^{\star}$, and (ii) a more gradual linear decay schedule, where $\alpha$ decreases to zero over several subsequent steps. As illustrated in Fig. [7](https://arxiv.org/html/2510.04072v2#S5.F7), both strategies yield similar accuracy and stability curves, suggesting that the decay schedule itself has negligible influence on the overall performance of SFPO. This result implies that once the entropy-trigger condition is met, the role of $\alpha$ becomes marginal, and any further adjustment has limited benefit. Consequently, for simplicity and wall-clock efficiency, we adopt the direct reset rule of setting $\alpha=0$ after $s^{\star}$. The formal procedure is summarized in Alg. [1](https://arxiv.org/html/2510.04072v2#alg1), and additional details can be found in Sec. [3.4](https://arxiv.org/html/2510.04072v2#S3.SS4).

## 6 Related Works

##### RL for LLM Reasoning.

OpenAI O1 (OpenAI,) introduced a paradigm shift in LLM reasoning by extending the reasoning horizon before final responses. DeepSeek-R1 (Guo et al., 2025) further advanced this line by open-sourcing both its training algorithm, the value-model-free GRPO (Shao et al., 2024), and model weights, achieving performance comparable to O1. Subsequent work has focused on stabilizing and simplifying GRPO: DAPO (Yu et al., 2025b) identifies entropy collapse as a key challenge and proposes effective remedies, while Dr. GRPO (Liu et al., 2025) removes normalization terms without sacrificing performance. In contrast, SFPO is orthogonal to both families: it leaves the objective unchanged yet restructures the update itself into a fast–reposition–slow decomposition, improving variance reduction, stability, and sample efficiency in LLM reasoning.

##### Data Efficiency for LLM Reasoning.

Despite focusing on designing novel training pipelines, a complementary line of work (Ivison et al., 2025, Xia et al., 2024, Muennighoff et al., 2025, Ye et al., 2025) improves the efficiency of LLM training through data filtering. One direction focuses on pruning data for supervised fine-tuning (Xia et al., 2024, Chen et al., 2023, Ivison et al., 2022).
Another direction targets reinforcement learning, where studies (Muldrew et al., 2024, Liu et al., 2024, Das et al., 2024, Li et al., 2025, Fatemi et al., 2025, Wang et al., 2025) show that GRPO requires only a small subset of the training data to improve reasoning ability. However, these methods largely optimize *which data* is used rather than *how updates are performed*. SFPO is orthogonal: it assumes no change to the data pipeline but restructures the policy update itself. By converting one-shot updates into a fast–reposition–slow trajectory, SFPO reduces variance and stabilizes learning.

##### Off-policy GRPO.

Original iterative GRPO  (Shao et al., 2024) tries to update policy model for multiple times with the same batch of rollout data. Mroueh (2025) provided a theoretical analysis for this off-policy policy gradient operation under verifiable rewards, reformulating GRPO as a KL-regularized contrastive loss and proving its iterative dynamics amplify success rates relative to the reference policy. Mroueh et al. (2025) examined both on- and off-policy variants, deriving reward improvement conditions, introducing clipped surrogate objectives to stabilize training, and empirically showing off-policy GRPO can match or exceed on-policy performance. Unlike these modifications, SFPO preserves the on-policy GRPO update but restructures one-step optimization into a fast-reposition-slow framework to tackle sample inefficiency and stabilize updates.

## 7 Conclusion

We proposed Slow-Fast Policy Optimization (SFPO), a simple three-stage update framework that stabilizes early training by combining a fast trajectory, a reposition mechanism, and a slow correction. SFPO directly addresses the instability of one-shot GRPO updates, achieving higher reasoning accuracy and fewer generated tokens without added wall-clock cost. Beyond these gains, its modular structure naturally opens avenues for curriculum or meta-learning extensions, and our entropy-triggered $\alpha$ schedule suggests richer adaptive rules. We believe SFPO offers not only an effective plug-in for current policy optimization, but also a foundation for scalable and data-aware optimization strategies in the LLM reasoning area.

## References

- Art of Problem Solving (2024a)
Art of Problem Solving.
Aime problems and solutions.
[https://artofproblemsolving.com/wiki/index.php/AIME](https://artofproblemsolving.com/wiki/index.php/AIME), 2024a.
Accessed: 2025-04-20.
- Art of Problem Solving (2024b)
Art of Problem Solving.
Amc problems and solutions.
[https://artofproblemsolving.com/wiki/index.php?title=AMC](https://artofproblemsolving.com/wiki/index.php?title=AMC), 2024b.
Problems and Solutions, 2024b. Accessed: 2025-04-20.
- Chen et al. (2023)
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al.
Alpagasus: Training a better alpaca with fewer data.
*arXiv preprint arXiv:2307.08701*, 2023.
- Dai et al. (2025)
Muzhi Dai, Shixuan Liu, and Qingyi Si.
Stable reinforcement learning for efficient reasoning.
*arXiv preprint arXiv:2505.18086*, 2025.
URL [https://arxiv.org/abs/2505.18086](https://arxiv.org/abs/2505.18086).
- Das et al. (2024)
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury.
Active preference optimization for sample efficient rlhf.
*arXiv preprint arXiv:2402.10500*, 2024.
- DeepSeek-AI et al. (2025a)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong
Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu,
Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025a.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- DeepSeek-AI et al. (2025b)
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang
Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang
Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan.
Deepseek-v3 technical report, 2025b.
URL [https://arxiv.org/abs/2412.19437](https://arxiv.org/abs/2412.19437).
- Engstrom et al. (2020)
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry.
Implementation matters in deep policy gradients: A case study on ppo and trpo.
*arXiv preprint arXiv:2005.12729*, 2020.
- Fatemi et al. (2025)
Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula.
Concise reasoning via reinforcement learning.
*arXiv preprint arXiv:2504.05185*, 2025.
- Guo et al. (2025)
D. Guo et al.
Deepseek-r1: Incentivizing reasoning capability in llms through rl.
*Nature*, 2025.
URL [https://doi.org/10.1038/s41586-025-09422-z](https://doi.org/10.1038/s41586-025-09422-z).
To appear / in press.
- He et al. (2024)
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.
OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems.
*arXiv preprint arXiv:2402.14008*, 2024.
URL [https://arxiv.org/abs/2402.14008](https://arxiv.org/abs/2402.14008).
- He et al. (2025a)
Jingkai He, Tianjian Li, Erhu Feng, Dong Du, Qian Liu, Tao Liu, Yubin Xia, and Haibo Chen.
History rhymes: Accelerating llm reinforcement learning with rhymerl, 2025a.
URL [https://arxiv.org/abs/2508.18588](https://arxiv.org/abs/2508.18588).
- He et al. (2025b)
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou.
Skywork open reasoner 1 technical report, 2025b.
URL [https://arxiv.org/abs/2505.22312](https://arxiv.org/abs/2505.22312).
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset, 2021.
URL [https://arxiv.org/abs/2103.03874](https://arxiv.org/abs/2103.03874).
- Ivison et al. (2022)
Hamish Ivison, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi.
Data-efficient finetuning using cross-task nearest neighbors.
*arXiv preprint arXiv:2212.00196*, 2022.
- Ivison et al. (2025)
Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi.
Large-scale data selection for instruction tuning.
*arXiv preprint arXiv:2503.01807*, 2025.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention, 2023.
URL [https://arxiv.org/abs/2309.06180](https://arxiv.org/abs/2309.06180).
- Lewkowycz et al. (2022)
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and et al.
Solving quantitative reasoning problems with language models.
In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 35, pp. 3843–3857, 2022.
- Li et al. (2025)
Xuefeng Li, Haoyang Zou, and Pengfei Liu.
Limr: Less is more for rl scaling.
*arXiv preprint arXiv:2502.11886*, 2025.
- Liu et al. (2025)
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin.
Understanding r1-zero-like training: A critical perspective.
*arXiv preprint arXiv:2503.20783*, 2025.
- Liu et al. (2024)
Zijun Liu, Boqun Kou, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu.
Enabling weak llms to judge response reliability via meta ranking.
*arXiv preprint arXiv:2402.12146*, 2024.
- (22)
MAA.
American invitational mathematics examination (aime).
[https://maa.org/math-competitions/aime](https://maa.org/math-competitions/aime).
Mathematics Competition Series; n.d.a.
- Mroueh (2025)
Youssef Mroueh.
Reinforcement learning with verifiable rewards: Grpo’s effective loss, dynamics, and success amplification.
*arXiv preprint arXiv:2503.06639*, 2025.
- Mroueh et al. (2025)
Youssef Mroueh, Nicolas Dupuis, Brian Belgodere, Apoorva Nitsure, Mattia Rigotti, Kristjan Greenewald, Jiri Navratil, Jerret Ross, and Jesus Rios.
Revisiting group relative policy optimization: Insights into on-policy and off-policy training.
*arXiv preprint arXiv:2505.22257*, 2025.
- Muennighoff et al. (2025)
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto.
s1: Simple test-time scaling.
*arXiv preprint arXiv:2501.19393*, 2025.
- Muldrew et al. (2024)
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber.
Active preference learning for large language models.
*arXiv preprint arXiv:2402.08114*, 2024.
- (27)
OpenAI.
Learning to reason with llms: Openai o1 preview.
URL [https://openai.com/index/introducing-openai-o1-preview](https://openai.com/index/introducing-openai-o1-preview).
- OpenAI et al. (2024)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo
Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan
Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez,
Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia
Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.
Gpt-4 technical report, 2024.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Shen et al. (2025)
Si Shen, Peijun Shen, Wenhua Zhao, and Danhao Zhu.
Mitigating think-answer mismatch in llm reasoning through noise-aware advantage reweighting.
*arXiv preprint arXiv:2508.05928*, 2025.
URL [https://arxiv.org/abs/2508.05928](https://arxiv.org/abs/2508.05928).
- Sheng et al. (2025)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
In *Proceedings of the Twentieth European Conference on Computer Systems*, EuroSys ’25, pp. 1279–1297. ACM, March 2025.
doi: 10.1145/3689031.3696075.
URL [http://dx.doi.org/10.1145/3689031.3696075](http://dx.doi.org/10.1145/3689031.3696075).
- Wang et al. (2025)
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al.
Reinforcement learning for reasoning in large language models with one training example.
*arXiv preprint arXiv:2504.20571*, 2025.
- Xia et al. (2024)
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen.
Less: Selecting influential data for targeted instruction tuning.
*arXiv preprint arXiv:2402.04333*, 2024.
- Yang et al. (2024)
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang.
Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024.
URL [https://arxiv.org/abs/2409.12122](https://arxiv.org/abs/2409.12122).
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report, 2025.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Ye et al. (2025)
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu.
Limo: Less is more for reasoning.
*arXiv preprint arXiv:2502.03387*, 2025.
- Yu et al. (2025a)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang.
Dapo: An open-source llm reinforcement learning system at scale, 2025a.
URL [https://arxiv.org/abs/2503.14476](https://arxiv.org/abs/2503.14476).
- Yu et al. (2025b)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025b.
- Zhang et al. (2019)
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton.
Lookahead optimizer: k steps forward, 1 step back.
*Advances in neural information processing systems*, 32, 2019.
- Zhao et al. (2023)
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li.
Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023.
URL [https://arxiv.org/abs/2304.11277](https://arxiv.org/abs/2304.11277).
- Zheng et al. (2025)
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen.
Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts.
*arXiv preprint arXiv:2506.02177*, 2025.

## Appendix A LLM Usage

Large language models (LLMs) were used solely to improve the writing of this paper, including grammar, clarity, and readability. They were not used for generating ideas, designing experiments, conducting analyses, or producing scientific content. All research contributions, technical claims, and conclusions are entirely the work of the authors.

## Appendix B GPU Memory Profiling Results

Figure: Figure 8: Comparison of GPU memory consumption during one RL training step between SFPO and GRPO for Deepseek-R1-Distill-Qwen-7B model.
Refer to caption: x8.png

From Fig. [8](https://arxiv.org/html/2510.04072v2#A2.F8), we can clearly observe that SFPO and GRPO demonstrate similar GPU memory consumption during one training step. This aligns with our expectation: SFPO does not need to store the heavy optimizer states and parameters but only need to store one copy of the model weight, which does not introduce significant overhead, especially when the model is sharded across GPUs.

## Appendix C Training Details

We conduct all experiments based on verl  (Sheng et al., 2025) Rl training framework. For training the policy model, we use Pytorch Fully Sharded Data Parallel (FSDP) distributed training backend  (Kwon et al., 2023). For inference, we use vLLM framework  (Zhao et al., 2023). The major hyperparameters used in reasoning RL training are listed in  Table [3](https://arxiv.org/html/2510.04072v2#A3.T3). Note that we do not use KL-divergence term in the objective loss function as suggested by previous works  (Yu et al., 2025a).

**Table 3: Hyperparameters for RL training.**
| Name | Value | Description |
| --- | --- | --- |
| Training batch size | 256 | Global training batch size for RL training. |
| Mini batch size | 64 | Sub-batches used to split the global training batch. |
| n | 8 | Number of responses to generate per sample. |
| Rollout temperature | 1.0 | Sampling temperature for rollout generation. |
| top_p | 0.7 | Top-p sampling parameter for rollout generation. |
| Optimizer | AdamW | The optimizer used. |
| lr | 1e-6 (Constant) | Learning rate. |
| grad_clip | 1.0 | Gradient clipping threshold. |

## Appendix D Training Dynamics

We illustrate the training dynamics of reasoning RL on the DAPO and math datasets covering entropy loss, mean response length, and reward for Qwen2.5-Math-1.5B, DeepSeek-r1-Distilled-Qwen-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base, as shown in  Fig. [9](https://arxiv.org/html/2510.04072v2#A4.F9),  Fig. [10](https://arxiv.org/html/2510.04072v2#A4.F10),  Fig. [11](https://arxiv.org/html/2510.04072v2#A4.F11), and  Fig. [12](https://arxiv.org/html/2510.04072v2#A4.F12), respectively. Overall, SFPO achieves higher rewards with smoother, more stable training with reducing entropy fluctuations and controlling response length, making it faster, more sample-efficient, and easier to tune than GRPO.

Figure: Figure 9: Training dynamics for Qwen2.5-Math-1.5B, comparing GRPO and SFPO across response length, entropy loss, and reward.
Refer to caption: x9.png

Figure: Figure 10: Training dynamics for DeepSeek-R1-Distilled-Qwen-1.5B, comparing GRPO and SFPO across response length, entropy loss, and reward.
Refer to caption: x10.png

Figure: Figure 11: Training dynamics for Qwen2.5-Math-7B, comparing GRPO and SFPO across response length, entropy loss, and reward.
Refer to caption: x11.png

Figure: Figure 12: Training dynamics for Qwen3-4B-Base, comparing GRPO and SFPO across response length, entropy loss, and reward.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: - Simple, drop-in recipe that practitioners can try with minimal code churn (Alg. 1).
- Large rollout/time reductions to reach GRPO’s best accuracy (Fig. 4), aligning with the observation that rollouts dominate wall-clock.
summary: The paper proposes Slow–Fast Policy Optimization (SFPO), it's a three-stage modification to on-policy GRPO-style training for reasoning LLMs: (i) a fast trajectory of $K$ inner gradient steps on the same rollout batch, (ii) a reposition interpolation back toward the start point controlled by $\alpha$, and (iii) a slow correction extra step. See Alg. 1 (p. 3). Authors also introduce an entropy-triggered schedule that sets $\alpha \leftarrow 0$ after a $z$-score threshold on policy entropy (Eq. 11), which effectively reverts to GRPO near convergence.
weaknesses: 1. GRPO-K: $K$ inner updates on the same batch without reposition or slow correction (a direct control for “just do more steps per batch”).  

2. Lookahead-GRPO applies only the Stage II interpolation around GRPO.  

3. Extra-grad-GRPO applies only the Stage III predictor–corrector step.  

4. Clarify whether per-iteration compute (backprop) increases due to $K$; if so, compare at matched FLOPs (or matched wall-clock) and at fixed accuracy targets, not “GRPO’s best accuracy”.

5. The theoretical section is insufficient. Make Eq. (10) precise (define $c$ and $F(K,\alpha)$; assumptions on clipping/KL; conditions handling negative curvature), or clearly mark it as heuristic.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How does the proposed direction-based extrapolation perform in non-mathematical reasoning tasks?

The token-replacement experiment convincingly shows that \Delta\log p identifies critical tokens. However, does this intervention sometimes harm performance? Are there cases were replacing a base model token with the RLVR model's choice leads to a wrong answer, and what characterizes those tokens?
soundness: 3
strengths: The paper introduces a novel and intuitive directional metric that effectively captures sparse, reasoning-critical updates, supported by rigorous token-replacement experiments and gradient analysis.

 The proposed methods are simple yet effective, demonstrating consistent gains across multiple models and benchmarks without requiring additional training data.
summary: This paper argues that the direction of token-level updates, measured by the signed log-probability difference between base and RLVR models, is more informative than magnitude-based metrics for understanding RLVR's effect on LLM reasoning. The authors propose and validate two methods, test-time extrapolation and training-time advantage reweighting, that exploit the directional insight to improve reasoning performance.
weaknesses: The test-time extrapolation method requires access to both the base and RLVR models, which may limit its practicality in settings where only the fine-tuned model is available.

    The paper focuses primarily on mathematical reasoning benchmarks (e.g., AIME, AMC); it remains unclear whether the findings generalize to other reasoning domains or more diverse tasks.

    The theoretical justification (Theorem 4.1) relies on a simplified tabular softmax bandit setting, which may not fully reflect the complexity of modern LLM training dynamics.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1.  Could you clarify the nomenclature for Stage I, the 'Fast Trajectory'? The term 'fast' is somewhat ambiguous. Does it refer to the speed of convergence, the magnitude of the parameter change, or some other property?
2.  A crucial baseline seems to be missing from the ablation studies. To isolate the benefits of Stage II (Reposition) and Stage III (Slow Correction), have you considered a baseline that only uses Stage I? This would be equivalent to applying GRPO with $K$ inner updates on the same batch, which would more clearly demonstrate the contribution of the reposition and slow correction steps.
3.  Stage II is designed to control off-policy drift via interpolation. A more common approach in policy gradient methods is to use a KL divergence penalty to constrain the policy update. Could you comment on the design choice of interpolation over a KL penalty? Have you compared the performance of your reposition step against a KL-constrained update in this multi-step update setting?
4.  The paper claims that SFPO addresses the high-variance gradient issue in GRPO. A significant source of variance in GRPO can come from batches with uniform rewards (e.g., all successes or all failures), which result in zero advantage for all samples. Does SFPO's performance gain stem primarily from better handling these 'zero-advantage' batches? To verify this, it might be necessary to conduct experiments with an algorithm like DAPO, which is explicitly designed to use *dynamic sampling* to mitigate this issue, to see if SFPO provides similar benefits in a setting where this specific source of variance is already addressed.
5.  Could you provide a derivation or at least a reference for the descent guarantee in Equation (10)? A more formal explanation of the assumptions and the term $F(K, \alpha) $ would strengthen the paper's theoretical grounding.
6.  Would you be able to correct the numerical errors in the text describing the results in Table 1 (lines 322-323)? The discrepancies between the reported gains and the values in the table are substantial.
7. The interpretation of lower entropy in SFPO as "more efficient exploration" (lines 374-375) is intriguing. Could you elaborate on this? Is it possible that the method simply encourages faster convergence to a more deterministic policy, which might be beneficial for exploitation but could also be interpreted as a sign of reduced exploration?

I look forward to the author's response. I am willing to reconsider my score if the questions above, particularly those concerning the experimental details, numerical correctness, and key design choices, are adequately addressed.
soundness: 3
strengths: 1.  The proposed method addresses a significant and well-recognized problem: the instability and sample inefficiency of on-policy algorithms like GRPO in the context of LLM reasoning. The proposed three-stage solution is simple, intuitive, and clearly motivated as a way to reduce update variance and make better use of collected data.
2.  A major strength of SFPO is its "plug-and-play" compatibility with existing policy gradient pipelines. By modifying only the parameter update rule while leaving the loss function, rollout generation, and regularization terms of GRPO intact, the method offers a practical, low-effort path to improving existing training setups.
3.  The paper is supported by a comprehensive set of experiments that demonstrate consistent and substantial gains over a strong GRPO baseline across multiple models and benchmarks. The reported improvements in sample efficiency (up to $4.93\times$ fewer rollouts) and wall-clock time (up to $4.19\times $ reduction) are highly significant. Furthermore, the ablation studies effectively justify the inclusion of each component of SFPO, particularly the adaptive entropy control and the slow correction stage.
summary: This paper introduces Slow-Fast Policy Optimization (SFPO), a three-stage update mechanism designed to improve the stability and sample efficiency of on-policy reinforcement learning for LLM reasoning. SFPO replaces the standard single gradient step with a fast trajectory of multiple inner updates, a reposition step to control off-policy drift, and a final slow correction step. Extensive experiments demonstrate that SFPO significantly outperforms the GRPO baseline in accuracy, convergence speed, and wall-clock time on various math reasoning benchmarks.
weaknesses: 1.  The reporting of experimental results contains significant numerical errors and omissions. In Section 4.2.1, the claimed performance gains for Qwen2.5-Math-7B (+1.80) and DS-distilled-Qwen-7B (+0.8) do not match the values in Table 1, which show gains of +0.83 and +2.57, respectively. 
2.  The reference list is not properly curated. For example, the citations "Yu et al., 2025a" and "Yu et al., 2025b" both point to the exact same arXiv preprint, which is misleading.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What does "Slow" refer to in stage 3? Does this stage require significantly more time to update the gradient?

2. Could you elaborate on the time consumption in Stage 1? Since gradient updates are computationally intensive, this stage might become slower as K increases. Additionally, in agentic scenarios involving multi-turn interactive trajectories, performing multiple policy gradient updates may significantly reduce optimization efficiency.
soundness: 3
strengths: 1. The method is well-structured and thoughtfully designed. The Fast Trajectory stage stabilizes the optimization process, while the Reposition and Slow Correction stages help to better align the optimization with the on-policy objective. The motivation behind each stage is clearly articulated, making the overall approach sound and convincing.

2. The empirical results are thorough and well-presented. The paper evaluates SFPO across 5 models, 3 methods, and 6 benchmarks, clearly demonstrating substantial improvements in both efficacy and efficiency. Furthermore, comprehensive analyses and ablation studies are provided, strengthening the validity of SFPO’s performance claims.

3. The paper is well-organized, featuring clear and informative figures, tables, and algorithm illustrations. Additionally, Section 3 provides intuitive explanations that greatly aid in understanding the underlying mechanism of the proposed SFPO.
summary: This paper introduces Slow-Fast Policy Optimization (SFPO), a plug-and-play method to stablize and accelerate the learning of on-policy RL algorithms such as GRPO. Specifically, SFPO has three stages for each training step: Fast Trajectory, Reposition, and Slow Correction, which mitigate the noisy single-step rollout updates and ensures a more stable optimization process. Empirical studies in mathmatical benchmarks demonstrates the consistent improvement of SFPO compared with GRPO.
weaknesses: 1. The empirical evaluation is somewhat limited in scope, as all experiments are conducted only on mathematical benchmarks, without including domains such as code or logic reasoning. Additionally, GRPO is the sole baseline used; incorporating additional baselines like DAPO and GSPO would further strengthen the evidence for SFPO’s effectiveness.

2. The ablation studies indicate that SFPO is sensitive to the hyperparameters $\alpha$ and $K$, which may hinder its direct applicability to domains beyond mathematical reasoning. Users may need to carefully tune these hyperparameters to achieve optimal performance.


Decision: Accept (Poster)