Single-stream Policy Optimization
Authors: Zhongwen Xu
Zihan Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Single-stream Policy Optimization
Large Language Models
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: We revisit policy-gradient optimization for Large Language Models (LLMs) from a single-stream perspective. Prevailing group-based methods like GRPO reduce variance with on-the-fly baselines but suffer from critical flaws: frequent degenerate groups erase learning signals, and synchronization barriers hinder scalability. We introduce Single-stream Policy Optimization (SPO), which eliminates these issues by design. SPO replaces per-group baselines with a persistent, KL-adaptive value tracker and normalizes advantages globally across the batch, providing a stable, low-variance learning signal for every sample. Being group-free, SPO enables higher throughput and scales effectively in long-horizon or tool-integrated settings where generation times vary. Furthermore, the persistent value tracker naturally enables an adaptive curriculum via prioritized sampling. Experiments using Qwen3-8B show that SPO converges more smoothly and attains higher accuracy than GRPO, while eliminating computation wasted on degenerate groups. Ablation studies confirm that SPO's gains stem from its principled approach to baseline estimation and advantage normalization, offering a more robust and efficient path for LLM reasoning. Across five hard math benchmarks with Qwen3-8B, SPO improves the average maj@32 by $+3.4\ \text{percentage points} (\mathrm{pp})$ over GRPO, driven by substantial absolute point gains on challenging datasets, including $+7.3\ \mathrm{pp}$ on BRUMO 25, $+4.4\ \mathrm{pp}$ on AIME 25, $+3.3\ \mathrm{pp}$ on HMMT 25, and achieves consistent relative gain in pass@$k$ across the evaluated $k$ values. SPO's success challenges the prevailing trend of adding incidental complexity to RL algorithms, highlighting a path where fundamental principles, not architectural workarounds, drive the next wave of progress in LLM reasoning.

Published as a conference paper at ICLR 2026
SINGLE-STREAM POLICY OPTIMIZATION
Zhongwen Xu" and Zihan Ding"
Tencent
ABSTRACT
We revisit policy-gradient optimization for Large Language Models (LLMs) from single-stream perspective. Prevailing group-based methods like GRPO reduce variance with on-the-fly baselines but suffer from critical flaws: frequent degenerate groups erase learning signals, and synchronization barriers hinder scalability. We introduce Single-stream stream Policy Optimization (SPO), which eliminates these issues by design. SPO replaces per-group baselines with persistent, KL-adaptive value tracker and normalizes advantages globally across the batch, providing stable, low- variance learning signal for every sample. Being group-free, SPO enables higher throughput and scales effectively in long-horizon or lool-integrated settings where generation times vary. Furthermore, the persistent value tracker naturally enables adaptive curriculum via prioritized sampling. Experiments using Qwen3-8B show that SPO converges more smoothly and attains higher accuracy than GRPO, while eliminating computation wasted degenerate groups. Ablation studies confirm that SPO's gains stem from its principled approach to baseline estimation and advantage normalization, offering more robust and efficient path for LLM reasoning. Across five hard math benchmarks with Qwen3-8B, SPO improves the average maj@ @ by +3.4 percentage points (pp) over GRPO, driven by substantial absolute point gains on challenging datasets, including +7.3 pp on BRUMO 25, +4.4 pp on AIME 25, +3.3 pp on HMMT 25, and achieves consistent relative gain in pass @ across the evaluated & values. SPO's success challenges the prevailing trend adding incidental complexity to RL algorithms, highlighting a path where fundamental principles, not architectural workarounds, drive the next wave of progress in LLM reasoning. Our code is publicly available at tts V/github cww. verl pro wowo://ee-oooooooeoooooooooooco htt//veeerrreeeeee ttreepppeeeeeeeeeeeeeeeeeeeeeeee htt.mmaiin////aaa spo
INTRODUCTION
Reinforcement learning (RL) (uttonn & Barto 2018) has become a cornerstone for advancing the reasoning capabilities of Large Language Models (LLMs), notably the Reinforcement Learning with Verifiable Reward (RLVR) paradigm (Lambert et al. 2024 DeepSeek] 2025). Methods like Group Relative Policy Optimization (GRPO) (Shao et a 2024 [eepSeek][00e
have achieved remarkable success by adopting multi-outcome approach, generating group of responses for each prompt to construct an on-the-fly baseline for variance reduction. While this "group-based" paradigm has pushed the state of the art, it suffers from fundamental inefficiencies. When all responses in group share the same outcome (c.g., all correct or all incorrect), the relative advantage collapses to zero, yielding no learning signal. This degeneracy represents fundamental waste of computation and data. To counteract this, a series engineering heuristics like dynamic sampling (Yu et al.) (005) have been developed. These workarounds, while functional, add significant complexity and create a less principled, more convoluted optimization process.
Group-based architectural choice also imposes a critical synchronization barrier. In distributed training, the entire group must wait for its slowest member, a bottleneck that becomes particularly acute complex agentic tasks requiring multi-turn tool use or long-horizon reasoning (Gao et al.) 2025; Xu et al.) 2025 Zhipu 2025). In these settings, interaction times are highly variable (c.g.. number of interaction turns, time per interaction, etc), and single slow-running agentic trajectory can stall its entire group, severely hindering training throughput and scalability.
We advocate for returning to the classic single-strean paradigm for policy gradient optimization (Sut) ton & Barto 2018). where each training sample is single stream of prompt-response pair. This is not mere simplification, but deliberate re-alignment with foundational RL principles to ad- dress the aforementioned architectural flaws. To overcome the critical challenge of high gradient
"quul contributions, corresponding to: zhongwensxu@ gmw.l.mmmmimmmmmmmmmmmmm dingzihan737 gmail.com

---

Published as a conference paper at ICLR 2026
variance in this setting, introduce Single-stream Policy Optimization (SPO). SPO replaces the noisy, on-the-fly group baseline with three synergistic components for stable and efficient learning. First, it employs a lightweight Bayesian value tracker to maintain a persistent, temporally-informe estimate of the success probability for each prompt, serving as a low-variance baseline. Second, it normalizes advantages globally across the entire batch, avoiding the instability of per-group statistics. Finally, this architecture naturally enables an adaptive curriculum via prioritized sampling, focusing computational resources on the most informative prompts.
The benefits of this principled approach are clear: SPO is inherently more scalable and eliminates the computational waste of degenerate groups. Our experiments confirm these advantages, demonstrating that SPO consistently outperforms GRPO on challenging reasoning benchmarks, improving the absolute point gains on challenging datasets, including 7-3 percentage points (pp) on BRUMO 25, 4.4 pp on AIME 25, 3.3 pp HMMT 25, and the pass @ curves of SPO are above GRPO for all ks. The scalability benefit is particularly pronounced in agentic settings. Our simulations, designed to model these variable-time scenarios, show that SPO's group-free design can achieve x.35x training throughput speedup by eliminating group synchronization bottlenecks. SPO thus provides a more robust foundation for modern LLM optimization, prompting re-evaluation of essential versus incidental complexity in the field.
2 RELATED WORK
Group Relative Policy Optimization (GRPO) (Sha et al. 2024) addresses the computational overhead and training instability of PPO-style algorithms (Schulman al. 2017) by eliminating the need for a separate critic network. Instead, GRPO constructs baselines on-the-fly using multiple responses generated for each prompt. Specifically, GRPO samples group of multiple responses for each prompt and normalizes the rewards within this group to have zero mean and unit variance, creating relative advantages for policy updates. However, this approach can be inefficient if all responses in group receive the same reward (e.g., all incorrect or all correct), resulting in a zero-advantage for all samples and providing no learning signal. To address this, DAPO (Yu et al., 2025) enhances GRPO with engineering treatments like dynamic sampling, which continues generating responses until non-zero advantages are achieved, ensuring meaningful gradients.
Several other works have proposed improvements to group-based methods. Zheng et al. (2025b) introduce GRESO, an online filtering algorithm that leverages reward training dynamics to predict and skip uninformative prompts before generation, Qu et al. (2025) introduce Bayesian estimation of the prompt accuracy and use it to form a bandit strategy, significantly reducing rollout overhead. Liu al. (2025c) propose "Lite PPO", which simplifies RLVR training only advantage normalization and token-level loss aggregation.
Other group-based approaches include RLOO (Ahmadian et al. 2024 which returns to the simpler REINFORCE (Williams] 1992, [Sutton & Barto 2018) algorithm using a Leave-One-Out baseline that treats entire generations as single actions. Similarly, Hao et al. (2025) propose On-Policy RL with Optimal Baseline (OPO), which uses eength-weighted average of rewards as an optimal simplified baseline. Despite these improvements, all group-based methods share fundamental limitations. They construct baselines from concurrently generated responses rather than persistent, historical estimates, inheriting the same core architectural constraints as GRPO: synchronization overhead and increased generation costs in distributed settings.
Moving beyond group-based based methods, Brantley et al. (2025) propose A'PPO, a two-stage framework for single-sample efficiency. In its first stage, -PO performs offline estimation of the optimal value function V*, not the policy-specific Vz The second stage then uses this pre-computed value to build optimal advantage estimates A". However, PPO has key limitations: its fixed, offline-compute estimate does not adapt as the policy evolves, and is constrained by KL-regularized optimization, restricting deviation from reference policy.
BACKGROUND
Reinforcement learning (RL) algorithms have been used to align Large Language Models (LLMs) with human preferences (RLHF) and to optimize verifiable reward signals (RLVR; e.g., (Lambert
et al. 2024 Shao et al. 2024
2

---

Published as a conference paper at ICLR 2026
3.1 POLICY GRADIENT AND THE REINFORCE ALGORITHM
The foundational method for this optimization is the policy gradient theorem (Williams) 1992 Sutton & Barto 2018). For LLMs, trajectory consists of generating a single response y from prompt The objective function is the expected reward:
(() = BewD.y (-wx) [R(,,
(aa)) v)],
where D is the prompt distribution and R(x, y) is the reward for generating response y for prompt æ. The gradient of this objective is given by:
(QJ = Ba-Day-aet m)) [[ y) V log To (3)22))
This formulation gives rise to the REINFORCE algorithm (Williams 1992. Sutton & Barto 2018).
which updates the policy by taking step in the direction of this estimated gradient. A significant drawback of REINFORCE is the high variance of its gradient estimator. The raw reward R(x, y) can fluctuate widely, leading to noisy updates and unstable training.
To mitigate high variance, a baseline b(x:) that is conditionally independent of the action y can be subtracted from the reward. This results in an unbiased gradient estimator with provably lower variance (Greensmith et al. 2004):
(0)
(t v) b(x)) Ve log no (v(x)]]
The term A(z, y) = R(2, y) b(x) is known as the advantage. The optimal baseline that minimizes variance is the true value function (+) = (1)=) [R(z;, [R(() v)], which is the expected reward for given prompt In practice, V=(2x) is unknown and must be estimated. The quality of this estimation is crucial for the stability and efficiency of the RL algorithm.
3.2 VARIANCE REDUCTION BASELINES FOR LARGE LANGUAGE MODELS
Several strategies have been developed to estimate the baseline b(x) in the context of LLM training. PPO (Schulman et al., 2017) trains parameterized critic network Up. However, learning Us notoriously unstable and resource-intensive, as & typically matches the size of the LLM policy
parameters
A common approach is to construct an empirical, on on-the-fh fly baseline from multiple samples. Group Relative Policy Optimization (GRPO) (Shao et al., 2024 [DepSSekk] 2025) generates group of G responses (yI...., ,G) } for single prompt then uses the mean rewards of the group as the baseline bGRPO- Another popular baseline is the Ceave-One--ut (RLOO). For a given sample yi. the baseline is the average reward of the other G' I samples in the group, denoted as bRLOO:
bGRPO(x) = T Σ R(I,yi), Wi),
bRLOO 9i) = G
Σ jfi
R(x, 4;).
(4)
3
The raw advantage for sample y; is then A(x, n.) R(r, v:) bGRPO(Z), then it is normalized with the standard deviation og. While simple to implement, this approach suffers from two key limitations. First, it is sample-inefficee the baseline estimated from a very small group (G), making a high-varianee estimate the true value function, which in turn leads to noisy advantage estimates.
requiring G > generations per prompt for each gradient step. Second,
METHOD
We introduce Single-stream Policy Optimization (SPO), method designed for policy optimization in settings with verifiable feedback (RLVR) (Lambert et al. 2024). We assume the feedback is binary i.e,, +1 for success and 0 for failure. SPO addresses the challenge of estimating a non-stationary success probability for policy that evolves over training iterations. It integrates a Bayesian value tracker with an adaptive memory mechanism into policy gradient framework. The core components are: (1) a KL-adaptive tracker that provides a low-variance, singic-sample estimate of the success probability: (2) a global advantage normalization scheme that ensures high sample efficiency and stable learning dynamics; and (3) prioritized sampling across training prompts to focus on prompts with high learning potential. The following subsections detail each component.
Generalizing non-binary rewards straightforward, discussed the end of Section
3

---

Published as a conference paper at ICLR 2026
4.1 A KL-ADAPTIVE VALUE TRACKER
The definition of a value function is the expected reward of the prompt x under policy и, i.e., (a(x)) = Ex [R(x, 1)]. We use i(x) to denote the tracker's running estimate of Va (x);; that is, o(x) = (z). To estimate the non-stationary success probability of a prompt 2, use Bayesian tabular tracker instead of a separate value network" For the binary success/failure rewards common in RLVR, this is elegantly modeled using Beta distribution, which is the conjugate prior for the Bernoulli process governing the outcomes. We therefore model the success probability i(x) using a Beta distribution; ((x) Beta(a(x), B(x))) where the value estimate is the posterior mean ((z) = a<z)/(((()) + {(z)).
The tracker adapts to policy changes by dynamically adjusting its memory of past rewards. When the policy changes significantly, older observations become less relevant and should be downweighted. After each new observation r(x, y) E (0, 1). we discount the prior Beta parameters (a-1, 8-1) by factor p(z) before incorporating the new evidence y))
a(x)
a<(x))
B(x)
6(x)
(5)
a(x) ((x)
The discount factor p(x) = Dmu) is determined by the KI. divergence ((x) between the currett policy and the last policy that acted prompt æ, causing the tracker to forget faster as the policy changes more significantly. The hyperparameter Dhaif controls this forgetting rate p E
[Pmin: Puax)
To initialize, we collect ny samples to compute an initial value estimate (r(x)) To avoid transient instability, we set the initial effective sample size its expected equilibrium, No = 1/(1 - pmin), where Pmnin is the minimum allowed forgetting factor. The initial parameters are then:
Initialization.
cr(x) = No 00(x),
Bo(c) = No (1 - (o(x))
This Bayesian update is equivalent to adaptive Exponential Moving Average (EMA) on the value estimate:
0(x) = (=) + m)xx)((xx,)) f-1())),
(7)
where the learning rate 7(x) - p(z)) and statistical confidence (via Ner = a(x) + (x))). This formulation highlights how our tracker balances new evidence against accumulated knowledge. For general rewards beyond binary ones, we can just use the same EMA formulation to directly track i, rather than relying a and / in
+ +) naturally adapts both policy shifts (via
the binary eases.
4.2 ADVANTAGE ESTIMATION AND POLICY OPTIMIZATION
SPO uses the tracker's s estimate i as baseline for advantage calculation in a policy gradient algorithm. At iteration i, for a single reward rax, y) obtained with policy no, the advantage is computed using the pre-update baseline (denoted with subscript -1):
A(z, v) = r(z, v) (z).
(8)
Using the baseline from the previous step ensures that it independent the action taken at step i, preserving the unbiasedness of the policy gradient estimate. While the reward r(z, y) is typically direct outcome signal, SPO's framework is also compatible with more sophisticated reward functions. For instance, recent work like InfAlign (Balashankar et al. (024) demonstrates how to calibrate and transform the reward signal to be "Inference-aware." " directly optimizing for procedures like Best-of-N sampling. Such transformed rewards can be seamlessly integrated into SPO by replacing the standard r(x, 3) in the advantage calculation. Since (=)(z) is independent of ~ Te, (-|æ),
EI(r- (x: )) V log T
VJ(O) (Williams) 1992). Instead of normalizing advantages on per-prompt basis in group (Shao et al. 22244] DeepSeek] 2025). SPO normalizes them across an
entire batch of prompts B (Hu et al. [025 Schulman et al. 2017 Andrychowicz al. 2020 Liu
al. 2025a). The normalized advantage A A(x.v) y) is computed as:
Á(x, v) A y) 148
σB
Thhe development of core RI. algorithms was tabular representation (Sutton & Barto 2018
4

---

Published as a conference paper at ICLR 2026
GRPO
y1
71
A
Group Norm
42
Ac
SPO
x
Figure 1: Illustrations of GRPO and SPO.
where jug and og are the mean and standard deviation of advantages in the batch {A(x, y)) We then apply the advantage A(x, 3) to each token in the response sequence and update the policy parameters using a standard PPO-Clip policy loss (Schulman et al. 2017
(a (t) Ti (at 88)
Te(a; sc) 1 1+5) A ]
L CLIP (0)
E.,t
min
Å, clip To (at 8t)
(10)
Methods like Clip-Higher (Yu et al. 2025) Clip-Cov (Cui et al. 2025) and KL-Cov (Cui al. 2025 to retain policy entropy are applicable here. Other policy optimization algorithms like CISPO (Mini
Max 2025) (similar to vtraae (Espeholt et al. 2018 Wu et 2025) and GSPO (Zheng al. 2025a (use sequence-level likelihood instead of token-level) are compatible with our advantage
estimator. Advanced methods to control policy behaviors like ASPO (Lin Xu 2025) can be utilized to modulate the advantage values. We note that if we use "no baseline" (i.e., O), it is an extremely simple and valid algorithm but may suffer from high policy gradient variance.
4.3 PRIORITIZED PROMPT SAMPLING
Algorithm 1 Single-stream Policy Optimization
1:
for iteration i = 1,,,..., do
2:
For each X, compute sampling weight wy(xr) according to Eqn. (11)
3: Sample a batch of B prompts B; C X' according to weights (w;(
4:
D ()
5:
for each prompt E B; do
6:
Sample action ~ To._ (- x) and observe reward r(z, y) (0,1}.
7:
Compute advantage A(x, y) + r(x,y) (_)(z).
8:
Store (x, y, A(x, y)) in D.
9:
Update tracker 6(z).
Normalize advantages: Á(x, y) + (A(x, v) /OB,
11: Update 0,-1 to 0, using mini-batcher with policy gradient algorithm (e.g., PPO-Clip).
To further enhance data efficiency, SPO employs a curriculum learning strategy by prioritizing prompts with the highest learning potential (Schaul al. 2015 Sutton & Barto 2018). At each iteration,
'The term "PPO" is frequently used with ambiguity. It may denote the entire algorithm suite (e.g., clipped policy and value losses), refer narrowly just the clipped policy objective, describe the broader training framework, including mechanisms like mini-hatch updates.
5

---

Published as a conference paper at ICLR 2026
sample batch of prompts based on score that emphasizes prompts with high uncertainty, while ensuring a minimum level of exploration. The sampling weight wy (2)) for prompt æ is defined as:
(a)(z)) - (1-0-1())
wx(x) x
+e.
(1I)
N 1
The first term corresponds the estimated standard deviation of Bernoulli outcome, which naturally allocates more weight to prompts that are neither almost always solved (0 = 1) nor almost always failed (ú s O). The exploration bonus e, set to 0.05 by default, prevents curriculum collapse by ensuring that every prompt retains a non-zero probability of being sampled, thereby maintaining broad coverage of the data distribution.
Here, (Neir a+ denotes the effective sample size of the Beta tracker and (7 E [0, 1J) controls how strongly we down-weight prompts with well-estimated success probabilities, allowing the sampler to balance value-uncertainty, driven prioritization against broader exploration.
The complete SPO training procedure is outlined Algorithm
4.4 ADVANTAGES OVER GRPO
Group-Free for Scalable Infrastructure. SPO's design is inherently "group-free", a significant advantage distributed training frameworks for L.MMs. Each sample, consisting of a single stream of (prompt, response) pair, is a self-contained data point for the policy update. GRPO, however, requires the generation and evaluation an entire group of G samples for single prompt before any training signal can be computed. We provide our illustrations in Figure [ In distributed setting, this introduces a synchronization barrier: the processing of a given prompt is not complete until all G responses have been generated. This is particularly problematic in the presence of long-tal tail generation times, where a single slow response generation can stall the processing for its entire group. For constructing training batch, SPO only needs to collect B independent (prompt, response) pairs, which is far more flexible and efficient than waiting for B entire groups to complete. This makes SPO's architecture significantly more infrastructure-friende in agentic training, especially in settings that require multi-turn interactions with tools (Gao et al. 2025; Chen et al. 2025b) or long-horizon agent rollouts (Zhipu [025 Xu et al. (025). The scale of these interactions can be substantial: state-of-the-art open-source models (gpt oss 2 20b) may average search turns per task (Chen et al. 2025b), with other agentic sessions reaching over 40 tool calls and generating up to 150,000 tokens of context (Gao et al. 2025).
and scalable. The advantage is amplified
Adaptive Curriculum. To further enhance training efficiency, SPO integrates a prioritized sampling scheme. This mechanism naturally creates adaptive curriculum by focusing computational resources on prompts with the highest learning potential. This ensures that the model's training is concentrated on the most informative examples at any given point in time. GRPO, in its standard formulation, typically relies on uniform sampling of prompts. This may waste computation on prompts that are already mastered
currently too difficult yield useful learning signals. While dynamic sampling (Yu et al.] 2025) and repeat strategies (An et al. (2025) have been proposed to mitigate this issue, they often discard samples after generation, wasting computation. SPO's prioritized sampling addresses the scheduling problem before response generation, leading to more
natural and efficient training process.
More discussions on the inefficiency of dynamic sampling and the variance reduction of policy gradient are outlined in Appendix D where we provide detailed analysis.
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
The SPO algorithm is broadly applicable in LLM reasoning tasks (DeepSeek] 2025) and Agentic training. We evaluate Inte-Inte Reasoning (TIR) (Feng et al. 2025 Lin & Xu (2025) scenarios, where the LLMs can utilize external Python interpreter to help solve hard problems. We conduct experiments using moderately sized LLM,, Qwen3-8B (Qwen 2025). For training data, use

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of GRPO and SPO on five benchmarks using maj @2 and avg 32. Averages are shown in the last column. Bold indicates the better-pent serfer-performing method for each metric.
Mathod
AIME 24
AIME 25
BeyondAIME
BRUMO 25
HMMT 25
Avcrage
maj @:32
avger
ma
ovg 32
maj66
avg
maj 62
avgel
maje: 32
uvg 32
maj 6:
avg
Qwea3-SB
77,8
64.4
70.5
58.4
45.2
38.0
55.1
49.4
36.8
30.3
57.1
48.1
GRPO
83.3
77.6
72.1
64.2
45.6
39.0
56.7
56.9
44.2
40.9
60.4
55.7
SPO
84,0
74.9
76.5
65.0
46.9
40.3
64.0
59.0
47.5
40.6
63.8
56.0
5P0
SPO
6P0
+Bw U1D
2.50
-0 URIO
CR.10
2.82
20.52
0.25
3.82
0.50
32
(a) AIME
(b) AIME
(c) BeyondAIME
0.83
S10 CRPP
G1P0
GAPO
2.72
9.54
5000000
0.52
2:72
0.70
5.75
0.50
(d) BRUMO 25
(e) HMMT 25
(i) Average
Figure 2: Pass @ l: plots comparing GRPO and SPO across five math competition benchmarks.
the English subset from the DAPO dataset (Yu et al. 2025). Only outcome reward is applied for RLVR, without the format rewards. We evaluate performance on the challenging math competition benchmarks, i.e., AIME 24, AIME 25, BeyondAIME (Seed) 2025). BRUMO 25 (Balunović et al.. 2025), and HMMT 25 (Balunović et al. 2025). See Appendix for training and evaluation details.
We distinguish our goal from that of "hill- "hill-climbing" on benchmark leaderboards. The latter often necessitates resource-intensive and highly specialized techniques, including SFT from frontier models (Liu al. 2025b), mid-training (Wang al. 2025), , multi-stage RL pipelines (Luo et al. 2025 He et al.] 2025: Chen et al., (2025a), curated hard datasets with intricate processing (An et al. 2025: (Shnn et al. 2025). test-time scaling techniques (Fu et al.) 2025) and extremely large
generation group sizes (Zhipul 2025). Our work, instead, concentrates on the fundamental efficiency and scalability of the algorith itself
5.2 EMPIRICAL COMPARISON WITH GRPO
Our experiments demonstrate that SPO outperforms the GRPO baseline aggregate metrics when training the Qwen-8B model. As shown in Table SPO achieves superior weighted average scores on both primary metrics. It obtains a maj@ mnj@32 32 of 63.8 compared to GRPO's 60.4, a significant improvement of +3..4 percentage points (pp). This aggregate strength is driven by remarkable consistency, as SPO outperforms GRPO on the maj @32 metric across all five benchmarks. The performance gap is most pronounced on BRUMO 25, where SPO achieves a substantial +7.3 pp (64.0 vs. 56.7). Further significant gains are seen on AIME 25 (+4..4 pp) and HMMT 25 (+3.3 pp points), underscoring the robustness of SPO's 's improvements. Notably, these benchmarks have minimal data contamination (Balunović et al. 2025), allowing them to serve as a true test of generalization. This demonstrates that our SPO method improves the model's ability to generalize rather than simply overfit to the training data, a risk exemplified by the DAPO dataset's 's strong correlation with AIME 24. While GRPO remains competitive on the avg @ 32 metric in some cases, SPO's 's consistent and significant advantage in maj @32 suggests it learns more robust and repeatable solutions, a key goal for reliable reasoning models.
7

---

Published as a conference paper at ICLR 2026
Degenerate Groups Ratio
0.18
Advantage Varlance
0.6
0.16
GRPQ
0.14
0.4
SPO [--0, (m-000011 000111
(ve0.92)
0.12
Bazelinc
0.10
0.08
20
40
60
80
100
120
240
20
40
60
80
100
120
140
Training Steps
Training Steps
(a) Ineffective Gradient Ratios
(b) Advantage Variance Comparison
Figure Signal Efficiency and Stability Analysis of SPO vs. GRPO. (a) GRPO suffers from high ratio of degenerate groups (blue), which yield no learning signal. In contrast, SPO's rate of near-zero advantages (red/green) increases as the model learns, reflecting prediction accuracy rather than wasted computation, (b) SPO's baseline (red) provides stable, low-variance signal, significantly reducing the raw reward variance (green). GRPO's effective advantage (blue), calculated only on non-degenerate samples, is highly volatile and unstable.
These findings mirrored in the pass k performance shown in Figure 2 The weighted average curve (Figure[1) 21) shows a clear and consistent advantage for SPO across all values of k, translating an average improvement of approximately 2.4 pp. While the performance on avg 32 more competitive on per-benchmark basis, SPO's s strong overall performance underscores the stability and effectiveness of its learning signal. We provide additional ablation studies on A A SPO with no baseline, and SPO with no offline initialization in Appendix
5.3 ANALYSIS OF SIGNAL EFFICIENCY AND STABILITY
To empirically assess the architectural advantages of SPO, we conduct two-part analysis of the unnormalized advantage signals produced by SPO and GRPÓ (Figure First, we quantify complete signal loss arising from degenerate groups. Second, we measure the variance of the remaining learning signals. Together, these metrics characterize each method's efficiency and stability.
Signal Efficiency and Information Loss. Figure 3al reports the fraction of ineffective samples. For GRPO (blue), the share of samples degenerate groups rises from roughly 60% to over 80%, yielding zero advantage and gradient. For SPO, instead track the proportion of near-zero advantages under two diagnostic tolerances, AA| S т, with values of T = 10 - (red) and = 0.02 10-4 remain while the |A| < 0.02 share (green) gradually increases as the value tracker i becomes more accurate and residuals shrink on mastered prompts. This trend expected and desirable: reflects accurate prediction rather than signal loss. Unlike GRPO's degenerate groups, these SPO samples are not discarded, they still produce well-defined gradients and contribute to learning. Notably, even under the stricter 0.02 tolerance, SPO's near-zero ratio remains far below GRPO's degenerate rate, underscoring SPO's efficient use of compute.
(green). Advantages under the tight tolerance
throughout training (red line),
Signal Stability and Advantage Variance. Figure 3b compares advantage variance across methods. As reference, the green line ("SPO No Baseline") corresponds raw rewards, i.e., the high-variance signal faced by vanilla policy gradient. SPO's history-informed baseline (red) delivers substantial, stable variance reduction of nearly 50%. For GRPO, computing variance only over non-degenerate samples ("GRPO Effective", blue) reveals highly volatile signal with the largest variance among all conditions, exceeding even "SPO No Baseline". We conclude that SPO's baseline is effective, yielding stable, ow-variance gradients, whereas GRPO's on-the-fly baseline is noisy and destabilizing when produces a signal. The apparent stability of GRPO's overall variance driven by the prevalence of zero- vorivvai degenerate samples and thus reflects inefficiency rather than robustness.
8

---

Published as a conference paper at ICLR 2026
Siisp conclatie 348
Total Interaction Time
4441
000-
Total Interaction
003-
302-
Predictable
High Bottlenack Effect
124a
137
100
100-
Sample
(a) Low-variance Group
(b) High-variance Group
Figure 4: The Bottleneck Effect in Group- Sampling. (a) In low-variance environment, sample completion times are predictable, and the group synchronization cost is minimal. (b) realistic high-variance agentic environment, three slow trajectories (444s, 508s, and 409s) create severe bottleneck, forcing the entire group to wait and wasting the compute used for the six faster samples.
500
15h
Group Completion Timee
Total Interadiion Time
Speedup::
Total Eompletion
Sescied
220-
L'aescid
Sesced turat!
120-
Group Index
Sample Index
Group Based
Group-Tree
(a) Group-base
(b) Group-free
(c) Strategy Comparison
Figure 5: Throughput Comparison: Group-Based -ased Group-Free Free. (a) A group-based strategy, when parallelized, is bottlenecked by its slowest group. taking 486s to collect a batch of 3 groups (24 samples). (b) A group-free strategy collects the 24 fastest samples from larger pool of 48, completing the batch in just 1 112s by avoiding stragglers. (c) The group-free approach achieves a 4.35x speedup, demonstrating its superior efficiency for agentic training.
5.4 AGENTIC TRAINING DEMONSTRATIONSS
We perform simulations demonstrate the practical implications of SPO's group-free design in agentic training scenarios, where interaction times can be highly variable. Group-based methods like GRPO suffer from critical scalability bottleneck due to their inherent synchronization barrier, problem that is particularly acute in agentic tasks involving multi-turn tool use or long-horizon reasoning.
Figure Aillustrates this fundamental issue. In an idealized low-variance setting (Figure 4a where all agentic trajectories complete in similar times, the group-based approach efficient. However, in more realistic high-variance setting (Figure 4b) characterized by long-tai tail latencies, a single slow-running trajectory (a "straggler") can stall the entire group. In our simulation, while most samples finish in under 133 seconds, the group must wait 508 seconds for its slowest member. This bottleneck effect forces faster samples to remain idle, severely hindering training throughput and wasting computational resources.
SPO's group-free architecture directly resolves this inefficiency. Figure compprrss the time required to assemble training batch of 24 samples using both strategies. The group-based approach (left), even when optimized by running 6 groups in parallel and selecting the 3 fastest, is still constrained by the slowest trajectory within those selected groups, taking 486s complete. In contrast, the group-free approach (middle) leverages asynchrony by starting 48 independent samples and simply collecting the first 24 finish. In our simulated scenario, this process takes only 1 112s, it naturally

---

Published as a conference paper at ICLR 2026
filters out the slow outliers. As shown the right, this architectural difference results in significant 4.35 x speedup in this realistic agentic simulation. Simulations show that SPO's s architecture can lead to significant throughput gains, making it a more scalable and robust foundation for training complex, long-horizon agentic tasks.
CONCLUSIONS
We identified critical inefficiencies in group-based policy optimization methods for LLMs, namely computational waste from degenerate groups and scalability bottlenecks from synchronization. To address these, proposed Single-stream Policy Optimization (SPO), principled return the classic single-strean paradigm. SPO replaces the noisy, per-group baseline with a persistent KL-adaptive value tracker and global advantage normalization, creating a more stable and efficient learning signal.
Our empirical results demonstrate that SPO's design is not merely simpler, but superior. It consistently outperformed GRPO on complex reasoning tasks while eliminating the systemic flaws of its group- based counterpart. By demonstrating that a well-designed single-stream approach can surpass more complex methods, our work challenges the prevailing trend of adding incidental complexity to RL algorithms for L.MMs. SPO provides a robust, scalable, and efficient foundation for future research in agentic and reasoning model training, highlighting the enduring power of foundational reinforcement learning principles. Future work can focus on refining the best practices for applying SPO and exploring its limits, pushing its effectiveness to power the next generation of reasoning and agentic LLMs.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in LLMs. arXiv preprint arXiv:2402. 14740, 2024.
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. POLARIS: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL https::/hkunlpp: githu b. http://loo//2225Pooaar.a
Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, al. What matters in on-policy reinforcement learning? A large-scale empirical study. arXiv preprint arXiv:2000.05990, 2020.
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. InfAlign: Inference-aware language model alignment. arXiv preprint arXiv:2412.19792. 2024.
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. MathArena: Evaluating LLMs uncontaminated math competitions. arXiv preprint arXiv:2505.2328/) 2025.
Kianté Brantley, Mingyu Chen, Zhaolin Gao, Jason Lee, Wen Sun, Wenhao Zhan, and Xuezhou Zhang. Accelerating RI. for L.M reasoning with optimal advantage regression. arXiv preprint arXiv:2505.20668 20686. 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2 003744, 2021.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-Nemotroon Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505. 16400, 2025a.
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. BrowseComp-Pine A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508 0660000 2025b.
10

---

Published as a conference paper at ICLR 2026
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025.
Team DeepSeek. DeepSeek-RI incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633-638, 2025.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Jiazhan Feng. Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. ReTool: Reinforcement learning for strategic tool use in LLMs. arXiv preprint ar arXiv:254..15556 2025.
Yichao Fu, Xuewei Wang. Yuandong Tian, and Jiawei Zhao. Deep think with confidence. arXiv preprint arXiv:2508. 00.5508..2600 15260, 2025.
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous RL. arXiv preprint arXiv:2508.0799 2025.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 1170v):44)
2004.
Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, and Furu Wei. On-policy RL with optimal reward baseline. arXiv preprint arXiv:2505 233555 2025.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. SkyWork Open Reasoner technical report. arXiv preprint wwXiv:2505.22312 2025.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE++: An efficient RLHF algorithm with robustness both prompt and reward models. arXiv preprint arXiv:2501.0322 2025.
Team Kimi. Kimi K1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12590 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training arXiv preprint arXiv:2411.15124, 2024.
Heng Lin and Zhongwen Xu. Understanding Tool-Integrated Reasoning, arXiv:2508. 00008.99200 19201, 2025.
arXiv preprint
Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Diyi Yang, Wee Sun Lee, and Min Lin. GEM: A gym for generalist LLMs, 2025a. /RRL/sttts///xoor111 iotionn sittiiiiossseee.eee.......ieeeee /gemm
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. AceReason-Nemottrn 1.1: Advancing math and code reasoning through SFT and RL synergy. arXiv preprint arXiv:2506. 13284, 2025b.
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, et al. Part I: Tricks traps? A deep dive into RL for LLM reasoning. arXiv preprint arXiv:2508.088 08221, 2025c.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa. and Ion Stoica. DeepScaleR: Surpassing ol-Preview with a 1.5B model by scaling RL. /
Diee/Dee O http::/ooo.aa.aaa.aaaa------aaaa
9681 3001111102224440005eeeaaaaa00100 2025. Notion Blog.

---

Published as a conference paper at ICLR 2026
Team MiniMax. MiniMax-MI: Scaling test-time compute efficiently with lightning attention. arXiv preprint Stti::2506.138883 2025.
Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Bjöm Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating RL finetuning of reasoning models? arXiv preprint 2:Xiv:2507.04632, 2025.
Team Qwen. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint a 1rXiv:1511.05952 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Team Seed. Seed1 5-tthinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.139/1 2025.
Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, et iStar2-Agent: Agentic Reasoning Technical Report. arXiv preprint arXiv:2508. 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv preprini arXiv: 2409. / 19256, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning language models. In The Eleventh International Conference on Learning Representations (ICLR), 2023. URL htttps:: http:///aaaeiiiieeeeeerrrrrrraaa
net for.mmiddyyyy1aaaa....
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Thinker: Mid- Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512. 2025.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-255 1992.
Bo Wu, Sid Wang, Yunhao Tang. Jia Ding, Elyk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, et al. LlamaRL: A distributed asynchronous reinforcement learning framework for efficient large-scale LLM training. arXiv preprint arXiv:2505.24034, 2025.
Zhongwen Xu, Xianliang Wang, Siyi Li, Tao Yu, Liang Wang, Qiang Fu, and Wei Yang. Agents play thousands of 3D video games. arXiv preprint arXiv:2503.13356 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint trXiv:2507.1807. 2025a.
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts. arXiv preprint arXiv:2506.022 02777, 2025b.
Team Zhipu. GLM-4.5: Agentic, reasoning, and coding (ARC) foundation models. 2025.
12

---

Published as a conference paper at ICLR 2026
A LLMS USAGE
LLMs employed solely to assist with minor editorial improvements to the text. Their was limited to polishing language, and they did not contribute to the conceptual or scientific content of the paper.
B SPO INITIALIZATION
We show the SPO initialization procedure in Algorithm 2 In the experiments, we use r4n = 8 to have a good estimation of initial baseline tracker. We ablate the setting where we use no offline estimation and rely on the online moving estimator in Section
Algorithm 2 SPO Initialization
Set initial effective sample size No 1/(1 Pmii). for each prompt E X' do
2:
3: Collect outcomes (r(() Jk= with an initial policy
4: Compute initial value estimate (z) - (A).
5: Set (x) No (z) and Bo(x) No (z)).
Practically, one may concern about the extra cost during the offline estimation of ig. We note that we share the offline estimation for experiments that people could skip this process and directly load our datasets, and there are datasets like Polaris (An et al. (0225) that pre-compute accuracy for Scepsoek-Rl-Distill D--Disiill-Qen--BB [DeepSeek] (025). The cost can be amortized across the experiments people run themselves, and we will share more (dataset, base_modes combinations to facilitate experiment efficiency.
C BATCH EXTENSIONS
We could adapt Single-strean Policy Optimization (SPO) into prompt-repetition scheme" process- ing each prompt G times per batch with shared baseline estimator to better handle sparse rewards. Our method's 's primary advantage the group synchronization barrier. Treating repeated prompts as independent trajectories unlocks two key efficiency improvements. First, enables robust handling of long-tail generation issues, slow or problematic trajectories can be terminated early, discarded, or managed via partial rollouts (Kimi) 2025) without delaying the entire batch. Second, facilitates more flexible batching strategy. By over-sampling the number of initial prompts (e.g., by 50%), a full training batch can be assembled from the first-finishing trajectories, allowing the optimization step to proceed immediately without waiting for stragglers. This design significantly reduces training latency compared to the rigid group synchronization required by GRPO. When tackling hard prompts, the batch extensions may help obtain learning signals more quickly.
GRPO lies in its asynchronous nature, achieved by removing
D COMPARISONS AGAINST GRPO
D.1 INEFFICIENCY OF DYNAMIC SAMPLING
To address the information loss from degenerate sample groups (where all rewards are identical), methods like DAPO (Yu et al. 2025) employ dynamic sampling. This strategy continues generating responses for prompt until the collected set contains at least one success and one failure, guarantecing non-zero advantage. While effective ensuring learning signal, this approach can be extremely data- and fime-inefficient. Note that when people report performance with dynamic sampling, the "steps" indicate the learning steps rather than the sampling steps, where the latter is normally multiple of the former (e.g., 5x).
Batch SPO BSPO
13

---

Published as a conference paper at ICLR 2026
We can formalize the expected computational cost. For prompt with true success probability p = (z),,, let N be the number of samples required to obtain a non-degenerate set. We have:
E[ |p]= - ( -0
p(1
This cost grows hyperbolically as the policy becomes either proficient (p - 1) or incompetent (p - 0). For example, if policy has a 10% success rate (p 0.1), the expected number of generations needed collect both success and a failure is E s 10.11. In contrast, SPO requires exactly one sample per prompt and uses its adaptive curriculum actively de-prioritize these inefficient prompts, allocating resources to where learning is most effective. This makes SPO fundamentally more scalable and computationally efficient.
D.2 VARIANCE REDUCTION FOR POLICY GRADIENT
The per-sample policy gradient is g = A(r, Vo log To (y/ (r), where the advantage A is an estimate of the expected return over baseline. The variance of this gradient, Var[g], is key driver of training efficiency. We analyze how the construction of the advantage A leads to significant variance differences between GRPO and SPO.
GRPO's High-Variance Variance Group-Based Advantage: GRPO computes advantages by comparing outcomes within small group of G (G = 8, 16.... samples generated for the same prompt. The normalized advantage for sample with binary reward {0, 1) is AGRPO =_'e where both the baseline μız (e.g., the group mean र Σ; rÿ) and the standard deviation og are estimated from the same small group of G samples. This coupled, small-sample estimation introduces three fundamental sources of variance:
ag+e"
Noisy Baseline (Numerator): The baseline Ag, estimated from only G samples, where G is small, a high-variance quantity. This inflates the variance of the unnormalized advantage (r -Pg) by a factor of (1 + )) compared to using an optimal baseline.
Noisy Scaling (Denominator): The standard deviation dg. estimated from only G samples, is also highly variable. Scaling the gradient by this noisy random variable further increases total variance.
Information Loss (Degeneracy): When all rewards in the group are identical (e.g., all Os or all 1s), the advantage for every sample becomes zero, providing
gradient signal. This where p = (x), effectively reduces the batch size and inflates variance by factor of 1/(1 Zp())), issue that is especially severe for easy (p R 1) or hard (p R 0) prompts.
event, which occurs with probability Zc(p) = pd + + - (1-p)*
SPO's Low- Vow-nariance Decoupled Advantage: In contrast, SPO designed to minimize these variance sources by decoupling the advantage calculation from the current group of samples. It uses an aetion-independent baseline = ((x:) from historical tracker, which provides a stable, low-variance estimate of the true success probability The advantage is simply Aspo www.chhnntrmpiii..eeeea w)
((x)). Crucially, SPO then applics global normalization (Schulman et al. 2017 Andrychowicz et al.
2020: Liu et al. 2025c), scaling all advantages in a large batch of size B > G by single, stable standard deviation Tbach- This design avoids GRPO's pitfalls: the baseline b is near-optimal, the normalization scaler o is stable, and there is systematic information loss from group-outcome degeneracy.
Quantitative Comparison: A simplified ratio of the reward-term variance quantifies the difference:
Var[g]GRPO
1 1+
1
tha
(12)
Var(g] sPP
T- Zo(p)
1/8
Nart1
Buseline Noise
Information Loss
Normalization
Here, Not is the effective sample count for SPO's tracker, and पh >> captures the excess variance from per-group, thB represents the excess variance introduced by estimating the normalization statistics (mean and standard deviation) from large global batch of size Ng (Eg z 0). For moderately difficult prompt (p = 0.5) with G = 8, the normalization noise dominates. However, for an easy/hard prompt (p- 0.9/p p= 0.1), the information loss term dominates, and the ratio swells to
14

---

Published as a conference paper at ICLR 2026
1.97. While increasing G' in GRPO mitigates information loss, it does at multiple generation cost and cannot fix the inherent noise from its smallesample baseline and scaling. SPO achieves lower variance more efficiently by design.
E TRAINING AND EVALUATION DETAILS
All experiments in this paper are implemented on top of verl (Sheng et al.) 2024) and ReTool (Feng et al. 2025) for the tool-integrated reasoning setup. During training, set the maximum response length to 16,384 tokens. The policy learning rate is fixed at x 10 6 Following DAPO Yu et al. 2025), we adopt the Clip-Higher mechanism, with clipping parameters flow 0.2 and Ehigh = 0.28, to balance exploration and exploitation. The sampling parameters are set to temperature 1.0, top-p 1.0, and top-k 11 The forgetting rate thresholds chosen Pmin 0.875 and and W max 25.
Pmax = 0.96, yielding window sizes W Wmin- =
Ponin
GRPO rollouts are collected with multiple responses per prompt, and training mini-batch sizes are chosen such that 8 gradient updates are performed per rollout step. For fair comparison, the prompt batch size in SPO is set equal to the total number of responses in GRPO, as SPO generates only a single response for each prompt. Specifically, GRPO uses prompt batch size of 256 with responses per prompt and training mini-batch size of 256, while SPO operates on 2, 048 256 x 8 prompts. Both algorithms are set with maximum of 8 Python interpreter interaction turns.
For evaluation on hard math competition benchmarks, i.e., AIME 24, AIME 25, BeyondAIME (Seed) 2025), BRUMO 25 (Balunović al., 2025) and HMMT 25 (Balunović al. 2025), we set sampling parameters to temperature 0.6, top-p 0.95, and top-k 20, officially recommended. We define a binary reward function Ta,j such that a response receives Tú = 1 if the final answer is correct, and Ti.j 0 otherwise. The same reward function is consistently used during training for policy optimization and during evaluation. We set the maximum response token to 32,768.
Given test set with M problems, and for each problem i we independently sample i responses with rewards {ri,1, ,.,,,..., we define:
avg@k: the expected correctness of an individual response:
M
avg @ k:
M
1=1
p the probability of solving a problem within k attempts. Directly computing 1( (maxxısj<* Fi.j = 1) can lead high variance. Following (Chen et al. 2021), we instead generate 2 & responses per problem, count the number of correct ones S n, and use the unbiased estimator:
M M Σ
1
pass
where & denotes the number of correct responses for problem
maj@ @k: the correctness of the majority- voted answer (Wang et al. 2023). This metric first identifies the most frequent answer among & responses for each problem. The score is if that modal answer is correct, and otherwise. Let a:.j be the final answer string for the j-th response to problem i, and let r(-) be the reward function for a given answer string. The metric is defined as:
M
maj@k @k
1
Σ
(modo (ai,5 11)
M
F ABLATION STUDIES
We conduct a series ablation studies to dissect the core components of SPO and validate our design choices. To facilitate efficient experimentation, these studies performed under streamlined setting compared to our main experiments. Specifically, we utilize a batch size of 256 prompt-resoonse pairs, and the model is updated with 4 gradient steps for each collected batch. All ablation results
15

---

Published as a conference paper at ICLR 2026
0.00
0.600
0.30
00000
0.45
0.500
0.00
0.475
SPO
w/ Baseline
4PPD
0,425
w/o Baseline
0.30
0
25
60
100
120
140
0,402
0
20
60
60
80
150
120
140
Training Steps
Training Steps
SPO -P-PO
Bascline Ablation
0.603
0.002
0.525
0.925
0.500
w/ Offinne Init
w/ Batch Norm
0.423
Init
0.423
w/o Norm
0.403
130
20
62
150
150
340
Training Steps
Training Steps
(c) Offline Initialization Ablation
(d) Batch Normalization Ablation
Figure 6: Ablation studies evaluating the core components of SPO. (a) SPO's adaptive baseline outperforms the static baseline of AO-PO, demonstrating the benefit of value function that evolves with the policy. (b) Removing the value tracker ("w/o Baseline") causes severe performance drop, confirming its critical role reducing gradient variance. (c) Eliminating the offline initialization step ("w/o Offline Init") leads to initial training instability and suboptimal convergence, highlighting the importance of a warm start for the value tracker. (d) Removing batch normalization ("w/o Batch Norm") results in slower convergence and reduced stability, showing the normalization's 's role in stabilizing updates.
reported on the AIME 25 benchmark, using the avg@ 16 metric with maximum generation length of 16, 384 tokens.
SPO vs. POO This experiment, presented in Figure 6a compares our proposed SPO with A"- PO (Brantley et al. 2025). -O-PO utilizes a static baseline derived from pre-computed optimal value function, V which is tied the KL-regularized objective with respect to an initial reference policy, Ref. While this approach is highly efficient, its central assumption may be challenged in tool-calling scenarios. In these tasks, learning involves acquiring new functional capabilities, leading to significant policy drift where the learned policy, Te, diverges substantially from Ted- Consequently, the pre-computed V may become a less representative baseline for the current policy's 's true value function, V=., potentially affecting the accuracy of the advantage estimates. In contrast, SPO's baseline is adaptive, dynamically tracking an estimate of Va, as the policy evolves. The empirical results, which show SPO's superior performance, suggest that this adaptability is crucial. By maintaining baseline that remains relevant to the current policy, SPO provides a more stable and effective learning signal in environments that demand significant policy evolution. Finally, from practical perspective, net computation during A -PO policy update occupies an extra trunk of GPU memory, making it less appealing than the proposed SPO algorithm.
Baseline Ablation. Figure 6b presents crucial ablation that validates the fundamental principle of using a baseline for variance reduction. In this experiment, we remove the value tracker com- ponent (z:) from the advantage calculation, causing the algorithm to rely solely on the globally barch-normalized raw reward r(z, y) as its learning signal. However, the substantial performance degradation observed is classic illustration of the remaining challenges. While global normalization effectively controls the overall scale of rewards, the raw reward signal is still noisy on per-sample basis it fails to account for ppemit-specific difficulty. SPO's cistory-informe to subtract this expected difficulty, thereby effectively reducing variance and providing a much cleaner, more reliable gradient for learning. This experiment confirms that the adaptive value tracker
baseline is designed
16

---

Published as a conference paper at ICLR 2026
is the most critical component for SPO's success, directly addressing the core challenge of variance in aingle-ttream policy optimization.
Offline Initialization Ablation. In Figure 6c| analyze the impact of the value tracker's initial- ization phase. The standard SPO algorithm iittializes the value tracker with estimates computed from small set of no offline samples, giving it a "warm start". The ablation removes this step, foreing the tracker to learn from scratch online. The results clearly demonstrate the benefit of the offline initialization. Without it, the tracker begins with a highly inaccurate baseline, leading to high-variance gradients and significant instability in the initial training phase, as evidenced by the performance dip. Although the model's performance is initially lower, eventually recovers and catches up to the level of the properly initialized model. This initial performance gap neverthe- less underscores the importance of good initial value estimate for ensuring training stability and accelerating convergence.
Batch Normalization Ablation. Figure 6d isolates the contribution of SPO's global advantage normalization (Eqn. (9) In this variant, we compute policy updates using the unnormalized advantages A(x, :) = r(a, y) v (x), omitting the batch centering and scaling. Compared to the full method, the training curve rises more slowly and exhibits larger oscillations, with a lower final avg @ 16. This behavior is consistent with our analysis: without standardizing by the batch statistics (uB, σg). the effective step size becomes entangled with the transient reward scale and the prompt-mix in each batch, inflating gradient variance and worsening PPO clipping dynamics. In contrast, SPO's batch-wise normalization stabilizes the update scale across iterations, yielding faster and smoother convergence. Together with the baseline and iilfine-initialization ablations, these results underscore that SPO's low-variance learning signal derives from both a history-informed baseline and global, batch-level normalization, as predicted by the variance decomposition in Appendix D.2
G ADDITIONAL EXPERIMENTS AND ANALYSES
This section reports additional experiments conducted after the initial submission. They probe the sensitivity of the KL-adaptive value tracker, the effect of the prompt curriculum, the robustness of the warm-start initialization, and the competitiveness of SPO against further baselines and model scales. Unless otherwise stated, all new studies are performed on Qwen3-8B with the Tool-Integrated Reasoning setup described in Appendix B using the DAPO training corpus and the same reward and evaluation protocols as in the main text.
Training setup. For the additional experiments we slightly reduce the per-step training budget due to time constraints. GRPO (with and without global normalization) and RLOO use prompt batch size of 128, group size G 8, and mini-batch size 16. SPO uses a single-stream batch of 1,024 prompts and mini-batch size of 128, so that the total number of sampled trajectories per update matches the group-based methods (128 prompts x 8 samples). In all settings, the maximum response length is set to 8,192 tokens; other hyperparameters follow Appendix
G.1 SENSITIVITY OF THE KL-ADAPTIVE VALUE TRACKER
The sampling weight in Eq. I1 priori- tizes prompts according to the Bernoulli standard deviation /B(1 - f). which is maximized when è ss 0.5 and down-weight prompts that are almost always solved or failed. As pointed out in the reviews, this heuristic does not explicitly account for the estimation confidence encoded by the effective sample size Not a + B of the Beta tracker.
Incorporating effective sample size into the curriculum.
To investigate this, we introduce an alternative sampling weight that down- weights prompts once their success probability is well-estimated:
V 0 01-1((12))
uy(x))
+c,
(13)
Nott.i-1(1))
where (et is the effective sample size of the tracker for prompt before iteration i and is the exploration bonus from Eq. [T] Table 2 compares the original and -ar-aware curricula on AIME 24 and AIME 25 at Dhatt 0.08.
17

---

Published as a conference paper at ICLR 2026
Table 2:
Effect of incorporating Netf into the sampling weight on AIME 24/25 maj 16 (Qwen3-8B, Dualt 0.08).
Sampling weight +E
AIME 24 maj@ 16
AIME 25 maj @ 16
55.95
52.75
Not tE
60.64
46.24
Table 3: Ablation on Dattt for Qwen3-8B with the Non- curriculum. Higher is better (maj @ 16).
Dhait
AIME 24 maj@
AIME 23 maj@
0.06
57.58
51.28
0.08
60.64
46.24
0.10
58.90
52.73
0.12
67.20
55.23
The mixed trend highlights nuanced exploration-cxploiatiio trade-off. Dividing by Neir encourages cycling through prompts more aggressively: once a prompt has been sampled many times, its weight decreases even when i x 0.5. This broader coverage appears beneficial for AIME 24, but less so for AIME 25, where spending more steps on a smaller of challenging prompts is advantageous. We therefore modify the sampling weight in the mainpaper to incorporate New in a manner that balances this need for exploration with the deep learning required for generalization.
Sensitivity to Dualt- The hyperparameter Drai controls how quickly the Beta tracker forgets past observations as the policy drifts: larger Daat yields longer effective memory, reducing variance but increasing bias with respect to the current policy. We ablate D E {0.06, 0.08, 0.10, 0.12) using the -wrre curriculum above. Results are summarized in Table
Performance relatively stable over broad range, Diair = 0.12 provides the best overall trade-off. In our setting (learning rate, batch size, and reward sparsity), this value implicitly defines "half-life" that allows the tracker to accumulate enough history to reduce variance, while not retaining excessively stale information once the policy has moved far in KL.
G.2 ANALYSIS OF THE PROMPT CURRICULUM
Effect of prioritized sampling. To isolate the contribution of the automatic curriculum, we compare the full SPO algorithm (with prioritized sampling from Eq. 11) against a variant without sampling mechanism, keeping all other components unchanged (including the value tracker and global nor- malization). Table 4 reports maj @ 16 on AIME 24 and AIME 25, Removing the curriculum leads to substantial drop (-10.8 and 9..3 percentage points on AIME 24 and AIME 25 respectively). confirming that focusing computation on prompts with high estimated learning potential is critical for both data efficiency and final performance.
Revisiting hard prompts. A natural concern with the variance-based weight / /0(1 6 is that prompts that are consistently failed (ō 0) could effectively disappear from training. Eq. (II) we explicitly prevent this collapse by adding an exploration bonus c, which ensures a non-zero lower bound on each prompt's sampling probability. As the model improves, such hard prompts are periodically revisited, allowing the tracker to update their value estimates if the policy's capabilities increase.
G.3 WARM-START SAMPLE SIZE no
Appendix already ablated the extreme case of removing the offline initialization entirely (ng = 0), which leads to unstable early training but eventually recovers. Here ask whether very small warm start is sufficient by comparing no against the default nig = 8.
The results are:
18

---

Published as a conference paper at ICLR 2026
Table 4: Effect of prioritized prompt sampling (Eq.
on maj (Qwen3-8B, Dhair 0.12).
Sampling scheme
AIME 24 maj @ 16
AIME 25 maj @
None Prioritized (SPO)
56.38
45.90 55.23
67.20
Table 5: Comparison with additional RI baselines on AIME 24/25 maj @ 16 (Qwen3-8B)
Method
AIME 24 maj@@ 16
AIME 25 maj 16
GRPO (standard)
55.02
44.71
GRPO Global Norm
60.59
48.32
RLOO
62.91
51.70
SPO (ours)
67.20
55.23
Early training (step 20): no = 8 reaches avg @ 16 of 40.42, whereas 2 lags behind at 36.46.
Convergence (step 120): the gap closes and slightly reverses: 47.50 for 710 versus 47.71 for no = 2.
Thus, even minimal warm start (ng = 2) is sufficient to reach essentially identical final performance, demonstrating that SPO's online tracker is robust. However, somewhat larger initialization (e.g.. ny = 8) remains preferable practice, as stabilizes the initial phase and accelerates early convergence.
G.4 ADDITIONAL BASELINES: RLOO AND GRPO WITH GLOBAL NORMALIZATION
RLOO. Beyond GRPO and PP (Appendix we evaluate SPO against RLOO, which uses leave-one-out group baseline in Eq. 4 with the same group size G 8. RLOO is strong group-based baseline but consistently outperformed by SPO.
GRPO with batch-level global normalization. To disentangle the effect global advantage normalization from the rest of SPO, we augment GRPO with the same batch-level normalization used by SPO: advantages are computed with the standard GRPO group baseline but are then normalized across the whole batch rather than within each group. This variant ("GRPO + Global Norm") substantially improves over standard GRPO, but still falls short of SPO, indicating that global normalization is an important but not exclusive contributor to SPO's gains. Table 5 summarizes the comparison.
These results confirm three points: (i) batch-level normalization alone yields tangible gains even for group-based methods, (ii) single-stream SPO remains competitive against strong variants such as RLOO, and (iii) SPO's remaining advantage primarily stems from its group-free data efficiency and KL-adaptive baseline, which enable 8x more unique prompts per update at fixed rollout budget.
G.5 EXPERIMENTS ON SMALLER BASE MODEL:: QWEN3-4B
To assess robustness across model scales, repeat the comparison between GRPO and SPO Qwen3-4B, using the same Tool-Integrated Reasoning setup and training hyperparameters (including D = 0.12 and the prioritized curriculum). reports maj @ 16 performance AIME 24 and
AIME 25.
Table 6: GRPO SPO Qwen3-4B (maj @ 16).
Method
AIME 24 maj maj@ 16
AIME 25 maj @ 16
GRPO
56.01
46.62
SPO
59.32
48.07
SPO again outperforms GRPO on both benchmarks, indicating that the benefits of single-stream optimization and the KL-adaptive value tracker specific the 8B scale. During Qwen3-4B
19

---

Published as a conference paper at ICLR 2026
training also observe that the model tends degenerate into pure-text solver, rarely producing successful tool calls; both methods eventually rely more on text-only reasoning. This suggests that smaller tool-augmented backbones may require more careful cold-start or auxiliary supervision to reliably acquire tool-use capabilities, which we leave as an avenue for future work.
20


Reviews:
[{"review_id": "55IlBujRPC", "reviewer": "Reviewer_ss5K", "summary": "This paper introduces Single-stream policy optimization (SPO) for RLVR (verifiable rewards) LLM training. It basically compares with GRPO,  and replaces GRPO's baseline with (i) a persistent, KL-adaptive Beta tracker for per-prompt success probability (ii) global advantage normalization, and (iii) prioritized prompt sampling that biases training toward prompts with higher learning potential.", "questions": "See above weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Very clear problem framing. It identifies two GRPO pain points, degenerate groups (no signal) and synchronization overhead and direct addresses both.\n2. The group-free simulation (4.35x faster) is compelling for tool-use/agentic regimes as low-latency is needed.", "weaknesses": "1. Prioritized sampling is not so valid and might fail. This is a proxy for Bernoulli uncertainty but ignores the effective sample size $N_{eff} = \\alpha + \\beta$ carried by the Beta tracker. A prompt with $\\hat{v} \\approx 0.5$ but huge $N_{eff}$ should be less prioritized than one with the same $\\hat{v}$ and tiny $N_{eff}$. The prioritized sampling can over-sample well-known, mid-probability prompts.\n2. For $\\rho(x) = 2^{\\frac{-D(x)}{D_{half}}}$, there's no derivation or sensitivity study for $D_{half}$. Can the authors provide more ablation study on that?\n3. The experiment is only on one base model (Qwen3-8B) and binary RLVR reward. Showing another base LLM or a non-binary reward variant would be more beneficial.\n4. Missing baselines. Can the authors provide some non-GRPO baselines? A*-PO, GRESO, RLOO? Why the authors only provide GRPO as baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHhmMv88kZ", "reviewer": "Reviewer_uRF7", "summary": "Recent propular RL methods like GRPO employ group-based value estimate and avoid maintainence and training overhead of a learned value model. However, they also come with some downsides. First, the value calculation in GRPO requires a group of rollouts for each update, which makes it expensive. This becomes especially problematic in agentic workflows when the slowest rollout in the group will slowdown the entire update. Second, there can be degenerate groups (all pass or all fail) which will lead to zero advantage and subsequent wasted computation.\n\nTo fix the above limitations, authors propose Single-stream Policy Optimization (SPO). The key innovations of this methods are as follows:\n1. It uses a KL-adaptive value estimate as the posterior of the a parametrized beta distribution. The alpha and beta values are updated based on sampled reward (binary 0,1) such that effective update rule for value follows an adaptive exponential moving average.\n2. Since each prompt has a unique moving value estimate, they only use single response per prompt and normalize the advantage over the entire training batch. They used PPO's clipped objective for training their policy.\n3. They also leverage the value estimate to construct a prompt sampling weight that puts highest weight for prompt close to 50% win rate and creates an automatic prompt curriculum.\n\nOverall, their method consistently improves over GRPO over a diverse range of cometition math evals. In appendix they also compare with A*PO, another method that uses fixed precomputed V* estimate and takes one sample per prompt during online learning. SPO performs very similar to A*PO while also giving the advantages of automatic prompt weighting and learning free dynamic value estimate.", "questions": "1. Would have preferred to see an ablation experiment of w/ and w/o prompt curriculum to see the effectiveness of automatic curriculum\n2. I'm curious if the method will very rarely re-visit harder prompts due to the weighted sampling scheme. For example, if a prompt is harder for initial checkpoint and consistently gets 0 reward, its value will drift towards 0 and will be less likely to get resampled in the future.\n\nTypos:\n1. (equation 7) $N_{eff} = \\alpha(x) + \\beta(x)$.. there shouldn't be a +1\n2. (equation 10) Advantage shouldn't have subscript $t$ as its only using terminal reward.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Clear motivation and a very innovative way of calculating a moving model-free value estimate via on-policy samples\n- Their normalized advantage calculation leads to low variance and stable training\n- Their dynamic value estimate paves way for automatic prompt curriculum", "weaknesses": "- Their method does require some value warmstarting to be effective (similar to the inital cost of A*PO).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d4IOGcgkRN", "reviewer": "Reviewer_GDbD", "summary": "The paper proposes SPO, a simplified and scalable alternative to group-based methods like GRPO for fine-tuning LLM. It eliminates group synchronization and potential baseline issues by using a KL-adaptive value tracker and global advantage normalization. Experiments with Qwen3-8B show faster convergence and higher accuracy on challenging math reasoning benchmarks, achieving significant gains in maj@32 and pass@k metrics.\n\n\nOverall, the idea is interesting and potentially offers significant advantages. The acceleration in training speed can be substantial; however, the number of experiments conducted does not seem sufficient. Beyond the limited number of datasets and models, the authors introduced several new hyperparameters but did not isolate their individual contributions. The algorithm is always presented as a whole, while it is limited in how each parameter impacts performance. It would also have been valuable to include an ablation study on a smaller, less computationally expensive experimental setup, as I understand that the current configuration may be time- and resource-intensive.", "questions": "In addition to clarifying the absence of ablation studies, please address the last weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Training speed is notably increased through the streaming of a single output.\n- The paper provides a variance analysis of GRPO.", "weaknesses": "- Only one model and one dataset were used for training; testing multiple models across different datasets would be necessary to validate the generality of the results. \n- No ablation study was conducted for the D_half parameter.\n- The effect of using the same prompt versus different prompts was not analyzed.\n- No ablation was performed for the w parameter.\n- Since the warm start seems effective, it would be helpful to include an ablation and clarification showing the minimal effective value,  for instance, demonstrating that a relatively small warm start is sufficient to gain its benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pJVuS8XXyM", "reviewer": "Reviewer_hydy", "summary": "This paper introduces Single-Stream Policy Optimization (SPO), a group-free fine-tuning algorithm that effectively improves the throughput and scalability of group-based methods (e.g., GRPO) in long-horizon or tool-integrated scenarios. The authors also propose a KL-adaptive value tracker and batch-level advantage normalization to provide stable and continuous learning signals. Experimental results show that SPO significantly enhances the model's reasoning accuracy in terms of pass@k.", "questions": "- I think the value tracker and batch-level advantage normalization could also be directly integrated into GRPO, would this lead to further improvements, would it outperforms SPO?\n- In a standard RLVR scenario, does single-stream result in $\\times G$ times the runtime compared to GRPO (where $G$ is the number of responses in a group)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The idea and algorithm framework is simple.\n- The motivation is clear.\n- The experimental results are significant.", "weaknesses": "- Judging from the title, the core contribution lies in single-stream. Although Section 5.4 discusses the agentic training scenario, I believe plotting runtime versus pass@k is important, as it helps highlight the advantages of single-stream (If the advantage of single-stream mainly lies in distributed training scenarios). The authors mainly analyze a specific example in Figure 4 and Figure 5, but I think providing the corresponding success rate metrics is also necessary.\n- I encourage the authors to plot ablation experiments similar to Figure 2, which would help everyone intuitively and quickly understand the contribution of each component in SPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)