Stepwise Guided Policy Optimization: Coloring Your Incorrect Reasoning in GRPO
Authors: Peter Chen
Xiaopeng Li
Ziniu Li
Xi Chen
Tianyi Lin
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning (RL) has proven effective in strengthening the reasoning capabilities of large language models (LLMs). A widely adopted method, Group Relative Policy Optimization (GRPO)~\citep{Shao-2024-Deepseekmath}, has shown strong empirical results in training recent reasoning models~\citep{Guo-2025-Deepseek}, but it fails to update the policy when all responses within a group are incorrect (i.e., all-negative-sample groups). This limitation highlights a gap between artificial and human intelligence: unlike humans, who can learn from mistakes, GRPO discards these failure signals. We introduce a simple framework to mitigate the all-negative-sample issue by incorporating response diversity within groups using a \textit{step-wise} judge model, which can be trained directly or adapted from existing LLMs. In a simplified setting, we prove that this diversification accelerates GRPO’s learning dynamics. We then empirically validate Stepwise Guided Policy Optimization (SGPO) across model sizes (7B, 14B, 32B) in both offline and online training on nine reasoning benchmarks (including base and distilled variants). Overall, SGPO improves average performance and is effective in early and mid-training when all-negative groups are prevalent, while improvements are not uniform across every benchmark and depend on the structure and informativeness of negative samples. Finally, SGPO does not require the judge model to generate correct solutions, distinguishing it from knowledge distillation methods.

Published in Transactions on Machine Learning Research (03/2026)
Stepwise Guided Policy Optimization: Coloring Your Incorrect Reasoning in GRPO
Peter Chen Deportment of Mathematics Columbia Universily
le3826@oolumbia.eti
Xiaopeng Li
ziaopenglil I &cuhk. edu.cn
School of Artificial Intelligence The Chinese University of Hong Kong, Shenzhen
Ziniu Li
lisineu1997@gmmillccom
School of Data Science The Chinese University of Hong Kong, Shenzhen
Xi Chen
ac13@stern.ny met.ssteennn.... edu
Stern School of Business New York Universily
Tianyi Lin
#3335 44335@ccouubbaa.co edu
Deportment of Industrial Engineering and Operation Research Columbia University
Reviewed OpenReview: https: openreviee net/ forum? id= ALnVAgtshR
Abstract
Reinforcement learning (RL) has proven effective in strengthening the reasoning capabilities of large language models (LLMs). A widely adopted method, Group Relative Policy Opti- mization (GRPO) (Shao et al., 2024), has shown strong empirical results training recent reasoning models (Guo al., 2025a), but fails to update the policy when all responsos within group are incorrect (i.e., sal-nel groups). This limitation highlights gap between artificial and human intelligence: unlike humans, who learn from mistakes, GRPO discards these failure signals. We introduce simple framework to mitigate the all- negativo-sample issue by incorporating response diversity within groups using a step-wise judge model, which can be trained directly or adapted from existing LLMs. In a simplified setting, we prove that this diversification accelerates GRPO's learning dynamics. We then empirically validate Stepwise Guided Policy Optimization (SGPO) across model sizes (7B, 14B, 32B) in both offline and online training on nine reasoning benchmarks (including base and distilled variants). Overall, SGPO improves average performance and is effective in early and mid-training when all-negative groups prevalent, while improvements not uniform across every benchmark and depend on the structure and informativeness of nega- tive samples. Finally, SGPO does not require the judge model to generate correct solutions, distinguishing it from knowlodge distillation methods.
1 Introduction
The rise of OpenAI-ol (Jacch al., 2024), DeepSock-R1 (Guo et al., 2025a), and Kimi-1.5 (Team et al., 2025) has highlighted the emergence of large Al reasoning models. Unlike instruction-tunet models (Brown al., 2020; Chowdhery et al., 2023; Touvron et al., 2023; Achiam et al., 2023), which produce quick responses by

---

Published in Transactions on Machine Learning Research (03/2026)
GRPO
PG
Response (Positive Sample)
Prompt
Prompt
Prompt
Incorrect Response
(Negative Sample]
Sampling
Sampling
Sampling
Reward
Reward
Uvn
Advantage
Advantage
RLAIF Process Supervision
A.85 Jut
(A-05) VOE
A-0
4-0
AND
A-0
1
Both positive negative samples are both utilized for mixed group
Samples discarded advantage for all-negative-groupp
Advantage
SGPO diversifies the rewards for all-negative-ggoupp to utilize them
Figure 1: Main pipeline for Stepwise Guided Policy Optimization. On each llama, T indicates the reward of the sampled response and indicates response's advantage through group relative computation.
statistically inferring the next token, these new reasoning models deliberately decompose complex prompts (c.g., mathematical problems) into intermediate steps and work through shain-of-thoughe reasoning (Wei et al., 2022; Yao al., 2023; Besta et al., 2024; Xiang al., 2025). This slower yet. more rigorous process yields greater accuracy and makes them more buman--lkee enabling success on more complex and challenging tasks (Yang et al., 2018; Shi al., 2024; Jain et al., 2025). As generative Al applications move beyond single- turn chat and guestion-answering, these reasoning models are poised to become more powerful and widely adopted, positioning them as foundational component of modern AI systems.
At the heart of this revolution lies post-training with outcome-based and verifiable rewards (Cobbe et al., 2021; Uesato et al., 2022; Zelikman et al., 2022; Singh et al., 2023; Hosseini et al., 2024; Lightman et al., 2024; Wang al., 2024; Sctlur al., 2025; Zhang al., 2025b), together with reinforcement learning (RL) methods (Schulman et al., 2015; 2017; Li et al., 2024b; Ahmadian et al., 2024; Shao et al., 2024; Xiong al., 2025a), appreciated for their simplicity, intuitiveness, and practicality. A leading approach proximal policy optimization (PPO) (Schulman et al., 2017), which relies critic (or value) model to estimate advantages. While essential in general RL tasks, this critic is often unnecessary in large language models (LLMs) due to their deterministic transition dynamics (Li et al., 2024b). This observation has inspired alternatives such as group relative policy optimization (GRPO) (Shao et al., 2024) and its extensions (Yu et al., 2025b; Liu et al., 2025b; Chu al., 2025; Zhang et al., 2025a), which estimate advantages directly in group-relative fashion (normalizing rewards across multiple samples for the same prompt).
A major limitation of these methods arises when all sampled responses in a group incorrect (i.e., all- negalive-sample groups), which eliminates the learning signal and halts policy updates. In GRPO, given a prompt responses { are drawn from the old policy Told and assigned rewards fri]. where r = 1 if y, is correct and 0 otherwise. Advantages are obtained by normalizing within the group. If = 0 for all i, the advantage vanishos, yielding no update. Such groups are frequent in early and mid-stages of training, when reasoning ability is weak!. This shortcoming highlights a gap between artificial and human intelligence: humans effectively learn from mistakes, which act essential signals during cognitive development (Chialvo
To reduce computational cost, training often uses small group sizes short rollouts, further increasing the likelihood of all-negative-sample groups.
2

---

Published in Transactions on Machine Learning Research (03/2026)
& Bak, 1999). In mathematical reasoning, all-negativo-sample groups prompt a child to revise rules and strengthen reasoning ability.
Recent studies suggest that negative samples in RL-based large reasoning model training carry more nuanced value than previously assumed (Xiong et al., 2025a). Instead of treating negative samples uniformly, they advocate for principled mechanisms to distinguish negative samples. One prominent direction process reward models (PRMs) (Lightman et al., 2024; Wang al., 2024; Luo et al., 2024; Setlur al., 2025; Zhang et al., 2025b), which estimate either the probability of final success or its change after each reasoning step. However, their reliance speculative value functions makes them prone to reward hacking (Skalse et al., 2022).
A common observation is that many reasoning tasks possess a structure where step-level correctness can be explicitly defined. This motivates the use of a step-wise judge model that evaluates trajectories by labeling cach step as correct (1) or incorrect (0). Such a model can be trained directly (Xiong et al., 2025b) or adapted from existing LLMs (Zha et al., 2025; He et al., 2025)². By grounding rewards in step- level correctness rather than speculative value estimates, our method mitigates reward hacking and yields clearer signals. Intuitively, this allows negative samples to be differentiated through their trajectories: while early-stage reasoning trajectories are of low-quality, these remain informative - much like partial credit in education, where intermediate steps still guide learning.
Our approach enables a holistic evaluation of multi-step reasoning by transforming negative samples from binary outcome rewards into graded step-level rewards. Consider a negative sample with five reasoning steps (11, 02, 03: 01 ag). Tf the first error occurs at 03, then a₁ and 02 are correct, yielding a correctness pro- portion of 3. To improve reliability, we adopt Grok4-Heavy -inspired strategy where multiple independent judgments are obtained from the judge model, and the error position is determined by the majority vole. We further introduce two scaling parameters / and y to downweight noisy or unreliable signals (see Eq. (2)). Un- like PRMs, our approach avoids memory overhead and does not require costly step-level human annotations, thereby accelerating training. In this work, we focus on outcome- post-training with group-relative updates (GRPO-style) for structured reasoning lasks; extending our approach to arbitrary reward settings is beyond the scope of this paper and left to future work.
Contribution. We propose and analyze simple and efficient framework that introduces response diversity within all-neeative--smmle groups. It is both theoretically grounded in the simplified setting and empirically effective on various models, distinguishing our approach from existing heuristics. Our contributions can be summarized as follows:
1. We propose a Stepwise Guided Policy Optimization (SGPO) framework that leverages a step-wisc judge model that identifies the first incorrect step that causes the trajectory to deviate from correct- ness. This makes evaluation computationally tractable and reliable. It is important to emphasize that our contribution lies not in designing effective judge models, but in introducing a framework that leverages step-wise judges to effectively distinguish negative samples. We also prove that SGPO outperforms GRPO in simplified setting.
2.
We conduct experiments demonstrating the effectiveness of our approach in improving LLM reason- ing. Evaluations are undertaken across various model sizes (7B, 14B, 32B) in both offline and online settings with nine benchmarks, including base and distilled variants, using GRPO as the primary baseline given our focus on oulcome-based group-relative RLVR. Our results reveal two key benefits: (i) SGPO delivers improvements beyond the reach of GRPO, especially in the early and mid-stages of training where all-ncgative-sampl. groups are common; (ii) SGPO does not rely on more powerful judge models generating correct answers, allowing it to be distinguished from knowledge distillation
The additional overhead from all-negative-ssmple groups remains modest, since the correctness can be ef- ficiently verified against reference solutions, enabling rapid assessment of reasoning steps. As the compu- tational and financial costs of closed-source judge models (o4-mini, Claude3.7) 77 rise, SGPO accelerates learning dynamics, making the tradc-off worthwhile. SGPO also outperforms GRPO with less powerful and We do not have access their judge models as it's not publicly released, so we adapt our own from existing L.Mss.
3

---

Published in Transactions on Machine Learning Research (03/2026)
more affordable open-source judge models (DeepSeek-V3-0324, Qwen3-235B-A225 QwQ-32B), confirming that SGPO remains effective even without cutting-edge LLMS and underscoring its practicality in lower- resource settings.
2 Preliminaries and Technical Background
LLM finetuning. LLM finetuning typically consists pre-training and post-training. Pre-training equips the model with broad language understanding and generation capabilities, while post-training adapts the model to specific downstream objectives (e.g., improving mathematical problem solving via reasoning). In post-training, a model usually first undergoes imitation learning (o.g., supervised finetuning human expert trajectories, direct distillation from stronger models), and then further improves by training on self- generated responses paired with feedback indicating whether the responses are good or bad. Two common feedback-driven paradigms are reinforcement learning from human feedback (RLHF), where a learned reward model (trained from human preferences) assigns rewards to model outputs, and reinforcement learning from verifiable rewards (RIVR), where rewards are computed by an aulomatic verifier (often used for math problems with checkable answers). We introduce GRPO, a representative RLVR method, below.
Policy optimization. Modern LLMs are built based the Transformer architecture (Vaswani et al., 2017) and generate responses denoting the vocabulary and V the set of all possible token sequences. We view the LLM as a policy No (y x) parameterized by 8, assigning probabilities to responses y given x. The policy operates in an auto-regressive way as follows (Agarwal et al., 2020; Mei et al., 2021; Li et al., 2024b):
(a1.....ar1) to user prompts x, where each token E V", with y
11 II ma x,a),...,, aa-1)) A=1
<o(y)x)
For prompt x with ground-truth response y* performance is evaluated using a regular-expression match on the final answer: r(x,y) = 1 if y matches yx and r(x,y) 0 otherwise (Hendrycks et al., 2021). We consider the reasoning tasks defined over dataset D = y)), where each x is a problem and y its ground-truth solution.
The policy gradient methods (Williams, 1992; Sutton & Barto, 1998) aim to maximize the objective J(0) = Ex==py===a (rx y)] where p the prompt distribution and me is an LLM policy. Parameters are updated via 0 < f + (()). In practice, trajectories are sampled from an old policy Tamas which is different from wa, motivating the use of importance sampling as follows:
J(() Exxxpy (xxx)
moly|x) Te (y|x) r (xx,))
However, this estimator suffers from high variance when To deviates from Tosia" To stabilize training, clipped surrogate objectives are used as follows:
((0) = Ex-p,y (+1x)
min
sta (y|x) Tosa [y/x) (x, y), clip
moly|x) Re, [y/x] 1- -,-,,1+ -
where clip(x, 1-, +))= = max (mm) The group relative policy optimization (GRPO) and its variants (Yu et al., 2025b; Liu et al., 2025b; Chu et al., 2025; Zhang et al., 2025a) adopt this framework but estimate gradients using groups of samples. For each prompt x, GRPO samples responses y1,..,, YG from Tema- We aim at maximizing the objective function in the form of
((0) Ex Sx (+1x)
Σ min
{
Nely. Te (v.
AG, clip
so(y. To (Yi/x) 1 e, + +e) Ai
where e E 1) and the advantage A; is computed as
A rix.y.) mem.([ std((r(x.y)) (xxxxxx)))
(1)
where r(x,y:) 1 if y: matches the ground-truth answer and 0 otherwise.
4

---

Published in Transactions on Machine Learning Research (03/2026)
Remark 2.1. When rewards are identical across all samples within a group, A, = and updote occurs. This is oppropriate for all-posilive groups but constitules a critical limilation for all-negalive groups, where GRPO fails to exploit mistakes as learning signals.
3
Main Results
We propose the Stepwise Guided Policy Optimization (SGPO) framework, which employs the step-wise judge model to detect the first. incorrect step that leads trajectory away from correctness. Tn simplified setting, prove that SGPO consistently accelerates GRPO's learning dynamics.
3.1 A Step-wise Judge Model
We propose a principled reward mechanism for negative samples, wherein the step-wise judge model differ- entiates between structurally sound but partially incorrect reasoning and entirely erroneous responses. This design is motivated by the intuition that an incorrect final answer does not invalidate the entire reasoning process. For instance, model may follow logically coherent sequence of steps yet make a minor error such as arithmetic slip that leads flawed or incoherent reasoning does not make sense. This refinement remains effective under constraints such as reduced output length, where a model may be unable to complete the full solution but still demonstrates a valid reasoning trajectory.
incorrect conclusion. Treating such cases the same fundamentally
Our step-wise judge model can either trained directly or adapted from existing LLMs. evaluates responses sequentially, identifying the first substantive error such as computational slip or logical fallacy that causes the trajectory to deviate from correctness. To formalize this, we define the Reasoning Trajectory Score (RTS) for incorrect response y, denoted as RTS(y) E [0, 1]. The judge model checks cach step in order, pinpoints the first error, and treats all preceding steps as the valid reasoning segment. RTS(y) is then computed as the ratio of the valid segment length to the total trajectory length. For example, if 3 y consists of five steps (41, 02, as d4, a5) and the first error occurs at d4, then RTS(y) = 5 indicating that three steps reasoning are correct before erroncous.
In our experiment, we adapt the judge model from existing LLMs, either dlosed-ource (o4-mini, Claude3. 7) or open-source (DeepSeex-V3-0324. Qwen3-235B-AMe A22B, QwQ-32B). To enhance reliability and further reduce variance in the reward signal, employ the following protocol: (i) alongside the candidate response, we provide the judge with reference solution (e.g., gold final answer, brief solution outline, or a full reasoning trace when available); in our experiments, we draw this solution from supervised fine-tuning dataset with correct answers and reasoning trajectorics, which anchors the intended solution path and enables error localization; and (ii) we elicit step-wise evaluation rather than holistic evaluation. The judge model justifies correctness or flags an error sentence by sentence, identifies the first clear mistake, and then traces how this error propagates the final incorrect conclusion.
Based on the reasoning trajectory score, introduce new outcome reward function:
if the final answer of y is correct, 1+exp( (8o0(((())))))))))) otherwise.
1,
Tsapo
(2)
where y > 0 and B > 0 (taken 10 and 0.5 in the actual implementation) are two parameters to decide scale threshold and scale intensity, respectively. This design ensures that the model receives a more informative gradient signal during training, thereby encouraging refinement. of partially correct reasoning rather than indiscriminate penalization of all incorrect outputs. This specification of TeaPo can be directly incorporated into the advantage calculation in Eq. (1). As consequence, SGPO keeps the same rollout pipeline as GRPO and the same outcome- based supervision, and only replaces the reward used in rithin-group advantage computation from r(x, v) to (y) in Eq. (2).
Remark 3.1. Our approach differs from process reward models (PRMs) (e.g. Lightman al., 2024). For prompt x and a prefir of reasoning steps (oI,.... ..), PRM typically predicts either (i) a prefix-level value V(x, a1x) Pfinnll answer correct. xx,)) or (ï) step-level progress signal such D. V(x,
5

---

Published in Transactions on Machine Learning Research (03/2026)
((x, 01.t-1). In practice, PRMs are trained by supervised ranking of intermediate steps and are used to re-rank trujectories or shope truining at the prefix level, acting approzimale value (or Q-value) functions over prefizes. In contrast, SGPO introduces a different way of producing and using feedback signals: (i) Policy-guided rollouts without search. All trajectories are sampled from the current policy, without PRM- guided exploration or Irajectory alleration; (fi) Post-hoc first-error identificntion. A step-wise judge inspects the entire trajectory, pinpoints the earliest error relative a reference solution, and converts this into a calibrated scalar reword r'sroly) via the reasoning trajectory score; (ttt) Stable credit assignment in all- negative-sample groups. By locating the first definitive mistoke only after observing the full trace, SGPO eliminates the look-ahead ambiguily and feeaback loops inherent to PRM-guided search (Zhang et al., 2024a), while avoiding the need for the judge to solve the problem or approximate a value function. We note that SGPO alleviales but does not fully eliminale degeneracy: if all trajeclories in a group receive identical rscro (e.g., they fail ot the same first-error position), the group-normalized advantages can still vanish, though this phenomenon is rarely observed our experiments.
Remark 3.2. Our approach differs from knowledge distillation (e.g. Kang et al., 2023; Gu et al., 2024). The student model trained via knowledge distillation inherits the judge the model's 's failure, since it only imitates the judge model's outputs. In contrast, SGPO leverages the judge model to identify mistakes in the student 's reasoning, providing learning signals that go beyond imitation and enabling improvements unattainable by knowleagge distillation.
3.2 Accelerating Learning Dynamics
We present a theoretical analysis to explain why SGPO outperforms GRPO. We study an idealized setting in which the step-wise judge provides accurate first-error localization. Even in this regime, establishing a general separation is technically subtle, so we focus on a stylized example. To this end, we consider the case when 2, 2, where each step admits two possible actions an E 1,2 for h = 1, 2. Extending the analysis to general horizons or action spaces and imperfect (noisy or biased) judges is left to future work. This configuration follows prior works (Dayan, 1991; Li et al., 2024b), in which analogous examples were employed to validate theoretical insights. Without loss of generality, we assume a unique ground-truth response yx (2 for the prompt x. For clarity, we restrict the sample space to (1, (2, 1). 2), excluding (1,2) since correct reasoning step is unlikely to, and should not, follow an incorrect precursor.
To illustrate the effect of SGPO, we analyze the learning dynamics of SGPO and GRPO in this simplified setting. Under GRPO, the rewards are assigned as 2)) = 1 and r((2, 1)) = 1)) 0, mcaning that only selecting the "good" action 2 at both steps yields a positive reward. In contrast, SGPO assigns (2, 2)) 1, T2 1)) = and (sc,o((1, 1)) = 0. The difference is that partial progress choosing the "good" action 2 in the first step but failing at the second receives no credit in GRPO yet proportional credit in SGPO. Here, 1/2 is chosen for illustrative purposes to convey the qualitative behavior of the reward mechanism, while the exact values used in experiments are determined by Fq. (2).
The algorithm iteratively updates the policy parameter 0 using samples drawn from the current policy me. We rewrite the generic GRPO update with a step size rj > 0 as follows,
N G
B sa(x 11b--1 As. )..
g(++1) =0() (()) +n.g((0) where g(0) = NGH
ΣΣΣ
where N is the number of prompts, G is the number of groups, H is the number of reasoning steps in i,k cach response, so(x', - == Volog (a x, 01 is the score function, and the advantage A is defined by Eq. (1) for each question x'. To distinguish, we denote gGroo (.) as the gradient estimator using classical outcome reward model r, and (sapo()) as the gradient estimator using the reward r'aPo as proposed in Section 3.1.
In our analysis, we examine the population-level learning dynamics with G = 2, omitting clipping and importance sampling. In practice, importance sampling and clipping are important for training stability; we omit them here to simplify the analysis and leave their theoretical treatment to future work. For simplicity, we perform our analysis in the likelihood space rather than in the parameter space directly. Indeed, we
6

---

Published in Transactions on Machine Learning Research (03/2026)
define the key quantities follows
6x...2
- ne, (a1 xx
Te, (a2 = 1,,1,x,0 = =
o
+e 2
2.2
Note that the original 4-dimensional parameter space defined by 0x 8x. 6%.2,1 and 6x.22 in is reduced to 2-dimensional likelihood space defined by p,9 [0, 1].
For our simple stylized model, compute the score functions in terms of likelihood parameters p.q follows,
p
1
s(a1 = 11x)
s(a1 = x
and
0
0
sa2 x, a1 = 2)
s(a2 = 2| a, = 2)=
-
The responses can be drawn i.dd. from the distribution as follows,
(1,1), w.p. p,
(2), = (2, w.p. p(1 - 9).
(2,2), w.p. pq.
The SGPO and GRPO training dynamics with population-level policy gradient can be computed exactly for the stylized model as follows,
p(p 1) p(1 p)
p(p - 1)9
p(1 - p)q
Jsapo (0) = E Jacpo (0) =
JaReo (0) = EE (CaPo (0) =
p²qlg - 1)
pg(g 1)
p²q(1 4)
pg 9)
Since (GRFO (0) and (G) concentrate around (aRFa(() and (8) when the number of samples in each group is sufficiently large, it is reasonable to analyze the population-level dynamics for illustration. Note that the high-pro brobbbbrbbaiiitty guarantees for the sample-level dynamics can be derived using concentration inequalities under certain conditions.
(k) Denote Paseo and CGRPo the value of quantity p and o at iteration under GRPO. Analogously, Psapo and (k)
(k)
(k)
Isapp are the corresponding probabilitics and at iteration under SGPO. We can explicitly write down
the SGPO and GRPO update rules with v = 1 as follows,
PaGeo exp(f11 (PsaPo)): (k+1) exp(f12 (Pscro- (
(k++1) I)
and PERPO cxp(fe1 (Paroo
(k++1)
(k) 4GRPO (k)
(3)
OsaPo
(k)
CGRPO
(k+1)
exp(fzel (Peaso 4GpP
(k)
where the functions fyz are defined by
fn1(p) log(p) + p11 p) log(1 p pe
(p,(p.) = log(p) + p(1 p)q - log(1 - pe -p)9).
(4)
f12(2, q) = log(g) + q² - q) log(1 - q + gee 9(1 ((1-))),
(2)(1.) log(g) + pg(1 - logg1 + mo -q)].
Our theoretical findings summarized in the following theorem.

---

Published in Transactions on Machine Learning Research (03/2026)
Theorem 3.3. Suppose that PaRPa 9aspo PsoPo UsaPo 1 and 1l 13 for GRPO and SGPO. Then,
(0) (0) (0)
(0)
as k - +00; (ti) SGPO outperforms GRPO in learning the "good" action in the first step: Pegeo > PeRpo for all k 2 1; (iii)
we have that (i) GRPO and SGPO achicve successful learning: PeaPo: Yenso: Psopo: Isapo
(k) (k) (k) (k)
(k)
SGPO oulperforms GRPO in learning the optimal policy: Pearollacro > PaRrollaro for all k 2 1.
(k) (k) (k) (k)
Proof. To show (i), recall that the sequence (PeGPo)KEN strictly incrcasing and bounded in (0. 1) from Lemmas (iA(i) and B.A(ii), it converges to some value c E (0, Taking limit as k + oo:
(k+1)
1= lim
Pairo
lim
(1 (1-e)e-00-004e)
8-+00
Panon
k000 (1 (1-Peora)
Monon s +
Using the simple Taylor lower bound
2 1 x2 have
(c c=
c=
(1-cle all +0
[T-c[[-a(--o)t]
This shows PsaPo + 1 as h + co. Similarly, we can show IaRPO Psapo IsaPa + 1 as k - 00.
(k) (k)
(R)
To show (ii), consider the base case:
(1) PsaPo
(Pscao))))))) = exp(log PsoPo + h (PsaPo (1 Psarol)) (0)
Psaro
exp(log PaRPO + h Powe (PoRPo (1 - PoRPD)))
(0)
exp(log PORPO + h (o) (PeNPo (1
(0)
PaRPo ManP)) exp(f22 lcapo)) PGRPO
(0)
(1)
where the incquality follows from Lemma Bii Thus, we use induction and assume Peapa k 1. Then, we have
(k)
PaRPO for some
(k)
(k+1) Psaro exp (pe)) (k)
(k) exp(log Parro + h (a)
(p()) (1 PGRFO )))
(k)
Paaro
exp(log PaRPo + h (peapo (1
PaRoo
(k) PaRPo PaRPo) McRPa) exp(/21 (PaPPo (dRPo))
(A++1)
Paws
where the first inequality uses Lemma (i) and the second one uses Lemma B.i(ii). Thus, Peara PGRPO (A:
(k++1)
(k++1)
and the induction is done. We have proved that Psaro > PoaPo for all k: 2 1.
To show (iii), first notice that can show PoRpo IoRoo for all & by induction. The base case is trivial
(k) (k)
by initialization. Suppose Peaeo = gORPO for some l 2 0, then by noticing that f21 p) = [22 P), we have
(k)
(k+1) = exp(f21 (PoRPo- (caPo)) )) = exp(f21 e Ponpo))
= exp((fue (2) palbo)) = exp(fz2(Pen))
(k) (ekka)) = 9ERPa
(k+1)
Thus, by induction, PORPO Jaaoo for all 1 Now,
(k)
(k)
reduce the update rule of Paapo
(k)
(k++1)
Parro
(1/Paah (4) (expp(((pait)) (2( (1-Ppa-p) Placo 1-11
In addition, recall the update rule of PsaPa and AsaPa as It sulfices to show Pscoo
(k)
(k)
(k+1)
Pecpo
(1/pb/) exp( Peseo Paseo 1)
(k+1)
4sapa
(1/25a0 exp( (Pooro) 12 Tooro (1
(A)
9o0r 11+1
(4)
(k)
(PaRpo) 2 for all h: 2 1. We prove by induction. For the base case,
(I)
PscPo IsaPo
V 14e-175 I+e-1/18 14e
PORPO
We the unit stepsize simplicity. Our results are valid for any sufficiently small step size.
8

---

Published in Transactions on Machine Learning Research (03/2026)
Evaluation results on offline RL training. For each model, we report the baseline performance before RL training. We then report RL training results that uses only negative samples and positive samples, respectively. Performance across validation and training dataset (LIMO) shown.
Table 1:
AMC23
AIME24
MATH500
Olympiads pass@1
LIMO
avg@16
avg016
pass@1
pass@1
Qwen2.5-14B-Inu
Baseline Negative Samples only
58.59
14.58
80.40
41.78
31.70
61.88
15.21
80.40
42.37
30.11
Positive Samples only
61.72
14.58
79.80
42.07
38.68
Qwen2.5-32B-Instruun
Baseline Negative Samples only
64.22
17.08
83.60
45.93
34.64
69.53
20.42
83.00
46.37
36.47
Positive Samples only
66.87
18.75
83.60
47.41
41.86
The above inequality holds true since Lemma (iv implies
2 log (1 + e log(1 + - + log(1 + -e-ee-111 1/16))
It remains show that Psaro Sscpo > (Pawo)) 2 implies Pearo [sapo > (Pea20 )2 for 1. By Lemma B.4(iii,
(k) (k)
(k)
(k+1) (k+1)
(
we know Psopo > IsoPo for all h 2 1. Thus, Lemma B.2 implies that
(k) (k)
(k++1) (k+1) PsaPa AsGPO
A(p B (Peco-
(V
c (Vrede
Pearo Searo
Using Lemma Bii(i,), we complete the induction by applying our induction hypothesis:
exp
fz1
(V
(k) (A) PscPo
Psapo IsaPo ))
(k)
(k)
(k) (1:) exp((22 Panpo)) (patoo ))
Poore Seora
This completes the proof.
Remark 3.4. Theorem 3.3 presents one of the first theoretical analyses of GRPO with multiple samples and multi-step reasoning in the contert of LLM reasoning. The first part establishes that SGPO converges to the optimal policy. The second and third ports demonstrate that SGPO both accelerates the acquisition of partially correct reasoning sleps and preserves partial reasoning ability even when the final answer is incorrect. Importantly, the theorem provides per-iteration comparison of learning under different reward mechanisms an aspect ravely eramined in previous works. The provable improvement in learning the oplimal policy is also consistent with
reumerical findings. We plot the resulting learning curves of our rumerical simalation in Figure 2. The left panel shows the probubility of selecting the "good" action in the first step at ileralion k (i.e., P3Gr us. ( while the right panel shows the probability of learning the optimal policy (i.e., PsopoAlscpo us. Paspolopaa) The results align with the predictions of Theorem 3.3, demonstrating that the likelihood of learning the optimal policy under SGPO consistently exceeds that of GRPO across training.
(k)
(k)
(k) (k)
4 Experiments
We present the benefits of differentiating negative samples through experiments in both offline and online settings. Offline RL more computationally efficient, offering faster training, reduced memory consumption, and improved stability. In contrast, online RL provides greater flexibility and learning capacity, and has become the standard approach in large-scale reasoning models such DeepSeek-R1 (Guo et al., 2025a).

---

Published in Transactions on Machine Learning Research (03/2026)
Comparison of evolution of
Comparison of evolution of p q
0.55
0.5
0.85
0.75
0.7
0.5
0.4
GRPO SGPO
GRPO SGPO
0.55
10'
10'
10'
Iteration k
Iteration k
Figure Learning dynamics of GRPO and SGPO in the simplified setting.
4.1 Offline Training
For baselines, we consider strong models without further fine-tuned on math-specific SFT datasets, namely 1eam2..--4B-Instrree and
Prior work showed that a small set of carefully curated prompts significantly enhance the reasoning capability. Accordingly, we adopt the GAIR/LIMO dataset (Ye et al., 2025) as the training set, which has demonstrated strong potential for improving the reasoning performance of large-scale (32B) models in offline SFT. Evaluation is conducted on four standard math reasoning benchmarks: AIME24, AMC23, MATH500 (Hendrycks et al., 2021), and OlympiadBencil (He et al., 2024). Our aim to highlight the rich learning signal contained in aenegativv-samplle groups, showing that training exclusively on them can still yield performance gains. For benchmarks with fewer than 100 questions (AMC23, AIME24), we report avg016 results with a decoding temperature of 0.6 and Top_P P 0.95. For benchmarks with more than 100 questions, we report pass@1 results using greedy decoding. Here, pass@k is the percentage of prompts for which at least one of the k sampled responses is correct, while avg@k is the average percentage of corrected samples among the & samples. The maximum decoding length is set to 32768 Lokens.
We conduct all response generation and model updates using offline RL (Peters & Schaal, 2007) with the standard GRPO mechanism. Specifically, the model is updated with advantages estimated from the offline dataset (see, e.g., Peng et al., 2019; Li et al., 2024b). For each prompt, we sample six responses per group and identify all-negative-sample groups in which all responses yield incorrect answers. Within these groups, we apply the step-wise judge model to assign differentiated rewards to negative samples, which are then used for offline RL updates. The model is trained for three epochs with a learning rate of 2 x 10-6, As a rastive baseline, we also perform offline RL using only positive rollouts with correct answers. This parallel sctup enables a dircct comparison between Icarning from exclusively negative reasoning trajectories and from exclusively positive ones.
We conduct offline RL training to demonstrate that utilizing all-negative--smple groups can enhance the reasoning abilities of LLMs. For comparison, we also include positivo-only offline RL training. As shown in Table 1, SGPO with negative samples improves average performance and performs compelitively on most benchmarks, and in some cases even surpasses models trained solely on positive samples. In particular, in the 14B model experiment, training on negative samples yields improvements on four benchmarks relative to the positivo-sample bascline. These findings underscore the utility of negative samples, which should not be discarded in online GRPO training; see further comments in Section 4.4.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Evaluation results on online RL training. We refer to BASELINE as the performance the orig- inal model without RL finetuning. Overall is average performance across all the benchmarks. Note that the training dataset is AIME1997-2023. For Serpseek-R--Disil1----0 including (i) compatibility with more judge models and (ii) ablation the stability paramcters B and y
we report additional results,
Kaoyan passo1
GradeMath paaае1
MATH500
Olympiads passo1
CBMatb24
AIME25
AIME24
GaoKao arg016
ANC23 avg016
Overall avg
paaso1
avg016
avg016
avgē16
DeepSeek Baieck-RR--Distte Distill-Qween
BASELINE
50.25
41.43
87.00
49.93
73.75
40.62
32.92
80.22
89.53
62.85
GRPO
55.78
43.33
89.40
56.00
71.04
36.08
52.08
80.30
88.91
63.72
SGPO+o6-NINI
46.19
90.80
54.67
75.00
38.33
54.58
81.33
90.00
65.41
SGPO DasF9EEx-V3-0324
51.77
47.17
55.11
77.29
40.42
56.87
82.28
90.83
66.19
(GPO+QQMx3323555
56.78
46.67
92.00
54.67
73.33
37.92
35.63
81.17
90.63
65.42
52.26
45.24
92.00
53.78
75.00
35,
56.46
82.28
91.88
64.91
SGPO+QvQ-325 w/o (8.n)
58.29
42.38
90.20
55.11
74.58
38.69
53.63
81.24
88.75
65.08
DeepSeck- Ht -iti---iillll--aaaaaaa
BASELINE
29.15
23.81
77.40
41.48
61.46
27. .22
42.29
72.78
87 87.97
51.58
GRPO
33.668
28.33
84.00
46.32
57.08
28.33
42.08
68.99
86.72
53.06
CLAUE-3
39.70
29.05
83.60
48.44
58.96
24.58
39.37
71.32
89.06
53.81
Qwen2.55 I4B-rurtuuctt
BASELINE
37.69
49.52
80.40
41.78
21.88
13.13
14.58
41.14
58.59
39.85
GRPO
43.
47.14
80.20
43.11
21.88
13.13
13.33
39. 16
59.84
40.11
-0P0-00--1XI-0416
38.09
53.33
81.00
44.00
22.92
16.67
14.17
39.00
59.22
41.00
BASELINE
45.73
53.81
83.60
45.93
26.87
12.29
17.08
41.15
641.22
43.74
GRPO
48.24 24
52.86
83.20
45.93
22.50
12.08
21.67
45.73
67.34
44.39
CGPO+04-NINI-O00
48.24
53.81
83.00
46.81
29.79
14.58
19.58
45.09
69.53
45.06
QwQ-32B
BASELINE
61.32
62.38
94.60
68,71
89.39
68.54
77.71
86.88
97.03
78.84
GRPO
71.36
63.81
94.60
69.48
88.75
64.38
75.83
87.1
97.03
79.15
SGPO+ DesPSEx--V3-037
64.76
95.00
70.22
66.46
78.33
87.11
97.97
80.17
4.2 Online Training
For baselines, we consider applying Oren2.5-14B-Instzree Qwen2.5-32B- and DepSSea--RI-Dssill--lee
5-222.5-33--20 --2..-IssttIItuutt..... QwQ-32B,
Online GRPO training is imple- mented using the verl framework (Sheng et al., 2025). For the step-wise judge model, we adopt diverse set of LLMs, ranging from closed-source models with strong reasoning capabilities (o4-mini, Claude3.7) 7) to open-source models that are more accessible to the community, including DeepSeek-V3-0322 100--235B-222222 and QwQ-32B.
DepSSel--RI-Dssiil--eer
Compared to oflline RL, online RL yields larger improvements a model's reasoning capabilities. Since our baselines already include strong distillation models, some benchmarks used in offline evaluation are nearing saturation. To provide better asscssment, expand our evaluation suite beyond AMC23, AIME24, MATH500, and OlympiadBench by inchuding AIME25, GradeSchool (Ye et al., 2025), CHMath24, Kaoyan, and Gaokao. Specifically, CHMath24 is the benchmark from the 2024 Chinese High School Mathematics League Competition, Gaokao from China's 2024 National College Entrance Examination, Kaoyan from the Chinese Graduate School Entrance Examinations, and GradeSchool targets elementary-level mathematical reasoning. Among these, CHMath24 and Gaokao each contain fewer than 100 questions, for which we apply the semperature-based dccoding for evaluation.
For GRPO training, use the AIME collections from 1997 to 2023 provided in DeepScaler (Luo et al., 2025b), training for 12 epochs. All training questions are in English, while evaluation benchmarks include multilingual questions. Notably, negative samples learned during training generalize well to out-of-domain mathematical reasoning lasks. SGPO training follows the same setup. With batch-simullaneons processing, judge model calls take 90 seconds per batch of negatives, adding 10% wall-clock time relative to rollout and update. Step-wise supervision is applied only all-negative-sample groups during the first three epochs, we expect this duration suffice for the model to internalize corrective signals; beyond this point, unresolved examples more indicative of model capacity limits than learnability. Accordingly, the end-to-end wall-
11

---

Published in Transactions on Machine Learning Research (03/2026)
Table 3: Evaluation results are reported for DeepSel--I--Dstill--eerr across four independent runs. First column indicates judge models and its corresponding reward stability setup.
Xaoyan passot
GradeMath passe1
Clyupiads
paxa01
paxad11
Avg316
Avg218
avg816
wope16
wvge116
Avg
DeepSeck- R1 Mai-Distiil--ee Qwen 7B-S7BB---R SGPO
-Qven3-2288
22388-2228
2.10
-QuQ-326
53.80 + 1.96
44.88
+ 1.36
01.15 + 0.84
63.71 + 0.74
74.33 + 1:29
37.00 + 1:23
54.76 + 1.87
$1.31 + 0.53
90.05 + 1.23
64.64 0.41
-QxQ-320 (8.r)
56.14 2.76
44.55
1 241
90.10
1 0.66
53.64
1,08
73.80
1.3 38.70 1 1.80
53.63
1 2.15
8121
0.50
68.85
0.81
61.52
1 0.57
Mean and standard devistion of performance across different judge setup over multiple runs
Parformans standard deviation
2.5
80
Mean performance
1.5
60
50
0.5
Kaoyan
GradeMath
MATH500
Olympiads
CHM4m124
AIME25
AIME24
GaoKao
AMC23
DvO-320 328
DvO 2.0-220-00
CaV--225 Jezng!
Figure 3: Mean and standard deviation multiple across different judge model setups.
Table 4: Evaluation results are reported for DeepSee--R--isttlll judge model with and without majority voting.
as base model and QwQ-32B
Kacyan paco01
GradeMath pare01
MATHE00
Olyspiads pacoo1
CHMath224
AINE2E
AIME24 avg016
GaoKao avg816
Overall
passe1
avg016
avgl16
avg216
avg
+QH0-320
33.89
74.33
+QH0-320 with voting 36 66
1.66
46.24
2.16
91.35 à 0.50
53.82 0.9
74.19 +: 0.92
$7.35 + 2.25
55.81 0.90
82.12 + 0.48
90.53 =: 1.07
63.23 : 0.18
clock overhead is upper bounded by approxinately 10% 3/12 2.5%, and in practice this overhead be offset by a small reduction of time needed to reach a target performance. For all models, rollout length is fixed at 8192 tokens and group size at 8. Models less than 8B are trained on 8 H100, 14B models on H100, and 32B models on 32 H200. We adopt the default KL cocfficient and lcarning rate from the verl training script (Sheng et. al., 2025), and use the LIMO evaluation scripl (Ye et al., 2025), both of which are standard practices in the community. That being said, these benefits are not uniform across every benchmark. When negative samples are short, highly noisy, dominated by carly derailments, step-wise judging provides less actionable signal and can even introduce additional variance through judge noise, SGPO may match or occasionally underperform GRPO or the baseline on specific tasks. This pattern is consistent with our empirical results and motivates a more nuanced view: SGPO is most effective when the model's failures retain informative intermediate structure (c.g., truncated near-correct trajectories or localized errors), rather than unstructured breakdowns.
A key insight from Table 2 is that stronger models generate higher-quality negative samples, which aid learning. As model capability improves, does the informativeness of its mistakes. Negative samples fall into two categories: (i) correct reasoning trajectories truncated by output length limits, and (ii) trajectories containing logical errors. The first type remains highly valuable- yet discarded in GRPO since it preserves meaningful reasoning steps, motivating our step-wise judge model. The second type, though incorrect, still provides informative signals, especially when all samples fail on genuinely challenging problems. Notably, stronger distilled models average 6K tokens per response, compared to only 1K tokens for weaker base models, making truncated but informative negative samples more common in the stronger case. Likewise, their erroneous responses tend to be richer and more useful for step-level judgment.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5: Evaluation results are reported terms of pass@16 across benchmarks. The first two columns show the total number of questions and the number solved within 16 attempts, while the last two columns report the number of questions solved by SGPO but not by GRPO (SGPO) GRPP), and vice versa (GRPO),SPPO)
SGPO pass@16
GRPO pass@16
SGPO
GRPO
GRPO \ SGPO
AIME24
23/30
19/30
4
AIME25
21/30
21 21/30
Gaokao
70/79
68/79
2
AMC23
39/40
38/40
1
0
CHMath24
27/30
25/30
2
4.3 Other Ablation Studies
To assess reliability judge models, we evaluate our approach not only with strong closed-source reasoning models but also with publicly available models of weaker capacity: DeepSeek-V3, Qwen3- 233BB and QwQ-32B. As shown in Table 2 (best-tuned results) and Table 3 (multiple runs with weaker judges), performance remains stable, indicating that weaker judges do not significantly degrade outcomes. We attribute this reliability to two design choices: (i) first-step error identification with the reference answer. SGPO requires the judge only to verily each step against the reference, not to solve the problem, thereby reducing lask difficulty and avoiding the pitfalls of generic PRMs; (ii) reward stability parameters 3 and 7, which set the update inertia and reduce sensitivity to rewards from carlier failed rollouts. As confirmed by ablations, removing 3 and y increases variance and weakens performance. To improve verification, we incorporate Fraok--Hevy--ispiiie
strategy: multiple independent evaluations by the judge model, with the error position selected by majority voting. Using QwQ-32B the judge model, ReepSeek-R--Dssill--- as the base model, and four rollouts per judgment, we obscrved noticcable gains in consistency and stability (see Table 4). Figure: 3 visualizes the aggregated results under different judge modes from Tables 3 and 4.
Although avg016 measures average performance across roll- outs, pass016 reflects the ability solve new questions with multiple attempts. As shown in Table 5, SGPO's gains in pass@16 stem directly from leveraging negative samples. Learning only from solvable problems reinforces existing abil- ity, whereas all-negative-sample groups correspond to genuinely difficult questions where GRPO consistently fails. These are preciscly the cases where additional feedback can be most valu- able. By providing step-level signals, SGPO rewards near- misses by reinforcing correct reasoning up to the first error, penalizes carly failures by discouraging persistent error modes, and exposes blind spots by turning hard cases into informative training signals. In this regard, SGPO can provide additional benefits over GRPO, covering more hard problems and provid- ing sharper credit assignment, which translates to more reliable learning under realistic compute budgets.
DeepSeek-RI-DDistll-llaaam
0.70
SCPO
CAPO
0.65
0.50
0.45
0.49
Epochs
Figure 4: Evaluation results GRPO and SGPO. SGPO rewards end at epoch 3.
By leveraging richer early-stage signals from negative samples, SGPO can achieve competitive performance with GRPO, and in some cases better performance improved coverage hard problems. As shown in Figure 4, SGPO continues improving beyond epoch 5 by solving several additional hard training problems, whereas GRPO platcaus. This improvement stems from informative negative samples that help resolve previously unsolved problems as also shown in Table 5. In line with our theoretical finding that SGPO converges faster than GRPO, empirical metrics offer supporting evidence. Prior work on RLVR. entropy highlights its link to performance: Cui al. (2025) showed that lower policy entropy under correct signals correlates with stronger policies, while Agarwal al. (2025) demonstrated that directly minimizing entropy improve performance. As shown in Figure 5, SGPO reduces policy entropy more rapidly than GRPO,
13

---

Published in Transactions on Machine Learning Research (03/2026)
DeepSeek-R2 Diz--ietll---amm--D
Ow0 -2-
5003
0600
0.22
GAPe
5000
0.28
0.37
0.25
0.22
0.18
0.08
0.31
0.11
0.34
0.44
0.32
003
0.32
0.04
Epecy
Oponn
Figure Policy entropy levels during training for GRPO and SGPO across different base models.
indicating faster convergence toward deterministic RLVR behavior with higher rollout confidence. This matches our theoretical results, confirming that step-wise signals accelerate convergence.
4.4 Discussions
We highlight the motivation for evaluating both offline and online RL. In the offline setup, training uses only negative samples, allowing to directly test whether incorrect incomplete reasoning trajectories can improve performance. In the online setup, we simulate realistic GRPO training, where batches contain a random mix of positive and negative samples. This demonstrates that negative samples are not only effective in isolation but also remain valuable in practical settings with noisier, mixed data. While mixing positives and negatives introduces noise, simply discarding negative samples does not stabilize training; in several cases, the performance of GRPO drops below baseline, as the models overfit to problems they can solve.
This instability arises from limited out-of-domain generalization and catastrophic forgetting. Without ex- posure to challenging or partially correct reasoning, the model risks overfitting to easy cases, reinforcing shallow heuristics instead of developing robust problem-solving skills. The absence of diverse failure cases can also cause catastrophic forgetting, degrading performance on previously solvable tasks. Incorporating negative samples mitigates these issues, and SGPO shows stronger robustness on the Chinese OOD math benchmarks we evaluate. We emphasize that SGPO does not. guarantee uniform improvements on every benchmark: its gains depend on the structure of negative samples. SGPO is most beneficial when failures are due to truncated but largely correct trajectories or localized mistakes, whereas when negative samples fail very early or lack meaningful structure, the step-wise signal can be less informative and gains may diminish. These observations motivate further work on more stable training frameworks, including richer reward diversification mechanisms for handling negative samples and efficient RL methods beyond GRPO.
Benchmarking scope. Our goal to isolate the benefit of learning from all-neeative--ample groups in outcome-basod, verifibble-ewardd post-training with group-relative updates. We therefore benchmark SGPO primarily against GRPO under matched training pipelines (same base models, data, rollout budget, group size, KL. control, and optimizer settings), since SGPO is designed as drop-in modification of GRPO's credit assignment only in all-negative groups. This choice makes the comparison controlled: improvements can be attributed to step-wise wise differentiation of negative samples rather than to changes in the broader RL recipe.
A natural question is how SGPO compares to methods that rely on process reward models (PRMs) or other step-level scoring mechanisms. Such comparisons would indeed further strengthen the empirical picture, but they introduce additional moving parts (PRM training data/labels architecture, calibration, and inference overhead) that are orthogonal to main question: given an outcome- outcome--erifiable setting and GRPO-style updates, can we recover learning signal from all-negative groups by cheaply localizing the first error? In this work, we keep the benchmark focused on GRPO and its outcome- based setting, and view PRM-based pipelines as complementary directions for future evaluation, especially when reliable PRMs and their training recipes are available and standardized. That said, we acknowledge that broader benchmarking (o.g., PRM- based pipelines) is an important next step, but given the recency and centrality of GRPO in outcome- basod RLVR, controlled GRPO-centered comparison is sufficient to establish relevance for the target setting.
14

---

Published in Transactions on Machine Learning Research (03/2026)
5 Conclusion
We proposc simple and efficient framework that introduces response diversity within all-negativo-sample groups and prove, in simplified setting, that such diversification can accelerate the learning dynamic of GRPO. Empirically, SGPO improves average performance and improves coverage on harder problems, with the largest benefits in early and mid-training when all-negative groups are prevalent; gains are not uniform across benchmarks and depend on the extending theoretical results to broader multi-step reasoning lasks, applying response diversity to accelerate other RL methods, and designing lightweight, task-specific reward models that evaluate reasoning steps correctly even if they cannot solve the full problem.
of negative samples. Future works include
Broader Impacts
Although large reasoning models enable substantial social and economic benefits, growing litera- ture also highlights potential negative impacts, including environmental costs from training and inference, concentration of power and access, labor displacement, and downstream misuse (e.g., for manipulation or disinformation). Our work improves the post-training procedure for such models; as a result, may con- tribute to both beneficial applications and accelerated adoption, which can amplify these broader concerns. We emphasize that SGPO is technical contribution and does address these systemic risks directly. Responsible deployment should consider energy and computing reporting, access and governance practices, and safeguards against misuse. More broadly, we view rigorous assessment of societal impacts alongside technical progress as essential.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Acknowledgment
We sincerely approciate Buzz High Performance Computing (bttps://www.bzzzr) ittppp/.a.i..nfnuuiiaaaaa info@buzzhpc.aat ai) for providing computational resources and support for this work.
References
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Alt- man, S. Anadkat, et al. GPT-4 technical report. ArXio Preprint: 2303.08774, 2023.
A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In COLT, pp. 64-66, 2020.
S. Agarwal, Z. Zhang, Yuan, J. Han, and H. Peng. The unreasonable effectiveness of entropy minimization in LLM reasoning. ArXiv Preprint: 2505. 15134, 2025.
Ahmadian, C. Cremer, M. Gallé, M. Fadace, J. Kreutzer, O. Pietquin, A. Üstün, and Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In ACI, pp. 12248 122677, 2024.
D. Arora and A. Zanette. Training language models to reason efficiently. ArXiv Preprind: 2502.04463, 2025.
M. G. Azar, Z. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, pp. 4447 4455, 2024.
Y. Bai, S. Kadavath, S. Kundu, A. Askell, Kermion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McK- innon, et al. Constitutional AI: Harmlessness from Al leedback. ArXio Preprinl: 2212.08073, 2022.
M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadonski, Nyezyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In AAAL, pp. 17682 17690, 2024.
X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al. Deepseek LLM: Scaling open-source language models with longtermism. ArXio Preprint: 2401.0295, 2024.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry,
A. Askell, et al. Language models are few-shot learners. In NewrIPS, pp. 1877 1901, 2020.
H. Chen, G. He, L. Yuan, G. Cui, H. Su, and J. Zhu. Noise contrastive alignment of language models with explicit rewards. In NewrIPS, pp. 117784 117812, 2024a.
H. Chen, H. Zhao, H. Lam, D. Yao, and W. Tang. MallowsPO: Finc-tune your LLM with preference dispersions. In ICLR, 2025a. URL
net:/forummiiddddceeeeaaaa
P. Chen, X. Chon, W. Yin, and T. Lin. ComPO: Preference alignment via comparison oracles. ArXiv Preprint: 2505.053 05465, 2025b.
P. Chen, X. Li, X. Chen, and T. Lin. Reward-free alignment for conflicting objectives. arXiv preprint arXiv:2602.02244 2026a.
P. Chen, X. Li, Z. Li, W. Yin, X. Chen, and T. Lin. Exploration exploitation: Rethinking RLVR through clipping, entropy, and spurious reward. In ICLR, 2026b. URI. https://openrev/// wwwp::wwwnnnnnenneeeeeee sE8DCSJTzd.
Q. Chen, I.. Qin, J. Liu, D. Peng, J. Guan, P. Wang, M. Hu, Y. Zhou, T. Gao, and W. Che. Towards reasoning cra: A survey of long chain-of-thought for reasoning large language models. ArXiv Preprint: 2503.09567 2025c.
X. Chen, J. Xu, T. Liang, Z. He, J. Pang. D. Yu, L. Song, Q. Liu, M. Zhou, Z. Zhang, et al. Do not think that much for 2+3 the overthinking of ol-like LLMs. ArXio Preprint: 2412.21187, 2024b.
16

---

Published in Transactions on Machine Learning Research (03/2026)
J. Cheng and B. Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. Xrxiv Preprint: 2412.33171, 2024.
D. R. Chialvo and P. Bak. Learning from mistakes. Neuroscience, 90(4):1137-1145, 1999.
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 242240):1-113, 2023.
P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NewrIPS, pp. 4302-4310, 2017.
X. Chu, H. Huang, X. Zhang, F. Wei, and Y. Wang. GPG: A simple and strong reinforcement learning bascline for model reasoning. ArXiv Preprint: 020460 2025.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, Н. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, al. Training verifiers to solve math word problems. ArXiv Preprint: 2110.14168 2021.
G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. ArXiv Preprint: 2505.22617, 2025.
P. Dayan. Reinforcement comparison. In Connectionist Models, pp. 45 51. Elsevier, 1991.
B. Ding, C. Qin, L. Liu, Y. K. Chia, B. Li, S. Joty, and L. Bing. Is GPT-3 good data annotator? In ACL, pp. 11173 11195, 2023.
H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, Diao, J. Zhang, K. Shum, and T. Zhang. RAFT: Reward ranked fine-tuning for generative foundation model alignment. Transactions Machine Learning Research, 2023. URL https: /ttp://oopeeeeeeeeeee meee/forumiii frrmmiddmm7p502bbix
Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang, RLHF workflow: From reward modeling to online RLHFF Transactions on Machine Learning Research, 2024. URL https: /ttps://openreviext htt.://toorummiinaa1a
K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Model alignment as prospect theoretic optimization. In ICML, pp. 12634-12651, 2024.
G. Feng, B. Zhang, Y. Gu, H. Ye, D. He, and L. Wang. Towards revealing the mystery behind chain of thought: theoretical perspective. In NearIPS, pp. 70757-70798, 2023.
K. Gandhi, D. Lee, G. Grand, M. Liu, W. Cheng, A. Sharma, and N. D. Goodman. Stream of search (SOS): Learning to search in language. ArXiv Preprint: 2404.09683, 2024.
L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In ICML, pp. 10835 10866, 2023.
F. Gilardi, M. Alizadeh, and M. Kubli. ChaiGPT outperforms crowd workers for texl-annotation tasks. Proceedings of the National Academy of Sciences, :20(30)):230001122, 2023.
Z. Gou, 7. Shao, Y. Gong, Y. Shen, Y. Yang, M. Huang, N. Duan, and W. Chen. ToRA: A tool-integrated reasoning agent for mathematical problem solving. In ICLR, 2024. URL https: https://openreview.. net. forum?id=EpoTtjVoaa
Y. Gu, L. Dong, F. Wei, and M. Huang. MiniLLM: Knowledge distillation of large language models. In ICLR, 2024. URL
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. DeepSeck- Incentivizing reasoning capability in LLMs via reinforcement learning. ArXiv Preprinl: 2501.12948, 2025a.
J. Guo, Y. Wu, J. Qiu, K. Huang, X. Juan, L. Yang. and M. Wang. Temporal consistency for LLM reasoning process error ideetifi. Ar Preprint: 2503. 14495, 2025b.
17

---

Published in Transactions on Machine Learning Research (03/2026)
J. Guo, L. Yang, P. Chen, Q Xiao, Y. Wang, X. Juan, J. Qiu, K. Shen, and W. Wang. Genenv: Difficulty- aligned co-evolution between Ilm agents and environment simulators. or preprint or 2rx:22512. 196822 2025c.
S. Hao, Y. Gu, H. Ma, J. Hong, Z. Wang, D. Wang, and Z. Hu. Reasoning with language model is planning with world model. In EMNLP, pp. 8154 8173, 2023.
S. Hao, Y. Gu, H. Luo, T. Liu, X. Shao, X. Wang, S. Xie, H. Ma, A. Samavedhi, Q. Gao, Z. Wang, and Z. Hu. LLM reasoners: New ovaluation, library, and analysis of step-by-stop reasoning with large language models. In COLM, 2024a. URL https: http:///pprreiiereee nwt/eeetnuuu poo//forumiidbbyyffosss
S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, and Y. Tian. Training large language models to reason in a continuous latent space, ArXiv Preprint: 067666766, 2024b.
A. Havrilla, Y. Du, S. C. Raparthy, C. Nalmpantis, J. Dwivedi-Yu, M. Zhuravinskyi, E. Hambro, S. Sukhbaalar, and R. Raileanu. Teaching large language models reason with reinforcement learn- ing. ArXiv Preprint: 2408,04642, 2024.
C. Hc, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. OlympiadBench: A challenging benchmark for promoting AGI with Olympiad-level bilingual multimodal scientific problems. In ACL, pp. 3828 3850, 2024.
T. He, R. Mu, L. Liao, Y. Cao, M. Liu, and B. Qin. Good learners think their thinking: Generative PRM makes large reasoning model more efficient math learner. ArXiv Preprint: 2507.23317, 2025.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NewrIPS Datasets and Benchmarks Track, 2021. URL /ttps://openreviev. aeee/ttorrmmdd--yytt2uuo
J. Hong, N. Lee, and J. Thoree ORPO: Monolithic preference optimization without reference model. In EMNLP, pp. 11170-11189 2024.
A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. V-STar: Training verifiers for self-taught reasoners. In COLM, 2024. URL
wtw.:rrrwweeeeeeoouuununaaaae hett/foruumiddssttgB3Wdd
J. Huang and K. C-C. Chang. Towards reasoning in large language models: A survey. In ACL, 1049-1065, 2023.
K. Huang, J. Guo, Z. Li, X. Ji, J. Ge, W. Li, Y. Guo, T. Cai, H. Yuan, R. Wang, Y. Wu, M. Yin, S. Tang, Y. Huang, C. Jin, X. Chen, C. Zhang, and M. Wang. MATH-Perturle Benchmarking LLMs' math reasoning abilities against hard perturbations. ArXiv Preprint: 2502.06453, 2025.
S. Huang, Z. Ma, J. Du, C. Meng, W. Wang, and Z. Lin. Mirror-consistency: Harnessing inconsistency in majority voting, EMNLP, pp. 2408 2420, 2024.
Hwang, D. Kim, S. Kim, Ye, and M. Sco. Self-explore: Enhancing mathematical reasoning in language models with fine-grained rewards. In EMNLP, pp. 1444-1466, 2024.
A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. EI-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, al. OpenAl ol system card. ArXiv Preprint: 2412. 16720, 2024.
N. Jain, K. Han, A. Gu, W-D. Li, Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. BiveCodeBench: Holistic and contamination free evaluation of large language models for code. In ICLR, 2025. URL setps://openreee
net wttp//ormmmddcch..cccccc
M. Kang, S. Loe, J. Back, K. Kawaguchi, and S. J. Hwang. Knowledg--nngmented reasoning distillation for small language models in knowledge-intentt tasks. Tn NewrIPS, pp. 48573 48602, 2023.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Accurate credit assignment in RL for LLM mathemati- cal reasoning. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS224, 2024. URL Aetps://openrevien
T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal. Decomposed prompt- ing: A modular approach for solving complex tasks. In ICLR, 2023. URL https: https://openreviiie net/ forum?id= naazQjzaRy.
M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. ICLR, 2023. URL https://openrevieet net/forummidee htt.:/oorrumn...10o0UUIIIIII
A. K. Lampinen, I. Dasgupta, S. C. Y. Chan, H. R. Sheahan, A. Creswell, D. Kumaran, J. L. McClelland, and F. Hill. Language models, like humans, show content effects on reasoning tasks. PNAS Nezus, 3(7): pgae233, 2024.
Y. LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(():
1 62, 2022.
H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, et al. RLAIF vs. RLHF:: Scaling reinforcement learning from human feedback with Al feedback. In 1CML, 2024.
pp.
L. Lehnert, S. Sukhbaatar, D. Su, Q. Zheng, P. McVay, M. Rabbat, and Y. Tian. Beyond At: Better planning with transformers via search dynamics bootstrapping. In COLM, 2024. URL https: /ttps://openrevier: nnt/ forum? Perum?id=SGolii SCoVCoo...
Z. Li, H. Liu, D. Zhou, and T. Ma. Chain of thought empowers transformers to solve inherently serial problems. ICLR, 2024a. URL https://openrevior mott/tormm/dd-3ETTyyee
Z. Li, T. Xu, Y. Zhang, Z. Lin, Y. Yu, R. Sun, and Z-Q. Luo. ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In ICML, pp.
2024b.
Z. Li, C. Chen, Xu, Qin, J. Xiao, Z-Q. Luo, and R. Sun. Preserving diversity supervised fine-tuning of large language models. In ICLR, 2025. URL attps://ri htt.r//reee..etttrtooooooceeeere
H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, T. Sutskever, and K. Cobbe. Let's verify step by step. In ICLR, 2024. URL https://openreevier.. httpr/iiww.neefoum/mttiie v8L0pN6ED1.
T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu. Statistical rejection sampling improves preference optimization. In ICLR, 2024a. URL https: bettps://opeee http////pp/eeeeeeeeee/eeeeeeeeee wwtp:/teoooooodddddd//oooooe
T. Liu, Z. Qin, J. Wu, J. Shen, M. Khalman, R. Joshi, Y. Zhao, M. Saleh, S. Baumgartner, J. Liu, et al. LiPO: Listwise preference optimization through Barming-to-rante In NAACL, pp. To appear, 2025a.
Z. Liu, M. Lu, S. Zhang, B. Liu, H. Guo, Y. Yang, J. Blanchet, and Z. Wang. Provably mitigating overopti- mization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In NewrIPS, pp. 138663 138697, 2024b.
Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, Du, W. S. Lee, and M. Lin. Understanding R1-zero-like training: A critical perspective. ArXiv Preprint: 2503.20783, 2025b.
H. Luo, L. Shen, H. He, Y. Wang, S. Liu, W. Li, N. Tan, X. Cao, and D. Tao. ol-pruner: Length-harmonizing fine-tuning for ol-like reasoning pruning. ArXie Preprint: 2501.12570 2025a.
L. Luo, Y. Liu, R. Liu, Phatale, M. Guo, H. Lara, Y. Li, L. Shu, Y. Zhu, L. Meng, et al. Improve math- ematical reasoning in language models by automated process supervision. ArXio Preprint: 2406.06592, 2024.
19

---

Published in Transactions on Machine Learning Research (03/2026)
M. Luo; S. Tan, J. Wong,
X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo,
L. E. Li, R. A. Popa, and T. Stoica.
DeepScaleR: Surpassing ol-preview wtw.rraddoooooooiioooiiiiieeee.e
with a 1.5B model by scaling RL. DeepScaleR-
rttp:: notion site/
2025b. Notion Blog.
A. Madaan, K. Hermann, and A. Yazdanbakhsh. What makes chain-of-though. prompting ellective? a counterfactual study, In EMNLP, pp. 1448-154 1535, 2023.
J. Mei, Y. Gao, B. Dai, C. Szepesvari, and D. Schuurmans. Leveraging non-uniformity in first-order non- convex optimization. In ICML, pp. 7555-7564, 2021.
Y. Meng, M. Xia, and D. Chen. SimPO: Simple preference optimization with a reference-free reward. In NewrIPS, pp. 124198-124235, 2024.
W. Merrill and A. Sabharwal. The expressive power of transformers with chain of thought. In ICLR, 2024. URL https: attpp://ooet /ttprr/ee/e//eeeeeeeee/ee www.:/eeeooommmidijjioie....
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, pp. 17730-27744, 2022.
A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. ArXiv Preprint: 2402. 19228, 2024.
R. Y. Pang, W. Yuan, H. He, K. Cho, S. Sukhbaatar, and J. Weston. Tterative reasoning preference opti- mization. In NewrIPS, pp. 116617 116637, 2024.
R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference opti- mization. In ACL, pp. 4998-5017, 2024.
X. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weightet regression: Simple and scalable off-policy reinforcement learning. ArXi Preprint: 1910.00177, 2019.
J. Peters and S. Schaal. Reinforcement learning by rewire-weighted regression for operational space control.
In ICML, pp. 745 750, 2007.
R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In NewrIPS, pp. 13728-537448 2023.
R. Rafailov, J. Hejna, R. Park, and C. Finn. From $r8 to Sq *S: Your language model is secretly Q-function. In COLM, 2024. URL /ttps://opts
N. Razin, S. Malladi, A. Bhaskar, D. Chen, S. Arora, and B. Hanin. Unintentional unalignment: Likelihood
displacement in direct preference optimization. In ICLR, 2025. URL http:::/pppeereee..eeeeeeeee id=uaMSBJDnRv.
P. Roit, J. Ferret, L. Shani, R. Aharoni, G. Cideron, R. Dadashi, M. Geist, S. Girgin, L. Husscnot, O. Kellor, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. ACL, pp. 6252-6272, 2023.
J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust region policy optimization. In ICML, 2015.
pp.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. Xie Preprint: 1707.06347, 2017.
A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisonstein, R. Agarwal, A. Agarwal, Berant, and A. Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In ICLR, 2025. URL https: / //openreview httpe//r..e/ooomm//////////////.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RL incorrect synthetic data scales the efficiency of LLM math reasoning by eight-fold. In NeurIPS, pp. 18000-430330 2024.
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Soug, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language modcls. ArXiv Preprint: 2402.03300, 2024.
G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. HybridFlow: A flexible and efficient RLHF framework. In EuroSys, pp. 1279-1297. ACM, 2025.
B. Shi, M. Tang, K. R. Narasimhan, and S. Yao. Can language models solve Olympiad programming? In COLM, 2024. URL https://
Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou V. Panneershelvam, M. Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529 (7587))) 489, 2016.
A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. T. Parisi, A. Kumar, A. A. Alemi, A. Rizkowsky, A. Nova, B. Adlam, B. Bohnet, G. F. Elsayed, H. Sedghi, I. Mordatch, I. Simpson, Gur, J. Snoek, J. Pennington,
Hron, K. Kenealy, K. Swersky, K. Mahajan, L. A. Culp, L. Xiao, M. Bileschi, N. Constant, R. Novak, R. Liu, T. Warkentin, Y. Bansal, E. Dyer, B. Neyshabur, J. Sohl-Dickstein, and N. Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. Thansactions Machine Learning Research, 2024. ISSN 2835- Expert Certification.
8856. URL https: http:::://pereeeeeeeeeee/ooo
Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. ProgPrompt: Generating situated robot task plans using large language models. In ICRA, pp. 11523 11530. IEEE, 2023.
J. Skalse, N. H. R. Howe, D. Krasheninnikor, and D. Krueger. Defining and characterizing reward hacking. In NewrIPS, pp. 9460-9441, 2022.
C. V. Snell, J. Lee, K. Xu, and A. Kumar. Scaling LLM test-time compute optimally be more effec- tive than scaling parameters for reasoning. In ICLR, 2025. URL http::s//ppeervveet www.://tffffrmiidce. formm?id= 4FWAwZtd2n.
F. Song, B. Yu, M. Li, H. Yu, F. Huang, Y. Li, and H. Wang. Preference ranking optimization for human aligument. In AAAI, pp. 18990 18998, 2024.
S. Srivastava, A. PV, S. Menon, A. Sukumar, A. Philipose, S. Prince, S. Thomas, et al. Functional bench- marks for robust evaluation of reasoning performance, and the reasoning gap. ArXiv Preprint: 2402.19450, 2024.
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human foedback. In NeurIPS, pp. 3008-3021, 2020.
D. Su, S. Sukhbaatar, M. Rabbat, Y. Tian, and Q. Zheng. Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces. In ICLR, 2025. URL https://openreview wwwpe/eewwwweeeeeeoiiieeee net/forum? id=
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction, volume MIT Press, 1998.
F. Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, S. Ermon, C. Finm, and A. Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In ICML, pp. 17441-47474. 2024.
Y. Tang, Guo, Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. Pires, and B. Piot. Generalized preference optimization: A unified approach to offline alignment. In ICML, pp. 47725 47742, 2024.
21

---

Published in Transactions on Machine Learning Research (03/2026)
K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1.5: Scaling reinforcement learning with LLMs. XrX Preprint: 2501. 12259. 2025.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, al. Llama: Open and efficient foundation language models. ArXiv Preprint: 2302.1391 2023.
J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Croswell, G. Irving, and I. Higgins. Solving math word problems with process-and outcome-based feedback. ArXiv Preprinl: 2211.14275, 2022.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, T.. Jones, A. N. Gomez, F. Kaiser, and L. Polosukhin. Attention is all you need. In NewrIPS, pp. 6000 6010, 2017.
H. Wang, C. Qian, W. Zhong, X. Chen, J. Qiu, S. Huang, B. Jin, M. Wang, K-F. Wong, and H. Ji. OTC: Optimal tool calls via reinforcement learning. ArXiv Preprind: 2504. 14870, 2025.
P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. Math- Shepherdpp Verify and reinforce LLMs step-by-step without human annotations. In ACI, pp.
2024.
X. Wang, J. Wei, D. Schuurmans, Q V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self- consisteney improves chain of thought reasoning language models. In ICLR, 2023. URL https: //openreview.netee ierpareriee..ttttoouurr-i---eeee net//orrum/dd1111NNNMM
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Lc, and D. Zhou. Chain- of-thought prompting elicits reasoning in large language models. In NeurIPS, pp. 24824-2488 2022.
R. J. Williams. Simple statistical maddennt-foloowng algorithms for connectionist reinforcement learning. Machine Learning, 8:229 256, 1992.
Y. Wu, Z. Sun, S. Li, S. Welleck, and Y. Yang. Inference scaling laws: An empirical analysis of compute- optimal inference for LLM problem-solving. In ICLR, 2025. URL JEH..
V. Xiang, C. Snell, K. Gandhi, A. Albalak, A. Singh, C. Blagden, D. Phung, R. Rafailov, N. Lile, D. Mahan, et al. Towards system 2 reasoning in LLMs: Learning how to think with meta chain-of chain-of-though. ArXiv Preprint: 5501.04683 2025.
J. Xiao, Z. Li, X. Xie, E. Getzen, C. Fang, Q. Long, and W. J. Su. On the algorithmic bias aligning large language models with RLHF: Preference collapse and matching regularization. ArXio Preprint: 2405.16455, 2024.
Y. Xie, K. Kawaguchi, Y. Zhao, J. X. Zhao, M-Y. Kan, J. He, and M. Q. Xie. Self-evaluation guided beam search for reasoning. In NewrIPS, pp. 41618 -46100, 2023.
W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KI-constraint. In ICML, pp. 54715- 54754, 2024.
W. Xiong, J. Yao, Y. Xu, B. Pang, L. Wang, D. Sahoo, J. Li, N. Jiang, T. Zhang, C. Xiong, and H. Dong. A minimalist approach to LLM reasoning: From rejection sampling to reinforce. ArXiv Preprint: 2504. 11943, 2025a.
W. Xiong, W. Zhao, W. Yuan, O. Golovneva, T. Zhang, J. Weston, and S. Sukhbaatar. StepWiser: Stepwise generative judges for wiscr reasoning. ArXiv Preprint: 2508. 19229, 2025b.
H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In ICML, pp. 2024.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Y. Yang, D. Campbell, K. Huang, M. Wang, J. Cohen, and T. Webb. Emergent symbolic mechanisms support abstract reasoning in large language models. ArXiv Preprint: 20332, 2025.
Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380) 2018.
S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Grilfiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NewrIPS, pp. 11809 11822, 2023.
Y. Ye, Z. Huang, Y. Xiao, E. Chern, S. Xia, and P. Lin. LIMO: Less is more for reasoning. ArXiv Preprint: 2502.03387 2025.
F. Yu, A. Gao, and B. Wang. OVM, Outcome-supervised value models for planning in mathematical reasoning. In NAACL, pp. 858 875, 2024a.
F. Yu, L. Jiang, H. Kang, S. Hao, and L. Qin. Flow of reasoning: Efficient training of LLM policy with divergent thinking. In ICML, pp. To appear, 2025a.
L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. Kwok, Z. Li, Weller, and W. Liu. MetaMath: Bootstrap your own mathematical questions for large language models. In ICLR, 2024b. URL https: / //openreview net/forum? N-N
Q. Yu, Z. Zhang, R. Zhn, Y. Yuan, X. Zuo, Y. Yuc, T. Fan, G. Liu, L. Liu, X. Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. ArXiv Preprinl: 2503.14 1403,14476, 2025b.
H. Yuan, Z. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. RRHF: Rank responses to align language models with human feedback. In NewrIPS, pp. 10935-10950, 2023a.
L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, B. Shan, Z. Liu, J. Deng, H. Chen, R. Xie, Y. Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun. Advancing LLM reasoning generalists with preference trees. ICLR, 2025. URL / wwwperiwwwwwweeeeeeeeeeeee net/foorunI.-22o5TTVV.
Z. Yuan, H. Yuan, C. Li, G. Dong, K. Lu, C. Tan, C. Zhou, and J. Zhou. Scaling relationship on learning mathematical reasoning with large language models. ArXiv Preprint: 2308.01822 2023b.
X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen. MAmmoTH: Building math generalist models through hybrid instruction tuning. In ICLR, 2024. URL https://openrovie: net// formm?td=yLCIce7701.
E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman. STaR: self-taught reasoner bootstrapping reasoning with reasoning. In NewrIPS, pp. 15476 15488, 2022.
K. Zha, Z. Gao, M. Shen, Z-W. Hong, D. S. Boning, and D. Katabi. RL Tango: Reinforcing generator and verifier together for language reasoning. ArXiv Preprint: 2505.15034, 2025.
D. Zhang, S. Zhoubian, 7. Hu, Y. Yue, Y. Dong, and J. Tang. ReST-MCTS': LLM self-training via process reward guided tree search. In NewrIPS, pp. 64735 64772, 2024a.
K. Zhang, Y. Hong, J. Bao, H. Jiang, Y. Song, D. Hong, and H. Xiong. GVPO: Group variance policy optimization for large language model post-training. ArXiv Preprint: 2504.19599, 2025a.
L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The NewrIPS Workshop on System-2 Reasoning at Scale, 2024b. URL https: http//vpeeeeeeeeeeeeeeeeee net/forum? aaggoon8.......
Z. Zhang, C. Zheng, Y. Wu, Zhang, R. Lin, B. Yu, D. Liu, J. Zhou, and Lin. The lessons of developing process reward models in mathematical reasoning. ArXio Preprint: 2501 5501007301, 2025b.
23

---

Published in Transactions on Machine Learning Research (03/2026)
H. Zhao, G. I. Winata, A. Das, S-X. Zhang, D. Yao, W. Tang, and S. Sahu. RainbowPO: A unified framework for combining improvements in preference optimization. In TCLR, 2025. URL httpp::///oeeereeiie nnt/ forumiid-trKee5pIFV
Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. ArXw Preprint: 2305.10423 2023.
Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. V. Le, and E. H. Chi. Least-to-most prompting enables complex reasoning in large language models. In ICLR, 2023. URL https://openreviex. /tt/fooommmd/////700tttttt
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. F'ne-tuning language models from human preferences. ArXiv Preprint: 1909.08593, 2019.
24

---

Published in Transactions on Machine Learning Research (03/2026)
A Related Works
We comment on all related topics, including reasoning through chain-of-thought and its variants, test-time compute, direct preference alignment methods, reward models and reinforcement learning from Al feedback. For an overview of more reasoning models and methods, we refer to two recent surveys (Huang & Chang, 2023; Chen ct al., 2025c).
Chain-of-Thought and its variants. Chain-offthought (CoT) refers to as a broad class of methodk that generate an intermediate reasoning process before arriving at a final answer. These approaches either prompt LLMs (Wei et al., 2022; Khot et al., 2023; Zhou et al., 2023) or train LLMs to generate reasoning chains through supervised fine-tuning (SFT) (Yue et al., 2024; Yu et al., 2024b; Li et al., 2025) and/or RL (Wang et al., 2024; Shao et al., 2024; Havrilla et al., 2024; Yu et al., 2025a). While CoT has proven effective for certain tasks, its auto-regressive generation nature makes it challenging to mimic human reasoning on more complex problems (LeCun, 2022; Hao et al., 2023), which require planning and search. Recent efforts were devoted to equipping LLMs with tree search methods (Xie et al., 2023; Yao et al., 2023; Hao et al., 2024a) or training LLMs on search trajectorics (Lchnert et al., 2024; Gandhi et al., 2024; Su et al., 2025). Several other works have investigated why CoT is effective. For example, (Madaan al., 2023) used counterfactual prompting approach to examine the relative contributions of prompt elements, including symbols (digits, entities) and patterns (oquations). (Feng et al., 2023; Merrill & Sabharwal, 2024; Li et al., 2024a) analyzed CoT from the perspective of model expressivity, and (Feng et al., 2023) showed that employing CoT increases the effective depth of a transformer since the generated outputs are looped back to the input. This insight motivated the olain-of-continuous proposed in (Cheng & Van Durme, 2024).
paradigm (Hao et al., 2024b), and a related approach has been
Reasoning through test-time compute. OpenAT-ol (Jaech et al., 2024) is among the first large-scale applications of RL to reasoning, and achieved shate-of-the-art performance upon release. Following this trend, DeepSeck-R1 (Guo et al., 2025a) is the first open- weight model to match or exceed OpenAI-ol. Their real-world success stories have involved several simple yet novel techniques that enhance LLM reasoning through more test-time compute, including chain-of-thought (Wei et al., 2022), self-consistency (Wang et al., 2023), best-of-N sampling (Snell et al., 2025), process reward models (Lightman et al., 2024), exploration- exploitation mechanism (Chen et al., 2026b), Monte Carlo tree search (Silver et al., 2016; Hao et al., 2023), tree-of-thought (Yao et al., 2023), and recent works on preventing overthinking (Chen et al., 2024b; Team et al., 2025; Luo et al., 2025a; Arora& & Zanette, 2025) and compressing chain-of-thought (Hao et al., 2024b; Cheng & Van Durme, 2024). More specifically, chain-of- ofthhullf is reasoning approach where intermediate steps are explicitly written to make complex problem-solving processes more transparent and logical. Self- consistency suggests generating multiple final answers and returning the mode of an empirical distribution, euhancing lest-time performance when test-time verifiers are unavailable. Unfortunately, it is computation- ally expensive and effective only when answers can be clustered. Best-of-N sampling resolves this issue by sampling answers from the model and selecting the best at test time according to the scoring function; however, it is sensitive to the accuracy of test-time scoring functions (Gao et al., 2023). Process reward models offer fine-grained supervision of chain-of-thouuht. reasoning, but they might be vulnerable to reward hacking and introduce computation overhead. Monte Carlo tree search is a generic technique that allocates computational resources toward the most promising regions of the search space, and tree-of-thoughl and its extension (Besta et al., 2024; Gandhi et al., 2024) simplified this idea by exploring multiple reasoning paths in specific structure, allowing language models to select the most promising line of thought for complex problem-solving. Both length regularization and compressed chain-of-thoughl are developed to reduce in- ference costs for reasoning, which is crucial for the economic feasibility, user experience and environmental sustainability of LLMs. In addition several works have focused on specific reasoning tasks (Lampinen et al., 2024; Yang et al., 2025; Srivastava et al., 2024; Huang et al., 2025; 2024; Guo et al., 2025b; Gou et al., 2024; Wang et al., 2025; Guo et al., 2025c), demonstrating promising performance. The recent findings Xiong et al. (2025a) have shown that the REINFORCE-tyDE methods (including GRPO (Shao et al., 2024)) can not ef- feetively learn from all-nelative-sammll groups. Our work alleviates this issue by leveraging AT feodback to differentiate negative samples. We also provide a theoretical analysis through a stylized model, explaining why such diversification improves GRPO's learning dynamics.
25

---

Published in Transactions on Machine Learning Research (03/2026)
Direct preference alignment methods. These methods (e.g., DPO (Rafailov et al., 2023)) are simple and stable offline alternatives to online RLHF. Various DPO variants with other objectives have been proposed, including ranking ones beyond pairwise preference data (Dong et al., 2023; Yuan et al., 2023a; Song et al., 2024; Chen et al., 2024a; Liu et al., 2025a) and simple ones that do not rely on a reference model (Hong et al., 2024; Meng et al., 2024). Since DPO does not train a reward model, the limited size of human labels becomes bottleneck. To alleviate this limitation, subsequent works proposod to augment preference data using a trained SFT policy (Zhao et al., 2023) or a refined SFT policy with rejection sampling (Liu et al., 2024a). The DPO loss was recently rederived and extended to a token-level MDP view (Rafailov et al., 2024), where the state is the token prefix and the transition is deterministic - which has covered the fine-tuning of LLMs and more general RL problems (Azar et al., 2024). There are other DPO variants (Ethayarajh et al., 2024; Park et al., 2024; Xu el al., 2024; Tang el al., 2024; Meng et al., 2024; Chen el al., 2025a; Zhao et al., 2025; Chen et al., 2026a). For example, Ethayarajh et al. (2024) designed a DPO-style loss variant using a prospoct theory, Tang et al. (2024) optimized a general preference loss instead of the log-likclhhodd loss, and Meng et al. (2024) aligned the reward function in the preference optimization objective with the generation metric. Dong et al. (2024) and Xiong et al. (2024) proposed to generate human feedback in an online fashion to mitigate the distribution-shiit and pverpparameteiiiaionnn phenomenon. This improves DPO for complex reasoning tasks (Pang et al., 2024). Several other works focus on unintentional alignment of DPO and developing new methods (Pal et al., 2024; Tajwar et al., 2024; Liu et al., 2024b; Xiao et al., 2024; Yuan et al., 2025; Razin et al., 2025; Chen et al., 2025b). Among these works, Razin et al. (2025) proposed to measure the similarity between preferred and dispreferred responses using the centered hidden embedding similarity (CHES) score and showed that filtering out preference pairs with small CHES score improves DPO, while (Chen et al., 2025b) proposed to use comparison oracles, and showed that combining it with DPO effectively alleviated the issue of unintentional aligument.
Reward models. For the prompt x with a ground-truth response yx, we evaluate by implementing a regular expression match on the final answer (Hendrycks et al., 2021): y(x,y)= =1 if y matches y* on the fonal answer and y(x = otherwise. An outcome reward model (ORM) (Cobbe et al., 2021; Uesato et al., 2022) is trained for estimating y). In particular, we first choose x E DD and collect training samples (x,y Ty(+(xx,, rxx, y)). Then, we take yx) as input and train an ORM to predict r(x,y). This can be done using binary classification (Cobbe et al., 2021; Yu et al., 2024a), direct preference optimization (Hosscini et al., 2024) next-token prediction (Zhang et al., 2024b). Previous works also train LLMs on self-generated data using the ground-truth outcome reward model with either supervised fine-tuning (Singh et al., 2024; Yuan et al., 2023b; Zelikman et al., 2022) or online RL (Bi et al., 2024; Guo et al., 2025a). A process reward model (PRM) is trained to score an at S/2- = (x, a1...., ah-1) either using human annotations (Lightman et al., 2024) or the value functions based on L.I.Mgenerated data (Wang et al., 2024; Luo et al., 2024; Setlur et al., 2025); indeed, PRMs estimate either the likelihood of future success or the change in the likelihood of future success before and after taking a₁ In addition, PRMs were also developed to improve search methods (Snell et al., 2025; Wu et al., 2025), and to identify the "first pit" in incorrect reasoning trajectory to construct preference pairs for direct preference alignment (Hwang et al., 2024; Setlur et al., 2024). Related work such as VinePPO (Kazemnejad et al., 2024) refines process-level credit assignment in PPO without explicitly training a PRM and avoids a learned value network by using Monte Carlo rollouts from intermediate prefixes; this is complementary to our GRPOO-orienedd setting.
Reinforcement learning from AI feedback. Reinforcement learning from human feedback (RLHF) uses numan--referennce-llinng reward models to evaluate response quality (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022). A key barrier to scale RLHF is the need for high-quality human labels. Previous studies (Gilardi et al., 2023; Ding et al., 2023) have shown that modern LLMs exhibit. strong alignment with human judgments, suggesting that AT-generated labels can serve as a viable alternative. In this context, (Bai et al., 2022) was the first to explore RLAIF, jointly optimizing helpfulness and harmlessness using both human and Al-generated labels, and (Roit et al., 2023; Kwon et al., 2023; Lee et al., 2024) showed that LLMs can produce informative reward signals for RL post-training. Our approach can leverage AT feedback to introduce response diversity within all-negative-samll groups by assigning intermediate binary rewards to reasoning steps. Indeed, one identifies the proportion of correct steps in the reasoning trajectory and use it to compute a reward ri E 1).
26

---

Published in Transactions on Machine Learning Research (03/2026)
B
Missing Proofs
In this section, we present the derivations underlying the update rules in Scction 3.2. In particular, we derive geaFa( (0) and (Scro(0), and obtain the corresponding update forms in Eqs. (3) and (4).
To derive garro ()) and Gacro (0), we start by restating the score functions:
p
p
s(a1 = 2)
p
s(a1
2
x))
p
0
0
0
0
0
0
s(a2 = 1 2, 01 = 2)
s(a2 = 2 0, = 2)
Let S(y) = Σ s(an, I. a..11:11) denote the scorc-sum for trajectory y (a1, az). Under the restricted we have
trajectory space [11,), (2, (2,
1), =
S(2, 1)
-
S(2,2)
p
q
1
-4
q
and the trajectory probabilities are
P(1,1)=1-P) p, P(2, = p(1 44,,
P(2,2) = pq.
Recall G 2 and H 2. For a single prompt, our estimator averages over both samples and steps:
s GH
ΣΣ 1ml
(k) (k) ah Ir a1 Ak
GH Σ A()
where Aş is the within-group standardized advantage computed from the two rewards. For G= 2, whenever the two rewards are distinct, standardization yields Anigh = +1 and Alow = -1; set A, = A2 =0 Thus, for any ordered pair (,(2) (2)) with distinct rewards,
1; if the rewards are equal we
s(0) c () (2) -))
In GRPO, r(2,2) 2) and r(1, 1) =2(2 1) = 0. Thus, the only nonzero contributions come from mixed pairs ((2,2), (1, (1] and ((2,2), (2, 1)). Using independence of the two samples and summing both orderings,
GRPO.
Janro E[g(()) (P.22 (P1, S(1, -(1,))))))))((((11)))))
()( S(2,
With G = H = 2, GH 2. Substituting the probabilities and S(.) yields
2
p(p 1)q p(1 - p)q
(G)
pola 1) pg(1 9)
SGPO. In SGPO, rsGPo(2,2) = rsGPo = 2, and rsGro = 0. Therefore, all three unordered
mixed pairs contribute: ((2,2), 1)). ((2,2), (1,1)), and (), Summing both orderings, we have
Jscro(()) (0) 3 (P(2 1) - P(2, (1P () 5(1, PP(2, 1)P(1, ()(5(2 1)))
27

---

Published in Transactions on Machine Learning Research (03/2026)
Again, we have 2 GH for G H 2, and substituting gives
p(p 1) p(1 p)
(0)
p¹qla 1) p²q(11 9)
Now, we derive the update rules in Eq. (3) and the functions in Eq. (4) from the population gradients. Let 01++1) ((k) + n where y is either jsCPO or SCRPO- For notational brevity, define the step-1 logits (01,02)= = (02,1,02, 02,2) and step-2 logits (0g. 04) 2,,, l2, In addition, we have
Update of p. Lot 91.92 92 denote the first two coordinates of g(0). Then, we have
p((t1)
600
(k) eton
2 e992
a and (k))
Using (8))
uot we can factor out e e to obtain
n000011))) p(k+1) (1-ppit))+p(())mooo==))
For SGPO, from (f) we have 91 ppppp 1) and 92 11p(1 p), hence 92 -2-91 91 p(11 - -p). For GRPO,
from (f) we have 91 píp 1)9 and 82 = 22(11 p)g. hence 82 91 p(1 - ppg.. Setting 1 gives the claimed update
(k+1)
(4) Pscpo scrO ( "Spo'
(k) exp(((u(scc)
PSGPO
1-PSGPO + PSPO sGro ( 2 saro
Panco GRPO ORRO PORPO
= (xp(fxx(PERPO ()RP)))
PGRPO
(a) 1- PORPO P CRPO (1 (6)
GRPO G
Taking log on both sides yields
f(((P) = log + p(1 p) log (1 + pep(2-pi) -p))
(p g) = log + p(1 p)q - log [1 - p pe
Update of An identical argument applies to q using the last two coordinates gs, 94 of g(0):
g c0 (++)) (1-q(t))
For SGPO, 9o = bpd - 1) and 94 = 4 - -),, 80 94 91 p²a(1 - 9)- For GRPO, ga = 22r - 1) and
9t = pp - 9), so 94 - 933 = pq - q). With 7 = this gives
ISGPO exp (app(fa(ps5a) PSSPP (s
(k) IGRPO = exp(f2e (GRPO)),
where by taking log both sides, have
(1) logq + p² p²q(1 9) log a qe 2o (11--))
(2) logg + pq(1 - 9) - llog - + gepa(1-9)
We then provide several technical lemmas that are important the proof of Thcorem 3.3. Indood, the first lemma summarizes the properties of particular functions related to the aforementioned functions fui, fui. f12 and f22 from Eq. (4).
28

---

Published in Transactions on Machine Learning Research (03/2026)
Lemma B.1. The following statements hold true,
(i) The fumction fi1 is strictly increasing on (0, 1).
(ii) The function bp(x)= := log(1 - + pe") is strictly increasing for any fixed E (0,
(iii) The function fu1 is strictly increasing in either p for any fized or for any fixed p on (0, 1).
(in) The function ((xx) == log(1 + ) strietly concave on (-50,0) 0).
Proof. First of all, we have
1+(1-2p)p(1-) 1-p) (p)
p1l pll-p-peR(-pp) 1p(1 per(linn 0.
Thus, the function fn is strictly increasing on (0. 1). Furthermore, we have
0<p<1
[1(2) =
pc
1 1-p 1-ptpe
0.
1-p+pe-
Thus, the function (x) is strictly increasing. Morcover, have
(p.4)
1+ (pppp))))
>
4p(1 poll-ptpeptt myj > 0,
Op
p(1 pl--pt 21)
a (n-
p1
0.
dg
1-p+petr(1-))
Thus, the function fai is strictly increasing in either for any fixed g for any fixed p (0, 1).
Finally, we have
("(x) = (aete* -1)0"
2oe +20=*
Since = ex E (0, 1) for < O, we have
(ue" - Ea 1)u (e - 1 1I) -u 0.
Thus, (x(x) for all x which shows that tp is strictly concave on (-o0,))
The second lemma presents an inequality which plays a key role in the proof of Theorem 3.3.
Lemma B.2. We define the auniliary functions follows,
A(x) = 1 ( - 1) e ==((==)) B(x,8) v) = 1+ (1 -1) e e-
C(z)=1+ =1 ( 1) -
Then, we have 2 > A(x) v) for all and y satisfying 1/2 1.
Proof. We consider the lower and upper bound of - when 0:
u +
- 1- 2
Since 1/x 1, 1/y - and Iry - 1 are all positive, we have
A(x) < 1 1-r (1 x=( 2) + -11=))
1 11 -1- x(1-a)²
x
2
2
() < + 1- (1 (1- x2-22y(1 - y) + 20
x² - -y) + x² 21
y
2
C(=) 2 1+ 1-2 11 - 21 - 2) +
(1==))
2 (1-
= 1 -=2(1 - ==2² 2
(1-2)
(1-=)*
29

---

Published in Transactions on Machine Learning Research (03/2026)
Set 2² = zy, the original statement is equivalent to (zC(z))3 (zA(z)) v)). Using the above upper
and lower bound, it suffices to show Ci(vry)² > A (2) (z, v) where
A1 = x=1 + -- /2, B1 (z, = 1 - y² y)² + x x²y²(1 32,
(=()==1 (1-))= +
2 20 2))/
By Lemma B.3, this is indeed true. This completes the proof.
Lemma B.3. Define functions
A1(x) = x01 - +=1 -1 12y(1 - -1) + x y2y2 12
=1
2)² + -1-1))1/ - -
-)=////00 2)*/66
Them, Ci(vTy) (r for all 1/2 < < <
Proof. Let x = u² and v², then > u u>v>>>> 1/v2 and == we. We next show the desired inequality holds on a larger region, i.c., 1 > u > v >> 2/3. On this larger region, we have the reparametcrization as follows:
2s+3 35431
V 2r+2
8, E (0, +00),
3r+30+3
or equivalently,
3(1-v)
Mu-vi 230-2)
1> u > U > 3/5
3u-2
It easy to see this defines a one-to-one correspondence from (u, sspace to sppce.. Thus, we aim prove the following function f is positive:
F(s,r) := C1 3 2r+2+43) Ai
((
2a+3 3s+3
B1
2++3 3s+33
2r+2o-3
3r 3r+33+3
By leveraging Sympy's symbolic engine, the function expands and simplifies to:
farr) F(s,r) ((+1) where [(,
Σ C00 x (08)
where e > 0 is universal constant, and single-variable polynomials c20(s),..., co()) 0 and A (1)) - (s) < O, for all >> (see Table 6 for details). Notice that from the table, only nontrivial parts are cs(s) > 0 and c2(s) > 0 because only these two contain negative coefficients. The positivity of ()()) is simple because there is only one term (s9) with negative coefficient and for all 8 > 0,
see the
19971456711054443055512264
+ 9984588877331664779977733 > 14418823100333222411111999999
To see this, simple estimation and AM-GM inequality yield
LHS > 1.9 1 9.6 1024g* 22/18.4 -0 > 2.7 1025 >RHS.
The positivity of c2(s) is more complicated because it has 4 negative terms 110 s9 s8 s7. However, we can
use similar idea, i.c., choosing pair of positive terms to bound negative term:
557910827865550007722888222222 + 571888995001111107777733999929 3
712733652864220052277777666
> 999540727744322775533322222
16220776855577122997777333334343 3348887756066642211112222222 9372603317788999908888889222
46088111905004422277777788833333 + 43622950855813311044992228 4|tp:///88........44400000033333
40900778553777774227776688
To see this, simple estimation and AM-GM inequality yield
LHS > 9.5 x 2 15 + 5.7 x 1023.5 > 2/541.5 x 20 10 > 4.6 10 > RHS;
LHS > 4.3 x 20 14 + 1.8 x 10 2v77.4 x 10a5, > 1.7 x 26 10 > RHS;
LHS > 1.6 x 2 + 3.5 10 2/5.6 x 20 > 4.7x 1025 RHS;
LHS > 4.6 x 12 +4. -0 3 2 19.7 x 10 > 8,8 x 10 77 RHS.
30

---

Published in Transactions on Machine Learning Research (03/2026)
In conclusion, we have all coefficient (s(s) positive except (s)), but it doesn't affect the positivity of f because A2 < O. This completes the proof.
31

---

Published in Transactions on Machine Learning Research (03/2026)
Table 6: Coefficient Lists of F(,,r)
Notation
Value
437675595662200444366333
2(2s
55019914985500523992244 4
57004119900031117333147
95661997881993223954410000 2372912168999379430017
1057096445222359114660006
C20
71871176709582275222222222
+ 1072082288115222000992222
$703002000402228877725555
$29901677337000225558888
2827999866005666227772222
151168270170893335500555
+
4913984991184555333366-
90996003279335662977
74248449483558322441
C19
859518
8395188710053322279992337711 5047771-220777777777777799 + 2122646177044842227770222
717567702233300222222229333 582166221099477008811133388 + 33441400002539005577000222
23080557/11800000944444444 106628801111662277771333333
+ 500769225560744144002224
2(2s
+
370225677220011112222000000
30739877899973112942200.
21591GGSS276666906555503333333
22606633447711002277311123
622117772627666361196222244
774933411072002255550000011111
C18
710091186117758880000999999
49704775766557777633195111144
20ND2387199335511142222222
188330092881011003310066655
452205553392477755722228888
522-1226200677773555228583
1355649725580119954111862
12(2s
+
590220221110002688553300222
317009900116113327703--66822
+
7797797887777955477449445---
1053187731744400433447777666
681099033316633333114444444444
13304114404330222200400229999943
443377111765552222526669777777
C17
1104411772229772555999819194
72556722442240994575500002222222
34533340770662271199411188833333
14033118771122333992557777999
15771999666199554455555
3(2s
+
WTS37241723777755555511100
5377700866667733522451112222
22401070118856774556667666144444
377847725501103321622522222
14448855226000117777766633333333
+
20089933881133277770444000000003
00064422280330003330001112222220
+
44744322226677440223377644444
C16
22216220177013366777555446888812
+
300S110881741199336444126666
17392357115072225554412557777777
7861
$45077228821106659666666661111
20T44466967200777777711155555555
15221077040122777700999661122173
943377911644788811225577888
4223342277299211187440088888
139833022552522250000010111
3041651 304163110338341914433666603
244588879953555500011222
32

---

Published in Transactions on Machine Learning Research (03/2026)
12(2s
+
37(43330066448884489990000000
+
01773622666377507219990000
661-499779900777799777722268
52902622224070063300019992222.
2215006666717211775887777046
+
71011076887744113373555444444444
TT455777771903344557999999534
BLI14677888002077777444006666699
537017997794000044322268777777
64490099114465277791122666633333
365254229-3311422255555555333333
C15
+
12200099840112277810111113333333
8556000087140445577777744444344
421887796832555550008888883
2798448222830554448116444777777
+
119444772005527482222300098
10526644962241175555524777723222
+
44911000911191118501179993
3(2s
+
37(45500766444000997777000000000
93377009000-40002279633300000
94658822314499777766611111233
+
601884116642332264440040222244
+
910082226662271113333//566466144
24160007188300188055557702222113
+
601177440060440003322226777777/
00060088887776622001118881141444
+
11820777221522266425577333336366
+
12223799755017777771111666666614
C14
+
33922665522000333442261111338883
30225335552577551177777911333333
L966117789922390511885110000
4113881999977622226667778133335
20033338777773225000776666666663
11884220022222222003000222221111
4
155539777420011255502222211821
6(2s
+
987755555514422277777777222
109211222342221005999188844433
+
227006000114490944223333662222
32500897220222100055591111000000
T8803322292225555667146222222222
13211122880199511110177775333333
C13
+
13227700772644455577704449999994
71993377778000777747777111666613
+
1112200-408966337744442277777999
+
11294482201122242223322733113313
9B921668881111100223335555555559
413729966537762244119733333333
199077668880031112533366677777
6856199832222317776661010000
179-0007000000033335443399944444
$112663377044311882222111333323
3(2s
332738814002222555666633000
66686606667722112115554422244444
+ 23071666375572226777666677066669
42223001666333557739977777077672
44254887766700554772552222222222
+
34039077774422033222111166699999
+
1417777788448886/48811222888885
1.611258734774404081115552222112
14311044180067777700000000055550
10972800044785770003355662222220
C12
+
62208879977777710048000677777772
50300115522999664473337770122129
+
32293778800111117990449999991123
182721236621122221577779922
7551446228770077777788887777766
32221222822003000379900122222299
+
9
33

---

Published in Transactions on Machine Learning Research (03/2026)
2(2s
+
69979966013337995771155600
716949944667220050022255000000
+
588011877548870001722119994444
044477966776653777111111555444
12071 72071112228110052266622666666610
37719975560222330555566662222223
+
35642218882966177700666000888888
46522099417222555000776662022220
522207776005222-4006777742222222
48599888224011777007777770000110
C11
35999444660992211044477755111119
+
+
22993333226444577444996551163119
+
01118002277711775000444777777770
474449817739999911334444449999
587761770822550027772217778881
2L0900514477775119221133335
(2s
3Y(133660688077775551100000
+ 15542299110057777222222111000100
159761100866446644000111222666
+
144189923411995555600002074444
93158117770037771111229999999444
+
3291 3221117700222008617111888888111
+
+
+
22946117799119999977777771811119
22566591112304499954444111111111
C10
+
11177777661166775555000000333331
27447000007154111000011011111113
1270460096527773000088884222222
+
715388882201447225550007766600
4499061110234411107777728---5551
266029998$7/07779977993333333332
72967707776611117070000000000210
+
44907797745148077555111222888888
1550722200731139099955552212225
2(2s
+
580651319935220222996441114444
74000022418880001111000366666113
32224990066633311473777222322333
+
18877887458777777777777777777771
+
7992700770077221154771177777774
+ 1 8611409906675007550000066660994
+
22201199229774612227755222222229
0g
74222256666777550004411111118800
88875888200771777033377766---8
11329923300770000663332777777764
+
+
+
(0665229992233733327444400000000
352977798860777777615357778999
(123306008800777777477777722244
144722255007663944477762222282
209245-7203372222255660089
34

---

Published in Transactions on Machine Learning Research (03/2026)
3(2s
+
737777066557774496222222000000
1148412288-331033114333377000000
24044457766177885555666611144444
01943444000223337744522777777779
55211225222293311122377772222222
12201700122244400332555447422223
+
20T72217726377770886777777766663
c8
1001-000002244000017799977777733
+
LT122550112207775444411117700000
520221119460241155777666777222
2ASSUULL887711122200447225555555
47741199888100622007765566666400
27017376622611333122296776111110
16543800007766233322116666333333
99998886100811550336611191000010
33941186880003117777777777776
13567180009222003227440000000000
43955000005000222337777777779
+
718003398222335-229999992222
12(2s
+
371221646442/02244455500000
+
6108667706372228774338800000
277000707330011126633444444444
+
7106448887-45544444003330000000
33910011771054660008111666666
+
22227771677700222222222007111333
24461887761333117770096666616112
88722994880055333777111999999999
33970001182222330445221222222293
50953220000277030022000001100000
36692284483000777507779998333333
+
22277077877777885555533222888882
+
44950022550133773334522243333333
13343811197311770555543311222129
85779977642444411226666077777111
48322817662233227114666611179199
23001888775502211199333333114113
15322211305500111999111777881111
64221220552266107723114488888888
212377407722209933644443333222
53940000466113311663333355555
11220888422114886662222222222333
135700992334422022717777777
3(2s
+
12759681144112799227333400
1771212266477732288772266000011
+
11770072800311222224444777777771
10IS510000066222255994447776---
1995588910.900077799777722266663
+
04999277986111334425999955556060
+
22 22119111877701512244977777722311
+ 3TT28655447550651112217722222222
C6
+
+
4133331199236666/77751112222222
18 18119460651220333111777677777277
10000724455241177444466600000060
60 602215449600992286144488001111
3US617180099044533227772222222
12344336611800224444995533333333
10228000441133350077700061118
231
35

---

Published in Transactions on Machine Learning Research (03/2026)
6(2s +
37(08661153369900033773300000
+
263399633754222000111440000
3T3-440018225533222222222222
+
94221466662211499344770003333334
22261211800166777199237777777666
+
1191 00011997770005530000646555577720
461
40112266220018811188222211133133
22372355643311111198844466333332
32266111001170022733334441001113
380-5583222022700555007777000003
8941183446000S117777777777744444
C5
$7778550067757399777777166677777
+
29778841998299111519998887111111
22666551229865311338222222222222
35598007765122277999999441111199
LD611224411777277666577792211111
553119227744705522244466664444
22477/881144233330007758889999
7D07120811332233330222999233
147788866612222110044446666
(166339977876634211666)
3(2s
37(55887210500003511666600000
+
I70000900012267778000000000000
33215522570022011115244441122222
+
16611224022947773322411799998
+
8822611984742222800222227766---4
+
+
200-4771177460003355077777744444
+
81121188877111777777777000111113
L1592222881122--0000777222222223
11073389076622222277111777771151
01222332220000111880000111112223
92663318887755500777411144440119
C4
77661966000911460001333161111112
(U566221122000666332447775444441
60099922666022217711555555522222
221922779650006111677687718...11
3300552773555225561107777777997
4112082216611550033000222766666
23116666277499779119992221222222
0B55-2772644119790444466333333
+
231132122011551033116664999999
73118778030443334447770111111144
1553372239412667744000038888
6(2s
206181199033666331186666600000
31162133199999222611111166333333
31133666118552220776000777002003
222297977641133332226622222294
1222256117399111881999888888811
+
+
+
67783411222225677444444111333332
71166001222457700099337777777776
24991 44184488866991177777222992888813
29945002200044774442237777771111
11556222993229114441111111111111
61
7145779977744433447117788222229
+
20010220857-50000003331121111110
LLLI1722700220-3555001116700002
+
32200788411711137777779993333303
VDT73300713663330000000777244
14118822110330022411119995555
96445588777314443712222288
12550221065333301555500000000000
6622883322110009994403333322222
113333790
36

---

Published in Transactions on Machine Learning Research (03/2026)
(2s
+
37(65166122211555002222220000000
+
21233012870500992244443300333
388171682991550455155544442
+
$4122888222344000311445770000049
2511411876627344777444444111111
#455224777122333321111111000
24530577007722111008137777772222
10444334422775111220000222272223
25002270000011334222227777777775
60077331666400333333777777222223
38277UU887777007992111066222244
40556680011000055444400005555553
499777622877799007777666662222
4B758866655007228336666666116010
$572550006GG22555000222255566669
+
98791002266625550077200882222222
1622779665332712299971116633
40062200660064332777777778833333
718998880255510177777776618888
7D27559622043200029977771---
995540777703332277333332444
$872208811777677790081182229
6099033799330777012227778899
38550770077112233335666666
2TT.T//2522220001177777075111133 +57780005504111177777775522222
354883375500022242211999222
138295085881311704499882
3206021409994391224555
10806881374138833677)))
22
()
01377661444555555770000
+
42716345557053366600088...
643344477111331170100055
62477666333772255333900000
49916766666229929916666666
2881229337752284442111154--44494
1036074222442222518811111122""""
3711932000688125577677712244,1
+
1118772666707777777777777774477
2880177022-111267773386664444444
+
633322911445770044442996777--774
221330100122001188/8775555333322
+
022262225111778881000668772227-4
+
8$21899960019944533000000666
C1
4229771667331550000001133333333
+
$19922778881/3355107711111001112
+
19001472241100011123353433333444
11440031902464442222222222299999
2413866779000077766440550050
22371888080000003227555522222211
212239111064555770995511333787
923164001159741111403416011111
188311889722966449733333834 387700619706111155901369
665566022419222155555600000
5²
11528166720887799922677777711111
$
55019977222001999:1333317117
6996844722411223110000627777271
+
7417377281801165011222228242
Co
8816662247103355566233112222
+
9670330977997717710022222222
018047731161446663555999990000
+
25703664447945555534333360668
4411452661141122600077777777
67779460033222884449991
95524992788899225556772
280305513911982215777222
+ 675662995331333355557766
130659999667125564706522
194895630566999554999
210830545504555509088
14659444800513998888
40413871702271776)
37

---

Published in Transactions on Machine Learning Research (03/2026)
+ TUU55512222220070000444444422559
+ 0F--5388777534422000022222336633
+ 0U003306552222000000002222222223
2
+ 012.6227122211000000110007777710
+ T2272//0111110007770000000222220
+
+ 41115222877220007770022222222221
+ FF555978662204411102928882828899

---

Published in Transactions on Machine Learning Research (03/2026)
+ 22726651177220777116111111167222
+ 94993339222240133377000072211111
+ 44425111106666333337700066666666
The last lemma presents some properties for the population-level SGPO and GRPO dynamics. Lemma B.4. Under the assumptions from Theorem 3.3, the following statements hold true,
(i) Psapo: Isgog, Paspo: lappo E (0,1) for all k 2. 0.
(k) (k) (k) (k)
(ti) Psapg: 430po: PaRPO 9apPo
(k) (k) (k] (k)
strictly increasing in and lie in (2,1) 1) for all 1.
(iii) Psopo 9soPo for all k 2 1.
Proof. We first rewrite the update rule in Eq. (3) as follows,
(k)
Psara
(k+1)
(k) PsGPO
where
A ScPO.p PeGPo
(k)
(k)
PscPo)
(k)
(k)
(8)
Parzo
Pepro
ISaPO
(k++1)
IsCPO
(k)
2ar0.9
where
A
(k) 8070.4
(Pacpo) "saro (1
(k)
) 1sapo)
(k)
Paora
DSPo.4
PGRPO
(k++)
PERPO
where
(k) A
C PaReo (1 PeRPo IaaPo
Parzo
(K)
Pegre"
(R)
G100.
4CRPO
(k+1)
4GRPO
(k)
GeF0.9
(R) where
GRPD,9 PaRPo (1 (eRPo).
(k)
(k)
(1e)
(k)
(k) First of all, the uniform initialization yields the desired result for k 0. Suppose PsaPo E (0, for some k 0. Then, we have
(k) (k)
(k)
PsaPa + Psupg sar.. Pecpue
s000, >
which implies Psapo E (0. By induction, we have PsaPa (0, 1) for all & 2 0. Similarly, can show that (k) (k) (k)
(k+1)
@sapo: PoRPo: TeaPo E (0, 1) for all #20 0.
Furthermore, we have (R)
> 0 since Paceo E (0.1). 1). This implies
(x+1)
1
1.
(K] Parpo
Pearo Ja
Papoo (M)
1- Pouso Paaso
(4) (6)
(k) Since PsePo = 3, we have PsaPo ( 1) for all l 2 1. Similarly, we can show that IsaPo Peaeo aawo are strictly increasing and lie in (2) 1).
(0) Finally, we have PsoPo 2
(R) OsaP0- Thus, suffices to show that Psapa 2 (k)
(0)
Isapo implics Psapo
(R)
(A++1) 1)
Asapo
(k++1
for all
k 2 0. Indeed, Lemma (i) and PeePo 2 4saPo yield
(k+1) PagPo = exp( (peer)) )) 2 exp(S1 (2)))) = exp(log(g6)) + h (x)
(43apo (1 4ecpo 1))-
Raseo
Then, Lemma (ii and Psapa (0, 1) yield
exp(log(9ea) + h Geore
(4saro (1 ))) > exp(log 4sara + h ((pear))
(k)
geapo (1 )
4sara ))).
(k)
Roaro
39

---

Published in Transactions on Machine Learning Research (03/2026)
In addition, we have
Osapa exp(f12(PsaP) (saPo)) = exp(log 93aPo + (2)
(6)
(k)
((PSOP))
4sapa (1
4sapo ))).
400rd
(k+1)
(k+1)
Putting these pieces together yields Psapa 4scpo
40


Reviews:
[{"review_id": "3p5MLPJyCa", "reviewer": "Reviewer_yPcN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a more fine-grained reward system for reasoning RL training that has a higher probability to distinguish responses for harder prompts to avoid \"all-negative and equal\" scores. It conducts empirical experiments together with theoretical analysis to demonstrate the effectiveness of the proposed method. Here are some key pros and cons:\n\nPros\n* The method is relatively easy-to-implement and seems effective. In general, the method is quite intuitive and results are reasonable. The experimental results shown in this paper can be a reference for future studies.\n\nCons\n* The proposed method is an intuitive yet straightforward extension to existing methods. In fact, it is only applicable in certain conditions that I am hesitated to call it a systematic method. First, it is pretty intuitive to see that when \"all negative\" signals occur there are at least 2 options you can do: (1) improve your reward system so that it is doing a better job distinguishing bad examples, or (2) design a whole new algorithm system that works differently from the existing RL. The proposed method falls into the first category, which is very reasonable but the results then become expected, as long as the reward is reliable. So I feel the baseline here is not really GRPO but rather the reward system used in solving Math/STEM problems. Yet the design of reward system is a creative and complicated topic, and I feel the proposed method is not abstract enough to generalize to all reward methods. Second, there are many limitations to the method proposed. For example, it currently only improves on step-wise reasoning models that are trained on rewards computed purely on final answers. It also relies on stronger models for reference and correct judgements for reasoning-step correctness. And most importantly, it has the same issue of the original system if all samples fails at the same steps, which results in same scores again.\n\nOverall, I believe that the method is effective and I think it is great for other researchers to see this results. However, I feel the method is only proposed for and studied under a set of restrictions, so it might be better to carefully position the true contribution of this paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper contains plenty of experimental and theoretical results.", "audience_yn": "Yes", "audience_explanation": "See above", "requested_changes": "None", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pgI1La2o6V", "reviewer": "Reviewer_4mMY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\n\n* GRPO samples a group of trajectories to compute advantage. However often times early on during training, it is possible that none of the policy rollouts in a given group end with the correct state. When this happens in an RLVR training situation no gradient update takes place leading to a wasted step.\n* This paper (SGPO) proposes modification to GRPO where a new equation for computing advantages is used which allows for gradient updates even when the entire group consists of negative samples (ie. rollouts that result in the incorrect answer).\n* The key mechanism for SGPO is to use a judge model (aided by a ground-truth reference response) which examines the policy rollout if it is negative and assigns a score (called Reasoning Trajectory Score or RTS) to incorrect rollouts. This score is computed based on what percentage of the reasoning trace was correct (by finding the first error and considering all prior actions correct).\n  * The RTS can be computed by a separately trained model or by an existing LLM, in this paper they simply use existing LLMs\n  * When using an existing LLM the technique is performing a kind of distillation since it is using an existing trained LLM but it doesn't follow the traditional knowledge distillation formulation since the \"teacher\" model doesn't need to produce correct answers (therefore it's not really a \"teacher\" and really acts more like a \"judge\").\n* The main strengths of this paper & the method are that it finds a way to take advantage of all negative groups in GRPO which allows for sped up convergence and faster training. The authors provide a simplified setting experiment in Figure 1 which clearly shows the improved learning dynamics from SGPO in theory.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In general I believe that most of the authors claims are supported. Specifically that SGPO provides faster convergence compared to GRPO seems to be supported well by their various experiments. Additionally it is clear from the final benchmark numbers on various mathematics benchmarks that SGPO does not degrade performance. However in section 4 the authors also claim that SGPO can provide stronger performance than GRPO and I am not 100% convinced on this claim by the evidence. The overall performance of SGPO models seems to be quite similar to those trained with GRPO and based on the main results in Table 2 it seems to provide marginal if any improvements. I believe the authors are focusing on Table 5 to make their claim of \"stronger performance\" which focuses on pass@16 vs the pass@1 and avg@16 reported in Table 2.", "audience_yn": "Yes", "audience_explanation": "Yes this paper is of clear interest to the TMLR community. It proposes a new mechanism for computing advantage in RLVR training which is able to use groups of rollouts where all members are incorrect. This fits with the broader themes called for by TMLR papers and the findings are properly supported through both theory and experimentation.", "requested_changes": "In order to better support the claim that models trained with SGPO are \"stronger\" simply reporting pass@16 for a subset of benchmarks is insufficient. Looking at Table 2, the gains for 8B, 14B, and 32B models are all <1% in the 'Overall' metric. The authors should clarify if these gains are statistically significant or if the primary benefit is strictly convergence speed. If the primary benefit is convergence speed then the \"stronger\" claim should be removed from the manuscript.", "broader_impact_concerns": "Given the focus on more efficient training with faster convergence it would be interesting for the authors to compare the extra compute needed to run the judge model vs the compute saved by the faster convergence of SGPO. The authors mention a 10% wall-clock time overhead for the judge model. I would like to see a more explicit trade-off analysis: Does the accelerated convergence of SGPO reduce total training compute by _more_ than this 10% overhead? Currently, it is unclear if SGPO is more compute-efficient net-net. Ideally the small amount of compute needed for a judge model is offset by the improved convergence properties (though I recognize this doesn't take into account the cost of training the judge model).", "additional_comments": "I'm not as familiar with the literature for RLVR in LLMs and thus am unable to adequately judge the novelty of this contribution in its entirety. Based on a cursory understanding the paper does not incorrectly claim any novelty and the authors make clear the distinctions from prior work in their remarks, however due to my lack of knowledge about all prior work I cannot evaluate this for certain beyond my brief literature review.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ty40TB6FoE", "reviewer": "Reviewer_UtfL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper highlights the issue with certain modern GRPO methods, that gradients are zero when all reasoning trajectories in a group represent failures, and then proposes an alternative method that solves this issue. Specifically, the authors propose a “step-wise” approach to group-relative policy optimization, where the rewards for performance are not given solely based on the ultimate success or failure of a reasoning trajectory, but based on the number of correct intermediate reasoning steps taken by the model. To obtain an estimate of the correct number of reasoning steps, the authors propose using a second “judge” model to estimate the correct steps in each trajectory. The authors evaluate their method numerically, primarily by comparing with GRPO, and theoretically, by comparing their method and GRPO via a toy model. They show that their model tends to perform better than GRPO, by using information gained from partial correctness on a reasoning task.\n\n## Strengths\nI believe there are two primary strengths of the paper. First is its effort to include theoretical analysis. Including theory, even with a toy model, can provide complementary insights that one might miss with purely numerical work, so I appreciate the authors’ efforts on this front. Second, I think the solution that the authors propose is elegant in its simplicity.\n\n## Weaknesses\nIn my view, the primary weaknesses of the paper are (1) not benchmarking relative to existing process models; (2) not always quantifying statistical significance (or variance of measurements) in empirical results, and limited discussion of variability in the results; (3) some issues with the theoretical results, particularly a lack of derivations for some of the main equations in the lead up to the main theorem and a toy model that seems to miss some of the more important aspects of the algorithm being studied; (4) clarity of presentation. I will elaborate on these issues in the requested changes section.", "claims_and_evidence_yn": "No", "claims_explanation": "I do not believe the “consistent gains” claim made in the paper is sufficiently supported. This is because of variability in the results, and baseline choices.", "audience_yn": "Yes", "audience_explanation": "Given modern interest in reasoning, LLMs, and feedback during learning, the subject matter of this paper is definitely of interest to many in the community. For this reason, I have answered \"yes\" above. However, because of issues around baseline choices (mentioned below) I have some concerns that researchers may be less interested in the papers results despite the subject matter.", "requested_changes": "I list the requested changes below. I have added “**C**” at the end of changes that I view as critical, for my review, and “**S**” after ones that I view more as suggestions. I have also divided my review up into sections based on each of the three weaknesses mentioned in ‘strengths and weaknesses’, above, and a fourth section highlighting minor (mostly spelling and grammar) issues.\n\n**Weakness 1: benchmarking and comparison to other models**\n1. The big issue the paper aims to address is the lack of partial rewards for groups with no successful reasoning trajectories that one sees in SGPO. As the authors mention, there are already existing methods that aim to resolve this issue—e.g. process reward models. While, as discussed in remark 3.1, there seem to be differences between their approach and these other methods, it seems to me that the correct benchmark in the paper is not GRPO but, rather, these other methods that already aim to address the problem of lack of partial rewards. Additionally, GRPO is known to exhibit certain problems (e.g. bias towards long response lengths), as mentioned in “Understanding R1-Zero-Like Training: A Critical Perspective” by Liu et al. For these reasons, I would like to see a comparison between SGPO and existing process reward type methods—instead of GRPO—either demonstrating the utility of SGPO from a performance or computational efficiency perspective. For example, “VinePPO: Refining Credit Assignment in RL Training of LLMs”, by Kazemnejad et al. could be worth looking at as a benchmark. **C**\n2. [Remark 3.1, (i)]: it seems that the step-wise value that the paper studies could just be represented as a simple value function that assigns monotonically increasing value with each correct step, no? **C**\n\n**Weakness 2: statistical significance, and variability in results**\n1. Many of the results presented in the paper do not have measures of statistical significance—for example tables 1 and 2, and figures 2, and 3. Given the, at times, small differences in performance between models, it would be really nice to have some quantification of whether or not result differences are significant. Even a measure of standard deviation could help with this. **C**\n2. It does not seem that the authors’ model, SGPO, universally outperforms GRPO—or even the baseline—in terms of performance (see e.g. Table 2). One would have expected, at first glance, that having the extra information from all-negative samples should, at worst, not drop below the baselines that do not address the negative sample issue. However, there is not much discussion of this in the paper. It would be good to have some deeper discussion of when and why the authors’ approach does not outperform these alternatives.  **C**\n3. It is claimed that $\\beta$ and $\\gamma$ filter and smooth noisy signals and that removing them weakens performance. Three questions related to this: first, are you using filtering and smoothing in the usual statistical time-series sense? If so, how are they performing these operations? Second, it is not clear to me that removing $\\beta$ and $\\gama$ reduces performance or increases variance in a statistically significant fashion. It would be good to have a statistical test to confirm this. Third, when you say you “remove” these parameters, you mean setting them equal to $1$ and $0$, correct? **C**\n\n**Weakness 3: theoretical results**\n1. Importance sampling and clipping is not included in the model. This is clearly done to make analysis more simple, but, as far as I have seen, there is no description of the impact of these simplifications. It would be great to at least speculate about this a little bit. **S**\n2. The theoretical section focuses on proving that including information in learning for partially correct answers yields benefits over not including it. However, this seems somewhat intuitive. To me, the bigger theoretical question associated with the authors’ new method is how using other large models as judges—even when they may provide potentially incorrect insight on partial correctness—can lead to failures or successes in learning. Have the authors considered ways of investigating this question? **S**\n3. In the third line of equations on page 6 the authors get factors of $½$ multiplying each gradient; I have tried to rederive these and obtain different factors (namely $2$ for GRPO and $2$ and $4$ respectively for the top two and bottom two elements of the gradient for SGPO). Could the authors add a section deriving these gradient forms to the appendix please? **C**\n4. Would it be possible to add derivations for equations (3) and (4) in the appendix of the paper? **C**\n\n**Weakness 4: clarity of presentation**\n1. Regarding reference solutions: to clarify, must there be a reference solution for every question in the dataset on which the model is trained? I.e. SGPO requires training datasets with fully answered train of thought solutions? **C**\n2. [section 4.1]: What do “avg@16” and “pass@16” mean? It could be good to briefly describe these. **C**\n3. [appendix A]: I wonder if the “related works” section could be more logically organized—it seems like the text jumps back and forth between concepts a little bit. For example, chain of thought reasoning is referenced before the “chain of thought and its variants” section, which makes me wonder if the chain of thought section should come earlier. **C**\n4. Perhaps it is just me, but I am somewhat confused by the choice of “colouring” in the title. What is the rationale for this? **S**\n\n\n**Minor issues**\n1. [Intro line 8 and elsewhere]: There are many places in the paper where the authors add “the” where it is not needed. For example, line 8 of introduction: “As *the* generative AI applications…”\n2. [Intro line 9]: “conversational interfaces” is a very strange phrasing; suggest reword.\n3. [Intro line 9]: “increasingly powerful” => more powerful\n4. [Intro line 10]: what does “...positioning them as a key frontier in practice” mean? Suggest rephrasing this\n5. [end of page 1]: first mention of “group-relative” training. I suggest including a quick description of this\n6. [page 2, top of 3rd paragraph]: “The key insight is that many reasoning tasks possess a structure where step-level correctness can be explicitly defined.” The key insight where? In the above paragraph or in your own work?\n7. [first line, page 3]: distinguish => distinguished\n8. [section 2]: could be useful to start by mentioning the focus on fine-tuning.\n9. [3rd equation in section 2]: define “clip” operation\n10. [Equation 2]: Why use the sigmoid function? If RTS is zero, shouldn’t the reward be zero?\n11. [Example in Remark 2]: the example starting with “The twelve letters {A, B, C, D, E, F, G, H, I, J, K, L} are randomly grouped” seems redundant.\n12. [2nd line after first equation section 3.2]: $\\theta$ in the definition of the score should be a subscript, right?\n13. In some of the results tables it could be useful to present results as bar plots. The tables are quite dense and difficult to take in at times.\n14. [Table 5]: what does “SGPO\\GRPO”, and vice versa, mean in the table heading? Somewhat confusing.\n15. [page 23]: what do you mean by “given that the transition is deterministic”? Are deterministic transitions required for DPO?\n16. [page 23]: when you say “the specific loss using a prospect theory…”, what do you mean by “specific loss”? Do you mean the DPO loss?", "broader_impact_concerns": "In addition to potential societal benefits of large AI models, there is a growing body of literature observing certain, seemingly severe, negative impacts—including energy consumption, power/wealth concentration, job loss, and threats to democracy. For easily digestible overviews of such impacts, one could look at the Atlas of AI by Kate Crawford or The AI Con by Hanna and Bender. For academic articles one could start with works by the same authors. Given this growing literature, and given the focus of the article on improving large reasoning models, I was surprised to see no mention of broader impact concerns in the paper. I would like to see at least some brief discussion of broader impacts in the paper before I can recommend it for publication.", "additional_comments": "I have two meta-notes regarding my review. First, reasoning in large models is not my domain of expertise, so please take this into account when considering my review. Second, I have not verified the proof of the main theorem. The reason for this is that I have some immediate problems with other aspects of the paper (see main weaknesses) that I believe need to be resolved. If the authors are able to sufficiently address these other issues I will then be happy to check through the proof.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)