Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
Authors: Christos Thrampoulidis
Sadegh Mahdavi
Wenlong Deng
Venue: Accepted by TMLR ()

Abstract: We unify two seemingly distinct approaches to policy gradient optimization for the Pass@K objective in reinforcement learning with verifiable  rewards (RLVR): direct REINFORCE-style methods and advantage-shaping techniques that modify GRPO.  By reverse-engineering existing advantage-shaping algorithms, we reveal that they implicitly optimize surrogate rewards.  We specifically interpret practical ``hard-example upweighting'' modifications to GRPO as reward-level regularization. Conversely, starting from surrogate reward objectives, we provide a simple recipe for deriving both existing and new advantage-shaping methods.  
This perspective provides a  lens for RLVR  beyond our original motivation of Pass@K.

Published in Transactions on Machine Learning Research (03/2026)
Advantage Shaping as Surrogate Reward Maximization: Unifying PassØK Policy Gradients
Christos Thrampoulidis
cthrompo@ece. ubc.ee.ubc.... ca smahdavi@ece. ubc. cn dwenlong@ece ubc. ca
Sadegh Mahdavi
Wenlong Deng
Department of Electrical and Computer Engineering University of British Columbia
Reviewed on OpenReview: https: opwneeveew....e....mm net/ for/m? id- R1RABFUK8t
Abstract
We unify two scemingly distinct approaches to policy gradient optimization for the Pass@K objective in reinforcement learning with verifiable rewards (RLVR): direct REINFORCE- style methods and advantage-shaping techniques that modify GRPO. By reverse-engineering existing advantage-shaping algorithms, we reveal that they implicitly optimize surrogate rewards. We specifically interpret practical "hard-cxample upweighting" modifications to GRPO as reward-level regularization. Conversely, starting from surrogate reward objectives, we provide simple recipe for deriving both existing and new advantage-shaping methods. This perspective provides lens for RLVR beyond our original motivation of Pass@K.
Surrogate Reward
max F(Po)
This paper
w(p) F'()) p(1 - p)
Weighted Gradient
(か)
|4 -V_]
w()) A: p(1 P)
(n: P)
Prior work
Σ LE(N)
Advantage Scores
A; Ve logno
Figure Illsstaatoon of bidirectional framework mapping Advantage Scores (A,) of RLVR algorithms
with 0/1 and Pass@K rewards, such GRPO Shao et al. (024)) Pass@KGGRPOCCaa et al. (2025): Mahdavi et al. (2025), and Prioritized Sampling Team et al. (2025 to Surrogate Rewards (F(p) via Gradient
Reweighting (w()) of the average gradient updates of correct
(V+) and wrong (V-) model responses. The
framework provides a recipe for engineering new RLVR algorithms by choosing surrogate reward objective (or equivalently, reweighting function).
1 Introduction
When evaluating large language models on math and coding lasks, standard practice is to generate K independent solutions and check whether any succeeds; the so-called Pass@K metric (Kulal et al., 2019 Chen al. (2021) Yet most policy gradient methods, from REINFORCE (Williams) 1992) to more recent variants like RLOO (Kool et al. 2019 Ahmadian et al., (024) and GRPO (Shao al. 2024). optimize for single-attempt reward creating a mismatch. Should instead train what test??

---

Published in Transactions on Machine Learning Research (03/2026)
Recent work has approached this question from two seemingly distinct angles. Mahdavi et al. (2025): Walder & Karkhanis (2025) derive policy gradients that directly maximize Pass@K reward, yielding REINFORCE. style updates with reweighted advantages. Meanwhile, Chen et al. (2025) propose modifying GRPO through advantage shaping, that is, directly adjusting advantage scores to account for the Pass@K objective. Both
improve P performance, but their relationship remains unclear. Are these fundamenlally different techniques, or two views of the same underlying principle?
This work unifies these seemingly distinct perspectives, showing that advantage shaping and direct optimiza- tion are two sides of the same coin. The key is recognizing that different policy gradient methods can be understood as optimizing different surrogate rewards; see Figure | and Table L More broadly, we establish surrogate rewards as design mechanism for policy gradient optimization with verifiable rewards, providing a recipe for algorithm interpretation via reverse-engineering and development via naward-engineering
1.1 Problem Setup: Reinforcement Learning with Verifiable Rewards
We study reinforcement learning with verifiable rewards (RLVR) (Lambert et al. 2024 Shao et al. 2024 Guo
et al., 2025), framework for training Large Language Models (LLMs) on tasks with objectively verifiable outcomes (see (Zhan et al., 2025) for recent survey). Concretely, are given distribution P over problem-answer pairs (I., a), where might be math problem and its final numeric answer. The goal is to train an LLM to maximize the expected reward of finding correct answers to problems from P:
maxo
E(r,a)>P (By- r(y. a)]
(1)
We abstract the LLM conditional distribution (response) promptt) parameterized by 8: Given prompt v, the model generates response y (a sequence of tokens) by autoregressive sampling. The reward function rly, a) assesses whether the model response y agrees with the reference answer a. If it does, the response is correct; otherwise it is wrong. RLVR assumes the reward can be verified externally. Note the nested expectation: first over problem-answer pairs (x, a) ~ P, then over model responses ~ =T((1x)) per problem.
1.1.1 The 0/1 Reward
The most popular reward function in RLVR is the binary 0/1 reward:
{1 if agrees with otherwise
(y) = {0
Model performance is measured by the expected 0/1 reward over examples (z, a) in the distribution: ((ra))P(PP) (z, a)), where the per-example expected 0/1 reward (equal to the probability that model 0 gen- erates a correct response for problem z) is:
pa(x, a) = E (y Toyi (3)
A popular algorithm for maximizing expected 0/1 reward is REINFORCE Leave-One-Out (RLOO) KKoo
et al. 2019 Ahmadian et al.) 2024), which extends the naive REINFORCE algorithm Williams 1992) using multiple generated responses to form baseline. At each iteration t, and for each pair (x, a) from training set D. RLOO updates the model parameters as
Oc+1
0, + GRIOO (r, a)),
where, 17 is the learning rate, and the gradient estimate is weighted average over N generated responses IID ys To (=()))):
1 (z,
EiEIN A ARLOO, V, log To (z)
(RLOO)
N
2

---

Published in Transactions on Machine Learning Research (03/2026)
The weights A ARL.O are called advantage scores. The construction of RLOO's advantages scores ensures GRLOO is an unbiased, low-variance estimate of Vapo(a,,)) a), the gradient of the population (expected) 0/1 reward. Intuitively, for large N:
(x,, = Vopo(z.a).
(2)
Despite its simplicity, RLOO is a strong baseline (Ahmadian et al. 2024 Its structure also underlies recent popular variants like Group Relative Policy Optimization (GRPO) (Shao et al. 2024). In full online mode (when clipping can be ignored), GRPO uses the same form as (RLOO) with normalized advantages A AGRPO ARLOO/ VP(1 ù), where á the empirical 0/1 reward for example (r,a).
1.1.2 The Pass@K Reward
The 0/1 reward, however, misaligns with practical LLM usage. Given problem æ; users typically prompt the model multiple times (say, K times) and examine the responses yI..... WK to find one that produces the desired answer. Accordingly, LLMs are often evaluated using the Pass@K metric (Chen et al. 2021 which measures whether at least one solution is correct among generated responses:
Pe (or ((U))EE(K),)) a) = { at least one yi agrees with a
otherwise
Since test-time performance is measured via Pass@K, we may train the model to directly maximize Pass@K reward rather than 0/1 reward by maximizing ((,,))~ ((k,0))/PPK,00z, a)], where we define the per-example expecteoo Pass@K reward as
Pk. a) == (aa( Paoou (Lv))E(K),
This equals the probability that model 0 generates at least one correct response in K samples.
1.2
Contributions
C-1. Direct Pass@K optimization. Building on Mahdavi al. (2025). we show that the exact Pass@K policy gradient equals the 0/1 gradient reweighted by per-example @ (K-1) probabilities. This yields Monte Carlo unbiased estimators REINFORCEx /RRLOO and a biased variant GRPOk, all expressible asymmetric reweightings of their 0/1-reward counterparts.
C-2. Advantage shaping = surrogate reward maximization. We reverse-engineer the advantage-
shaping al Chen et al. 2025 and find that it asymplotically (in the number of generated responses) maximizes a surrogate reward arcsin( (PKK.).
Conversely, we provide a forward-cnginnering recipe: starting from any differentiable surrogate F [0, and surrogate-reward maximization objective
max8
E(r, [F (Bynro(1z)) rly, r(y,a))
(3)
advantage-shaping heuristics can be systematically derived as follows:
Differentiate the surrogate reward: (pF( = FPo) Vopo
(ii) Substitute all population quantities with their empirical analogues:
(a) Replace F'e) with FP)
(b) Replace the population reward gradient Vopo with its RLOO proxy GoLOO(() (Eq. 2
Put together, this yields per-example empirical gradient Gp(0; (z, a)) = a EieN 4 V, with advantage shaped as Af = F'(P) / /) ARRO Thus, a practical implementation can reuse existing GRPO code by multiplying the GRPO advantages (before clipping) by this prefactor.
As corollary, when K=1, we show GRPO is (up clipping) effectively RLOO applied the surrogate reward F(po)= 2 arcsin / variance-stabbllziing transformation of the 0/1 reward.
3

---

Published in Transactions on Machine Learning Research (03/2026)
C-3. Reward-level regularization. We interpret commonly used heuristics that downweight "easy" examples (those already solved with high probability) and upweight "hard" ones (Chen et al. 2025 Team et al. 2025). Concretely, we show that multiplying the GRPO gradients by 1 - p is effectively equivalent to optimizing a regularized surrogate reward aresin( (Po) V/o(1 - po). More generally, this can be interpreted as special instance of regularized surrogate reward maximization:
max9
E(a.a)D F (Ey--n((aa) r(y.a)) a)) + A 2 ((y)))))))
(4)
where the stochastic training objective over examples (I, ~ 'D consists of two components: a data-fitting term F (e.g., the GRPO aresin()Po) surrogate) and an additive regularizer Ω that encourages maintaining some probability mass on wrong responses, thereby exploring alternative solution paths that may generalize beyond the training set. Unlike typical policy-level or parameter-level regularization, this form of regulariza- tion operates simply at the reward level. To demonstrate its use, we instantiate the recipe of Contribution C-2 with entropy regularization, yielding a new tunable advantage-shaping algorithm.
Table 1: Unified view of main algorithms studied in this work. Rows grouped by target evaluation metric. P, PK are per-example expected 0/1 and PassOK rewards; ý, pK are their empirical estimators.
Algorithm
Advantage Scores (A+, ,-) Weighted Empirical Gradient
Population Surrogate Reward
Torgets 0/1 evaluation
(K=1)
RLOO
Kool et al.
2019 2019
NG-08 N-1 N
AL A) 1) 6-1 vØi1 - M/ - 6-]
GRPO
Shao et al
2024
2 (rc)
p.
I-p
Sleww-R
This works]
(1- Bly 1 -(1 AI
(1 8) vā(I 7,18--B
arcsi(( vpll-p)
Torgets Pass&8 ecaluation (K>2)
RL00x This work]
--1 x-1 ,
NO -
H
Pi -p) 8, PR-IM(1 11)6 A/
PK
GRPOx Chen al..222000 2025
Gx L1
x: -5x V
V Vi p -8-1
k aresiul /PK)
GRPOx This work]
xx 1 V p. -Ja- 12.)
Ph 8i 0 IK-IVAL- B(1 1 (1-1(1 K-
-
Table | unifics the main Pass@K policy-gradient algorithms studied in this work. Starting from their ad- vantage scores, we express every method (middle column, ignoring normalization constants and clipping) as W4 v_-w__ V_, where effective gradient weights wy multiply the average empirical gradients over correc- 1/wrong responses. Leveraging this unified representation, we reverse-engineer the corresponding population surrogate reward that each algorithm implicitly optimizes (last column). Conversely, using the forward- enginoering recipe from Contribution C-2, we can map chosen surrogate reward back to advantage scores, establishing a bidirectional relationship (see also Figure
w_
advantage scores
weighted gradients
surrogate reward
Roadmap. The rest of the paper is organized as follows. We start in Sec. 2 by reviewing the 0/1 0/1-reward setting and fixing the unified gradient/ /advantage notation that we use throughout. Sec. 3 then develops C-1: we directly differentiate the Pass@K objective leading to the REINFORCE /RLOOK estimators and the GRPOk variant. Sec. presents the complementary advantage-shaping route of GRPOk Chen 2025) and contrasts its symmetric reweighting with the asymmetric reweighting arising from direct Pass@K optimization. Sec. establishes C-2 by bridging these views: we reverse-engineer GRPOx as (asymptotically) optimizing an arcsin- transformed surrogate Pass@K reward (Sec. 5.1 recover it by forward- engineering from this surrogate (Sec. 5.2). and then generalize the reverse/ffowaan correspondence to broad class of binary-reward RLVR gradients (Sec. 5.3). Finally, Sec. G develops C-3 by interpreting practical hard-example upweighting as reward-Level regularization and instantiating the forward-engineering recipe
et al.
Bffectively equivalent to Kimi 1.5 Prioritized Sampling
Team et al
2025 soe Sec. 6.2
_ f= are leave-one-out Fail((K-1) scalers (Eq. 15)). Mahdavi et 2025
implements biased version; App. C.3
"The scaler Wx is defined in Eq. 21
4 Incomplete beta function B ax $S0 aa-1(1 w) 1--100 du.
4

---

Published in Transactions on Machine Learning Research (03/2026)
with concrete regularizers. Sec. i collects practical considerations and limitations of the surrogate-reward viewpoint. Sec. g reviews concurrent work and connects our surrogate-reward formulation of RLVR to conditional stochastic optimization in the optimization literature. We conclude in Sec. with an outlook on our perspective and its prospects. The appendix contains deferred derivations and proofs.
Notation. We define a training example as a tuple (z, a), where x is the prompt and a the reference answer. For this example (x, (x) let yI,,..,, yN denote N responses generated IID from model TT(-|a)). Let T4 To: a) denote the 0/1 reward of response i, and p == NV EiE(N) rs denote the empirical 0/1 reward. We denote the per-example expected 0/1 reward as Po Po(2,0) == E ( (a)) (1..11yy))))) a)]. When clear from context, we simply write p to denote po(2, a). Let N = ΣiE[N) r3 and N == N N+ denole the number of correct and wrong responses. We use the shorthand V. == Vo log no (vilz)) for the log-probability gradient of individual response y/,, and for the average ww-pprbbabiity gradient over correct/ wrong responses:
Σ Vo Vologmo(ss and V N+
Σ 6r:=0
Vo
(5)
Throughout, A, denotes the advantage score of the i-th response to problem x and we silence its dependence on when it will be clear from context. Finally, we reserve K to denote the argument of Pass@K and quantities with subscript K denote quantitics related to Pass@K reward. In particular, PK,0 denotes the per- example expected Pass@K reward and ~K ils empirical counterpart. Notation specific to different algorithms we study in this paper appear as they are introduced in the text.
2 Warm-up: Optimizing the 0/1 Reward
With access to finite training set D of problem-answer pairs (z,a), which we call examples, natural approach to maximize the average (over D) reward is stochastic gradient ascent. At each iteration t, we update o.t1 = 0g + 17 D) with learning rate and gradient.
G(O; E (icee>>D/Vopole a)] By (2)) a) Ve log [o(lac)
(6)
where C denotes empirical averages and we have applied the log-derivative trick: Vonaly/>) = (g(aa)
Vo log (y]xx)) The REINFORCE-style algorithms reviewed below provide different approximations of the per-example expected gradient Goy1 (x, (o)o
REINFORCE. Classical REINFORCE (Willaams 1992
approximates the expectation over y <T(-12)
in using the empirical average over N sampled responses. This yiclds the per-example gradient:
CREINFORCE = N
Σ r V, = V.
N
While simple, this estimator suffers from high variance and intuitively provides poor updates since it entirely ignores wrong responses.
RLOO. REINFORCE with Leave-One-Out (RLO0) Kool et al. 2019 A Ahmadian et al. 2024h reduces
variance by replacing in Eq. 7 rewards r; with advantage scores that subtract a leave-one-out baseline:
A N
Σr, r,
jti
This also introduces learning signal from wrong responses. Since (0, 1}, the advantage scores depend only whether response y; is correct or wrong:
N -1
N(1 P)
N
A
and A
(8)
N
N
1
N
5

---

Published in Transactions on Machine Learning Research (03/2026)
Ignoring the common factor N/(N- 1) across all examples, the RLOO gradient simplifies to:
Goooo (2, (z) = A v + A (1- -ー
= p (1 - D) | - - 8-
(9)
This is an unbiased estimator of Vape(z, a) with reduced variance compared to REINFORCE.
GRPO.
GRPO Shao et al. 2024 normalizes advantages by their empirical standard deviation:
A Ti n-P
V
This yields correct/ pereet/wrong advantage scores
p-p
A -
p
A+
and
(10)
VPLL-1
V VPLL
P)
1
Thus, in the fully-online Säingle-update--er-rlloooo regime where the PPO-style clipping term is inactive, and ignoring the KL regularization term (see App. for the full GRPO objective and discussion), the GRPO
gradient can be expressed conveniently as (Deng al. 2025a
(z, vp (1 P) | - -+----1.
(11)
- scaled by function of p. RLOO scales by ((1-p) p), while GRPO uses Vā(1 - P). Throughout, we call such multiplicative factors
Comparing Eqs.
9 and 11 we see both RLOO and GRPO have the form -
the effective gradient weights of the respective algorithm.
3 Pass@K by Direct Differentiation
In analogy to the 0/1 case, the natural approach to optimize the expected Pass&K reward is stochastic gradient ascent with gradient
Grawak (0) (n,a)~ (e
To compute the gradient, rewrite the per-example Pass@K reward in terms of the per-example 0/1 reward. Using the IID nature of generated responses:
(K,
Pr (3i E [K] is correct) = =1 II (-)la))
Pr (y is not correct) = 1 111--111 ())( ()
(E(K)
Thus, following Mahdavi et al. 2025
VepK,o(z, = K(1 a)) Vepa a) = K(1
(a( k)) -a))))GG)))) Go/n(( (0; a)). Now, we recognize that the term in parentheses equals the per-example Fail@
1)= 1 --Pass@ (as) (K-1) reward. Hence, we obtain the final expression for the per-example (population) gradient:
Grassuxc (2, a)) = .(111 -1- 1L-PK-I-C (l-K-- Go/1 (0; (x, a)).
(12)
(ail
This is reweighting of the respective gradient Go/1 (6; (z, a)) = (y [ron a) log [o(2))] for the 0/1 reward. For K == the gradient updates are indeed equivalent.
Note that GRPO directly uses mean estimates across responses (rather than leave-one-out). More importantly, normalizes standard deviation, making the gradient biased estimator of Vapa(z,

---

Published in Transactions on Machine Learning Research (03/2026)
REINFORCEK. With access to a finite number of responses per example, need to approximate Growe (0; (x, a)) with empirical estimate. To arrive at an unbiased estimator, we require an unbiased estimator for the Fail@ (K-1) term in Eq. [2). For this, we leverage the following unbiased estimator of PK,0(2, (z, a) Chen et al. 2021:
N K
N K
PK = (K,((,)))
(13)
representing the probability of drawing K responses without replacement from N total responses such that at least one is correct. Having unbiased estimators for both terms in Eq. 12 we combine them via the leave-one-out trick to form an unbiased estimator of their product:
GREINFORCE (8: (z) = Σ
K
(lood) (1-PR-1) KE[N]
(1
T3 V,,
(14)
N
where 1 1-pk-1 is the leave-one-out unbiased estimator of Fail@(K-1) excluding the i-th response. For binary rewards, simple algebra (deferred to Appendix B.2 shows that
fk-1= =1-==11- (1-px) p- k=) if is correct
1oo,5 PK-1
(15)
= (1-
ys is wrong
The weights - and - are leave-one-out empirical estimates @ail@(K-)), both expressed in terms of the empirical Pass@K reward ọK. With these, the REINFORCE& update takes the following final form:
GREINFORCE, (0; (x. a)) fk-1' p. 4 = f (GREINOORCC(((( a)).
(16)
Reweighing the vanilla REINFORCE gradien by 1 the empirical Fail@(K-1) of the other N- samples, amplifies gradients of examples with rare correct responses (large N) and suppresses gradients of redundant examples (small N)). When N < K-1, 1, the weight is 0 since Pass@K is alrcady guaranteed. As in the 0/1 case, REINFORCE is unbiased but high-variance estimator. We can reduce variance by substituting rewards T; in Eq. 14 with appropriately computed advantages. This way, we also introduce negative gradients.
RLOO k. Using the RLOO advantage scores from Eq. 8 in place of r; in Eq. [4], the RLOO algorithm for Pass&K optimization has gradient (proportional to, ignoring constants N/(N and K):
GiLOok (0; (z, = 7( P) 4 Jn-1 M( P) V_
(17)
Note the asymmetric weighting for correct versus wrong responses due to Pass@K maximization. This is unbiased estimator (sce App. B.3 for proof) of Gr (I, with lower variance compared to GREINFOnCE, (0: (I).
GRPOk. Analogously, we obtain GRPO update for Pass@K optimization by using the GRPO advantage scores from Eq, (10) in place of r3 in Eq. 14h:
GERPON (0: (z.a)) fK-1 Vp V -
-) - V_
(18)
4 Pass@K by Advantage Shaping
The two Pass@K optimization algorithms, RLOOk and GRPOk. of Section 3 differ from their vanilla counterparts through asymmelric advantage reweighting. Specifically, their effective advantage score is:
A + f A
(19)
where f defined in Eq. 15 and = the vanilla advantage scores (c.g., Eq. 10 for GRPO).
We have dropped constant factor across all examples, which can be absorbed learning rate.

---

Published in Transactions on Machine Learning Research (03/2026)
Comparison of Pass@K Effective Gredient Weights (N=16)
K-1
K-
K-
K-0
K- 8
ERocze GrolertWeggh
0.2
-04
0.00
325
230
0.73
180
0.00
1.25
0.30
0.75
1.22
0.00
0.25
1.50
0.75
1.00
0.00
0.5
0.00
0.75
102
0.00
0.25
0.50
0.75
1,00
P
GRPOg
Positive Advantage 1A"1
Negativee Advantage (A")
vs. GRPOx (4 in Eq. 21 These weights scale the gradients of correct (solid lines) and wrong (dashed lines) responses by factor A 0 and
Figure 2: Effective gradient weights of GRPOx (A in Eq. 19
A (1 - p), respectively. The scores are plotted against the empirical 0/1 rate (p) for a fixed sample size N = 16 and varying K. For K 1, both weights coincide with vanilla GRPO weights vp - p) (Eq.
Both methods downweight easy examples (large P), but GRPOk more aggressive and uses asymmetric correct/wrong weights, unlike the symmetric scaling in GRPOk-
GRPOx via advantage shaping. We now review the GRPO variant proposed by Chen et al. 2025
which we denote GRPOk, and which directly shapes GIPOO's advantages to favor Pass@K improvements. Their proposed advantage scores, which we denote A are given in (Chen al. 2025 App. B3) as:
Atteeca v (1 PK)/PK and
À ATawO
(1-PK FR-1)/VK /kl pk),
(20)
expressed in our notation of empirical Pass@K reward PK (Eq. 13) and the ceave-onewroong-uu F estimator fK-1 (Eq. 15 As noted in Deng et al. 2025b), this can be simplified: From Eq.
15), we have
1 - 7K- (1 p) JK Thus, A
At Atwaak and in terms of vanilla GRPO advantage:
1-p
1 1- PK p PK
A Apaan WK A# where
WK "k:=1
(21)
V p
Overall, the per-example gradient of GRPOx is:
& (0: (z; = Wg (C) (z, =
1 1- pK
[ p.
p-(0+ [
(22)
pK
Comparing the two approaches. Both methods reweight vanilla advantages and have similar effect: compared to the vanilla methods of Sec. 2 that optimize 0/1 reward, they aggressively downweight gradient contributions from examples with medium-to-high empirical 0/1 rewards, thereby favoring more difficult examples. Yet, they differ structurally: The reweighting Eq. (21 is symmetric (same multiplier Wk for correct and wrong responses), whereas Eq. optimization more aggressively dampens contributions when the empirical reward is high (and amplifies more when low). Figure visualizes the effective weights each algorithm applics to correct vs. wrong gradients.
[19 uses asymmetric weights (fk-1 * fx-1).
Moreover, direct
5 Bridging the Views
5.1 Reverse- Engineei-n Surrogate Reward from Advantage Shaping
4 showed that both Pass@K methods, direct optimization (GRPOk) and advantage shaping (GRPOk): can be expressed as reweightings of vanilla GRPO. We now tighten this connection by showing that advantage shaping also admits an interpretation as direct optimization of surrogate Pass& reward.
Section
Claim (Reverse- Enneess-Engineriing GRPOk). The GRPOx algorithm by Chen et al. (2025) rescales the vanilla GRPO empirical gradient, yielding updates shown in Eq. (22 For sufficiently large sample size N > K,

---

Published in Transactions on Machine Learning Research (03/2026)
this corresponds to direct mazimization of the per-ezample surrogate reward
2 aresin ( (K.o(z,)))
(23)
K
where (K, == heper P Franoxx (1 is the expected Pass0K reward per exomple.
Thus, the advantage-shaping approach of Chen et al. (2025) implicitly optimizes a smooth, differentiable transformation of the Pass0K reward, providing further justification for the validity of the different approach by which the authors arrive at it. Note that the surrogate reward is maximized when the Pass reward equals 1, and is actually strictly monotone, so both the original and surrogate objectives share the same optimal solution, though the aresin surrogate reward could lead to different optimization path.
Connection to variance stabilizing transforms. The arcsin transform that emerges is noteworthy: it is a variance-stabbliznng tronsformation (VST) for the binomial distribution. Specifically, for X ~ Bin M, p). the arcsin transform satisfies Var(vM arcsin( /X) 1/4 independent of p (Ansommbe 1948). This connection not coincidental: [Chnn et al. (2025) Soc. 2.2) arrived their method through arguments using batched sampling, where N responses are partitioned into N/K groups of size K. In this setting, the number of successful groups follows binomial distribution with M N/K trials, and the arcsin transform would stabilize the variance of this estimate. Formalizing potential the connection between VSTs and optimization stability when applying empirical gradients to transformed rewards is open direction for future work.
5.2 Corward-Engineering Advantage Shaping from Surrogate Reward
GRPOx be reverse-engineered optimizing an implicit population-leva surrogate reward. Now, we address the forward direction: How to arrive at GRPOk starting from the trosin-tr surrogate reward?
Claim 2 (R.OOx=RT.0 on Surrogate Reward). The GRPOk (2025) equivalent an RLOO-style policy gradient update to the surrogate per-enample reward arcsin( arcsin(V (VPK,o(z,a) from Eq.
update by Chen et al.
To this, we work as in Sec. 3 but now with the aresin-rranssormed objective. We proceed in two steps:
(i) By direct differentiation, the population per-example gradient is
1 (1 pa) Kl1 Vepe
K V arcsin (VKK,)) = K v Vk PK,0)
V8PK,8
VPK,0(1 -Kx00)
1 1 PKK0 Vapa,
(24)
Vp -KK,)
Po
(ii) To obtain an empirical version, we (a) approximate the multiplicative term of the gradient by substituting PK.8 and pe with their empirical counterparts Px and p. and (b) approximate the expected 0/1 reward
gradient term with the RLOO gradient update from Eq. (9] (recall Eq. (2)) The update then becomes:
1- 1-px
PK
7(1 -(1- 1-7
p.
VTx(1 px)
PK
and we recognize G GRPOx (0; (2, a)) in the final expression (see Eq.
22
Alternative surrogate rewards. This connection invites exploration of alternative variance-stabillzingg Iransformations. The aresin transform is a VST for the binomial distribution, which aligns with the batched sampling scheme of Chen et al. (2025 Sec. 2.2). However, the standard combinatorial Pass@K estimator does not strictly follow a binomial distribution. Also, the analysis above assumes N grows large while К remains constant. Designing surrogate rewards that account for joint scaling of N and K an interesting direction, particularly for the practically relevant small-N regime where computational budgets are limited.
9

---

Published in Transactions on Machine Learning Research (03/2026)
Biased versus unbiased estimation. In the derivation of Claim 2 contrary to construction of RLOOx in Sec. 3 we did not insist on maintaining unbiasedness of the gradient estimator. Lok is biased estimator the multiplicative term in Eq. (24), and even replacing Px with a leavo-one-out estimate would not remove this bias since the product of two unbiased estimators is generally biased. However, note that
1-p
even the strong vanilla GRPO baseline does not implement an unbiased empirical estimate of the gradient.
Special case: GRPO. In fact, applying the logic of Claim to the K case provides an alternative interpretation to GRPO's normalization by the reward's standard deviation.
Corollary (GRPO as Surrogate Reward Optimization). The GRPO update (Eq. 11 (2024) is equivalent to an RLOO-style policy gradient update for the surrogate per- per-ezample reward
by Shao el al.
aarcsin
( (Vpu(s,a)
where po(x, a) == (1 (v, a) is the per-erample expected 0/1 reward.
Indeed, the population gradient of this surrogate is Vop/ hpp1 - p). Approximating the leading multi- plicative factor by 1/1p(1 - p) and applying an RT.00-style estimator for Vop (which is proportional L-.. matching that of vanilla GRPO.
P(1 -1---p11 -DV + - - by Eq. 9 yields update
This surrogate reward perspective should be distinguished from GRPO's original formulation. As presented by Shao et al. (2024), GRPO maximizes a PPO-style objective designed for the multi-epoch off-policy setting, where the expectation is over the old policy. Our surrogate reward, in contrast, identifies the on-policy population objective that the GRPÓ gradient ascends in the fully online setting. We detail this distinction in App. A showing why the PPO-style objective cannot directly yield this on-policy surrogate.
5.3 Equivalence of Advantage Shaping and Surrogate Reward
The surrogato-reward perspective developed in the previous part of this section for GRPOk (and its special case vanilla GRPO for K 1) applics more generally to a broad class of binary-rewarl RLVR policy gradient algorithms whose per-example gradient be expressed as:
G(: (I) = wi ( V+ -- (p) V_,
(25)
for some functions w+ : [0, 1] + R which determine the effective gradient weights multiplying the average empirical gradients of correct and incorrect responses respectively. This form corresponds to the middle column (Weighted Empirical Gradient) of Table 1 as seen, all algorithms we study here fit this expression.
The surrogate reward perspective provides an interpretation of any such algorithm in the population limit of large N. Specifically, the following claim generalizes Claim !
Claim In the population limit N + oo, a policy-gradient algorithm with per-erample gradient of the form 25 performs gradient ascent on a per-ezample surrogate reward (2(p(z a)), where F [0, 1] R is differentiable and satisfies for 10:
F'(u) (+(x) + w_(w) u
Consequently, the updote averaged over eramples targets mazimizing the population surrogate objective
(26)
This objective is surrogate transformation of the conventional RLVR objective in Eq. 1
Proof. Replacing empirical quantities with their population counterparts, the population-level per-example gradient
G(0; a)) w+ (og)
Iryy, w- (--(po) (oo))-yyrro((l) Vo ()()))))))))))))))))))))))))))) r a) =0],
10

---

Published in Transactions on Machine Learning Research (03/2026)
where we use our shorthand po po(z, a). Evaluating the conditional expectations (Lemma in App. B.1 yields
Vopo
Vopo [w+(po) w w-(po)
Vepo FPo) Vapo- Thus, in the population limit, the algorithm updates are of the form
((0:(z,a)) (Po)
w wp()
Pa
1 pg
pa
pa
B++1 = 0: + F' Vopo.
(27)
which is gradient ascent on the surrogate reward Fooo) (up to an additive constant in F).
Examples (reverse-engineering)) For (u) w+ (u) u(l-u) (RLOO, Eq. (9 ) we obtain F" hence F(u) = (up to constant). Similarly, for w4 (u) = and (u) = 0 (REINFORCE, Eq. 7 we
also have F (u) and thus F(u) = u. For GRPO, where w- ww(w) w- (u) = v u(1 v) Eq. we have and thus F(u) = 2 arcsin(V/ (up to a constant), matching Corollary
F'(u)= = /a(1 u)
Conversely, starting from differentiable surrogate reward F [0, + R, can derive RLVR policy- gradient update that (in the population limit) performs gradient ascent on F(pa). At the population level, gradient ascent corresponds exactly to 27 To obtain practical finite-N algorithm, we replace population quantitics by empirical analogues: (a) replace F'(po) with (P),, where p is the empirical (finite-N) estimate
of pa: and (b) replace Vepe with its RLOO praxy (o) = P(1 P) (V+ - V-) (Eq. (2).
This yields the surrogatc-driven finite-N gradient estimator
(p(0: (z, w ( --), where w(u) = F'(u) - (1
Equivalently, the conventional advantage form,
-1 p) r n= 1, Σ A v, =f== (n+ - p) = ( [
(z, = N
(28)
0.
28 is a GRPO-style update in which the surrogate choice F shapes (i.e., reweighs) the binary learning signal through the scalar factor FP))). It can be easily checked that
Up to clipping and KL regularization,
A = F' ())(())
P)
AGRPO
with AGRPO the vanilla GRPO advantage scores in Eq. (10
Examples (forward-engnnerrin). Letting F(u) = u recovers (up to a constant scaling N/(N - 1)) the vanilla. RLOO update. Letting F(u) = (arcsin(Vu) recovers the GRPO advantage shaping (Corol- lary D. For Pass@K optimization, letting F(u) 1 (1 u)* yields RLOOK, while letting F(u)
ldd for any 2 arcsin (1-11-ulus))) u)*) yields GRPOx (Claim 2. As another example, letting F(u) = -
> 0 yields onc-paramcter family of strictly increasing surrogate objectives with F'( = pa-1, i.c., (p) = -1 - 7): <1 upweights low-ō examples, while >> - upweights high-ō examples.
In all these cases is strictly increasing: F(p9) is maximized at 1 just like pe, but induces possibly different optimization dynamics via shaped advantages. In the following section, we provide additional examples of surrogate objectives that yield new RLVR algorithmic instances while possibly relaxing the requirement of the optimization objective being strictly increasing. We then conclude with practical remarks in Sec. 7 including the generally biased nature of the finite-N update (28). and how one can incorporate clipping and KL regularization to obtain practical GRPO-like algorithms in offline settings (at the cost of sacrificing the exact surrogale-maxinizalle and their potential practical benefits.
guarantee). We also invite the community to explore alternative choices of F
TThis empirical substitution represents practical algorithmic choice. While highly sensible and consistent within our analysis (for instance, when applied to F(u) = 2 2arssi((/w) it exactly recovers the strong GRPO bascline), it is not the only option. Alternative gradient estimators, such as those leveraging variance reduction debiasing techniques from the conditional stochastic optimization literature (see Sec. R could also be employed to optimize the same underlying surrogate reward.
11

---

Published in Transactions on Machine Learning Research (03/2026)
6 A Regularized Surrogate Rewards Perspective
Both Pass@K routes examined in Secs. and ! tilt the learning signal toward the hard cases, Chen et al. 2025) argue this is aligned with the exploit/ haploit/xxplone explore roles of the two metrics. Maximizing the 0/1 reward emphasizes exploitation: making individual responses more often correct, even on already-easy examples. In contrast, Pass@K encourages exploration: for a given example, only one of K responses must be correct, so once an example is "solved" in the Pass@K sense, further pushing its 0/1 reward is less valuable than allocating gradient signal to unsolved examples where more search could pay off. Ideally, want controlled balance: retain sufficient signal on solved examples while injecting additional search signal on unsolved ones. To achieve this, they propose mixing Pass@ K-saaed and vanilla advantages.
Using teverse-nngineering, we arrive at a complementary interpretation of their method as regularized sur- rogate reward mazimization. This perspective decomposes the RI. objective into a data-fitting term (the transformation of the mcan reward that the GRPO baseline optimizes) and a regularizer (an uncertainty term such as the reward's standard deviation). While the data-fitting term can be interpreted as exploitation and the regularizer exploration, the regularization viewpoint itself provides a possibly more elementary justification for practical advantage-shaping heuristie-modfficttinn of GRPO.
6.1 Combined 0/1 and Pass@K Advantage Shaping
Having observed that Pass@K optimization can aggressively favor difficult examples (small p) while poten- tially zeroing out useful signal on easier ones (medium/ /large p), Chen et al. (2025) propose a simple blend between optimizing the 0/1 reward and optimizing Pass@K. They combine the PassüK advantage with the vanilla advantage using the empirical 0/1 reward p as mixing weight. In our notation:
mix = P A PaNSK + (1 (1-7) A+ = (1 - p + P kk) A
(29)
where we used Eq. optimization view presented in Sec. 3 Since A Atwook
21 to write Abuwaxx = "K AL. The same idea can be instantiated for the direct Pass@K f -KK- = by 19 an analogous combination yields
A A.rmaa + (1- = (1 +
A=
(30)
Figure visualizes the resulting effeclive gradient weights (i.e., PA for correct and (1-7) for wrong responses). Observe this blending avoids the overdamping of mid/ mid/large-p training signal that can occur under pure Pass@K optimization. Interestingly, the two combinations become numerically very close K grows. Moreover, except when K is small relative to N, both essentially collapse to the lower envelope (1 A- That is, they essentially no longer exploit an explicit. Pass@K signal.
Comparison of Combined Pass@K Effective Gradient Weights (N=16)
Ab, Effcctre Abvarage
0.3
0.2
3.00
1.25
0.50
0.75
0.50
0.25
1.00
3.25
0.50
0.75
0.59
0.25
0.30
3.75
1.00
0.00
0.25
1.0
PGRPO, (1 PIGRPO,
PGRPO (1 PIERPO,
Skew-Rt [1- BD
Fostive Advantage (41)
Negativve Advantage (AT)
Figure 3:
Combined Pass@K effective gradient scores for N = 16 and various K. Green uses Eq. 30 Blue
uses Eq. 29:; Dotted gray is the skew-R (1 - A= For K = 2, the two combinations differ slightly at
mid/ hadhhhhh/p p (blue is modestly larger); for K 2 they nearly coincide across p. This is because, for most P, the
Pass@K scalers vanish for p < K/N, roducing both combinations to the same envelope (1 (1- (1PP)p(t P)
6.2 Reward Regularization Interpretation of S
What reward does the combined 0/1 and Pass@K advantage shaping actually optimize? Because Sec. con- nects advantage shaping to direct P optimization, one might guess it maximizes convex combination
12

---

Published in Transactions on Machine Learning Research (03/2026)
of the 0/1 and Pass@K rewards. The catch is that in (29 the mixing weight p is adaptive to the instance "difficulty" (as measured by p), so the combination is not fixed convex blend at the objective level.
To simplify reverse engineering and obtain a more interpretable surrogate reward, we leverage the empirical observation from the previous subsection: except when K is small relative to N, both combined schemes mix and 4 mix are numerically close terms of effective weights to the simpler right-skew shaping
A Aow-RR (1-
Claim 4 (Skew-R= Regularized Surrogate Reward Maximization). The skew-R (pronounced "skewer"" method rescales the vanilla 0/1 advantage by (1- -), yielding gradient updates G (0; (x,a)) (1- GGRPo(O: (2) When N > 1, this corresponds to mazimization of the per- per-ezample reword
arcsin(VP) + vp(1-p),
(31)
where p: pa (z, a) = Ey (n) (ro/1 a)] is the (population) 0/1 reward per example.
Reward-rregularization perspective. Vanilla REINFORCE and RLOO maximize the expected 0/1 reward p (pure exploitation). As we saw in Corollary 1 vanilla GRPO maximizes strictly monotone trans- form of the mean, arcsin(v/),, so it has the same manimizers as P and likewise corresponds to exploitation. In contrast, the skew-R reward can be interpreted as stochastic gradient ascent on a regularized surrogate reward that combines a data-fitting term and a regularizer. The data-fitting term incentivizes finding a policy that maximizes the reward over the training examples (x, a) E D. In contrast, the regularizer alone is maximized at p 1/2 where the binary correct and wrong policy outcomes are equally probable. An intuitive explanation of this term, aligning with the regularization effect in classical supervised learning, is that it inceutivizes the model to not "overfit" on the training examples, maintaining diversity in wrong response that may explore different solution paths generalizing to unseen examples. Formalizing such a notion of overfitting and its connection to regularization could provide deeper insights into RLVR learning tradeoffs. We note that the argmax of the combined reward in Eq. (31) is still at p = 1, reassuring here that we incentivize ultimately finding parameters that maximize the average reward. However, mirroring lessons from supervised fine-tuning, this might not always be the best strategy for generalization, and stronger regularization with an explicit parameter controlling the strength could be beneficial (sce next subsection).
Implications for the "Prioritized Sampling" strategy of Kimi 1.5. An interesting observation potentially supporting the practical value of skew-R: Kimi 1.5 TTamm et al. in their RL experiments, reweighting each example by 1 1-p to allow harder examples to appear more frequently. This corresponds to the advantage shaping of skew-R (Eq. (31 applied at the example level rather than within gradient computation. Thus, our interpretation as regularized reward maximization provides direct justification for their empirically motivated practical choice.
2025 performs priority sampling
Consistency of interpretations. In the large-sample limit N >>>>>>>>>>> 1, skew-R is equivalent to GRPOK=2 In Appendix C.2. we claborate on this comparison with emphasis on interpreting the reverse-ongincerod surrogate reward. show that GRPOk=2 effectively optimizes an incomplete-beta-ffncce of the Pass@2 (p2) reward, B31 - (1 (1-p)) /22 1/2,3/2), and that this objective is algebraically identical to the skew-R surrogate in Eq. (31 Thus, the population limit, both methods are in fact maximizing the same scalar objective. This validates our ourrogato-reward revarse-engineering framework, but it also highlights subtle interpretive question about what constitutes "regularization. " The interpretation depends which performance metric we take as the target: (1) Pass@K taryet. If Pass@K (pk) is treated as the true performance metric, then GRPOk should be viewed as directly maximizing a smooth, monotone surrogate of Pk- In this view there is no extra penalty term: we are simply doing surrogate maximization of the evaluation metric itself. (2) 0/1 target. In contrast, if 0/1 reward is the ultimate objective (as with skew-R), the interpretation changes. The baseline GRPO already optimizes a surrogate arcsin( (VP), and the extra multiplicative (1 - reweighting is interpreted as adding a regularizer term to that baseline.
transformation
6.3 An Example of Reward-Lever Regularization
To further illustrate the use of reward regularization, we show how to optimize surrogate reward that combines the data-fitting term with an entropy regularizer H(p) -plog(p) - -(1 - -p) log(1 p) of the binary
13

---

Published in Transactions on Machine Learning Research (03/2026)
(a] Effective Gradient Welght
Reward Value
caro
Ereetrve Grodkent WN
Emppiccal Roward
Pepulation Reword ES:
Figure 4: Comparison of (regularized) surrogate objectives (GRPO, Skew-R, and Entropy-Augmented). (a) Effective Gradient Weight: Empirical gradient's 's scaling factor as a function of the empirical reward p. The vanilla GRPO (blue, dashed) is symmetric. Skew-R (green) and the antropy-augmented methods (red, purple) are asymmetric, suppressing gradients for high P. Skew-R gradients are multiplied here by 2, so all four curves have the same arca under the curve. (b) Surrogate Reward Value: Corresponding surrogate reward functions, normalized that they evaluate to 1 at p = 1. Takeaway: these regularized surrogates prioritize improving
prompts. The regularization strength (larger X) controls how strongly updates are shifted away from high-Ō prompts, while the regularizer type controls the shape of this reweighting (e.g.. entropy retains a small but nonzero weight even at high p).
0/1 reward. Intuitively, similar to the standard-deviation regularization of skew-R, H(p) favors examples where the model is still "undecided," redistributing gradient effort toward those rather than repeatedly polishing ones it already gets right. The overall surrogate reward is 2 Paresin( + À Hp)). Applying the
forward-engineering recipe of Sec. 5.3 yields tunable advantage-shaping rule that scales the vanilla. GRPO gradient as follows:
1 A 11-AAvA(1-77 log p (1= GERPO.
Gentropy =
(32)
This mirrors the structure of skew-R, but with a different weighting function. Whereas skew-R uses (1 - p) and therefore drives the weight for already-solved examples (β x 1) essentially to zero, the entropy-based weight retains significantly reducod, but nontrivial, weight even for casy prompts, while still sharply boosting harder examples. See Fig. for illustration. Note also that the entropx-augmented objective need not be globally maximized at p p= 1 for all A: For À S 1.5, its maximizer remains at p 1, so ultimately pushing toward perfect success. For larger regularization strengths, the entropy term is strong enough that the overall maximizer shifts to some p < 1, providing a mechanism to penalize perfect fitting of the training data, stronger form of regularization.
still
7 Practical Considerations
Biased vs Unbiased Scalings. In Sec. 3 we derived Pass&K advantages by using a leave-one-out con- struction to oblain unbiased estimales of the two components in (12 Fail@(K -- and Go/1 (0: (x, a)), resulting in the unbiased empirical gradient GRLOOx in Eq. [17 In contrast, Mahdavi et al. (2025) apply biased empirical gradient with advantage scaling (1-p) A This remains biased even if A cor- respond to an unbiased method, such as RLOO. Reevrsse-nngnneriing their biased estimator shows that it maximizes the same surrogate reward as our unbiased RLOOx. In fact, the advantage scores resulting from our forward-engineering of surrogale rewards are all biased. Thus, while the surrogate reward provides a unifying perspective to advantage scores, it does not differentiate between biased vs unbiased implementa- tions. While unbiasedness intuitively desirable property, see App. C.3 for a discussion on regimes where biased variants can be attractive. Finally, we remark that when the number of responses N per prompt is small, plug-in estimates such as (or (K) can be noisy, and the resulting reweightings may deviate from
14

---

Published in Transactions on Machine Learning Research (03/2026)
their large-sample forms assumed by our surrogate-reward interpretations. Characterizing these finite-N effects is an important direction for future work; we view the surrogate-rewarl lens as still useful here as an organizing principle that identifies the population objective being approximated.
Normalization Factors. The main difference of GRPO from classical RLOO is, up to the clipping, the normalization with respect to the reward's standard deviation. Cor. | provides an alternative interpretation of this scaling: at the level of the training objective, it maximizes a transformation of the 0/1 reward that stabilizes the variance of its empirical estimator. More broadly, our forward-engineering of reward surrogates F()) uses RLOO empirical gradients as a robust estimate of the population reward gradient Vep and scales them by (p) that generalizes the specific GRPO normalization via standard deviation. On the empirical side, there is ongoing debate about whether such normalizations are necessary for good performance. For example, in support of earlier works [Ahmadian et al. 2024). a recent study by bh(Kat et al. (2025) finds that RLOO is competitive to GRPO, with zero-mean advantages being the main driver behind its success (Xiong et al.) 2025). This raises an interesting question: Could we thcorctically characterize how properties of a surrogate F relate to favorable optimization dynamics? Such an analysis might shed light on if, when and why these normalizations matter in practice.
Clipping. Our analysis omits the clip operation of GRPO (Shao et al. 2024), which is itself inherited from PPO (Schulman et al.] 2017). This is accurate when GRPO operates in full online mode (single update per model generation), as originally done by Shao et al. (2024) However, if multiple updates are performed per generation, the GRPO gradient expression in Eq. (11) that our analysis uses is only a proxy. That said, current systems are often optimized be on-policy (Liu et al.) Qi et al., 2025). Either way, all advantage-shaping methods studied here (e.g., GRPOk, skew-R, entropy-augmented) can be empirically applied in practical, multi-epoch training regimes by simply substituting the respective advantage scores into the original GRPO formulation with the clip operation and/or KL regularization (or follow-up variants, e.g., YYu et al. 2025a Zheng et al. 2025 ). Specifically, using a surrogate F' is essentially a one-line change in existing GRPO code: following (28). set A; + F' F'()) F'(D)/((1 - p) A) AGRPO (with ş the batch success rate) before clipping, then apply the usual clipping/K (KL.
Granular Advantage Shaping. The methods we study shape the advantage at the example level. This is the case for all methods that result from forward-onginecring a (regularized) reward surrogate (as in Eq. +): they scale vanilla. RLOO by a factor ('( + x (p) that depends only on the per-example empirical 0/1 reward p. The only exception is the Pass@K methods designed in Sec. 3 which introduce asymmetric weights
for correct versus wrong responses. Here, advantage shaping is done at the response level. F Finer-grained
shaping is also possible. For example, Deng et al. (2025b); Cheng et al. (2025) : Cui et al. 2025 investigate
token-level advantage shaping and would be interesting to reverse-engineer those as regularized surrogate
rewards, albeit in the form of finer-grained regularization at the policy/ / reoeesereeeeesnntttoonn level.
Empirical evaluation and surrogate selection. Beyond providing a unifying lens for interpreting exist- ing RLVR SLVRt/G policy-gradient methods, the established bi-directional relation "advantage shapes ++ weighted gradients + surrogate reward" offers an algorithmic opportunity: a forward-engineering recipe for designing advantage-shaping rules. As noted above, the updates derived from this recipe are plug-and-play in existing GRPO implementations. In Secs. 5 we introduce concrete surrogates as starting points, and in App. D we provide proof-of-concept experiments comparing the performance of variants like Skew-R, GRPO K. and entropy-regulan regulariee methods against vanilla GRPO. For designing future algorithms, the most convenient objects to work with are the surrogate reward F(po) or the associated weighting functions (Pa), as these directly specify how the update reweights correct vs. incorrect responses (see Sec. 5.3 Conceptually, the equivalence "weighted gradients <> surrogate reward" mirrors the duality in supervised learning bet ween modifying gradients and modifying loss functions. This suggests that well-studied robust or imbalance-aware loss-design principles from supervised learning can serve as starting points for designing corresponding RLVR updates. Overall, the recipe is intentionally general; a systematic empirical study of this design space that involves identifying robust families across tasks/com /compute regimes and characterizing trade-offs is an impor- tant next step, which we leave to future work. Combined with the theoretical unification through surrogate
FIn this specific case, the weights of correct (fx and wrong gradients (xx1) have the same population limit. The reverse- engineering approach of Claims 4 can still be applied even if these limits are themselves asymmetric (see Appendix C.4.
15

---

Published in Transactions on Machine Learning Research (03/2026)
rewards, such study may also yield concrete guidelines for when properties of the data distribution, model capacity, or task structure favor one algorithm over another.
8 Related Work
Our work is motivated by the rapid development over the past year of variants of the GRPO algorithm. from vanilla binary reward maximization to Pass@K optimization. These developments typically result from selecting appropriate advantage scores, as detailed in the preceding sections. Our work provides a comple- mentary perspoctive via weighted gradients and surrogate rewards, unifying existing algorithms and offering an alternative recipe for engineering new ones that is more akin to loss design and gradient reweighting in supervised learning (sec discussion in Sec. 9
Contemporaneous work by Davis & Recht (2025) independently arrives at the same reverse-enginerring of algorithms such as GRPO as surroaate reward maximization. Since we were motivated by the suite of Pass@K-tailore algorithms, the set of algorithms we study here is broader. Additionally, beyond reverse- engineering, we put forth a specific forward-engineering recipe that forms the basis for using the framework to design new RLVR algorithms. That, said, the expositions, even at the level of reverse-engineeringg and the motivations between our paper and theirs are qualitatively different; we invite the interested reader to consult their paper as well for complementary perspective on this topic.
The surrogate reward maximization objective in Eq. 3 falls under the umbrella of what the optimization literature calls conditional stochastic optimization (Hu et al. 2020a), which has been studied algorithmically in a series of works, c.g.: (Hu et al. 2020b Wang & Yang. 2022 He & Kasiviswanathan 2023). The algorithm resulting from our lorward-enginnerrigg is essentially the same as the biased stochastic gradient ascent algorithm proposod and studied in Hu et al. (2020b), but not identical: here we use the RLOO empirical estimator for Vaps rather than the REINFORCE estimator (no baseline), which would make the two identical. This connection of RLVR algorithms to the conditional stochastic optimization literature opens the possibility of applying techniques from it (e.g., variance reduction and debiasing) directly to RLVR implementations, and at the same time further motivates the study of the conditional stochastic optimization framework.
9 Conclusion
Motivated by an effort to reconcile recent Pass@K policy gradient mothods, we show that advantage shaping is (asymptotically) equivalent to RLOO-style optimization of surrogate rewards F(p). This equivalence extends beyond Pass@K: for instance, "hard-example upweighting" heuristics correspond adding reward- level regularization s2(p) to the training objective. While F(p) is maximized at p p= 1, acting as data-fitting term, the regularizer M(p) provides an opposing force that can be interpreted preventing overfitting, particularly to easy training examples.
Advantage shaping and surrogate reward design are two sides of the same coin. This equivalence mirrors classical supervised learning, where modifying the training objective analogous to directly modifying the gradient. Both design approaches have proven effective, with strong parallel supervised classification for imbalanced data, egg. (Lin al., 2017 Menon et al., 2020 Ye et al.) 2021 Kini et al.) 2021). At a conceptual level, understanding this equivalence is instructive: irrespective of the design choice- whether starting from advantage shapes or surrogate rewards-analyzing analyzing the counterpart can provide valuable insight. Looking forward, while direct parallels to supervised lcarning should be drawn cautiously, the (regularized) surrogate reward lens may facilitate (i) theoretical bridge between ilerative policy-gradient optimization and the population performance metrics these algorithms ultimately aim to optimize, (ii) designing new practically relevant advantage shapes, with surrogate losses and reweighting schemes from supervised lcarning as potential starting points, and (iii) gaining insight into what properties (e.g., of the data distribution or model) determine when existing new algorithms perform well.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Acknowledgments
This work supported by NSERC Discovery Grant No. 2021-03677, Alliance Grant ALLRP 581098- 22, a gift from Google.org, and UBC 4YF Departmental Fellowship. Additional computational resources were provided by the UBC Advanced Research Computing (ARC) Sockeye cluster. The authors thank the anonymous revicwers for the useful feedback that led to improving the manuscript. CT especially thanks Reinhard Heckel (TU Munich) and Mahdi Soltanolkotabi (USC) for inspiring and helpful discussions and ideas on the topic.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in lms. arxiv preprint 2rXiv:240.. 14740, 2024.
Francis J Anscombe. The transformation of poisson, binomial and nogative-binomial data. Biometrika, 35 (3/4):246-20) 25444,,, 1948.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. Xáv preprint arXiv:2107.05:: 03374, 2021.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass k training for adaptively balancing exploration and exploitation of large reasoning models. arXie preprint arXiv:2508.1071 2025.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint ar 14758, 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.2200 22611, 2025.
Damek Davis and Benjamin Recht. What is the objective of reasoning with reinforcement learning? arXiv preprint 2rxi:22510. 13651, 2025.
Wenlong Deng, Yi Ren, Muchen Li, Danica Sutherland, Xiaoxiao Li, and Christos Thrampoulidis. On the effect of negative gradient in group relative deep reinforcement optimization. Xiv preprint arXiv:2505. 18830, 2025a.
Wenlong Deng, Yi Ren, Yushu Li, Boying Gong, Danica J Sutherland; Xiaoxiao Li, and Christos Thram- poulidis. Token hidden reward: Steering exploration-exxlootatioo in group relative deep reinforcement learning. arXiv preprint arXiv:2510.0300 03669, 2025b.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lmms via reinforcement learning. arXiv preprint 200i:22501.1044, 2025.
Lie He and Shiva Kasiviswanathan. Debiasing conditional stochastic optimization. Advances in Neural Information Processing Systems, 36:78846 788083 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Fric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math datasct. NeurIPS, 2021.
Yifan Hu, Xin Chen, and Niao He. Sample complexity of sample average approximation for conditional stochastic optimization. STAM Journal on Oplimization, 30(3):2103 2133, 2020a.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning. Advances in Newral Information Processing Systems, 33: 2759 2770, 2020b.
Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for llms. arXiv preprinl arXic:2510. 13786, 2025.
Ganesh Ramachandra Kini, Orestis Paraskovas, Samet Oymak, and Christos Thrampoulidis. Label- imbalanced and group-sensitiv classification under overparameterization. Advances in Neural Information Processing Systems, 34:18970-1899 2021.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a bascline for free! 2019.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Facze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint or 11 15124, 2024.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp.
2017.
Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When speed kills stability: Demys- tifying rl collapse from the inffrenn mismatch, september 2025. URL https://richardli, zyz/rl-
collapse.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint or arXiv:2503.20783, 2025.
Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Renjie Liao, and Christos Thrampoulidis. Beyond accuracy: A policy gradient reweighting approach for pass@ k maximization in llms. In 2nd AI for Moth Workshop@ TCML 2025, 2025.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiw:2007.09914 2020.
Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and success amplification. arxiv preprint arXiv:2503.066 06639, 2025.
Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Defeating the training-inferrncce mismatch via fp16. arXio preprinl ar arii::2510.2677 2025.
John Schulman, Sergey Levine, Picter Abbecl, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. arXiv preprint arxiv: 10r..777.oo 003477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning open language models. arXiv preprint arXiv:2402.0 03300, 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lms. arXiv preprinl xrx::22501.125 2025.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Opennathinstruct-.: Accelerating for math with massive open-source instruction data. arXiv preprint arX1::2400001560 2024.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Christian Walder and Deep Karkhanis. Pass@ k policy optimization: Solving harder reinforcement learning problems. arXiv preprint arxi:2505. 15201, 2025.
Bokun Wang and Tianbao Yang. Finite-sum coupled compositional stochastic optimization: Theory and
applications. arXiv preprint exXx:2222.123990 2022.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256, 1992.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Jumnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arviv preprinl ar Xiv:2500. 11343, 2025.
Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2412.151 2024.
URL https://api: www.stiismntntaaaea.... orgg/orrpu1D:227855222
Han-Jia Ye, De-Chuan Zhan, and Wei-Lun Chao. Procrustean training for imbalanced deep learning. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 92-102, 2021.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xisochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint Xiv:2250 14476, 2025a.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint Xiv:2550. 144 76, 2025b.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Koqing Hc, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and laming zero reinforcement learning for open base models in the wild. arXio preprinl arXiv:2503. 18892, 2025.
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, et al. survey of reinforcement learning for large reasoning models. arXio preprint arXiv:2509.088 088877, 2025.
Chujic Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yugiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. Xiviv preprinl 2rx1 18071, 2025.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Summary of Notations
Notations. For the reader's convenience, we list the core notation used throughout the paper.
(z, a)
A problem-prompt (z) and reference-answer (a) pair.
wo (-(2))
The policy (LLM) parameterized by 0.
Number of IID responses generated per example.
Vi
The i-th response, ~ x-(+2²).
ri
The binary 0/1 reward ra/1 (Vi, a).
p p= pa
The (population) expected 0/1 reward (o a)].
P
The empirical 0/1 reward N Σm= Tr.
N
Number of correct (r; = 1) and wrong (r; = 0) responses.
The (population) expected Pass@K reward.
PK PK.
PK
The unbiased empirical estimator of PK,0-
V:
The log-probability gradient. V l
V_
Average empirical gradient over correct responses:
N+ Eiras V..
V
Average empirical gradient over wrong responses: Eir=o V..
A
GRPO's PPO-Style vs Surrogate Reward Objectives
In the original paper, Shao et al. 2024) present GRPO as optimizing a PPO-style objective (Schulman et al. 2017). This section reconciles that formulation with the surrogate reward objective derived in our Corollary
PPO-style objective. Following common practice Yu et al. 2025a Liu al. 2025 Zeng al., 2025
Deng et al. 2025a), omit the KL regularization term and the length normalization (which is found responsible for length-bias) from the original GRPO objective of Shao et al. 2024). yielding:
N Iv.l
E (ui)e(t)) Fo (1a))
1 N
ΣΣ
min(y, As, Isseltt 14c A1)
(33)
where [:)2 is the clip function at levels a and b, and 7ax no (vi (r, Wi. csss/moas (Mir (r, vi,et) is the policy ratio between the current model (to be updated) and the model lota that generates the responses. For clarity, further omit the clip operation, as done for example in Shao et al. 2024 App. A.1.6). Then the above objective simplifies to
N
1
E By (-bx))
Tae A
N
Note that the expectation here is defined with respect to samples from the old policy Tass following the
TRPO and PPO frameworks (Schulman et al. 2015 2017). Thus, when taking gradients, Te is held fixed; only the numerator Ta depends on
Taking the gradient with respect 0 and using the log-derivative trick, the gradient becomes
Iud
Fitt A; Vo lolo B,, vl (,
By nott (112t)
(34)
N
We refer the reader Mrouch (2025) for complementary analysis of GRPO's optimization dynamics focusing the trajectory the probability success under explicit. KI. regularization towards reference policy. Instead here, focus solely the implicit objective optimized by the gradients correspanding the "advantage term" the GRPO objective.
20

---

Published in Transactions on Machine Learning Research (03/2026)
The fully online setting (corresponding to iterations in (Shao et al. 2024 Alg. 1)) implies no R0ald Then, by 7i,t 1, this simplifies to
E By g A. Ve Vo Tog
(35)
The empirical version of this, + Em A; Vo log (V)llr)) is precisely the GRPO gradient we analyze in Sec. and Eq.
Difference to our surrogate reward. A natural question arisos: How does the PPO-style objective in Eq. 34 relate to the surrogate reward arcsin (po) that Corollary 7 identifies as the function optimized by the GRPO gradient (Eq. (35) )
The resolution lies in recognizing that these are fundamentally different types of objectives:
As mentioned, the PPO-style objective (Eq. 34 is defined over fired distribution Teai
Our surrogate reward identifies the on polpolicy population objective that the gradient in Eq. 35 ascends.
To illustrate this distinction, consider what happens we incorrectly interpret the PPO objective as fully on-policy by substituting #eai = ne everywhere (both in the integrand and the sampling distribution) Eq. 34.. This would yield:
N A
Ry (-12))
(36)
However, recall that A; = (n:-0)/A011))) by construction, the advantages are zero-centered: EKENN A; = 0 for any batch. Therefore, this naive on-policy interpretation yields trivial objective that is identically zero; clearly not what GRPO optimizes.
As established in Corollary [ the actual population objective that the on-policy GRPO gradient (Eq. 35 ascends the surrogate reward: 2 arrcsin /pe).
Perspective Beyond the Fully Online Sctting. This surrogate reward perspective provides principled to interpret and design advantage functions. Our analysis shows how algorithms like GRPO and GRPOk correspond to specific surrogate rewards in the on-policy setting. This link constructive: new advantage- shaping methods, derived from new surrogate rewards (as in Sec. 3 and G). are not limited to the fully online setting. Their resulting advantage scores, A,, can be seamlessly embedded within the PPO-style objective allowing them to be used in practical, multi-cpoch training regimes.
(Eq.
33
B Missing Proofs
B.1 Auxiliary Technical Results
Lemma 1 (Relating fk-1 fx-1- and PK). With
Āx - K
K-1
FR-1
K-1
K-
K
and the conventions NN) ) = 0 when N < and (?) = 1, we have for p =
N+
K -
px = (1 (1-p) p) F = (1
Fk-1
N
21

---

Published in Transactions on Machine Learning Research (03/2026)
Proof. Use the standard identities (x)
N (2-1) and (x) = N (x-1). Then
Nー N (N-
K K
N
(N-11)
K-1
N
KK-1
R k-1
N
K-1
(N-1
N
which implies 7K- N N
JK- -K-1-
Alternatively, using ( = N
N
with N = = and k = K - 1,
N (K 1)
N
N (K -1)
K
K
N-
N-
--1
K
Substitute this into the previous display to get
N
N
N - (K (K-1) 1)
N (K 1)
K
p
N
N
N
K
hence PK - 1)
SK-1- The boundary conventions cover cases
(both weights = 1) and
N
N <K K (ratio- 0).
Lemma (Conditional Coonditionllgraddien identity). Let r Toyi a)) and p = po(x,, a) == (1o With expectations over y no(+(zr), define
E Voloooolylo)) r r= 1] V ())l) r= =0] 0].
Then
A
p(1-p) Vop.
Proof. By conditioning and the log-derivative trick,
logooo)))) r Bir V log [b(()=)]
Vap
p
Using E(V, log [b(x=))] 0,
V ())l)))) rr=0 B[(1 Vo log (y(12))
Vop
p
1 -p
Subtracting gives = v p11-p Vap.
B.2 Proof of Equation
15
The leave-one-out unbiased estimator of Fail@(K-1) excluding response i is:
(NN (N-1))1 1)(1) (1-1)(11pboo)) pwo..)
-0oo PK -1
K
where the leave-one-out empirical 0/1 reward is
if is correct
ploo.
yi is wrong
For correct responses: (N - 1)(1 - (100.0) = (N - 1) (N+ - N7
For wrong responses: (N- 1)(1 - p1oo..)
(N - 1) N -
Apply now Lemma to yield Eq. 15)
22

---

Published in Transactions on Machine Learning Research (03/2026)
B.3 Proof of Unbiasedness for RLOO
Fix an example (z, a) and an index [V]. Let y == fustees denote the other 1 IID samples, and define for convenience the leave-onc-out (K-1 weight WRLOO, := 1 - PK-1 and the RLOO bascline Boo: == N-1 Ejft / "j. As mentioned in the main body, the RLOOK gradient results from replacing in the gradient formula of GREINFORCE in Eq. 14h the reward ry with the vanilla RLOO advantage A, = ri bioo. Thus, the RLOOx estimator for sample is defined as:
g. = WRLOO, (r: - -booo Vi,
(r:-ban
V = Ve log (Vil
Since WRLOO., and blog are functions of y only, they are independent of (ra, V.). Hence, conditioning on yoo,
[ = E B[WiLLOO: 1[((;: - bloo vu 1 y = B[W (Blr.V.l - loo E E[
E[ E (1 PK-10((,,)) a)) (epo(z. a),
where we used E = 0, Elr, V. = Vepa (z, a), and E|WRLLOO = - PK-19 a) since WRILOO, is the standard unbiasod estimator of Fail@(K computed from N 1 samples. Averaging over and multiplying = Vopk,0, proving unbiasedness.
by K yields E[GRLOo (8: (z, a)]] = K(1
B.4 Proof of of Claim Reverse-Engineerie GRPOk
Recall the empirical update for the advantage-shaping method from Eq.
211
G
=WK-
GGRPO
1 PK
p
GGRPO
GRPÓx
PK
1-p
PK
Vp (1))) (f+ P-)
(37)
PK
V
1
Replacing empirical quantities with their population counterparts, the population-level gradient (call it)
G GGrox
is:
= (VI PK
V vo(1 D): By Be)m))() [ - -P-1
G CaRPx
PK
The expectation in the last term becomes
E Ey Va log ()((() | r(y; a) 1] - yy (u) Vo log (o(()>)) r(y, a) = 0).
Using Lemma 2 this equals (epi(e(1- - p)). Thus,
G GRPOK =
TvT PK
P VT-p)
(v/l1 pp) vep)
Vep = ([V
PK
1 p Vap-
(38)
[v(l1
PK
We relate P to PK- Recall that Pk VOpk = K(1 ( K(1 1-)K- Vap.
(1 - k)k
1 - p (1 pk
I/K From this also
Substitute these two relationships back into 38 :
G
GenTox
1
PK
VaPK
K
PK
11 pk 1/K(1 (1--111--1177) 1K/K
1
VOPK
(39)
K V PK Prc(1 (1-Pk)
23

---

Published in Transactions on Machine Learning Research (03/2026)
This is now in the form G GRPOx
We integrate in px to find the surrogate F:
VPk(1- -K) dpx = 2arcsin(yPR) + +C aresin arpsin(2px - 1) C²,
(40)
where C,C² are constants. This completes the proof of the claim.
B.5 Proof of Claim Reverse-engineering Skew-R
Recall the empirical GRPO update (no clipping) from Eq.
GEaPO- = (Ā
V+ L-
We have in the proof of Claim | that, the population level this update becomes
1 Vap.
GaRPO
Vp(1 p)
Then, the skew-R update at the population level is
1 1- Vep. V
V
Gskew-R (1 p) GGRPO
P
We can more conveniently express this as
1 1- 2p
Gekew-R 2vp(1- Vept +
Vap.
2vp(1
We know from Corollary ! that the first term corresponds to surogate reward arcsin(VP), Analogously, the second term corresponds to P(()) for Ω such that ('(p))
1-2p Integrating gives (up to a constant)
201(1
(2()) v p(1 - p) completing the proof of the claim.
B.6
Lewerd-Level Entropy Regularization
We start by computing the population gradient of the per example entropy-surroge reward aaresin( +
X (p), where recall H(p) pplog(p) - (1-p) log(1 p). By direct diffcrentiation
+ X log 1
(--m) Vop p
Gentropy (0; (x,
[Pp(1-) [
Following the recipe of transforming the population objective into an empirical estimate we (1) substitute population quantitics with their empirical counterparts in the multiplicative factor of the gradicnt, (2) substitute the population gradient with the an RLOO empirical gradient. We then gel
Gentropy (0; (xt, a))
+ A log (1))
p( -11-
2) 5 -V_]
Vp(1 P)
Recalling that GeRrO = / ((1 8+ - -V-]. we find
Gentrony (0; (x. a)) = 1+ Aog log (2-))- v v /) GaRro
Equivalently, the ad advantage scores of the new method are
(10) vpl 7) 4
eutropy
+ A log
24

---

Published in Transactions on Machine Learning Research (03/2026)
C Additional Analysis and Discussion of GRPO
C.1 Revines-Enginneerin GRPOK
Taking limit of N + oo and replacing empirical quanlities with population counterparts in Eq. 18 we find
1 1 PK p(1 v(l
GerPOR
1
where the term A Vopp(p(1- p)) is the difference of perret/wroony population gradients defined in Lemma Simplifying,
(1 - p) V p(1 -
GGRPOX
Vop.
Integrating the multiplicative factor (1 K over P the per-example surrogate reward that gives this gradient the incomplete beta function with parameters 1/2 and K 1/2:
1/2 -1)2)
Written in terms of Pass@K reward, the surrogate reward
B(1 (1 1/ 1/2, 1 1/2).
(41)
By symmetry properties the incomplete beta function, this up to (a K-dependent) constant equivalent 1/2, K 1/2).
to -B((1 [(]] 1/K,
For K 1=
L/- Vi(1-r) dr (arcsi(V/pk))
B(1 - (1- pk) -K) 1/2, 1/2) B(p -/px::/2221/2
-
Thus, the surrogate function coincides with that GRPOk (Claim 2 However, for K 2 2, the two surrogates differ. Specifically, GRPOx's surrogate is adaptive to the value of K. See Fig.
C.2 GRPO K=2 vs. Skew-R
For K 2: fk (1 (1-P2 P2 1/N)/(1 p) and fk-1 = (1-1111))) p1) P), which applied to Eq. 18 gives
GGRPOK=2 1 1- 1/N
Vp(1 pp) p). V.
1- p(l -
1 1-6
1-
For > 1, for which 1 1- P2 1/N x - 82 2 p), this becomes the same as the skew-R ob- jective in Eq. 31 : (1 p) p) p) v This raises a question with regards to our reverse-
engineering perspective: On the one hand, we have argued that at population level skew-R maximizes aresin( + vp(1 p). On the other hand, we have seen in the previous subsection that GRPO maximizes ((1 (1 /2:2/333))
First, note that the two surrogates are in fact algebraically identical. To see this, we can express B(1 - (1- (1-11) 1/2 1/2, 3/2) in terms of (using the definition = I (1-p)
[' V -
dr 1," 2
dr /'
2 - dr.
2T
B B(p:
The first integral is aresin((P)) and the second is pp(1 p), confirming the identity. This means there is no inconsistency in our reverse-engineering strategy. At the same time, the simple three-step enwgrrrenngnnering recipe operates by substituting population quantitics with empirical estimates without accounting for unbi- asedness (e.g., via leave-one-out): thus cannot recover the finite-sample (N 1) version of GRPO and arrives at skew-R.
25

---

Published in Transactions on Machine Learning Research (03/2026)
Incomplete Beta vs. Arcsine Surrogate
1.0
K=2
K=4
K=8
2/t Pp
Function Value
0.4
0.2
Pk 0.6
0.8
Figure 5: A comparison of surrogate reward functions, plotted against the Pass reward Pk- The colored lines show the GRPOx surrogate, Eq. (41 for various K values. The black dashed line shows the (nor- malized) GRPOk surrogate, 2 arcsin( PK) (All curves are normalized to reach value 1 at PK = 1.) The surrogates agrec for K = 1 (dashed and purple solid lines coincide), but for K 1 the GRPOx surrogates become concave providing stronnger optimization incentive for hard prompis.
#
This algebraic identity, however, raises a subtle question of interpretation for how we talk about "regu- larization." One reading (a 0/1- 0/1-reward-centric lens) is to say: GRPOk=1 already optimizes the surrogate arcsin(/P): moving to K = simply adds the extra term vp(1-11111111 which one might then call reward-level regularizer
However, for the Pass@K methods, we argue is more appropriate adopt PasstK-centric lens. For Pass@K methods, PK is the actual performance metric of interest, and the 0/1 reward (p) is merely intermediate quantity used in its calculation (since Px =
k)* From this perspective, the objective in Eq. [1 is not a 'rgullarized 0/1 surrogate' but rather a direct surrogate for pk: a monotone transformation of the Pass@K objective itself. Under this view, GRPOx is just doing standard surrogate maximization of the performance metric (analogous to cross-entropy as a surrogate for 0/1 risk), rather than "optimizing p with an added regularizer."
Both interpretations are mathematically consistent; the choice depends on whether one views the opti- mization goal improving the 0/1 reward (and thus interpreting K > 1 regularization) directly optimizing the Pass@K metric (and thus interpreting the objective as a simple, monotone transformation of pk).
C.3 Comparison to Biased Scaling
In Sec. applied the log-derivative trick directly to the Pass@K objective, yielding the expected gradient Gea.DR (0; (z, a)) in Eq. [2).. We then derived Pass@K variants of REINFORCE by using leave-one-out construction to obtain unbiased estimates of the two components in 12 Fail@ ( 1) and Goy1 (0; (z, a)),
resulting the unbiased empirical gradient GRLOOR in Eq. (17
In contrast, Mahdavi al. (2025) apply biased empirical gradient that estimates the Fail@(K - 1) factor (1 2) K-T, The corresponding advantago-score scaling is (up to constant multiplicative factor K)
(1- A
(42)
26

---

Published in Transactions on Machine Learning Research (03/2026)
Comparison Pacc@KK Effective Gradient Weights (N-16)
Alc. ee Grodeet weignt Hog)
K+2
K-4
K-6
K-
K-
2011
10-1
0.25
0.30
0.25
0.00
1.00
0.75
100
GRPO Blased: --p-1
Positive Adivantage (A*)
Negative Aduantage
Figure 6: Comparison of absolute effective gradient weights (log scale) for Pass@K training with 16. Each panel shows a different and plots the magnitude applied to correct (solid) and wrong (dashed) responses under three schemes: GRPOx (Eq. 19h; green), GRPOx (Eq. (21); blue), and the biased scaler (Eq. 42 red). For small أ. GRPOx assigns slightly larger weights than the biased scaler. As á increases, GRPOx enforces a hard zero once á > 1- K-1, whereas the biased scaler decays smoothly. GRPOx applies a symmetric, example-level scale and often yiclds the largest mid-ő weights until PK saturates. Only the GRPOx exhibits asymmetric scaling between correct and wrong responses.
N
Unlike the estimators in Sec. 3. this remains biased even if the advantage-scores A correspond to an unbiased method (c.g., RLOO), because (1 K- is itself biased estimator of Failã - 1). Replacing it by leave-one-ont version (1 (loo) does not remove this bias.
Note that such biased scaling is also present in advantage-shapes resulting from our nawaad-engineering of surrogate rewards in Sec. 5 Even our reverse-engineering operates at the population level. Thus, here, it would tell us that both GRPOk (which applies unbiased reweightings to vanilla GRPO) and the method of Mahdavi et al. (2025). both implicitly maximize the same surrogate reward (as derived App. C.I. In the practical regime of few samples though the two methods could lead to different behaviors. While unbiasedness is intuitively good property, discuss below regimes where the biased variant can be particularly attractive. It is worth noting that for K 2 the biased variant coincides with skew-R.
When a biased reweighting factor can help. There are practical regimes where biased scalings are intuitively attractive. Specifically, for PassuK optimization this includes (sce also Fig. G (1) Compute- constrained N < K. The unbiased Pass estimator in Sec. 3 (and the grouping- based method of Sec. 4 is undefined when K > N, whereas (1- ~) still yields a meaningful signal.
(2) Small N relative to K. When N > K but not by much, the unbiased combinatorial weight becomes exactly zero whenever N < K 1, prematurely killing gradients; the smooth scaler (1 1--p) p) k avoids this carly saturation and provides a gentler shaping A = (1 7) K- A that maintains nonzero updates. (3) Very large N
relative to K. When N > K, fk-1 = fK-1 (N- /(N - 1)) K-1 (1 kk))== the two estimators nearly coincide.
C.4 Comparison to Further Related Work
In the main body, we compared GRPOk to its biased variant from Mahdavi et al. (0225 and to GRPOk as implemented by Chen et al. 2025 Policy optimization for K has also been studied by Walder &
Karkhanis 2025 Their approach differs from both ours and Mahdavi et al. (2025) in two important ways.
First, Walder Karkhanis 2025 do not formulate GRPO-style estimators. The fair comparison therefore with our REINFORCE, RLOO estimators from Sec. which directly target the Pass@K objective. Like us, they construct unbiased estimator of VoPK (I, a), but the structure of their estimator is different.
27

---

Published in Transactions on Machine Learning Research (03/2026)
In our notation (and dropping an overall prefactor K that is common across examples and matches Sec. their per-example estimator for Vopk,o(x, a) has the following form:
3
if To a) 1, - (R-i))/GEE) il roya = 0.
1.
Σ Vo log log :E[N]
(43)
This should be contrasted with our REINFORCE estimator which can be interpreted as a reweighting of vanilla REINFORCE. By contrast, the estimator in Eq. (43) assigns a positive coefficient not only to correct responses but also to incorrect responses. That is, it explicitly takes positive gradient steps in directions corresponding lo some incorrect samples in the batch. Thus, it is also nol comparable to RLOOk.
Besides, it is not immediately clear from 43 how to subtract baseline to reduce variance while maintaining
unbiasodness. Instead, going from REIN FERNFORCEK to RLOOk Sec. g was rather intuitive. We refer the
reader to (Walder & Karkhanis 2025
Sec. 4) for more discussion on variance reduction of their method.
For completeness, we can apply our reverse-engineeringg lens Eq. (43 and verify that it indeed ascends the Pass@K objective in expectation. Rewriting Eq. (43) more compactly using notation for conditional gradients V+ and V_ and for fK-1, we obtain (for a single (2, a))
4 + (1 - D) (1 FR P_
Passing to population quantities (Lemmas and and using PK = 1 - (1-p) this expression becomes
Vag (1 (1
1 PK 1-
Vap
1 (1- 1-p
PK
Vap (1- k) Vep
1 VePK.
p.
1 1-p p
K
Thus, in population, the estimator from Walder & Karkhanis (202 is indeed maximizing PK.0, consistent with our own derivations. The difference is at finito-samples: (i) how gradient weight is assigned across samples (especially incorrect ones), and (ii) how variance reduction is handled at finite N.
Finally, Walder & Karkhanis (0025 explicitly emphasize extending Pass@K-style optimization to non- an-bbinary continuous rewards. It would be interesting future work to extend the surrogate reward lens to such reward settings beyond binary.
D Empirical validations
Here we provide preliminary empirical validation of several policy-gradient methods discussed in this paper on controlled mathematical reasoning tasks. The goal of these experiments is not comprehensive benchmark, updates can be implemented and can be competitive
but rather to demonstrate that the derived/identionn with the strong GRPO bascline in controlled setting. As discussed in Sec. 7 systematic empirical study across lasks, compute regimes, and hyperparameter choices is beyond the scope of this paper and is left to future work.
D.1 Synthetic Data
Setting. We perform lightweight round of supervised fine-tuning (SFT) on Qwen2.5-0.5BBBaa. Yang ee al. (2024) using 1,000 randomly sampled problems from OpenMachlnstruut-2 (Toshniwal et al. 2024) This SFTee model serves the baseline and primarily learns proper formatting and instruction following behavior for mathematical problems.
We Uhen apply RL lo a synthetic arithmetic task of the following form: What is a x x d? The training, validation, and test splits consist of 50,000, 500, and 500 problems, respectively. For training, we use a learning rate of 10-6, weight decay 0.01, sampling temperature 1, unique prompts per optimization step, and N 8 rollouts per prompt (i.e., batch size of 128 sequences per optimization step). The training done on-policy for eight epochs, and without KT. regularization term.
28

---

Published in Transactions on Machine Learning Research (03/2026)
K= 1.0
- K 2.0
==3.0
3.0
4=0.0
k= 5.0
Temperature = 0.8
Temperature = 1.0
100
100
95
95
90
90
Pass@k (%)
85
85
Pass@k
80
80
75
75
70
70
21
22
23
24
25
25
2
2
23
24
2
K
K
Figure 7: Performance of (biased) GRPO (Eq.
42 with K < R). k= corresponds to vanilla GRPO,
while K 2 is equivalent to skew-R. Larger values of K explicitly target higher Pass@K. Empirically, increasing K improves PassOK at larger evaluation values of K, but degrades performance at smaller K. Gains diminish as K increases.
To avoid ambiguity between the PassüK metric used for evaluation and the targeted Pass@K objective used during training, denote K the target value of K used for training, and K as the value used for evaluation. For evaluation we use a test-time rollout size of M 128.
We evaluate the following algorithms: GRPOx (Eg. 118 ), the biased GRPOx implementation of Mahdavi et al. 2025) (Eq. (4),, GRPOk K of Chen et al. 2025 (Eq. (22 our skew-R variant (Eq. 31 and the
entropy-regularicn method (Eq. 332 For unbiased and biased GRPOk and GRPOk, set + K (with K (2,3)) in Eqs. 18
42 and 22 respectively. For the entropy method, set X =1
Higher values of K lead to higher Pass@K. Figure 7 shows the effect of varying the target K during training on downstream Pass@K performance. We observe that increasing Y generally improves Pass@K at larger evaluation values K. However, this improvement comes at the cost of reduced performance for smaller K, suggesting a trade-off between single-sample accuracy and multi-sample coverage. Moreover, the gains diminish K increases, indicating diminishing returns for very large target values.
Algorithm comparisons. Figure compares the performance of the algorithms across values of K used for evaluation and for sampling temperatures 0.8 and 1. Recall that for K = 2, the biased GRPO variant is identical to R.
For k = (Figure Saj, the unbiased estimator outperforms the biased estimator (which in this case coincides with skew-R) across most evaluation values K, suggesting that reducing finite-N bias can be beneficial when optimizing for relatively small target K. Recall from Sec. C.2 that for K 2 skew-R and the biased estimator correspond to the same population surrogate objective, which does not capture the finite-N effects reflected in this experiment.
In contrast, for K = (Fig. 8b), the biased estimator achieves better performance at larger evaluation values of K, indicating that the finite-sample trade-off can shift with K. This may be due to the higher variance of the unbiased estimator in this regime, or because the unbiased estimator too aggressively dampens the advantage scores for high-Ō prompts, as seen in Fig.
Across both settings: the entropy-based method achieves stronger Pass@1, indicating improved singlo-sample accuracy. However, expected, at higher Pass@K values, it underperforms both the biased and unbiased
29

---

Published in Transactions on Machine Learning Research (03/2026)
GRPO estimators which aim to directly maximize surrogate of the Pass@K reward PK,0- Furthermore, GRPOx performs on par with the biased GRPOx estimator.
Temperature 0.8
Temperature 1.0
95
Pass@k (%)
40
Pass@k (%)
90
85
85
Bliwed 2)
Base Of =2)
Unbiased (K=2) 2)
Unbiasec (X=2)
Entropy 1A
Entropy M-
CoEPy (R 2)
CEEd (R=2)
22
21
27
2
(a)
Temperature
Temperature 1.0
P (%)
90
p (%)
85
85
80
80
Biased IK 3)
Based (K
75
Unbiases (K- 3)
75
Unbiased (K
Entropy
Entropy
08
20
22
22
22
23
21
22
22
24
24
K
K
(b)
Figure 8: Performance comparison different methods on the synthetic dataset of Sec. D.1 using N 8 responses per prompt and test-time sampling temperature 0.8 (Left) and (Right). (a)
K 2: The
unbiased GRPOR is best performing across K > 2. The entropy-based method achieves stronger 0/1 reward
(i.c., Pass@1) (b) K 3: The biased GRPO is best performing for K 4. GRPOR'S performance is comparable. The entropy- based method yields again the strongest @ performance.
D.2 Real math problems
Here, we conduct additional initial evaluations on real-world mathematical reasoning tasks. Specifically, we compare three methods: GRPO, P (Eq. (29). K 4), and Skew-R (Fq. (31) Following the training protocol of Deng al. (2025b), we fine-tune Qwen2.5-Math-1155) (Yang al.j 2024 the MATH dataset (levels 3-5) (Hendrycks al. 2021). To accelerate training, adopt dynamic sampling (Yu et al. 2025b), which discards samples with zero advantage and resamples until full batch is formed. All methods share identical reinforeement-leariing hyperparameters. Concretely, training is performed on four A100 GPUs with a prompt batch size of 256 and N = 8 rollouts per prompl. We use a learning rate of x 10 and a mini- batch size of 64, yielding 32 gradient updates per step. Training runs for 40 steps, corresponding to more than two effective epochs due the increased throughput from dynamic sampling. We the sampling temperature to 1.0, the clipping ratio to 0.2, and the KL regularization cocfficient 10-1 We evaluate the challenging ATMF2024, ATME2025, and AMC23 benchmarks using unbiased
30

---

Published in Transactions on Machine Learning Research (03/2026)
Method
Qwen2.5-Matt--1.52 Pass@K
16
32
64
128
256
ATME 2025
GRPO
15.0
26.5
33.6
49.8
56.7
Pass@ K- n-mied
5.6
9.6
14.6
20.1
26.1
33.3
41.7
50.0
56.7
skew-R
5.7
9.7
14.9
20.4
25.9
31.8
37.8
44.8
53.3
AIME 2024
GRPO
11.4
17.7
24.3
30.5
36.7
43.4
50.0
56.0
63.3
Pass@K K-mixed
10.6
16.7
23.5
30.3
37.1
44.3
51.2
57.5
63.3
skew-R
10.2
16.0
22.3
28.3
34.7
42.3
50.8
60.3
70.0
AMC23
GRPO
46.6
59.1
70.0
78.9
85.5
90.2
93.7
96.0
97.5
Pass@K-mixxo
69.4
78.4
85.3
90.2
95.2
98.5
100.0
skew-R
15.7
58.3
69.0
77.6
84.5
90.0
94.8
98.6
100.0
Average
GRPO
21.3
28.9
36.4
43.3
49.6
55.7
61.7
67.3
72.5
Pass@K-mixed
20.5
28.1
35.8
42.9
49.5
56.1
62.7
68.7
73.3
skew-R
20.5
28.0
35.4
42.1
48.4
54.7
61.1
67.9
74.4
Table 2: Performance comparison of different methods (see text) on AIME 2025, AIME 2024, and AMC23
under Qwen2.5-Me 1ron2.5-Maate --tth----.
P accuracy with temperature 1.0, which better reflects exploration settings. Final performance is reported using the unbiased Pass@K metric with test-time rollout size of M 256.
Table 2 shows that Skew-R achieves strong average performance at K = 128 and K = 256, matching surpassing pass@K-mixed. Since large-K primarily reflects the ability to discover diverse correct solutions rather than improving single-sample accuracy, these gains suggest that Skew-R enhances exploration. This observation supports our interpretation that reallocating gradient emphasis toward low-success prompts leads broader solution coverage. We expect these effects be amplified when the rollout-size N is larger; such ablations out of scope and we leave them to future work.
31


Reviews:
[{"review_id": "m4qaVExpn1", "reviewer": "Reviewer_Wzcq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis work proposes to unify the two views for optimising Pass@K objectives in Reinforcement Learning with Verifiable Rewards (RLVR) by showing that advantages shaping techniques derived from GRPO, and direct optimisation algorithms derived from REINFORCE, are trying to address the same problem. This unified view is built on the introduced surrogate rewards. Furthermore, this work propose to a way to derive both existing and new advantage-shaping methods from a chosen surrogate reward, and vice-versa.\n\n**Strengths**\n\n- (S1. readability): This work makes a lengthy introduction on concepts in RLVRs for readers not so familiar with the related subjects. This work is well motivated and is generally well written. I especially find Table 1 very helpful to understand the newly derived methods using this work's proposed framework. \n\n- (S2. importance): Unifying the two views in related works can be a key step in the developing new algorithms in the future. The reasons behind where algorithms down-weight easier examples while over-weight harder ones when optimising for the pass@K metric can be generalised by the bidirectional framework introduced in this work.\n\n\n**Weaknesses**\n\n- (W1. Readability): the flip side of S1, this lengthly introduction sometimes may cause confusion of where the central contribution of this work is proven. I had to re-read quite a bit to find where the contributions listed in sections 1.2 are proven. \n\n- (W2. Missing introduction): For the lengthy introductions on the REINFORCE-style direct optimisation techniques and advantage shaping techniques, the surrogate rewards were not properly defined. Some texts addressing how surrogate rewards are utilised can be included. Furthermore, how to use a chosen surrogate reward to derive new advantage-shaping techniques can be explained further in detail. Currently it is only briefly explained in Section 1.2. \n\n- (W3. Effectiveness of derived algorithms): I think this is the main weakness of this work. Although this work is more about theoretical contribution, I think it would be good to have some experiments, for example accuracy on some toy tasks, when directly optimising for pass@k using the proposed methods (i.e. $GRPO_k$). I think this would significantly strengthen this work by showing the effectiveness of proposed bidirectional relationship. Basically, to show that newly derived algorithms can work. \n\n- (W4. Guidance for future practitioners): I feel somewhat confused on how to utilise these claims going forward. Given the new bidirectional relationships, where should we start if we were to design new algorithms? To my understanding is to somehow choose a surrogate reward; it would be great if the authors can include some starting points on how to do so.", "claims_and_evidence_yn": "Yes", "claims_explanation": "To my understanding these claims are well supported by theories introduced in relevant sections. One small caveat is that the new algorithms, introduced by the framework of this work, were not empirically verified but only theoretically.", "audience_yn": "Yes", "audience_explanation": "Researchers studying LLM agents especially RLVR would find this work relevant. This unified view and the introduced framework can be quite impactful in deriving new algorithms.", "requested_changes": "- (R1. missing references): There are quite a few missing references in-between the manuscript, especially regarding proofs referring to the appendices. For example, proofs in Appendix B are not referred/mentioned at all in the main texts. This somewhat reduces readability. \n\n- (R2. add description): Address W2 by adding some descriptions regarding surrogate rewards, and how these surrogate rewards are utilised/can be utilised. Currently, the only description is in Section 1.2, which can be explained more in detail.\n\n- (R3. add toy experiments): Address W3 by adding some toy experiment to show efficacy (doesn't have to be too big an experiment) comparing derived direct pass@k optimisation versus related work (i.e. versus $\\widetilde{GRPO_k}$), or the derived 0/1 reward optimisation (i.e. $\\text{Skew-R}$) versus related works.\n\n- (R4. guidelines): It would be great if the authors can address W4 by including some guidelines for practitioners. \n\n- (R5. minor, clarity): The captions in the figures can be improved by adding some conclusion in the captions to facilitate faster understanding. Especially Figure 3. \n\n- (R6. minor, clarity): I might have misunderstood, however, the main discussions (sections claim 5) and plots (figures 1, 2, 3) assumes N>>K. In the proofs, this assumption is not clearly written. Furthermore, it is unclear what specific number of N and K can be expected. It would be great if the authors can clarify in the texts.", "broader_impact_concerns": "None as far as I am concerned.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7UXZlrZrYU", "reviewer": "Reviewer_QEqk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies policy-gradient optimization for Pass@K objectives in reinforcement learning with verifiable rewards. It shows that direct Pass@K policy gradients and heuristic advantage-shaping methods (e.g., GRPO^K) can be unified through a surrogate reward perspective, where advantage shaping is asymptotically equivalent to optimizing smooth transformations of the Pass@K success probability. The authors further propose a forward-engineering framework for designing new advantage-shaping methods from surrogate rewards and interpret hard-example upweighting as reward-level regularization, with an entropy-augmented variant as an illustrative example.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes — the paper’s claims are supported by clear and technically sound theoretical derivations, with assumptions and limitations explicitly stated.", "audience_yn": "Yes", "audience_explanation": "Yes — the results are directly relevant to researchers working on RLVR and LLM training.", "requested_changes": "1. It would help to add a short discussion of how the surrogate-reward view should be interpreted in the settings people actually use (e.g., small $N$, and GRPO implementations with clipping and multi-epoch updates).\n2. (optional) Beyond the analytical weight-shape plots, a simple toy RLVR simulation showing how the different reweightings change the actual update behavior (and/or Pass@K trajectories) would make the paper more concrete.", "broader_impact_concerns": "N/A", "additional_comments": "**Strengths**: The paper provides a clear and technically sound unifying perspective on Pass@K optimization and advantage shaping in RLVR. The surrogate-reward viewpoint is elegant and helps reconcile several recent methods under a common framework. \n\n**Weaknesses**: \n\n1. In practice GRPO is usually run in a PPO-style way (old-policy rollouts, multiple epochs), but the surrogate-reward story is derived in a clean fully-online/on-policy setting. \n\n2. Several of the paper's main correspondences rely on large-$N$ plug-in asymptotic approximations. In typical training, however, the group size is often small (e.g., $N \\in \\ {8,16\\ } $), so the empirical success rate $\\hat{\\rho}$ can be noisy and the resulting reweighting can deviate from the asymptotic picture. \n\n3. The forward-engineering recipe makes it easy to map a chosen surrogate $F$ to a reweighting rule via $F'(\\rho)$, but the paper does not spell out practical design criteria for picking $F$ (e.g., avoiding overly sharp weights near $\\rho\\approx 0$ or $1$, or robustness to plug-in noise when $N$ is small). A short discussion of surrogate-selection trade-offs (stability vs.\\ aggressiveness of reweighting, sensitivity to plug-in noise, behavior near $\\rho\\approx 0/1$) would make the recipe more actionable.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pefGLl5goY", "reviewer": "Reviewer_Rab9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a theoretical bridge between Reinforce-style gradients to GRPO for optimizing the Pass@K metric in LLMs.\n\n*Strength*\n\n1. Method is indeed novel and theoretical grounded. \n2. Applied to different methods from reinforce, rloo to GRPO.\n\n*Weaknesses*\n\n1. Some derivation isn't rigor. See my detailed comment below.\n2. Presentation issue exists. See my detailed comment below.\n3. Pass@k reward has already been utilized RLVR, but this work didn't justify the effectiveness of the proposed method in actual training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "To some extent, yes. \n\nWhile the paper provides a compelling framework for unifying Pass@k objectives, the central claims of theoretical equivalence are occasionally undermined by heuristic derivation steps and notation inconsistencies that introduce statistical bias not fully resolved in the proofs.", "audience_yn": "Yes", "audience_explanation": "Yes.\n\nLLM RLVR with pass@K reward is a timely topic, and many readers will be benefited from the theoretical analysis in this paper.", "requested_changes": "1. Writing issues, please ensure rigor as this seems to be a more theoretical work\n- Please correct all the citation format, i.e., using \\citep and \\citet properly, currently one affects reading. \n- The definition of Pass@K sampling isn't clear in terms of RLVR practice (page 2 below section 1.1.2); it would be better to define sth like \"independently sampling K response in xxx temperature from prompt x\". The current explanation of repetitive prompting reads less rigorous.\n- Please rigorously define all notations at the beginning of the main-body work (move the one from appx to main body) and double check all the expressions and phrases to make sure they are **consistent**. What do you mean by \"per-example\" expected pass@k reward, as $x$ is defined as \"problem\" instead of \"example\".\n- I don't quite follow the derivation in eq23 to eq24, somehow confused by $\\hat{f}_{K-1}$ and $\\omega$.\n\n2. Theoretical rigor \n- In eq6, there's a sample-level coefficient $\\frac{N}{N-1}$ and you say you can \"ignore it\" below the line under eq6, but I don't quite understand why this is removed in GRPO eq8? Simply dropping it, the estimator then becomes biased by a constant multiplicative factor.\n- For eq15, those quantities are over batch, which $\\hat f$ and $\\hat \\nabla$ are calculated over the same batch. How does this ensure the unbiasedness?\n- Sec5.2 claim 2 might have some issue. Could you please clarify the following questions: If I understand correctly, the derivation relies on substituting population quantities ($F'(\\rho)$ and $\\nabla \\rho$) with empirical, batch-dependent estimators ($F'(\\hat{\\rho})$ and $\\hat{G}_{RLOO}$). Since these two terms are computed from the same sample batch, they actually are statistically dependent, so it yields a biased estimation, which seems to be a heuristic approximation? How it makes sure that this batch-dependency with RLOO estimator to maintain independence between the weight and the sample $y_i$? If so, can the error towards population-level surrogate be bounded?\n- GRPO has KL, clipping and importance ratio sampling. I didn't find any assumption or clarification regarding the GRPO objective that you are analyzed until page 12. This should be moved to the beginning to avoid misleading confusions.\n\n3. Actual training effectiveness.\n\nCould you provide some actual experiment by comparing to other pass@K RLVR method (e.g., SimKO)? The readers will be more interested in how the derived surrogate would work actual training. This is similar to how DPO simplifies from RLHF, the authors not only show the nice reparameterization of RLHF objective but also demonstrate it via actual experiment result.", "broader_impact_concerns": "N/A", "additional_comments": "There are some parts that I didn't yet follow and may need author's clarification to further carefully check the full proof, will update by adding new thread of comment if I have any concerns or questions.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)