GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
Authors: Kaichen Zhang
Yuzhong Hong
Junwei Bao
Hongfei Jiang
yang song
Hong Dingqian
Hui Xiong
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Post Training
Reinforcement Learning
Reasoning
GVPO
GRPO
Primary area: deep_learning

Abstract: Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.

GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainie
Kaichen Zhang!.
Yuzhong Hong²
Junwei Bao Hongfei Jiang?
Yang Song" Dingqian Hong" Hui Xiong
Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou)
Zuooybbang Education Technology Department of Computer Science and Engineering, HKUST
Dahhangb@cconneee wareeecttt.t.htt ust.hk eugene h. h.giitee t@gmail com baojunwei001 @iweei000@mmil..... sagmail..cmm .com
{j (jinghonggeei, songyang, hongdingginn mongddnggaan/ezuuyebaaaa @zuoyebang.... com Mlonghui@ust.o
Abstract
Post-training plays crucial role refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post- training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adop- tion. As a next step, present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring aligmment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) guarantees a unique optimal solution, exactly the Kl-cconttriined reward maximization objective, (2) supports flexible sampling distributions that avoids importance sampling and on-policy limitations. By unifying theoretical guarantees with practical adaptability, GVPÓ establishes new paradigm for reliable and versatile LLM post-training.
Introduction
Large language models (LLMs) [59, trained on extensive datasets, exhibit impressive general- purpose capabilities, yet their practical utility and aligmment with human values depend critically post-training [46] refinement. While pre- pre-training (60] 33 equips LLMs with broad linguistic patterns, post-training techniques-Sut such as supervised fine-tuning (SFT) 32 and reinforcement learning
30 from human feedback (RLHF) K are indispensable for adapting these models to specialized applications and ensuring their outputs align with ethical, safety, and user-centric standards.
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large - -Rich Sutton, 2024 Turing Award winner
margin."
This principle outlined The Bitter Lesson
which advocates for scalable, computation-dies -diven approaches is exemplified by recent advances in post-training, particularly Group Relative Policy Optimization (GRPO) 39 Diverging from conventional reinforcement learning frameworks (37 that depend on training separate value function, GRPO directly optimizes advantage by standardizing reward scores across samples. This approach eliminates the need for an auxiliary value model, which typically demands computational resources comparable to those of the policy model itself. As result, GRPO significantly reduces memory and computational overhead, enabling more efficient sampling and scalable training. Deepseek-Rl
[ leverages GRPO and achieves significant performance.
*Corresponding Authors. Code available at https///gttubb.com/ss//c//
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VeLovpo(0) = Ey-r, V log no
Valroing Gradient (0)
Ex VV log no
π g
Thuman
R R- log
Bynn.
1 log
log
Ey
R log
TUPT
Tep
Constant
(oLovro
loss
Orgninn orginnng Pa
Vo VoEn,yx [Ro y Ro) - (R B. R2)2
aaree n [aa(Tol(Tr)
Rg EyRo log T6.
Ey log 7i
Re = R. TT = π
x
2
(80 2.
VeLovro(0) Vob
Term (R E, log ne llggh R. Low no II
I(R Ey log mo
Term Con(log ne, log mo))
Term
0.5V (log my)
Fe
Cov(log ITo log
0.5V Ma5Var(logg) T9)]
Eavaa
Eg
Bavzo
Figure 1: Three equivalent loss functions of GVPO offer distinct interpretations: (1) The Negative log-Likelihood perspective (top) illustrates that GVPO accommodates broader sampling distributions compared to conventional policy gradient methods; (2) The Mean Squared Error interpretation (middle) reveals GVPO's unique optimal solution, which simultaneously maximizes reward under a KL constraint; and (3) The Reinforcement Learning viewpoint (bottom) highlights GVPO's implicit regularization terms that ensure stable policy optimization. We assume
for simplicity.
However, GRPO has been documented experience issues with training instability in prior literature (54 [28]. Specifically, GRPO is highly sensitive to its hyperparameters, such as the clip threshold and the KL--divergence coefficient. These limitations undermine the robustness GRPO and hinder its broader practical adoption. As next step, we propose Group Variance Policy Optimization (GVPO). a novel approach for reliable and versatile LLM post-training.
Our analysis begins with a key observation: post-training SFT, Reject Sampling 488) and GRPO-share a unified mathematical structure their loss gradients 39 10 Specifically, each method's gradient can be expressed as weighted sum of the gradients of the log-likeele likelihoods of responses. This unified framework reveals that we can directly design weights to encode preferences positive weights amplify gradients for favored responses, while negative weights suppress disfavored ones, with magnitudes modulating strength of preference.
crarithms-including but not limited to
Motivated by the success of Direct Preference Optimization (DPO) 34 which utilizes a closed- form link between reward models and the optimal policy under KI. div-dieeeee constraints 23 explore how to leverage this analytical relationship. A central obstacle arises from the partition function in the closed-form formula, which requires intractable expectation calculations over all possible responses. To address this, we identify a critical condition: when the sum of assigned response weights within prompt group equals zero, the partition function becomes invariant across compared responses, effectively canceling out in the policy update rule. This insight eliminates the need for explicit estimation of the partition function, thereby enabling deployment of the closed-form optimal policy while retaining its theoretical advantages.
Based the previous findings, we design GVPO's weighting scheme where the gradient weight of response in a group is the difference between the central distance of implicit rewards-which derive from the current policy and the reference policy-and that of actual rewards, illustrated in Figure [ (top panel). The loss is computable because the sum of weights in a prompt group equals zero.
We demonstrate that GVPO loss function carries physically meaningful interpretations. Specifically, we establish that its gradient equals that of a mean squared error loss measuring the discrepancy between implicit and actual reward central distances, illustrated in Figure [
(middle panel).
Furthermore, the loss function GVPO can be decomposed into three distinct components, as visualized in policy, and (3) the covariance between the current policy and reference policy. The first component directly promotes advantage maximization by prioritizing responses with higher expected returns.
panel): (1) group-relative reward term, (2) the variance of the current

---

The covariance term acts as regularizer, mitigating excessive deviations from the reference policy to ensure stable policy updates. Meanwhile. the variance term encourages moderate entropy, thereby naturally balancing exploration and exploitation. We systematically analyze GVPO's structural similarities with conventional policy gradient reinforeement learning methods.
We demonstrate that GVPO offers two key advantages:
GVPO has unique optimal solution, which coincides precisely with the optimal solution of the KL-constrained reward maximization. This guarantee confers a significant theoretical advantage over DPO. Prior work 4 [6] highlights that DPO may fail to converge to the optimal policy for the co-ctrnrrr reward maximization problem, because of the inherent flaw Tradley-Terry model [52]] In contrast, GVPO guarantees that its loss function is aligned with the original constrained optimization problem, ensuring convergence to the globally optimal policy. This theoretical robustness positions GVPO as more reliable method for policy optimization.
GVPO supports flexible sampling distributions that avoids importance sampling and on-policy limitations. Beyond the common practice of sampling from the previous step's policy, GVPÓ retains theoretical guarantees for the unique optimal solution under any sampling distribution satisfying mild condition. This property provides notable theoretical advantage over policy gradient methods [44). Unlike on-policy approaches [4] (0)] which require fresh trajectories for updates, GVPO facilitates off-policy training using reusable or heterogeneous datasets. Further- more, in contrast to off-polic poliiolly methods (39,37 (37 reliant on importance sampling, GVPO inherently
avoids gradient explosion risks without introducing bias through clipping techniques.
As result, GVPO emerges competitive online RL algorithm, capable of leveraging diverse data sources, sustaining stable policy updates, and preserving convergence to optimality.
2 Preliminary
Large language models take a prompt x as input and generate a response as output. A policy To(unl y<e)) with parameter maps sequence of tokens generated (x: and y<ee) probability distribution over the next token Yt- We also denote Th (y(y)z) as the probability of generating the response y from æ. A reward model R(x, y) scores the response y as the reply to the prompt x.
reward model can explicitly be evaluation ratings of human beings; or a trainable function that implicitly reflects human preferences;
predefined rule, such correctness, accuracy.
The general purpose of post-traine post-training of large language model is summarized as following: Given an initial policy To.... a dataset of prompts x ~ D, reward model R, the objective is to train new policy Ty that generates responses with higher rewards, that is, maximize Eegz a) R(x,y).
2.1 Towards better computation leverage in post-training
The initial stage of large language model post-training typically involves Supervised Fine- Tunnng (SFT) (32. In this phase, dataset comprising input prompts paired with exemplary responses y used to optimize the pre-trained model. The training minimizes the negative log-likelihood loss:
E (0) =
Σ log Tog(())
(x.y)E
Recent advancements, such as GRPO 39 56). better leverage the computation by incorporating multiple sampled responses with their standardized reward scores as weights:
(ORPO( = Σ Σ Clip
To (u(l))
Ra (R))
log mo (x) - BKL(mol|mn (2)
(a.()R))ED
Rewards below average lead negative weights, minimizing their likelihoods. However, minimizing likelihood can result in unstable training 11. Additionally, off-polic training of GRPO, which becomes unstable when no significantly deviates
involves importance sampling with the weight meatd
from Tel potentially causing gradient explosions. To mitigate these issues, GRPO uses gradient clipping and KI. constraint between the updated and reference policies. Nevertheless, empirical results [54] 17
show that GRPO still exhibits training instability, which undermines its performance.
3

---

2.2 Optimal solution to the KL-constrained reward maximization
In human preference alignment scenario 32 the ideal reward model would directly reflect human evaluative judgments. However, obtaining explicit human ratings is often unavailable in practice. Instead, contemporary approaches typically leverage pairwise response preferences (æ, yw: Wt where You denotes the preferred response and yı the dispreferred response to prompt æ, to approximate human preferences through reward model training. The resulting reward model subsequently enables policy optimization through the following KI.-regularized objective:
maLm E gr [[x v)] 3D (TO (y/ [T) (w]x²]
(3)
where 3 > 0 controls the divergence penalty from policy πT. In preference alignment scenario, Ter set to reference policy T
Rather than employing separate reward modeling and policy optimization stages, DPO derives single-stage training paradigm by exploiting the analytical relationship between optimal policies and reward functions. The optimal solution Equation satisfies:
(u|x)
((x) no (ul R(u,w/9
(4)
which implies the corresponding reward function:
R(x, u) = 3 log (g/lr) To (ulæz)
+ B log ((x)
(5)
where ((x) -, Te (e represents the partition function. DPO circumvents explicit
computation of (zx) by substituting the reward expression from Equation 5 into the Bradley-Terr Terry loss [5], yielding the final objective:
To To (x) Traf (yrol [x)
To ( (x) Tref (y1
CDPO (0)
Σ DED
log (33 log
3 log
The success of DPO has been proven be both efficient and effective. We attribute its achievements to its direct incorporation of the optimal policy's ' closed-form solution into the training objective.
2.3 Unified framework of post-training
As far we know, post-training algorithms share unified framework (39 10 in which their losses' gradients share same format:
Vol(O)= =
Σ
Σ
wi Va log (ynl2)
(I. DED 1=1
SFT only has one response per prompt, and its w1 1. GRPO's essential weights are the standard
scores of its rewards in a prompt group. Though it is not obvious for DPO, its gradients also share the same format, in which w.e aB3 log
xo( # (m)((() (2)(=) 3 log mree (ye (a)) and un Ww- Such the unified framework post-training holds, because of the chain rule derivatives.
ne (u (x)
Algorithm 1 Group Variance Policy Optimization
Require: initial policy no: prompt distribution D; hyperparameter 3
1: for step = 1..... n do
2: Sample batch D, from D
3: Update the old policy model Ke.u < x
4: Sample responses (y)k- ~ (-(x)))) for each prompt e Db
5: Compute rewards {R(x, y:) 1 for every sampled response y/ and prompt
6: Iteratively update policy by minimizing the GVPO loss (Equation 8 setting тे Te
7: end for
8: Return m

---

3 Group Variance Policy Optimization
3.1 Motivation
The unified post-training framework (Equation incorporated through the assignment of weights w0ç.
7 indicates that response preferences can be directly
To determine appropriate weights, we draw inspiration from the success of DPO, In particular, we aim to exploit the closed-form relationship between rewards and the optimal solution to the KL-constrained reward maximization objective: Ro(æ, y) = 3 (og(=o(yla mo (ylz)) + 8 log (x:).
However, the closed-form formula contains a partition function ((x) that is expensive to estimate in practice, because the function requires calculating the expectation of all possible responses.
To address this issue, we identify critical condition: when the sum assigned response weights within a prompt group equals zero, Ei Wi 0, the partition function becomes invariant across responses: Σ to Ro Ro, vz) Σ w log log(To(Us)))/ (1i(z))).
3.2 Method
Build on this insight, propose Group Variance Policy Optimization (GVPO), whose gradient weight w; is the difference between the central distance of implicit rewards-which derive from policy Th and policy ne and that of actual rewards. Formally, GVPO's gradient (f)
(U( log To {e((U))>)) Vo log (V())))
Σ
(R(a, vi) R(x, (urH)) -(log
Tgg (u/lz)
TT TO (fvi)(z)
(n.(wx))ED
(8)
where R(z, (1:1) 1 E R(x, yı). and log
mel (
E log
no (a( (vila))
We note that
oo Live m))
GVPO's gradient satisfies E W/ = Algorithm shows our proposed algorithm.
We demonstrate that GVPO's objective carries physically meaningful interpretations:
V VoLovro (0)
k
Σ
R(x, (vi1)) (Ro(x, Ro (br1)) log No (v)la) (R(z,y))
&
Σ
(R(x,0)) R(x, (mi))) (Ro(s, vi) - Ro(o, (nr Ro vs)
n. (1A) ==1
Σ
(R(z,v) v.) -R(x, (v.))) (Ro(x,y) (c) - Ro(x, (o(o:tt - R2 {vcH))
1. 1=1
k (Ro(2, v:) Ro( (vr))) (R(x, yi) R(x, {1:1)))2
Σ
2
[ 1=1
The first and second steps hold because 0 log Z(x) can cancel out. The second step holds because Σk_ w, VoR(z, { O. The third step holds because (zf(=)²² 2f())
Essentially, we have established that GVPO's gradient mathematically equals that of a mean squared loss measuring the discrepancy between implicit and actual reward central distances. Intuitively, when implicit rewards equal actual rewards or with a constant group shift, the GVPO's loss is minimized. This interpretation also implies that the response with higher actual rewards in group is also encouraged to have higher implicit rewards, indicating higher log se(w]]>)
Furthermore, by rearranging the mean squared error loss, we can derive a variance-based formulation, which represents the "Variance" term in the name GVPO:
V 2
Σ (
1 RR( v.) R(x, v4)) Re (x. (vr)) R(x, {v.h)
1=1
5

---

3.3 Theoretical guarantee
We show that GVPO has an unique optimal solution, and this unique optimal solution is exactly the optimal solution of reward maximization with KL constraint (Equation 4. Formally,
Theorem 3.1. The unique optimal policy that minimizes Lovro(0), defined
(0) BE (1 By Ro(I, v)) (R(zz, y) By R(x, R(1, (9)
is given by mo(y)>) = n ( [r) ,ं nor (y) R(z.)) for as = mp-
We prove the theorem by establishing both necessity and sufficiency, provided in Appendix B.1
Theorem 3.1 implies that the parameters minimizing (ovro(()-guaraattee)) to be the sole global optimum also maximize the expected rewards while maintaining proximity reference policy.
The uniqueness of the solution ensures the optimization landscape is well-behaved, avoiding sub- optimal local minima and guaranteeing convergence to single, interpretable policy that optimally balances reward maximization with behavioral consistency relative the reference. Consequently, this theorem bridges GVPO's practical algorithmic performance with theoretical guarantees.
Theorem 3.2. The Theorem 3.1 also holds for any sampling distribution Ta satisfying Vr, {y To (yl x)) 0} T 0}.
Beyond the conventional practice of sampling from the reference policy (T, = Tor), GVPO retains the theoretical guarantee unique optimal solution under any sampling distribution that satisfies mild condition. This condition is readily met by any policy where ( x) 0, criterion inherently fulfilled by contemporary LLM policies utilizing softmax decoding.
The Theorem 3.2 of GVPO opens new methodological avenue for off-policy LLM post-training. Prior off poolcy methods have relied heavily on importance sampling [47], which suffers from two key limitations: (I) when diverges substantially from T., the importance weight iee can become either excessively large or vanishingly small, destabilizing training: and (2) when sampling involves heuristic or non-paarmtttiic components, Tr becomes intractable compute, thereby prohibiting techniques such as experience replay [9] in modern LLM post- post-training In contrast, GVPO supports highly flexible off off-policy sampling strategies while maintaining strong theoretical guarantees, enabling more robust and practical post-traie tettrraining paradigms for large language models.
3.4 Discussions with DPO
We begin by analyzing the foundational commonality between GVPO and DPO: both methods integrate the closed-fome form solution to the reward maximization problem under a KL divergence constraint into their training objectives. This integration establishes a direct relationship between learned policy xa and the implicit reward function Rg, yielding two key advantages:
It ensures an optimization process that inherently respects the KL divergence constraint, thereby preventing excessive deviation of the policy Ty from the reference policy Tref-
It reduces the joint optimization over policies and rewards to simpler problem focused solely on rewards. The latter is more tractable, as it requires only aligning the implicit rewards Rş (z., y) with the true reward function R(z, y)-
To design effective methods leveraging this closed-form solution, two critical insights emerge:
1. Computational Tractability: The method must avoid intractable terms such as the partition function (xr). For instance, naive loss L (Ro(r. y) R(x, y))² fails because Rs (a,, y) im- plicitly depends on 7(x), which is computationally infeasible to estimate. DPO circumvents this by adopting the Bradley-Terry preference model, where Z ((x) cancels out in pairwise comparisons. GVPO proposes novel zero-sum property across groups of responses, enabling cancellation of Z (xx) in broader multi-sample scenarios.
2. Alignment with Desired Optimality: The loss function must enforce meaningful convergence.
For example, minimizing L = Σ log ze(z.))
R(x,y)) yields a suboptimal solution Frer(a.9)

---

Ra(a,3) y) R(æ,8) y) + 5 log (zx), which deviates from the true reward R(z, y). A well-designed objective must avoid such misalignment. The method should adapt to available supervision. DPO leverages pairwise preference data without explicit rewards, while GVPO generalizes to group-wise responses with reward signals.
Moreover, GVPO demonstrates stronger theoretical robustness compared to DPO:
Prior work 4 T6 highlights that DPO may fail converge the optimal policy for the KL- constrained reward maximization problem, because of the inherent flaw of Bradley-Tere model This arises because the DPO loss admits multiple minimizers, and its correlation with the true reward objective can diminish during training [45
152
In contrast, as formalized in Theorem 3.1 and Theorem 3.2 GVPO guarantees that its loss
function is aligned with the original constrained optimization problem, ensuring convergence to the globally optimal policy. This theoretical robustness positions GVPO as more reliable method for policy optimization in practice.
3.5 Discussions with GRPO and Policy Gradient methods
Structural similarities. Sceing the forest for the tree, we compare GVPO not only with GRPO but also with the broader family of policy gradient-based based RI. methods, beginning with their superficial structural similarities.
For simplicity, we assume 3 1 without loss of generality. Then (o) Y.
(-1z) (1a)) v) - E, Ra v))² - 2(R((z, v) R,, y) Ro v)
" [Var (log mo) - 2Con no, log To 2 R(x, v) - Rz, B)) log (v)rr)
(10)
-2E (R(,,, F.g R(r, u)) log (v)l>) + Cov mp, log ).) - 0.5V arol mo)]
where Var(log nO) )= (log ( (v) By log (v )z)² and Cov ma. log ) = (log (y)z))
Ey log no (y)zz) (log no (w|x)) By log oe. (ylr)) As shown Equation TO
the term (R(x,y) F, R(ar, y)) log (ylar (r)) encourages advantage maximization. Unlike conven- tional policy gradient methods that rely on explicit value function approximation (36), GRPO directly optimizes advantage by standardizing reward scores across samples. distinction lies in GVPO's 's omission of standard deviation normalization Prior research [29] has also demonstrated that such scaling introduces bias by conflating prompt-level difficulty with reward signals.
the term Cov(log Na. log Tgr)) serves to constrain deviations of the policy Ty from policy we. corresponding to [xI.[Te|le [T Moreover, in GVPO, where me = Teau this term essentially aligns with the trust-region constraint (35), that ensures robustness between policy updates.
the term Vloglog =e) strikes a balance between exploration and exploitation. We juxtapose this term with entropy regularization -Ey log π (y (x)
Increasing entropy encourages diversity by driving the policy toward uniform distribution, but risks suppressing the likelihood high-quality responses. Conversely, reducing entropy concentrates probability mass on narrow of outputs, diminishing diversity and potentially inducing entropy collapse. Consequently, entropy regularization proves highly sensitive to its coefficient, complicating practical implementation.
In contrast, V ar (log πo) circumvents this issue without requiring ad-hoc tuning by enabling scenarios where some responses receive zero probability, while other responses retain parable probabilities. As the term R - R maximizes advantage, undesirable responses will receive zero probability, while favorable responses will maintain similar probabilities.
To illustrate the benefit, consider toy example involving generation over the tokens a, b, c, d, where To (a) = (b)
and (c) (d) (e(e) = 1/3. In this case, Var (oo()oo(TT)) = 0, indicating minimum variance and thus no penalty on this distribution. In contrast, entropy
regularization only reaches its minimum either when the distribution uniform or when
one-hot, depending whether one encourages higher or lower entropy.
This omission is not heuristic design but rather consequence of the theoretical formulation.
7

---

In-depth similarities. In addition to structural similarities. we now highlight their key theoretical and practical distinctions. Modern policy gradient methods [35
optimize the expected reward under the current policy Ty while constraining updates to avoid excessive deviation from the previous policy Te.-- This typically achieved by optimizing a objective that combines the reward R(x, y) and Kl.-divergence penalty term DKL (Ta|l Teaa) yielding the gradient expression:
37 39
To (1)y22 23)
V (l [R(z, v)] - DKL [To Teaall = V VelE re(ulz)(((z)) - log
To. (y/
(11)
To (u(())) ne (2)
EE (o (R(x; v) log
1) log (1(
However, estimating this expectation requires on-policy sampling from Te (ula), leading to low sample efficiency- -ell-eooeeeneed limitation of policy gradient methods. Reusing stale samples from prior policies introduces bias, degrading optimization stability and final performance.
To mitigate this, prior works
39
employ importance sampling, rewriting Equation as:
xx)
To (ylxr) To (y/z))
E By (vll)) 00a (r)
R(x, y) log
Vo log (y)l)))
(12)
This allows off-policy gradient estimation using samples from z0a.- However, ()(u
Rea becomes
unstable when ny deviates significantly from Thl. risking gradient explosion. Heuristics like gradient clipping (37 address this at the cost of biased gradient estimates, undermining theoretical guarantees.
GVPO circumvents these issues because it does necessitate on-policy sampling in the first place. By rearranging Equation II we observe that the policy gradient can be expressed as:
To (y) x)) Teau (3)(z)
Thau (y x } V log To(y))).
To( (x)
(2.yr-m(((la)) R(x,8) y) log
Byy-m((ula)) [[xx,) y) log
where the baseline term (subtracted expectation) arises because Eyyrro (cVo log To (y|z²) = Voc for any constant c. Crucially, the GVPO gradient generalizes this structure, (oLGvpo(8)=
Ez, Be.grvas R(x, v) - log (M( To.. (yll=) Engna (ul=) R((x v) log
Te (v([))
V log (y())))
To (w/x:)
This reveals that classical policy gradient under trust-region constraint is special case of GVPO gradient with To. As proven in Theorem 3.1 and Theorem 3.2 GVPO retains the same optimal solution the policy gradient method while decoupling the sampling distribution म, from the learned policy πp. GVPO's decoupling addresses two critical limitations:
Sample Efficiency: Unlike on-policy methods 41 50], GVPO supports off-policy training with reusable or mixed data (e.g., expert demonstrations, historical policies, or model distillations).
2. Stability: By avoiding importance sampling weights me GVPO eliminates gradient explosion risks without biased clipping.
By synergizing these advantages, GVPO emerges as a competitive online reinforcement learning algorithm capable of leveraging diverse data sources, sustaining stable policy updates, and preserving convergence to optimality-& combination previously unattained in prior policy gradient methods.
Table 1: Algorithm Performance Comparison on Mathematical Datasets
Algorithm
AIME2024
AMC
MATH500
Minerva
Olympiadbench
14.68
38.55
64.00
27.20
30.66
+GRPO
14.79
55.42
80.00
41.17
42.07
GRP.
16.56
48.19
81.20
44.48
43.40
+Remax
17.19
60.24
82.00
40.44
45.19
+Reinforce++
16.67
54.22
80.40
43.01
41.78
+GVPO
20.72
62.65
83.80
45.95
46.96

---

Experiments
Task. Following the established experimental setting of GRPO, conduct comprehensive evalua- tion on math reasoning. Specifically, we post-train the Qwen2.5-Mand M-tthh--- model on Competition Math dataset [14] and assess performance AIME2024 [26). AMC [26], Math500 [13], Minerva [25] and OlympiadBench [13] For answer verification, we utilize the x Verify framework [6]. We
adopt the pass@1 accuracy for all benchmarks except AIME2024, where we report avg accuracy
to account for its limited size (30 problems) and high difficulty.
In addition math reasoning, we also evaluate GVPO on the summarization task in Appendix
Setup. To ensure fair comparison across methods, we maintain identical experimental settings while only modifying the algorithmic component. For GVPO, we employ 3 = 0.1 and T. Te in the main experiment. For competing approaches, we utilize hyperparameters specified in their original publications. All experiments generate h = 5 responses per prompt. A comprehensive description of the training details is provided in Appendix[A.1]
Main Result. Table [ shows the main experiment result, which demonstrates that GVPO achieves the best performance, outperforming both the base model and other variants in all benchmarks particularly in complex problem-solving scenarios. We attribute its effectiveness to its strong theoretical guarantees of convergence.
28 27
Ablation on 3. Figure analyzes the sensitivity of GVPO to variations B. The results demonstrate little fluctuation in performance across 8, suggesting GVPO exhibits robustness this hy pyprparametee. This stability may reduce the need for exhaustive tuning and enhance its practical utility.
Ablation on k. Figure 3 examines how GVPO scales with k. evaluated on Swee2.5-Math-11. Top and bottom panels show results for MATH500 and AIME2024 respectively. GVPO consistently outperforms GRPO across all k and demonstrates superior scalability. Notably, GVPO matches the AIME2024 performance
7B model on the 1.5B architecture through increased k, highlighting its potential for reducing inference costs in practice.
Ablation on ... Figure Biguueeiinveseee investigates GVPO's versatility on sampling distributions, evaluated on 1ren2.5-Math-1-e and MATH500. We propose a heuristic тт that mixes responses from Tewu with historical responses. Results demonstrate GVPO's robust performance across mixing proportions, highlighting: (1) this n, can reduce sampling costs during training, and (2) it suggests GVPO's potential to bridge modern LLM research with previous RL research on exploration strategies.
0.8
0.8
Accuracy
Accuracy
Accuracy
0.6
0.76
0.6
0.2
0
3
3.5
6
0.2
0.15
0.2
0.01
0.05
0.1
0.2
0.5
21
#Traning Steps
Figure 2: Ablation on 3. Each
Figure 3: Ablation on k. Blue Figure 4: Ablation on T.. #(his-
line represents dataset.
line: GVPO; Red line: GRPO.
torical y) #(y from T
Table 2: Ablation on Regularization Terms
Algorithm
AIME24
AMC
MATH500
Minerva
Olympiadbench
GVPO
20.72
62.65
83.80
45.95
46.96
GVPO Var
0.00
0.00
0.00
0.00
0.00
GVPO Cov
0.00
0.00
0.00
0.00
0.00
GVPO Var Cov
7.19
39.76
73.00
34.93
35.70
GVPO- Var + Entropy
0.00
0.00
0.00
0.00
0.00
GVPO Var + Entropy (LR=1c-6)
3.02
39.76
33.20
23.16
8.89
we decompose the GVPO loss into three components: an advantage maximization term, variance regularization term (Var) on the current policy, and covariance regularization term (Cov) between the current and reference policies.
Ablation on Regularization Terms. In Section 3.5

---

Table 2 presents the ablation results for these regularization terms. We first remove Va Var(log To) and Cov(Log Ty. log Fg.) individually. In both cases, the model fails to converge and generates incoherent outputs, indicating that each term plays crucial role in stabilizing training. When both regularization terms are removed but diverges after approximately 10% of the training steps, further confirming that these regularization components are essential to GVPO's stability.
salllaaeoooss--rduuciii the objective to At - Ri-te model initially converges
Study Var(log me). We further investigate the role of Var(log TT) by replacing it with entropy regularization term. As shown in Table 2 the model again fails to converge and produces incoherent outputs. Lowering the learning with entropy regularization still lead to divergence or suboptimal performance. These findings suggest that although entropy regularization serves similar stabilizing purpose, cannot fully substitute Var(log mæ). The degradation likely arises from entropy regularization being either crak--insufficient suppress extreme updates too crrong-impeding convergence. This underscores a key limitation of entropy regularization: its sensitivity to coefficient tuning. In contrast, the coefficient for Var(log πo) in GVPO is derived analytically via the optimal solution theorem, eliminating the need for manual tuning and enhancing overall robustness.
to Te-6 improves stability marginally, yet substituting
Table 3: Algorithm Performance Comparison with Different Random Seeds
Algorithm
AIME2024
AMC
MATH500
Minerva
Olympiadbench
GRPO
13.59
2..94
16.3 77.36=0.98
35.40+1.004
38.65-10.74
GVPO
15.56=1.005
45.78=2.05
36.03=1.15
39.58:11.08
Robustness Check with Different Random Seeds. We conducted additional experiments using 10 random seeds for both methods on Qwen2.5-Math-1.5B The results show that GVPO consis- tently outperforms GRPO in overall performance while exhibiting comparable standard deviations, indicating stable results across different runs.
Table 4: Algorithm Performance Comparison with
Algorithm
AIME2024
AMC
MATH500
Minerva
Olympiadbench
Llama-3.1-BB-Instree
5.00
19.27
50.40
22.42
17.04
+GRPO
8.54
19.27
52.60
25.37
16.15
+GVPO
11.56
20.48
56.60
29.41
20.59
Robustness Check with a Different Foundation Model. To further assess generalization, we repeated the experiments using Llama-3. 1-8maa-----B--nstttuue as the foundation model. As shown in Table GVPO again outperforms GRPO, reaffirming the robustness and effectiveness of our approach.
Related Work
This paper closely relates the LLM post-training literature, including SFT 32 8. RLHF 3), PPO
37]. TRPO [35), DPO 33), GRPO 33). Dr.GRPO (28). Remax [27). Reinforcet+ [17]. DAPO [4]:
and debates on whether RL improves LLM reasoning capacity (55). spurious reward issue 38 data
contamination issue [51. and @ K optimization [56].
Beyond post-training, LLMs have demonstrated broad applicability across domains, including vision- language modeling [12). graph learning [4 19 AI safety research [53], evaluation frameworks
leveraging LLM-as-a-Judee paradigms [22 21, retrieval-augmented generation [7], AI for Education
20 49 and economics of Al [57] 58) These broader developments underscore the promising role of
post-training in enhancing the adaptability of LLMs across growing range of applications.
6 Conclusion
In this paper, we present Group Variance Policy Optimization (GVPO). GVPO guarantees a unique optimal solution, exactly the Kl-constrained reward maximization objective. Moreover, it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. Through systematic comparisons with other prominent methods both theoretically and empirically, we establish GVPO paradigm for reliable and versatile LLM post-training
10

---

Acknowledgments and Disclosure of Funding
We thank all the reviewers, the AC and program committee members for constructive feedback.
This work was supported in part by the National Key R&D Program of China (Grant No.2023YFF0725001 in part by the National Natural Science Foundation of China (Grant
No. No.92370204), in part by the guangdong Basic and Applied Basic Research Foundation (Grant No.2023B15151200577 in part by the Education Bureau of Guangzhou.
References
[1] A. Abdolmaleki, B. Piot, B. Shahriari, T. Springenberg, T. Hertweck, R. Joshi, Oh, M. Bloesch, T. Lampe, N. Heess, et al. Learning from negative feedback, or positive feedback or both. arXiv preprint arXiv:2410004166, 2024.
[2] Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans. Understanding the impact of entropy on policy optimization. In International conference machine learning, pages 151-160.
PMLR, 2019.
[3] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wtX.::2204.05666 2022.
[4] H. Bong and A. Rinaldo. Generalized results for the existence and consistency of the mle in the bradley-terry-lue model. In International Conference on Machine Learning, pages 2160-2177. PMLR, 2022.
[5] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[6] D. Chen, Q. Yu, P. Wang, W. Zhang, B. Tang, F. Xiong, X. Li, M. Yang, and Z. Li. xverify: Efficient answer verifier for reasoning model evaluations, 2025.
[7] L. Dai, Xu, J. Ye, H. Liu, and H. Xiong. Seper: Measure retrieval utility through the lens of semantic perplexity reduction. arXiv preprint arXiv:2503.0144 2025.
[8] Y. Fan, Y. Hong, Q. Wang, J. Bao, H. Jiang, and Y. Song. Preference-orienteè supervised fine-tuning: Favoring target model over aligned large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 23859-23363 23869, 2025.
[9] W. Fedus, P. Ramachandran, R. Agarwal, Y. Bengio, H. Larochelle, M. Rowland, and W. Dabney. Revisiting fundamentals of experience replay. International conference on machine learning, pages
PMLR, 2020.
[10] B. Gao, Song, Y. Miao, Z. Cai, Z. Yang. L. Chen, H. Hu, R. Xu, Q. Dong, C. Zheng, et al. Towards a unified view of preference learning for large language models: A survey. arXiv preprint arXiv:2409.02799 2024.
[11] D. Guo, D. Yang, H. Zhang, Song, Zhang, Xu, Q. Zhu, S. Ma, Wang, X. Bi, al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[12] W. Guo, Z. Chen, S. Wang, J. He, Y. Xu, Ye, Sun, and H. Xiong. Logic-in-framme Dynamic keyframe search via visual semantic-logical verification for long video understanding. arXiv preprint arXiv:2503. 13139, 2025.
[13] C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
[14] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang. D. Song, and J. Stein- hardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0333 038774 2021.
[15] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Stein- hardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0388 2021.
11

---

[16] Y. Hong, H. Zhang, J. Bao, H. Jiang. and Y. Song. Energy- based preference model of- fers better offline alignment than the bradley-terry preference model. In Proceedings of the 42nd International Conference on Machine Learning. volume 267 of Proceedings of Machine Learning Research, pages 23787-23804. PMIR, 13-19 Jul 2025.
[17] J. Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03266 2025.
[18] Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H.-Y. Shum. Open- meen-reasnner-zero An open source approach to scaling up reinforcement learning on the base model, 2025.
[19] Z. Hu, Y. Li, Z. Chen, J. Wang. H. Liu, K. Lee, and K. Ding. Let's ask gnn: Empowering large language model for graph in-context learning. arXiv preprint arXiv:2410.07074, 2024.
[20] Z. Hu, J. Lian, Z. Xiao, S. Zhang. T. Wang, J. Yuan, X. Xie, and H. Xiong. Unveiling the learning mind of language models: A cognitive framework and empirical study, arXiv preprint
arXiv:2506. 1 13464, 2025.
[21] Z. Hu, L. Song, J. Zhang, Z. Xiao, T. Wang, Z. Chen, N. Yuan, J. Lian, K. Ding, and H. Xiong. Explaining length bias in Ilm-based preference evaluations. arXiv preprint arXiv:2407.0100 2024.
[22] Z. Hu, Zhang, Z. Xiong, A. Ratner, H. Xiong, and R. Krishna. Language model preference evaluation with multiple weak evaluators. arXiv preprint arXiv:2410. 12699, 2024.
[23] N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645-1654. PMLR, 2017.
[24] W. Jiang, W. Wu, L. Zhang. 7. Yuan, J. Xiang, J. Zhou, and H. Xiong. Killing two birds with one stone: Cross-modal reinforced prompting for graph and language tasks. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 1301-1312, 2024.
[25] A. Lewkowyez, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[26] J. Li, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. Huang, K. Rasul, L. Yu, A. Q. Jiang, Z. Shen, et al. Numinamath: The largest public dataset in ai4 with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
[27] 7.. Li, T. Xu, Y. Zhang, Z.. Lin, Y. Yu, R. Sun, and Z.-Q. Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310. 10505, 2023.
[28] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding r1 ri-erollikke training: A critical perspective. arXiv preprint arXiv:2503.2078, 207883, 2025.
[29] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang. C. Du, W. S. Lee, and M. Lin. Understanding rl-zero- training: A critical perspective, 2025.
[30] R. Miao. Optimizing the Unknown: Reinforcement Learning and Energy- -Bsee Model for Black Box Bayesian Optimization. University of California, Los Angeles, 2024.
[31] S. Minace. T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao. Large language models: A survey, 2025.
[32] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[33] J. Qian, Z. Zhu, H. Zhou, Z. Feng, Z. Zhai, and K. Mao. Beyond the next token: Towards prompt-robust zero-shot classification via efficient multi-token prediction. arXiv preprint arXiv:2504.03159, 2025.
[34] R. Rafailov, A. Sharma, F. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances Neural Information Processing Systems, 06:53728-53741 2023.
12

---

[35] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In F. Bach and Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.
[36] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1556.02444 2015.
[37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1o0Kiv:1007.06 707.06347, 2017.
[38] R. Shao, S. S. Li, R. Xin, S. Geng, Y. Wang, S. Oh, S. S. Du, N. Lambert, S. Min, R. Krishna, al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.1099 10947, 2025.
[39] Z. Shao, P. Wang. Q. Zhu, R. Xu, J. Song, X. Bi. H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning open language models. arXiv preprint (xKi::2402.033000 2024.
[40] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. Hybridflow: flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256 2024.
[41] S. Singh, T. Jaakkola, M. L. Littman, and C. Szepesvári. Convergence results for single-step on-policy veinforeement-leanning algorithms. Machine learning, 38:287-308, 2000.
[42] N. Stiemnon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and F. Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
[43] R. Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019.
[44] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for rein- forcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[45] Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, H. Richemond, M. Valko, B. Á. Pires, and B. Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.0577
2024.
[46] G. Tie, Z. Zhao, D. Song, F. Wei, R. Zhou, Y. Dai, W. Yin, Z. Yang. J. Yan, Y. Su, Z. Dai,
Y. Xie, Y. Cao, L. Sun, P. Zhou, L. He, H. Chen, Y. Zhang, Q. Wen, T. Liu, N. Z. Gong, J. Tang,
C. Xiong, H. Ji, P. S. Yu, and J. Gao. A survey on post-training of large language models, 2025.
[47] S. T. Tokdar and R. E. Kass. Importance sampling: review. Wiley Interdisciplinary Reviews: Computational Statistics, 2(1):54-60, 2010.
[48] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Stti::3007009288 2023.
[49] T. Wang, Y. Zhan. J. Lian, Z. Hu, J. Yuan, Q. Zhang, X. Xie, and H. Xiong, Lim-powered multi-agent framework for goal-oriented learning intelligent tutoring system. In Companion Proceedings of the ACM on Web Conference 2025, pages 510-519, 2025.
[50] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
[51] M. Wu, Z. Zhang, Q. Dong, Z. Xi, Zhao, S. Jin, X. Fan, Y. Zhou, Y. Fu, Q. Liu, et al. Reasoning memorization? unreliable results of reinforcement learning due to data contamination. arXiv preprint arXiv:2507. 10532, 2025.
[52] W. Wu, B. W. Junker, and N. M. D. Niezink. Asymptotic comparison of identifying constraints for bradley-terry models, 2022.
[53] Y. Xu. A. Liu, X. Hu, L. Wen, and H. Xiong. Mark your IIm: Detecting the misuse of open-souree large language models via watermarking, arXiv preprint arXiv: 044660, 2025.
[54] Q Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arxiv:2503. 14476. 2025.
13

---

[55] Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.155 13837, 2025.
[56] K. Zhang, S. Gao, Y. Hong, H. Sun, J. Bao, H. Jiang, Y. Song, H. Dingqian, and H. Xiong. Rspo: Risk-seeking policy optimization for pass@ k and max @ metrics in large language models, arXiv preprint arXiv:
2025.
[57] K. Zhang, Z. Yuan, and H. Xiong. The impact of generative artificial intelligence on market equilibrium: Evidence from natural experiment. arXiv preprint Stti:2311.000711,
(58) K. Zhang, 7. Yuan, and H. Xiong. Optimized cost per click in online advertising: A theoretical analysis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4232 4243, 2024.
[59] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang. Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Du, Yang. Y. Chen, Z. Chen, Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, and J.-R. Wen. A survey of large language models, 2025.
[60] C. Zhou, Q. Li, C. J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, H. Peng, J. Li, J. Wu, Z. Liu, Xie, C. Xiong, J. Pei, P. S. Yu, and L.. Sun. A comprehensive survey pretrained foundation models: history from bert to chatgpt, 2023.
14

---

Limitation
in complannee with page limit requireient, this paper does not extensively diseuss experimental results. Instead, have prioritized an in-depth analysis of GVPO's essence, posit that such a discussion offers greater scholarly value to readers. We acknowledge this structural limitation and note that it will be addressed in subsequent revisions, particulally in the event of acceptance with an additional page alowaanee
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and introduction, we introduced GVPO and discussed GVPO in detail in the subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have a limitation section before the checklist.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
15

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We clarify the assumption along with each theoretical result, and provide detailed proof' in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly introduced GVPO's loss and algorithm process in the main body of this paper, and showed how to implemented GVPO by using open-sources RL framework in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
16

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In Appendix A.2, we show the code to implement GVPO and share instructions to reproduce experimental results in Appendix A.I
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
cablic/guides http///ooosoissoonnoiioo onooooiyllll for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips wttp://pppppp...dd...oosppps htapl/ic/guides//ooedeiee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix A.I provides a comprehensive description of the training details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
17

---

Justification:
We do not have this result because training large language model is very costly.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported experiments compute resources Appendix A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tn:r//nsusis// wwterrisscccppuulie cc/publlcc/Ehhhccddddddll1
Answer:
[Yes]
Justification:
We followed NeurIPS Code Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [NA]
Justification: GVPO does not tie to particular applications, and does not lead to societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
GVPO algorithm poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, credited them appropriately.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development of GVPO does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy httt::///earrrppcccc////ofeeeeee for what should or should not be described.
21

---

Appendix
A Supplementary Experiment Information
A.1 Experiment Settings
pyperaarameter Recipe. For each step, sample 1024 prompts from the training set and set the mini-batch size each step to 256. We repeat the whole training set for 10 epochs and set the warm-up ratio to 5%. We grid-search the learning rate in {5e 7, 1e 6, 5e 6, Le - 5) and find 5e - to the best setting. We conduct the main experiment using an Deepseck-Ri template top of Qwen2.5-Math-7B
chat in [8). In the ablation experiments, for faster training and GPÜ memory limitations, we use the original Qwen chat template on top of Qwen2.5-Math Math--.5B..
Compute Resources, We conduct our experiments using server with eight 80GB H800 GPU cards. For Qwen2.5-Math-71 experiments with k: = 5, takes to 8 minutes per training step and approximately 12 hours experiment. For 1wen2.5-Math-1.55 1.5B experiments with A: = 8, it takes 4 to minutes per training step and approximately hours per experiment.
A.2 Code Implementation
It is easy to implement GVPO based on open-source RL framework. For example we show the minimum viable implementation of GVPO that only modifies few line of GRPO loss in verl 40):
def compute_policy. ** **argg)))
(old_log-proy log_prob, advantages, eos_ cos_mask,
(log-prob eos_mask) sum (dim=-1)
scores_old (old_log las_llog-proo prob eos mask) sum (dim=-1) advs (advantages eos ma_k) sum (dim=-1) / eos mask. sum (dim=-1)
beta = 0.1
size (0)
scores_nev scores ()
loss - -1 * beta * scores * (advs beta ((scores_nex scores_new mean ())
(scores_old scores mean )))
12
return loss sum () (k-1)
Listing 1: A Simple GVPO Code Implementation
Moreover, provide implementation based verl http://gittubbcom/sssc///VV
B Proofs
B.1 Proof of Theorem
3.1
Theorem
3.1
The unique optimal policy that minimizes Lavro(0), defined as
(0) = Ex--DEE-r..lee
u) - Ey~a, Ro(r, u)) (R(x, y) - E Ey,r,, y))
is given by To(y|=) = ()(()) = 2(x) Tg. (yl (c) R(s.w)
for n, = Tpp-
Proof. We prove the theorem by establishing both necessity and sufficiency.
Necessity: If To(y)=)) n (y Ir), then it is optimal policy solution.
The loss function (o) is non-negative because it represents the expectation of squared term:
Levpo = Em.y ((Ra (x, v) Ro v)) - (R(x, v) - E., R(x, u)) 2
Make sure that 1) each input batch correspond to the A responses of a prompt and 2) the std normalizer in the GRPO advantage calculation has been removed.
22

---

When No (v) = (B)2), we have Ro(z,y) R(x, y). Substituting this into the loss function gives (0) = 0, confirming that nt achieves the minimum loss.
Sufficiency: If policy xy optimal, then (o(a)))) ' (y/æz)
Assume for contradiction that there exists an optimal policy To # n² Since To optimal, (f) = 0. This implies:
(Ro(z, v) By (o = (R(a,, v) R(x, v)), Va, U.S. TT (T(1)) > 0
Rewriting Rg and in terms of their respective policies:
(r, y) = Blogn' (y|æ) Eg)(o,)) Bloggo(ylæ) Ro
Rearranging terms yields:
By [a R(r,,)) w)] x² (v(xx)
To(u|o)) = exp
B
Since Ene (o) (b(()=) = (1|=)>03
T (ulax) = 1, we must have:
By Ro y) R(x, w)]
Σ To(()=) = exp
Σ x (ulæz)
B
Ey Ro y) R(x, v)
exp
8
Thus, To(y)>)= TT" (y/az) for all z, y. contradicting the assumption Ty # x²
Since both necessity and sufficiency hold, the optimal policy is uniquely x².
C Alternative Task on Summarization
To evaluate GVPO in the context of alignment conduct experiments summarization task following the experimental setup of DPO [34]. Specifically, we used the Reddit dataset from [42] and the reward model OpenAssistunttreward wttpta/////wwaaddmmmmeeeeeee -ebelll-ddbbrtt---largge 1.5B using the dataset to obtain an SFT reference model. We then applied both DPO and GVPO for post-training. The training data was sampled from the reference model and scored using the reward model.
First, fine-tuned Qwen2.5
For evaluation, considered:
The average reward of generated summaries the test split.
Win rate against human human-written written demonstration answers, as scored by the reward model.
Preference accuracy using human-labeled comparison data: both DPO and GVPO are trained [a(x, y) 3 log melvle)
+ 3 log (xr). Given a preference pair (yw, 31) where yoo is preferred,
log marly)) (x)
we compute whether log (ymll))
log (ravvla)) (a). log me(()])))
Benchmarking by set of powerful LLMs.
Human evaluations by crowd-soureing workers. We hire three workers to label their preference on 100 prompts on Prolific platform.
Table S: Algorithm Performance Comparison on Summarization
Algorithm
Reward
Win Rate
Accuracy
Gpt-4o
Gemini-2.5-pro
Deepseek-RI
Human
SFT
2.84
41. 41.85%
+DPO
4.83
68, 18.28%
60.43%
31%
40%
25%
34%
+GVPO
5.75
79.49%
64.93%
69%
60%
75%
66%
The results indicate that GVPO outperforms DPO across these metrics, supporting the view that improvements in policy search methods positively correlated with alignment quality.
Thank Reviewer X8wt for sponsoring this study.
23


Reviews:
[{"review_id": "57ipj55Bsi", "reviewer": "Reviewer_X8wt", "summary": "This paper presents Group Variance Policy Optimization (GVPO), a method for KL-constrained reward maximization. To derive GVPO, the paper identifies that when the sum of assigned weights to all responses within a prompt equals zero, then the partition function becomes invariant across the compared responses. The paper presents a clean interpretation as well as connections with existing literature on policy optimization methods. The paper also presents empirical results highlighting the usefulness of the proposed policy search method on math reasoning tasks.", "questions": "While the paper presents clear benchmarking for math reasoning tasks, it is worth asking the broader question -- what is the role of policy optimization in general natural language generation tasks? Does better policy optimization lead to better alignment? This question is more nuanced since there are other factors at play about quality of reward models, how they align with human preferences, reward hacking etc. I think the paper benefits a discussion surrounding these issues, and additional experimentation to highlight the nuanced nature of the discussion surrounding development of better policy search methods. Towards this, I'd like to see (a) additional results on natural language generation tasks such as summarization, question answering etc., (b) whether a better policy search method naturally translates to better evals on automated/human eval on alignment quality. My sense is these results will be illuminating and may not fully line up in terms of improvement in metrics, yet will be beneficial to guide subsequent research in this topic.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Important problem, well motivated and well written paper\n- Clearly well situated with important (and most well known) related works such as GRPO, DPO etc.\n- clear derivation and strong empirical results\n\nWeaknesses:\n- Insufficient empirical benchmarking", "was_revised": true, "final_justification": "The authors agree to expanding their results to handle alignment benchmarks and engaged with addressing comments about insufficient benchmarking. I have updated my score based on these discussions.", "rebuttal": "We appreciate your recognition of the importance and clarity of our work, as well as the relevance of our positioning within the literature. We are particularly encouraged by the acknowledgment of our theoretical derivations and empirical results. \n\nBelow, we address your concerns and provide clarifications to further strengthen our contribution.\n\n**Insufficient empirical benchmarking**\n\nWe have added three more recently popular benchmarks, including DAPO [1], Remax [2], and Reinforce++ [3]. \n\n| Algorithm           | AIME2024 | AMC    | MATH500 | Minerva | Olympiadbench |\n|--------------------|----------|--------|---------|---------|----------------|\n| Qwen2.5-Math-7B     | 14.68    | 38.55  | 64.00   | 27.20   | 30.66          |\n| +GRPO              | 14.79    | 55.42  | 80.00   | 41.17   | 42.07          |\n| +Dr.GRPO           | 16.56    | 48.19  | 81.20   | 44.48   | 43.40          |\n| +DAPO              | 3.23     | 20.48  | 50.40   | 28.68   | 16.59          |\n| +Remax             | 17.19    | 60.24  | 82.00   | 40.44   | 45.19          |\n| +Reinforce++       | 16.67    | 54.22  | 80.40   | 43.01   | 41.78          |\n| **+GVPO**          | **20.72**| **62.65** | **83.80** | **45.95** | **46.96** |\n\nThe results demonstrate that GVPO continues to retain its effectiveness.\n\n[1] Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. arXiv preprint arXiv:2503.14476, 2025.\n\n[2] Li, Ziniu, et al. \"Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models.\" arXiv preprint arXiv:2310.10505 (2023).\n\n[3] Hu, Jian, Jason Klein Liu, and Wei Shen. \"Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models.\" arXiv preprint arXiv:2501.03262 (2025).\n\n**I think the paper benefits a discussion surrounding these issues, and additional experimentation to highlight the nuanced nature of the discussion surrounding development of better policy search methods. Towards this, I'd like to see (a) additional results on natural language generation tasks such as summarization, question answering etc., (b) whether a better policy search method naturally translates to better evals on automated/human eval on alignment quality.**\n\nThank you for your valuable suggestions, which help broaden the impact of this work within the alignment research community.\n\nFrom a theoretical perspective, when an RL algorithm is trained using a proxy reward model instead of the true reward (e.g., actual human preferences), its performance on the true objective depends critically on the fidelity of the proxy [4]. This challenge of proxy rewards is well recognized in reinforcement learning literature [5].\n\n[4] Wang, Binghai, et al. WorldPM: Scaling Human Preference Modeling. arXiv preprint arXiv:2505.10527, 2025.\n\n[5] Skalse, Joar, et al. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems 35 (2022): 9460–9471.\n\nFollowing your advice, we conducted additional experiments on a summarization task. Specifically, we used the Reddit dataset from [6] and the reward model OpenAssistant/reward-model-deberta-v3-large-v2. First, we fine-tuned Qwen2.5 1.5B using the dataset to obtain an SFT reference model. We then applied both DPO and GVPO for post-training. The training data was sampled from the reference model and scored using the reward model.\n\nFor evaluation, we considered:\n\n+ The average reward of generated summaries on the test split.\n\n+ Win rate against human-written demonstration answers, as scored by the reward model.\n\n+ Preference accuracy using human-labeled comparison data: both DPO and GVPO are trained via\n$R_\\theta(x, y) = \\beta \\frac{\\log \\pi_\\theta(y|x)}{\\log \\pi_{\\text{ref}}(y|x)} + \\beta \\log Z(x)$. Given a preference pair $(y_w, y_l)$ where $y_w$ is preferred, we compute whether\n$\\frac{\\log \\pi_\\theta(y_w|x)}{\\log \\pi_{\\text{ref}}(y_w|x)} > \\frac{\\log \\pi_\\theta(y_l|x)}{\\log \\pi_{\\text{ref}}(y_l|x)}$.\n\n| Algorithm       | Reward | Win Rate | Accuracy |\n|----------------|--------|----------|----------|\n| SFT            | 2.84   | 41.85%    |   ---    |\n| +DPO           | 4.83  | 68.28%      | 60.43%    |\n| +GVPO          | 5.75  | 79.49%      | 64.93%    |\n\nThe results indicate that GVPO outperforms DPO across these metrics, supporting the view that improvements in policy search methods are positively correlated with alignment quality.\n\n[6] Stiennon, Nisan, et al. Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33 (2020): 3008–3021."}, {"review_id": "jRz38ZJkoM", "reviewer": "Reviewer_riT8", "summary": "This paper presents a new method for policy optimization for LLM post-training, referred to as Group Variance Policy Optimization (GVPO).\nGVPO puts together various ideas from policy optimization and preference learning in a new way.  In particular, via a clever trick that centers the weights of gradients (where the weights are differences in log-likelihood ratios), a partition function can be made to vanish in the gradient.  This yields not only an efficient method, but an interpretable one.", "questions": "It would be nice to understand whether there is a more general principle at work here.  As currently presented, the idea (center the weights) comes across as a (nice) trick.  Are there analogous methods in the operations research literature?  Or the optimization literature?\n\nIn particular, the objective function studied here is of the form of mirror descent, which may provide a path towards a deeper understanding and even obtaining theoretical convergence rates.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths:  The authors do an excellent job at situating their method in the landscape of existing methods, and pointing out its relative advantages, which include a lack of a need for importance sampling and on-policy gradients, and therefore a lack of need for various ad hoc adjustments including clipping.  The authors also do an excellent job at interpreting the gradient of their method, in three different ways, all insightful.  The experiments also seem compelling.\n\nWeaknesses:  None that I can see.", "was_revised": true, "final_justification": "I view this as a valuable contribution to the literature on LLM post-training with a useful and interesting algorithmic innovation of weight-centering.", "rebuttal": "Thank you for engaging with our work and highlighting the novel aspects we take pride in. We appreciate your recognition of the clarity with which we position our method in relation to existing approaches, as well as your appreciation of the three complementary interpretations of the gradient. We're also pleased that you found the experimental results compelling.\n\n**It would be nice to understand whether there is a more general principle at work here. ... In particular, the objective function studied here is of the form of mirror descent, which may provide a path towards a deeper understanding and even obtaining theoretical convergence rates.**\n\nThank you for highlighting this promising direction for future research. Based on our current understanding, we are not aware of a unifying principle in OR or optimization literature that directly explains this method. Nonetheless, we share your interest in theoretical convergence rates and find questions—\"Which sampling distributions exhibit faster convergence? What characteristics do they share?\"—both insightful and significant. We will keep your suggestion regarding \"mirror descent\" in mind and plan to explore this perspective further in future work."}, {"review_id": "IrYpOOnZN1", "reviewer": "Reviewer_u3Wn", "summary": "This paper develops a method for mitigating instability in GRPO, which it claims arises from exploding gradients that are a consequence of minimizing log likelihood, called GVPO. GVPO is a weighted policy gradient objective, where weights sum to zero and are derived from the closed form of the KL-regularized policy objective. There are a number of theoretical results showing that GVPO recovers the unique optimal policy, which rely on a reduction from GVPO to squared-loss regression. Experiments are provided in a number of mathematical reasoning datasets.", "questions": "Why does incorporating the log-policy ratio terms into GVPO increase stability?", "limitations": "See comments above", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Overall, I think that the proposed method is interesting, but requires more in-depth empirical investigation. The clarity of the writing, motivation for the paper, and theoretical results are somewhat weak in my view, and as a result I would have really liked to see ablations targeting the purported variance-reduction benefits of GVPO. \n\n**Motivation**\n\nTo begin with, I found the motivation for the paper to be underwhelming and largely unsupported. The motivation for the method is that GRPO is unstable because \"minimizing the log-likelihood of those responses can lead to exploding gradients due to the convexity of the logarithmic function\" (L107-108), but there is no evidence within the paper that this is true, nor citations for this claim. \n\nIn addition, the motivation for the method is to reduce policy learning to reward learning (L148, 151-152), but why is this better than optimizing the policy? And what is the relationship with GVPO (Eq. 9) itself?\n\n**Method and theoretical results**\n\nThere are interesting manipulations and interpretations of the gradient objective: using policy gradient weights that sum to 0, the partition function no longer needs to be estimate, which allows the objective to be converted to squared loss regression. This seems possibly straightforward given the form of Eq. 9 and the definition of $R_\\theta = \\log \\frac{\\pi_\\theta}{\\pi_{\\theta'}}$, and the theoretical results are somewhat simple and repetitive (in my opinion, not generating much more insight) beyond the sq-loss interpretation. \n\nAs a result, I wonder if the paper might be made stronger if Sections 3.1 and 3.3 were shortened in favor of more insightful experiments and ablations, discussed below. \n\n**Experiments**\n\nI appreciate that the authors included both theoretical and empirical results, and considered a few datasets. However, it appears that no standard deviations or averages over seeds are reported, which I find discomfiting given the variance of policy gradient methods for training language models (e.g., GVPO, by the paper's own admission). While I understand that better performance can reflect increased stability, I think that the experimental results would be more interesting if they explicitly highlighted the intended benefits of GVPO.\n\nThe objective resembles the GRPO objective, except it doesn't divide by the standard deviation, and has an additional pair of log policy-ratio terms that can be viewed as implicit rewards. But it has been observed empirically in DPO that such policy-parameterized rewards can be suboptimal or less stable compared to reward models (Swamy et al. 2025; Lin et al. 2024). As such I am curious about the effect of the policy ratio terms, I would think that there could be interesting ablations to conduct on their role in stability, as well as in probing the predictions in Section 3.5. It's not really clear to me how much benefit comes from removing the standard deviation normalization in GRPO (as discussed in L245-249) versus adding the log policy ratios.\n\n**References**\n\nSwamy, Gokul, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell. \"All roads lead to likelihood: The value of reinforcement learning in fine-tuning.\" _arXiv preprint arXiv:2503.01067_ (2025).\n\nLin, Yong, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. \"On the limited generalization capability of the implicit reward model induced by direct preference optimization.\" _arXiv preprint arXiv:2409.03650_ (2024).", "was_revised": true, "final_justification": "I have considered the authors' rebuttal and other reviewers' comments; further justification is provided in my response to the rebuttal.", "rebuttal": "We appreciate your feedback and the recognition of the potential interest in our method. However, we believe there may be a misunderstanding regarding the motivation and contributions of the paper. \n\nWe clarify this further below and address your questions and concerns.\n\n\n**Overall, I think that the proposed method is interesting, but requires more in-depth empirical investigation. ... I would have really liked to see ablations targeting the purported variance-reduction benefits of GVPO.**\n\nWe fully understand your concerns. However, we believe there may have been a misunderstanding. This paper does not aim to reduce variance, nor have we claimed that it can. Our primary objective is to propose a better post-training algorithm compared to GRPO. To that end, we have thoroughly compared GVPO and GRPO to highlight the benefits of our approach, and other reviewers have acknowledged this aspect positively.\n\nUpon reflection, we recognize that the term \"Variance\" in GVPO’s name may have led to confusion. Our goal was to create a counterpart to GRPO's \"Relative\" term. Specifically, while GRPO's weight contains $R-\\overline{R}$ (a relative term), GVPO’s MSE loss $[(R_\\theta-\\overline{R_\\theta})-(R-\\overline{R})]^2$ can also be expressed as a variance-like term, $[(R_\\theta-R)-\\overline{(R_\\theta-R)}]^2$. We will clarify this distinction in the revised draft.\n\n**Motivation**\n\n**The motivation for the method is that GRPO is unstable ... but there is no evidence within the paper that this is true, nor citations for this claim.**\n\nStill, we believe there may have been a misunderstanding. The motivation of this paper is not: because GRPO is unstable, so we improve its stability. But instead is: GRPO are reported not effective in practice (and many people attributes to its instability), so we propose a more effective one. In other words, this work focuses on better performance rather than low variance. Framing our motivation as “variance reduction” risks underestimating the true contribution of our work.\n\n\nTo clarify, we will add relevant references and provide a more clear explanation in Lines 106-111:\n\n\\begin{equation}\n\\mathcal{L}(\\theta) = -\\sum_{(x,\\{y_i\\}) \\in \\mathcal{D}} \\sum_{i=1}^k \\frac{R_i-\\overline{R}}{\\sigma(R)}\\log \\pi_\\theta(y_i|x)\n\\end{equation}\n\nRewards below average lead to negative weights, penalizing their likelihoods. However, minimizing likelihood can result in unstable training [1]. Additionally, off-policy training of GRPO, which involves importance sampling with the weight $\\frac{\\pi_\\theta}{\\pi_{\\theta_{old}}}$, becomes unstable when $\\pi_\\theta$ significantly deviates from $\\pi_{\\theta_{\\text{ref}}}$, potentially causing gradient explosions. To mitigate these issues, GRPO uses gradient clipping and a KL-divergence constraint between the updated and initial policies:\n\\begin{equation}\n\\mathcal{L}(\\theta) = -\\sum_{(x,\\{y_i\\}) \\in \\mathcal{D}} \\sum_{i=1}^k Clip(\\frac{\\pi_\\theta(y_i|x)}{\\pi_\\text{old}(y_i|x)})\\frac{R_i-\\overline{R}}{\\sigma(R)}\\log \\pi_\\theta(y_i|x)- \\beta KL(\\pi_\\theta||\\pi_\\text{ref})\n\\end{equation}\nNevertheless, empirical results [2,3] show that GRPO still exhibits training instability, which undermines its performance. \n\n\n[1] Abdolmaleki, Abbas, et al. \"Learning from negative feedback, or positive feedback or both.\" ICLR. 2025.\n\n[2] Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. arXiv preprint arXiv:2503.14476, 2025.\n\n[3] Hu, Jian, Jason Klein Liu, and Wei Shen. \"Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models.\" arXiv preprint arXiv:2501.03262 (2025).\n\n**Why is this (reduce policy learning to reward learning) better than optimizing the policy? And what is the relationship with GVPO (Eq. 9) itself?**\n\nReducing policy learning to \"reward learning\" is also a key advantage of DPO: it does not replace policy optimization but rather provides a more smart pathway to it. This is because this method simplifies the joint optimization over policies and rewards to a simpler problem focused solely on rewards. This makes the problem more tractable, as it only requires aligning implicit rewards $R_\\theta(x, y)$ with the true reward function $R(x, y)$ (Line 215-217). For instance, in joint optimization, the decision of whether to add standard deviation normalization is ambiguous. In contrast, \"reward learning\" only necessitates designing a loss function that trains $R_\\theta$ to align with $R$, on which the question of whether adding standard deviation will depend. GVPO reduces policy learning to \"reward learning\" through an MSE loss.\n\n\n**Method and theoretical results**\n\n**This seems possibly straightforward given the form of Eq. 9 and the definition of $R_\\theta=\\frac{\\log \\pi_\\theta}{\\log \\pi_{\\theta'}}$, and the theoretical results are somewhat simple**\n\nWhile the necessity of the theorem ($R_\\theta = R$ is an optimal solution) follows directly, establishing its sufficiency (the only optimal solution is $R_\\theta = R$) is significantly more challenging and is addressed in detail in Appendix B.1.\n\nThe sufficiency result is novel—for instance, DPO establishes only necessity, not sufficiency, and admits multiple optimal solutions, some of which does not correspond to the optimal solution of the KL-constrained reward maximization objective (see Lines 231–238).\n\n**I wonder if the paper might be made stronger if Sections 3.1 and 3.3 were shortened in favor of more insightful experiments and ablations**\n\nWe agree with your suggestion and will prioritize experiments over less important theoretical discussions.\n\n**Experiments**\n\n**It appears that no standard deviations or averages over seeds are reported, ..., I think that the experimental results would be more interesting if they explicitly highlighted the intended benefits of GVPO.**\n\nWhile the objective of this work is not to reduce variance, we have nonetheless conducted additional experiments using 10 random seeds for both methods on the 1.5B model with $k=4$ as a robustness check: \n\n|Algorithm|AIME2024|AMC|MATH500|Minerva|Olympiadbench|\n|-|-|-|-|-|-|\n| GRPO| 13.59$\\pm$1.11 | 44.34$\\pm$2.19 | 76.30$\\pm$1.35 | 35.40$\\pm$1.04 | 38.65$\\pm$0.74 |\n| GVPO| 15.56$\\pm$1.05 | 45.78$\\pm$2.05 | 77.36$\\pm$0.98 | 36.03$\\pm$1.15 | 39.58$\\pm$1.08 |\n\nThe results indicate that GVPO consistently outperforms GRPO in terms of performance, while maintaining comparable standard deviations.\n\n**The objective resembles the GRPO objective, except it doesn't divide by the standard deviation, and has an additional pair of log policy-ratio terms that can be viewed as implicit rewards.**\n\nWe would like to respectfully point out that your comparison may overlook a critical component of GRPO—the clipped importance sampling weight, $Clip(\\frac{\\pi_\\theta(y_i|x)}{\\pi_\\text{old}(y_i|x)})$. This clipping mechanism is essential in GRPO to prevent gradient explosion due to large importance weights. While it effectively stabilizes training, it introduces bias into the gradient estimates. In contrast, GVPO does not rely on importance sampling from the outset, thereby avoiding these issues entirely (see Lines 270–285). This represents a significant advantage over GRPO that may not have been fully considered in your comparison.\n\n**But it has been observed empirically in DPO that such policy-parameterized rewards can be suboptimal or less stable compared to reward models (Swamy et al. 2025; Lin et al. 2024).**\n\nThank you for providing the relevant literature. Our interpretation is as follows: Although DPO employs log-ratios as implicit rewards—thereby functioning similarly to reward models—it is, at its core, a policy that generates responses. It is therefore reasonable to expect that it may be suboptimal for the specific task of reward estimation, especially when compared to reward models explicitly trained to output scalar values.\n\n**As such I am curious about the effect of the policy ratio terms, I would think that there could be interesting ablations to conduct on their role in stability, as well as in probing the predictions in Section 3.5.**\n\nWe conduct comprehensive ablation studies on the regularization terms.\n\n|Algorithm|AIME2024|AMC|MATH500|Minerva|Olympiadbench|\n|-|-|-|-|-|-|\n| GVPO| 20.72 | 62.65 | 83.80 | 45.95 | 46.96 |\n| GVPO $w/o$ Var| 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |\n| GVPO $w/o$ Cov| 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |\n| GVPO $w/o$ Var $w/o$ Cov| 7.19 | 39.76 | 73.00 | 34.93 | 35.70 |\n\nFirst, we remove $Cov(\\log\\pi_\\theta, \\log\\pi_{\\theta^\\prime})$ and $Var(\\log\\pi_\\theta)$ individually. In both cases, the models fail to converge and produce incoherent outputs, indicating that these terms are essential for stabilizing training. Next, we remove both regularization terms simultaneously, i.e., removing the ratio terms and reducing the objective to $R - \\overline{R}$. While this model initially converges, it begins to diverge after approximately 10% of the training steps, further confirming that the regularization terms are critical components of GVPO.\n\n**Questions**\n\n**Why does incorporating the log-policy ratio terms into GVPO increase stability?**\n\nFrom the perspective of regularization, the log-policy ratio terms are equivalent to the sum of $Var$ and $Cov$ after taking gradients. These two components help discourage extreme probability assignments and prevent the current policy from deviating excessively from a reference policy. Both effects contribute to improved stability."}, {"review_id": "7Yd7NVBxgz", "reviewer": "Reviewer_eBjY", "summary": "This article introduces Group Variance Policy Optimization (GVPO), a new approach for post-training LLMs that ensures reliable and robust model alignment with specific tasks and human preferences. GVPO directly incorporates the analytical solution to KL-constrained reward maximization into its gradient weights, offering two key advantages: a unique optimal solution aligned with the KL-constrained reward maximization objective and support for flexible sampling distributions that avoid limitations of on-policy and importance sampling methods.", "questions": "See Weaknesses. \n\nFor instance,\n\n1. **Separation of Reward Modeling and Policy Optimization**\nDoes GVPO require a separate training phase for the reward model before applying the algorithm?\nHow does the performance of GVPO depend on the quality of the separately trained reward model?\n\n2. **Insufficient Experimentation:**\nHow can the experimental section be expanded to provide more thorough empirical validation of the proposed method ?", "limitations": "See Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n\n1. GVPO builds on the insights from Group Relative Policy Optimization (GRPO) but addresses its training instability issues. By designing a weighting scheme where the gradient weight of a response is the difference between the central distance of implicit rewards (derived from the current and reference policies) and actual rewards, GVPO ensures that the sum of weights within a prompt group equals zero. This design eliminates the need for explicit estimation of the partition function.\n\n2. Theoretical guarantees of GVPO are rigorously established. It is proven that GVPO’s loss function is aligned with the original constrained optimization problem, ensuring convergence to the globally optimal policy. \n\n3. A Few experimental results demonstrate GVPO’s effectiveness. \n\n**Weaknesses**:\n\n1. **Separation of Reward Modeling and Policy Optimization**:\n    GVPO appears to reintroduce the separation between reward modeling and policy optimization. Unlike DPO, which integrates reward model training and policy optimization into a single process by exploiting their analytical relationship, GVPO seems to require a predefined reward model  $R$  (as indicated in Equation (9), not $R_{\\theta} = \\frac{\\log(\\pi_{\\theta})}{\\log(\\pi_{\\theta^{'}})}$). This raises questions about whether GVPO necessitates a separate training phase for the reward model before applying GVPO. If so, this could be seen as a limitation, as the performance of GVPO might become heavily dependent on the quality of the separately trained reward model, potentially undermining its overall effectiveness.\n    \n 2. **Redundant Presentation of Equations**:\n    The presentation of equations in the paper could be streamlined for better clarity. For instance, the two equations introduced between lines 142--152 appear to primarily serve the purpose of supporting the equation presented between lines 159--160 that illustrates that GVPO is equivalent to optimizing a mean squared error (MSE) loss that measures the discrepancy between implicit and actual reward central distances. To enhance the flow, the authors could first introduce the symbol $R_{\\theta} = \\frac{\\log(\\pi_{\\theta})}{\\log(\\pi_{\\theta^{'}})}$, then present the equation between lines 159--160. Following this, they could explain how the equation between lines 142--152 arises from the condition $\\sum w_{i}=0$ for verifying the equation between lines 159--160.\n    \n 3. **Redundant Theoretical Proofs in Section 3.3**:\n    The theoretical proofs in Section 3.3 feel repetitive, as they focus extensively on retelling the result that GVPO is equivalent to optimizing an MSE loss measuring the discrepancy between implicit and actual reward central distances. \n    \n \n  4. **Misleading Name (\"Variance\")**: \n    The choice of the name \"Variance\" for the proposed method might be misleading. The new weight introduced in GVPO is actually a difference of two deviations from the the mean rather than a pure variance. This could cause confusion regarding the method's core mechanism and its relationship to statistical variance.\n\n5. **Insufficient Experimentation**:\n    Since this paper focuses on proposing a new post-training algorithm rather than being a theoretical reinforcement learning(RL) paper, the experimentation section feels somewhat insufficient. The proposed method lacks thorough empirical validation across diverse LLMs, which would be crucial for demonstrating its effectiveness and generalizability in practical applications.", "was_revised": true, "final_justification": "My concerns have been addressed, and I will raise my score.", "rebuttal": "We appreciate your recognition of GVPO’s advancements over GRPO. We also thank you for acknowledging the rigorous theoretical guarantees of our paper.\n\nBelow, we address your concerns and provide clarifications to further strengthen our contribution.\n\n\n**1. Separation of Reward Modeling and Policy Optimization**\n\n**Does GVPO require a separate training phase for the reward model before applying the algorithm?**\n\nNo, GVPO does not require a separate training phase. GVPO supports flexible sampling distributions, including those derived from human ratings or preferences, similar to DPO, and thus does not necessitate training a separate reward model (Lines 278–283).\n\nIn our paper, the reward model can take various forms: it may explicitly consist of human evaluation ratings, a trainable function that implicitly captures human preferences, or a predefined rule—such as correctness or accuracy (Lines 94–95).\n\n**How does the performance of GVPO depend on the quality of the separately trained reward model?**\n\nLLM post-training scenarios generally fall into two categories: reinforcement learning with verifiable rewards (RLVR) and reinforcement learning from human feedback (RLHF).\n\nIn RLVR settings, there is no need for a trained reward model. In RLHF, if GVPO uses human-labeled data directly—similar to DPO—it likewise avoids reliance on a reward model's quality. However, when GVPO is trained using a learned reward model as a proxy, its performance, like that of other RL algorithms, will depend on how accurately the proxy reflects true rewards. This proxy reward issue [1] is a well-known challenge in reinforcement learning. Consequently, it is common practice to evaluate post-training algorithms in the RLVR setting.\n\n[1] Skalse, Joar, et al. \"Defining and characterizing reward gaming.\" Advances in Neural Information Processing Systems 35 (2022): 9460–9471.\n\n**2. Redundant Presentation of Equations**\n\nWe appreciate your detailed suggestion. We will revise the manuscript to first introduce $R_\\theta$, followed by the MSE equations. We will then explain how these equations naturally arise from the condition $\\sum w_i=0$.\n\n**3. Redundant Theoretical Proofs in Section 3.3**\n\nThank you for this helpful suggestion. We will move less essential proofs (such as Lemmas 3.4 and 3.5) to the appendix to allocate more space for experimental results.\n\n**4. Misleading Name (\"Variance\")**\n\nWe apologize for any confusion caused by the use of “Variance” in GVPO’s name. Our intention was to create a counterpart to GRPO’s \"Relative\" terminology. Specifically, while GRPO’s weight includes a term $R - \\overline{R}$, GVPO’s MSE loss $[(R_\\theta - \\overline{R_\\theta}) - (R - \\overline{R})]^2$ can be rewritten in a variance-like form: $[(R_\\theta - R) - \\overline{(R_\\theta - R)}]^2$. We will clarify this interpretation in the revised manuscript.\n\n**5. Insufficient Experimentation. How can the experimental section be expanded to provide more thorough empirical validation of the proposed method ?**\n\nWe have strengthened our empirical evaluation in the rebuttals by\n+ adding experiments on an additional base model (in the Response to Reviewer KXfz)\n+ incorporating a broader range of benchmarks (in the Response to Reviewer X8wt)\n+ conducting ablation studies on the regularization terms (in the Response to Reviewer u3Wn)\n+ performing robustness check across seeds (in the Response to Reviewer u3Wn)\n+ including results on an alternative summarization task (in the Response to Reviewer X8wt)."}, {"review_id": "HGS4KZqDtB", "reviewer": "Reviewer_KXfz", "summary": "This paper introduces a new loss function for post-training LLMs that incorporates KL-constrained reward maximization directly into its gradient updates. In particular, the paper first identifies that common post-training methods such as SFT, rejection sampling, GRPO, etc, have a unified structure, as a weighted sum of log-likelihood gradients. Then, improving on this structure, they propose a novel loss with weights as the difference between implicit rewards defined in terms of $\\pi_{\\theta}$ and $\\pi_{\\theta'}$ and actual rewards. The implicit reward expression is derived from the optimal solution to the KL-constrained reward maximization problem in line to the DPO framework. They then meaningfully reinterpret this as the GVPO loss which minizes the squared distance between implicit rewards and actual rewards. \n\nThey analyze the theoretical guarantees of this GVPO loss in terms of optimal solution, different sampling policy, convergence guarantees, and unbiasedness of the proposed estimator. They detail two discussion section where they this loss function with DPO and GRPO respectively. They evaluate their proposed loss against GRPO and Dr. GRPO losses on the Qwen2.5-Math-7B model. They further conduct ablation studies on Qwen2.5-Math-1.5B model.", "questions": "1. From my understanding, $Var(\\log(\\pi_{\\theta}))$ is not related to rewards at all. Then how can one claim in Lines 259-261, that this expression is better than entropy regularization in terms ofensuring undesirable responses receive zero probability, while favorable responses retain comparable probabilities? Are there any experimental validation to this claim?\n\n2. Were more experiments conducted comparing GRPO to GVPO apart from Qwen series of models? This can be crucial to understand the efficacy of the proposed loss. I am willing to change my score based on such evaluation.", "limitations": "Yes, but not in the main text.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. A novel loss formulation that incorporates the implicit reward expression from the optimal solution to the KL-constrained reward maximization problem into the GRPO loss framework \n2. Clear analysis of the loss and its solution, along with a proper discussion comparing it to DPO and GRPO losses\n\nWeaknesses:\n1. Limited Experimental Evaluation (experiments run only on Qwen2.5-Math-7B model)\n2. Limited theoretical novelty. Note that the unified loss gradient framework in terms of weights has already been introduced in [1]. The primary novelty in this work seems to be novel weight assignment in terms of implicit and actual rewards. It straightforwardly follows from the definition of implicit rewards that the optimal solution to such a loss is also the optimal solution to the KL-constrained reward maximization problem. \n\n\n\n[1] Gao, B., Song, F., Miao, Y., Cai, Z., Yang, Z., Chen, L., ... & Chang, B. (2024). Towards a unified view of preference learning for large language models: A survey. arXiv preprint arXiv:2409.02795.", "was_revised": true, "final_justification": "New experimental evidence has been provided to further justify the method's efficacy. However, the concerns regarding theoretical novelty and the utility of the variance term in practice still persist. As such, I cautiously increase my score to 4.", "rebuttal": "We appreciate your recognition of the novelty in our loss formulation and the clarity of our analysis.\n\nBuilding on this foundation, we respectfully address the following points of concern to further clarify our contributions.\n\n\n**1. Limited Experimental Evaluation (experiments run only on Qwen2.5-Math-7B model)**\n\nWe have included experiments on an additional base model as presented below. \n\nFurthermore, we have expanded our evaluation in the rebuttals by incorporating more benchmarks, conducting ablations on the regularization terms, performing robustness check across seeds, and evaluating an alternative task of summarization.\n\n**2. Limited theoretical novelty.**\n\n**It straightforwardly follows from the definition of implicit rewards that the optimal solution to such a loss is also the optimal solution to the KL-constrained reward maximization problem.**\n\nWhile the necessity of the theorem ($R_\\theta = R$ is an optimal solution) follows directly, establishing its sufficiency (the only optimal solution is $R_\\theta = R$) is significantly more challenging and is addressed in detail in Appendix B.1.\n\nThe sufficiency result is novel—for instance, DPO establishes only necessity, not sufficiency, and admits multiple optimal solutions, some of which does not correspond to the optimal solution of the KL-constrained reward maximization objective (see Lines 231–238).\n\n\n**$Var(log(\\pi_\\theta))$ is not related to rewards at all. Why is this expression better than entropy regularization in terms of ensuring undesirable responses receive zero probability, while favorable responses retain comparable probabilities? Are there any experimental validation to this claim?**\n\nWe justify this expression both theoretically and empirically.\n\nFirst, we clarify that $Var(\\log(\\pi_\\theta))$ is intended to ensure some responses receive zero probability, while other responses retain comparable probabilities. As just introduced in the paper, the term $R - \\overline{R}$ promotes advantage maximization; therefore, $Var(\\log(\\pi_\\theta))$ serves to ensure that undesirable responses receive zero probability, while favorable responses maintain similar probabilities. We will revise the manuscript to make this point clearer.\n\nTo illustrate the benefit of $Var(log(\\pi_\\theta))$, consider a toy example involving generation over the tokens $a, b, c, d, e$, where $\\pi_\\theta(a) = \\pi_\\theta(b) = 0$ and $\\pi_\\theta(c) = \\pi_\\theta(d) = \\pi_\\theta(e) = 1/3$. In this case, $Var(\\log(\\pi_\\theta)) = 0$, indicating minimum variance and thus no penalty on this distribution. In contrast, entropy regularization only reaches its minimum either when the distribution is uniform or when it is one-hot, depending on whether one encourages higher or lower entropy.\n\nWe further validate this claim through ablation studies.\n\n| Algorithm           | AIME2024 | AMC    | MATH500 | Minerva | Olympiadbench |\n|--------------------|----------|--------|---------|---------|----------------|\n| GVPO          | 20.72 | 62.65 | 83.80 | 45.95 | 46.96 |\n| GVPO $w/o$ Var          | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |\n| GVPO $w/o$ Var $w/$ Entropy          | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |\n| GVPO $w/o$ Var (LR=1e-6)          | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |\n| GVPO $w/o$ Var $w/$ Entropy (LR=1e-6)          | 3.02 | 39.76 | 33.20 | 23.16 | 8.89 |\n\nIn the first set of experiments, we (a) remove the $Var(\\log(\\pi_\\theta))$ term, and (b) replace it with entropy regularization. In both cases, the models fail to converge and produce incoherent outputs, demonstrating that $Var(\\log(\\pi_\\theta))$ plays a crucial role in stabilizing training by discouraging extreme probability assignments.\n\nIn the second set, we lower the learning rate to $1\\text{e}{-6}$ and repeat the experiments. We observe that removing the term still leads to divergence, whereas replacing it with entropy regularization shows slightly improved but still unsatisfactory performance. These results suggest that while entropy regularization serves a similar purpose, it does not fully substitute $Var(\\log(\\pi_\\theta))$. Moreover, the poor performance may stem from entropy regularization being either too weak—failing to suppress extreme values—or too strong—hindering convergence. This highlights a key drawback of entropy regularization: it requires careful coefficient tuning. In contrast, the coefficient for $Var(\\log(\\pi_\\theta))$ is determined analytically via the optimal solution theorem, thereby avoiding this sensitivity and improving robustness.\n\n**Were more experiments conducted comparing GRPO to GVPO apart from Qwen series of models?**\n\nYes. We conducted additional experiments using Llama-3.1-8B-Instruct as a new base model. The results consistently show that GVPO outperforms GRPO under this new setting, demonstrating its effectiveness again.\n\n| Algorithm           | AIME2024 | AMC    | MATH500 | Minerva | Olympiadbench |\n|--------------------|----------|--------|---------|---------|----------------|\n| Llama-3.1-8B-Instruct       |  5.00   | 19.27  | 50.40   | 22.42   |  17.04         |\n| +GRPO              | 8.54    | 19.27  | 52.60   | 25.37   | 16.15          |\n| **+GVPO**          | **11.56**| **20.48** | **56.60** | **29.41** | **20.59** |"}]

Author rebuttal:

Meta-review: Building upon previous work of GRPO, the paper utilizes multiple rollouts per prompt for robust advantage evaluation in RLHF. The paper proposed to use the squared difference in the central distance of implicit rewards and the actual rewards as the minimization objective and proved several nice properties of such optimization objective including: (1) the solution is the minimizer of a KL regularized reward maximization problem; (2) the method is robust to mis-specification of the sampling distribution by avoiding the importance sampling term. The derivation relies on the zero-sum property of the group calibrated rewards to avoid the computation of the partition term for reward-tilted distribution, which is a nice insight and could be of independent interest. The paper also demonstrates empirically that the method outperforms GRPO, and a few other baseline methods (a few added during discussion) on standard benchmarks.

A majority (4 out of 5) of the reviewers like solid theoretical justification of the method and its strong empirical performance against other baselines. During the discussion phase, the authors further addressed reviewers' concerns on the empirical results on the paper, including providing comparison to other baseline algorithms, experiments on non-math tasks, ablation studies on different characteristics of the algorithm, and benchmarks on non-Qwen models. Most of the reviewers liked the rebuttal and mentioned that their concerns were addressed if the authors could include these new experiments in the revised version.

Reviewers also raised the concern that the presentation of the paper could be improved by shortening the redundant equations in the main proof and add more ablations and empirical validations of the algorithm. I agree with the reviewers' suggestion that the presentation of the theoretical part needs some shortening for precise presentation.

After rebuttal, one reviewer raised a concern about the lack of solid theoretical justification on why the method works, which is a fair point. But I believe this point doesn't outweigh the paper's theoretical and empirical contributions.

Overall, I believe the paper is a solid contribution to the field, and hence I will recommend acceptance. The authors should include the new experiments provided in the discussion and revise their paper based on the suggestion provided by the reviewers.
Decision: Accept (poster)