Title: GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Post Training

Reinforcement Learning

Reasoning

GVPO

GRPO
Abstract: Post-training plays a crucial role in refining and aligning large language models to meet specific tasks and human preferences. While recent advancements in post-training techniques, such as Group Relative Policy Optimization (GRPO), leverage increased sampling with relative reward scoring to achieve superior performance, these methods often suffer from training instability that limits their practical adoption. As a next step, we present Group Variance Policy Optimization (GVPO). GVPO incorporates the analytical solution to KL-constrained reward maximization directly into its gradient weights, ensuring alignment with the optimal policy. The method provides intuitive physical interpretations: its gradient mirrors the mean squared error between the central distance of implicit rewards and that of actual rewards. GVPO offers two key advantages: (1) it guarantees a unique optimal solution, exactly the KL-constrained reward maximization objective, (2) it supports flexible sampling distributions that avoids on-policy and importance sampling limitations. By unifying theoretical guarantees with practical adaptability, GVPO establishes a new paradigm for reliable and versatile LLM post-training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new loss function for post-training LLMs that incorporates KL-constrained reward maximization directly into its gradient updates. In particular, the paper first identifies that common post-training methods such as SFT, rejection sampling, GRPO, etc, have a unified structure, as a weighted sum of log-likelihood gradients. Then, improving on this structure, they propose a novel loss with weights as the difference between implicit rewards defined in terms of $\\pi_{\\theta}$ and $\\pi_{\\theta'}$ and actual rewards. The implicit reward expression is derived from the optimal solution to the KL-constrained reward maximization problem in line to the DPO framework. They then meaningfully reinterpret this as the GVPO loss which minizes the squared distance between implicit rewards and actual rewards. \n\nThey analyze the theoretical guarantees of this GVPO loss in terms of optimal solution, different sampling policy, convergence guarantees, and unbiasedness of the proposed estimator. They detail two discussion section where they this loss function with DPO and GRPO respectively. They evaluate their proposed loss against GRPO and Dr. GRPO losses on the Qwen2.5-Math-7B model. They further conduct ablation studies on Qwen2.5-Math-1.5B model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. A novel loss formulation that incorporates the implicit reward expression from the optimal solution to the KL-constrained reward maximization problem into the GRPO loss framework \n2. Clear analysis of the loss and its solution, along with a proper discussion comparing it to DPO and GRPO losses\n\nWeaknesses:\n1. Limited Experimental Evaluation (experiments run only on Qwen2.5-Math-7B model)\n2. Limited theoretical novelty. Note that the unified loss gradient framework in terms of weights has already been introduced in [1]. The primary novelty in this work seems to be novel weight assignment in terms of implicit and actual rewards. It straightforwardly follows from the definition of implicit rewards that the optimal solution to such a loss is also the optimal solution to the KL-constrained reward maximization problem. \n\n\n\n[1] Gao, B., Song, F., Miao, Y., Cai, Z., Yang, Z., Chen, L., ... & Chang, B. (2024). Towards a unified view of preference learning for large language models: A survey. arXiv preprint arXiv:2409.02795.\n\nQUESTIONS:\n1. From my understanding, $Var(\\log(\\pi_{\\theta}))$ is not related to rewards at all. Then how can one claim in Lines 259-261, that this expression is better than entropy regularization in terms ofensuring undesirable responses receive zero probability, while favorable responses retain comparable probabilities? Are there any experimental validation to this claim?\n\n2. Were more experiments conducted comparing GRPO to GVPO apart from Qwen series of models? This can be crucial to understand the efficacy of the proposed loss. I am willing to change my score based on such evaluation.\n\nLIMITATIONS:\nYes, but not in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces Group Variance Policy Optimization (GVPO), a new approach for post-training LLMs that ensures reliable and robust model alignment with specific tasks and human preferences. GVPO directly incorporates the analytical solution to KL-constrained reward maximization into its gradient weights, offering two key advantages: a unique optimal solution aligned with the KL-constrained reward maximization objective and support for flexible sampling distributions that avoid limitations of on-policy and importance sampling methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. GVPO builds on the insights from Group Relative Policy Optimization (GRPO) but addresses its training instability issues. By designing a weighting scheme where the gradient weight of a response is the difference between the central distance of implicit rewards (derived from the current and reference policies) and actual rewards, GVPO ensures that the sum of weights within a prompt group equals zero. This design eliminates the need for explicit estimation of the partition function.\n\n2. Theoretical guarantees of GVPO are rigorously established. It is proven that GVPO’s loss function is aligned with the original constrained optimization problem, ensuring convergence to the globally optimal policy. \n\n3. A Few experimental results demonstrate GVPO’s effectiveness. \n\n**Weaknesses**:\n\n1. **Separation of Reward Modeling and Policy Optimization**:\n    GVPO appears to reintroduce the separation between reward modeling and policy optimization. Unlike DPO, which integrates reward model training and policy optimization into a single process by exploiting their analytical relationship, GVPO seems to require a predefined reward model  $R$  (as indicated in Equation (9), not $R_{\\theta} = \\frac{\\log(\\pi_{\\theta})}{\\log(\\pi_{\\theta^{'}})}$). This raises questions about whether GVPO necessitates a separate training phase for the reward model before applying GVPO. If so, this could be seen as a limitation, as the performance of GVPO might become heavily dependent on the quality of the separately trained reward model, potentially undermining its overall effectiveness.\n    \n 2. **Redundant Presentation of Equations**:\n    The presentation of equations in the paper could be streamlined for better clarity. For instance, the two equations introduced between lines 142--152 appear to primarily serve the purpose of supporting the equation presented between lines 159--160 that illustrates that GVPO is equivalent to optimizing a mean squared error (MSE) loss that measures the discrepancy between implicit and actual reward central distances. To enhance the flow, the authors could first introduce the symbol $R_{\\theta} = \\frac{\\log(\\pi_{\\theta})}{\\log(\\pi_{\\theta^{'}})}$, then present the equation between lines 159--160. Following this, they could explain how the equation between lines 142--152 arises from the condition $\\sum w_{i}=0$ for verifying the equation between lines 159--160.\n    \n 3. **Redundant Theoretical Proofs in Section 3.3**:\n    The theoretical proofs in Section 3.3 feel repetitive, as they focus extensively on retelling the result that GVPO is equivalent to optimizing an MSE loss measuring the discrepancy between implicit and actual reward central distances. \n    \n \n  4. **Misleading Name (\"Variance\")**: \n    The choice of the name \"Variance\" for the proposed method might be misleading. The new weight introduced in GVPO is actually a difference of two deviations from the the mean rather than a pure variance. This could cause confusion regarding the method's core mechanism and its relationship to statistical variance.\n\n5. **Insufficient Experimentation**:\n    Since this paper focuses on proposing a new post-training algorithm rather than being a theoretical reinforcement learning(RL) paper, the experimentation section feels somewhat insufficient. The proposed method lacks thorough empirical validation across diverse LLMs, which would be crucial for demonstrating its effectiveness and generalizability in practical applications.\n\nQUESTIONS:\nSee Weaknesses. \n\nFor instance,\n\n1. **Separation of Reward Modeling and Policy Optimization**\nDoes GVPO require a separate training phase for the reward model before applying the algorithm?\nHow does the performance of GVPO depend on the quality of the separately trained reward model?\n\n2. **Insufficient Experimentation:**\nHow can the experimental section be expanded to provide more thorough empirical validation of the proposed method ?\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a method for mitigating instability in GRPO, which it claims arises from exploding gradients that are a consequence of minimizing log likelihood, called GVPO. GVPO is a weighted policy gradient objective, where weights sum to zero and are derived from the closed form of the KL-regularized policy objective. There are a number of theoretical results showing that GVPO recovers the unique optimal policy, which rely on a reduction from GVPO to squared-loss regression. Experiments are provided in a number of mathematical reasoning datasets.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I think that the proposed method is interesting, but requires more in-depth empirical investigation. The clarity of the writing, motivation for the paper, and theoretical results are somewhat weak in my view, and as a result I would have really liked to see ablations targeting the purported variance-reduction benefits of GVPO. \n\n**Motivation**\n\nTo begin with, I found the motivation for the paper to be underwhelming and largely unsupported. The motivation for the method is that GRPO is unstable because \"minimizing the log-likelihood of those responses can lead to exploding gradients due to the convexity of the logarithmic function\" (L107-108), but there is no evidence within the paper that this is true, nor citations for this claim. \n\nIn addition, the motivation for the method is to reduce policy learning to reward learning (L148, 151-152), but why is this better than optimizing the policy? And what is the relationship with GVPO (Eq. 9) itself?\n\n**Method and theoretical results**\n\nThere are interesting manipulations and interpretations of the gradient objective: using policy gradient weights that sum to 0, the partition function no longer needs to be estimate, which allows the objective to be converted to squared loss regression. This seems possibly straightforward given the form of Eq. 9 and the definition of $R_\\theta = \\log \\frac{\\pi_\\theta}{\\pi_{\\theta'}}$, and the theoretical results are somewhat simple and repetitive (in my opinion, not generating much more insight) beyond the sq-loss interpretation. \n\nAs a result, I wonder if the paper might be made stronger if Sections 3.1 and 3.3 were shortened in favor of more insightful experiments and ablations, discussed below. \n\n**Experiments**\n\nI appreciate that the authors included both theoretical and empirical results, and considered a few datasets. However, it appears that no standard deviations or averages over seeds are reported, which I find discomfiting given the variance of policy gradient methods for training language models (e.g., GVPO, by the paper's own admission). While I understand that better performance can reflect increased stability, I think that the experimental results would be more interesting if they explicitly highlighted the intended benefits of GVPO.\n\nThe objective resembles the GRPO objective, except it doesn't divide by the standard deviation, and has an additional pair of log policy-ratio terms that can be viewed as implicit rewards. But it has been observed empirically in DPO that such policy-parameterized rewards can be suboptimal or less stable compared to reward models (Swamy et al. 2025; Lin et al. 2024). As such I am curious about the effect of the policy ratio terms, I would think that there could be interesting ablations to conduct on their role in stability, as well as in probing the predictions in Section 3.5. It's not really clear to me how much benefit comes from removing the standard deviation normalization in GRPO (as discussed in L245-249) versus adding the log policy ratios.\n\n**References**\n\nSwamy, Gokul, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell. \"All roads lead to likelihood: The value of reinforcement learning in fine-tuning.\" _arXiv preprint arXiv:2503.01067_ (2025).\n\nLin, Yong, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. \"On the limited generalization capability of the implicit reward model induced by direct preference optimization.\" _arXiv preprint arXiv:2409.03650_ (2024).\n\nQUESTIONS:\nWhy does incorporating the log-policy ratio terms into GVPO increase stability?\n\nLIMITATIONS:\nSee comments above", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents a new method for policy optimization for LLM post-training, referred to as Group Variance Policy Optimization (GVPO).
GVPO puts together various ideas from policy optimization and preference learning in a new way.  In particular, via a clever trick that centers the weights of gradients (where the weights are differences in log-likelihood ratios), a partition function can be made to vanish in the gradient.  This yields not only an efficient method, but an interpretable one.

STRENGTHS AND WEAKNESSES:
Strengths:  The authors do an excellent job at situating their method in the landscape of existing methods, and pointing out its relative advantages, which include a lack of a need for importance sampling and on-policy gradients, and therefore a lack of need for various ad hoc adjustments including clipping.  The authors also do an excellent job at interpreting the gradient of their method, in three different ways, all insightful.  The experiments also seem compelling.

Weaknesses:  None that I can see.

QUESTIONS:
It would be nice to understand whether there is a more general principle at work here.  As currently presented, the idea (center the weights) comes across as a (nice) trick.  Are there analogous methods in the operations research literature?  Or the optimization literature?

In particular, the objective function studied here is of the form of mirror descent, which may provide a path towards a deeper understanding and even obtaining theoretical convergence rates.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 1
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Group Variance Policy Optimization (GVPO), a method for KL-constrained reward maximization. To derive GVPO, the paper identifies that when the sum of assigned weights to all responses within a prompt equals zero, then the partition function becomes invariant across the compared responses. The paper presents a clean interpretation as well as connections with existing literature on policy optimization methods. The paper also presents empirical results highlighting the usefulness of the proposed policy search method on math reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Important problem, well motivated and well written paper\n- Clearly well situated with important (and most well known) related works such as GRPO, DPO etc.\n- clear derivation and strong empirical results\n\nWeaknesses:\n- Insufficient empirical benchmarking\n\nQUESTIONS:\nWhile the paper presents clear benchmarking for math reasoning tasks, it is worth asking the broader question -- what is the role of policy optimization in general natural language generation tasks? Does better policy optimization lead to better alignment? This question is more nuanced since there are other factors at play about quality of reward models, how they align with human preferences, reward hacking etc. I think the paper benefits a discussion surrounding these issues, and additional experimentation to highlight the nuanced nature of the discussion surrounding development of better policy search methods. Towards this, I'd like to see (a) additional results on natural language generation tasks such as summarization, question answering etc., (b) whether a better policy search method naturally translates to better evals on automated/human eval on alignment quality. My sense is these results will be illuminating and may not fully line up in terms of improvement in metrics, yet will be beneficial to guide subsequent research in this topic.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}