RiskPO: Risk-based Policy Optimization with Verifiable Reward for LLM Post-Training
Authors: Tao Ren
Jinyang Jiang
Hui Yang
Wan Tian
Minhao Zou
Guanghao Li
Zishi Zhang
Qinghao Wang
Shentao Qin
Yanjun Zhao
Rui Tao
Hui Shao
Yijie Peng
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning with Verifiable Reward
Risk-Sensitive RL
Primary area: reinforcement learning

Abstract: Reinforcement learning with verifiable reward has recently emerged as a central paradigm for post-training large language models (LLMs); however, prevailing mean-based methods, such as Group Relative Policy Optimization (GRPO), suffer from entropy collapse and limited reasoning gains. We argue that these issues stem from overemphasizing high-probability output sequences while neglecting rare but informative reasoning paths. To address these challenges, we propose Risk-based Policy Optimization (RiskPO), which substitutes classical mean-based objectives with principled risk measures. Specifically, we introduce a Mixed Value-at-Risk objective that integrates weighted attention over multiple regions of the reward distribution, thereby amplifying gradient signals on challenging instances and preventing overconfident convergence. We further design a bundling scheme that aggregates multiple questions into bundles, thus enriching the feedback signal and yielding more stable and informative training dynamics. Theoretically, we prove that the risk-averse update alleviates entropy collapse and promotes exploration. Numerically, RiskPO achieves consistent and significant improvements in mathematical reasoning, multi-modal reasoning, and code generation benchmarks, surpassing GRPO and its variants on both Pass@1 and Pass@k metrics. Our results demonstrate that risk-based optimization provides a rigorous and effective paradigm for enhancing LLM reasoning capabilities. The implementation is available at https://github.com/RTkenny/RiskPO.

Published as a conference paper at ICLR 2026
RISKPO: RISK-BASED POLICY OPTIMIZATION VIA VERIFIABLE REWARD FOR LLM POST- TRAINING
Tao Ren' Jinyang Jiang'
Hui Yang" Wan Tian'
Minhao Zou'
Guanghao Li
Zishi Zhang' Qinghao Wang' Shentao Qin Yanjun Zhao" Rui Tao?
Hui Shao Yijie Pengl.6j
Peking University Tsinghua University OriginFlow *Xi'an Jiaotong University
5 Zhejiang University Xiangjjang Laboratory
ABSTRACT
Reinforcement learning with verifiable reward has recently emerged central paradigm for post-training large language models (LLMs); however, prevailing mean-based methods, such Group Relative Policy Optimization (GRPO), suf- fer from entropy collapse and limited reasoning gains, which stem from overem- phasizing high-probability output sequences while neglecting rare but informative reasoning paths. To address these challenges, we propose Risk-based Policy Op- timization (RiskPO), which substitutes classical mean-based objectives with prin- cipled risk measures. Specifically, we introduce a Mixed Value-at lalue-at-Risk objective that integrates weighted attention over multiple regions of the reward distribu- tion, thereby amplifying gradient signals on challenging instances and preventing overconfident convergence. We further design a bundling scheme that aggregates multiple questions into bundles, thus enriching the feedback signal and yielding more stable and informative training dynamics. Theoretically, we prove that the risk-averse update alleviates entropy collapse and promotes exploration. Numer- ically, RiskPO achieves consistent and significant improvements in mathematical reasoning, multi-modal reasoning, and code generation benchmarks, surpassing GRPO and its variants on both Pass @ and Pass@k k metrics. Our results demon- strate that risk-based optimization provides a rigorous and effective paradigm for enhancing LLM reasoning capabilities. The implementation is available at https: com Co..///oo.co///..ccc/////EEE
INTRODUCTION
Since reinforcement learning (RL) provides unified framework that flexibly accommodates diverse train- ing targets and feedback, it has become a key tech- nique for the post-training of large language models (LLMss. Based on such a foundation, RL with veri- fiable reward (RLVR) has recently been recognized as an effective paradigm for enhancing the reasoning abil- ity of LLMs. Unlike traditional RL from human feed- back, it leverages objective and binary reward signals, providing clear optimization feedback. Maximizing the expected average reward is anticipated improve task performance of LLMs. Within this framework, series of -fficiency-orientedd extensions have been developed from the classical policy-based RL method. Among them, Group Relative Policy Optimization (Shao al.) 2024 Guo et al.) 2025 GRPO) achieves substantial ef-
Performance on AIME2024
GRPO passli32 0.52
Pase/Aog llk Store
0.5
0.4
GRPO ag 832 1.195
REAPO puaiitt
RS-P0 20g
00
100
30C
350
400
100
Training Steep
Figure 1: Pass 32 and Avg@ 32 learning curves of DeepSeek-RL -i-eek----issilll DeiisiillQwenn 1.5B trained by RiskPO on AIME2024. ficiency gains by discarding redundant structures originally designed for standard RL tasks, and has
Equual contribution.
Corresponding author: hui shaoo nt z.j...dee edu.cn en, pengyi jinryijjeiee jieēpku. edu.cn.

---

Published as a conference paper at ICLR 2026
become the de facto baseline in this area. Since then, several GRPO variants have been proposed; see Section for details.
However, RLVR methods that maximize average performance suffer from the fundamental issue of entropy collapse. Prior work shows that models trained via RLVR often experience rapid en- tropy collapse in the early stages of training, leading premature convergence and plateau in performance with little subsequent improvement (Cui et al. 2025 Gao et al.] 2025). Entropy, as emphasized by several studies, serves as a key indicator of exploration capacity in RL (Wang al.) 20256 Cheng el al., 2025 Hou et al. 2025). Once entropy collapses, the model becomes over- confident, reduces exploration prematurely, and fails to acquire new knowledge effectively. This constrained exploration ultimately limits its reasoning capabilities and overall performance. As consequence, I.LMs do not truly expand their intrinsic reasoning capacity or boundary; the ob- served improvements often reflect more efficient sampling of known answers rather than genuinely stronger reasoning skills (Yue et al. 2025a Xiong et al. [225] Chen et al. 2025 Gao et al.) 2025). This boundary effect implies that GRPO may only enhance short-horizon performance metrics (c.g., @ass@) I) without significantly lifting the capability of the base model.
We argue that key reason behind these chal- lenges is that GRPO employs the mean as its ob- jective, which is inherently misaligned with the goal of improving reasoning ability. mean- based objective disproportionatelly emphasizes common, high-probability generation paths while neglecting rare yet informative reasoning trajec- tories, leading to premature convergence and lim- ited exploration. Even worse, if the model con- sistently generates all wrong answers to ques- tion, the estimated GRPO advantage collapses to zero, leaving the model without any effec- tive learning signal its weakest areas. Over- concentrating gradient updates on casier ques- tions brings only marginal gains, as it mainly re- inforces what the model already knows instead of improving ability on harder problems. In contrast, risk-averse optimization objectives, such as Conditional Value-at-Risk (CVaR) or Range Value-at Matie-at-Reee -isk (RVaR), can encourage the model to explore difficult problems and enhance rea- soning abilities. By amplifying gradient signals from low-reward answers, these objectives natu- rally encourage the policy to reduce overconfidence, diversify its search, and promote novel reason- ing strategies. Consequently, risk-based objectives provide an effective handle for better mitigating entropy collapse, preventing overfitting to easy problems, and driving genuine improvements of the reasoning boundary.
Reward Signal Enrichment
Question
Question
Question
Spurse
Rg
RS
RS
Bundle
Dundle
Densse
RS
2. Risk Policy Optimization
Risk profile (a..,,.m.))
Risk profile (1g.)
(12.85.015)
Evalisatc Bg $ Track 96.4g
Pette 17 Reward dist.
Update 6g
Policy entropy
Risk-scoking
Rink-ncutral
Risk-averee
Figure The framework of RiskPO.
We propose Risk baskd Policy Optimization (RiskPO), which employs a novel risk-sensitive objec- tive termed Mixed Value-at-Rish (MVaR). Compared with mean-based post-training methods, risk-based approach demonstrates superior performance in encouraging exploration and fostering stronger reasoning capabilities. The overall framework of RiskPO illustrated in Figure We summarize our contributions as follows:
1. To the best of our knowledge, we are the first to incorporate risk measures into the training objective. Since the reward for a single question is binary, we propose grouping multiple ques- tions into bundle to enrich the feedback signal.
shown to avoid the zero advantage issue and strengthen gradient signals for hard problems, thereby facilitating exploration.
2. We provide theoretical results that explain the superiority of the proposed MVaR objective. By analyzing the entropy mechanism. we demonstrate that the risk-averse configuration can effectively mitigate entropy collapse.
3. We conduct extensive numerical experiments to evaluate the performance of our algorithm. RiskPO consistently outperforms GRPO and other baselines multiple mathematical reason-
2

---

Published as a conference paper at ICLR 2026
ing tasks. On Pass @k metries, RiskPO even achieves better performance, indicating its strong capacity for exploration and acquisition of new reasoning skills.
2 RELATED WORKS
RL for LLM Post- Training. RL has played critical role in the post-training phase of LLM (Shao
al. 2024 Christiano al. 2017 Lambert et al. 2022). Through verifiable reward, the LLM learn
inapeicated reasoning skills across solving math problems and coding (Zhao et al. 2025a Chen
al. 2024 Huang et al. 2025). Originating from the PPO, much literature has proposed new meth-
ods to cater to the requirements of RLVR. ReMax (Li et al., 2024) proposes to use the deterministic output of the LLM as the baseline to reduce variance. DAPO (Yu et al., 2025) incorporates four engineering tricks to improve GRPO. VAPO shows that the value-based RL method can also per- form well in RLVR (Yue et al.] (0255). GPG (Chu et al. (2025) investigates the normalizing factor in GRPO. Several literatures (Wang et al., [025b] Cui et al. 2005; Cheng et al. 2025 Wang al,, 2025a) investigate the entropy mechanism in RLVR, pointing out the significance of exploration in RLVR. GSPO (Zheng al. (0255) and GMPO (Zhao et al.) 2025b) focus stabilizing the RL. training. ProRL (Liu et al. (025a) shows that with stabilized training, the performance gain would have a log-scale relationship with the training time.
Risk-Sensitive RL. Risk-sensitive RL (see, e.g.. Ren et al. 2024 Petersen et al., 2019) seeks to
shape the entire reward distribution rather than merely optimizing mean. Chow et al. (2015) investigates the Markov Decision Process (MDP) under CVaR objective and proposes dynamic- programming based solution. La & Ghavamzadehi (2033) and PPashhnnhh et (2016) use finite difference to optimize risk measure under the MDP setting. Dabney et al. (201 8b a) introduces state- action value distribution approximation techniques to improve the effectiveness, which is referred to distributional RL. CVaRPG (Tamar et al. 2015) and QPO (Jiang et al.) (022) design policy gradient style algorithm to optimize CVaR and quantile, respectively. Jiang et al. (2024) considers more general case, optimizing the distortion risk measure in policy gradient manner. There is also literature considering risk level as a constraint. [Bertsekas] (1997) use a Lagrangian approach to [ndCCow solve RL problems. BBorarr & Jain 2014) use CVaR constraint, and Chow et al. (2018) develop actor-critic algorithms under quantile and CVaR constraints.
3 RETHINKING RLVR FROM A DISTRIBUTIONAL PERSPECTIVE
We formalize the post-training problem of RLVR as follows. Given an input problem x sampled from dataset D, an LLM parameterized as my generates response y ~ (=(2))) A rase- verifier RZ(-) then evaluates the correctness of the response, returning one if y is correct and zero otherwise. Notably, no intermediate process-level feedback is provided. The standard objective in this setting is to maximize the expected reward: J J()) EEDD, y--m(--la) [[y)]. With score- function method, its gradient is given by VoJ (0) = TR. R(y)) Vo In ()]rr)) resulting in a standard RL framework, where a baseline or so-called value model is used for variance reduction.
As widely adopted baseline for RLVR, GRPO (Shao et al., 2024) replaces the value model with sequence-level standardized rewards computed within a group of responses. We denote by Vee the partial response consisting of the first / tokens, i.e., To (y]=c) = II, 7 (xl,,, Vee). Specifically, given query and group of G' responses { sampled from a reference model ne (-Iz), the GRPO objective is defined as
Iel
Σ min (w), clip(w,.:00),1 e, c,1+ E) A.)]
(8) E
IND {vr 12-11 mor
where A, R(y+)- à E M(ys) denotes the standardized feedback, while (0) wh,
no (v.. M...
Stdl[ [1,;(1,
Ho (35, 12.8
is the importance sampling ratio that enables multiple parameter updates per group of generated data. Despite these modifications, GRPO remains fundamentally a method for optimizing the mean performance of LLMs. Since the reward provided by the verifier is an indirect objective, it may not be the best practice for RLVR to optimize its expectation. Instead, we propose to adopt distributional perspective. The most challenging problems correspond to the left tail of the reward
argue
3

---

Published as a conference paper at ICLR 2026
distribution. These samples represent the questions that the model has not yet mastered. Such hard cases often lead to gradient vanishing in GRPO. For example, when all responses are incorrect, the computed advantage collapses to zero, which provides no meaningful training signal. As result, the model fails to improve on its weakest regions of the distribution.
Therefore, beyond optimizing the expectation, claim that it is more beneficial to consider the distributional structure of performance, particularly the lower tail. Incorporating risk measures, such as CVaR or RVaR, into the training objective emphasizes hard problems in the tail of the distribution and provides finer-grained and more robust learning signal for RLVR.
MASTERING THE UNCERTAINTY VIA RISK-BASE POLICY OPTIMIZATION
In this section, we introduce our risk-based objective for RLVR and the associated post-training methodology, establishing principled framework for enhancing LLM reasoning ability.
Denote the RLVR reward signal distribution by Fo()), where the parameter reflects the stochasticity induced by the LLM No (-(xz)) RVaR is defined to capture the average performance within specified
quantile interval of the distribution. Let Fg (a) be the a-level quantile of R(y). Then, for 0 S <
< 1, RVaR on the interval [a, 3] is written as
1 :R e (0) = B (R(y) R(y) E Fa), 3 a P (a)
(3)
rdFo(r),
(I)
that is, the conditional expectation of R(y) given that it falls between its and B-quantiles. To optimize the RVaR through gradient descent algorithms, we first derive the gradient of RVaR as shown in Theorem [ The proofs of all theoretical results in this work can be found Appendix B
Theorem 1. Assume Fo(r) is continuously differentiable with respect to both the parameter A and the variable r: the density is positive at the quantiles, i.e., fo (Fr (a)) O and fo (B)) O;
and that the differentiation under the integral sign is justified. Then the gradient of RVaR is given by
VoJTVaR.... (0) =
E [[g(R(y)) Fg (a), F7 (o
where g(2,a,b) a. b) (z -2)+ - (==-b) b)+ +a b. b. and (2)+ max(220).
O. RVaR coincides with the lower-tail CVaR at level B, and the gradient in Theorem reduces to VoJRVaRO (0) = ' (B) ( V In [y(x)))] Since RVaR effectively places window for control the reward distribution, it is natural to further combine several such segments to better shape the overall distribution. Accordingly, we introduce a new objective into RLVR, namely Mixed Value-at-Risk (MVaR), which integrates metrics over multiple distributional segments as follows:
Note that when
f }
F5 (a)
Fo (8)
((++w)
J M o.
rdF
(0)
Fg (T)
where w 0 controls the emphasis placed tail samples during optimization, and high- performance samples are excluded from the current training process. Note that JNNAR= (0)
(1 + JJJJava (0) + (B JRV.Rv (0). The gradient of 2 can be derived by Theorem !
However, the distributional information from a single question I is limited, since the feedback is binary and offers only coarse signals. To obtain a richer source of information, we propose to group several questions into a bundle, i.c., X := { ~ D&B and calculate the advantage based on the sum of the individual question scores within the bundle. This aggregation transforms sparse binary feedback into a more informative distribution over bundle scores, enabling finer distinctions between different levels of performance and avoiding zero gradient on difficult questions. We then focus on optimizing the MVaR of the bundle score:
E Ex-D8" (x))
RB + ( (a)) + I (003(00e0
4

---

Published as a conference paper at ICLR 2026
where Rg Σド= R(y') denotes the bundle score. For each i fl...., B), sample Y, { with yj TT(-12;) ii.d., and define Y {r
Then we generate G bundles without overlaps from the G x responses of questions. The gradient can calculated by
B
Vo Volnmo(Ne,, l2x)
EX-DOB (
Aj
B
1=1
1=1
where Aj -1 -1++ - RR,) + oRR Fo (a), Fo (B)) is the bundle-wise advan- tage under MVaR objective, RB, = {1....,G] G} that independently draw & ~ Unaf(Ga) for every i, GG is the symmetric group on G element, and E., is the j-th elements in the permutation. This construction yields G disjoint bun- dles: the j-th bundle uses (ue,, 1 so that for each fixed f. V. is permutation of wui.-..9h))- i.e., every answer used only once (without replacement)
is the bundle- wise score, ç is permutation of
To ensure stable improvement (Schulman et al. 2015 2017) with multiple updates per bundle- wise MVaR objective evaluation, we adopt a trust-region style update with clipping and sequence- level importance sampling (Zheng et al. 2025). Since the reward in RLVR is only available the sequence level, i.e., y", it is natural to define importance weights also at the sequence (response) level and then aggregate them into the bundle objective. Formally, given B problems X {a:) and G responses per problem Y; {uj 1
independently draw & Unif(Sa) for each i, yielding G bundles: P3 = {vÊ., } 1= 1,....C G, where every responses is used without replication. We then construct the clipped MVaR objective at the bundle level, which constitutes the final loss for backpropagation:
min (z)(0) A - A
Mvan (0) EE.Y,IE.
(3)
G
B
no(uE, (1x)) For (V.,, [z,)
1/1mé,,
where s)(
is the sequence-wise importance sampling ratio.
Every token within the same bundle shares the same M MaR-bbase advantage A ensuring that op- timization is aligned with the unit of reward (the bundle score) and directs training toward the left tail of the performance distribution. We track +aa(a) and Fo (B3) in online manner. After substitut- ing the tracked quantiles into the advantage and deriving the gradient, we update model parameters accordingly. Therefore, RiskPO can be implemented as two-timescale stochastic approximation algorithm. The pscudocode of the proposed algorithm is provided in Algorithm
Algorithm Risk-Based Policy Optimization (RiskPO)
1: Input: quantile levels o; 3, weight w, policy æ., learning rates (re), {mk), and iterations K
2: Initialize: policy parameter fg. and quantile trackers 88 of
3: for k 1, K do
4: Sample B questions, X = {x from the dataset D
5: Generate G responses for each question, { ~ . (z,) and evaluate the reward R(v²)
6: Sample from the symmetric group for B times, & ~ Unif(Ga), yielding G bundles
7: Track quantiles of bundle scores: of+1 = of + (s
ES 1 (Rn, < <vg)), E 3)
8: Backpropagate the clipped MVaR objective
33 to compute V Joini (0)
9: Update policy parameter: 0k+1 0 Th Vo Jolelip MVaR (0)
10: end for
Output: Final policy parameter 0K+1
5 ENTROPY MECHANISM FOR
OBJECTIVE
well-known issue GRPO is entropy collapse, where the policy entropy rapidly decreases dur- ing training. This premature reduction in entropy limits exploration of alternative reasoning paths,
5

---

Published as a conference paper at ICLR 2026
thereby constraining performance improvement and reducing the likelihood of discovering correct solutions. In this section, we conduct a per-step analysis for the change of policy entropy in the optimization and give theoretical guarantee that our RVaR policy gradient can mitigate the entropy collapse issue.
Following the standard framework in policy-gradiens literature (see, e.g.. Agarwal et al. 2021 Shani et al. 2020; Abbasi- Yadkori et al. 2019), we conduct theoretical analysis under a tabular softmax formulation with deterministic sequence-level rewards. Specifically, we consider an input set X and an output set y. The actor is parameterized by matrix 0 E R| where each entry BR,y is
the logit for choosing y given æ. The policy is thus To(y|æ) = Ener. exp(=z.u) and its conditional entropy is H( (r) = - -y (2(2) log To (v(a)). Let Ao(x, y) be the advantage value associated with the chosen algorithm. We begin with the following proposition, which links entropy dynamics to the covariance between the advantage and the log- bog-prbbability of the output.
exp(#r.g)
Proposition 1. Fix a prompt z and a finite set of complete sequences Ya. Consider a natural- gradient step, i.e., Bx+1 = 0% + nAs where A.. = Ao, (æ, y) otherwise zero, then
H (To 1z) H(FTO.1)) = -1 Covy y (-w)) (log To, (ylz); Ao, (x. + O(1Axx12). (4)
The proposition indicates that the correlation between the advantage and the log probability of the output affects entropy changes: a positive correlation leads to entropy decrease, and vice versa. High advantage and high log probability suggest that the model is very confident about samples with high advantage. Therefore, over-optimizing already well-learned problems accelerates entropy collapse, reducing the model's opportunities for trial and error on difficult problems and ultimately limiting policy performance. RiskPO, which uses MVaR as its objective, can alleviate this issue. It focuses more on difficult problems, i.c., samples from the left tail of the reward distribution, and clips the gradient signal for well-learned problems. In the following theorems, we provide theoretical jus- tification by analyzing the correlation between advantage and log-likelihood, showing that RiskPO induces higher-entrope policy updates. Before that, we present the following assumption, which cap- tures the most intuitive scenario but is not the only suitable one. Let (r)) = Bllog no (y|r)) = r] denote the conditional logeliiel..ood.
Assumption 1. The conditional log log-probability of output (br) is non-decreasing for r 2 Fø (B), non-increasing for r I Fg (a:), and (Fa (F)))))) ((F) (Fg (B)) N Bllog mo (y)
Intuitively, this assumption captures the behavior of a pre-trained base model. For relatively easier problems in the upper tail of the reward distribu- tion, the model exhibits well-calibrated confidence, assigning higher probabilities to correct answers (upper-tail monotonicity). In contrast, for harder problems in the lower tail, the model often fails systematically: it allocates substantial probability mass to incorrect outputs, effectively making con- fident but consistently wrong predictions (lower- tail monotonicity). We empirically validate this assumption using DeepSeek-RR RRI-Disiill-Ween-l on the training set. For each question, the model generates 16 responses and computes the mean re- ward across these responses. Recall that the length- normalized sequence-level
Log Probubility vs Reward on DAPOMATH-17K
Log Probubiitt
0...111
0.002
0240.
0.004
0.40.
05508
0.6.777
074s
8s.0.
09.10
Reward Range
Figure 3: Log-probabilities as a function of reward quantile levels for DeepSeek-RI- Betill-Qwen-Lee on DAPOMATH-17K.
proopbbbaylltyy is de-
fined as log (b)zz) = |y| -' log " x). As shown in Figure the sequence-wise logprob exhibits monotonicity in 0.3) and [0.7, which justifies Assumption 1 Denote the ad- vantages in the associated algorithms as AMVAR and A Mean. see Appendix BB.3 for details. Next, we present comparison of the correlations between the advantage values and the log- likelihoood
Theorem 2. If Assumption 7 holds and E|| log T [yylz)] < oc, then the covariance between MVaR- based advantages and output log- pe-probabillitiss is smaller than that of mean-based methods, i.e.,
Cov ( (log To (y)xr), Anavate & Cov (oVgr/tl()1)) (log No (3) (r), A AMean)-
(5)

---

Published as a conference paper at ICLR 2026
The combination of Proposition and Theorem 2 implies that, under the same update setting, RiskPO leads to higher output entropy compared with mean-based methods. Similarly, we can conclude that risk-seeking objectives assign greater weight high-reward outcomes, thereby am- plifying the covariance between log-probabilities and advantages. This stronger coupling causes entropy decrease relationship between correlation and objective design, beyond the specified MVaR objective, we can generalize more general transformation of the reward or so-called advantage value. This general form allows us to assess the strength of the correlation and, in turn, reason about the resulting policy entropy. Please refer to Appendix B.4 for details.
rapidly and results in severe entropy collapse. To further elucidate the
EXPERIMENTS
To comprehensively evaluate the effectiveness of RiskPO, this section conducts systematic exper- iments across a broad spectrum of tasks, including mathematical reasoning, code generation, and multi-modal reasoning. We benchmark on more than ten datasets that span different levels dif- ficulty. Through these experiments, our goals are threefold: (i) to verify that risk-based objectives consistently outperform mean-based methods across domains, (ii) demonstrate that the proposed distributional perspective leads to tangible improvements on the hardest reasoning problems, and (iii) to provide empirical evidence that RiskPO mitigates entropy collapse and enables genuine ex- pansion of reasoning capability rather than merely improving sampling efficiency. Detailed experi- mental settings and supplementary results are provided in Appendix A
6.1
MAIN RESULTS
Table reports @ass@1 I accuracy across six hard-level mathematical reasoning benchmarks. We observe that RiskPO consistently achieves the best performance among all methods, outperforming both the base models and recent GRPO variants. In particular, RiskPO attains an average score of 46.65, representing +2.78 absolute improvement over the strongest baseline DAPO (43.87) and +6.24 improvement over vanilla GRPO (40.41). The gains are especially pronounced on the most challenging AIME datasets, where RiskPO surpasses DAPO by nearly +6.7 points (33.3 vs. 26.6). These results demonstrate that emphasizing distributional risk through our MVaR objective substantially improves reasoning ability, not only enhancing performance on easier datasets like AMC and MATH500 but also pushing the frontier on the hardest Olympiad-style tasks.
Table I: Pass@ performance on hard-level mathematical reasoning benchmarks.
Model
AIME25
AIME24
AMC
MATH500
Minerva
Oly.
Avg.
Owen2.5-Math-11.50
6.6
10.0
43.4
61.8
15.1
28.4
27.55
Awen2.5-Math-1.5B--mee
10.0
48.2
64.2
26.5
35.2
32.35
DeepSeek- -Distiil Swen---ennnS
1.5B
13.3
13.3
32.5
59.8
20.3
30.5
28.28
Dr.GRPO-1.58 [Liu et al.
2025b
20.0
23.3
54.7
77.4
26.3
38.1
39.97
GRPO-1.5B (Shan ct al. 2024)
20.0
20.0
56.6
79.2
27.1
39.6
40.41
GPG-1.5B (Chu et al. 2025
16.6
20.0
55.7
74.5
28.8
37.6
38.90
DAPO-1.5B ( Yu
2025
30.0
26.6
58.6
78.2
29.2
40.6
43.87
GMPO-1.5B (Zhao el al. 2025b
23.3
54.2
29.2
39,2
40.90
RiskPO-1.5B (Ours)
33.3
33.3
60.8
81.8
29.5
41.2
46.65
We complement these findings in hard-level math tasks with results easier mathematical rea- soning, multi-modal reasoning, and code generation tasks in Table [ While performance gaps on GSM8K are naturally small due to the dataset's 's simplicity, RiskPO maintains measurable ad- vantage on MATH and yields consistent improvements on both LiveCodeBench and Geometry3K. Together, these results underscore the broad applicability of risk-sensitive objectives and their ability to enhance reasoning capacity across diverse domains.
We argue that the RiskPO is expanding the reasoning boundary of the base model. To support the claim, we present the evolution of Pass@1 1, Pass@8, and Pass 16 on AMC and MATH500 during the training process in Figure 4 A clear widening gap emerges as & increases, with RiskPO steadily surpassing GRPO across all evaluation points. This pattern indicates that the model is not merely improving its sampling efficiency problems it could already solve, such turning "one success

---

Published as a conference paper at ICLR 2026
Table 2: @ass@1 results on easy-level math benchmarks and muuli-mooallccodiing benchmarks.
Easy-level math. reasoning
Multi-modal coding
Method
MATH
GSM8K
Avg.
LCB
Geo3K
Avg.
GRPO
54.3
78.8
66.55
25.8
53.7
39.75
DAPO
55.2
80.3
67.75
26.2
54.3
40.25
RiskPO (Ours)
56.2
80.3
68.25
26.8
54.5
40.65
0.0
Aocurey
Acarney
Accurcy
0.6
0.7
0.7
0.0
0.4
0.5
0.0
2000
300
200
300
500
100
2001
-000
500
Step
Stop
Step
Pass @ on AMC.
(b) P on AMC.
(c) Pass @ on AMC.
1.00
0.96
Acctracy
0.80
0.75
Acntary
0.90
Accuracy
GRPO
0.55
Ri
0.00
Step
Step
Step
(d) Pass@ on MATH500.
(e) Pass@ on MATH500.
(I) Pass 16 on MATH500.
Figure @a learning curves on AMC and MATH500 datasets.
in sixteen attempts" case into reliable single-shot success, but is fact acquiring genuinely new solution strategies. Notably, RiskPO is able to solve instances that remain persistently unsolved under GRPO, even after sixteen attempts, all within the same computational budget. These results substantiate our claim that RiskPO expands the reasoning boundary of the base model, enabling progress beyond the capabilities achievable by conventional mean-based objectives.
Entoops
RVVR
MYAR
Rewand
-nu-
IDsPPO
-05
Step
Starp
Stege
Stego
(a) Entropy.
(b) Lower-tail RVaR.
(c) MVaR reward.
(d) Mean reward.
Figure 5: Learning curves on DAPOMATH-I7K. the RiskPO mitigates the entropy collapse and shows better performance on difficult problems, which is indicated by risk measures.
We investigate how the training dynamics of the RiskPO differ from GRPO, and Figure 5 de- picts the trajectories of different objectives and entropy during training on the DAPOMATH-177 dataset. We contend that the mean reward an inadequate training objective. The mean reward learning curves of GRPO and RiskPO are almost indistinguishable, with RiskPO exhibiting slightly greater fluctuations, likely due to its inherently higher-entropy behavior. In contrast, the lower-tail RVaR JRVaRo... (0) and MVaR curves demonstrate a pronounced advantage for RiskPO. Since these risk-sensitive measures emphasize the lower tail of the reward distribution, higher values indicate stronger performance
the more challenging problems. Consistently, RiskPO maintains substan- tially higher entropy throughout training, whereas GRPO's entropy collapses early on, curtailing exploration and limiting its ability tackle difficult instances.
8

---

Published as a conference paper at ICLR 2026
Ealeope
Reoard
Accarsey
Accuraey
0.1
12
Pask Seekeng
Rick
130
120
150
USE
Step
Step
Step
Stop
(a) Entropy.
(b) Mcan reward.
(c) Pass@1 on MATH.
(d) Pass on GSM8K.
Figure 6: Ablation of risk profiles for training.
Table Ablation of different quantile levels (a, 3) on easy-level mathematical reasoning.
Levels
(0.2, 0.8)
(0.1, 0.8)
(0.3, 0.8)
(0.4, 0.8)
(0.2, 0.6)
(0.2, 0.7)
(0.2, 0.9)
MATH
56.2
55.1
55.8
55.6
55.5
56.0
55.0
GSM8K
80.3
78.7
79.2
79.6
79.0
79.5
78.9
Avg.
68.25
66.90
67.50
67.60
67.25
67.75
66.95
6.2 ABLATION STUDY
We conduct the ablation study on the easy-level mathematics reasoning tasks. We start our analysis with the contrasting version:risk-seeking the reward distribution will catalyse the entropy collapse. Similar to the MVaR objective, we use the counterpart risk-seeking objective, (1 + w) 3) JRVaR (0) + (3 a JRVaR...: (0), to train the model and keep other parameters the same as the risk-averse version. The training curves are shown in Figure 6 The entropy of the risk-seeking version decreases sharply the training proceeds, whereas the entropy of the risk-an avverse version decreases at the beginning, then remains stable around 0.2. In the Pass @ I curve on MATH, the risk-averse version exhibits clear advantage over the risk- seeking. Before 50 steps, the risk-seeking has a better Pass value. However, after the 50th step, the risk-sceking struggles keep improving because fails optimize on those difficult problems (from 52% to 54%). The risk-averse version continues to improve on the Pass @ (from 52% to 56%), showing 1.5 times improvement. This observation further justifies our theoretical results, which suggest the risk-averse objective is better than both the mean and risk-seeking objectives.
As indicated by Section 3 focusing on the upper tail of
We further conduct a systematic investigation of the parameterization of the risk-averse algorithm, focusing on the quantile level (a, 33). In the main experiments, we adopt (0.2, 0.8) as the default configuration and independently perturb a and [3 to validate this choice. The results of the quantile- level ablation are summarized in Table 3 Deviations from the configuration (0.2, 0.8) consistently lead to deterioration in performance. In particular, the configurations (0.1,0.8) and (0.2, 0.9) exhibit more degradation. Reducing to 0.1 implies diminished emphasis on the lower tail, whereas increasing to 0.9 intensifies the emphasis on the upper tail. These observations underscore the importance of maintaining risk-averse objective.
Table 4: Ablation of different bundle size B on easy- mathematical reasoning.
Bundle size
2
3
5
7
8
9
10
MATH
53.6
55.4
55.8
56.0
56.2
55.8
55.8
55.2
54.9
54.7
GSM8K
78.0
79.0
79.5
80.1
80.3
79.9
79.4
78.8
78.6
78.6
Avg.
65.80
67.20
67.65
68.05
68.25
67.85
67.6
67.00
66.75
66.65
We also report the ablation results on the bundle size in Table 4 The best performance is achieved at bundle size of B = 5. This result can be intuitively explained by the trade-off in gradient esti- mation. Since all instances in bundle share the same advantage estimation, using an overly large bundle dilutes the gradient signal, as too many instances rely on single advantage value. Con- versely, when the bundle size is too small, quantile tracking becomes unstable, which also weakens the gradient signal. The results in Table 4 corroborate this intuition: at B 2 and B 10. the average performance drops by 1.05% and I.6%, respectively. Setting B (no bundling) leads to the most severe degradation, with 2. 2.45% drop. These findings highlight the necessity of bundling for RiskPO and suggest that bundle size should carefully tuned, as both overly large and overly small values harm performance.

---

Published as a conference paper at ICLR 2026
7 CONCLUSIONS
In this paper, we introduced RiskPO, a distributional alternative to mean-based objectives for rein- forcement learning with verifiable reward. By leveraging MVaR and bundling multiple questions into informative training units, RiskPO effectively mitigates entropy collapse and strengthens ex- ploration. Our theoretical analysis establishes that risk-averse updates weaken the coupling between policy log- probbabbbilissss and advantages, thereby preventing premature overconfidence, Empirically, RiskPO achieves consistent and significant improvements across a wide range of mathematical rea- soning, code generation, and multi-modal benchmarks, outperforming GRPO and its strongest vari- ants. These findings highlight that risk- averse objectives not only improve sample efficiency but also expand the reasoning frontier of LLMs.
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grants 72325007, and 72250065, and the Science and Technology Innovation Program of Hunan Province under Grant 2024RC7003.
REFERENCES
Yasin Abbasi-Yadiie Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference Machine Learning, pp. 3692-3702. PMLR, 2019.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1300 2025.
Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3): 334-334 1997.
Vivek Borkar and Rahul Jain. Risk-constrained markov decision processes. IEEE Transactions on Automatic Control, 599))2574-2579, 2014.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass @ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508. 10751, 2025.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint 231.3.2401.01335, 2024.
Daixuan Cheng, Shaohan Huang, Xuckai Zhu, Bo Dai, Wayne Xin Zhao Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective on reinforcement learning for 1lms, 2025. https:s////axx org/ abs/2500 14758
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. Advances neural information processing systems, 28, 2015.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained rein- forcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167): 1-51, 2018
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- fems, 30, 2017.
10

---

Published as a conference paper at ICLR 2026
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:250..02446, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprinl arXiv:21 10.1:2210.14166, 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv: 22617,, 2025.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference machine learning, pp. 1096- 1105. PMLR, 2018a.
Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018b.
Michael C Fu, L Jeff Hong, and Jian-Qiang Hu. Conditional monte carlo estimation of quantile sensitivities. Management Science, (5(1)):0019-200 2009.
Zitian Gao, Lynx Chen, Haoming Luo, Joey Zhou, and Bryan Dai. One-shot entropy minimization. arXiv preprini arXiv:2505.20282: 2025.
D. Guo, D. Yang, H. Zhang, and et al. Deepseek-rl incentivizes reasoning in Ilms through re- inforcement learning. Nature, 645:633-638, September 2025. doi: https://doi.org/ 1031/ -4158--22--99422- URLhttpss:/ /ooi wwg/Ic. 1ew..10011118880 025-00255001 - z Re-
ceived: 14 February 2025; Accepted: 17 July 2025; Published: September 2025; Issue Date: 18 September 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. T1: Advancing language model reasoning through reinforcement learning and inference scaling, 2025. httpppss//aarii..orr///////////0
Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang. Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning Ilm from zero data, arXiv preprint arXiv:2508.0550 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Le7ama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
Jinyang Jiang, Yijie Peng, and Jiaqiao Hu, Quantile-based policy optimization for reinforcement learning. 2022 Winter Simulation Conference (WSC), pp. 2712-2723. IEEE, 2022.
Jinyang Jiang, Bernd Heidergott, Jiaqiao Hu, and Yijie Peng. Distortion risk measure- deep reinforcement learning. In 2024 Winter Simulation Conference (WSC). IEEE, 2024.
Prashanth La and Mohammad Ghavamzadch. Actor-critic algorithms for risk- mdps. Ad- vances in neural information processing systems, 26, 2013.
Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. Illustrating reinforcement learning from human feedback (rlhf). Hugging Face Blog, 2022. http://uuggingffcc..ooo//o/g////
11

---

Published as a conference paper at ICLR 2026
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Soke -Sonan-Sooo,,a et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In International Conference on Machine Learning (ICML.). 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong- Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint urXiv:2505. 24864, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 2025b.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.041100 2021.
MAA. American mathematics contest 12 (amc 12), november 2023, 11 2023. URL
ht https: http:///rttofpooommooooo
com/ wiwkwwikki// /wnd/xinxxxx.//m php/ AMC Peoc_11_Probemmss
and Solutions
MAA. American invitational mathematics examination (aime), february 2024, 2024. URL Problems_
and Solutions
MAA. American invitational mathematics examination (aime), february 2025, 2025. URL
https: /u.ps://arte http://rrtoorrooeoooooioooo
com/ htmm/wiii/inneen wwt.aadex..ppp.pppmpp AAIEE Probl ems
and Solutions
Brenden K Petersen, Mikel Landajuela, T Nathan Mundhenk, Claudio P Santiago, Soo K Kim, and Joanne Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk risk-seeking policy gradients. arXiv preprint arXiv: (91.:::99920487/// 2019.
LA Prashanth, Cheng Jic, Michael Fu, Steve Mareus, and Csaba Szepesvári. Cumulative prospect theory meets reinforcement learning: Prediction and control. In International Conference on Machine Learning, pp. 1406-1415 PMLR, 2016.
Tao Ren, Ruihan Zhou, Jinyang Jiang, Jiafeng Liang, Qinghao Wang, and Yijie Peng. Riskminer: Discovering formulaic alphas via risk seeking Monte Carlo tree search. In Proceedings of the 5th ACM International Conference on Al in Finance, pp. 752-760, 2024.
John Schulman. Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Lior Shani, Yonathan Efroni, and Shie Mannor. Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp- 5668-5675, 2020.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint 20Xiv:2402.03300, 2024.
12

---

Published as a conference paper at ICLR 2026
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the via sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, and Guorui Zhou. Stabilizing knowledge, pro- moting reasoning: Dual-token constraints for rlvr. arXiv preprint arXiv:2507. 15778, 2025a.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning, 2025b. htthhhtss//axii.....
2.506.01933
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504. 1134,, 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, and al. Qwen2.5-math technical report: Toward mathe- matical expert model via self-improvement. arXiv preprint arXiv:2409. 12/22, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does inforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504.10 13837, 2025a.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118. 2025b.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthicu Lin. Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with data. arXiv preprint arXiv:2505.08:: 03335, 2025a.
Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, Fang Wan, and Furu Wei. Geometric-mean policy optimization, 2025b. htlt:htts:://aaaa
org/ httr/bbs/25000 2207. 20673
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arX1v:2507.1807,, 2025.
13

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL SETUP AND SUPPLEMENTARY RESULTS
A.1 EXPERIMENTAL SETUP
Model. We focus on mathematics reasoning, code generation, and multi-modal reasoning. We use DeepSeek-RII R--Dissiil----e----e
(Guo et al. 2025) as our base model to evaluate different algorithms on hard-level mathematics reasoning and code generation. On easy-level mathematics reasoning, we use Qwen2.5-Math- 1.5--ath--,55-Insrre I.S--uuttt (Yang et al. modal reasoning,
2024) as the base model. On multi- Owwn2.5-VL-BB........ VL-Inssruct-3B (Bai et al. 2025) the base model.
Training. For the hard-level mathematics reasoning tasks. We use the DAPO-math- 17k as the training set. For the easy-level, we use MATH (Hendrycks et al. 2021) and GSM8K (Cobbe et al.) 202T]. For multi-modal reasoning, we use Geometry3K (Geo3K, Lu al. 202T). For code genera- tion, we train the models on Archer-6K (Wang et al., 2025a). We set the clipping threshold KL penalty and entropy regularization are omitted from the loss objective. We use vLLM as the inference backend and FSDP as the training backend. We set the temperature to 0.8 and top-p to 1.0, and maximum output length as 3072. We generate 10 responses for each problem. The batch size is 512, the mini batch size is set to 128. For quantile levels, we set to 0.2 and [ to 0.8 corre- spondingly. The bundle size B is set to 5. The mixing parameter of MVaR is w = 0.5. All training procedures carried out a Linux server equipped with 8 NVIDIA H20 GPUs, each providing 96 GB of memory.
0.2.
Evaluation. For hard-level mathematics reasoning, We evaluate on six math reasoning datasets: AIME24 (MAA 2024) and AIME25 (MAA 2025) with 30 problems from the American Invita- tional Mathematics Examination, both targeting advanced pre-collegiate reasoning; AMC23 (MAA) 2023) with 83 problems from the American Mathematics Competitions, testing creative algebraic, geometric, and number-theoretic skills; MATH-500 (Lightman et al., (023) with 500 graduate-level problems from the original MATH dataset covering algebra, geometry, and number theory: Minerva Math (Lewkowycz et al. (002)) with 272 undergraduate-level quantitative reasoning problems; and OlympiadBench (He et al., (0024) with 675 Olympiad-styy problems. For easy-level math tasks and the multi-modal task, we follow the train-test split in the original datasets. For code generations, evaluate trained models on LiveCodeBench v5 (LCB, Jain et al. 2024).
A.2 ABLATION ON THE MIXING PARAMETER
We fix the quantile level, and perturb the w investigate how different attention on the lower tail influences the performance. The ablation of w is shown in Table Setting w = 0.0 would reduce the MVaR objective, JMVaR= (6), to JRvaRo.; (0). which does not have extra attention on the lower tail even though it is still a risk-averse objective. The variant with w 0.0 has the largest performance decrease, indicating the significance of extra attention on the lower tail. When setting w 1.0, the variant also suffers from a mild performance drop. Overall, the phenomena suggest that the level of risk aversion needs to be properly tuned; both indifferent level and excessive focus would lead to undesirable performance.
Table 5: Ablation of the mixing parameter.
Settings
0.0
0.1
0.5
0.6
1.0
MATH
54.8
55.1
56.2
56.0
55.7
GSM8K
79.6
80.0
80.3
80.2
80.0
Avg.
67.20
67.55
68.25
68.10
67.85
A.3 EXTENSIVE AvG AND PASS@K RESULTS
Since both AIME2024 and AIME2025 contain only 30 questions, the @ass@k metric exhibits high variance and fluctuates significantly. To obtain a more stable evaluation, we report the Avg@k re- sults in Figure Across both datasets and different & values, RiskPO consistently outperforms
14

---

Published as a conference paper at ICLR 2026
GRPO, achieving higher Avg@k scores and demonstrating ing. The advantage of RiskPO is especially pronounced in the later training stages, where it con- tinues to increase while GRPO tends to plateau. These results further confirm the effectiveness of our risk-sensitive optimization in enhancing reasoning performance on small-scale but challenging benchmarks like AIME
stable improvements during train-
Figure reports @ass@k for E {1,8, 16). Across both datasets and all k, RiskPO consistently outperforms GRPO throughout training. The margin is modest but stable at k (especially on MINERVA, where variance is higher), and becomes clearly larger for k 8, 16. This widening gap at larger indicates that RiskPO not only improves the best single prediction, but also spreads probability mass over a broader set of valid solution paths, thereby increasing the likelihood that at least one sampled response correct. In effect, the risk-sensitive objective enhances coverage and diversity of reasoning, pushing the success frontier on problems that initially have low correctness probability and yielding larger gains higher
0.20
0.20
Accuracy
Accuracy
0.35
0.15
0.30
20001
300
200
1.00
1000
3000
Step
Step
(a) @v 32 on AIME2024.
(b) Avg@64 on AIME2024.
0.20
Accarncy
Accuracy
0.15
0.15
0.30
130
200
200
400
200
100
200
300
400
500
Step
Step
(c) Av on AIME2025.
(d) Avg@64 AIME2025.
Figure 7: Avg@k @k learning curves on AIME2024 and AIME2025 datasets.
0.50
Accuracy
0.35
Arcurncy
0::00
Accuracy
0.20
0.50
300
400
300
400
500
100
500
Step
Step
Step
(a) Pass Minerva.
(b) @ on Minerva.
(c) Pass on Minerva.
0.55
0.50
Acemaey
0.35
Acemary
0.000
0.30
0.45
0.50
0:25
0.00
0.15
GRPO
Pi
Step
Step
Step
(d) Pass@ 1 on Olympiad.
(c) Pass @8 on Olympiad.
(f) Pass @ 16 on Olympiad.
Figure 8: @ass@k learning curves on Minerva and Olympiad datasets.
15

---

Published as a conference paper at ICLR 2026
A.4 DYNAMICS OF COVARIANCE AND ENTROPY
BIAPO GEPO
-VEunr
3000
300
Dettrog
X055
385
300
315-
1000
3007
3000
Step
Stop
Stop
(a) Dynamics of GRPO.
(b) Dynamics of RiskPO.
(c) Comparison of Covariances.
Figure 9: The dynamics of the covariance and the entropy difference. We calculate the one step
entropy difference and the covariance between log-prob and advantage.
In the Figure 9 we verify the validity of the Proposition and Theorem We record the covariance between log- prg--rbbbbilityy and the advantage, and calculate the entropy difference during the training on easy-level math task. Figure 9a and 9b] shows that the covariance and the entropy difference move in synchronicity during both the GRPO and RiskPO training, which validate our Proposition show the comparison of covariance between RiskPO and GRPO in FigurePec The covariance of RiskPO is consistently smaller then the GRPO throughout the training, which coincides with the conclusion in the Theorem
I
To validate the Theorem
A.5 JUSTIFICATION OF ASSUMPTION 1
Log Proladality vs Reward on AMC
Log Prolabality- vs Bewoar on MATH
1000
Lng. Protebility
Log Probabiley
16.0
-17.8
-25
-2
0.422
0.40
0.4.
04485
a.4.
0.248
u.641
6910
a0g1
0.482
0.3d
0.445
0x60.7
0.54
9.849
0.1.0
Bemand Bange
Arwond Range
(a) AMC.
(b) MATH.
Log Probalality va Reward on Olympiad
Log Prolaciliiy ws Reward on Minerva
Los, Probstakit
Log Proboleity
-16
-20
-20
0b81
0.4t2
0280
dade
844es
0648
ezis
8843
8%
81482
B28d
B.45
01-00
de4er
87-45
8340
Arwand Ramge
Rewnnd Range
(c) Olympiad.
(d) Minerva.
Figure 10: The output log probability on various evaluation datasets.
Figure 10 presents the output log probability stratified by reward ranges across evaluation datasets. On Minerva and Olympiad datasets, the patterns closely align with Assumption T the output log probability is monotone with respect to reward in both the lower- and upper-tail regions, approxi- mately on (0, 0.3) and (0.7, 1). Results on AMC show a similar monotone trend, although fluctua- tions appear in the mid-reward ranges, suggesting mixed difficulty and solution modes. For MATH
16

---

Published as a conference paper at ICLR 2026
which is comparatively easier, the upper tail exhibits strong monotonicity, while the lower tail is less pronounced liikoll due to a scarcity of truly difficult items that would populate that region. Im- portantly, these evaluation sets not used for model training; the observed regularities therefore provide additional evidence that Assumption holds broadly for the pretrained base model across diverse benchmarks.
17

---

Published as a conference paper at ICLR 2026
B THEORETICAL DETAILS
B. PROOF OF THEOREM
[
Proof. Recall the definition of the RVaR functional:
1 Fo (9)
JR (0) = E R E (o), 3 (a)
rfa (r)(rd
To compute the RVaR gradient, apply Leibniz's rule for yiceen
V VeJRvaR... (0) 3-a (1 rVofo(r)dr
Po (0)
( (=) (2)
FF'((a)
Note that, by the implicit function theorem, the quantile gradient can be expressed as (see, e.g., Fu et al. 2009) Vo (z) = - -ooFo Fā (z))laa fo (Fo (2)). Substituting this identity into our previous expression, we can obtain
( rVofa(r)dr Vz)o (2))
(A)
0=0 !
VeJRVaR (0) = 3 3-a F.)(
By the definition of CDF, have VoFo(r) VEE ()))))) E [[1 (R()S))) Vo In fo (R(()))) Thus, with the score-function method, we rewrite the RVaR gradient expectation form as
VoJRvaR (0) E
: (R(v)1 (RW))
FF
Ve Fo B a
Finally, since the distribution of R(y) is induced by the LLM ne (+I-), we can apply the score-function transformation to yield the final expression
V VoJaVaR (0)
3
which completes the proof.
B.2 PROOF OF PROPOSITION
Proof. With Lipschitz-continuous entropy gradient and a bounded Hessian, the first-order Taylor expansion yields H(To,,+1 (z:) = H(mo, (x) VoHH(TT, (x), +O(|AxI1))))) The entropy gradient can be written
VoH((To x) = Ve Ey (+ log To (3)(2)))
= Eyyno Ve log To (y( + log To (ylx) (x) Ve log [u((=)
-E Eyyno llog no ( log (y]c=))
where the second equality comes from the score-function method, and the last equality is due to the identity Ey)rs(g)=)) V log [y(()=) = O. Note that 36_.
log Te ())(z) = (v=v') To (e(3((z)
Taking the inner product with A gives (V (()l(()))))) A) (Byyno (-120) [log (o log ( (y/=) A}
(x log [)) V log To (v)xx), A)]
ò (v)
Eg
log
20..y'
2.9
V'Ey
- --Eyoro(1o)
log (v(l)
Σ (1(w=') To( (o(B((x)))4 (x))
V'EY
(1x))
log (T())))) Az.))
Ex log T0())2)
Σ
To ( D Az.y'
wey
= - Cov (log fog An.y)
(6)
In tabular softmax policy, a natural policy gradient update step admits Az,y = 17 Ag (x, y) (Agarwal
al. 202T). Plugging this into (6) yields the equality (4), which completes the proof.
18

---

Published as a conference paper at ICLR 2026
B.3 PROOF OF THEOREM
2
Before proving Theorem 2 we first prepare the following lemma, which provides convenient representation of covariance.
Lemma 1. Let X,Y Y be real-valued random variables satisfying B||XY|] <o0. co. Then
L Cov (1(x>1): Y) dt.
Cov(X,
Proof. We start from the layer-cake representation X = (1 (1xx>-- (-xxx)) dl, which holds for any real-valued X. Multiplying by Y and taking expectations, may apply the Tonelli-Fubini theorem under the integrability condition E[|XY|] < co, which yields
E = /. E [ - [[ 1 (-xx0)) dt.
Applying the same transformation Y and subtracting, we obtain
Cov(X, 1) f. (Cor
(1(x21), (Cor(((
- Cov (1(-x>1), X)) dt.
Changing the integral variable in the second term and using the identity Cov(1 - Z, Cov(Z, Y). merge the integrals and obtain the equality 7 The distinction between strict and non-strict inequalities only affects a countable set of values and does not change the integral under the Lebesgue measure, which completes the proof.
Next, present the proof of Theorem 2 For notational clarity, focus single repre- sentative output from the model (T(- (r) rather than a bundle. The advantage values for the MVaR- and mean-based objectives are given by AMvaRe = -(1 (L' (a) R(y))+ + +
( F7 (B)) and AMean R(y) [[[(()]]
Proof of Theorem 2 Recall that the positive part function can be expressed via the Mayer-cake rep-
resentation: (z aa 1+ 1/200 dt. With Lemma
we can derive the covariances below
(a)
(1)
COv(AMVaR: SF) (1 (1+w] w)
Cov (I(()))>)) SF) dt,
(8)
(F)
where, for notational convenience, we denote SF = log log=o(y(zx) Define k(t) :=
Cov(1 (Ri)s))))) and recall that the density of R(y) is fe- Then, compute the derivative of l:(t) as follows:
((() d(E[1 (R(g) SF)
((P((R(y) > t)E(SF])
dt E (F - E d
dt
d
dt J. fe(r)dr
dt
d
R( [[()]] - [E(FF] dt J. fo(r)dr
E
di
dt J. 1 R(5) = rlfo(r)dr E(SF d 1. fo(r)dr dt
d.
-(0))(())((EE(S)))) (-E(SF),f(()
(o()) E[SF)).
Under Assumption (b()) 2 E(SF) for / 2 Fo (3) and / 1 (a). Consequently, k(()) S 0 for t t> Fo (33) and t < (a). which implies that k(t) is non-increasing in both the upper and lower tails. Moreover, since E[[SF]] < oc, the dominated convergence theorem implies that
lim k(t) lim (ov((1R())22)) SF)
lim [[ SF] lim Pr(R(y) > [)E[SF]
0.
19

---

Published as a conference paper at ICLR 2026
Analogously, show that lime- k(t) E[SF] - E[SF] 0. By the monotonicity in the tails, obtain k(t) 0 for & 2 (8) and k(t) 0 for t S Fg (a). Therefore. noting that k(t) = Cov (I(m6)>e)) preserves its sign on both tails, we can further obtain
/
Cov(AMvany, SF) -
Cov ( dt Cov SF), R
which completes the proof.
B.4 SUPPLEMENTARY THEOREM OF SECTION 5
With the different treatment of the tail in the reward distribution, we obtain the following covariance result between the resulting advantage value and the output log-probability.
Theorem 3. If Assumption 7 holds with [[[SF]] < oo, and 91, 92 are nondecreasing and differen- tiable with gi (t) 2 (2) on Fø 00), (t) = 92(() on (-00, 1 then we have Cov(SF, 92 (v))) 2 (oo(SF,g:(R()))
Proof.
Note that
Cov(SF, (o.:((())))))dd f CSv ((R())a))dgg:()
Cov(SF,
Cov(SF, (()())))
where the first equality is due to Lemma D and the second equality is integration by substitution.
The proof of Theorem implies that under Assumption
we have k(t) 2 0 for 2 F (8), thus
00
Cov(SF, (1(SF.g2(R()))) Cov(SF (R =
((t) (t)) 2 0,
(T)
which gives the desired result.
A symmetric conclusion can be derived analogously for the treatment of the other tail.
20


Reviews:
[{"review_id": "aV7pos4R5o", "reviewer": "Reviewer_TQ2d", "summary": "The paper proposes a risk-sensitive alternative to mean-based RLVR (Reinforcement Learning with Verifiable Reward) objectives. It introduces a Mixed Value-at-Risk (MVaR) loss that emphasizes the lower tail of the reward distribution and a “bundling” trick that aggregates multiple questions to avoid vanishing advantages when all samples for a question are wrong. The theory links advantage–logprob covariance to entropy changes and argues MVaR mitigates entropy collapse. Empirically, the paper reports Pass@k gains over GRPO and recent variants on several math benchmarks, plus smaller gains on coding and multimodal tasks.", "questions": "Specific requests / experiments that would change my mind\n\nAdd Putnam-AXIOM (and other unseen-variation splits) with 95% CIs and p-values; show RiskPO beats strong GRPO/DAPO/GSPO baselines and a prompt-diversity baseline at matched compute.\n\nAblate attribution: (a) Mean + Bundling, (b) CVaR-only + Bundling, (c) MVaR without bundling.\n\nForgetfulness audit: pre/post retention on easy skills and previously-mastered bins; report calibration changes.\n\nStatistics: ≥5 seeds, paired bootstrap CIs for Pass@k and Avg@k, and a preregistered evaluation script.\n\nCompute parity: confirm identical budgets, temperatures, decoding, and training steps for all baselines.\n\nPrompt baseline: add a well-tuned diversity-prompting method that explicitly enumerates multiple reasoning paths with probability assignment, to test whether RL is necessary for the claimed effects.\n\nMinor comments\n\nClearly define Avg@k in the main text; it is referenced but not formalized.\n\nExplain the length-normalized sequence IS ratio exponent (the 1/|y| in Eq. (3)): why this choice? Any stability issues without it?\n\nAcknowledge AIME’s small-n variance in-paper (not just in the appendix) and report CIs in all main figures.\n\nTighten text and fix minor wording (“catalyse”, “papers” vs “bundles” in Algorithm 1 Step 6).\n\n\nNote: all writing was assisted by LLMs, but all the thoughts were my own.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Motivation is reasonable: mean objectives over-weight easy, frequent modes; a lower-tail-aware loss could push exploration of harder cases.\n\nSimple to implement: the method is a drop-in change atop GRPO-like training (quantile tracking + bundle-level advantage).\n\nSome positive results: the paper reports consistent improvements on hard math suites and provides ablations (quantiles, bundle size, risk-seeking vs risk-averse).", "weaknesses": "Novelty feels incremental.\nApplying standard risk measures (CVaR/RVaR/MVaR) to RLVR is a natural extension, and the “bundling” device is an engineering fix to sparse binary rewards. The paper would read more convincingly if it isolated how much of the gain comes from risk weighting versus bundling. A missing baseline is Mean + Bundling (i.e., GRPO-style mean objective but with bundles). A direct CVaR-only baseline adapted to RLVR is also needed to understand if “mixed” VaR really matters.\n\nBenchmark choice is dated and misses stronger generalization tests.\nMATH500 and GSM8K are saturated; GSM8K-Platinum exists, and Putnam-AXIOM (arXiv:2508.08292) directly targets extrapolative generalization with functionally novel problems. If the claim is “expands the reasoning frontier,” I would expect evaluation on Putnam-AXIOM and related unseen-variation splits. As written, improvements on older suites are less persuasive.\n\nCatastrophic forgetting is not ruled out.\nRL-trained reasoning models are known to regress on distributional pockets they previously handled well. The paper attributes entropy dynamics to the objective, but the same curves could reflect forgetting. Please include a retention analysis: pre/post accuracy on held-out skills or difficulty bins (and on easier sets that should not degrade), plus calibration/entropy before/after. Without this, the claimed mechanism is under-identified.\n\nPrompt-level diversity controls are a missing baseline.\nThere is emerging evidence that you can preserve output diversity/entropy via prompted diversity instructions (e.g., “enumerate diverse solution paths; assign probabilities”) and light sampling changes—no RL required. Please include a strong prompt-only diversity baseline, matched on compute, to test whether RiskPO’s benefits exceed careful prompting.\n\nFigure 1 is not convincing; statistics are thin elsewhere.\nThe AIME learning curves (Pass@32/Avg@32) are noisy and, as plotted, do not clearly outperform GRPO in a way that rules out randomness; the panel lacks confidence intervals. AIME’s n=30 also makes variance high. Later figures (e.g., Figure 4) are more favorable but still fairly close on some panels. Please add multi-seed CIs, paired bootstrap on Pass@k, and exact tests (e.g., permutation) for headline numbers across all datasets, and report standardized effect sizes. For small suites (AIME), de-emphasize them in the main paper or aggregate multiple seeds/splits.\n\nTheory–practice gap.\nThe entropy analysis uses a tabular softmax and a natural-gradient step; actual training is transformer + Adam with sequence-level clipping and IS. While this is common in theory, please add practical diagnostics (token-level entropy, solution-path diversity, coverage metrics) to demonstrate that the claimed mechanism holds in the real training dynamics observed here.\n\nVerifier details and noise sensitivity.\nBecause RLVR relies on automatic verifiers, please specify the exact checkers per dataset (math normalization rules, code execution policy, multimodal grading) and include a robustness check to verifier noise. Tail-emphasizing objectives can amplify reward mislabels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "affYwltBdY", "reviewer": "Reviewer_GymV", "summary": "This paper proposes a new RL algorithm with verifiable rewards. The authors argue that GRPO mainly optimize mean rewards, leading to model overfitting to easy prompts and entropy collapse. The authors propose to replace mean-based objectives with risk-sensitive measures that emphasizing the lower-tail of the reward distribution. To obtain distributional information on the reward, authors propose to group multiple prompts and compute a shared reward distribution. With extensive experiments, the authors demonstrate that the proposed method improves performance across reasoning tasks compared to GRPO.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Novel design: This work proposes a novel RL objective that leverages distributional information of reward to effectively enhance model's capability in solving difficult tasks. \n* The work is very well written, presenting complex ideas with clarity that makes both the theoretical analysis and experimental results easy to follow.", "weaknesses": "* The proposed framework adds complexity to RL training with additional hyper-parameters to tune.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wzOg2Pp2UT", "reviewer": "Reviewer_Hp4w", "summary": "This paper introduces RiskPO, a reinforcement learning algorithm method for LLM post-training. RiskPO replaces the original objective in GRPO with a risk-sensitive objective called Mixed Value-at-Risk (MVaR). The key motivation is that methods like GRPO suffer from entropy collapse and poor exploration. RiskPO uses bundled questions and tail-focused optimization to amplify gradient signals from rare but informative low-reward cases, thereby promoting exploration and better reasoning. Experiments show that RiskPO outperforms other baselines on several benchmarks and mitigates the overconfidence issue.", "questions": "- Can you further explain why the bundle grouping makes sense? \n- What does “papers” refer to in lines 251 and 252?\n- I think the connection between Sec. 5 and Secs. 3,4 is somewhat weak. Could you further explain why strengthening the gradients on hard questions can address the issue of overconfident convergence?\n- Please see Weaknesses for the other questions mentioned.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper investigates a question of significant concern to the community: the entropy collapse of GRPO during training and the sparse reward issue on difficult problems.\n- It’s very interesting to incorporate a risk-sensitive objective based on the Mixed Value-at-Risk into the advantage estimation. The detailed theoretical explanation makes it clearer.\n- RiskPO shows promising performance gain compared to other baselines. The ablation study is thorough.", "weaknesses": "- The motivation is not convincing enough. The authors argue that GRPO suffers from entropy collapse, which is indeed a consensus in the community. However, they attribute this to overemphasizing high-probability output without any experimental or theoretical verification. Due to the design of the GRPO advantage, the model's gradient is 0 on both all-correct and all-incorrect problems, only updating samples with performance differences within the same group. I'd like to know how the authors correlate these non-zero advantage samples with high probability; is there any experimental verification or theoretical derivation?\n- I think considering GRPO as a mean-based method is debatable, since there is a significant difference between directly using reward signals for gradient optimization and using advantages. Therefore, I do not consider GRPO to be truly maximizing the expected average reward. In my view, RiskPO essentially extends the original group definition in GRPO, using different baseline definitions to mitigate the sparse reward problem in difficult tasks. In this context, RiskPO and GRPO should be classified similarly in terms of their objectives.\n- Although the proposed bundling strategy is intended to approximate a continuous reward distribution, the underlying feedback remains binary in nature. This discreteness introduces a potential mismatch between the analytical assumptions in Section 4 (which rely on smooth reward distributions) and the practical training setup. Moreover, the quality of the MVaR-based advantage estimation may depend sensitively on the bundle and generation sizes (B and G). Given that both are relatively small in the experiments, it remains unclear whether the estimated quantiles accurately reflect the model’s true reward distribution at each training step. Additional evidence or analysis would strengthen the authors’ claim that MVaR optimization remains valid under such discrete and limited sampling conditions.\n- The experimental results should be expanded, e.g., on larger models. Additionally, the settings of the current experiments should be described more clearly. For Tables 1 and 2, how many samples were used to compute Pass@1? Some baselines perform worse than vanilla GRPO, and I wonder why. Disclosing this information would make the results more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bjfGAOnSvu", "reviewer": "Reviewer_yqFg", "summary": "The paper introduces Risk-based Policy Optimization (RiskPO), a post-training reinforcement learning (RL) framework for large language models (LLMs) that replaces the standard mean-based GRPO objective with a risk-sensitive objective called Mixed Value-at-Risk (MVaR).\nThe motivation is that GRPO’s mean-based objective focuses on high-probability “easy” reasoning trajectories, causing entropy collapse and limited exploration. RiskPO instead amplifies gradient signals from low-reward or rare reasoning paths, encouraging exploration of harder problems.\nThe authors bundle multiple questions to enrich the binary reward signal, analyze entropy theoretically, and show that the risk-averse formulation mitigates overconfidence and improves reasoning diversity.\n\nExtensive experiments on over ten benchmarks—including AIME24/25, Minerva, Olympiad, MATH500, and AMC—show that RiskPO outperforms GRPO, DAPO, and Dr.GRPO, achieving higher Pass@k and Avg@k metrics. The improvement is more pronounced at higher k, indicating broader coverage of valid reasoning trajectories.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clearly identifies the limitations of mean-based GRPO objectives and motivates a principled alternative.  \n- Proposes a novel risk-sensitive objective (MVaR) that enhances exploration and mitigates entropy collapse.  \n- Shows consistent performance improvements across reasoning benchmarks, especially on harder problems.", "weaknesses": "- Experiments are mostly limited to mathematical reasoning tasks, with unclear generalization to other domains.  \n- The empirical analysis of how risk sensitivity affects exploration or diversity is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)