Risk-Sensitive Reinforcement Learning for Alleviating Exploration Dilemmas in Large Language Models
Authors: Yuhua Jiang
Jiawei Huang
Yufeng Yuan
Xin Mao
YuYue
Qianchuan Zhao
Lin Yan
Venue: ICLR 2026 Poster (2026)
Keywords: RLVR
Large Language Model
Reinforcement Learning
Pass@k Optimization
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for enhancing Large Language Models (LLMs) on complex reasoning tasks. Yet current methods face an exploration dilemma: standard RL struggles to escape the local optima of pre-trained LLMs’ sharply peaked initial policies, boosting single-solution accuracy (pass@1) but suppressing solution diversity and multi-solution performance (pass@k). As a result, RLVR often distills existing capabilities rather than discovering new reasoning strategies.  We address this with a Risk-Sensitive Reinforcement Learning framework. By adopting a risk-seeking objective that interpolates between mean and maximum rewards, we derive a novel Risk-Sensitive GRPO (RS-GRPO) algorithm that emphasizes hard prompts to drive exploration. Across six mathematical reasoning benchmarks and five LLMs, RS-GRPO consistently improves pass@k performance while enhancing or maintaing pass@1.

Published as a conference paper at ICLR 2026
RISK-SENSITIVE RL FOR ALLEVIATING EXPLORATION DILEMMAS IN LARGE LANGUAGE MODELS
Yuhua Jiang' Jiawei Huang? Yufeng Yuan Yu Yue Qianchuan Zhao
Xin M
Lin Yan 2ETH Zurich
TTsinguua University
BytteDance Seed
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for enhancing Large Language Models (LLMs) complex reasoning tasks. However, existing methods suffer from an exploration dilemma: the sharply peaked initial policies of pre-trained LLMs confine standard RL algorithms to narrow of solutions, boosting single-solutio accuracy (pass @ 1) but suppressing solution diversity and multi-solution performance (pass@k).
a result, RLVR often distills existing capabilities rather than discovering new reasoning strategies. To overcome this, we introduce a Risk-Sensilive Reinforcement Learning framework. Our approach employs a risk-seeking objective that interpolates between mean and maximum rewards, leading to a novel algorithm, GRPO), which drives deeper exploration by amplifying learning from challenging prompts. Remarkably, RS-GRPO is simple to implement, requiring only minor code modifications. On six mathematical reasoning benchmarks and with five different LLMs, RS-GRPO consistently improves pass @k performance while maintaining enhancing pass@) I accuracy.
GRPO (RS-
G.T N
A logma (x), (m) r(a)
1
Σ r(s))
N
Reward
Policy
Standard RL
Mial
Red (36)
All (ui log mo 1)) Ag (vd)
* 2 Miuu
Risk-Sensitive RL
LA Lu
Trasssng
Figure 1: Starting from a sharply peaked policy whose probability mass is concentrated on suboptimal rewards, standard RL further collapses the policy and fails to converge to the optimum, whereas risk- sensitive RL promotes exploration and converges to the optimal solution. The training configuration described in Section 3.I
INTRODUCTION
Reinforcement learning (RL) with verifiable rewards has recently emerged highly effective paradigm for enhancing large language models (LLMs) in complex reasoning domains, enabling models achieve superhuman performance (Jaech et al.) 2024; Guo al., [025 Kimi al.
2025 Comanici et al. 2025) Seed et al.) 2025). However, growing body of evidence reveals critical failure mode in this approach, which we term the exploration dilemma: while current methods improve average accuracy (pass @ l), they often achieve this by simply sharpening the policy distribution around a limited number of homogeneous solutions. This concentration of probability mass leads to a collapse in solution diversity, causing performance on the more general pass @k
Work done BytelDacce Seed. Corresponding to ji jiangyh22@mal wwt.nnnnuuee.. edu.cc

---

Published as a conference paper at ICLR 2026
metric to stagnate or even degrade compared to the base model Yue al. 2025 Wu al. 2025
He et a 2025a; Liu et al. 2025a] [Shah et al.) (025). Rather than discovering genuinely novel reasoning strategies, existing methods merely reinforce pre-trained biases and fail to expand the policy's capability frontier, posing a significant bottleneck to progress.
We argue this dilemma arises from a fundamental mismatch between the optimization landscape of LLMs and the dynamics of standard RL algorithms. In contrast to traditional RL settings (e.g., game playing (Mnih al. 20155 Silver et al. (017)) where training starts from a randomly initialized policy, LLMs begin with a highly specialized policy distribution that is already sharply peaked around certain solutions. If those initial peaks are not supported in the regions that yield optimal rewards. standard RL optimizers face a significant challenge: they struggle to escape the gravitational pull of the pretrained model's biases and tend to converge to a nearby, but often suboptimal mode (Wu et al.. 2025
He et al. 2025a). This prevents the discovery of more diverse and powerful reasoning paths.
To address this limitation, we introduce Risk-Sensitive RL framework designed to enhance exploration in LLM training, enabling policies to escape local optima induced by the initial bias. Our core idea is to replace the standard risk-neutral objective, which optimizes the mean of the reward distribution, with risk-seeking one that instead interpolates smoothly between the mean and the maximum reward. By employing an exponential utility function, we derive a new formulation of policy gradient with corresponding risk iis-stenitivv advantage. This advantage function dynamically re- re-weights optimization, placing greater emphasis on hard prompts where the model performs poorly, thereby driving the policy to explore under-explored regions of the solution space.
Our approach is instantiated as a simple yet powerful algorithm Risk-Sensitive GRPO (RS-GRPO), which can be implemented as a drop-in replacement for the advantage calculation in standard RL for LLM pipelines. Through extensive experiments on six mathematical reasoning benchmarks with a diverse set of six LLMs, we demonstrate that ES-GRPO consistently and significantly improves pass @k performance over both the base models and the standard GRPO baseline. Crucially, RS- GRPO achieves this while maintaining or even improving pass @ accuracy, striking a more effective balance than prior methods. Our main contributions are summarized as follows:
We introduce risk-sensitive RL framework address the exploration dilemma in LLM fine- tuning and instantiate as a simple yet powerful algorithm, Risk-Sensitive GRPO (RS-GRPO). (Section 2
We provide theoretical and empirical evidence that standard RL. often fails to reach the global optimum when the initial policy is sharply peaked and far from optimal, whereas our risk-sensitive formulation avoids this pitfall. (Section
We demonstrate through large-scale experiments on mathematical reasoning that RS-GRPO sig- nificantly improves pass @k k performance while maintaining or even enhancing pass@1 1 accuracy, achieving a superior trade-off compared to existing methods. (Section
1.1 RELATED WORK
RL Exploration Exploration remains a central challenge in reinforcement learning (RL), but its nature differs significantly between traditional applications and LLMs. In domains like game-playing. where policies are often trained from random initializations, broad exploration is essential and often encouraged by intrinsic motivation based on state novelty (Oudeyer et al., 2007]: [ellemare et al.) 2016 Pathak et al. 2017, Burda al.) 2018 Henaff et al. [022 Yang al. 2024b) Jiang et al. 2025a). While some have adapted intrinsic motivation to LLMs (Bai et al. 2025 Gao et al. 2025)
they often introduce auxiliary networks, complicating training and scaling.
The most direct method to encourage exploration in LLMs is to maximize policy entropy as an auxiliary objective, but its effectiveness can be limited (Cui al. 2025] Chen et al. (2025), spurring research into alternative directions. Some approaches focus on enhancing the reasoning process
through self-reflection (Jiang et al. (2025b) Kumar et al., 2024 Ma et al. 2025a Yea al. 2025). while others investigate policy entropy dynamics to prevent mode collapse Yuu et al. 2025 Cui et al. 2025; Cheng et al., 2025). Orthogonal to these methods, our work contributes to a line of research focused on directly optimizing for inference-time objectives. This can be viewed as form of lisk-sensitive learning, where early efforts on Best-of-N (BoN) aligmment (Gui et al. 2024 Amini et al. 2025 Chow et al.) 20025] Balashankar et al. (022)) have evolved into policy gradient methods for pass @ k optimization. Notable developments include various policy gradient formulations for
pass @e k (Tang et al. 2025 Walder & Karkhanis, 2025 Mahdavi et al. 2025
Chen et al. 2025). As
2

---

Published as a conference paper at ICLR 2026
Table I: Comparison of @ass@k optimization methods with Risk-Sensitive RL.
Methods
Binary Rewards
Continuous Rewards
Dense Signal
Tang et al. (0255)
X
Walder Karkhanis
2025
X
Mahdavi et al. 2025
X
Chen et (2025)
X
Risk-Sensitive (ours)
shown in Table and detailed in Appendix D our risk-sensitive framework offers two key advantages over these pass @ k optimization methods. First, our formulation naturally handles continuous rewards, whereas prior methods are often restricted to binary signals (Chen et al. 2025 Mahdavi et al. (025). Second, our method yields a denser advantage signal. In most of pass @ methods, the optimization weight vanishes once prompt accuracy surpasses a given threshold (e.g.. - N which can hinder Pass @ improvement. Our risk-sensitive formulation, by contrast, sustains a non-zero gradient even for high-accuracy prompts, thereby facilitating more effective trade-off between Pass @k and @ass I performance. Detailed comparsion can found in Appendix D
and
Risk-Sensitive RL Risk-sensitive RL (Howard & Matheson 1972, Heger 1994 Neuneier &
Mihatsch 1998 Mihattsh & Neuneier 2002) aims to model and manage the risks associated with
decision- making under uncertainty, moving beyond the standard expectation-basee objective. While early work focused on risk-averse strategies for safety-critical domains such as financial trading (Filos)
2019), energy storage (Liu al. 2024), and robotics (Nass al.) 2019; Noorani al. 2022 Shi et al. 2024), the advent of distributional RI. (Bellemare et al. (017)) has enabled more nuanced approaches. This paradigm facilitates not only risk-averse but also risk-seeking behaviors, which have been shown to promote exploration in domains like game playing (Jiang et al. [2024] Ma et al.
2025b; Mavrin et al. 2019). Our work posits that risk-secking optimization critical for escaping the sharply peaked initial policy distribution of pre-trained models and enabling broader exploration.
1.2 BACKGROUND
We formulate language generation as a reinforcement learning (RL) problem. A language model acts as a policy тe. which generates response y prompt x with probability Ne (y/ x). The quality of each response is measured by reward function r(æ, y). The standard objective is to maximize expected reward:
(mT) (o [r(z, v)].
This objective typically optimized via policy gradient, as stated by:
VoJ(TO) EE. o (y) V log [o(y)²)]
(2)
where Ah denotes the advantage function. Empirically, for each prompt we sample N responses (y.) N. from (-|x)) and construct stochastic gradient estimates:
N
1
= VOT(TT))
A To (ys Vo (o
N
where AT (y:) = r(y)) 1/2 Σ r(y)). with y TH (-(xx)), is the advantage estimate For clarity, omit terms common in RLHF, such regularization and importance sampling, and drop the explicit dependency on x when unambiguous.
Pass @k. @ass@k (Chen et al.) [021 Kulal et al. 2019) estimates the probability that at least one of generated responses is correct. It serves as a key inference-time objective, reflecting exploration ability and approximating best-of-k under a reliable reward model:
Pass@k = E (u romg [max (r(m)),... ryn)))]
We omit the standard deviation normalization used in the original GRPO Shao al. which, as noted by DrGRPO (Liu et al. 20256), introduces bias.
2024 algorithm,
3

---

Published as a conference paper at ICLR 2026
Positive
Negative
Cumulative
8=0
Adventage
0.5
Advantage
B=4
-0.5
0.0
8=0
-075
0.5
0.0
0.2 0.4
0.2
0.2
0.8
Reward
Prompt Accuracy
Prompe Accuracy
Prompt Accuracy
(a) Continuous Reward Setting
(b) Binary Reward Setting
Figure 2: Analysis of the risk-sensitive advantage function with varying iisk-sensttivity B.
RISK-SENSITIVE REINFORCEMENT LEARNING
Desiderata. Standard average reward optimization is often insufficient for tasks where exploration is critical, particularly when the initial policy distribution is sharply peaked. To this end, we design a new objective to promote exploration, for which establish two primary desiderata. First, it should value all high-reward outcomes, not just the most probable one, thereby moving beyond simple mean-reward maximization and toward should provide a flexible and principled mechanism to interpolate between optimizing for the mean reward and the maximum reward, balancing exploration and exploitation. We find that risk-sensitive RL provides natural framework for designing such an objective.
cetttuu----eaa---eeeeiiie objective. Second, the objective
Objective. To meet these desiderata, we employ the disk-sensitive objective derived from exponential utility (Howard & Matheson 1972). This objective provides principled way to control the trade-off between exploration and exploitation. For given policy xo and prompt æ, the risk-sensitive objective is defined as:
1
log Eyywro a)) [comt]]]
(Ts) = EZnD
(5)
[B
where the hyperparameter 3 R controls the risk-sensitivity level:
Risk-Neutral (B - 0): Recovers the standard expected reward, E[r(v)]-
Risk-Seeking (3 - +oo): Approaches the maximum reward, max, r(y), encouraging exploration.
Risk-Averse (3 -oo): Approaches the minimum reward, min, r(y), promoting robustness.
To effectively explore the solution space, we adopt a risk-seeking objective (3 > 0). As 3 increases, the objective places greater weight on high-reward outcomes, smoothly interpolating from the mean to the maximum reward. As 3 - 0, it recovers the standard mean-reward objective. We now derive the corresponding policy gradient.
2.1 POLICY GRADIENT FOR THE RISK-SENSITIVE OBJECTIVE
We first derive the risk sensitens policy gradient in thcorem below, and defer its proof to Appx.
Theorem 1. The policy gradient of the risk-sensitive objective in Eq. (5) is given by
VoJRS(TTO) = E.WD. (-mo(()))
A (य) V log no ( [z)
(6)
where the risk-sensitive advantage A is
(r()
A (v) 3 (es -1).
Practical Implementation practice, we approximate the gradient for each prompt æ using N samples [ ~ T=(+|x), yielding vnJn(TT)) = ΣN Age (v.) Vo log To x), where the empirical advantage is defined as
N N eBr(vs) -1)
(r(w)
A (ys) =
(8)
3
4

---

Published as a conference paper at ICLR 2026
1.0
0.20
Initial Polley Distribution
1.0
0.8
0..5
Reward
Reward
0.8
0.6
0.10
8
0.4
0.6
8
0.2
0.05
B-64
0.4
0.0
0.00
50
100
200000
400000
Action Index
Steps
Figure 3: A bandit experiment demonstrating that risk-sensitive RL can escape local optimum that traps its standard RL counterpart. Left: The reward landscape shows global optimum and distinct local optimum where the policy is initialized. Right: standard risk-neutral policy (B = 0) trapped locally, while risk-sensitive policies (B Z 4) converge to the global optimum.
key feature of this formulation is that only alters the advantage computation while leaving the policy gradient structure intact. This allows our risk-sensitive advantage to serve as a drop-in replacement in existing GRPO-based RL algorithms (Shao et al. 2024 Yu et al. 2025 Liu et al. 2025b), requiring only minimal code modifications.
2.2 ANALYSIS OF THE
ADVANTAGE FUNCTION
The effectiveness our method stems from the behavior of the risk-sensitive advantage function. To analyze its properties, we examine how the distribution of advantage values changes with the risk-sensitivity level, B3 as illustrated in Figure 2
Continuous Reward Setting. As shown in Figure aa) in a continuous reward space, the standard policy gradient (3 - 0) yields an advantage that is linear with the reward. As 3 increases, the function sharpens into a step-like curve. This transformation amplifies the advantage for high-reward samples and suppresses for low-reward ones.
Binary Reward Setting. Figure 2bb illustrates the advantage dynamics in binary reward setting (common in RLVR) as a function of prompt accuracy- fraction of correct responses out of 16 samples. As 3 increases, the advantage function increasingly prioritizes correct responses hard prompts (low accuracy) while reducing the penalty for incorrect ones on easy prompts (high accuracy), as seen in the Positive and Negative subplots. Consequently, the Cumulative plot shows that the total advantage magnitude per prompt (sum of absolute advantages) shifts from peaking 50%-accuracy prompts (for B + 0) toward lower-accuracy ones. This demonstrates that as increases, risk-sensitive RL re-weights the advantage signals to prioritize harder prompts.
3 WHY TISK-SENSIIVE RL IS BETTER
Fine-tuning LLMs with RL often starts from sharply peaked pretrained policy. Standard mean- reward optimization methods can be trapped in local optima corresponding to high-probability regions of this initial distribution, failing to discover global optima in proobabbbylity areas. The risk-sensitive approach we adopt is designed to overcome this limitation. This section provides both empirical and theoretical support for this claim.
3.1 EMPIRICAL PERSPECTIVE
To illustrate the exploration dilemma, we design 100-armed bandit problem where each action yields deterministic reward. Figure 3h visualizes the reward landscape, which features two distinct peaks: a global optimum (reward 1.0) and prominent suboptimal one (reward 0.6). We deliberately initialize the policy as sharp distribution centered the suboptimal arm. This setup analogous to the LLM fine-tuning challenge, where pretrained models may exhibit a bias toward solutions that are good but may not globally optimal.
We employ our risk-scnsitive policy gradient algorithm to train policies with varying risk-sennitivity parameters (B > 0). The learning curves in Figure reveal evident divergence in performance. The
5

---

Published as a conference paper at ICLR 2026
standard risk-neutral policy (3 = 0) and its low-risk-sensitiiiy counterparts (e.g., 3 4) rapidly converge to the suboptimal reward of 0.6, becoming trapped in the local optimum by exploiting the initial policy's nigh-probabiligg region. By contrast, policies with sufficient risk-sceking behavior (3 2 4) successfully escape this trap and converge to the globally optimal reward of 1.0. The evolution of the policy distribution during training is illustrated in Figure T
3.2 THEORETICAL PERSPECTIVE
In this section, We examine the one-step policy update in simple multi-armed bandit setting, demonstrating the fundamental advantage of the risk-sensitive objective from Eq. (5). For clarity, we assume the uniqueness of the optimal action. Our results can be generalized to settings with multiple optimal actions, which we defer to Appendix [
Setup and Notation We study the K-armed bandit problem with action space A == f(a.. .... ak and denote the unique optimal arm by a" E A. We assume a bounded reward function r A - [0, 1. and consider the softmax policy To parameterized by 0 E RR: Vi E [K], To(a;) = e. / 2ー=1 e
we denote the updated parameters after one step of standard policy gradient (Eq. (2) as 0, and after one step of risk-sensitive policy gradient (Eq. 6 as 8 The learning rate is assumed to be the same and omitted for simplicity,
Let's compare single policy update step. Given a starting policy
Our first result highlights critical flaw in the standard policy gradient: it can decrease the probability of the optimal action. This happens if a suboptimal action exists that is nonetheless better than the average, which can misdirect the update.
Lemma 2.
If there is an action og with reward r(as) such that r(a") ) > r(as)) > ming r(az), then there exist policy parameters 0 for which the standard policy gradient update decreases the probability of the optimal action, i.e., Tā(a") < Te Ta(a").
In contrast, our next lemma states that the risk-sensitive policy gradient guarantees an improvement for the optimal action, as long as 3 is sufficiently large.
Lemma 3. For any policy ua and reward function r, there is risk risk-sensitivity level B such that for all B >>>>> B, the risk-sensitive update increases the probability of the optimal action: mg=(a") > To =o(a²)
Together, Lem. 2 and Lem. 3 exxlaiin why increasing 3 helps to escape from local optima in Fig. and provide theoretical insights into the benefits of risk-sensitive policy gradients.
This raises a natural question: should we always use the largest possible 3? The next lemma gives negative answer: once B exceeds certain threshold, the policy improvement on while still positive positive-dderrnnee decreases as grows.
Lemma 4. For any policy and reward function т, there is threshold 3 such that for any 31 > B2= B, the improvement on the optimal action is smaller for the larger 3: Tgo, (a*) To(a") < Tgaz (a*) To (a")
This result aligns with the convergence speed shown on the right of Fig. 3 where increasing B eventually slows down the convergence. This provides crucial guidance for tuning 3 in practice: should be large to enhance exploration, but not so large that it hinders convergence speed.
EXPERIMENTS
4.1
SETUP
Training Setting We focus on mathematical reasoning tasks and train our approach on six base models: -0wen2.5-Math11.3.
Owen2.5-MMath771 Qwen2.5-7B, Qwen3-4B-Bant and L.Lam331-88B Instruct (Yang et al. 2024a 2025 Grattafiori et al. 2024). We name our method RS-GRPO, which extends the GRPO algorithm with the risk tisk-sensttive ad advantage function (Eq. 8). The training framework is built upon VeRL (Sheng et al. 2024) and incorporates techniques from DAPO (Yu let al.) 2025), such as dynamic sampling (filtering samples with all-O all-1 accuracy in each rollout) and clip-higher. We keep the shared hyperparameters identical across all comparative experiments. Full details of the training are provided in the Appendix F

---

Published as a conference paper at ICLR 2026
Base
GRPO
HMMT RS-GRPO
HMMT Feb24
20-
97-
2222222'' 2727
2222222 7770
227227222
AIME24
AIME25
HMMT Feb24
HMMT Fab25
CMIMC25
MATH500
22222222222 21220
22222 22
22222 22 2-222
2222222 2722
AIME24
AIME25
HMMT Fab24
HMMT. Fab25
CMIMC25
MATH500
90
20
76
2322233''''
2
HMMT Feb24
HMMT_ IMMM__ee222 Feb22
CMIMC25
MATH500
Passak (%)
21
222222222
HMMT
HMMT Feb25
(9)
12-
Figure 4: @ performance of RS-GRPO, GRPO, and base models. (Qwen2.5- 1.5B,
Qwen2.5-Math-77 Qwen2.5-7B, Qwen3-4B-Bas and Instruct from top to bottom.)
Camulative Solve Rate
Training Rewards
Teatiing Pass@l (%)
22
913
20
ng Passig 32
16
27
13
200
200
300
Cumulative Solve Rate
Training Rewards
8-0
B-2
Testing Pass@1 (%6)
8
8-8
gistin Pes5@332 1%6)
34
50
22
8
20
8
34
609
200
400
200
600
Steps
Steps
Steps
Figure 5: Ablation Study of 3 in RS-GRPO on Qwen2.5-Math- 1.5B (top) and -7B (bottom).
Evaluation Setting Our evaluation is conducted on six widely-used mathematical reasoning bench-
marks: MATH500 (Cobbe et al., (021), AIME24, AIME25 (Mathematical Association of Amer
ical 2025), HMMT-Feb24, HMMT-Feb25 (Harvard-MIT Mathematics Tournament] 2025), and
CMIMC25 (Carnegie Mellon Informatics and Mathematics Competition 2025). The MATH500 benchmark contains 500 problems, while the other datasets consist of 30 or 40 problems each. For most benchmarks, we generate 1024 candidate solutions per problem. However, for the larger
MATH500 dataset, N 32 ensure the evaluation remains computationally feasible.
7

---

Published as a conference paper at ICLR 2026
4.2 PERFORMANCE ON PASS @K EVALUATION OF SEN-SSENI RL
We present the pass @ k performance for k E {1, 1024] across five LLMs and six benchmarks in Fig. 4 The results reveal that RS-GRPO consistently and significantly outperforms both the the standard GRPO baseline, showing comprehensive improvements on the pass metric. Notably, for several models (c.g., Qwen2.5-Math-L5 1.5B, Qwen2.5-Math-27
and Qwen3-4B), GRPO underper- forms the base model at high values of k (k > 256). This suggests that GRPO merely sharpens the existing policy distribution rather than discovering novel solutions. In contrast, RS- RS-GRPO surpasses the base model's performance, demonstrating its ability to expand the model's exploratory boundaries. However, for some models, such as Qwen2.5-7B and Llama3. I-maa...IBBB.. I-8B-Instruct, RS-GRPO fails to outperform the base model at high values of k. We speculate this occurs when the optimal policy is prohibitively distant from the initial distribution, causing RS-GRPO to converge to a local optinum. Nonetheless, this still represents significant improvement over GRPO.
4.3 DIFFERENT IMPACT OF SENS-SENEI HYPERPARAMETER /
We conduct an ablation study on the risk-sensitive parameter to analyze its impact on training dynamics. We track several key metrics including the cumulative solve rate the training set (a problem is counted as solved if at least one generated response is correct; this equals the fraction of training problems solved), training reward, and test performance (pass@) I and pass @ (20@222fff- for 8 E {0, 2, 8). The case where 3 = 0 is equivalent to standard RL (i.e. GRPO).
and Qwen2.5 -Math-5-M models. We observe that as 3 increases, the cumulative solve rate on the training data improves, while the training reward grows more slowly. This result aligns with the theoretical analysis in Sec. 3.2 This slower reward growth is not necessarily drawback, as it may indicate regularization effect that prevents overfitting On the test benchmarks, RS GRPO yields substantial gains in pass 32 performance, with an improvement of approximately 5%. While pass I performance is maintained relative to standard RI., an appropriate choice of / (e.g.. 3 = 2) can lead to a 1-2% improvement, as observed on 0wen2.5-Math-77 This suggests that balancing the objectives of optimizing for the mean reward (pass @ I) and the maximum reward (pass@k) k) is crucial. We conclude that 3 = 2 offers an effective trade-off, achieving strong pass @ k performance while simultaneously enhancing pass@1 @ 1.
Figure 5 illlssrattes the training dynamics for the Owwe2.5-MMth-11..
4.4
COMPARISON TO OTHER PASS @K OPTIMIZATION BASELINES
Chenetal.
Mahdavi al.
Chen et
Mahdavi st al.
Walder Karihanis
Rick-Sensitvv (Ours)
Karkhanis
Sensitiee
1esting Pass 32 (%)
Testing Passx@ (96)
Testing Pacc@832 (%)
20
30
26-
40-
24
Texding
266
22
36
26
20
34
200
400
100
200
300
400
200
400
100
200
300
400
Steps
Steps
Steps
Steps
Figure 6: Training dynamics of Dessitensttive RL vs. other Pass optimization methods. (Left:
Qwen2.55-Mtt--1.55. Right: -wahh.5-
We compare RS- GRPO with several pass@k optimization baselines (Walder & Karkhanis 2025
Mahdavi et al. 2025 Chen al. 2025 on the 00we2.5-MMat- and Owen2.5-Maah-777 models using the deepmath 103k dataset. ssows the training dynamics: RS-GRPO generally matches the @ performance of baselines while consistently outperforming them in pass@1 1. We attribute this improvement to the denser advantage signals provided by risk-sensitive objective, as discussed in related work.
Table 2 provides more comprehensive evaluation, covering five base models and three training datasets (math 12k, deepmath 103k, dapol7k). While many pass @ k-oriented baselines fail to improve @ 1 over GRPO, RS- RS-GRPO achieves least comparable Pass @ performance and exceeds GRPO by an average of about 2% across three models Doween..57B--.. In addition, RS-GRPO consistently improves pass @ 32 over GRPO by an average of about 4%.
Qwen2.5-7B, Qwen3-4B).
8

---

Published as a conference paper at ICLR 2026
Table 2: Main results on mathematical reasoning benchmarks, reporting pass@11 I and pass@33 ( (%) for five models and three training datasets. Subscripts denote improvement over GRPO. RS-GRPO consistently outperforms the GRPO baseline on pass 32, while maintaining or improving pass @ accuracy. RS-GRPO also achieves a better trade-off than prior pass pass@k k optimization methods.
AIME24
ADME25
HMMT
HMMI
CMIMC15
MAT1550
Assragy
5 2 wwB.B//ooo///BBB
Piar Paar
3.5/25.6
1.3/17.
260/848
63/
GRee
166/364
16.24114
13.4
3.422544
W33/440
37.4/3755
Walder Karklann 2025
13.7/34.5
10.03882
5.7 IS7
1.5 13.7
4.3/25.0
64.7/926
1677
15,9/450
152/434
687222 22.7
1.27 15.5
74/314
75.5/95.
20.67
Charenat
16.2/46.1
186/41.
5.1 20.9
1./ 15.4
$.8/2933
79.77443
20.0/412
167 453
163 on 47.
72 on 19
Tuuns: 17.6..m
76 9SA In
Quee22 S-Mah---
(deopnuthii22g Pazeie Porvke22
288
1.
1.17 14
0.766
04127
22-57913
GRPO
15.7
193//412
7.5/
11.2/241
15.0
62.3
21.1/4211
5.2 10.9
5.K/ 24
10.9/M 16.2
836 92.8
26 47
50 10000.0
20 Fo5
12 294433
&.oes 26
147.00//328..
28.621
(heec
Mosh-7 (23
Foceed Parste
Bonc
2.9 29.8
15/22.0
1.1 14.8
0.7/6.9
04722.
22.579122
4/9 28-3
GRBO
31.1 6610
18.7/326
23./22.55
3.07 13.8
2.7/113
72519223
24.5 44L/
Cheatoea 2003
26.7/67.4
176/4866
13.8/25.8
4B/ 19.3
5/2/22.8
2307798
24.5 45.
25.8/68.9
15
19.6
3.3/1566
164.0%/ 26.
54hmm 2014ue
Queen2 Sree2.5-Monh M-Mhh- (manh)22 22y Parel /Parie
CIRO
36.0/58.3
139/365
/
4./23.9
23.7/418
RSS-GRPO
167.075
732A
RLS
Deve2.5.2 78 (kk Pus
9.47958
3.7
9.
62.57
GEPO
13:7/360
6.4/250
91.4
RS-GRPO
/387....
17.4.am
215....
Qxen2 4- Suse (muath Fus Pus4
0.57
5.0/30.3
2.7/26.5
66.77938
148/384
GRPO
23.0/5669
46.5
136/31
9.5
136/355
K5.5 94.3
RS-GRPO
2677.12 68.3.k
26.1.03 /48.3.11
14 35....
13.
14.
7418
87.
99.
30.3.11 B3L
Aiowa- J.7 4B butract (wuut) Pourth Pusstt
Banz
5./3310
0.77 10.1
0.266.1
13.3
50.47 99.6
GRID
9MV 25:8
1.7755
25/39
0.100.1
06 FM1
50.81 788
118/2110
RS-CRPO
9:4.357 / (33
B. 19.3
12 54mi
8..es /54.53)
8.788.1
59.9.au R3.8.5 12...02 72.5
We observe that the approach of Walder & Karkhanis 2025) performs unsatisfactorily, mainly because its advantage estimates remain strictly positive (see Appendix D). The absence of negative advantages causes rapid entropy collapse and poor training performance, consistent with prior findings on the importance of negative signals (Zhu et al. 2025).
4.5 ANALYSIS OF PASS @K K IMPROVEMENT
0.30
We analyze how resk-sensitive RL enhances both pass @k and pass@1 1 performance using the Qwen2.5-Math 7B model trained on the deepmath 103k dataset. For each problem in five benchmarks-AlMaa AIME24, AIME25, SHAMT_Febb24 HLMMT_Feb25, and CMIMC25 -w sample 1024 solutions, each corresponding to single final answer. The left of Figure illlstrates the unique answers ratio. We observe that after RS-GRPO training, the num- ber of unique answers shows a significant increase com- pared to that of GRPO. This indicates that risk-sensitive RL enhances the diversity of reasoning paths.
Riato
0,00
1.950 823 191 131
0.25
(20,10-008 911 111 0.3
0.20
12.4 244-260--0 8:0 382 031
0.15
6.01
12
012 031
0.10
00.5
3.0
0.00
06.44
Qualter Acarsyy (CaPO)
Figure 7: Left: RS-GRPO finds more unique solutions. Right: Accuracy tran- sition map from GRPO to RS-GRPO. The heatmap in Figure provides detailed view of the prompt accuracy transitions from GRPO to RS-GRPO. We observe that 8% of prompts with an accuracy of O under GRPO achieve an accuracy in the (0, 0.4] range with RS-GRPO, while only 4% show the opposite change. This shift is the primary contributor to the improved pass @k performance. Simultaneously, 3% of prompts with an accuracy of are shifted to the (0.6, 1) range, while only 1% move in the opposite direction. This performance trade-off explains why pass @ improvements are more modest than gains in pass @ k: while RS-GRPO can solve more problems than GRPO, occasionally makes errors on simpler ones.
4.6 PERFORMANCE OF RISK-SENSITIVE RL ON OOD TASKS.
We directly evaluated Owen2.5-MM. M-tth7BB models trained on the deepmath 103 dataset (whose math performance was already reported in Table 2) GPQA-Diamone (Rein et al. (0244), which covers biology. physics, and chemistry. The results show that RS- GRPO consistently outperforms GRPO from pass @ I to pass 32,, aligning with our findings on math tasks.

---

Published as a conference paper at ICLR 2026
Table 3: Pass@k comparison on GPQA-Diamond
Model
Pass@1 1 (%)
Pass 2 (%)
Pass@4 (%)
Pass@8 (%)
Pass 16 (%)
Pass@32 (%6)
4.4
8.4
15.5
44.5
GRPO
41.2
52.2
62.8
78.5
84.8
RS-GRPO (ours)
41.8
54.1
63.5
75.2
83.1
88.9
Table 4: Pass comparison on LiveCodeBenchvS..
Pass@1 (%)
Pass@2 2 (%)
Pass@4 (%)
Pass@8 (%)
Pass 16 (%)
Pass@32 (%)
Base
22.6
27.4
31.8
35.6
39.0
42.1
GRPO
28.5
30.4
31.7
32.7
33.6
34.3
RS-GRPO (ours)
29.7
32.5
34.6
36.2
37.7
39.1
4.7
PERFORMANCE OF RS-GRPO ON CODING TASKS.
on the Hug- conducted additional code generation experiments with Qwen2.5-7B-Instuee gingFace dataset ganler/ code r1 11-11kk 12k, and evaluated on LiveCodeBench v5 (Jain al. 2025) (880 questions in total). We set the maximum response length to 4k tokens and keep all other hyperparameters identical to Table 3. The results show that RS- RS-GRPO outperforms GRPO coding tasks, with a 1.2% improvement in pass@) I and a 4.5% improvement in pass@88. While the base model's pass @8 exceeds GRPO's trained result, RS-GRPO's pass@8 still surpasses the base model.
4.8 ABLATION KL.
Our main experiments omit KL regularization, following the prevalent RLVR practice. To verify that RS-GRPO's exploration benefits persist under explicit KI. control, we additionally compare RS-GRPO+KI. with GRPO+KL. KL. by tracking the evolution of the average Pass (1,..., 33] across five benchmarks (AIME24, AIME25, CMIMC25, HMMT WWMMM_Fee24,, HMMT Feb25) during training. As shown in Figure g trained Qwen2.5 5-Ma55- (coefficient 0.001). RS KS-GLRO++L consistently surpasses GRPO+KL from Pass to Pass @ 32, although the KL term slightly lowers the overall performance ceiling.
HMMT_Feb25)
the dapo 7k dataset with KL regularization
Po00e1 (5))
Pogge (5)
156
220
180
760
150
220
165
280
106
706
150
750
Stopo
Stops
Staps
Stops
St050
Figure 8: Ablation KL on RS-GRPO.
CONCLUSION
In this paper, we aim to address the exploration dilemma in fine-tuning large language models with reinforcement learning, where existing methods often improve pass @1 accuracy the expense of solution diversity, leading to stagnation or even degradation in pass@k performance. We argue that this arises from the failure of standard RL algorithms escape the local optima defined by the charply-peaked initial policy of pretrained models. To overcome this, we introduce risk-sensitive reinforcement learning framework, instantiated the RS-GRPO algorithm. By optimizing risk- seeking objective, our method encourages the policy to explore under-explored regions of the solution space, discovering novel reasoning paths. Our experiments on mathematical reasoning benchmarks demonstrate that RS-GRPO significantly improves pass@) @k performance while maintaining or im- proving pass accuracy, achieving a more favorable trade-off than prior methods. Future work could explore the application of risk-sensitive objectives to other generative modeling domains and investigate their interplay with other exploration techniques (see Appendix for limitations).
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research. Our novel algorithm, RS-GRPO. detailed in Section with theoretical claims and their complete proofs provided in Appendix [C] All datasets used for training and evaluation are publicly available, with specific sources and ver- sions documented in Appendix F This appendix also contains a comprehensive description of our experimental setup, including all hyperparameter settings (Table and training configurations. To facilitate direct replication of our results, we provide the key source code as part of the supplementary material.
ACKNOWLEDGEMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62192751, in part by the Key R&D Project of China under Grant 2017YFC0704100, in part by the 111 International Collaboration Program of China under Grant B25027. in part by BNRist Program under Grant ENR2019TDO1000 R&D Project under Grant CX/Ki-2020-0006, in part by the InnoHK Initiative, The Government of HKSAR, and in part by the Laboratory for Al-Powered Financial Technologies.
in part by the National Innovation Center of High Speed Train
REFERENCES
Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https
//openeevew. net /
Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. Online prefer- ence alignment for language models via count-based exploration. In The Thirteenth International Conference on Learning Representations, 2025. URL CRLhttpss: https:://ppeneeveettt www.w/wwwwwww..eeeeeeeeeeeeeeeee for
id=cf
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, Ananda Theertha Suresh, and Ahmad Beirami. Infalign: Inference-awarc language model alignment. In Forty-second International Conference Machine Learning, 2025. URL https: /owpereeevvveeeeeoeeeoeeeeeeeeee net. / forum? d= hInfvt7c4p
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unilying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International conference on machine learning, pp. 449-458. PMLR, 2017.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810,1244,, 2018.
Camegie Mellon Informatics and Mathematics Competition. Cmime 2025. Inthh. cmu. www.ccm....eeeeeemmmmm 2025. Accessed: 2025-09-15.
https: emime
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. @a training for adaptively balaneing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508. 10751 2025.
Daixuan Cheng, Shaohan Huang, Xuckai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu (5l.:2556..4555 14758, 2025.
Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.
11

---

Published as a conference paper at ICLR 2026
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Aviral Kumar, Rishabh Agar- wal, Sridhar Thiagarajan, Craig Boutilier, and Aleksandra Faust. Inference- aware fine-tuning for best-of-n sampling in large language models. In The Thirteenth International Conference on Learn- ing Representations, 2025. ht /wwpoopeeeeeeeeeeeeeeeeeeeeeeeee net/ ftr/foormmmm/ d=gg7==gghhh
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint wwxXiv2505.226117 2025.
Grégoire Delétang, Jordi Grau-Moya, Markus Kunesch, Tim Genewein, Rob Brekelmans, Shane Legg, and Pedro A. Ortega. Model-free risk-sensitive reinforcement learning, 2021. URL https wwwxiv..rg//ab.a...a.o
02907
Angelos Filos. Reinforcement learning for portfolio management. arXiv preprint arXiv:1909009571 2019.
Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xiangyu Zhao. Navigate the unknown: Enhancing 11m reasoning with intrinsic motivation guided exploration. arXiv preprini arXiv:2505.17621 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Lin Gui, Cristina Garbacea, and Victor Veitch. BoNBon alignment for large language models and the sweetness of best-of-n sampling. In The Thirty-eighuh Annual Conference on Neural Information Processing Systems, 2024. URL https::
hwt.:/.e//eeee. Eorum aediidhhhhiMM.. rbx5
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Harvard-MIT Mathematics Tournament. Hmmt february 2025.
https: wwww hmmnt org/
2025. Accessed: 2025-09-15.
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening. arXiv preprint (iXXi:25506003555 2025a.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-1003 A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504. /456, 2025b.
Matthias Heger. Consideration of risk in reinforcement learning. In William W. Cohen and Haym Hirsh (eds.), Machine Learning Proceedings 1994, pp. 105-111. Morgan Kauf- mann, San Francisco (CA), 1994. ISBN 978-1-55860-335-6 doi: htpp://ddoi.or/11..000 1016/ URL https: / www.:science... rect. wtt.:co..ccccceee
13778-1-55800-335665000 Sttic/e///1//978115500333333
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631-37666, 2022.
Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science, 18(7):356-36,, 1972.
12

---

Published as a conference paper at ICLR 2026
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free eval- uation of large language models for code. In The Thirteenth International Conference Learning Representations, 2025. URL htttt:://ppenreeiieeeee
Yuhua Jiang, Qihan Liu, Xiaoteng Ma, Chenghao Li, Yiqin Yang. Jun Yang. Bin Liang, and Qianchuan Zhao. Learning diverse risk preferences in copulation-based self-play, In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1991-12 2024.
Yuhua Jiang, Qihan Liu, Yiqin Yang, Xiaoteng Ma. Dianyu Zhong, Hao Hu, Jun Yang, Bin Liang, Bo XU, Chongjie Zhang, and Qianchuan Zhao. Episodic novelty through temporal distance. In The Thirteenth International Conference on Learning Representations, 2025a. URL https
//openreveew net wwwpa//iiwe..ett/oumneeeeee... worrmmmdd-I/eaajaa
Yuhua Jiang, Yuwen Xiong, Yufeng Yuan, Chao Xin, Wenyuan Xu, Yu Yue, Qianchuan Zhao, and Lin Yan. Pag: Multi-turn reinforced IIm self-correction with policy as generative verifier. arXiv preprint arXiv:2506.1044 2025b.
Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.
2025.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Bearch-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv: 12917, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint X51v:2505.22844, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Zifan Liu, Huan Zhao, Guolong Liu, Gaoqi Liang, Junhua Zhao, and Jing Qiu. Risk-sensitive mobile battery energy storage system control with deep reinforcement learning and hybrid risk estimation method. IEEE Transactions on Smart Grid, 15(4):4443-4158, 2024. doi: 10. 109/ 3358838.
Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S r. Teaching llms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502. 12855,, 2025a.
Xiaoteng Ma, Junyao Chen, Li Xia, Jun Yang, Qianchuan Zhao, and Zhengyuan Zhou. Dsac: Distributional soft actor-critic for risk-sensitive reinforcement learning. Journal of Artificial Intelligence Research, 83, 2025b.
Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Renjie Liao, and Christos Thrampoulidis. Beyond accuracy: A policy gradient reweighting approach for pass maximization in LLMs. In 2nd Al for Math Workshop @ ICML 2025, 2025. URL Dn3gk
net ffr.umiidddd
13

---

Published as a conference paper at ICLR 2026
Mathematical Association of America. 2025 american invitational mathematics examination (aime).
https::/ www maa. org cow.ttttooomttttiooono. aime 2025. Accessed: 2025-09-15.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International conference on machine learning. pp. 4424-442333 4434. PMLR, 2019.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. Hal Daumé and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2229. PMLR, 13-18 Jul 2020.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49 (2):667-290, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 2015.
David Nass, Boris Belousov, and Jan Peters. Entropic risk measure in policy search. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1101-1106. IEEE, 2019.
Ralph Neuneier and Oliver Mihatsch. Risk sensitive reinforcement learning. Advances in Neural Information Processing Systems, 11, 1998.
Erfaun Noorani, Christos Mavridis, and John Baras. Risk-sensitive reinforcement learning with exponential criteria. arXiv preprint arXiv2221.009010 2022.
Pierre-Yver Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions evolutionary computation, 11(2):265-266, 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2172-777777 PMLR, 2017.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024. URL httttps::/opeerevviieii
ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504. 13914, 2025.
Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre- training, arXiv preprint wxxv:2504..0002. 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 0009.19256 2024.
Jiyuan Shi, Chenjia Bai, Haoran He, Lei Han. Dong Wang, Bin Zhao, Mingguo Zhao, Xiu Li. and Xuelong Li. Robust quadrupedal locomotion via risk-averse policy learning. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 11459-1 1466. IEFF, 2024.
14

---

Published as a conference paper at ICLR 2026
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self--pay with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. 2017.
Yunhao Tang, Kunhao Zheng, Gabriel Synnaeve, and Remi Munos. Optimizing language mod- els for inference time objectives using reinforcement learning. In Forty-second International Conference on Machine Learning, 2025. URL https: /tEE::://ppeneview..r mettfoormmli ZVWJ05YTZ44
Christian Walder and Deep Karkhanis. Pass@ policy optimization: Solving harder reinforcement learning problems. arXiv preprint arXiv:2505. 2025.
Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin. arXiv preprint vrXiv:2507.14844 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self- eeff-improvement arXiv preprint arXiv:2409. 2024a.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.099 033888 2025.
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti-exploration with distributional random network distillation. arXiv preprint arXiv:2401.097500 2024b.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning llms. arXiv preprint arXiv:2502. 033333 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint 2503::22003.447000 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 250.::2750.....7,,, 2025.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness negative reinforcement in Ul reasoning. arXiv preprint 3rXi::2506.01347,200 2025.
15

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
limitation of our current work is that all experiments were conducted with a fixed risk-seeking parameter B. A natural extension is to dynamically adjust during training to better balance exploration and exploitation. We experimented with several heuristics, including:
Initiating training with a high /3 value, followed by a linear or cosine decay schedule after initial training period.
Beginning with a large 3 and subsequently switching to standard mean-reward optimization after an initial phase.
Employing an adaptive / based on prompt difficulty, assigning larger values to harder prompts and smaller values to easier ones.
However, none of these strategies yielded superior pass performance compared to training with a fixed, well-chosen 3 (i.e., / 2). Devising optimal dynamic strategy balance exploration and exploitation remains a challenging open problem.
Another limitation is that methods tailored bandit settings. Extending them general Markov Decision Processes (MDPs) is non-trivial, because credit assignment in MDPs requires learning value function, and a principled design for risk-sensitive value functions remains challeng- ing (Jiang et al. [224]] Delétang et al. [202T). Fortunately, most RLVR tasks tasks-including multi-turn ones such as coding and search agents only reveal a reward at the end of generation, so they can be naturally modeled as bandit problems and handled by our approach. In short, whenever GRPO-style algorithm is applicable, our risk- fisk-sensitive RL can be directly deployed.
B THE USE OF LARGE LANGUAGE MODELS (LLMs)
We utilized a large language model to enhance the quality of this paper. Its applications included refining the manuscript for clarity, conciseness, and grammatical correctness.
C MISSING PROOFS
C.1 PROOF FOR THEOREM
Proof. The proof relies on the log-derivative trick (VoTg = Ty Vo 1og The gradient of J. log E Byyna (rl)) is:
VoJa VolEyaao le (r())
Ey (e Vo Volog (v(lr)
(Log-derivative trick)
E Erywor (r())
Eaynnto ( (r(''))
(r()))
= EYnTT
V log To())
Ey (r(y)))
Here, y' is a dummy variable for the inner expectation. Subtracting the baseline 1/3 from the advantage term gives the final form in Eq. ( which is an unbiased estimator with reduced variance.
C.2 THEORETICAL ANALYSIS OF -IS-STENE POLICY GRADIENT
In this section, we provide the formal version of lemmas in Sec. 3.2 and the detailed proofs.
We study the general cases without restricting the uniqueness of the optimal arm. Recall that we consider the bandit setting with K actions A == {ai,..., ax). We will use I = {i [K]]r(a)) = max r(a,)} to refer to the collection of indices of all optimal actions. With a bit abuse of notation, we denote (I*) = Σiez- (aa) be the total mass of on optimal actions.
We consider the softmax policy To parameterized by = 101, 0x] E R e
Vi [K], no (as))
EjE(keeP ""
16

---

Published as a conference paper at ICLR 2026
Following the notation in Sec. 3.2
denote 0: := 101...., [x......]] RK and 68 : 10 ..... 0g R
to be the parameters after performing one-step standard PG and risk-sensitive PG on 8, respectively. Combining with the policy gradient theorem for softmax policy in (Mei et al. 2020h. implied by Eq. ② and Eq. (6), elementwisely, the updates of the policy parameters follow:
Vi E [K],
d, 8 (TT A (Fa);
8, レ 0º + ato(o:) (a:),
(10)
where a > denotes an arbitrary and shared learning rate.
For simplicity, we use (a():) and A, == A (a:) as short notes of policy and advantage values regarding 0, and use क. == (aa))) and मे = "ga (a:) as the short note of policy value w.r.t. the parameters after being updated. Similarly, T2. Ry- and # denote the total policy density assigned the set of optimal actions.
By Eq. 9 the dynamics of i and # follow:
CitoxiA:
c
1T: E5e0ttanAA) Ej e0, t A
E, mr A, A,
(11)
Remark 5. Note that min r(a:) max r(a;) trivial case where every action optimal. We only focus on cases where min r(a:) max r(a;).
Lemma 6. [Formal Version of Lem. 2 As long as 3i' [K] satisfying max; r(a:) > r(ar) > min; r(ai), there exists 0 (or equivalently, Te), s.t., 7z. < TI or even, Tú < Ti for any i E I' and any learning rate > O.
Proof. The proof is by construction. By Eq. 11 7. n. as long as E3 T An > 1. By the convexity of exponential function,
Σn Tejenm and 2 e (E, -,,)),,;))
and all we need to do is to construct s.t. the RHS above is larger than 1.
For convenience, denote i :- arg min r(a;) to be (one of) the worst action(s), and denote /' be (one of) the second optimal actions such that r(az) max{e[K]]z== r(a;). Besides, we denote Tmax := max; ra,)) and Tmin == min; r(a;) the maximal and minimal policy values, respectively.
Note that,
Av = r<=)(a)) - Σ n,r(a,) > n; (r(az) - rminn) TI- (Fmax - -((a))).
KE[K]
Consider an arbitrary policy satisfying the following constraint:
r(ax) Tiin
0 < TI
Tmax r(ay) mT-
(12)
which implies A, > ,,, (r(ax) Tmin).
Since A, E 11, under the constraints by Eq. (12 and that Vi, > and E, n. 1, long as
Ty 2 1. for any optimal action index E I", we have:
Σ - A. ni=-
5" - (r(az) rmin) TT= Σ
(13)
j
We consider the following constraints, which are feasible when my is large enough and n;- is appropriately chosen:
Tt- (r(ar) -mmia),
x 16
Σ
2 -
(r(az) -mmin).
16
J jti
17

---

Published as a conference paper at ICLR 2026
In this case, conclude Σ, T}A, Ar from Eq. (33 This implies #.- nT for any optimal action di-, and therefore,
Tz. < mI.
Lemma 7. [Formal Version of Lem. 3 For any given v and 0, consider the risk-sensitive update (Eq. 6 Eq. (10), there always exists B, for any 3 > B and a > 0, have = II. > WI-
Proof. In the risk sensitive setting, recall the advantage function takes
er(as)
A Ea, [eSr(as)] 1)
For convenience, we use Ag,; := A (a:) short note.
By Eq. 10), the risk-sensitive policy gradient yields:
m T Tye A.
(14)
Σj
Z
Here z: Ej T Asi denotes a normalization term independent of i.
Now, let's denote i' to be the second optimal action satisfying r(ap) = max r(a:) and
denote A == max; r(a;) - r(ar) > 0 to be its value gap.
Easy to see that, for any i I', As, 0, while for any & I
Ag,i
E Eujara (Br(a,)-Dr(a.) 1) S TI. 88 1).
Therefore, as long as 3 2 log 1, we have Ag, O, which implies
az.
e Aaa T3z
vi E I Aea & Ts.
By Eq. 114 we must have #1. > Tr.
Lemma 8. [Formal Version of Lem. For any given and 6 there exists B, s.t., for any B1 B2 B,
0 < 8 =B for any fixed learning rate >> 0.
Proof. We view Ag,i := AT (as) as a continuous function in B::
1 er((a))
Ag,i = B E Eaasrs [eSr(as) 1) Ea, NTo [eBA,.] 1),
where we use Aj, = r(a;) r(a;) as a short note. By taking the derivative w.r.t. 3, we have:
A =
1
1)
1 E Eay
~To
g2
Es Eagnno [eBd,.]]
3
122
~ AA
(B2
Baynto le BA - Eajwao [11 + 34 (A
32B2 (A g.s]
We first check A'2, for E TT. Since A₁ S for any j. we have:
E²
Buyax ~ [A - Eay [11 + 33 3
= Tzz +
Σ
(1 + 3A j.s BA TI-
jAA
<mz. + 2TTI-
Σ
Te .I- +
Tye 2..22
B Σ
nz Ay
AAA,.500
j:A5,:<0
18

---

Published as a conference paper at ICLR 2026
Now, denote 3' to be the minimal value, s.t., for all [ B> B²,
Σ Tye
(15)
6
and we denote 8 be the minimal value, s.t., for all 3 8".
T2- m
Σ JA
Tye A.
+
Σ
Tyl
(16)
Since the RHS of both Eq. 15 and Eq. (16) independent w.r.t. B, such B¹ and 8" always exists.
Then, for any > max B", 3") J. we have:
71. TI- < 0,
E M le | BayneH (1 3A A
which implies that, although Ag, > 0, it decreases 3 increases. Secondly, we check A'2,; for all the other & I* As discussed in the proof of Lem. when B 2 A log L, we have A3,i < 0 for all + i.. Note that,
Tzz
E2
(e (A5,)) Baywao [1 + 8B (A,
Bo vat
2(
T +
Σ
==
f:Ar1200
(Dropped positive terms Σj:d, The in E2 (elds.))
(e
Σ
Jj
Σ
T + BA
j.
)e
Σ
TT(1 + (Bj
;:A,
f:A,..0
J:A
2(
n
Σ
x, 31 je
Σ
=:
P2
Pg (a2 + 12 S (a b)2 for a, b,b>0 0)
As we can sec, P1 is negative but fixed; for ps. consider Jt == max [ we know 0 Ps S long as Bt. Then, we check P2, which is dominated by e There exists 8tt, s.t., P2 1p + > |pl| + Ps as long as B Z max (Bt fft), which implies Ag,s will stay negative but increasing when 3 is large enough.
Combining all the discussion above, as long as B 2 B = {B', 8" St gt we have:
I A/ < 0, therefore, Ag,: but decreases increases:
II 2/2 > 0, therefore, Ag.i 0 but increases as 3 increases;
Combining with Eq. 14 we have o.. Tr. is decreasing as 3 increases when B2 B.
19

---

Published as a conference paper at ICLR 2026
D DETAILS ABOUT OTHER PASS K OPTIMIZATION
Positive
Negative
Cumulative
0
k=1
10-
Advantage
k=4
km8
-0.5
2
0
0.2
0.4 0.6 0.8
0.2 0.4 0.6 0.8
0.2 0.4 0.6 0.8
Prompt Accuracy
Prompt Accuracy
Prompt Accuracy
(a) Mahdavi et al 2025
Positive
Negative
Cumulative
4
k=1
15
Advantage
kmd
k=8
10
2
5
0
-4
0.6
0.8
0.2 0.4
0.8 Prompt Accuracy
0.2 0.4
0.6
Prompt Accuracy
Prompt Accuracy
(b) Chen et al 2025
Positive
Negative
Cumulative
0.05
k=2
Advantage
k-8
k-16
0.5
0
0.5
-0.05
0.2 0.4 0.6 0.8 Prompt Accuracy
0.2 0.4 0.6 0.8
0.2 0.4 0.6 0.8 Prompt Accuracy
Prompt Accuracy
(c) Walder & Karkhanis 2025
Positive
Negative
Cumulative
3
7.5
Advantage
B=2
-0.25
2
6+4
f=8
B-16
1
2.5
-0.75
0
0.2 0.4
0.8 Prompt Accuracy
0.6
0.2 0.4 0.6 0.8
0.2 0.4 0.6 0.8
Prompt Accuracy
Prompt Accuracy
(d) Risk-Sensitive RL (Ours)
Figure 9: Comparison of advantage estimations across different inference- cineeennee--imeeeeee objective methods under the binary reward setting with N 16. Left Positive: Advantage estimation for positive responses. Middle Negative: Advantage estimation for negative responses. Right Cumulative: Cumulative absolute advantage value per prompt.
We compare different methods in a binary reward setting (i.c., r E (0. For a given prompt that generates N responses, let F [0, be the mean reward and (r)) be its standard deviation.
20

---

Published as a conference paper at ICLR 2026
We define the positive advantage, Aposs the advantage for responses with reward of 1, and the negative advantage, Aneg. for those with reward of O. The cumulative advantage is the sum of the absolute advantage values over all N responses.
Mahdavi et al. (2025) employs reweighting policy gradient in the context of pass@ optimization, under the assumption of binary rewards (0 or 1). The expressions for the positive and negative advantages are follows:
A Aos k(1 f)
(17)
Aneg k(1 - -r)
However, increases and the prompt accuracy F' approaches 1, (1 F) rapidly tends O, causing the positive advantage to vanish, as illustrated in the Cumulative column of Fig. 2
Chen al. (2025) proposes a pass@k training objective, adopting the binary reward assumption. Among the N responses, let Nneg be the number rewarded with 0 and Nps N Nneg be the
number rewarded with 1. Define F and negative advantage values:
The objective defined by the following positive
A pas
(1 (F))
Naw h
(18)
Anog
1 Fk
(Fx))
1-11
Tang et al. (2025) introduces best-of-N training objective and utilizes leave-one-out strategy to reduce the variance of the policy gradient. The advantage A, for this objective is defined as:
A,
max
r(a,
max
r(z,
(19)
T=.1......7
[= NYA
Walder & Karkhanis
(2025) builds upon the work Tang et al (2025) and further generalizes the method to a smoothed maximum objective, where k < N. This generalization involves considering the maximum reward within subsets size k. The policy gradient for this smoothed objective is given by:
Σ Z IET
max
Ui) max r(a.w))) jEI((i)
(20)
jel
(C(1,2....N)
Walder & Karkhanis (2025) and Chen et al. (2025) investigate pass @ass@k k-style objectives estimated by uniformly sampling size-k subsets from the generated responses and computing the probability that subset contains at least correct response. The corresponding advantage is then derived from these subset-based pass @ estimates. For a given prompt, let E [0, 1] denote the empirical fraction of correct responses among the N candidates; then N11 - F) is the number incorrect responses. Selecting > [11 F) guarantees that every size-k subset contains at least one correct response. Consequently, the best-of-k: score equals for every subset; hence, by Eq. 18 and Eq, 20 the advantage collapses to zero and the gradient vanishes. Therefore, the advantage estimates of these methods are not dense; when
N. i.e.. the prompt accuracy exceeds N ), the advantage estimate also becomes zero, confirmed in Fig. 9
As illustrated in Fig. we compare various methods based on their positive, negative, and cumulative advantages (the sum of absolute advantage values) in binary reward setting with N 16. Our approach overcomes two key limitations of prior work. First, methods such as those in (Mahdavi et al.) 2025 Chen et al. 2025) are confined to binary rewards and do not naturally extend to continuous reward spaces. Second, in existing pass @k optimization techniques, the advantage estimate vanishes when the sample accuracy exceeds (1 K) ). This limitation is highlighted in the "Cumulative" column Fig.9) where the magnitude of the advantage estimate, which dictates the optimization weight, drops to zero.
21

---

Published as a conference paper at ICLR 2026
In our comparative analysis of different methods, select hyperparameters to ensure fair compar- ison. For Risk-Sensitive RL, we set 3 2, which strikes a balance between pass@) I and pass@k performance. For the bascline methods, tion in the "Cumulative" column of Fig. 9 where the peak advantage for RS-GRPO with / is approximately 0.2, which aligns with the peak advantage of other methods when & experimental results, as shown in Fig. 6 indicate that the method from Walder & Karkhanis (2025) yields unsatisfactory outcomes. Its advantage estimates are persistently positive, and observe that this absence of negative ad advantage leads to rapid entropy collapse and poor training performance, finding consistent with prior work on the importance of negative advantage (Zhu et al. (025). While other methods (Chen et al. 2025 Mahdavi et al., 2025) achieve pass @ 32 performance comparable to RS-GRPO, their pass@1 performance is substantially lower. This highlights the benefit the denser advantage signals provided by RS-GRPO.
use k 4. This choice is motivated by the observa-
Our
E
ADDITIONAL EXPERIMENTS
E. ENTROPY ANALYSIS
We investigate the connection between entropy changes and Risk-sensitive RL. As shown in Sec- tion 4.3, a larger 3 value typically leads to higher cumulative solve rate on the training set and encourages stronger exploration. Figure 4.3 illustrates the entropy loss dynamics during training. Our findings indicate that while correlation exists, entropy does not consistently increase with larger values. This suggests that entropy loss may be a biased indicator and might not fully capture the extent of exploration. Moreover, we observe a relationship between optimizing the risk-secking objective and an increase in entropy, as evidenced by the lowest entropy levels occurring when 3
8 0
20
Entropy Loss
20
B m 2
Entropy Loss
18
B 4
17
16-
B
14
12-
12
10
10
100
200
300
400
100
200 Steps
300
400
Steps
(a) Qwen2.5-Math- 1,5B
(b) Qwen2.55-ath-77
Figure 10: Entropy Analysis under RS-GRPO Training with Different / Values
E.2 ABLATION OF RESPONSES PER PROMPT
Unless otherwise stated, all reported experiments rollout N 16, i.e., responses per prompt N 16. In this section, we ablate the effect of responses per prompt N on model performance. We on the dapol7K 7k dataset and plot training curves of the average pass @k for k E {1. 2, .2} across five benchmarks (AIME24, AIME25, CMIMC25, HMMT Feb-24, HMMT Feb-25). Because MATH500 contains many problems and computing pass @ 32 is costly, we exclude MATH500 from these training checkpoints. Hyperparameter settings follow Table 5 We keep the total batch size fixed 512 16; thus for N 8 the training batch size 1024, for = is 2048, and for N = 32 the prompt batch size is 256. We also adopt dynamic sampling so that each prompt has neither all-correct nor all-incorrect accuracy, ensuring non-zero gradients. Our results show that RS-GRPO (3 2) consistently outperforms GRPO (3 0) across all choices of N. delivering higher pass @2 through pass@ @3,, which further demonstrates the robustness of our method.
fine-tune
E.3 COMPREHENSIVE PASS @K COMPARISON WITH BASELINES
We train Owen2.5-Math-77 7B on the deepmath 103k dataset and plot the evolution of the aver- age Pass @ (1,...332 32) across five benchmarks (AIME24, AIME25, CMIMC25, CMMMT_eeb24.
22

---

Published as a conference paper at ICLR 2026
73008 1961
2000 1941
PS002- (161
Pacoge 1941
Goog (Wa
%
350
250
100
202
370
250
20
182
209
300
200
320
280
SiMpp
Steps
Steps
Steps
Staps
(a) N=4
RS RS-GRPO
GAPO
12
P05299 (30)
Paizy (4)
Ruszge (44)
Rossg332 (54
PG12204
200
156
200
162
204
216
200
(b) N
GAPO
GRPO
P020 (96)
82 (N4
Podo 24 (N6)
Padle ge 1k
R05822 1961
Nave
376
Steps
Steps
Steps
Steps
Staps
Sieps
(c) N=16
Pozz89 (56)
Pas (54
Russp4 (8)
Pussge (8)
Pusre32 (44)
126
220
150
700
156
250
185
204
166
200
126
280
St550
Stops
Stops
Stopo
Stopo
(d) N=32
Figure 11: Ablation the effect of responses per prompt (N). We vary N 16, 32) while keeping across AIME24/25, the total batch size fixed, and report average pass @ for k trained on lk. RS-GRPO CMIMC25, and HMMT Feb-24/Feb-25 using
(3 2) consistently outperforms GRPO (B 0) for all N, highlighting the robustness of the method.
HMMT Feb25) during training. RS-GRPO uses 3 and Pass @k baselines (Chen al 2025 Mahdavi et al. 2025) use k=4, which is fair comparison discussed in Appendix D Compared
@ methods, only RS-GRPO surpasses GRPO on Pass@ and RS- RS-GRPO shows stronger performance for Pass@ 81..8} against other baselines, highlighting the benefit of its denser ad advantage, which better balances Pass @ and @a
R5-CIPO (Gun)
Mihidei
GRPO
15
20 (5))
Heuugin NN
P50204
%
1
610
610
15
610
610
660
Steps
Stepp
Steps
Steps
Steps
Steps
Figure 12: Comprehensive Pass @k comparison with baselines.
F IMPLEMENTATION DETAILS
Datasets We trained our models using the following datasets from Hugging Face:
12k (Cobbe al. 2021 hi youga mmmtmmath111
23

---

Published as a conference paper at ICLR 2026
dapo17k 77 (Yu et al. 2025 ByteadSsinnhuuu---1///A22-------
deepmath 103k (He et al.
2025b): 2whe99/ 2wwee9/DeepMaceeeee wet//eeeetta111...
For evaluation, we used the following datasets, also from Hugging Face:
MATH500: math-
AIME24: #usgginggaccfft//aimee
AIME25: math-aimaimmee.
HMMT_Feb24: hataa/aaaaaaeeeamaa__eee__e__
HMMT_ Feb25: Lttpi/rrrnnaann_eee_______
CMIMC25: MathArena
Training Details Our implementation is based on the VeRL framework (Sheng et al. 2024). and we utilize vLLM 0.8.5 (Kwon al.) 2023) for our experiments. During reinforcement learning training. we do not apply KL. regularization. The maximum response length (in tokens) varies by model: and Swen2.5-Mathh
3,072 for Qwen2.5-Math- Llama-3. 1-8aaa3.------ntttteee 1-8t-tnsttuct. We use Math_Verifyy as the ground-truth reward model (reward = for conrect answer, 0 otherwise). For every question, we append the string \nPlease reason step by step, and put your final answer within \boxed } as the prompt.
and 8,192 for Qwen2.5-7B. Qwen3-4B, and
Table S summarizzes the hyperparameters employed in our experiments. All the experiments keep these identical. For the experiments in Fig. 4 set for RS ES-GRPO. For the comparison with other @ass@k methods Tab. 2 we set 4 for all pass @k methods and = 2 for RS-GRPO. This comparison is fair, further discussed in Sec.
Table 5: Hyperparameters used in experiments During RL Training.
Hyperparameter
Value
Temperature
1.0
Top-p
1.0
learning rate
x 10
Training prompt batch size
512
Responses per prompt N
16
PPO epochs
PPO mini-batch size
32
PPO dii_
0.28
PPO clip low
0.2
Entropy loss coefficient
KI. coefficient
0
Evaluation Details The MATH500 benchmark contains 500 problems, while the other datasets consist of 30 40 problems each. During inference, set the sampling temperature 1.0 and use top-p value of 0.7. For most benchmarks, we generate N 1024 candidate solutions per problem. However, for the larger MATH500 dataset, computationally feasible. For the training curve metrics recording (like Figure and 5). we set N for MATH500 and N 32 for the other datasets. Thus, the testing pass metric records the average across 6 benchmarks, and the testing pass @ 32 metric records the average across the benchmarks excluding MATH500. We compute the pass@k metric using the unbiased estimator proposed in (Chen et al. 2021)
N 32 ensure the evaluation remains
def pass_at_k(nn
k);
:para n: total number of samples
:param c: number of correct samples
:param kt k in passåk
24

---

Published as a conference paper at ICLR 2026
if < k: return 1.0
return 1.0 mp.ppppr pro.prodii (1.0 - k
ap.arange (n + 1, + 1)
Bandit Experiment Details In the experiments Section we consider bandit setting with 100 actions, denoted as A = {aI..... 0100). We employ softmax policy wo parameterized by 0 E R 100 where 0 0100]. The probability of selecting action a; is given by:
e
To(a;) = 1000 e0,"
for all E [100].
For each stochastic policy gradient update, we set the batch size N 16 and the learning rate to 0.1.
25


Reviews:
[{"review_id": "CCEJkgAowo", "reviewer": "Reviewer_roMf", "summary": "This paper addresses the mismatch between the risk-neutral training objectives commonly used in RLVR (e.g., GRPO) and the risk-sensitive evaluation metrics such as pass@k. The authors propose a risk-seeking RL formulation based on an exponential utility function, leading to a risk-sensitive variant of GRPO, termed RS-GRPO. The paper derives the policy gradient for this formulation and demonstrates its benefits over standard GRPO in both bandit experiments and math reasoning tasks across seven base LLMs, with evaluation over $k \\in [1, 1024]$. An ablation study finds that $k=2$ yields a good trade-off across different $k$ values in pass@k.", "questions": "- Figure 1 lacks clear axis labels and is difficult to interpret.  \n- Please clarify how “dense signal” in Table 1 is defined.  \n- What is the “cumulative solve rate” in the training metric?\n- In Figure 3, the y-axis should be labeled “reward” rather than “reward distribution.”?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly articulates the problem of mismatch between training and evaluation in RL for LLMs and motivates the use of risk-sensitive objectives.  \n- The derivation from exponential utility to the risk-sensitive policy gradient is mathematically sound and well presented.  \n- Section 3 provides an excellent illustrative toy example that clarifies the intuition behind the approach and shows how risk-seeking objectives can encourage better exploration.  \n- The method addresses the vanishing advantage issue for prior pass@k optimization methods.\n- Extensive experiments across multiple base models and math reasoning benchmarks convincingly demonstrate consistent improvements over GRPO baseline.  \n- The paper is well written, logically structured, and easy to follow, which is commendable given the technical depth of the topic.", "weaknesses": "**Lack of formal connection between risk-seeking objective and pass@k:**  \n   The paper repeatedly motivates its approach by the mismatch between risk-neutral objectives and pass@k evaluation, yet it provides no theoretical derivation linking the risk-seeking objective to pass@k performance. While the intuition that risk-seeking encourages exploration is sound, a formal connection would substantially strengthen the argument.\n\n**Lack of formal understanding for the vanishing-advantage analysis:**  \n   Figure 8 shows the “vanishing advantage”. A formal proof or at least a more detailed analytical explanation of why prior works have this issue. \n\n**Fairness of baseline comparison:**  \n   The paper compares to baselines trained with $k=4$ but evaluates at $k=1$ and $k=32$. Since prior works *directly* optimize pass@k, such comparison can be misleading. The authors should either match $k$ values during training or explicitly state that the comparison involves different optimization targets. Additionally, analysis of sensitivity to $N$ (number of samples) is needed, as the claimed robustness of RS-GRPO to vanishing advantage depends on this factor.\n\n**Scope:**  \n   While the method is domain-agnostic as claimed, experiments focus solely on math reasoning. It would be valuable to discuss whether RS-GRPO naturally extends to multi-turn RL in theory and other reasoning domains like coding in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxQZ3JEI6x", "reviewer": "Reviewer_mSfu", "summary": "This paper introduces RS-GRPO, a method to address the \"exploration dilemma\" in RL fine-tuning of LLMs. It argues that standard RL improves pass@1 at the cost of pass@k by overly sharpening the policy. RS-GRPO uses a risk-seeking objective to encourage exploration of diverse solutions, aiming to improve pass@k while maintaining strong pass@1 performance.", "questions": "1. Regarding the comparison with other pass@k methods: Could you comment on the fairness of comparing your single-stage method against baselines [1, 2] that often benefit from a second, pass@1-focused fine-tuning stage? Would applying a similar two-stage approach with RS-GRPO provide a more direct and fair comparison of the methods' full capabilities?\n2. Regarding the continuous rewards: The ability to handle continuous rewards is presented as a significant advantage. Could you provide any experimental results, even on a smaller scale or a synthetic task, to demonstrate the practical benefits of RS-GRPO in such a setting?\n3. The paper's motivation relies on the premise that pre-trained LLMs have a \"sharply peaked\" and \"suboptimal\" initial policy. Could you provide more direct evidence to support this? For instance, an analysis of the policy entropy or showing that the most probable reasoning paths are indeed suboptimal would greatly strengthen the paper's foundation.\n\n[1] Chen, Z., Qin, X., Wu, Y., Ling, Y., Ye, Q., Zhao, W. X., & Shi, G. (2025). Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508.10751.\n\n[2] Li, Q., Xue, R., Wang, J., Zhou, M., Li, Z., Ji, X., ... & Yang, J. (2025). Cure: Critical-token-guided re-concatenation for entropy-collapse prevention. arXiv preprint arXiv:2508.11016.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Applying the established theory of Risk-Sensitive RL to LLM fine-tuning is a well-motivated.\n\n2. Most claims are supported with both theoretical analysis and comprehensive experiments across multiple models and benchmarks.", "weaknesses": "Despite its strengths, the paper has several weaknesses that temper its overall contribution:\n1. Unclear Improvement and Potentially Unfair Comparisons: The central claim of improvement is not consistently strong when viewed in context.\n- If the main goal is to improve pass@k, the experimental results (Table 2, Figure 6) show that RS-GRPO performs comparably, and sometimes worse than, other baseline methods designed specifically for pass@k optimization.\n- More importantly, the comparison to other pass@k methods may be unfair. Other methods like [1, 2] often employ a two-stage training strategy: first, they run pass@k training to enhance exploration and diversity, and then they perform a second stage of fine-tuning to distill this diversity into a strong pass@1 policy. This paper compares its single-stage method against only the first stage of these baselines, which may not represent their full potential, especially on the pass@1 metric.\n2. Unverified Advantage on Continuous Rewards: The paper claims the ability to handle continuous rewards as an advantage, but all experiments use binary rewards, leaving this claim empirically unsubstantiated.\n3. New Hyperparameter: The method introduces a hyperparameter, β, which requires tuning and adds complexity for practical application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iNW8EVjssJ", "reviewer": "Reviewer_aF1e", "summary": "The paper directly addresses a critical issue in Reinforcement Learning with Verifiable Rewards (RLVR) for LLMs. The exploration dilemma, where LLMs get stuck in suboptimal solutions due to their pre-trained biases, is clearly articulated.", "questions": "/", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduction of a risk-sensitive RL framework for LLM fine-tuning.\n2.Theoretical and empirical support for the risk-sensitive approach.\n3. Demonstration of significant improvements in pass@k performance with maintained pass@1 accuracy.", "weaknesses": "1.The paper focuses on mathematical reasoning. Investigating the applicability of RS-GRPO to other OOD tasks is necessary.\n2. While the paper presents an entropy analysis, it acknowledges that entropy loss may be a biased indicator of exploration. Exploring alternative metrics for quantifying exploration in this context could be beneficial.\n3. The paper notes that for some models (e.g., Qwen2.5-7B, Llama3.1-8B-Instruct), RS-GRPO failed to outperform the base model at high values of *k*. Further investigation into the reasons for this behavior could lead to more robust performance across different LLM architectures.\n4. A key limitation is the use of a fixed risk-seeking parameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jno7jfgu9J", "reviewer": "Reviewer_586J", "summary": "This paper proposes a risk‑sensitive RL objective for LLM fine-tuning with verifiable rewards. Concretely, the authors replace the mean-reward objective with the exponential-utility formulation that interpolates between mean and max reward, yielding a simple \"drop‑in\" advantage estimator for GRPO. The paper argues this encourages exploration from sharply peaked pretrained policies and improves pass@k while preserving pass@1. Empirically, the method is evaluated on six math benchmarks and several LLMs, reporting consistent pass@k gains and mostly non‑degrading pass@1. A small bandit study and lemmas provide intuition/theory.", "questions": "1. Did you sweep k for the pass@k baselines and report the best? Why is k=4 the sole setting (Appendix D)? Could stronger tuning reverse the small pass@1 gap you observe in Fig. 6?\n\n2. What are the training FLOPs / tokens and wall‑clock for RS‑GRPO vs GRPO? How do gains scale with fewer rollouts (smaller N) or shorter training?\n\n3. You set KL coefficient to 0 (Table 3). Do results hold with a modest KL (as in many RLHF pipelines)? How sensitive are the conclusions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear, readable paper with helpful figures. The exposition around the exploration dilemma and how the exponential utility shifts gradient mass toward hard prompts is crisp. The overview figure and the derivation culminating in Equaqtion 7 are easy to follow.\n\n2. RS‑GRPO only alters the advantage computation, making it nearly \"plug‑and‑play\" for GRPO pipelines. This is appealing for practitioners.\n\n3. The study covers multiple LLMs and benchmarks; pass@k curves across k=1..1024 are shown in Fig. 4, and $\\beta$‑ablations (Fig. 5) help quantify the trade‑off between exploration and convergence.\n\n4. Lemmas in section3.2 show that, in a bandit, risk-neutral PG can reduce probability on the optimal arm (Lemma 2), whereas risk‑sensitive PG increases it for sufficiently large $\\beta$ (Lemma 3), with diminishing gains if $\\beta$ is too large (Lemma 4). The toy bandit (Fig. 3) visually supports this.", "weaknesses": "1. Limited novelty. The exponential-utility objective and risk‑sensitive policy gradients are classical (e.g., Howard & Matheson; Mihatsch & Neuneier), and the paper essentially instantiates that objective as a reweighted advantage for GRPO (Eq. 7-8). The algorithmic delta is small and conceptually straightforward---primarily a different per‑sample weighting. Theoretical results are restricted to a bandit setting; there is no analysis in sequence‑generation or RLVR settings beyond intuition. This makes the contribution marginal.\n\n   - Table 2 (p.8) shows RS‑GRPO usually adds ~+3–5 points in pass@32 over GRPO (sometimes less), while pass@1 is often flat and occasionally lower. For instance, on Llama‑3.1‑8B‑Instruct AIME24, pass@1 drops from 9.9 (GRPO) to 9.4 (RS‑GRPO); on AIME25 it drops from 1.2 to 0.6. The Qwen2.5‑Math‑1.5B MATH500 pass@1 also decreases slightly (80.2$\\rightarrow$78.1). These are small but non‑negligible regressions.\n\n2. The comparison to pass@k optimization baselines uses k=4 for those methods and $\\beta$=2 for RS‑GRPO. Although the paper argues this equalizes the advantage magnitude (Fig. 8), the mapping between k and $\\beta$ is not unique. Baselines might benefit from a small hyperparameter sweep; it is unclear if they were tuned comparably to RS‑GRPO.\n\n3. There are some design choices that look unnatural to me.\n\n   - No KL regularization is used during RL (Table 3). This can lead to policy drift; evaluating safety, verbosity, or formatting regressions would strengthen the case that risk‑seeking exploration is benign beyond accuracy.\n\n   - Risk‑seeking emphasizes tail rewards and may amplify false positives from the verifier. There is no experiment where the reward is intentionally perturbed or partially noisy to test robustness.\n\n   - Many benchmarks use N=1024 samples per problem at evaluation. It would help to quantify gains at small k (e.g., k in {4,16}) under fixed compute budgets and to report training FLOPs/tokens; the training curves (Fig. 5-6) stop at a few hundred steps but do not translate to compute cost.\n\n4. There is a gap between the theory statements and experiments. The claims are theoretically justified only for bandits and do not consider sequence-level credit assignment, KL penalties, or importance sampling present in practical RLVR. Extending the analysis (even approximately) to token-level updates or to a simple MDP with stochastic rewards would increase significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)