No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
Authors: Thanh-Long V. Le
Myeongho Jeon
Kim Vu
Viet Dac Lai
Eunho Yang
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
reinforcement learning with verifiable rewards
llm reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: Zero-Variance Prompts is a valuable source of learning signals for RLVR to improve LLM Reasoning.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful framework for improving the reasoning abilities of Large Language Models (LLMs). However, current methods such as GRPO rely only on problems where the model responses to the same input differ in correctness, while ignoring those where all responses receive the same reward — so-called zero-variance prompts. In this work, we argue that such prompts are not useless but can, in fact, provide meaningful feedback for policy optimization. To this end, we introduce Reinforcement Learning with Zero-Variance Prompts (RL-ZVP), a novel algorithm that extract learning signals from zero-variance prompts. RL-ZVP directly rewards correctness and penalizes errors even without contrasting responses, modulating feedback with token-level characteristics to preserve informative, nuanced signals. Across six math reasoning benchmarks, RL-ZVP achieves significant improvements of up to 8.61 points in accuracy and 7.77 points in pass rate over GRPO, while consistently outperforming other baselines that filter out zero-variance prompts. These results highlight the untapped potential of learning from zero-variance prompts in RLVR.

Published as a conference paper at ICLR 2026
No PROMPT LEFT BEHIND: EXPLOITING ZERO- VARIANCE PROMPTS IN LLM REINFORCEMENT LEARNING VIA ENTROPY-GUIDEI ADVANTAGE SHAP-
ING
Thanh-Long V. Le', Myeongho Jeon', Kim Vul, Viet Lai", Eunho Yang'
KAIST
EPFL "Adobe Research
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) powerful frame- work for improving the reasoning abilities of Large Language Models (LLMss). However, current methods such as GRPO rely only on problems where the model responses to the same input differ in correctness, while ignoring those where all responses receive the same reward - so-called zero-variance prompts. this work, we argue that such prompts not useless but can, in fact, pro- vide meaningful feedback for policy optimization. To this end, we introduce Reinforcement Learning with Bero-Variance Prompls (RL-ZVP), novel algo- rithm that extract learning signals from zero-variance prompts. RL-ZVP directly rewards correctness and penalizes errors even without contrasting responses, mod- ulating feedback with token-level characteristies to preserve informative, nu- anced signals. Across six math reasoning benchmarks, RL.ZVVP achieves sig- nificant improvements of up to 8.61 points in accuracy and 7.77 points in pass rate over GRPO, while consistently outperforming other baselines that filter out zero-variance prompts. These results highlight the untapped potential of learning from zero-variance prompts RLVR. The project page is available at https: //bltnnnkk.githu...... cttaauu.oo/////oo http://w......a.....ooooo..ttta.
Objective
RL-ZVP
losl
min
(ia10) A.s. clip(ra.(0), 1- 1 Ass)
70
69.2
65.3
Å.. Rt mean (1:
std( (
Acc@ia (96)
Zero-Variance Prompts: std((R.) =0
50.2
50
GRPO
RL-ZVP
45.6
A
R. >
Azvr(Ora)
(aHH,
R
0
40
if 0
max lol
(Ha))
Has R,
1.78
88
Figure I: Left: RL-ZVP employs an entropy-guided advantage shaping formula to extract learning signals from vero-variance prompts. For ton--zero-varinnee prompts, it reverts the standard GRPO formulation. Right: RL-ZVP demonstrates significantly higher average accuracy than GRPO on both Qwen3-1.7B--ase and Qwen3-8B-Ban across six math reasoning benchmarks.
Correspondence t thanhlong07800e ong07800ka kr, dacla @adobe.com

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent frontier models such as OpenAl ol (Jacch et al., 2024).
Rollaut Generation Time (%%
Claude 3.7 Sonnet (Anthropic, 2025) DeepSeek-RI (Guo et al.
50
2025), and Kimi k1.5 (Team et 2025) provide compelling proof
of concept for leveraging long chain of thought (Wei et al. 2022
48
M
Kojima et al. 2022) to enhance the reasoning capabilities of Large
Language Models (LLMs). Following these advances, Reinforce- ment Learning with Verifiable Rewards (RLVR) techniques such
44
Group Relative Policy Optimization (GRPO) (Shao et al. (2024) and Dynamic Sampling Policy Optimization (DAPO) (Yu et al. 2025
42
have emerged as the de facto standard for strengthening LLM rea- soning. In RLVR, the policy model is trained by sampling mul- tiple responses (rollouts) to a prompt and then reinforcing correct answers while penalizing incorrect ones. While this approach is ef- fective when responses for the same prompt contains both right and wrong responses, it does not account for scenarios when group of rollouts are entirely correct or entirely incorrect (with identical rewards), referred to as zero- variance prompts (Zheng al., 2025). In such situations, the normalized rewards (i.e., advantage values) collapse to zero, resulting in training signal, rendering all rollouts for those prompts un- informative. While it may seem conceptually reasonable to ignore these prompts, doing so remains problematic because rollout generation itself is major bottleneck in RL training (Zhong et al.) 2025 Noukhovitch et al.] 2025, Sheng et al. 2025). ). Our preliminary experiments show that re- sponse sampling typically takes up about 50% of the total time per training step, which emphasizes the significant throughput overhead introduced by rollouts (Figure 2)
150
300
450
600
Gradient Steps
Figure 2: Rollout generation overhead as percentage of total time of the training step.
Recent methods acknowledge the inefficiency of zero-variance prompts and attempt filter them out, either pre-rollout (Zheng et al. 2025] Quu et al. (2266) or post-rollout (Yu et al. (2025) However, to the best of our knowledge, no prior work has investigated whether zero-variance prompts themselves can serve as a valuable source of learning signals for policy optimization.
In this paper, we challenge the prevailing consensus regarding zero-variance prompts and the com- mon practice of filtering them out. We argue that, with an appropriate strategy, sero-variance prompts can still provide meaningful learning signals that benefit policy optimization during LLM training. We begin by analyzing the objective formulation of GRPO and identify the advantage van- ishing issue that prevents zero-vare vrr-vvaiaecee prompts from contributing to learning. Specifically, GRPO imposes implicit constraints on the policy optimization process: for given prompt, the policy model is rewarded for correct responses only when at least one incorrect response exists within the group, and penalized for incorrect responses only when there is at least one correct response in the same group. Based on this observation, we develop an intuition for the high-level role of advantage in guiding token-level optimization, and how its interaction with response correctness influences the learning dynamics of language models.
Building these insights, we introduce RL-ZVP (Reinforcement Learning with Prompts), a novel algorithm that extracts useful learning signals from zero-variance prompts to im- prove the effectiivnness of RL training. Our approach rests on two key intuitions: (i) the policy model should still be rewarded for correct answers even if no incorrect answers are present in the same group (and penalized analogously), and (ii) the degree of reward penalty should be determined by the characteristics of sampled tokens, inspired by recent advances in reward shaping (Wang) al., 2025 Devidze al., [022,] Ng et al.) 1999). Concretely, we formulate the advantage for varianar prompts around two properties: direction, which aligns with the correctness of the response; and magnitude, which quantified by the entropy value of each token within the response (Figure D].
Experiments on six math reasoning benchmarks across two model scales - Qwwn3-1.7B-B.. 1.B--Base and Qwen-8B-Base (Yang et al. (025) show that RL-ZVP significantly outperforms GRPO, with an average improvement of 4.00 points in accuracy (Ace@8) and 4.28 points in pass rate (Pass @8), 8). Notably, RL-ZVP achieves gains of up 8.66 points in Acc@8 (on AIME25) and 7.77 points in Pass@8 (on OlympiadBench). RL-ZVP also consistently surpasses other baselines that filter out zero-variance prompts, even under unfavorable settings where these baselines are given 3x-5x
2

---

Published as a conference paper at ICLR 2026
more rollouts. Moreover, across both scales, RL-ZVP demonstrates more stable and consistent learning process. These results establish zero-variance prompts as a valuable resource for learning signals and open perspective on how to utilize them, rather than discarding them, in RLVR.
2 PRELIMINARY: GROUP RELATIVE POLICY OPTIMIZATION
Group Relative Policy Optimization (GRPO) (Sha et al. timization (PPO) (Schulman et al. 2017) tailored for LLM post-training. Given a crestion-answer pair (g. a) from a dataset D, group of G responses (rollouts) {or is sampled from the old policy Teas (typically the policy from the previous training iteration). Each response compared against the ground-trun -rrut answer a to produce final rewards {R, Instead of relying on a costly value model like PPO, GRPO normalizes these rewards to compute the advantage as:
2024 a variant of Proximal Policy Op-
R, mcan ( (R.
where t denotes token position o₁. Note that the advantage A, computed at the response level and subsequently broadcast to all tokens the response, i.e., A., Å ...== A.. Aq. The policy objective for single prompt then defined as:
0. min (0) Ar,, clip(ra,(0)) 1 1+ A.t)
JARPO (0 I a.a, (or)(=))
1
(2)
T 1=1
To(one 1 o(,cr) Te (o,, q, 01,4t)
where
(0)
Here, (f) represents the importance sampling ratio that adjusts the objective account for the off- off-policy distribution of the training batch. The final GRPO objective for the whole dataset is:
(O, D) = -B(ga))D, (. ( (11) [JARPo(O)s: (ot)
REINFORCEMENT LEARNING WITH ZERO- VARIANCE PROMPTS
In this section, introduce our algorithm, Reinforcement Learning with Vero-Variance Prompts. We begin by motivating the approach through the advantage vanishing problem in zero- vero-nariance prompts (Section[3.I) We then present our entropy-guide -uriddd advantage formulation (Section 3.2 and
conclude with the full training objective and algorithm (Section 3.3
3.1 MOTIVATION: ADVANTAGE VANISHING IN ZERO- VARIANCE PROMPTS
During GRPO training, the policy model may encounter questions
100+
Zero-Variance Ratio (%)
Qwen3-88B-Base DAPO-Math-17k
where all sampled responses in group are either entirely correct entirely incorrect, resulting in identical rewards. Following Zheng
et al. (025). call these cases aero-variance prompts. They typ- ically arise in two situations: when the dataset is either (i) too difficult or (ii) too easy for the model. From training perspec- tive, zero-variance prompts appear most frequently (i) at the begin- ning of training, when the model often fails solve problems even once across multiple responses, and (ii) near convergence, when the model has become strong enough solve many questions entirely.
75+
Qwen3-1. 1.-B-a MATH
50+
25
100
300
500
Gradient Steps
To understand the prevalence of gero-variance prompts, we exam- trained on MATH. and Qwen3- 8B-Base trained on DAPO-Math- 17k. We found that zero-vas variance prompts account for roughly 30% to 99% of each batch during train- ing (Figure 3).. This high proportion suggests that if these prompts could be leveraged effectively, they potentially represent an untapped source of useful learning signal for the policy model.
Figure 3: The ratio of zero- variance prompts across two experimental scales.
ine two settings: Qwen3-1..7-BB
We omit the KI. penalty here for clarity.
3

---

Published as a conference paper at ICLR 2026
Remark 1 (Advantage Vanishing). In Equation ① and 2 if all responses within a group receive identical rewards, their advantages degenerate to zero. In this case, the GRPO objective reduces to Jearo Pup: a, (o,))=))) 0, effectively discarding all the potential learning signal.
From Equation
2p. taking the gradient of both sides gives:
G
1od
1
V VoJaRro(q, (o.1)=1)
(0) As, Vo oo(o q, oc,ct)-
0
From this expression, the gradient of the GRPO objective with respect to the policy parameters can be interpreted as weighted sum of the log- probbabbiyyy gradients for all tokens in response. Intu- itively, GRPO raises the probabilities of tokens in correct responses while lowering those of tokens in incorrect ones. Under GRPO advantage formulation, however, if we ignore the importance-sampling ratio, each token in a response receives the same weight, regardless of its role or informativeness.
Remark 2 (The Role of Advantage). The sign of the advantage År, determines the direction of the gradient update for each token since the importance sampling ratio Ti in Equation 4 is strictly positive, while its absolute value [Ai,l governs the magnitude of the update in conjunction with Ti,
Recently, several methods have excluded zero-variance prompts from RL training Yu et al. 2025 Zheng et al., 2025 Cheng et al.) 2026), viewing them merely way to reduce rollout cost and implicitly assuming they offer no value for policy optimization. We challenge this assumption and argue that, when handled appropriately, zero-variance prompts could provide valuable learning signals for RL training.
3.2 ADVANTAGE FORMULATION
We propose Reinforcement Learning with Zero-Variance Prompts (RL-ZVP), a novel algorithm designed to exploit zero- variivari prompts. The key intuition is to address the limitation noted in Remark M the policy should be rewarded or penalized based solely on the correctness of its re- sponses, even when contrasting answers are present. RL-ZVP behaves identically to GRPO on non-zero- non-zero-variance prompts, but adopts a different strategy for zero-variance ones, thereby general- izing GRPO. We first classify zero-variance prompts with all correct responses as positive prompts and those with all incorrect responses as negative prompts. Based on Remark we frame advantage shaping as developing appropriate scaling factors, with focus on two properties: direction and magnitude.
Advantage direction. The sign of the advantage is straightforward: positive (+) for positive prompts and negative (-) for negative ones. Intuitively, when all sampled responses are correct, the model should increase their probabilities, reinforcing certainty and encouraging exploitation. Conversely, when all responses are incorrect, their probabilities should be reduced, discouraging repeated mistakes and promoting exploration of alternative sampling paths.
Advantage magnitude. Inspired by recent works that incorporate token-level signals to augment verifiable rewards (Cui et al., 2025) Fang et al., [0025] Agarwal et al.] 2025 Deng al.) 2025] Lei et al. 2025). we leverage token entropy in the formulation of the advantage magnitude for zero-variance prompts. The key idea to scale each token's gradient update in proportion to its entropy. For correct responses, high-entropy tokens
typically those crucial for reasoning, such as connectors or branching points Wang et al.) 2025) - receive larger updates than semantically trivial tokens. This encourages reasoning behaviors such as reflection and verification (Guo al.) 2025), while avoiding over-exploitation of trivial text completion patterns. For incorrect responses, the probabilities of all tokens reduced, but high-entropy tokens (that initiate reasoning paths and behaviors) are penalized less severely, preserving flexibility for the model to revisit and explore those reasoning paths in future iterations.
We omit the clipping factor before gradient derivation for clarity.
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: Reinforcement Learning with Zero- Variane Prompts
Input:
initial policy model тe; training dataset for deration ..... M do
Update the previous policy Now <- mp;
Sample (w/o replacement) a prompt batch B D;
foreach (q, a) do
Sample G outputs {o;};-1 Tew (+1
Compute rewards (R, G { by verifying each o; against o;
for gradient step 1, ........ do
Sample (w/o replacement) a mini-batch Buini C B;
foreach (g, a, {R] E Bm do
if (t ( [ > 0 then compute advantages {Ai.) using Equation 1
else compute advantages {Azv)(o,,)) using Equation 5
Update the policy model To by maximizing the RL-ZVP objective
Output: final policy model IT
Ad vdvantage formula. Combining these design choices, our final token-level advantage formu- lation for zero-variance prompts, replacing the zero assigned in GRPO, is:
if std( = A R, 0, Azvp(ora)= = -a max loil (Hir.) - Has). if std([ 0 A R. S0
aHH
(5)
IV] with ---=-x Toluy I q 0z,ee) log [To(Us) g. o1,4))
3=1
where a denotes the scaling factor hyperparameter, and H: is the entropy each token 0s, in the Jo.l (H.
response o given the policy model's vocabulary set V {or, U2, ...). The term max reverts the effect of token entropy in negative prompts to align with our intuition: tokens with higher entropy incur smaller penalties, while low-entropy tokens are penalized more.
3.3 FINAL OBJECTIVE
Let Dzve be the set of veri-neriance prompts encountered during training (Dzvp C. D). Incorporat- ing our advantage shaping for zero-variance prompts, the RL-ZVP objective becomes:
t D 1 E!'l min (0).(() Azvp(o,,), C..
JRL-ZVe(Olq.a (orl
clip(r.,(O), 1 e, 1 e) Aevr(ou.)). if D2ve, (6)
JaRro a, (o.1
otherwise.
Remark 3 (Relationship GRPO). RL-ZVP does not introduce any loss term beyond the GRPO objective. It simply reformulates the special case where the prompt is cero-variance. In fact, the GRPO objective from Equation (2) can be equivalently expressed in branching form:
(o. 0.
if DzvP,
( g. a. (o))
[o q, ( otherwise.
In this regard, our formulation serves as generalization of GRPO-reducing exactly to GRPO when prompts non-zero-variance while resulting in non-trivial update for zero-vuriance prompts rather than collapsing to zero.
In the implementation, detach every occurrence of His from the computational graph that it acts as scalar factor rather than differentiable parameter.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
In this section, we first outline our experimental setup, including models, datasets, and implementa- tion details (Section 4.1). We then present validation results and analysis, comparing RL-ZVP with GRPO and prompt filtering baselines (Section 4.2) Finally, we conduct ablation studies to assess the contribution of each sub-component RL-ZVP (Section 4.3
4.1 EXPERIMENTAL SETTINGS
Models & Datasets. We conduct experiments on both Qwen3-1.7B-Bas and Qwen3-8B-
Base Yang et al. 2025 Liu et al. 2026) to assess performance across model scales. For Qwen3- 1.7B-Base, train on the MATH dataset (Hendrycks et al. 2021 Lightman et al., (024). which we adopt the more challenging DAPO-Math-17K dataset YYu al., 2025) better exploit the capacity of larger models. Within the scope of this work, we refer to these two setups small scale and large scale, respectively. We then evaluate the trained models on six standard math reasoning benchmarks: Minerva, AMC23, MATH (Hendrycks) et al. 202T), AIME24, AIME25, and OlympiadBench (He et al. 2024). For each question, we gen- erate eight responses and report both the accuracy (Ace@8) and the pass rate (Pass @8). To further evaluate the generalization of RL-ZVP, we also report additional experimental results in Appendix
contains 7.5k problems. For Qwen3-8B-Bass
Baselines. Since RL-ZVP the first method that attempts to explicitly leverage zero-variance prompts, we compare it against GRPO (Shao et al.] 2024) and two representative methods that filter out zero-variance prompts: GRPO with Dynamic Sampling (GRPO-DS) (Yu et al.) (025) and GRESO (Zheng et al., 2025). GRPO-DS repeatedly samples questions, generates group of responses, discards sero-variance cases, and constructs batches containing only non-zett non-zero-variance prompts, ensuring that every prompt contributes to the gradient signal. In contrast, GRESO predicts and skips vero-variance prompts without responses sampling, using a probabilistic filtering strategy based historical training dynamics. For GRPO-DS and GRESO, settings: (r) using the same number of rollouts as RL-ZVP for a fair comparison, and (g) using the same number of gradient steps. The latter favors the prompt filtering baselines, allows roughly 3x-5x more rollouts than RL-ZVP and GRPO (Table I
report results under two
Table 1: Number of rollouts and gradient steps of RL-ZVP compared to the baselines. For GRPO-DS-r and GRESO-r on the large scale, the policy initially solves only ~2% of problems per training batch. This leads to extremely poor data efficiency, requiring over-sampling of ~500 to form full con-zero-variance batch for each gradient step. As a result, these methods achieve far fewer gradient updates than GRPO and RL-ZVP, despite using nearly 1.5x more rollouts. This highlights the severe inefficiency of dynamic sampling on challenging training data.
Small Scale
Large Scale
Method
Rollouts (xGRPO)
Rollouts (xGRPO)
#Rollouts
Grad Steps
#Rollouts
Grad Steps
GRPO
172,032
1.00x
672
540,672
1.00x
1,056
GRPO-DS-r
172,032
1.00x
320
827,392
1.53x
32
GRPO-DS-g
421,888
2.45x
672
2,859,008
5.29x
1,056
GRESO-r
173,600
1.01x
320
815,232
1.51x
32
GRESO-g 272,030
1.58x
672
2,159,104
3.99x
1,056
RL-ZVP
172,032
1.00x
672
540,672
1.00x
1,056
Implementation Details. All experiments implemented using the verl framework (Sheng et al., 2025), following the standard hyperparameter settings from recent works (Yu et al., 2025 Zheng et al.) 2025). We train with a batch size of 512 and mini-batch size of 32, resulting in 16 gradient steps per training iteration. Full implementation details are provided in Appendix
4.2 RESULTS & ANALYSIS
RL-ZVP achieves improvements of up to 8.61 points in accuracy and 7.77 points in pass rate GRPO. We present main results in Table 2 and Figure Across six benchmarks, RL-ZVP

---

Published as a conference paper at ICLR 2026
Table Detailed evaluation results on six math reasoning benchmarks. The best and second best performance across all settings are bold and underscored, respectively. Under a fair setup of equal rollout budget, RL-ZVP achieves the best performance across all datasets, heavily outperforming GRPO. (*) Even in the unfavorable setting where Dynamic Sampling baselines use 3 x 5x more rollouts, RL-ZVP still outperforms them on the majority of benchmarks.
Minerva
AMC23
MATH500
AIME24
AIME25
OlympiadBench
Method
Acc@8 Pass
Acc@ Pass
Acc@
Pass@8
Acc@8
Pass
Ace@8
P
Acc@8 Pass 688
Qwen3-1.
1.m.3-1.7-BBB
1.7B-Basc trained with MATH (small scale)
GRPO
29.09
43.19
42.19
72.00
69.09
84.05
8.75
17.24
4.17
6.67
33.20
50.52
GRPO-DS-r
29.96
44.02
42.50
73.84
68.31
83.04
7.50
17.41
2.92
8.73
32.67
49.89
GRPO-DS-g*
29.96
73.84
7.50
7.50
12.95
35.68
53.03
GRESO-r
27.99
43.39
40.00
70.25
67.53
83.07
7.08
18.48
3.75
12.89
31.30
49.13
GRESO-g"
30.19
43.39
45.00
73.86
69.77
84.77
9.58
19.51
5.00
11.00
34.68
52.43
RL-ZVP (Ours)
29.96
44.76
48.75
77.19
70.98
87.35
12.50
23.99
6.25
13.23
35.11
54.84
AwwEt GRPO)
+3.75
+6.75
+2,08
+6.56
+4.32
Qwen3-8B-- BaBe trained with DAPO-Math- 17k (large scate)
GRPO
44.99
58.42
73.75
92.48
83.00
92.40
28.33
50.64
20.00
31.59
49.59
66.20
GRPO-DS-r
29.96
50.85
47.19
76.70
68.20
87.26
12.08
31.33
9.17
22.02
43.88
58.16
GRPO-DS-g*
44.76
59.60
76.25
92.43
84.98
92.42
27.08
42.72
19.58
35.93
44.83
60.42
GRESO-r
30.74
50.53
49.38
80.53
67.40
87.79
12.08
29.31
10.83
26.26
44.92
$9.66
GRESO-g"
45,50
58.69
78.13
92.90
83.23
92.24
25.00
41.80
17.92
32.12
44.14
59.79
RL-ZVP (Ours)
45.96
58.83
78.75
94.18
89.73
96.13
33,33
54.66
24.58
39,36
$8,20
72.28
A(w.r GRPO)
+0.97
+0.41
+5.00
+1.70
+6.73
+3.73
+5.00
44.02
+4.58
+7.77
+8.61
+6.08
consistently delivers significant gains over GRPO, ranging from 0.87 to 6.75 points on Qwen3- Base and from 0.41 to 8.61 points on Qwen3-8B Base. On small scale, the average improvements are 2.84 points in Ace @8 8 and 4.62 points in Pass while on large scale, the average gains are 5.15 points in Acc@8 and 3.95 points in @ These results indicate that leveraging zero-variance prompts provides strong and informative learning signals that substantially enhance reasoning per- formance, with improvements generalizing across both model scales and datasets.
RL-ZVP consistently outperforms prompt-fittering baselines, even under visadvantageous set- tings. Under the fair setting with the same rollout budget, RL-ZVP substantially surpasses both GRPO-DS-r and GRESO-r across all benchmarks, achieving average gains 3.28 points Acc@8 and 4.03 points in Pass @ the small scale, 19.20 points in Acc@s 8 and 13.56 points in Pass @8 on the large scale (Figure 4. Even in the less favorable setting with the same number gradient steps RL-ZVP still heavily outperforms GRPO-DS-g and GRESO-g on the majority of benchmarks, with average improvements (over their best performance) of 0.99 points Acc@8 and 1.19 points in Pass@8 on the small scale, and 5.51 points in Acc@8 8 and 5.32 points in Pass@ @8 on the large scale. These results not only reaffirm the importance of learning signals from zero-vare variinae prompts, but also provide a new insight: lever-
where the baselines require 3 more rollouts [Table
GRPO
GRPO-DS-g GRESO
GRESD-g
GRPO
GRPO-DS-g
GRESO-3
GRPO-DS-T
RL-ZVP
GRPO-DS-r
GRESO
RL-ZVP
75
70-
50
65
Acci8 (%)
45-
60-
11-7
40-
55
35-
32.1
30.9
50-
66.6
90.2
14.
25
40
Qwen3-1.77 Wwen3-1.78-Base
Qwen3-8B- B-en3-8B-BBse
Qwen3-1.78-Baal
Qwen3-88-Baan
Model
Model
(a) Accuracy
(b) Pass rate
Figure 4: Average accuracy (a) and pass rate (b) on six math reasoning benchmarks. RL-ZVP consistently delivers the strongest performance among all baselines.
7

---

models exhibit Published as a conference paper at ICLR 2026
aging such prompts brings greater benefits than discarding them, directly challenging the prevailing practice adopted in recent works.
Response Length
AccBB (%)
RL-ZVP
1.00
1250
45-
0.75
1000
0.50
40-
750
0.25
200
400
600
800
200
400
600
800
Gradient Steps
Gradient Steps
Gradient Steps
Acc@8 8 (small scale)
(b) Entropy (small scale)
(c) Response Length (small scale)
50
1.00
Response gen
3000
www
Acc@8 (%6
40-
Entropy
0.75
2000
K30
1000
0.25
200 400 600 800
1000
200
400
600
800
1000
200 400
600 800 1000
Gradient Steps
Gradient Steps
Gradient Steps
(d) Acc@8 (large scale)
(e) Entropy (large scale)
(f) Response Length (large scale)
Figure 5: Validation accuracy and training dynamics at different experiment scales. Each row shows Acc@8, entropy, and response length during training for Qwen3-1.7B-BB (top) and Qwen3-8B-Base (bottom). RL-ZVP exhibits more consistent and stable trends than GRPO.
RL-ZVP demonstrates more stable and consistent learning process. Figure present the validation accuracy and learning dynamics across benchmarks of RL-ZVP and GRPO as training progresses. On 1.7B scale, RL-ZVV initially shows competitive results with GRPO, but after roughly 500 gradient steps, it achieves a clear performance boost, creating a significant performance gap. On 8B scale, the difference is even more pronounced: RL-ZVP exhibits smooth and steadily improving curve, while GRPO suffers sharp drop during the first 200 gradient steps, then recovers and but eventually underperforms RL-ZVP. This highlights RL-ZVP's superior effectiveness and stability, which avoids the performance collapses observed at the start of training with GRPO on large scale.
We attribute this robustness the quantity and quality of learning signals each method receives at different training stages. In small scale experiments, the number of zero-variance prompts increases as training progresses (Figure 3. The model improves and produces more positive prompts (all- correct responses), which RL-ZVP can leverage as useful signals, while GRPO discards them. This leads to a widening gap in later stages of training. For large-scale experiments, the situation is reversed at the start of training: the model initially solves fewer than 5% of problems per batch (Figure 3). leaving GRPO with sparse and noisy feedback. In contrast, RL-ZVP can still exploit the abundance of negative prompts (all call-incorrect responses), obtaining denser and thereby resulting in greater training stability in the early stages. Although the proportion of zero- varoinarancce prompts decreases as training continues, it remains non-negligible, sustaining huge advantage for RL-ZVP over GRPO. Overall. across both scales, RL-ZVP consistently benefits from the additional signals provided by zero-variance prompts, ensuring stronger and more stable performance than GRPO.
RL-ZVP encourages longer and more confident reasoning across model scale. As training pro- gresses, RL-ZVP steadily reduces entropy while producing increasingly longer responses. Both trends align with significant gains in validation results (Figure 5) These patterns, along with the qualitative demonstrations (Appendix]s [F). suggest two complementary effects related to the policy model: (i) enhanced capabilities due to the emergence of long-form reasoning behaviors the so- called aha-moment (Guo et al.) 2025 Jin et al. 2024) and (ii) greater confidence exploratory
decisions, reflected in the low entropy (Cheng et al.) 2026 Kadavath et al. 2022).
On the small scale, models trained with RL-ZVP show a significant increase in response length over training. indicating the ability to generate extensively long chains of reasoning that greatly improves its reasoning abilities (Jin et al. 2024 Guo et al. 2025). In contrast, GRPO- minimal growth in response length, aligning with their bounded validation performance and poten- tially their inability to generate multi-step, well-structured solutions. On the large scale, entropy initially drops sharply but then stabilizes after roughly 300 gradient steps rather than collapsing.
8

---

Published as a conference paper at ICLR 2026
Combined with the consistent growth in response length and accuracy. this rules entropy col- lapse and emphasizes RL-ZVP's ability to promote detailed and confident reasoning (Cui et al. Wang et al. 2024a).
2025
RL-ZVP facilitates more sophisticated reasoning even with small policy models. In many cases, especially on 1.7B scale, ERann-r models produce simple and direct solutions, jumping straight into problem-solving with little structure. Even when they show glimpses of more com- plex behaviors such as reflection, these behaviors are shallow and disorganized. In contrast, models trained with RL-ZVP display much more sophisticated and systematic reasoning strategies: they restate and reflect on the key hypotheses, carefully interpret the clues, plan solutions step by step, organize their reasoning into major steps with detailed sub-solutions, and reasonably attempt alter- native strategies when encountering dead ends. Illustrative examples are provided in Appendix [
For instance, during exploration, train-t models often rely on shallow trial-and-error heuris- ties (e.g., "let's try another set of values", " "let's try different method"), which resemble second- guessing and lack of direction. In contrast, models trained with RL-ZVP adopt more principled and comprehensive strategies (e.g.. "let's double check", "assume symmetry to simplify the problem". "Iry to factor directly", "let's Iry to express the sides using the Law of Cosines"), demonstrating focused and systematic exploration. Remarkably, such behaviors emerge even models as small as 1.7B when trained with RL-ZVP, whereas GRPO fails to elicit them at this scale. On 8B scale, both GRPO and RL-ZVP produce systematic solutions with clear structure. However, GRPO outputs tend to be repetitive and rigid in tone, while RL-ZVP exhibits more flexible and exploratory style of reasoning, reflecting a deeper ability to search for and evaluate alternative solution paths.
4.3 ABLATION STUDY
To better understand the contribution of each design choice in RL-ZVP, we conduct ablation exper- iments on our small scale setting Sowen3-1.7B-Ba. trained with MATH.
Impact of zero-variance type and entropy scaling. We examine the contribution of three sub- components in our advantage formulation for RL-ZVP: negative prompts, positive prompts, and the entropy scaling factor. Table the performance when each component is removed, compar- ing against full RL-ZVP and GRPO. The results are consistent across benchmarks: full RL-ZVP achieves the best performance in all evaluations. Ignoring either negative or positive prompts yields comparable performance, confirming that both play similar roles in providing learning signals for the policy models. Importantly, removing the entropy scaling factor (e.g., using only sample-level +1/-1 as the advantage) causes the most severe degradation. This emphasizes the importance of advantage magnitude design for zero-variance prompts learning signal: simply promoting correct answers and penalizing incorrect ones is insufficient. Rewards and penalties must also be weighted appropriately to be effective.
Table 3: Ablation on the subcomponents of RL-ZVP. Full RL-ZVP achieves the best average performance, while the second-best results mostly occur when single component is removed.
Minerva
AMC23
MATH100
Average
Method
Acc@8
Pass
Acc@8
Pass @8 8
Acc@8
Pass
Acc@8
Pass @
RL-ZYP (Ours)
29.96
44.76
48.75
77.19
70.98
87.35
49,90
69.77
w/o negative prompts
29.92
43.87
44.69
70.51
68.64
84.40
47.75
66.26
w/o pusitive prompts
29.50
43.88
41.56
70.40
70.70
85.81
47.25
66.70
w/o entropy scaling
28.95
43.02
42,50
70.76
69.19
84.37
46.88
66.05
GRPO
29.09
43.19
42.19
72.00
69.09
84.05
46.79
66.41
Impact a. At the beginning of training, the top 20% of tokens with the highest entropy often take values between and 10, which can result in very large advantages if left unscaled. To avoid explod- ing updates and potential training instability, we introduce a as the scaling factor in the advantage computation. Table shows the performance of RL-ZVP under different a values. We observe that RL-ZVP achieves its best results with a = 0.10 or a = 0.20, while performance drops sharply at both 0.05 and a 0.30. This demonstrates that plays critical role in balancing stability and effective learning: overly small values fail to sufficienlly utilize learning signal from zero-variance prompts, while overly large values may cause unstable training with large gradient updates.

---

Published as a conference paper at ICLR 2026
Table Validation results under different a values. Both excessively small and excessively large values degrade performance. Across all experimental settings, we empirically find that a 0.1 yields the most consistent and best performance.
Minerva
AMC23
Math500
Average
Alpha
8
Pass
Acc@8
Pass @
Acc@8
Pass @8
Acc@8
Pass @ 8
a 0.05
29.60
43.78
42.19
69.44
68.56
82.96
46.78
65.39
0.10
29.96
44.76
48.75
77.19
70.98
87.35
49.90
69.77
a = 0.20
29.60
44.54
43.75
74.14
71.10
86.47
48.15
68.38
= 0.30
28.68
42.61
41.56
71.22
67.20
83.46
45.81
65.76
RELATED WORK
Reinforcement Learning for LLM Reasoning. With the rise of test-time scaling and large reason-
ing models (Muennighoff et al.
2025 Guo et al. 2025). the post-training stage of large language
models has increasingly shifted toward Reinforcement Learning with Verifialle Rewards (RLVR).
However, these methods either discard zero-variance prompts implicitly (Shao et al.) 2024 Liu
iet al. 2025) or explicitly (Yu et al. 2025 Zheng et al.] 2025). This leads to wasted computation and reduced data efficiency during online training rollouts remains major bottleneck in RL train- ing (Zhong et al. [2025] Noukhovitch et 2025). Our work addresses this gap by extracting useful signals from zero-variance prompts, thereby mitigating useless rollout costs.
Online Prompt Selection. Zero-variance prompts have been identified as key weakness of GRPO in recent works (Yu al. 2025 [Zheng al.] [025 Qu et al. 2026). DAPÒ addresses this issue with post-rollout oversampling: repeatedly sampling until batch containing only non-zero-variant
prompts is obtained (Yu et 2025). GRESO improves efficiency with a pre-rollout probabilistic filter, leveraging training history to predict whether a prompt is likely to be zero-variance (Zheng al. 2025). While these approaches can be broadly viewed as online p selection methods, our work differs fundamentally: instead of filtering out zero-variance prompts, we propose extracting useful learning signals from them, thereby improving the effectiveness of RL training.
Fine-grained Reward Shaping. In contrast to traditional RL algorithms such as PPO (Schulman al. 2017), recent RLVR methods remove the value model and compute sample-level advantages, broadcasting the same advantage to all tokens in response. This coarse-grained approach has motivated research into fine-grained reward shaping, such as token-level adjustments based on to- ken roles (Cui et al. 2025 Agarwal et al., 2025: Wang al. 2025 Deng et al.) 2025 Lei et al.) 2025). However, these methods still suffer from advantage vanishing when faced with vari-var prompts. Our work builds this line of research by leveraging entropy mechanism extract meaningful signals from zero-variance prompts, addressing this limitation directly.
6 CLOSING REMARKS
In this paper, we introduced RL-ZVP, a novel algorithm that extracts useful learning signals from vero-variance prompts enhance policy optimization. Within these prompts, RL-ZVP directly rewards correct responses and penalizes incorrect ones, without requiring contrastive answers within the same group. Furthermore, RL-ZVP adopts a fine-grained optimization strategy, scaling updates by token entropy to elicit stronger reasoning behaviors in the policy model. Empirical results show that RL-ZVP delivers significant improvements, substantially outperforming GRPO and other strong baselines. To the best of our knowledge, this is the first work to demonstrate that zero-variance prompts can be exploited rather than discarded. We believe this perspective opens a new direction for RL training, with possibilities for larger gains through improved formulations in future research.
Limitations & Future Directions. Due to computational constraints, our experiments are limited to models up to 8B parameters; future work should investigate how RL-ZVP scales to larger models (e.g., 14B or 32B). Furthermore, we only validate RL-ZVP on verifiable tasks with binary rewards; extending it to settings with graded or ambiguous feedback remains open challenge. Finally, promising direction is to generalize our advantage formulation beyond zero-variance prompts or to combine with existing token-level reward shaping methods to further improve GRPO.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We thank Brano Kveton, Nikos Vlassis, Yoonjeon Kim, Doohyuk Jang, Gyouk Chu, Minh-Hai Nguyen, and Tri Le for their valuable feedback and discussions. We are also grateful to Trung Bui for supporting compute allocation.
This work was supported by the National Research Foundation of Korea (NRF) grant (No. RS-2023- 00209060, A Study on Optimization and Network Interpretation Method for Large-Scale Machine Learning) and the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant (No. 955-2019-II19000
Artificial Intelligence Graduate School Program (KAIST); No. RS-2024-00457882, AI Research Hub Project) funded by the Korea government (MSIT). Myeongho Jeon was supported by the Sejong Science Fellowship of the National Research Foun- dation of Korea (NRF) (No. RS-2024-00421200 and the InnoCORE Program of the Ministry of Science and ICT (No. N10250156).
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in Ilm reasoning, Advances in Neural Information Processing Systems, 38, 2025.
Anthropic.
Claude 3.7 claude--- 7- sosnnett] 2025.
sonnet.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 40, 2026.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. 2025.
Jia Deng, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, and Ji-Rong Wen. Decomposing the entropy- performance exchange: The missing keys to unlocking effective reinforcement learning. arXiv preprint arXiv:2508 022600, 2025.
Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guidee reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 35:5829-5842. 2022.
Lizhe Fang, Yifei Wang. Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? In The Thirleenth International Conference on Learning Representations, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in llms through reinforce- ment learning. Nature, 645(808) 1):63)66:33 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 3828-3850, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol-
ume 1, 2021.
11

---

Published as a conference paper at ICLR 2026
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16722) 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025.
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 1830-1842, 2024.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield- Dodds, Nova DasSarma, Eli JonnsJonnoon, et al. Language mod- els (mostly) know what they know. arXiv preprint axXiv:220.00522 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35:22199-22221 2022.
Shiye Lei, Zhihao Cheng, Kai Jia, and Dacheng Tao. Revisiting Ilm reasoning via information bottleneck. arXiv preprint arXiv:2507. 18391, 2025.
Jia LI. Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen. Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. hetttps:/huugingeee hetp::://gtttubb..cm//oo/o// aimo progress http::/pppppp...///a/////////aaa
2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du Wee Sun Lee, and Min Lin. Understanding r1 r1-zero-like training: critical perspective. In Second Conference on Language Modeling, 2025.
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, et al. Tricks or traps? a deep dive into RL for LLM reasoning. In The Fourteenth International Conference on Learning Representations, 2026.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zeulemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. sl: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20286 20286-20332. 20332. 2025.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In Proceedings of the Sixteenth International Confer- ence on Machine Learning, volume 99, pp. 278-287, 1999.
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Faster, more efficient RLHF through off-policy asynchronous learning. In The Thirteenth International Conference on Learning Representations, 2025.
Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Bjöm Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models? In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2026.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-prov q&a bench- mark. In First Conference Language Modeling, 2024.
12

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thoughe can solve them. In Findings of the Association for Computa- tional Linguistics: ACL 2023, pp. 13003-13051, 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 12599, 2025.
Binghai Wang, Rui Zheng. Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint 2rXiv:2401.000 000000, 2024a.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effecttive reinforcement learning for IIm reasoning. Advances in Neural Information Processing Systems, 38, 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weim- ing Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95266 95290, 2024b.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thougee prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38, 2025.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised 11m reasoning incentivization. Advances in Neural Information Processing Systems, 38, 2025.
Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for Ilm reasoning via selective rollouts. Advances in Neural Information Processing Systems, 38, 2025.
Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, and Xin Jin. Optimizing RLHF training for large language models with stage fusion. In 22nd USENIX Symposium on Networked Systems Design and Im- plementation (NSDI 25), pp. 489-503, 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A FULL IMPLEMENTATION DETAILS
We present the full experimental setup details here. RL-ZVP and all baselines implemented on top of the veri framework, ensuring that every experiment is conducted within the same pipeline. Training is performed on 4xRTX 3090 GPUs for Qwen3-1 1.oen-1.7B--- 1.7B-Base and 8xA100 GPUs for Qwen3- 8B-Base.
For evaluation, initially attempted to measure validation results after every training iteration (16 gradient steps), but this proved extremely inefficient given the size of the validation set (1534 problems with 8 responses each). Instead, the small scale, we evaluate the model performance on Minerva, AMC23, and MATH after every iteration and report the best result, while for AIME24, AIME25, and OlympiadBench we use the final checkpoint. On the large scale, thanks to greater training efficiency on more powerful GPUs, we evaluate Minerva, AMC23, MATH, AIME24, and AIME25 after every iteration and report the best results, while for OlympiadBench we use only the final checkpoint.
The detailed hyperparameter settings are provided in Table We mostly follow the standard hyper- parameter settings from recent works (Yu et al., 2025; Cheng et al., 2026 Zheng al. 2025). We use the clip-higher scheme with thigh = 0.28 and Elow 0.2, apply token-level loss, and exclude KL divergence term from the objective.
In addition to the common settings shared across methods, we also adopt the unique hy hyprraaraeeter configurations of each baseline. For GRESO (Zheng al. 2025). we follow their probabilistic filtering setup exactly: Oggas 12 Chard = 6 Peasy 0.5, Phard 0.5, Ap = 0.01, and 3 1.25. For GRPO-Da the only deviation from DAPO (Yu et al. 2025 is that omit the soft overlong punishment to ensure fair comparison with other methods, which also do not include this form of length-based reward shaping.
Table 5: Full hyperparameter setting.
Hyperparameters
Qwen3-1.7B-Bal + MATH
Qwen3-8B-Baid Ba-e + DAPO-Math-177
Training batch size
512
Mini batch size
32
32
Samples per prompt
16
Max response length
4096
8192
Clip ratio
(0.20, 0.28)
(0.20, 0.28)
Scaling factor a
0.10
0.10
Training temperature
1.0
1.0
Training top-p Validation temperature
1.0
1.0
Validation top
0.7
0.7
Number of epochs
Total gradient steps
672
1056
Optimizer
AdamW W
Adam W 10-6
Learning
10-6
LR warmup steps
LR scheduler
cosine
constant
B ADDITIONAL EXPERIMENTAL RESULTS
B. ALTERNATIVE TRAINING DATA
To further validate the effectiveness of RL-ZVP alternative training dataset, conduct periments using a random cok-prompt subset NuminaMath-Con (LI et al.] [024]] Zhang et al.] 2025), comparing RI-ZV with GRPO under the same setup main experiments.
14

---

Published as a conference paper at ICLR 2026
Table 6: Additional evaluation results on six math reasoning benchmarks when the model is trained with NuminaMath-Cof.
Minerva
MATHS50
AIME24
AIME25
OlympiadBench
Method
Acc Pass
Pass@8
Acc@8
Pasc@8
Acc
Pass88
Gasc A
8
Qwen3-1.7B:
NuminaMath-Con subsel
30.01
44.88
43.75
72.20
67.53
83.12
RL-ZVP
30.56
45.96
44.69
70.81
70.80
86.78
8.33
18.32
5.42
13.94
32.78
51.95
Awwr GRPO)
+0.55
+1.08
+0.94
-1.39
+3.17
43.66
+1.66
-0.27
+0.84
40.30
+2.33
+4.27
Qwen3-8B- trained with NuminaMath-CoT subset
GRPO
42.42
54.41
66.88
90.10
83.63
93.32
25.00
45.40
20.83
38.49
54.01
68.29
RI-ZVP
56.15
28.75
A(we GRPO)
+2.30
+1.74
+6.25
+3.81
+3.60
+1.88
+3.75
66.14
+2.09
+2.12
+1.20
+1.38
Res Response Length
1.00
Acc@8 (96)
45-
1250
RL-ZVP
GRPO
Entropy
0.75
40-
1000
0.50
200
400
800
200
600
800
200
800
Gradlent Steps
Gradient Steps
Gradient
(a) Acc@8 (Qwen3-1.. Beana--1.B--Baa. 7B-Basc)
(b) Entropy (Qwen3-1.77) 1-77--Bscc)
(c) Response Length
Jywen3-1.7B-BBl
Response Length
AcciB (96
Entropy
0.75
1500
40
0.50
1000
200
400
600
800
200
400
600
800
200
400
600 800
Gradient Steps
Gradient Steps
Gradient Steps
(d) 8cc@8 (Qwen3-8B-Base)
(c) Entropy (Qwen3-8B-Base)
(f) Response Length (Qwen3-8B-Baan
Figure 6: Validation accuracy and training dynamics at different experiment scales.
The detailed evaluation results are shown in Table and the corresponding average validation ac- curacy and training dynamics are provided in Figure D RL-ZVP consistently outperforms GRPO all benchmarks, surpassing it throughout the training process. Moreover, RL-ZVP exhibits dis- tinct training dynamics, characterized by lower entropy and longer response lengths. These findings confirm that RL-ZVP is effective not only across evaluation benchmarks but also across different training datasets, further strengthening LLM reasoning capabilities.
B.2 OUT-OP-DOMAIN EVALUATION
To assess whether RL-ZVP remains effective beyond math reasoning, we evaluate Qwen3-1.77 Base trained with the MATH dataset on several out-of-domain benchmarks. Our evaluation suite includes LiveCodeBench for coding (Jain et al,, 2025). MMLU-Pro for general knowledge (Wang for scientific reasoning (Rein al. 2024). and BIG-Bench Hard for logical reasoning Suzgun et al. (2023). For MMLU-Pro, we use a subset containing economics, history, and other subjects; for BIG-Bench Hard, we evaluate on the formal fallacies and logical deduction subset. This avoids overlap with our in-domain math benchmarks, keeps evaluation size manageable, and focuses on reasoning-oriee oriented tasks. As shown in Table 7 training on math al- ready provides noticeable cross-domain gains, and RL-ZVP consistently outperforms GRPO and all baselines on
al., 2024b), GPQA-Diamone
benchamrks. These results demonstrate that RL-ZVP not only improves in-domain math reasoning but also generalizes effectively to other forms of complex reasoning.
15

---

Published as a conference paper at ICLR 2026
Table Out-of-domain evaluation results of trained models on four general reasoning bench- marks. The targeted domains are coding (LiveCodeBench), general knowledge (MMLU-Pro)), sci- entific reasoning (GPQA-Diamond) and logical reasoning (BIG-Bench Hard). For MMLU-Pro, the test subsets of economics, history, and other are utilized. For BIG-Bench Hard, the used subsets are formal fallacies and logical deduction.
LiveCodeBench
MMLU-Pro
GPQA-Diamond BIG-Bench Hard
Average
Method
Acc@8
@
A
Pass@8 Acc@8 Pass@8 Acc@
Pass@8
Acc@8 Pass@8
Qwen3- 1.7B-Base trained with MATH (small scale)
Base model
2.30
8.63
25.79
52.79
19.92
59.93
9.98
21.42
14,50
35.69
GRPO
13.03
24.17
38.92
59.88
26.56
63.03
32.84
64.27
27.84
52.84
GRPO-DS-r
13.27
23.70
35.47
56.84
25.59
62.89
28.64
54.49
25.74
49.48
GRPO-DS-g*
11.69
21.61
37.55
59.02
27.21
66.12
39.61
65.87
29.02
53.16
GRESO-r
12,99
23.03
38.33
60.17
28.19
65.91
36.16
64.61
28.92
53.43
GRESO-g*
11.58
22.75
38.91
59.86
26.82
63.31
36.60
68.46
28.48
53.60
RL-ZVP (Ours)
13.61
24.46
39.42
62.18
28.97
69.47
38.74
65.34
30.19
55,36
A(wrt GRPO)
+0.58
+0.29
+0.50
+2.30
+2.41
+6.44
+5.90
+1.07
+2.35
+2.52
Table 8: Evaluation results on Llama-3223BB-nstruus demonstrating the generalization of RL- ZVP across model families.
Minerva
AMC23
MATH500
AIME24
AIME25
OlympiadBench
Method
Acc@8 Pass @
Acc@ Pass @8
8 Pass@8
@ Pass
Pass Acc@8
Acc@8 Pass @
Llama-3.2-31 Instruct trained with
GRPO
21.23
35.64
28.13
56.39
51.84
71.08
14.58
28.79 29.79
2.50
7.78
19.43
32.19
GRPO-DS-r
21.65
34.44
33.75
58.70
51.23
70.77
12.92
28.84
0.83
4.38
18.75
31.18
GRPO-DS-g*
21.88
36.50
33.75
58.70
52.44
70.77
14.58
28.84
1.67
6.67
19.82
30.14
GRESO-r
21.09
35.41
29.69
58.95
51.64
69.91
15.42
26.52
1.67
7.43
18.49
30.92
GRESO-g"
21.55
35.41
32.19
58.95
52.16
69.91
16.25
27.65
1.67
7.43
18.96
30.72
RL-ZVP (Ours)
22.43
36.50
30.63
58.44
52,87
70.46
17.08
30.48
2.50
7.43
19.47
32.31
AWwIt GRPO)
+1.20
+0.86
+2.50
+2.05
+1.03
-0.62
+2.50
+1.69
0.00
-0.35
+0.04
+0.12
B.3 PERFORMANCE DIFFERENT MODEL FAMILY
All experiments presented so far are conducted on models from the Qwen family (Yang al. (025). To examine whether the effectiveness of RL-ZVP extends beyond this setting, we additionally eval- uate our method on Llama-3. 1i-aa...-3---nstteeee (Grattafiori et al. 2024), representing a different model family. We follow the same experimental setup and compare RL-ZVP against GRPO and other prompt filtering basclines.
As shown in Table 8 RL-ZVP continues to deliver consistent and meaningful improvements over GRPO and other baselines across most benchmarks, achieving higher average performance overall. These results indicate that the benefits of RL-ZVP not specific to particular model family, and that the proposed approach generalizes well to models with distinct pretraining and alignment characteristics.
C MORE ABLATION STUDIES
C.1 EFFECT OF DIFFERENT a SCALES
We already provided the ablation study the scaling factor a in Section 4.3 analysis by examining wider range of a values - varying them across several orders of magnitude - to understand their impact on RL-ZVP performance more comprehensively. As shown in Table setting a too small diminishes the gains over GRPO, as the contribution of zero-variee vero-nariance prompts signals becomes negligible. In contrast, overly large values of destabilize optimization and lead to noticeable performance degradation. These results are consistent with, and further reinforce, the conclusions drawn from the ablation in the main text.
Here, we extend this
16

---

Published as a conference paper at ICLR 2026
Table 9: Validation results under a wide value range of in order of magnitude.
Minerva
AMC23
MATHS00
Average
Alpha
A
@
Acc@8
Pass 8
Acc@8
Pass @8
Acc@8
Pass@2
a
29.09
43.19
42.19
72.00
69.09
84.05
46.79
66.41
0.001
29.41
43,45
43.44
71.08
70.65
86.38
47.83
66.97
a 0.01
29.09
43.61
43.44
69.75
69.77
86.19
47.43
66.52
0.10
29.96
44.76
48.75
77.19
70.98
87.35
49.90
69.77
Q 1.0
26.98
41.64
39.38
67.30
66.60
84.00
44.32
64.31
10.0
24.86
42.01
34.69
65.42
60.34
81.97
39.96
63.13
Table 10: Comparison between GRPO, RL-ZVP and symmetrie RL-ZVP
Minerva
AMC23
MATH500
Average
Method
Acc@8
Pass@8
Acc@8
Pass@8
Acc@8
Pass@8
Acc@8
Pass@8
RL-ZVP (Ours)
29.96
44.76
48.75
77.19
70.98
87.35
49.90
69.77
with symmetric 7VP advantage
29.60
43.76
41.88
70.26
66.44
84.44
46.64
66.
GRPO
29.09
43.19
42.19
72.00
46.79
66.41
C.2 COMPARISON WITH SYMMETRIC VARIANT OF RL-ZVP
The advantage formulation for zero-variance prompts in Equation is intentionally asymmetric between positive and negative prompts. To validate this design choice, we additionally compare RL-ZVP and GRPO against a modified "symmetric" variant of RL-ZVP which the advantage for negative prompts is defined as the exact opposite of that for positive prompts.
(oHi,
if
= A R, 7>00 0,
Azve(or.)
-aHa,
if std std( A R
(8)
Table re the results. We observe that enforcing symmetry significantly harms performance: the symmetric variant underperforms RL-ZVP across all benchmarks and even falls below the GRPO baseline. These empirical findings highlight that a symmetrie penalty is counterproductive setting and further justify asymmetric formulation, consistent with the intuition provided in the main text regarding the distinct roles of positive and negative prompts in our advantage design.
this
D FULL TRAINING DYNAMICS OF THE MAIN EXPERIMENTS
Figure 7 shows the full training dynamics for all methods, including GRPO-DS and GRESO, in our main experiments (Section 4
We observe that at the small scale, the prompt-fiteriing baselines (GRPO-DS and GRESO) perform better than GRPO but remain consistently worse than our RL-ZVP method. In terms of entropy behavior, although GRPO-DS and GRESO exhibit increasing response length, their entropy rises sharply, suggesting a form of entropy explosion. contrast, RL-ZVP achieves longer responses while maintaining stable and non-collapsing entropy, indicating substantially healthier learning dy- namic. Combined with the steady performance gains, these trends reflect the emergence of stronger forg-form reasoning ability with high certainty.
At the larger scale, prompt-filtering baselines lag significantly behind RL-ZVP, though they perform slightly better than GRPO. Their training dynamics also show limited growth in response length relative to RL-ZVP, and together with interior evaluation results, this suggests that GRPO-DS and GRESO fail to learn effective reasoning behaviors efficiently RL-ZVP.
We emphasize that this plot inherently favors GRPO-DS and GRESO, because under the same num- ber of gradient steps, these methods perform many more rollouts than GRPO and RL-ZVP (see
17

---

Published as a conference paper at ICLR 2026
GRPO-DS
Response Length
1500
MA
GRPO
1.00
Arciiia
85
Eatrop
0.75
1250
GRESO
G 0.50
1000
40
0.25
750
200
200
400
600
800
200
400
600
800
Gradient Steps
Gradient Steps
Gradient Steps
(a) Aco@8 (small scale)
(b) Entropy (small scale)
(c) Response Length (small scale)
50
1.00
Response Length
:3000
www.
Acc@@a (%)
Entropy
40
0.75
2000
0.50
30
1000
0.25
200 400 600 800
1000
200 400 600 800
1000
200 400
600 800 1000 Gradient Steps
Gradient Steps
Gradient Steps
(d) Acc@8 (large scale)
(c) Entropy (large scale)
(f) Response Length (large scale)
Figure 7: Validation accuracy and training dynamics at different experiment scales, The figure
with here is essentially updated version of Figure with of GRPO-DS and GRESO training dynamics.
Table D. Despite this advantage for prompt-fiterrin baselines, RL-ZVP still shows consistent and substantial improvements over them throughout training.
PROPORTION OF ZERO VARIANCE PROMPTS AT SCALE
In Figure X we provide zero-variance prompt statistics for our ex- periments from Appendix B.I. where train Qwen3-8B-Base on a 20k subset of NuminaMath-CoT. This dataset more challenging
100
Owen3-88-Bare DAPO-Math-177
Zero-Variance Ratio
than MATH but substantially easier than DAPO-Math- 1..--M lead- ing to a much lower gero-variance ratio - approximately 30% prompts per batch at the start of training, compared to nearly 99% when training on DAPO-Math-17k.
75
Owan3-8B-Base NuminaMath-Coo
50
25
100
500
Gradient Steps
Figure The percentage of zero-variance prompts when training 8B model with challenging dataset (DAPO- Math-17k) and easier dataset (NuninaMath-Cof)),
18

---

Published as a conference paper at ICLR 2026
F QUALITIVATE EXAMPLES
This appendix demonstrates some output examples generated by policy models trained with GRPO and RL-ZVP.
Minerva: Question and Ground Truth
Question: A particular star has absolute magnitude M = -7 If this star is observed in galaxy that a distance of 3Mpc, what will its apparent magnitude be?
Ground Truth: 20.39
Minerva: Response from GRPO (Qwen3-1.7B-Base)
To determine the apparent magnitude of star with absolute magnitude of M -7 that is observed at distance of 3Mpc, we can use the distance modulus formula:
m M = 5 (d) 5
where:
- is the apparent magnitude, -M the absolute magnitude,
d is the distance to the star in parsecs.
Given: .M -7
d =3Me
First, we need convert the distance from Mpc parsecs. Since 1Mpc 3.2616 1028 parsecs, we have:
d 3 3.2616 10 parsees
Now, plug the values into the distance modulus formula:
(-7) 5 log (3 x 3.2616 x 10
Simplify the expression inside the logarithm:
x 3.2616 x 1 9.7848 x 10
Now, calculate the logarithm:
(9.78488 10 log (9.7848) log 1o (1018) = 0.9908 + 18 18.9908
Next, multiply by 5:
18.9908 94.954
Now, add 5 to the result:
7 94.954
94.954 - 87.954
So, the apparent magnitude m of the star is approximately 87.954. Therefore, the final answer is:
87.95
19

---

Published as a conference paper at ICLR 2026
Minerva: Response from RL-ZVP (Qwen3-1.7B-Bas
Alright, let's solve the problem step by step.
**Problem Statement:** *0
We have star with an absolute magnitude M = -7. This star is observed in galaxy that is at a distance 3Mpc. We need to find apparent magnitude.
** Understanding the Concepts:**
1. ** Absoluute Magnitude (M))** This is the brightness of star if it were placed at standard distance of 10 parsees (pc) from Earth. It gives us a measure of the star's intrinsic brightness. 2. * Apparent Magnitude (m):** This is the brightness of star seen from Earth. It depends on both the star's intrinsic brightness (absolute magnitude) and the distance to the star.
3. **Distance Modulus Formula:** The relationship between absolute magnitude (M), ap-
parent magnitude (m), and distance (d) in parsecs is given by:
m M 5 logno (d)
where d the distance the star in parsecs.
**
Absolute magnitude M = 7
Distance d = 3Mpc
**Step 1: Convert the Distance to *
First, we need to convert the distance from megaparsecs (Mpc) to parsecs (pc) because the distance modulus formula uses parsecs. 1Mpc = 10 pc So,
d= 3Mpc 10 pc
**Step 2: Plug the Values into the Distance Modulus #ormula Using the formula:
M 5 log (d)
Substitute M 77 and d = 3 10
m (-7) (3 100) 5
m + 7 5og 5lg (10%) 5
**Step 3: Simplify the Logarithms** (...)
Step 5: Substitute and Solve for ** Now, substitute (0)10(3) into the equation:
+ 7 0.4771 25
2.3855 +5
m 7 27.3855
m = 27.3855 - 7
m = 20.3855
**Final *n
The apparent magnitude of the star is approximately
20.39
20

---

Published as a conference paper at ICLR 2026
AIME24: Question and Ground Truth
Let B the set of rectangular boxes with surface area 54 and volume 23. Let be the radius of the smallest sphere that can contain each of the rectangular boxes that elements of B. The value y2 can be written as P where and are relatively prime positive q²
Question:
integers. Find +
Ground Truth: 721
AIME24: Response from GRPO (Qwen3-1.7B-Base)
To solve the problem, we need find the radius of the smallest sphere that can contain each of the rectangular boxes with surface area 54 and volume 23. The radius of the smallest sphere that can contain rectangular box is the maximum distance from the center of the box to any of its vertices.
Let's denote the dimensions of the rectangular box by a. b, and c. The volume of the box is given by:
(...)
We need to find the roots this polynomial. By trial and error or using numerical methods, find that possible set of roots is 1, b and 23. Let's check if these values satisfy the equations:
abc
23 23
ab + bc + ca = 1- 23 23 1 23 23 47 27
This set of values does not work. Let's try another set of values. Suppose a 1, b = 1, and c = 23 not solution, let's try another approach. We can use the fact that the box symmetric and assume b. Let's assume b and solve for c:
a²c = 23
2a² + 2ac 54
From the first equation:
23 c a²
Substitute into the second equation:
23
2a + 2a
54
a²
46
2a 2 +
54
a
Multiply through by a:
2a 46 54a
2a - 54a + 46 = 0
By trial and error, we find that a = 2 is a root:
(...)
The answer is 673
21

---

Published as a conference paper at ICLR 2026
AIME24: Response from RL-ZVP (Qwen3-1.7B-Base)
AIright, have this problem about rectangular boxes and spheres. Let's break it down step by step.
** **Problem Statement: **
We have a set of rectangular boxes with:
1. Surface area 54
Volume = 23
Let B be the set of all such boxes. We need to find the radius of the smallest sphere that contain each of these boxes. Finally, we need to express r² as a fraction where p and q are coprime positive integers, and then find +
** UUnderstanding the *
First, let's understand what's given: A rectangular box has dimensions b, and
The surface area a rectangular box is given by:
S 2(ab + be + ea) 54
-The volume V of the box is:
V abc 23
We need to find the smallest sphere that can contain any box The radius r this sphere will be the maximum distance from the center of the sphere any point the box. So, r half the length of the space diagonal the box.
** Finding the Space Diagonal:***
The space diagonal d of rectangular box is given by:
= v a2 - +
(...)
We need to maximize a² 63 + c², Let's see if we can express a² + 6² + c² in terms of ab be + and abc. Recall the identity:
(a + b + c) a² 12 2 + 2(ab + be + ca)
But we don't 't know Lagrange multipliers or express two variables in terms of the third, but that might get compli- cated.
directly. Alternatively, we can consider using the method of
assume symmetry to simplify the problem. Let's assume that two of the dimensions are equal, and the third is different. Let's s set a = b, and express in terms of a.
Wait, perhaps
(...)
We need to find 2
2
3
73
9 x 73
657
22
64
Here, p 657 and g 64, which are coprime.
#*Final *
p + q
657
64 721
721
22

---

Published as a conference paper at ICLR 2026
AIME25: Question and Ground Truth
Question: Let A ... An be 11-sided non-convex simple polygon with the following properties: * 'The area of A,41 for each 2 S i < 10, * cos A, = 13 for each 2 < S 10, 'The perimeter of A1 ... A11 is 20. If A + A1 can be expressed m.y- for positive integers m, n, with squarefree and no prime divides all of m,,, 4.
12
find m + n2 + + 4-
Ground Truth: 19
AIME25: Response from GRPO (Qwen3-8B-Base
### Step-by-Step Solution:
** "Understanding the Given Information:
We have an 11 1--iiddd non-convex simple polygon A A2... An-
The of triangle A is for each < 10.
The cosine of angle 1 A4+1 12 for each < < 10.
13
The perimeter of the polygon 20.
We need to find A₁ A2 + A An expressed in the form myT--p and compute m + n + p +
2. ** *Aallyzing the Triangles:**
For each triangle A we know:
Area = cos( 13 where LA;A1A+1
(...)
However, this approach seems too complex. Let's try a different method.
**Using Trigonometry and Perimeter:**
Let'ss denote A₁ A2 and y A1A11
The perimeter equation:
10
T +
A1A; + AnA = 20
(...)
-Tii seems too complicated. Let's try a different approach.
10. **ninal Calculation: Let's assume A₁ for i = 2, 10.
(...)
-This approach is too complex. Let's try different method.
11. **Simpler Approach:*3 Let'ss denote S = A,A2 + AiA
(...)
-Therefore:
m. n +
80 + 26 + 260 + 19 = 385
#*# Final Answer: 385
23

---

Published as a conference paper at ICLR 2026
A1ME25: Response from RL-ZVP (Qwen3-8B-Base)
Alright, have this problem about an I-sided non-convex simple polygon with some specific properties. I need to find the value of A1 + A1A11 expressed in certain form and then compute m + n + p + q. Let's break it down step by step.
**Given:*
1. A1A2.... A1 is 11-sided non-convex simple polygon.
2. The area of triangle A, is for each 2 S S 10.
3. (1 = 12 for each < i S 10.
4. The perimeter of the polygon is 20.
** Find A A2 + A₁ An expressed as my/nPP and then compute m + n + p + Alright, let's tackle this step by step.
### Step I: Understanding the Triangle Areas and Angles
First, we have triangles A, As+1 for i = to The area of each of these triangles is and we know the cosine of the angle at A₁ in each triangle is 12 13"
(...)
But from the triangles we have relationships between 2; and ys- Let's see how x; and ys relate to the sides of the polygon. Wait, perhaps it's better think in terms of vectors the Law of Cosines. Alternatively, perhaps we can express everything in terms of A A2 and A₁ An1.
### Step 3: Expressing the Polygon in Terms of Vectors
(...)
Wait, perhaps we can find + y or another relationship. Alternatively, perhaps we can sum up all the triangles. Wait, let's think differently.
### Step 4: Summing the Areas (...)
But how do these sides relate to x and y? Wait, perhaps we can express each side A;A44 in terms of x and Let's see.
(...)
Wait, perhaps we can express a and lo in terms of x and y. Alternatively, perhaps can find + directly. Let's think differently.
### Step 8: Finding + k
(...)
Thus,
m m+n+p+g=
**Final **
19
24

---

Published as a conference paper at ICLR 2026
MATH500: Question and Ground Truth
Question: 19 1 and + 1, then evaluate
+ w
1 w
+ w 1997
Ground Truth: 1997
2
MATH500: Response from GRPO (Qwen3-8B-Base
### Step-by-Step Solution:
** "Undrrstadiing the Problem:** **
We are given that w 1997 1 and w & 1. This means w is a primitive 1997th root of unity.
We need evaluate the sum:
1997
=
Σ
1 + wk
R=1
Simpllff the Sum
Notice that w 1997 1 implies w for k 1996 are all distinct roots of unity, and
w 2997 = 1 repeats the root
However, since + 1, 1997 is already included in the sum.
3. **Pairing *s -Consider the sum:
1997
=
1 + w
R= 1
Notice that w 1907-k w-k (since 1997 1). The term I can be paired with Too
1
1
1 wh + 1
1 + wk
+ 1997-
Simplify the pair:
(1 + w
+ (1 + w
wh + w
2 2 w +
++ w
1 +
(1+ w+ w
1 w + + +1
2 + wh +w
-Therefore, each pair sums 1.
4. ** **Cunnting the Pairs:*
Since there 1997 terms in the sum, and each pair consists of terms, there are 1997-1 998 complete pairs.
The remaining term is when k 999, but since 1997 is odd, there is unpaired term.
5. **Calculating the Total *um:**
-The sum of 998 pairs, each summing to 1, is:
998 X 998
### Final Answer: 998
25

---

Published as a conference paper at ICLR 2026
MATH500: Response from RL-ZVP (Qwen3-8B-Base
Alright, have this problem to solve:
**Prblemm*** **
If w 1997 and w # 1, then evaluate
+
1t
1 1 w
+ 1997
Alright, let's break this down.
### Understanding the Problem
(...)
So, the total sum S is:
1
1996
1997
S 998 998
2
2
Wait, that's 1997 But let's double-check this.
### Double-Checking the Calculation
(...)
Let's multiply S' by (1 - w)):
1996 (1-w)) (1-
w
+ w
k=
This seems complicated. Maybe there's better approach.
### Using the Identity for the Sum
I recall that for any primitive r-th root of unity the sum Exel
a-1
Final Answer: 1997
But since the problem likely expects an integer, perhaps missed something. Wait, let's think differently.
### Final Verification Let's consider the sum:
1997
w
k=1
(...)
Therefore, the correct answer is indeed 1997
2
1997
Final Answer:
2
26

---

Published as a conference paper at ICLR 2026
G THE USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were employed in a limited capacity during the final preparation of this manuscript. Their role was restricted proofreading and stylistic refinement, including the correction of grammatical errors, improvement of sentence fluency, and minor rephrasing to enhance readability.
All significant aspects of the work - including research conception, methodological design, experi- mental implementation, analysis, and the drafting of the manuscript were conducted solely by the authors. The use of LLMSs did not contribute to the ideation, technical content, scientific writing of the paper, and was confined strictly language polishing at the concluding stage.
27


Reviews:
[{"review_id": "CO6kmcO7EI", "reviewer": "Reviewer_sAk1", "summary": "The paper targets a failure mode in RLVR training (e.g., GRPO) where prompts whose rollouts are uniformly correct or uniformly incorrect yield zero reward variance, causing normalized advantages to collapse and producing no gradient signal. The authors propose RL-ZVP, a simple modification that preserves the standard GRPO update when variance is nonzero and substitutes an entropy-guided, sign-corrected token-level advantage when variance is zero. Intuitively, positive zero-variance prompts should reinforce the behavior, while negative ones should be discouraged; using token entropy modulates the update to emphasize informative positions. Experiments on multiple math benchmarks with Qwen backbones report consistent improvements over GRPO under comparable training setups and show smoother training dynamics.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Well-motivated objective:** The paper clearly identifies a practical inefficiency in GRPO, zero-variance prompts producing zero gradients, then directly targets it with a minimal change to the loss. Conceptually, reclaiming learning signal from these otherwise “wasted” batches is appealing because it improves sample/compute efficiency without redesigning the overall RLVR pipeline. The proposed branch only activates when the variance is zero and leaves GRPO behavior untouched elsewhere, so the method respects the existing training regime rather than replacing it.\n- **Solid empirical results:** Across several math datasets and multiple model sizes, the method reports consistent gains over GRPO under comparable setups. The improvements appear meaningful rather than marginal, and the training dynamics presented suggest more stable progress rather than spiky or brittle behavior.", "weaknesses": "- **Checkpoint selection leans to optimistic:** The evaluation mixes “best checkpoint during training” with other reporting choices, which can yield optimistic numbers and makes comparisons harder to interpret. It would be beneficial to follow the ML practice where checkpoints are chosen based on a held-out validation set, and results are reported from a single, consistently selected checkpoint (not a mixture).\n- **Limited training-curve comparisons:** Training curves (Fig. 4) are shown only for GRPO and the proposed method. Including additional baselines would clarify whether RL-ZVP’s dynamics are uniquely beneficial or simply reflect broader trends across methods, which helps to better assess the proposal approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ABsPBNAYx", "reviewer": "Reviewer_xC6w", "summary": "The paper aims to leverage Zero-Variance Prompts (consisting of fully correct or fully incorrect responses) during RL to enable more efficient learning. The authors propose using token-level entropy-based advantage updates for these responses. Extensive experiments are conducted to demonstrate the effectiveness of the method on reasoning tasks.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow.\n\n- The proposed method of assigning advantages to tokens in responses to zero-variance prompts is intuitive and well motivated.\n\n- Extensive experiments are conducted to demonstrate the effectiveness of the method on reasoning tasks.", "weaknesses": "- Significance of the problem: Are there any empirical statistics on the proportion of zero-variance prompts? I notice that the authors only train 8B models on the harder DAPO-Math-17k dataset. It would be helpful to also show empirical results on Math with 8B models (scenarios with fewer zero-variance prompts) to evaluate the effect of the newly designed advantage. We might expect that it does not hurt performance in scenarios with fewer zero-variance prompts, while providing greater benefits when the number of zero-variance prompts is larger.\n\n- How is it ensured that the advantages assigned to tokens in zero-variance prompts are on the same scale as normal advantages? Could differences in scale affect training stability?\n\n- How should an appropriate value of α be chosen? Are there any guidelines or heuristics for selecting it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4gCLA4JQHz", "reviewer": "Reviewer_3GQ5", "summary": "This paper addresses a critical inefficiency in Reinforcement Learning with Verifiable Rewards (RLVR), known as the \"advantage vanishing\" problem. Standard methods, such as GRPO, fail to learn from \"zero-variance prompts\"—instances where all sampled responses are either uniformly correct or incorrect—because their advantage calculation collapses to zero. The authors compellingly argue that this constitutes a significant source of inefficiency, as these prompts are both computationally expensive (rollouts consume approximately 50% of training time) and highly prevalent (comprising 30-99% of the data). The proposed method, RL-ZVP, leverages these prompts rather than filtering them. It achieves this by defining a new advantage signal: the direction is set to positive for all-correct groups and negative for all-incorrect groups, while the magnitude is modulated by token-level entropy. This \"entropy-guided\" shaping mechanism rewards high-entropy (uncertain, complex) tokens in correct responses and penalizes low-entropy (confident) tokens in incorrect responses. Rigorous experiments demonstrate that RL-ZVP significantly outperforms GRPO. Notably, it also surpasses modern filtering-based baselines (e.g., GRPO-DS, GRESO), even when those baselines are allocated 3–5 times more computational budget for rollouts.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's primary strength lies in its novel and intuitive solution to a practical and well-defined problem. The entropy-guided advantage shaping mechanism is a well-motivated and insightful design that effectively rewards productive, uncertain reasoning steps while penalizing confident errors. This strong methodology is validated by a particularly rigorous experimental setup. The authors compare RL-ZVP not only to the standard GRPO but also to two strong baselines (GRPO-DS, GRESO) specifically designed to filter the problematic prompts. By testing under both equal-rollout (fair) and equal-gradient-step (baseline-favored) conditions, the paper offers compelling evidence for RL-ZVP's superior performance and efficiency. This quantitative strength is substantiated by in-depth analysis, including an examination of training dynamics that explains its enhanced stability (leveraging signals from both easy and hard prompts at different training stages) and qualitative examples (Appendix D) that demonstrate a tangible improvement in reasoning correctness, not just stylistic complexity.", "weaknesses": "1）The method proposed is highly sensitive to the $\\alpha$ hyper-parameter. The experiments results in Table 3 demonstrate that the optimal performance window for the scaling factor $\\alpha$ is very narrow. This raises concerns that the method may require extensive and precise fine-tuning when applied to different datasets or models.\n\n2) The description on negative prompts seems confusing. The logic for advantage shaping (Section 3.2) is asymmetrical. Positive prompts reward high-entropy tokens far more than  negative prompts penalize high-entropy tokens. Could the author explain why use this setting?\n\n3 I think it will be better to provide more sufficient ablation Studies. The ablation study Table 2, while strong, omits a key comparison. It does not test a symmetrical penalty formula for negative prompts (e.g., $\\hat{A} = -\\alpha H_{i,t}$, which would penalize high-entropy error tokens more to see how it would affect performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)