Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
Authors: Junkang Wu
Kexin Huang
Jiancan Wu
An Zhang
Xiang Wang
Xiangnan He
Venue: ICLR 2026 Poster (2026)
Keywords: RLVR
LLM reasoning
entropy explosion
advantage estimation
Primary area: reinforcement learning

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning but training often oscillates between {entropy collapse} and {entropy explosion}.
We trace both hazards to the mean-baseline used in value-free RL (\eg GRPO/DAPO), which improperly penalizes negative-advantage samples under reward outliers.
We propose {Quantile Advantage Estimation} (QAE), replacing the mean with a group-wise $K$-quantile baseline.
QAE induces a response-level, two-regime gate: on hard queries ($p \le 1{-}K$) it reinforces rare successes, while on easy queries ($p > 1{-}K$) it targets remaining failures.
Under first-order softmax updates, we prove {two-sided entropy safety}, giving lower/upper bounds on one-step entropy change that curb explosion and prevent collapse.
Empirically, this minimal modification stabilizes entropy, sparsifies credit assignment (with tuned $K$, roughly 80\% of responses receive zero advantage), and yields sustained pass@1 gains on Qwen3-8B/14B-Base across AIME'24/'25 and AMC'23.
These results identify {baseline design}—rather than token-level heuristics—as the primary mechanism for scaling RLVR.

Published as a conference paper at ICLR 2026
QUANTILE ADVANTAGE ESTIMATION: STABILIZING RLVR FOR LLM REASONING
Junkang Wu' Kexin Huang' Jiancan Wu' An Zhang' Xiang Wang's Xiangman He
!University of Science and Technology of China
MoE Key Lab of BIPC, University of Science and Technology of China {j [jkwu0909, wujcan, xiangnanhe @ com
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM rea- soning but training often oscillates between entropy collapse and entropy explo- sion. We trace both hazards the mean-baseline used in value-free RL (e.g.. samples under GRPO & DAPO), which improperly penalizes Estimation (QAE), replacing reward outliers. We propose Quantile Advantage the mean with a group-wise K-quantile baseline. QAE induces a response-level, two-regime gate: hard queries (p 1-K) it reinforces rare successes, while on easy queries (p > 1-K) it targets remaining failures. Under first-order soft- max updates, we prove two-sided entropy safety, giving lower/upper bounds on one-step entropy change that curb explosion and prevent collapse. Empirically, this minimal modification stabilizes entropy, sparsifies credit assignment (with tuned K, roughly 80% of responses receive zero advantage), and yields sustained pass gains on across AIME'24H 25 and AMC'23, These results identify base- line design rather than token-level heuristics as the primary mechanism for scaling RLVR
INTRODUCTION
Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024; DeepSeek-Al et al., 2025; Yang et al., 2025a) enhances Large Language Models (LLMs) by rewarding verifiable cor- rectness (Phan et al., 2025; Rein et al., 2023). Yet neward-driven optimization often triggers entropy collapse (Yu et al., 2025; Cui et al., 2025): the policy distribution sharpens prematurely. suppress- ing exploration and ultimately limiting performance. This exposes fundamental tension between maximizing reward and preserving policy diversity during RLVR fine-tuning.
Prior work focuses almost exclusively on preventing collapse, e.g., uplifting low-probabilit tokens (Yu et al., 2025), penalizing collapse-inducing tokens (Cui et al., 2025), or preserving policy di- versity by primarily learning from negative samples (Zhu et al., 2025). While effective at avoiding collapse, these methods address only one side the problem and largely overlook its symmetric counterpart: entropy explosion. Uncontrolled entropy growth is equally harmful, leading to ineffi- cient exploration and stalled progress.
This risk is practical, not merely theoretical. On Qwen3-8B- BB-Base with DAPO, Figure (left) shows that Clip-Hi averts collapse but induces an early entropy spike (steps 10 - 80) that, while immediately harming performance, creates long-term instability. After step 100, entropy re- mains high and volatile, while performance plateaus. These dynamics highlight key shortcomings of unconstrained entropy growth: (i) higher policy entropy does guarantee continued effective ex- ploration performance can plateau despite ongoing behavioral variability reflected in high entropy; and (ii) the initial entropy spike indicates a period of exr-explonntion that, though not immediately destructive, ultimately undermines the model's ability to consolidate learning from high-reward rea- soning trajectories. The dual challenge, therefore, is to avoid both premature convergence (collapse)
"Jiancan Wu and Xiang Wang are the corresponding authors. 'The code is available https:tpp //gi www..coooooc...ccooooooccctoeeco www.:/ooojjjnnoooo juaa//guunaaggga......

---

Published as a conference paper at ICLR 2026
R mean((Rdee))
Au Quantilex ((RD)-))
ALt
std((R
std(
DAPO
DAPO+QAE
Explosioo Phase
12
Exploration Phase
Purform Purformancee corformance (AIME)
(AIME)
0.0
2.3
Entropy
Entropy
Sptral Zore
8
02
0.3
Russ:: Parfommence
[eadd: Suataiee Inprovemenn
50
156 Step
2003
332
105
236
Step
Figure 1: Entropy- peroooranee dynamics Qwen3-8B-Base Left: DAPO with Clip-Higher prevents early collapse but triggers an early entropy spike (steps 10-80) and later performance plateau. Right: our quantile baseline (QAE) stabilizes policy entropy and sustains pass@ I gains by steering training into a balanced exploration regime.
and unproductive, signal-degrading divergence (explosion). Merely avoiding collapse is therefore mnsufficient-effeenii RLVR requires keeping entropy within productive range.
We address this dual challenge with Quantile Advantage Estimation (QAE), which dynamically regulates policy entropy by replacing the conventional mean reward baseline with group-wise K- quantile. The key idea is that the baseline choice controls how many samples receive positive vs. negative advantages, which directly impacts exploration behavior. Specifically, a lower K marks more samples having positive advantage, encouraging the model to exploit these successful pat- terns and reducing entropy. Conversely, higher К makes fewer samples appear successful, pushing the model diversify its behavior patterns, thereby increasing entropy. By tuning the quantile pa- rameter K, we can control the an appropriately chosen K, this mechanism steers training toward stable entropy regime- neither collapsing nor exploding - enabling sustained performance gains beyond the prior plateau. This mechanism has striking empirical consequence: it naturally sparsifies updates. With tuned K, roughly 80% of responses receive zero advantage. This concentrates computational effort on the most informative samples and revealing deep redundancy standard mean-baselin approaches.
balance. As shown in Figure 1 (right), with
We trace both early entropy spikes and late plateaus to the mean-baseline in value-free RL; substi- tuting K K-quantile baseline (QAE) implements a response-level gate that routes updates rare suc- cesses on hard queries and to remaining failures on easy ones. We prove a two-sided entropy safety guarantee and derive a discriminative objective that explains the observed stability, which leads to significant pass@) gains and solid pass 16 performance. Empirically, the one-line swap boosts clip-Hi gher (Yu et al., 2025) on QWEN3-8B/14B -ASE,, pairs well with C1 cwwppCoo/KL.-OOO (Cui et al., 2025) QWEN3-8B --BEN--BB-- 3-B-BBS,,, and works with GSPO (Zheng al., 2025) on QWEN3- 30B-A3B- BASE, yielding consistent pass@ gains and strong pass @ on AIME 24, AIME' 25, and 23. Overall, QAE reframes entropy regulation baseline-design problem rather than a teven-level tuning problem.
2 PRELIMINARIES
In this section, review the policy optimization algorithms that form the foundation of our work, starting with Proximal Policy Optimization (PPO) and its value-free variants, GRPO and DAPO.
Proximal Policy Optimization (PPO) PPO (Schulman et al., 2017) is a foundational on on-policy algorithm that stabilizes training by constraining policy updates trust region around the previous
2

---

Published as a conference paper at ICLR 2026
policy Tew- It maximizes clipped surrogate objective:
JPPo (0) = -
min (r:(0)Ar, clip(r: -100)),1,,111 -,,,,))))
(I)
where r0) mefor q.oc)) is the probability ratio. The advantage A, is typically estimated by =a (or g.0c))
value network, and the clipping hyperparameter (e.g., 0.2).
Group Relative Policy Optimization (GRPO) To eliminate the need for value network, GRPO (Shao et al., 2024) adapts the PPO objective by proposing a relative advantage estimator. For each query, GRPO samples group of G responses {or from Tdw. Each response is as- signed binary reward RG based on its correctness against a ground-truth answer a. The advantage for the i-th sample is then estimated by normalizing its reward against the group's statistics:
A R4 - mean ( (Rx)
1.0 is equivalent(a, o,),
where R4 =
(2)
( )i
0.0 otherwise.
GRPO further incorporates KL divergence penalty against regularize the policy update.
Dynamic Sampling Policy Optimization (DAPO) We use DAPO (Yu et al., 2025), a state-of- the-art falue-free method, our baseline. DAPO refines GRPO with several key modifications. removes the KL penalty but introduces an asymmetric clipping range (1 - Clow 1 1 Chigh), ) allowing larger updates for advantageous actions. The objective also normalized at the token level:
o.
1
min (rt:(0)A,,, (0). Clows + ehigh Ar.t
JDAPO =E
(g.a)D, (-1))-1- aa (+1a)
Z
where Z = -E |o:1 is the total number of tokens in the group, and the advantage A.,; is computed as in GRPO. Crucially, DAPO employs dynamie sampling constraint:
(o; issequivalent o,) < G.
This ensures that each training batch contains both positive and negative examples, guaranteeing a meaningful advantage signal and stable gradients.
THE ENTROPY DILEMMA IN RL SCALING: FROM COLLAPSE TO EXPLOSION
Policy entropy is central to reinforcement learning, governing exploration-exploittiie trade-off. This balance is especially fragile in RLVR for large models. When entropy is too low, the policy converges prematurely to suboptimal behaviors (entropy collapse); when is too high, uncontrolled stochasticity attenuates learning signals (entropy explosion). Navigating this entropy dilemma is therefore pivotal for scaling RLVR.
3.1
THE Two PERILS OF POLICY ENTROPY
Entropy collapse. Well documented in RLVR (Yu et al., 2025; Cui al., 2025; Zhu al., 2025), collapse occurs when the policy becomes overly deterministic too early. The resulting loss of explo- ration traps training in narrow reasoning modes and limits generalization.
Entropy explosion. At the other extreme, the policy becomes overly stochastic: gradients are swamped by noise, credit assignment deteriorates, and learning turns unstable and inefficient-an equally limiting regime that has been comparatively underexplored (Ahmed et al., 2019; Geist et al., 2019; Haarnoja al., 2018; Xu et al., 2021; Zhang et al., 2025).
The dilemma. Most prior work targets collapse alone. Treating it the sole bottleneck is critical oversight: in practice, mitigating collapse with existing techniques can inadvertently induce explosion. Addressing only one side is insufficient; effective RLVR requires keeping policy entropy within a productive. stable range. We next analyze the mechanisms that drive entropy explosion and motivate remedy.
3

---

Published as a conference paper at ICLR 2026
Artheppmmqnic paesa) Performaoot
DAPO (without Clip-Higher)
DAPO (with Clip-Righer)
0.5
Thai Creath
Plane 21 Deceaping Natina
Corsitted
Saceplig & Pattns
230
250
0.3
1.0
0.5
Training Step
Training Step
Figure 2: DAPO training dynamics on Qwen3-8B. Left: without Clip-Higher; Right: with clip-Higher. In both settings we observe two phases- an early correlated growth between anthropomorphic token frequency and pass@1 1, followed by a decoupling then plateau. While Hlip-Higher averts collapse, it does not prevent the later performance stall.
Aassenig
Other tokera
Step 20
Step 80
Stop 200
6.53
555
6540
0.34
3.40
000
0.000
1150
00
0.00
109
Figure 3: Evolution of high-entropy token usage under DAPO (steps 20/80/200 Early training exhibits diverse anthropomorphic tokens (e.g., wait, perhaps): by steps 80-200 the distribution ho- mogenizes around rigid reasoning templates (e.g., so, let), indicating reduced exploratory diversity consistent with entropy explosion.
3.2
AN ANALYSIS OF ENTROPY EXPLOSION IN RLVR
To investigate the drivers of entropy explosion, we analyze prevalent class of value-free RL meth- ods that apply policy gradients at the token level. We use DAPO (Yu et al., 2025) as a representative case, focusing on its C1 ii-iip-hhhe gher mechanism-al token-level control designed to prevent entropy collapse but, as we will show, one that also illustrates the pitfalls of fine-grained control. Unless oth- erwise noted, we follow the recommended configurations in Yu et al. (2025); full details appear in Appendix B.
Observation 1: Token-level control does not guarantee sustained reasoning gains. In Fig- ure 2, clip-Higher triggers an early spike (steps 20-80) in anthropomorphic tokens- proopoee by Yang et al. (2025b) as markers of "aha-moment" reasoning- coincides with sharp pass @ gains. However, after step 150, anthropomorphic token frequency returns toward baseline while performance plateaus.
Observation 2: Token-level control yields homogenized, low-quality exploration. To probe the stall, we examine the distribution of high-entropy tokens at steps 20, 80, and 200 (cf. Figure 3). Early in training, diverse markers such as wait and perhaps are frequent. By step 80, usage concentrates on assertive, formulaic tokens like and let. This convergence reflects a loss of diversity in high- entropy states: the model increasingly relies on rigid reasoning templates rather than exploring alternatives, aligning with the observed plateau.
4

---

Published as a conference paper at ICLR 2026
(a) Entropy Dynamics
(b) Weight Positive Samples
(c) Weight ht Negative Samples
2.5
Net
GAPO 3/R0 -RR
G8P0 R
Thresho di bo -.,4
Thrediis fased of EAi
Weight
0.0
Figure 4: Quantile baseline reshapes weighting and entropy dynamics. Left: policy entropy over training split by advantage sign negative-advanteee advantagge samples drive the surge. Middle/Right: query- level weights vs. success rate y; GRPO & DAPO use symmetric v p) weighting, whereas our method applies thresholded scheme (K 00.4).
Table 1: Different
chigh values DAPO.
Observation 3: Entropy explosion is disproportionately driven by negative-advantage samples. We decompose entropy dynamics by sam-
Chigh
AIME24
0.20
32.29
18.6%
ple advantage, where positive-advantage samples contribute positive up-
0.22
34.90
12.1%
dates and negative-advaanage samples contribute non-positive updates.
0.24
34.17 13.9%
As shown in Figure 4 (Left), entropy growth is dominated by negative- advantage samples, which show both the steepest increase and the largest share of entropy early in training. Positive-advantage samples remain compafa@8ely Kaall. This
0.26
40.63+2.4%
imbalance indicates over-exploration induced by regative-advantage samples in the early phase, followed by insufficient exploitation later.
Tuning token-level hyperparameters is insufficient. One might lower the token-level high clip threshold thigh to curb update magnitude. Table (varying thigh from 0.20 to 0.28) shows only marginal effects: performance peaks near fhigh = 0.26, but the overall improve- ment is limited and the stae- plateau persists. Simply adjusting token-level clipping cannot resolve the core exploration-exploittiie tension.
Observation 4:
TAKEAWAY
Our analysis indicates that fine-grained, token-level controls provide temporary fix with notable side effects:
They prevent entropy collapse but can inadvertently induce a performance-limitinn en- tropy explosion.
The explosion is mechanically rooted in the advantage baseline, which systematically mishandles negative-advantage samples under reward outliers.
The issue is therefore a baseline-design flaw, not hyperparameter tuning problem the token level.
4 METHOD: QUANTILE-BASED ADVANTAGE ESTIMATION FOR ENTROPY REGULATION
Building on the analysis in Section 3, we identify the advantage baseline as the primary source of instability in RLVR. Value-free methods such GRPO (Shao al., 2024) and DAPO (Yu et al., 2025) use an empirical mean baseline that is sensitive to reward outliers: a few high-reward samples can inflate the baseline, turning otherwise competent responses into regative-advantaa examples and penalizing useful exploration, which induces entropy collapse.
We address this by quantile-be muadtile-based advantage estimation. Replacing the mean with distributional quantile yields a baseline that (i) statistically robust and (ii) explicitly controllable. A single hyperparameter K E E00, 1) shifts the update focus between exploration and exploitation.
5

---

Published as a conference paper at ICLR 2026
4.1 FORMULATION AND INTUITION
For a query y, sample G responses {(o;, R:) with or Toial 1) and binary rewards Rt E {0, 1}. Let
p(a)
1
Σ
R
G
be the empirical success rate under Told- Define the group empirical CDF
1
Pa(x)
1Rg S x))
G
and the (right-continoous) -qquantile baseline
(g) = QK ((R)))=)) inffæ : Fg(x) 2 K}.
K (0.1).
We then define the standardized advantage
R1 bx(g)
A std((Ry) (R +
E> 0.
(3)
where E prevents division by zero when E {0, 1}. For binary rewards, the baseline reduces to threshold on p(g):
0, p(g) S 1-K, bx(4) 1, p(g) > K. This yields two regimes governed by the difficulty threshold 1 K- K:
(4)
Hard (exploitation-focuseed) p(q) S 1-K. The baseline is Incorrect responses (R 0)
have A= 0, while rare correct responses (R receive À 0, reinforcing nascent successful trajectories.
Easy (exploration-focusel) p(q) > 1-K. The baseline is 1. Correct responses have A 0, while remaining failures (R=0) yield Å < 0, discouraging residual failure modes already- solved queries.
Hence K acts a direct lever that regulates policy entropy by switching updates between rare successes (hard) and remaining failures (casy).
4.2 GRADIENT ANALYSIS
We adopt the discriminative perspective of GRPO introduced by DisCO (Li et al., 2025), which separates a query-level weight from a discriminative term. Let क ( q) and Faia (. 4) denote the conditional distributions of responses with rewards 1 and 0, respectively. For response o, let s (o, q) and sg (o, 9) denote score functions based on token- normmnzoma policy ratios for posi- tive/negative examples (see Appendix A.2 for exact forms).
GRPO revisited. ei al. (2025) show that the GRPO objective can be written as
(c) = Eg Vl - p(g)) E
Bown o' a [st (a, 8g (o', 91)
(5)
weight
discriminative term
with symmetric weight that down-weights both very easy and very hard queries (cf. Fig. 4).
Quantile- based objective. Under Eqs. 3-4, the standardized advantage is non-zero only one outcome type per regime. Substituting into a GRPO-style objective yields:
Proposition 4.1 (Quantile-regulated objective). Assume binary rewards, group size G 2, and the right-continuous empirical quantile. Using the standardized advantage in Eqs. 3-4, the learning objective is (up to a constant factor depending equivalent to
Jquantite (0) Eeg 1(p(g) S 1 1-K) /
p(a)
E
E
tul 10) 8/ (0,
I I-p(g)
- -1pp(g) > 1 K} V
1-p(g) p(a)
E (19)) (0'.0)
(6)

---

Published as a conference paper at ICLR 2026
Remark. Please check Appendix A for all proofs. Compared to the GRPO objective in Eq. 5, QAE makes two crucial changes: (i) it selectively nullifies one of the discriminative terms based on query difficulty, and (ii) it replaces the symmetric, bell-shaped weight pp(1 p) with asymmetric, monotonic factors-eithn (p p) for hard queries or v(1 (1-p)/p for easy queries. This trans- forms the update mechanism from focusing on moderately difficult problems to amplifying signals from rare successes residual failures (cf. Fig. 4).
4.3 THEORETICAL ANALYSIS: TWO-REGIME ENTROPY SAFETY
Setup. Adopt a bandit reduction in which producing a full response to is a single action. Let T. (1) be the current softmax policy and (g) the token-averaged (ength-normalize tropy. Let Á denote the GRPO/DAPO token-nowe mormalzzomal advantage (Sec. 4.2); more generally, write As(y.g) 9) = rly. q) - b(4) for the response-leve advantage with baseline b(g). For binary re- wards with group success rate p()), we use the right-continuous K K-quantile baseline bx(g) (Eq. 4), iee, (g) = 0 if p(q) < 1-k
policy en-
K and otherwise. Under first-order logit updates of a softmax policy with step size > 0, the entropy-covarine
identity (adapted from Cui et al. (2025)) yields.
(g -1 Cov π (y) g), Tly q) As(v.))) 17 7>0 0.
Baseline linear knob. For be [0, 1], define (b) Cov=(log (r b)) for (0, By linearity,
Pb(() (O bov (log T), Cov whenever T- ()) is non-uniform. Hence AHl (q: b) = -1 Fb is strictly increasing in be [o, 1].
Proposition 4.2 (Two-regime entropy safety of K-quantíle). Fix and non-uniform x(- q). Then:
1. Low-success (explosion-proof) p(q) S-K so bx(g) = O, then for any baseline b E [0, (including the mean b=p(g) or token-level clipping/KL that keep b unchanged).
AH(q: bx) s (H(
2. High-success (collapse-proof) lfp(g) 1-K so bx(g) = 1, then for any be [0,
AMl (1:bb) bx) 2 ().
Sequences token-level controls. Existing token-level controls are one-sided: they rescale step sizes but leave the response-level baseline b(q) unchanged, so they cannot prevent explosion driven by negative-advantage samples. In contrast, the K-quantile baseline is two-sided (Prop. 4.2): by -0 when p(g) < 1-K (explosion-prool) and bx = when p(q) > 1-K (collapse-proof), matching the two training regimes in Fig. 4.
TAKEAWAY
Method takeaways (QAE).
K-quantile as response-level gate. A single parameter yields a deterministic switch (Fqs.3-4) hard queries (p(g) - update on rare successes only; easy queries (p(g)> 1-K) update on remaining failures only (Fig. 4).
Two-sided entropy safety (provable). Under first-order softmax updates, the K K-quantile baseline attains the extremal one-step entropy miftt-minimal at p(g) < 1-K (prevents explosion) and maximal at p(g) > 1 K (prevents collapse); see Prop. 4.2.
Note: Token-level mechanisms only rescale steps and do not change the response-level base- line, so they cannot realize these guarantees.
EXPERIMENTS
Evaluation protocol. We evaluate three standard math reasoning benchmarks: AIME'24, AIME'25, and AMC'23. All evaluations are zero-shot. For each query we sample k= comple- tions with temperature T=0.7. We report pass @ 1 and pass@ 16 accuracy metrics, together with
7

---

Published as a conference paper at ICLR 2026
Table 2: Overall performance on the 24'2'24/22 and AMC'23 benchmarks. Our drop-in QAE consistently improves pass @ across different models and methods, while maintaining comparable pass@ 16 scores. Red denotes an improvement and blue decline.
AIME25
AIME24
AMC23
Model
Method
Pass@11 1
Pass 16
Pass@!
Pass 16
Pass
Pass 16
Clip-Higher QAE
32.71
56.66
39.69
71.23
92.11
97.50
34.90 -4.7%
57.92+2.25%
48.23-21.5%
+1.63+0.8%
92.97 +..9%
97.50 97.50+0.0%
Qwen3-
CLIP-Cov
33.02
52.27
42.40
68.58
87.42
96.25
8B-Base
QAE
37.40-13.3%
56.29+7.7%%
46.04
66.16+6
90.23+33229
96.25+
KL-Cov
33.33
45.86
44.90
73.00
86.02
95.00
QAE
33.44-0.3%
51.62+12.6%
44.69 -..5%
77.08
87.97 +2.3%
96.25 +.33% 1.33
Qwen3-30B-
GSPO
31.15
46.59
43.75
67.91
90.00
99.39
A3B-Base
QAE
32.50 -4.8%
48.01 +3300%
47.50 8.66%
71.72 66.89
89.38
0.77%
97.21
2.25
Performance Comparison (41MB24)
Entropy Dynamics During Waining
Propertion of Advantage Values
Nostire Advantape Sumples
25
Adsartage
Ceherrs
DAPO
Figure 5: Training dynamics and sparsity. (a) AIME'24 (Qwen3-8B): QAE boosts pass@) I while keeping pass@ 16 comparable--showing higher sample efficiency. (b) Entropy by sign: DAPO's explosion stems from negative-advantage samples; QAE suppresses it. (c) Response sparsity: 80% responses have zero advantage, focusing updates on informative subsets.
the average tokens per response. Unless noted, keep all training and decoding hyper-parameter identical across baselines and our method, changing only the response-lewe baseline from the mean K -quantile (default K -00.4). This value chosen robustly balance exploration and exploita- tion; we present detailed sensitivity analysis on K in Appendix B.3.
5.1 OVERALL PERFORMANCE ACROSS MODELS & RECIPES
Drop-in gains across model sizes. Table summarizes results on Qwwn3-8B-Bas and Qwen3- 30B-A3B-Base Replacing the mean baseline in DAPO with our K-quantile baseline (QAE) yields consistent pass@1 improvements across datasets and model sizes, while keeping pass@ 16 perfor- mance highly comparable. The stability of this process is further illustrated by the training dynamics curves for both 8B and 14B models in Appendix B.4, which show QAE consistently mitigates the entropy explosion seen in the bascline.
Compatibility with strong recipes. QAE orthogonal to token-level controls (e.g. CLIP-COV, KL-Cov) and sequence-level optimization (GSPO). When layered top of these methods, QAE consistently provides further gains without altering their syper-parameterss
5.2 TRAINING DYNAMICS & ENTROPY SAFETY
Pass @ improves while pass @ 16 stays comparable. Figure 5 (Left) plots AIME'24 performance over training for Qwen3-8B-Banl From sstep 100, DAPO exhibits an entropy surge and pass @ stalls, while QAE maintains stable training and continues improve. Pass @ 16 remains similar,
Negative-advantage entropy is the driver of instability. Figure 5 (Middle) decomposes entropy by the sign of the advantage. The growth is dominated by negative-advannage samples; QAE sup- presses this component and keeps the overall entropy within a productive range. This behavior follows directly from using a quantile baseline that weintweights uninformative negatives.
8

---

Published as a conference paper at ICLR 2026
248 Model
[b) Ablation Study
Ablation Study 0.20)
DAPO DAPGHQAE
Postiive Mack
Fostiive Mask
16.46
Mask
Performann
50
39
20
Metrics
Metrica
Figure 6: Performance and ablations. (a) QAE improves DAPO on the 14B model for both AIME'25 and AIME'24 (pass@1). 1). (b) With weaker high-end clipping (Chigh =0.28), controlling updates (NEG-MASK) most critical, closely tracking full QAE. (c) With stronger clipping (chigh-0.20)) positive-advantage control (POS-MASK) dominates.
Response-level sparsity: the 80/20 rule. Figure 5 (Right) shows that x80% of sampled responses have zero advantage throughout training. This "response-level 80/20 rule" focuses updates on the informative minority, explaining QAE's stability and efficiency. In contrast to the bascline, which leads to homogenized exploration (Sec. 3.2), QAE sustains a productive co-growth of diverse ex- ploratory tokens and reasoning accuracy, detailed in Appendix B.2.
5.3 ABLATIONS COMPOSITION
Masking mechanisms. QAE can be viewed as selectively masking updates. To disentangle their roles, we define two one-sided objectives:
JAOS-MASK (0) = =E,, (plo)s1-K) V
plg) T- p(1) o
+8 (0, -
1-pla) p(g)
50 (d) (7)
0 D'
JNEG-MASK (0)=
p(g) =0= [ [V I-pla) Bons (o. (p(g)>11K)
1-pla) p(g)
E Borna Sp (0). (8)
Masking mechanisms. QAE can be interpreted masking positives easy queries and nega- rives on hard queries. We isolate each side by constructing two objectives: POs-MASK (Eq. 7) and NEG-MASK (Eq. 8), leaving the other side unmasked.
Explosion collapse regimes. As shown in Fig. 6 (b-c), when the high-end clipping is weak (0.2), the dominant failure mode entropy explosion; NEG-MASK nearly matches QAE and outperforms POS-MASK. With strong clipping (0.20), collapse pressure dominates and the ordering flips (POS-MASK > NEG-MASK). This matches the two-regime analysis in Sec. 4.3.
DISCUSSION
K entropy-guided exploration-exploitatoo knob. We single hyperparame- ter K to control how many responses receive nonzero advantage, thereby steering the explo- ration-exploitation trade-off by modulating entropy. Operational rule-of-thumb: per baseline by inspecting the entropy of the baseline policy, rather than the evaluation metric. When entropy is low (risk of mode collapse), choose K 0.6 to inject diversity; when entropy is high (risk of unstable updates), choose K = 0.4 to temper exploration. Since all our recipes use Clip-Hi Hi gher, we default to 0.4; finer-grained tuning can yield further gains.
select K once
QAE prioritizes who learns over how much. Updating only a small subset of samples (~ 20%; Figure 5 (Right)) makes RLVR more stable and improves scaling behavior, indicating that selection, not update magnitude, is the primary bottleneck. QAE implements a binary reward with a quantile baseline at the query level: for difficult queries it assigns credit successes, whereas for easy queries assigns credit to failures. By adjusting the masking range- or by introducing dual masks- naive DAPO/GRD PO reduce special cases, providing safe fallback within the same framework.

---

Published as a conference paper at ICLR 2026
Baseline design third knob for entropy control. Prior work has studied positive/negative ratios (Zhu et al., 2025), entropy dynamics (Cui et al., 2025), and advantage shaping (Cheng et al., 2025). QAE is orthogonal: it uses baseline design-a shift from the mean to the --quantile-ar the primary entropy lever and composes cleanly with existing techniques (Table 2). Related to Arnal et al. (2025), which analyzes tunable baselines in REINFORCE, our quantile baseline is data-adaptive, group-level instantiation that improves robustness while preserving standard policy- gradient updates.
RELATED WORK
Reinforcement learning for LLM RL has become key technique for eliciting advanced reason- ing in large language models (LLMs), a paradigm shift from its earlier applications preference alignment via RLHF (Ouyang et al., 2022). This modern approach, termed Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024; Mroueh, 2025), leverages outcome-bart based optimization to achieve state-of-thh-an art performance in complex domains like mathematics and pro- gramming. Seminal works, including OpenAl's ol (ope) and DeepSeek R1 (DeepSeek-Al et al., 2025), demonstrated that RI can effectively scale reasoning capabilities, spurring new line of research (Yang et al., 2025a; Team et al., 2025). Central to this progress are online, value-free al- gorithms that have generally outperformed offline preference optimization methods (Rafailov al., 2023; Wu et al., 2024; 2025). In particular, Group Relative Policy Optimization (GRPO) (Shao al., 2024) and its successor, Dynamic Sampling Policy Optimization (DAPO) (Yu et al., 2025), have emerged as foundational baselines for many contemporary reasoning systems (Yue et al., 2025; Zeng et al., 2025; Hu al., 2025). Our work uses DAPO as a representative algorithm to investigate critical, unresolved challenge in this domain: the training instability caused by dysregulated policy entropy, which limits the performance and scalability of current RLVR methods.
Exploration, entropy dynamics, and collapse/explosion in RLVR. Existing RLVR entropy search follows three strands: (i) Mechanistic analyses identifying where exploration concentrates, such high-entropy "forking" tokens (Wang al., 2025b) "thinking tokens" (Qian al., 2025), and the dynamics of sequence-level collapse or explosion (Cui et al., 2025); (ii) Objective-level reg- ulation steering entropy via modified optimization targets (Agarwal et al., 2025; Zhu et al., 2025; Zhang et al., 2025) or regularized MDP scheduling (Geist et al., 2019; Ahmed et al., 2019; Xu et al., heuristics injecting exploration via advantage shaping (Cheng et al. 2025), Pass @1 k training (Chen et al., 2025), agentic scaffolds (Zhou et al., 2025; Shang et al., 2025), and modulated gradients (Wang et al., 2025a; Song al., 2025). Despite these advances, a baseline-level entropy control that is data-adaptive yet preserves standard policy- -riiieggraieen updates remains missing. Our work fills this gap using quantile baseline and binary masking, offering a drop-in lever that complements existing methods while explicitly targeting stability.
2021); and (ii) Recipe/system-Iert
8 CONCLUSION
Conclusion We propose Quantile Advantage Estimation (QAE), replacing the mean baseline with group-wise K-quantile implement a two-regime gate that amplifies rare successes and sup- presses residual failures. Under first-order policy updates, QAE provides two-sided entropy control with bounded one-step entropy change, curbing both collapse and explosion. Empirically, QAE stabilizes entropy, sparsifies credit assignment, and improves @ass@1 I across reasoning benchmarks while composing cleanly with standard sequence- and token-level controls.
Limitations and Future Work (i) Dynamic K: Beyond fixed K, explore simple schedules or
two-phase curricula to better balance exploration and exploitation; (ii) Automatic K: Adapt K to model state (e.g., success rate, entropy, gradient variance) to remove manual tuning; (ii) PPO integration: Embed the quantile-baselin baseline idea into PPO's whiteeingg/oommlizattoonn -en.. batch- wise quantile baselines- test robustness across algorithms and scales.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research supported by the National Natural Science Foundation of China (U25A20445, 62572449, 62525211, I, 62302321). This research also benefited from the advanced computing re- sources provided by the Supercomputing Center of the USTC.
REFERENCES
Learning to Reason with LLMs.
URL
learning to- recoo-resooo--ee wreessnwwtth-aeeeeeeeeeaeeeeeeee with-11ms//
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effee- tiveness of entropy minimization in LLM reasoning. arXiv preprint 2rXiv:5555.51344 2025.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In JCML, 2019.
Charles Arnal, Gaëtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Rémi Munos. Asymmetric REINFORCE for off-policy reinforcement learning: Balancing positive and negative rewards. CoRR, abs/2506.20520, 2025.
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Zheng Liu, and Wayne Xin Zhao. Pass@ training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508. 10751, 2025. URL https: / attpaaiiiiiv..i org/ abs/ 10751.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758, 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models. CoRR, abs/2505.220 22611,, 2025.
DeepSeek-AL, Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang. Minghui Tang, Meng Li. Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Yc, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and Li. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. CoRR, abs/2501. 12948, 2025.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In ICML, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Godfrey Harold Hardy, John Edensor Littlewood, and George Pólya. Inequalities. university press, 1952.
Cambridge
Jingcheng Hu. Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. rean-rrasone--zroo An open source approach to scaling up reinforcement learning on the base model. CoRR, abs/2503.24200 2025.
11

---

Published as a conference paper at ICLR 2026
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. CoRR, Abs/2411.15122 2024.
Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, and Tianbao Yang. Disco: Reinforcing large reasoning models with discriminative constrained optimization. CoRR, abs/2505. 12366, 2025.
Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and success amplification. CoRR, abs/2503.06m5 066699, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Daron Anderson, Tung Nguyen, Mobeen Mahmood, Fiona Feng. Steven Y. Feng, Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Jessica P. Wang, Pawan Kumar, Oleksandr Pokutnyi, Robert Gerbicz, Serguei Popov, John-Clark Levin, Mstyslav Kazakov, Johannes Schmitt, Geoff Galgon, Alvaro Sanchez, Yongki Lee, Will Yeadon, Scott Sauers, Marc Roth, Chidozie Agu, Søren Riis, Fabian Giska, Saiteja Utpala, Zachary Giboney, Gashaw M. Goshu, Joan of Arc Xavier, Sarah-Janc Crowson, Mohinder Maheshbhai Naiya, Noah Burns, Lennart Finke, Zerui Cheng, Hyunwoo Park, Francesco Fournier- Facio, John Wydallis, Mark Nandor, Ankit Singh, Tim Gehrunger, Jiaqi Cai, Ben McCarty, Darling Duclosel, Jung- bae Nam. Jennifer Zampese, Ryan G. Hoerr, Aras Bacho, Gautier Abou Loume, Abdallah Galal, Hangrui Cao, Alexis C. Garretson, Damien Sileo, Qiuyu Ren, Doru Cojoc, Pavel Arkhipov, Us- man Qazi, Lianghui Li, Sumeet Motwani, Christian Schröder de Witt, Edwin Taylor, Johannes Veith, Eric Singer, Taylor D. Hartman, Paolo Rissone, Jachyeok Jin, Jack Wei Lun Shi. Chris G. Willcocks, Joshua Robinson, Aleksandar Mikov, Ameya Prabhu, Longke Tang, Xavier Alapont, Justine Leon Uro, Kevin Zhou, Emily de Oliveira Santos, Andrey Pupasov Maksimov, Edward Vendrow, Kengo Zenitani, Julien Guillod, Yuqi Li, Joshua Vendrow, Vladyslav Kuchkin, and Ng Ze-An. Humanity's 's last exam. CoRR, abs/2501. 2025.
Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. Demystifying reason- ing dynamies with mutual information: Thinking tokens are information peaks in Ilm reasoning. arXiv preprint (xXX::2506002666 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. CoRR, abs/2311.12022 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, fbs/1707.06344 2017. URL http://aaxxv..rg/// 1707.06347.
Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, Ying Xin, Ziming Miao, Scarlett Li, Fan Yang, and Mao Yang. rstar2-agent: Agentic reasoning technical report. arXiv preprint arXiv:2588.20722 2025. URL https:/::
org/ abs/ 2508 20722.
Zhihong Shao. Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu. and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.033000, 2024.
12

---

Published as a conference paper at ICLR 2026
Yuda Song, Julia Kempe, and Rémi Munos. Outcome-based exploration for Ilm reasoning. arXiv preprint arXiv:2509.0660 06941, 2025. URL
org. atd//abs/2500 2509. 06941.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun
Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu. Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang. Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang. Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianging Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du Yuxin Wu, Yuzhi Wang. Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with llss. CoRR, abs/2501. 122599.,.. 2025.
Jiawei Wang, Jiacai Liu, Yuqian Fu, Yingru Li, Xintao Wang, Yuan Lin, Yu Yue, Lin Zhang, Yang Wang, and Ke Wang. Harnessing uncertainty: Entropy- policy gradients for long- horizon Ilm agents. arXiv preprint arXiv:2509.99265 2025a. URL https: /aaxivxxvvv../// org// abs/2509 09265.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01933 2025b.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 8-dpo: Direct preference optimization with dynamic B. In NeurIPS, 2024.
Junkang Wu, Kexin Huang, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, and Xiang Wang. Repo: Relu-based preference optimization. CoRR, Abs/2503.07426 2025.
Yaosheng Xu, Dailin Hu, Litian Liang, Stephen McAleer, Pieter Abbeel, and Roy Fox. Target entropy annealing for discrete soft ccior- In NeurIPS 2021 Workshop, 2021.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao. Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505.09 09388. 2025a.
Shu Yang, Junchao Wu, Xin Chen, Yunze Xiao, Xinyi Yang, Derek F. Wong, and Di Wang. Understanding aha moments: from external observations to internal mechanisms. CoRR, abs/2504.02956, 2025b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system at scale. CoRR, abs/2503. 14476, 2025.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Cheng- Xiang Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang. Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks. CoRR, abs/2504.00 1.10000011111 18, 2025.
13

---

Published as a conference paper at ICLR 2026
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- Z0o: Investigating and taming zero reinforcement learning for open base models in the wild. CoRR, 1ass2203311892. 2025.
Ruipeng Zhang, Ya-Chien Chang, and Sicun Gao. When maximum entropy misleads policy opti- mization. In ICML, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, Yang. Jingren Zhou, and Junyang Lin. Group sequence policy op- timization. CoRR, abs/2507. 18071, 2025.
Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Jiale Zhao, Jingwen Yang, Jianwei Lv, Kongcheng Zhang, Yihe Zhou, Hengtong Lu, Wei Chen, Yan Xie, and Mingli Song. Breaking the exploration bottleneck: Rubric-scaffolded reinforcement learning for general Ilm reasoning. arXiv preprint arXiv:2508. 16949, 2025. URL wwwtthttss//aarte org /ab////////// 2508 16949.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness negative reinforcement in LLM reasoning. CoRR, abs/2506.01 05344767777, 2025.
14

---

Published as a conference paper at ICLR 2026
A PROOF
A.1 PROOF OF PROPOSITION 4.1
Proposition 4.1 (Quantie-regalateed objective). Assume binary rewards, group size G22, 2, and the right-continuus empirical quantile. Using the standardized advantage in Eqs. 3-4, the learning objective is (up to a constant factor depending on E) equivalent to
Iquantilo (0) = Egg [(p(g) s 11-K)
p(g)
A (0,
I-P())
- -1(p(g) > 1- K}
1-p(g) p(g)
I
(10) (0) (0'00)
(6)
Proof. Write p = p(g) for brevity. Recall the torma-nor surrogate
lol
To (o, q. g,0ct) Tor q 0et)
A(o l0))
O(0) By (o
and the positive/negative homogeneous scaling of f (the same convention as in the main text):
(e ((x,0) (100) 5-x,
e (eft(r, 1x,
<> 0,
=
(c > 0). (10)
0,
For the binary reward r(o 9) (0, 1} and the group statistics Eoo-so((lo)) 9) p and Vato q) p(1 - p). the standardized advantage used in the paper takes the form
r(o ②) =1
A(o ②)
(11)
r(o q) 0.
Under the K-quantile baseline described in Section 4 (right-comtinuous)) responses are masked asymmetrically by the regime of p:
ff K: A+ = vp(1 p) p) (4 =
(12)
1
> 1-> A (*) = 0, (g vp(1 p)
(13)
Equivalently, among (r 1,r 0} only one label contributes in each regime.
Plug equation 12 into equation 9 and decompose over E (1.0} (writing कt (- g) and mo (- (1]g)
for no(- q) conditioned on and r = O, respectively):
(o gg,0c))
[(0) E. 110 1 K} p omar (:10)
(14)
mn(o: q, ((e)
Vp(1 -p)p
+ 1p K} (1 p) E o (:1g
Σs
To(or 9. g,0c))
1
To(at g. (.0)))
Vp(1
Apply the homogeneity equation 10 separately to the two terms in equation 14. For porp& S 1 scalar is positive, and for p > 1 is negative, hence
K the
To(o, q, (1,021) To(o: (,00ee)
O(0) F. 11p 1
E V VI-p
Σ
(15)
om
(-10) Tol
To(or g,ocz) Tolor q, q,
-1pp> 1-p>1-K]
E on oe (110 1o
E
p
15

---

Published as a conference paper at ICLR 2026
Equation 15 is the claimed quantile-regulate objective: compared with the symmetric GRPO/DAPO weight /p(L p), the quantile bascline (i) masks one side (positives easy queries
with > K or negatives on hard queries with I 1 K) and (ii) re-weights the active side by the asymmetric factors /p/(1 - p) or v(1 - p)/p. p. This completes the proof.
Instantiating f for GRPO. For GRPO
(*, 1) min clip(r, 1 + c)) = min(2, , 1 + c); ((x 1) max clip - 1 e)) max(2z c),,
(16)
(17)
which can be plugged into equation 15 directly.
A.2 PROOF OF PROPOSITION 4.2
Proposition 4.2 (Two-regime entropy safety of K-quantile). Fix q and a non-uniform x(: Then:
q).
1. Low-success (explosion-rroof). If p(g) 1-K so (g) 0. then for any baseline b E [0, (including the mean b=p(g) or token-level clipping/KL that keep b unchanged),
AH( s (H)
2. High-success (collapse-proof). If p(g) 1-K so bx (g) = then for any be [0, 1].
AH( 2 A (g
Proof. Fix q and non-uniform softmax policy x(- q). For any baseline b E [0, 1] and binary reward {0, 1}, write
An(y, g) r(y,g) bb, b2(b) = Cov Covggn((-9g (log ( 4), ( q) (r(v. q) - b)).
The entropy-covariance identity for softmax policies under first-order logit updates (adapted from Cui et al. (2025) gives
() -n Fg()),
7 ワ> 0.
(18)
Step 1: Baseline monotonicity. By bilinearity of covariance,
(b) = Cov Cov,(log n, ar) b bCv v, π) =: Fq b..
(19)
Let U' = =T q) for Y ~ T-- g). Then C. = Cov U, U). Since u + log and u + strictly increasing (0, 1], they co-monotone; hence Cov U) 0 whenever U is non-constant, i.c., whenever x(. q) is non-uniform (see, e.g., Chebyshev's sum / rearrangement inequality (Hardy et al., 1952)). Therefore C2 and equation 19 shows that (b( is strictly decreasing in b, so by equation 18 the entropy change AH (g; b) is strictly increasing in e [0, 1].
Step 2: Two-regime extremality of the K-quantile baseline. For Bernoulli rewards with success rate p(g). the K -qanttile baseline is
[0, p(g) - K, [1, p(g) -1-, K,
bk(g)=
(Eq. 4).
Because AH (g: b) increases in b (Step 1), we have, for any b E [0, 1].
p(g) S 1 K + (4)
min (0, 1] + AH bx) S AH(gb))
p(g) 1--K K (x( Strict inequalities hold whenever IT.- q) non-uniform and b + bx (g). These exactly Items (1) and (2) of Proposition 4.2.
max[0, AHl bx) 2 AHl b).
This establishes the claimed two-regime entropy safety: in the low-success regime (p S 1 K)
the quantile choice by = minimizes the entropy increment (explosion-proof), whereas in the high-success regime (p > 1 - K) the choice bx maximizes it (collapse-proof)
16

---

Published as a conference paper at ICLR 2026
Trahhing Stop
Figure 7: High-entropy token diagnostics under QAE. Green bars: counts of anthropomorphic high-entropy tokens; orange line: overall pass@1. Early coupled growth transitions to later de- coupling-toker counts plateau while accuracy improves-indicaltie entropy-safe, selective explo- ration.
B EXPERIMENTS
B.1 IMPLEMENTATION DETAILS
Experimental Setup:
Our configuration includes clip-higher, dynamic sampling, token-level pol- icy gradient loss, and overlong reward shaping, as proposed in DAPO. We use the recommended hyperparameters: Chigh 0.28 and éjow 0.2 for clip-higher, and maximum response length of 20,480 with a 4.096-token cache for reward shaping.
Training Details: We train with a global batch size of 512, using 16 gradient accumulation steps with mini-batch size of 32. The learning rate is fixed at 10-6 with no warmup or decay schedule. Importantly, we exclude both KL divergence and entropy losses.
Evaluation: To analyze scaling effects, we apply this method to the Qwen3-32B and Qwen3-8B base models, training them on the DAPO-Math- 17K dataset (Yu et al., 2025).
Additional Experiments:
We also conduct a cold-start experiment with the GSPO algorithm, ini- model. In this configuratio,, we use four gradient aceu-
tializing from the Qwen3-30B-A3B. mulation steps per batch. The GSPO clipping ranges are set to 3 10-4 (left) and x 10 (right), aligning with the official VERL implementation script".
B.2 MORE EXPERIMENTS
QAE sustains co-growth of "aba" markers and accuracy. Contrasting with Clip-Higher, Fig. shows that under QAE the amthropomorphic token count and pass@) I rise together across training. From early to late steps, the green bars ("aha" markers) increase and remain elevated, while the orange curve improves monotonically, indicating that exploration is converted into productive reasoning rather than unchecked entropy.
High-entropy token diagnosties under QAE (fine-grained snapshots). A finer finer-grained inspec- tion at representative steps 20/80/200 Fig. 8 corroborates this interpretation. At step 20, an- thropomorphic markers sparse, consistent with exploration just being activated; by step 80, these tokens separate more distinctly, aligning with the performance uptick seen in the coupled-growth regime; by step 200, their counts stabilize despite continued pass @ gains, evidencing shift from "more randomness" to targeted refinement. Taken together with the trajectory view, these snapshots confirm that QAE leverages high-entropy branches when beneficial and then curbs their proliferation once they cease to deliver marginal utility.
httpps: / thub. www.vol math.sh
17

---

Published as a conference paper at ICLR 2026
Step 20
Step 80
Step 200
0.50
0:64
2223
0.00
1221
339
0.110
1809
Figure& 8: Token-level diagnostics. Probability mass top high-entropy tokens different train- ing steps. Under QAE, exploratory tokens increase in a controlled manner, aligning with the stable- entropy regime in Fig. 5b.
Entrapy
Accuracy
Responss Langth
C-03
C-0.1
Neag Length
100
10:00
155
2.0
2.5
380
Figure 9: Training curves under different K Qwen3-8B-Base. Leftt: entropy; middle: accuracy
(AIME24@ 32); right: response length.
B.3 QUANTILE PARAMETER ANALYSIS
Trade-off's governed by K. Figure 9 (leftmmiddlerrihh)) shows how the quantile K tunes entropy, accuracy, and response length for Qwen3-8B-Base Large K (e.g.. 0.8) marks most samples as rggative-avvantage driving entropy upward, inflating response length, and yielding volatile training with an early accuracy plateau. Small (e.g.. 0.2) marks most samples as positive-advantage producing low-entropy, over-regularized regime that is stable but exploration-pont with limited accuracy gains. These trends align with Sec. 4: K simultaneously sets the share of responses updated and the direction of entropy flow.
Stability at 0.4 (with Clip-Higher). All main experiments use =004 with ehigh = 0.288 This configuration avoids the high-entropy instability seen at K = 0.8 while maintaining sufficient stochasticity to prevent collapse. Empirically yields bounded entropy (Fig. 9, left), stable lengths (right), and sustained accuracy improvements (middle), striking a robust balance and matching the two-sided entropy safety predicted by our analysis.
B.4 ANALYSIS TRAINING DYNAMICS ON 8B AND 14B MODELS
QAE stabilizes entropy and sustains performance gains across model scales. We compare the baseline DAPO with DAPO+QAE on Owen3-8B-Bass (Figure 10) and Qwen3-14B-BB Base (Fig- ure 11). Across both model sizes, QAE consistently reduces and stabilizes policy entropy, keeps response length bounded, and yields smoother, longer-lasting accuracy improvements.
On Qwen3-8B, the mean-baselin variant exhibits a pronounced entropy surge around step 100, accompanied by divergence in response length and a subsequent accuracy plateau. In contrast, QAE maintains entropy within productive range throughout training and avoids the plateau, leading to sustained accuracy gains in later stages.
18

---

Published as a conference paper at ICLR 2026
Entragy
Accuracy
Response Length
0.45
0.00
0.0
040
680
Step
Figure 10: Training curves under DAPO and DAPO + QAE on Qwen3-8B-Ban Left: entropy; middle: accuracy (AIME24@32); right: response length.
Estropy
Accuracy
Response Longth
Nespasa Langsh
2030
142 140.00
3tap
STop
30ap
Figure 11: Training curves under DAPO and DAPO + QAE on Swen3-14B-Base Left: entropy; middle: accuracy (AIME24032) right: response length.
The same pattern appears on Dwen3-14B. Although the entropy spike under the baseline is less severe, entropy remains higher and more volatile than with QAE. QAE again moderates entropy and response length and produces smoother, more monotonic accuracy trajectory. Taken together, these results indicate that QAE addresses the sensitivity of the mean baseline in value-frec RL train- ing and that principled baseline design provides effective mechanism for scale-robust entropy control in RLVR.
19


Reviews:
[{"review_id": "VovCkfpEzu", "reviewer": "Reviewer_LBMg", "summary": "This work proposes an alternative to the mean baseline commonly used in RL advantage calculations. It shows that the mean baseline focuses learning on moderately difficult problems, while the proposed approach focuses it on rare successes and residual failures, sparsifying the LLM outputs that contribute to the gradient. The method is well supported by theoretical and empirical evidence, which comes from evaluations on three math reasoning benchmarks.", "questions": "Line 115: this should be $\\pi_{ref}$, not $\\pi_{old}$, for the KL reg.\n\nLine 146: Can you provide evidence/citations/intuition for statements about the dangers of entropy explosion like this one?\n- “Gradients are swamped by noise” \n\nLine 200: In this sentence, the idea that entropy explosion limits training is stated as if there is a proven causal relationship, but the experiments only show correlation between entropy growth and limited performance. \n- “Thus, although Clip-Higher mitigates early collapse, its rapid escalation is coupled with entropy explosion that ultimately limits scaling.”\n\nLine 244: I think a comparison of entropy between DAPO with and without clip-higher is needed to support this statement. Currently, only anthropomorphic token counts are compared (Figure 2), not entropy.\n\nKL divergence regularization can mitigate over-optimization and preserve diversity. Can it help address the entropy explosion that you find to be driven by negative samples?\n\nHow would the proposed approach perform on non-Qwen models, on models that have very low initial success rates, or in settings with fewer responses sampled per prompt? You could address this question by training Llama 3.1 8B base on MATH with k=16, for example. It would also be interesting to see Reasoning Gym results.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality** Addressing entropy collapse/explosion by altering the advantage’s baseline is novel.\n\n**Quality** The two regimes induced by the proposed baseline are intuitively beneficial and supported by experiments with multiple models and training objectives. Also, mathematical analysis motivates and supports the approach well.\n\n**Clarity** The paper is well written. The figures and tables clearly communicate results.\n\n**Significance** RL with LLMs is a highly impactful area, and this work illustrates a simple way to keep entropy in a helpful regime, avoiding harmful growth/collapse. The community will likely be interested in this.", "weaknesses": "Some of the statements lack sufficient support. See the questions/suggestions section below for more details and how to address this.\n\nIt's left unclear if the proposed method's gains appear with other models and with other, non-math RL problems (e.g. coding or game playing). I think the method is still valuable if it's only relevant to a subset of RL problems, but its broad applicability (or the absence thereof) should be mentioned in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P0P5qzPPMs", "reviewer": "Reviewer_Z6EA", "summary": "They propose Quantile Advantage Estimation (QAE), which replaces the mean with a K-quantile baseline, forming a two-regime gating mechanism that reinforces rare successes on hard queries and penalizes remaining failures on easy ones. Theoretically, QAE provides two-sided entropy safety, bounding both over-exploration and over-exploitation. Experiments on AIME’24/’25 and AMC’23 with Qwen3-8B/14B/30B show consistent pass@1 gains while keeping pass@16 stable.", "questions": "listed in weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The presentation of the paper is clear.\n\n2. QAE requires minimal modification, just a quantile replacement, but achieves consistent empirical improvements and stability, making it highly practical for large-scale RLVR systems.", "weaknesses": "1. The model settings appear inconsistent across the paper. The abstract mentions Qwen3-8B/14B-Base, while Table 2 reports results for Qwen3-8B-Base and Qwen3-30B-A3B-Base. In Appendix D.1, Qwen3-32B is also referenced. However, only two of these models are actually reported in Table 2, and the appendix does not include the missing results. This inconsistency makes the experimental setup confusing.\n\n2.  The benchmarks used (AIME-24 (30 problems), AIME-25 (30), AMC (40)) have small test sizes and inherently high variance (even with repeated sampling), meaning that a difference of one or two problems could significantly change the reported pass@1 score. Since the reported gains are typically around 1–2 examples, it would strengthen the paper to include larger or more diverse benchmarks to verify the robustness of the improvement.\n\n3.  In DAPO training, responses with zero advantage are discarded and resampled to maintain batch size (dynamic sampling). Since QAE introduces more zero-advantage responses, does this increase training overhead or cause slower convergence? Clarifying this would help assess the practical efficiency of the method.\n\n4. Do the authors consider evaluating the methods in other areas other than math?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fWPeFoxWY9", "reviewer": "Reviewer_4HDB", "summary": "This paper proposes Quantile Advantage Estimation (QAE) for RLHF. The key insight is that existing mean-based advantage estimation often suffers from either entropy collapse (limited exploration) or entropy explosion (limited exploitation). QAE addresses this by replacing the mean with a K-median estimator, enabling a better balance between exploration and exploitation through the choice of K. The paper also provides analytical insights into the dual benefits of this approach and presents encouraging empirical improvements.", "questions": "Thank you for submitting your work to ICLR 2026. The paper is well-written, clearly motivated, and proposes a friendly modification that addresses a meaningful issue in RLHF training stability. However, the contribution is somewhat incremental, and the evaluation setup makes it difficult to assess the robustness of QAE's improvement over state-of-the-art (SOTA) methods.\n\n- Q1: How does QAE perform on Qwen3-8B with GSPO? The reported results are promising, but given the incomplete baseline coverage across experiments, it is hard to gauge its competitiveness against SOTA approaches.\n\n- Q2: Can the authors include additional datasets or tasks, such as LiveCodeBench, to test the generality of QAE beyond math reasoning tasks?\n\n- Q3: It would strengthen the paper to align the experimental settings with SOTA baselines (e.g., using GSPO configurations and pass@32 metrics as reported in prior work). Even partial replication or ablation under comparable setups would make the evaluation more convincing.\n\nFinally, I appreciate the discussion on future directions, especially the idea of dynamic $K$. Exploring adaptive quantile strategies could meaningfully enhance both the novelty and practical utility of the approach.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ QAE is compatible with many existing RLHF algorithms, including GRPO and GSPO.\n+ The design is simple, intuitive, and easy to integrate.\n+ Evaluation results show good accuracy gains.", "weaknesses": "- The novelty is incremental, primarily modifying mean advantage estimation to a median-based variant.\n- Evaluations are limited to three math reasoning tasks, and the baselines are not fully consistent across settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjNiGJ75OK", "reviewer": "Reviewer_GY1t", "summary": "This paper addresses entropy instability in Reinforcement Learning with Verifiable Rewards (RLVR). The authors observe that standard training algorithms can suffer from either entropy collapse or entropy explosion. The paper traces this instability to the mean-based advantage baseline which is overly sensitive to reward outliers. The authors then propose Quantile Advantage Estimation (QAE) that replaces the mean baseline with a group-wise quantile. For hard queries, it exclusively reinforces successes, while it exclusively penalizes failures for easy ones. This mechanism is shown to stabilize entropy and produce good performance gains on challenging reasoning benchmarks.", "questions": "- The paper argues that anthropomorphic tokens are markers of an unproductive entropy explosion, yet Figure 2 shows their initial rise correlates strongly with the initial pass@1 gains. This feels strange to me. How would Figure 2 and 3 look like if you had applied QAE? Does it reduce the frequency of anthropomorphic token frequency initially? Does that affect performance?\n\n- I am not sure about the statement in Line 201. I am convinced that ClipHigher mitigates early collapse. However, in both DAPO with and without Clip-Higher, the pass@1 performance seems stable after a few steps (although a bit less without Clip-Higher). Hence, it is not clear to me that entropy explosion is the culprit behind limited scaling. Also, out of curiosity if you have results, how would the training dynamics in Figure 2 and 3 look like for pass@16? And what was the $\\epsilon_\\text{high}$ value you considered in Figures 2 and 3?\n\n- Negative-advantage samples drive entropy explosion, but what drives the collapse? Both positive and negative samples have virtually the same collapse in Figure 4.\n\n- Can you explain line 428? What is the sample efficiency reading?\n\n- Can you explain your results in Figure 6 (b, c)? I encourage you to elaborate on the connection with your theoretical results.\n\nI would suggest citing [1] because they study the learning dynamics in RLVR very carefully. The results there might be interesting to you. \n\n[1] Mroueh, Y. (2025). Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification. arXiv preprint arXiv:2503.06639.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "A primary strength of this work is its clear and insightful diagnosis of entropy instability, both its collapse and explosion, in RLVR. This allows the authors to provide an intuitive, theoretically motivated  (Proposition 4.2), and practically appealing solution. \n\nThey replace the advantage baseline with a quantile. The decomposition of the learning objective (Proposition 4.1) further clarifies why the quantile baseline reshapes the learning signal. I also especially like Figure 4 for showing that negative-advantage samples drive entropy explosion.\n\nThe experiments are well-done, illustrative, and extremely beneficial to the readers!\n\nI also particularly appreciate that QAE is a tool that fits nicely in the the broader RLVR ecosystem. The results in Table 2 are particularly strong, showing that QAE can be layered on top of diverse and powerful existing methods", "weaknesses": "- The core mechanism is a hard threshold based on the quantile K. This means a tiny change in the success rate around the threshold can cause a dramatic flip in the learning objective. This could introduce its own form of training instability, especially with noisy or borderline cases. This may be fine in the case of binary rewards might will be an issue for more general setups with continuous rewards.  Even with binary rewards, the paper uses a fixed G=32. How does the method's performance change with smaller G? For a small G (e.g., 4 or 8), the empirical p(q) will be extremely noisy, which could make the hard-thresholding gate flip erratically,\n\n- The paper frames the ~80% zero-advantage rate as a positive aspect. This can also be viewed as inefficient data use. In tasks that are less sparse or require learning a wider variety of skills, ignoring the learning signal from 80% of generated samples could be highly detrimental.\n\n- I am not exactly sure that the entropy instability is causing the shy gains in performance. Can you pinpoint to which of your experiments tell us we need to control entropy because it is shown to causally worsen performance? My impression is  that QAE's success doesn't come from \"taming entropy.\" It comes from creating a more effective learning curriculum. By setting the baseline to 1 on easy queries, QAE effectively tells the model, \"Stop wasting capacity on problems you've already solved.\" By setting the baseline to 0 on hard queries, it tells the model, \"Focus all your attention on learning from the rare successes you find on these hard problems.\" I would love your thoughts on this based on your experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)