Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
Authors: Peter Chen
Xiaopeng Li
Ziniu Li
Wotao Yin
Xi Chen
Tianyi Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning with Verifiable Rewards
Group Relative Policy Optimization
LLM Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: This paper examines the exploration–exploitation trade-off in reinforcement learning with verifiable rewards (RLVR), a framework for improving the reasoning of Large Language Models (LLMs). Recent studies suggest that RLVR can elicit strong mathematical reasoning in LLMs through two seemingly paradoxical mechanisms: \textit{spurious rewards}, which suppress exploitation by rewarding outcomes unrelated to the ground truth, and \textit{entropy minimization}, which suppresses exploration by pushing the model toward more confident and deterministic outputs, highlighting a puzzling dynamic: both discouraging exploitation and discouraging exploration improve reasoning performance, yet the underlying principles that reconcile these effects remain poorly understood. We focus on two fundamental questions: (i) how policy entropy relates to performance, and (ii) whether spurious rewards yield gains, potentially through the interplay of clipping bias and model contamination. Our results show that clipping bias under spurious rewards reduces policy entropy, leading to more confident and deterministic outputs, while entropy minimization alone is insufficient for improvement. We further propose a reward-misalignment model explaining why spurious rewards can enhance performance beyond contaminated settings. Our findings clarify the mechanisms behind spurious-reward benefits and provide principles for more effective RLVR training.

Published as a conference paper at ICLR 2026
EXPLORATION VS EXPLOITATION: RETHINKING RLVR THROUGH CLIPPING, ENTROPY, AND SPURIOUS REWARD
Peter Chen'
Xiaopeng Li²
Ziniu Li
Wotao Yin
Xi Chen'
Tianyi Linè
CColumbia
CUHK SZ 3AMO. Alibaba US
4NYU Stern
ABSTRACT
This paper examines the iuploration-cxplottaiii trade-off in reinforcement learn- ing with verifiable rewards (RLVR). framework for improving the reasoning of Large Language Models (LLMs). Recent studies suggest that RLVR can elicit strong mathematical reasoning in LLMss through two seemingly paradoxical mech- anisms: spurious rewards, which suppress exploitation by rewarding outcomes unrelated to the ground truth, and entropy minimization, which suppresses ex- ploration by pushing the model toward more confident and deterministic outputs, highlighting a puzzling dynamic: both discouraging exploitation and discouraging exploration improve reasoning performance, yet the underlying principles that reconcile these effects remain poorly understood. We focus on two fundamental questions: (i) how policy entropy relates performance, and (ii) whether spu- rious rewards yield gains, potentially through the interplay of clipping bias and model contamination. Our results show that clipping bias under spurious rewards reduces policy entropy, leading to more confident and deterministic outputs, while entropy minimization alone is insufficient for improvement. We further propose a reward-me reward-misalige misalignment model explaining why spurious rewards can enhance perfor- mance beyond contaminated settings. Our findings clarify the mechanisms behind spurious-reward benefits and provide principles for more effective RLVR training.
INTRODUCTION
The recent emergence of Large AI Reasoning Models (e.g., Kimi-K2, OpenAl-ol, and DeepSeek- RI (Kimi, 2025; Jaech et al., 2024; Guo et al., 2025)) has been driven by reinforcement learning with verifiable rewards (RLVR) (Li et al., 2025c). In RLVR, a verifier compares the model's ss rollout against deterministic ground- ground-truth solution, especially in mathematics and other STEM domains, providing outcome rewards. This verifiability has enabled models to achieve competitive and human-level performance on challenging benchmarks, such the International Mathematical Olympiad (Huang & Yang, 2025). Among RLVR methods, Group Relative Policy Optimization (GRPO) (Shao et al., 2024) has become particularly popular due its computational simplicity and memory efficiency.
In reinforcement learning, the exploration-explootaiio trade-off is framed within Markov decision process with per-step or shaped rewards. Exploration is typically promoted through stochastic policies or explicit bonus terms for underexplored actions (e.g., entropy regularization), while exploitation reinforces high-return actions via accurate value estimation. RLVR for LLMs departs from this paradigm in three respects: (i) rewards are outcome-level, extremely sparse, and verifiable only at the end of long rollouts, rendering unfolds in sequence space and is governed by decoding temperature rather than state-local bonuses; and (iii) policy updates rely on ratio clipping with group norppnoomal advantages, making them more sensitive to importance ratios and relative ranks than to absolute reward values.
intermediate token-level actions equivalen; (ii) exploration
These properties give RLVR a distinctive exploration exxioottatioooiiiiiiinnnn regime. In classical RL, spurious rewards, which misaligned with the true outcome reward (e.g., random noise), would be expected to hinder exploitation by injecting randomness that encourages suboptimal actions. Yet in RLVR, they have been observed improve performance in Qwen-Math models (Shao et al., 2025),

---

Published as a conference paper at ICLR 2026
phenomenon attributed to upper-clipping bias that disproportionately amplifies high-prior responses, consistent with contamination effects reported on MATH5CO (Wu et al., 2025). Conversely, entropy minimization, which reduces policy entropy to yield more deterministie, high-confidence rollouts, has been widely adopted in RLVR and empirically linked to consistent gains (Zhang et al., 2025b; Zhao al., 2025b; Cui et al., 2025; Fu et al., 2025). Notably, Agarwal et al. (2025) and Gao et al. (2025) directly optimize entropy as an objective and report substantial improvements even without verifiable feedback. These findings point to an RLVR-specific paradox: discouraging exploitation through spurious rewards and discouraging exploration through entropy minimization can both enhance validation accuracy, underscoring learning dynamics that depart from classical RL intuitions.
In this paper, we investigate how clipping, policy entropy, and spurious (random) rewards jointly shape model performance in RIVR. We show, both theoretically and empirically, that under random rewards, which discourage exploitation, clipping bias alone provides no meaningful learning signal and cannot directly improve performance. Instead, we establish a direct connection between clipping and policy entropy: clipping reduces entropy and drives the policy toward more deterministic, higher-confidence outputs, thereby inducing an murroyymmiiimization effect. Importantly, reduced entropy by itself does not guarantee performance gains. To clarify when spurious rewards can be beneficial, we introduce simple reward- masaannmnsll model. Our analysis overturns the prevailing view that improvements under spurious rewards are limited potentially contaminated Qwen models; similar gains also arise in the 1.1 ama and QwQ families, revealing a more nuanced exploration-exploiatiio dynamic that cannot be explained by contamination alone.
Contributions. We focus on two fundamental questions: (i) how policy entropy relates to perfor- mance, and (ii) whether spurious rewards yield gains, potentially through the interplay of clipping bias and model contamination. Our contributions can be summarized as follows:
We advance the theoretical foundations of RLVR by deriving explicit bounds on clipping bias and showing, under spurious rewards, this bias does not constitute meaningful learning signal. To capture its effect more precisely, we introduce a novel one-step policy-entropy shift formulation, which establishes deterministic link between clipping and policy entropy: clipping systematically reduces entropy and drives the policy toward more deterministic, higher-confidence rollouts.
2. We conduct extensive experiments across multiple model families -Math, Llama, QwQ) and sizes (7B, 8B, 32B), including both base and distilled variants. These results reconcile conflicting reports in the literature, demonstrating that performance improvements under spurious rewards are robust and not tied to any single model or dataset.
3. We show that these gains cannot be attributed to clipping bias or to causal effects of policy entropy, thereby overturning the prevailing view that improvements under spurious rewards are confined to potentially contaminated Owen-Math models. Instead, our findings reveal broader and more nuanced exploration-exxplotttion dynamic unique to RLVR.
2 PRELIMINARIES AND TECHNICAL BACKGROUND
2.1 GROUP RELATIVE POLICY OPTIMIZATION
RLVR assigns a binary outcome- outcome-based reward r(x, y) to a sampled response y from prompt x by comparing it against the ground-truth answer y*. To learn optimized policy via these reward, policy gradient methods (Williams, 1992; Sutton & Barto, 1998) aim to maximize
J(0)= Ex [)],
where p is the prompt distribution and Ty denotes the LLM policy. The parameter update at each iteration is 0 + 0 + 1/ v)) In practice, the trajectories are generated by an older policy To... but we wish to estimate the gradient at current policy no. By using the importance sampling technique with per-token ratio rr (0) noly, h.),
be rewritten as
lyl (e()))(b),))))
((Ø) = Ex (1xx
2

---

Published as a conference paper at ICLR 2026
where yı is the f-th token of y, which has ly] tokens in total, and h, fx.y<i with h1 Importance sampling might suffer from large variance when Ty drifts from Te..- To stabilize training, optimize the clipped surrogate objective follows,
((0) = Ex-roy-rrappp (px)
min (r.(0)r((,, clip(n + te
In this context, GRPO (Shao et al., 2024) and its variants (Yu et al., 2025; Liu et al., 2025b; Chu al., 2025; Zhang al., 2025a; Chen al., 2025c; Li et al., 2025b) estimate policy gradients using groups of samples. For each prompt x, GRPO draws a set [y from Tew. We denote y! as the f-th token of i-th sample y(1) and h = fx, and optimize the clipped objective follows,
( yx
J(0)- E 12
AED
min (1100
where ri (0)
x Raa (y p & E (0, 1) is hhper--parmnnn and the advantage A, == A(x, (()) is
computed from the group rewards as follows,
y0x, r(x.y9) meoo)())-mmo((((i))) (r(x))( ((.
(I)
ata(( ).... rxx,,)xxxx()))))
with r(x, y()) = if matches the ground-truth final answer and r(x, (0) 0 otherwise.
Remark 2.1. Under the GRPO update, the token-level advantage equals the response-level advantage A, and is independent of token index t.
Policy update. Following Cui et al. (2025), we use the softmax policy update framework, and one typical iteration amounts one-step exponentiation update with rollouts {y(")) ] as follows,
"e (alb) exp(nA(h,a)
(a h) Ev'cv uu (a' [h) exp(nA(haa')) OC Te.e (a h) exp(1)A(,.))) a)),
(2)
where 7) O is the step size and the advantage of an arbitrary token V is given by
A(h a) = Σ
Σ
by Aj.
(3)
Tar(a/)
Throughout the paper, we assume there exists Tiin > such that wold (a h)) 2 Tmin for all (h, a). For the case of presentation, Building upon Eq. (2), we derive the following reparamcterization for token-level importance ratio, with its proof presented in Appendix C.I.
abbreviated He and New Nnow and Told in the subsequent analysis.
Lemma 2.2. Suppose that 2 0. Then, we have
log(Fnew(a (a h)) log( (a h)) (((Ā(,, (h))))))))hh>)>>>.h) 0 S Cnp²
where p(h) Ea-maat (b) (h, a)), (2) Var (-1b) [n)[A(h, a)) and C 36v
does
not depend on n. Equivalently, we have log(r(h, a)) ((hh, a) - =(h)) + ) < Orp As a consequence, under the standardized setting with ((h) = 0 and o² 1, have
log(r(h, a)) nA(h, + of s CTp².
(4)
2.2 SPURIOUS REWARD FOR RLVR
Spurious reward arises whenever the feedback signal is misaligned with the ground truth reward. A random reward is a canonical example of such misalignment. In the context of RLVR, we formalize this notion as follows,
Definition 2.3 (Random reward). We consider the binary reward r(x, reward is feedback signal independent of (x, y()) and follows that r(x, (()) ~ Berrnoull( 1), i.e.
r(x,y()) y() in Eq. (1). A random
Pr(r(x, (0) 1) Pr(r(x,y(1) 0)
3

---

Published as a conference paper at ICLR 2026
Based on Definition 2.3, we obtain the following lemma for the GRPO advantage mechanism. These properties form the foundation for our subsequent analysis. The proofs are deferred to Appendix C.I.
Lemma 2.4. Fixing group size G 2 2 and denoting r; := r(x, and A, == A(x, y! where (r.)ez are group of random rewards, we define = è/ Ei= r4 S, = v v Ei (r. - -r)
and A, =, Then, the following statements hold: (i) A is symmetrically distributed
around 0) and thus E[A,h 2 GT >x= (2) VKGG K); for all integers &s 2, E 2 1 1 21-
0 for all k E N+; (ii) |A, & VG - 1/VG; (ii) E[[A,]] =
We examine several empirical findings related to random rewards. Notably, Shao et al. (2025) reports striking performance gains on MATH500 for the Owen-Math family when models are fine-tuned using the random reward defined in Definition 2.3. However, similarly large improvements observed for several other model families. Wu et al. (2025) likewise find substantial contamination in Qwen-Mat t h the MATH500 validation benchmark, hypothesizing that the apparent gains under random reward largely stem from reinforcing memorized or contaminated trajectories. In particular, Shao et al. (2025) attributes these gains to the PPO-style upper-clipping bias, formalized as follows,
not
Remark 2.5 (Upper-clipping bias). The upper clipping enforces Tt (0) = new (y (*) 11))
I 1 +
and implies that nnew (y: h S (1
é Tou (6)
(.) Equivalently, we have
Amax(y? (0) = nnew (y! h
(y (9) [ ) S emaaly! (0) [ (
If Told (y (6) Amax Ay
h ( 2 Tou hy and the upper clipping are active for both tokens, we have
The above interpretation indicates that upper clipping permits larger absolute increases for tokens that already have relatively high probability, whereas probbabbyyy tokens reach the clipping threshold much earlier. This asymmetry can preferentially amplify high-prior responses, potentially exploiting latent knowledge rather than fostering new reasoning ability.
However, Oertell et al. (2025) challenge this interpretation, arguing that the reported gains arise from algorithmic heuristics and evaluation artifacts; in their experiments, handom-rewat fine-tuning does not consistently improve reasoning and can even degrade it. These conflicting findings highlight how little is currently understood about RLVR learning dynamics and motivate two central questions: (i) Can random rewardy improve model performance, and under what conditions? (ii) Does clipping bias provide meaningful learning signal, and if not, what role does it actually play? Following prior work, our empirical analysis also focuses primarily on MATH500. We further discuss the broader implications of random-rewan training for general reinforcement learning settings in Appendix A.
2.3 LLM POLICY ENTROPY
Policy entropy H (me) quantifies the diversity of a policy's 's action distribution. A high-entropy policy allocates probability more evenly across actions, producing a wider variety of sampled responses, whereas a low-entropy policy concentrates probability on a small subset of actions, resulting in more deterministic behavior (Li et al., 2025a).
Definition 2.6 (Policy entropy). For any given policy Te, its entropy over a rollout trajectory space y E ) given prompt can be defined as follows:
H((Ta) = -by-a<((ooloo(TTT(T x))] = Σ
(y xx) log(me( x))-
yEy
Recent works in RLVR has begun to examine how policy entropy influences model performance. common perspective emphasizes avoiding "entropy collapse" to prevent premature convergence
to a low-diversity, suboptimal policy (Yu et al., 2025). At the token level, Wang et al. (2025b)
similarly highlight the importance of minority high-entropy tokens for effective reasoning. Yet several studies report the opposite pattern: reducing entropy can be beneficial. Agarwal et al. (2025) explicitly optimize an entropy-minimizze minimization objective and observe performance improvements, and Cui et al. (2025) even propose monotonic relationship in which lower entropy yields better
4

---

Published as a conference paper at ICLR 2026
performance. These conflicting findings raise a second fundamental question: (iii) Is there direct causal relationship between policy entropy and policy performance?
Beyond empirical observations, Cui et al. (2025) provide theoretical analysis by deriving the following estimate of the one-step change in policy entropy:
(Teww) - H(mTou)) -Covy-maa
(y x)), A(x,)))
(5)
Intuitively, if the reward positively correlated with the rollout probability, meaning sigh-probability responses tend to receive reward 1 while low-probability responses receive reward O, the policy becomes more peaked, leading to a decrease in entropy. Conversely, low-probability responses receive reward and high-pun trobbbbbbaiilitty responses receive reward 0, the policy is pushed toward flatter distribution, increasing its entropy. However, we emphasize that the approximation Eq. (5) does not apply for analyzing RLVR with random rewards.
Remark 2.7. Under random rewards, because A(x, y) is independent of Rold x) and has zero mean, substituting into Eq. (5) yields H(Tnew) (TTTaa)) = 0 (see Appendix C.3 for details). This implies that policy entropy should remain constant throughout training. However, this prediction contradicts empirical observations, which exhibit clear interaction between clipping and entropy dynamics. The discrepancy arises because Eq. (5) (i) retains only first-order terms in the policy expansion, ignoring higher-order contributions, and most importantly and (ß) assumes an unclipped formulation. Our theoretical results in $4.1 provide a more complete picture of how clipping interacts with and modulates policy entropy.
3 CLIPPING AND MODEL PERFORMANCE
We provide rigorous analysis of the upper-clipping bias from Remark 2.5. Indeed, we derive explicit bounds on the magnitude of the clipping bias and describe its effect on the learning signal. We further validate our theoretical findings with extensive empirical experiments.
3.1 THFORETICAL RESULTTS
We begin by decomposing the upper-clipping surrogate into two components: the raw term No. corresponding to the unclipped surrogate, and the cipping-correction term Neelip.
Definition 3.1. Suppose rollout y of length I, is sampled from a prompt and the clip ratio is e E (0, 1). For simplicity, we denote the token-level ratio r r(h,yr) as rt- Then, we define the clipped token-level ratio F, clip(r,, E, 1 E) max max(min(r::
e), - e), the raw surrogate as N; r2 A(x, y), the clipping-correcteo surrogate as Nº = F, A(x, y), and the upper activation indicator T,- = 1 (rx)14e)- The corresponding total upper clipping correction Cite is defined as
Σ (Nveli t=1
No). (F. A(xx,))
wl
For simplicity, we omit the superscript since it can be applied to any sample of the response group. The following theorem provides an upper bound on E||Cuul]; its proof is deferred to Appendix C.2.
Theorem 3.2. Let a prompt X have a response group of size G. each rollout has length L, and the clip ratio is E E (0, 1). For any rollout y, write = A(x,y). Denote p+ E[ and
DJ = (F. rT such that Cūu = Et DJ A. Then, for all > O, we have
El M M 22p LReaax o5 (Rtppaxx + ML min { (Pt,,
d(R5un)) (lI+s) f
(6)
where I2max = e//l Tmii,, M = IG-I, d(u) log u + 1, and A = (Te - 1 E) For
sufficiently small 1J, we have E < L m " where C1 M /
Co = -M(e- 1) and C3 = M(e- 1)o(1 + - min.
Remark 3.3. Theorem 3.2 shows that the upper bound the total clipping-correction term depends on the (empirical) expected token-level activation rate p: larger p would bring more clipping correction. p varies across model families but can be directly monitored during training. This motivates a general, model-aggoste framework for analyzing clipping effects one that applies uniformly across architectures by expressing all bounds in terms of the observable activation rate p.
5

---

Published as a conference paper at ICLR 2026
To quantify the effect of clipping, establish the following bound relating the magnitude of the raw surrogate sum to the total clipping correction. For the proof, please refer to Appendix C.2. Theorem 3.4 (Law of Clipping). Under the same settings Theorem 3.2, define the surrogate sum Naw Ei=1 rtA. Then, for all 17 > we have
Ba 2 LE[|A|]e - 2 LB] Cn²2,
On
where C
Furthermore, we have
min
B[[AI](1 C212)
E| C|
L L-1/2M,
v 2p+ Rigax of ( Repax) + M MD min (v++,
di Reuas
# 1++)
In addition, under practical hyperparameter settings, we have Frw > evaluation using the parameters from our actual training setup given in Corollary 3.6.
EllCall |] A quantitative
3.2 MODEL-SPECIFF EVALUATION
Following the hyperparameter configuration of Shao et al. (2025), we train .wen2.5 5-Math-7B on the DeepScaleR dataset (Luo et al., 2025) using random rewards drawn from (ernoulli The training setup uses batch size of 128, group size of 16, decoding temperature 1.0, clipping ratio 0.2, learning rate x 10-7, and KL coefficient
We run multiple consecutive experiments with and without clipping using the verl framework (Sheng et al., 2025). The resulting training trajectories on the MATH500 validation set, together with the clipping activation fraction over training. are shown in Figure 1. We adopt the default training prompt from verl, which instructs the model to enclose validation (see Appendix A for further discussion). Notably, for Qwen2. 5 -Math-7B, the clipping activation rate substantially lower than what is typically observed other base models:
final answer in a box for verifier
Remark 3.5. Empirically, the clipping activation ratio is usually below 1% for general GRPO training. For specific Qwen2. 5-Ma h- h-7B training, the clipping activation ratio never exceeds 0.2%, with expected activation probability [[[e] 0.001.
Unclipped- Qwenn2.55 Math-78
Clipped- Qwenn2 5- Math-78
Ratio of clipping activation OwenZ, 5-ath--
2.2 250
13.5 300
Figure 1: Independent trials over Owen2.5-Math- 5-Mathh- the MATH500 validation set. For per- formance validation subpanels (Left & Middle), each color represents a different run; the bold line shows the smoothed trajectory, and the faint line of the same color shows the corresponding raw individual run. All later figures follow the same plotting convention. Unclipped training (Left); clipped training (Middle); and clipping activation ratio during training (Right).
As shown in Figure 1, enabling clipping lead to decline in validation performance, whereas disabling clipping often results in improvement. These findings suggest that upper clipping bias not the mechanism driving the observed gains under random rewards. To illustrate this point, we provide a numerical instantiation of Theorem 3.4 using the training hyperparameters of Qwen -Math:
Corollary 3.6. Suppose that 7 5 10-7 ε 0.2, p+ 0.001, G 16, Tmin = 10 then by Theorem 2.4, M = 3.75 and E[A[]] s 0.967; by Theorem 3.2, Rmmax s 1.649, Φ( [Rmaa) x 0.176, and A se 0.449. Thus, Theorem 3.4 implies C = 1.25 x 10 and
4096, and
ELLNN
B[AAI(( -
17.15.
L 1/2M 22p Rmax (R + MA min
of (RRmA vP+.
<[+e))
This confirms that BE((Nr EllCitill in magnitude for hyperparameters used in practice.

---

Published as a conference paper at ICLR 2026
Remark 3.7. As consequence of Corollary 3.6, the upper-clipping bias fails to provide meaningful learning signal towards the gradient, even under contaminated model and benchmark. This result is supported by empirical observations and theoretical justifications. We present further analysis over clipping threshold and group size in Appendix B. Nonetheless, even though clipping does not directly correlated to performance, $4 shows that it still has causal effect on policy entropy under random rewards, shaping the structure of the outcomes without enhancing learning.
further ablation
4 CLIPPING AND POLICY ENTROPY
We provide two theoretical results describing policy entropy under unclipped and clipped training ($4.1). As discussed $2.3, the approximation in Eq. (5) from Cui et al. (2025) becomes inaccurate when clipping random rewards are present. Our analysis incorporates both clipping and initial policy skewness, yielding a more precise characterization of entropy dynamics. these results through extensive experiments and targeted case studies. In $4.3, we interpret clipping as a mechanism that implicitly reduces entropy and caution supported by empirical evidence - against conflating entropy reduction with improved performance.
$4.2, we validate
4.1 ONE-STEP POLICY ENTROPY CHANGE UNDER RANDOM REWARDS
We analyze entropy dynamics under unclipped and clipped training in Theorem 4.1 and Theorem 4.3, and hope these results motivate new ways to modulate entropy using spurious-reward setups alongside explicit entropy regularization.
We first present the unclipped trriiinng dynamics in Theorem 4.1, with detailed statement and proof in Appendix C.3, where we also identify the conditions that permit entropy growth.
Theorem 4.1. With update in Eq. (2) and cG == (1 11-22-0//G, all > 0.
[[H((Teww) H(TTua)) -CG [[((())
where & measures the skewness of aaa and [R(1)] = O(1))) for small n7.
Remark 4.2. Theorem 4. shows that the one-step entropy change under unclipped training depends critically on the initial policy distribution; indeed, more skewed policies can exhibit entropy increases during training. As concrete example, we consider a two-armed policy Told = (3, - B3 for some 8
E (0, 1). In this case, using the definition of 4, one compute 4 ((Tota) = 1+ (1 (1-23) 2,3) log( 1-8 ).
Moreover, (Taaaa 2 0 if and only if B E 10. 0.824). Thus, up (11?)) term, entropy decreases in expectation when 8 E [0.176, 0.824] (a less skewed policy) and increases when 3 0.824 or < 0.176 (a more skewed policy). Figure 9 illustrates this behavior: for a less-skewed initialization (Figure 9, Left), spurious rewards do increase entropy under unclipped training, whereas with a sufficienlly skewed initialization (Figure 9, Right), entropy increases over training. This is also consistent with the entropy growth observed in experiments (Figure 2, Left).
Actual policy (xr) evaluation. Apart from the two-armed example in Remark 4.2, we further evaluate ((T) for the actual Qwen-Math-79B policy in Figure 8, which helps readers better perceive the policy skewness and its corresponding ((T) associated with entropy increases during training. For detailed setup and results, please refer to Appendix B.
Next, we analyze training dynamics with upper clipping in Theorem 4.3; detailed statements, proof and entropy decay verification under practical parameters are deferred to Appendix C.3.
Theorem 4.3. Define C;:= {A, > O, r(y(0) > + + Let == P(C1) and 8 = E(r(y(U) -
<) Ct). Then for yj > 0 small enough and any p E (Tmin, 1).
E(H(Tnow)) - H( I 2C E[R(r)] + c(p) (p6ent - Xmaa 1)p)
where CG. tb and R(1) are defined as the same as in Theorem 4.1; 1; for other constants, see Ap- pendix C.3.
Remark 4.4. As shown in Figure 2 (Middle), our experiments confirm that policy entropy consistently decreases over time under random rewards. contrast, disabling clipping leads to entropy increasing during training (Figure 2, Left). Existing approaches to counter early-stage entropy collapse rely on
7

---

Published as a conference paper at ICLR 2026
Unc Unclloped Policy Entrapy
Clpppdd- Policy Entropy
Undlipped R2 R3-Giatil-Liamm
211
340
140
Policy entropy evolution of Qwen2. 5-Math-78 under random-reward training, with results for unclipped training (Left) and clipped training (Middle); Unclipped training with 11-Distill-Llame= an example that leads to the gradient explosion (Right).
Figure 2:
regularization techniques that merely slow the decay ( (Wang et al., 2025a; Yao et al., 2025; Zheng al., 2025; Cheng al., 2025). Our finding that also improving validation performance suggests complementary strategy: using spurious-reward setups more effectively preserve and modulate entropy. This highlights promising direction for combining true and spurious rewards to better balance exploration and exploitation in RLVR.
actively increase policy entropy while
4.2 EMPIRICAL EVALUATION
Figure 2 (Left & Middle) shows that, under random rewards, disabling clipping can cause policy entropy to increase over training, reflecting progressively greater exploration. In contrast, enabling clipping constrains this behavior and leads to monotonic decrease in entropy. This pattern highlights that clipping functions primarily as a form of regularization: by capping per-token likelihood ratios, it effectively reduces the update step size and prevents the policy from drifting too far from its previous distribution. Beyond its regularization effect, clipping also fulfills its original purpose of preventing gradient explosion, thereby adding further training stability.
When gradient magnitudes grow large, clipping protects the optimization process by preventing abrupt, destabilizing updates. Without clipping. this safeguard disappears: the optimizer may take oversized steps that inject excessive exploration and destabilize training. Thus, clipping does introduce additional learning signals; its primary function to maintain optimization stability by enforcing local trust region. Models with sufficiently large single-step gradient norms can collapse entirely. A failure case is shown in Figure 2 (Right): training R1 -iiststt ill L-L-laa ama -BB without clipping initially raises the MATH5 0O validation accuracy from 65.6% to 76.6% within 100 steps, but around step 150 the gradients explode, causing a sharp drop in performance. For comparison, the shown Figure (Middle).
clipped-training training counterpart for ai-Distill-LLi
4.3 POLICY ENTROPY AND MODEL PERFORMANCE
OwO-328 JIME Tainting Se
0B 13:2otti-Jamm-lf 18. AIN Twasing Set
Ower2 D.en2:5-tab--7D JINE Training Set
Cloped -wenn S-Nth-795--- JINE Trannig Set
Keu
0.0
(Middle- L). Qwen2. 5-Mathh----- 73 (Middle-R). With one specific example that shows entropy minimization would lead sub-optimal policy under noisier and more difficult training environment (Right).
Figure 3: Results on AIME training set OwQ-320 32B (Left), R1-Distill-Llant
Figure shows that both higher and lower entropy can achieve improved performance. practice, higher entropy reflects stronger exploration: the policy is flatter and thus more capable of discovering new trajectories. Lower entropy corresponds to greater confidence, with the policy becoming more concentrated small set of trajectories; in RLVR, such concentration may also correlate with better performance. However, this connection not guaranteed: convergence to a highly skewed, low-entropy policy does not necessarily improve accuracy, demonstrated in Figure 3 (Right).
8

---

Published as a conference paper at ICLR 2026
This suggests that methods explicitly minimizing policy entropy should be applied with caution. Additional evidence from unclipped training under the same setup is provided in Appendix B.
Under random rewards, clipping acts as an implicit entropy minimization mechanism, pushing the policy toward a more peaked distribution that concentrates probability mass on a small set of trajectories. Whether this effect is beneficial depends the model's 's initial policy distribution and the difficulty of the training data. For a strong model on an easy dataset, the policy is already concentrated on correct trajectories; additional concentration can be sufficient and may appear advantageous. We provide a simple theoretical explanation of this phenomenon in $5.
However, as the training data becomes difficult, the policy may place most of its probability mass on incorrect trajectories. This produces the noisy rollouts and unstable updates, often driving the model toward an incorrect low-entropy solution. To illustrate, for Qwen2. 5-Math-7B, we replace the milder DeepSca leR curriculum with the harder AIME Past series. As shown in Figure 3 (Middle-R), after 20 training epochs (with the same hyperparameters as Figure 2), the trajectory resembles random walk with little meaningful improvement in validation accuracy. In contrast, stronger QwQ-32B and R1 ail-Dsstee Distill-Llama 8B models (rollout length 8192, with all other settings identical to the 7B configuration) trained on A AIM dataset exhibits steady early-epoch gains (Figure 3, Left & Middle-L). These results indicate that the effectiveness of entropy minimization is regime-dependent: for strong models on casier data, it can further concentrate mass correct trajectories, whereas for weaker models or harder data, it may reinforce incorrect modes and stall, or degrade performance. Thus, entropy minimization mechanisms (including clipping under random rewards) can be interpreted as regularization rather than universally beneficial learning signals.
5 REWARD MISALIGNMENT: WHO CAN BENEFIT FROM RANDOM REWARDS?
From empirical observations in this and prior work, we note two consistent patterns under random- reward training. First, in line with Shao et al. (2025), weaker models tend to improve less-and importantly, model strength is dataset-dependent: model that performs well on an easier benchmark may struggle on a harder one. Second, as baseline accuracy increases (e.g., approaching 70%).. training dynamics become noticeably smoother, whereas models starting around 50% accuracy exhibit substantially more oscillation. To explain when and why a model may improve under random rewards, we analyze the phenomenon through the lens of reward misalignment. As a warm-up, we introduce simple probabilistic model that captures this mechanism in the binary outoome-reward (ORM) setting, converting the observed behavior into a tractable misalignment analysis.
For prompt x. draw G rollouts (y-(), ..... correct and incorrect sets C,I C {1...., .......,) with |C] = fg.. [Z] = n2; and n. + n, G. We analyze two label errors: (i) False positives (FP):r, negatives (FN):
.......(.)) y()) } from current policy πT. Partition the indices into
for j E (an incorrect rollout is rewarded); (ii) False 0 for le E C (a correct rollout is not rewarded). Specifically, we aim to explain: (i) why validation curves fluctuate less when accuracy is high but become noticeably unstable when accuracy is low, and (ii) why stronger models are more likely to improve under random rewards. Our starting point is to formalize reward misalignment: the loss of advantage mass that should have been assigned to correct rollouts but is instead diverted due to random reward mislabeling.
Definition 5.1 rCosonct-response advantage loss). Let {r be i..dd. with r, ~ Bernoulli(3) for all j. independent of correctness. We define the event counts f == Ejez 1(r 1} and
9 = ExEc ( = 0}. and let T' Σj=rr = f (n. (me-9) 9) be the total number of +1 rewards. We write r- := T
& for the group-averaged reward. The class-wise centered reward sum over C is (n.-g)- g)- " As an "ideal" reference with no mislabels (f g 0), we have Sideall Ekec(1 [)) n1((1 (x)) Finally, we define the damage (advantage loss) as
(f.g)) == E Exec(Tx
-f) (
A(f.g) = Sideal Ec(f.g)-
(8)
Proposition 5.2. For any ne, n. 2 1 and G The + nhi let f Binomial(n,, 1), g Binomial(n.. 1) be independent, and A = Alf.g) be defined in Eq. (8). Under i.i.d. Bernoulli() rewards, we have
E A
GG (n) Var(A)= 40
Ro(G-no)

---

Published as a conference paper at ICLR 2026
Qwen22.5 MattMMMth.333.
R1.00all--ama-85
Percenta Average Improvement
250
112
250
305 200
210
Cee
Figure 4: Results of Qwen2. -.tt5-5-Mtt5..555 under clipped training (Left); results of RI-Distill- Llama-8B under clipped training (Middle); percentage improvement (averaged over six independent runs) for different models under the same training and validation setup (Right).
The expected damage decreases as the number correct rollouts ne increases, and its variance likewise shrinks with The, explaining why the stronger models exhibit more stable validation curves. The largest fluctuations occur near the symmetric regime ne RS ng. This consistent with our empirical results in Figure 1. We further refine this characterization by decomposing the damage into conditional means in Theorem 5.3. The proofs are given in Appendix C.4.
Theorem 5.3. Let f Binomial(m, n, 2) and g Binomial(ne: 2)) be independent, and let A be defined in Eq. (8). For policy with more correct rollouts (n, > nzd), we have
[12) < E[A1 (o>1)}
As ne increases on (g G). we have E[ (f>n)) constitutes strictly smaller fraction of B[A].
Theorem 5.3 retines Theorem 5.2. As the overall damage E| decreases with Re. the composition of that (shrinking) damage shifts: for stronger models (those with ne > n4;), FN-dominated regions (g > f) contribute larger share than FP-dominated regions (f > g). and the FP-dominated portion decreases monotonically. Practically, this means that training stronger models on datasets where n. n, incurs less total misalignment damage particularly fewer FP misallocations -add is therefore more likely to yield improvements under random rewards. This effect persists even beyond contaminated-rewarde settings. We further corroborate this trend through experiments on stronger distilled L1 ama model and a weaker Qwen Math model, with results shown in Figure 4.
As reported by Shao al. (2025), base ama models reliably degrade during random-reward training across trials. Under the reward-misalignment perspective, stronger models should benefit more and are thus more likely improve. We test this by evaluating a stronger distilled ama variant, whose base and teacher models both exhibit contamination on MATH500. As shown in Figure 4 (Middle), using rollout length of 8192 tokens and matching all other hyperparameters the Qwen-Mate h configuration, we observe improvements comparable to those in Figure 2. In contrast, the weaker and potentially contaminated Qwen-Math model (Figure 4, Left) fails to achieve similar gains. These results indicate that validation-set contamination does not account for the improvements under random rewards, nor is the effect specific to Qwen Taren-Mathh Math. Figure 4 (Right) summarizes the percentage improvements across the model results in Figure (Left) and Figure (Left and Middle).
6 CONCLUSION
We now revisit the three guiding questions posed in $2: (i) Can random rewards improve model performance, and under what conditions? (ii) Does clipping bias provide a meaningful learning signal, and if not, what purpose does it serve? (iii) Is there direct causal relationship between policy entropy and policy performance?
First of all. random rewards can improve model performance. As shown in $5, the benefits depend on model strength: stronger models are more likely to realize gains from random reward, whereas weaker models become unstable when trained on harder datasets. Second, clipping bias does not supply useful signal ($3.1); instead, its function is to regulate policy entropy in the presence of spurious training signals ($4.1). Finally, as demonstrated in $4.2 and $4.3, policy entropy and performance do not exhibit a deterministic causal relationship: entropy decreases may accompany performance collapse, while entropy increases may coincide with improvements. Overall, theoretical and empirical analyses disentangle the complex interplay between exploration and exploitation in RLVR, offering principled foundation for future work to understand the alignment dynamics.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
We sincerely appreciate Buzz High Performance Computing (https: /www. wuzzhpc..a... .ai, info@buzzhpc.co ai) for providing computational resources and support for this work.
REFERENCES
A. Agarwal, M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98): 1-76, 2021.
S. Agarwal, Z. Zhang, L. Yuan, J. Han, and H. Peng. The unreasonable effectiveness of entropy minimization in LLM reasoning. ArXiv Preprint: 2505. 15134, 2025.
A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back basics: Revisiting REINFORCE optimization for learning from human feedback in LLMs. In ACL, pp. 12248-12266 2024.
2024.
M. G. Azar, Z. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, pp. 4447-4455, 2024.
Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In ICLR, 2019. URL httph:h//ppeeeee/eeeee
net. wet.://oorormmii--1111-0-
H. Chen, G. He, L. Yuan, G. Cui, H. Su, and Zhu, Noise contrastive alignment of language models with explicit rewards. In NeurIPS, pp. 117784-117812 2024a.
L. Chen, M. Prabhudesai, Fragkiadaki, H. Liu, and D. Pathak. Self-questioning language models. ArXiv Preprint: 2508.03682, 2025a.
Chen, Y. Xie, and Q Zhang. SICNN: Sparsity-induced input convex neural network for optimal transport. In OPT 2024: Optimization for Machine Learning, 2024b.
P. Chen, X. Chen, W. Yin, and T. Lin. ComPO: Preference alignment via comparison oracles. In NeurIPS, 2025b.
P. Chen, X. Li, Z. Li, X. Chen, and T. Lin. Stepwise guided policy optimization: Coloring your incorrect reasoning in GRPO. In The 5th Workshop on Mathematical Reasoning and Al NeurIPS 2025, 2025c. URL https: epeenevveww net / forum? ? d=WLe InJaUe.
D. Cheng, S. Huang, X. Zhu, B. Dai, W. X. Zhao, 7. Zhang, and F. Wei. Reasoning with exploration: An entropy perspective on reinforcement learning for LLMs. ArXiv Preprint: 2506. 14758, 2025.
X. Chu, H. Huang, X. Zhang, F. Wei, and Y. Wang. GPG: A simple and strong reinforcement learning baseline for model reasoning. ArXiv Preprint: 2504.02546, 2025.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, al. Training verifiers to solve math word problems. ArXiv Preprint: 2110. 14168, 2021.
G. Cui, Y. Zhang, J. Chen, Yuan, Z. Wang. Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, Z. Liu, H. Peng, L. Bai, W. Ouyang, Y. Cheng. B. Zhou, and N. Ding. The entropy mechanism of reinforcement learning for reasoning language models. ArXiv Preprint: 2505. 2361,2 2025.
H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang RAFT: Reward ranked fine-tuning for generative foundation model alignment. In Transactions on Machine Learning Research, 2023.
H. Dong, W. Xiong, B. Pang. H. Wang. H. Zhao. Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. RLHF workflow: From reward modeling to online RLHF, In Transactions on Machine Learning Research, 2024.
11

---

Published as a conference paper at ICLR 2026
Y. Fu, X. Wang, Y. Tian, and J. Zhao. Deep think with confidence. ArXiv Preprint: 2508.15260, 2025.
Z. Gao, L. Chen, H. Luo, J. Zhou, and B. Dai. One-shot entropy minimization. ArXiv Preprint: 2505.20282, 2025.
D. Guo. D. Yang. H. Zhang. J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, al. Deepseck-R. incentivizes reasoning in LLMs through reinforeement learning. Nature, 645(8081): 633-638, 2025.
D. Hendtycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Mcasuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks, 2021. URL https:::pp www.eeveeeeeeeeeeee/ee/pe net/ forum? d= (ddd=Bytttaaccc
J. Hong, N. Lee, and J. Thome. ORPO: Monolithic preference optimization without reference model. In EMNLP, pp. 1 11110-11111 170991189,11111111 2024.
A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. V-STar: Training verifiers for self-taught reasoners. In COLM, 2024. URL https: /ttpss:/oppnzeevert net/ forum? wtw.smad.....
Y. Huang and L. Yang. Gemini 2.5 pro capable of winning gold at imo 2025. ArXiv Preprint: 2505. 2025.
A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. OpenAl ol system card. ArXiv Preprint: 2412.16720, 2024.
Kay, G. Van Horn, S. Maji, D. Sheldon, and S. Beery. Consensus-driven active model selection. ArXiv Preprint: 2507.23771, 2025.
Team Kimi. Kimi k2: Open agentic intelligence. ArXiv Preprint: 2507.20534, 2025.
J. Koch, L. Langosco, J. Pfau, J. Le, and L. Sharkey, Objective robustness in deep reinforcement learning. ArXiv Preprint: 2105.14111, 2021.
L. Langosco Di Langosco, J. Koch, L. Sharkey, Pfau, and D. Krueger. Goal misgeneralization in deep reinforcement learning. In ICML, pp. 12004-12019, PMLR, 2022.
G. Li, Y. Wei, Y. Chi, and Y. Chen. Breaking the sample size barrier in bodel-based reinforcement learning with a generative model. Operations Research, 72(1):203-221, 2024a.
Z. Li, Xu, Y. Zhang, Z. Lin, Yu, R. Sun, and Z-Q. Luo. ReMax: simple, effective, and efficient reinforcement learning method for aligning large language models. In ICML, pp. 29128-209 29163, 2024b.
Li, C. Chen, T. Xu, Z. Qin, J. Xiao, Z-Q. Luo, and R. Sun. Preserving diversity in supervised fine-tuning of large language models. In ICLR, 2025a. URL https: wpwpeeveen.eeeeeeeeeeee.. net/ forum? Pid=NQEe 7B7bSw.
Z. Li, C. Chen, Yang, T. Ding, R. Sun, G. Zhang, W. Huang, and Z-Q. Luo. Knapsack RL: Unlocking exploration of LLMs via optimizing budget allocation. ArXiv Preprint: 2509.25849, 2025b.
Li, P. Wang. T. Xu, Ding, R. Sun, and Y. Yu. Review reinforcement Icarning for large language models: Formulations, algorithms, and opportunities. Submitted Transactions Machine Learning Research, 2025c. URL https: http:///e...peeeeeeeeeeeeeaeaaaa jxJJ.. Under review.
www.:/ewwwwwwww.eeeeeeeeeeee.eee net www.ee//oooidnoooeooooooo
H. Lightman. V. Kosaraju, Y. Burda. H. Edwards, and K. Cobbe. Let's verify step by step. In ICLR, 2024. URL https: wwtp///innweee.e.eeoeeee.. net/ forum? 1ew.mm.dddv1....ER0.
B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever,
T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, J. Liu, and J. Liu. Statistical rejection sampling improves preference optimization. ICLR, 2024a. URL https: /pwpeevveeeeeeeeeeeeeeeee. net/ forum? d=xb jSwrrgee.
12

---

Published as a conference paper at ICLR 2026
T. Liu, Z. Qin, J. Wu, J. Shen, M. Khalman, R. Joshi. Y. Zhao, M. Saleh, S. Baumgartner, J. Liu, et al. LiPO: Listwise preference optimization through learning-to- In NAACL, pp. To appear, 2025a.
Liu, M. Lu, Zhang, B. Liu, H. Guo, Yang. J. Blanchet, and Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In NeurlPS, pp. 138663-138697, 2024b.
Z. Liu, C. Chen, W. Li. P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin. Understanding R1-zero-lide training: critical perspective. ArXiv Preprint: 2503.20783 2025b.
L. Luo, Y. Liu, R. Liu, S. Phatale, M. Guo, H. Lara, Y. Li, L. Shu, Y. Zhu, L. Meng, et al. Improve mathematical reasoning in language models by automated process supervision. ArXiv Preprint: 2024.
M. Luo, S. Tan, Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, Luo, L. E. Li, R. A. Popa, and I. Stoica. Deepscaler: Surpassing ol ol-preve with a 1.5b model by scaling RL. Notion Blog, 2025.
H. Ma, G. Fu, Z. Luo, J. Wu, and T.-Y. Leong. Exploration by random reward perturbation. ArXiv Preprint: 2506.08725 08737, 2025.
Y. Meng, M. Xia, and D. Chen. SimPO: Simple preference optimization with reference-fre free reward. In NeurlPS, pp. 124198-124255 2024.
A. Y. Ng. D. Harada, and S. J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, pp. 278 778-287, 1999.
O. Oertell, W. Zhan, G. Swamy, Z. S. Wu, K. Brantley, J. Lee, and W. Sun. Heuristics considered harmful: RL with random rewards should not make LLMs reason. Notion Blog, 2025.
A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. ArXiv Preprint: 3402.13228, 2024.
A. Pan, K. Bhatia, and Steinhardt. The effects of reward misspecification: Mapping and miti- gating misaligned models. In ICLR, 2022. URL https: https////pppreeee..eo font/forummid JYtwewiL7ye.
R. Y. Pang. W. Yuan, H. He, K. Cho, S. Sukhbaatar, and J. Weston. Iterative reasoning preference optimization. In NeurlPS, pp. 116617-116637, 2024.
C. Park, S. Han, X. Guo. A. Ozdaglar, K. Zhang, and J. K. Kim. MAPoRL: Multi-agent post- co-training for collaborative large language models with reinforcement learning. In ACL, pp. 30215-30248, 2025.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity- driven exploration by self-supervised prediction. In ICMI., pp. 2778-2787, 2017.
M. Prabhudesai, L. Chen, A. Ippoliti, K. Fragkiadaki, H. Liu, and D. Pathak. Maximizing confidence alone improves reasoning. ArXiv Preprint: 2505.22660, 2025.
R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model secretly reward model. NeurlPS, pp. 53741,, 2023.
R. Rafailov, Hejna, R. Park, and C. Finn. From Sr$ to Sq Your language model is secretly a Q- function. In COLM, 2024. URL https: http//veevvveeeeeeeeee
htttps://pperrevieee ew wttpeewwwwweeeeeeeeeeeeeeeeeeeee rumm?? d=kkVcNxt.gX..
N. Razin, S. Malladi, A. Bhaskar, D. Chen, Arora, and B. Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. In ICLR, 2025. URL https:// openreview net forum
tett//orrmmiddmuaas...00 Lid
D. Russo and B. Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 1214):1221-1243 2014.
13

---

Published as a conference paper at ICLR 2026
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. ArXiv Preprint: 1707.06347, 2017.
A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and
A. Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In ICLR, weteefoorume http:///ppe//eeeeeeeeeee/eee//ee /RLLtpss//ppenreeieee.
2025. URL https:
d-A.Y 7A41 ZLW.
R. Shao, S. S. Li, R. Xin, S. Geng, Y. Wang. S. Oh, S. S. Du, N. Lambert, S. Min, R. Krishna, et al. Spurious rewards: Rethinking training signals in RLVR. ArXiv Preprint: 2506. 10947, 2025.
Z. Shao, P. Wang, Q. Zhu, R. Xu, Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, et al. Math Pushing the limits of mathematical reasoning in open language models. ArXiv Preprint: 03300, 2024.
H. Shen. On entropy control in LLM-RL algorithms. ArXiv Preprint: 2509.03493, 2025.
G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng. H. Lin, and C. Wu. Hybridflow: A flexible and efficient RLHF framework. In ECCS, pp. 1279-1297, 2025.
I. Singh, Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, Thomason, and A. Garg. ProgPrompt: Generating situated robot task plans using large language models. In ICRA, pp. 11523-11530. IEEE, 2023.
F. Song. B. Yu, M. Li, H. Yu, F. Huang, Y. Li, and H. Wang. Preference ranking optimization for human alignment. AAAI, pp. 18990-18998, 2024.
Y. Song, J. Kempe, and Munos R. Outcome-based exploration for LLM reasoning. ArXiv Preprint: 2509.06941, 2025.
R. Sutton and G. Barto. Reinforcement Learning: An Introduction, volume MIT Press, 1998.
Tajwar, A. Singh, Sharma, Rafailov, Schneider, T. Xie, Ermon, C. Finn, and A. Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In ICML, pp. 47441- 47474, 2024.
J. Tien, J. Z. He, Z. Erickson, A. Dragan, and D. S. Brown. Causal confusion and reward misiden- tification in preference-based reward learning. In ICLR, 2023. URL https::/openrevit/ net/forum? dd=ROxxvr_X322 ZZA.
J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and Higgins. Solving math word problems with process-and outcome-basee feedback. ArXiv Preprint: 2211. 2022.
C. van Niekerk, R. Vukovic, B. M. Ruppik, H. Lin, and M. Gašic. Post-training large language models via reinforcement learning from self-feedback. ArXiv Preprint: 2507.21931, 2025.
J. Wang, J. Liu, Y. Fu, Y. Li, X. Wang, Y. Lin, Y. Yue, L. Zhang, Y. Wang, and K. Wang. Harnessing uncertainty: Entropy-modulater policy gradients for long-horizon LLM agents. ArXiv Preprint: 2509.09265. 2025a.
P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Li, D. Chen, Y. Wu, and 7. Sui. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. In ACL, pp. 9426-9439, 2024.
S. Wang, L. Yu, C. Gao, C. Zheng, A. Yang, A. Zhao, Y. Yue, S. Song, B. Yu, G. Huang, and J. Lin. Beyond the 80/20 rule: High- entropy minority tokens drive effective reinforcement learning for LLM reasoning, ArXiv Preprint: 2506.0199 01939, 2025b.
S. Liu, R. Lu, K. Dang, X. Chen, Yang. Z. Zhang, Y. Liu,
Y. Wang, M. Yang, R. Dong, Sun, F. Liu, and I.. H. U. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. In NeurlPS, pp. 38786-38797, 2023.
R. Williams. Simple statistical gradient followwig algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
14

---

Published as a conference paper at ICLR 2026
M. Wu, Z. Zhang, Q. Dong, Z. Xi, J. Zhao, S. Jin, X. Fan, Y. Zhou, Y. Fu, Q. Liu, S. Zhang, and Q. Zhang. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. ArXiv Preprint: 2507.10532, 2025.
Xiao, Z. Li, X. Xie, E. Getzen, C. Fang. Q. Long, and W. J. Su, On the algorithmic bias of aligning large language models with RLHF: Preference collapse and matching regularization. ArXiv Preprint: 2405.16455, 2024.
W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint, In ICML, pp. 54715-54754, 2024.
R. Xu, K. Li, H. Wang, G. Kementzidis, W. Zhu, and Y. Deng. RL-QESA: Reinforcement-leaminng quasi-equilbriim simulated annealing. In JCML Workshop. 2025.
A. Yang, B. Zhang, B. Hui. B. Gao, B. Yu, C. Li, D. Liu, Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, and Z. Zhang. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. ArXiv Preprint: 1212,, 2024.
Yao, R. Cheng, X. Wu, J. Wu, and C. Tan. Diversity-aware policy optimization for large language model reasoning. ArXiv Preprint: 2505.23433, 2025.
Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. DAPO: An open-source L.MM reinforcement learning system at scale. ArXiv Preprint: 2503. 14476, 2025.
H. Yuan, Z. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. RRHF: Rank responses to align language models with human feedback. In NeurlPS. pp. 10935-10950, 2023.
L. Yuan, G. Cui, H. Wang. N. Ding, X. Wang. B. Shan, Z. Liu, Deng. H. Chen, R. Xie, Y. Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun. Advancing LLM reasoning generalists with preference trees. In ICLR, 2025, URL ht / //pp////epeeeieieeeeemeem net. htt///oommmmmddaaaamee
E. Zelikman, Y. Wu, Mu, and N. D. Goodman. STaR: self-taught reasoner bootstrapping reasoning with reasoning. In NeurlPS, pp.
2022.
A. Zhang, N. Ballas, and J. Pineau. dissection of overfitting and generalization in continuous reinforcement learning. ArXiv Preprint: 1806.07937, 2018.
K. Zhang. Y. Hong. J. Bao, H. Jiang, Y. Song. D. Hong, and H. Xiong. GVPO: Group variance policy optimization for large language model post-training. ArXiv Preprint: 2504. 19599, 2025a.
Q. Zhang, H. Wu, C. Zhang, P. Zhao, and Bian. Right question is already half the answer: Fully unsupervised LLM reasoning incentivization. ArXiv Preprint: 2504.05812, 2025b.
T. Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian. NovelD: A simple yet effective exploration criterion. In NeurlPS, pp.
2021.
7. Zhang, C. Zheng, Y. Wu, B. Zhang, R. Lin, B. Yu, D. Liu, J. Zhou, and J. Lin. The lessons of developing process reward models in mathematical reasoning. ArXiy Preprint: 2501.07301, 2025c.
W. Zhao, P. Aggarwal, S. Saha, A. Celikyilmaz, J. Weston, and I. Kulikov. The majority is not always right: RI training for solution aggregation. ArXiv Preprint: 2509.06870 2025a.
X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song. Learning to reason without external rewards. ArXiv Preprint: 2505.19590. 2025b.
Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv Preprint: 2305. 10425, 2023.
Zheng, Xing, Q. Gu, T. Liang, X. Qu, X. Zhou, Y. Li, Z. Wen, C. Lin, W. Huang, Q. Liu, G. Zhang, and Z. Ma. First return, entropy-eliciting explore. ArXiv Preprint: 2507.07017 2025.
15

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
We provide a technical review clarifying the differences in experimental setups and summarizing insights from recent advances in RLVR for LLM post-training.
Spurious reward in classical RL. We provide broader context on how prior work in reinforcement learning for classical settings has leveraged spurious rewards to facilitate training. Spurious reward signals are closely linked to challenges in generalization (Zhang et al., 2018; Koch et al., 2021; Langosco Di Langosco et al., 2022). While these works illustrate deliberate uses of such signals, spurious rewards may also arise unintentionally, leading reward misspecification and reward hacking (Pan et al., 2022); similar reward hacking has been documented in Tien et al. (2023). A second relevant thread traces back to potential- reward shaping (PBRS) (Ng al., 1999), which introduces additional or misaligned rewards in principled ways to preserve the optimality of desired behaviors. More recently, Random Network Distillation (RND) (Burda et al., 2019) emerged as leading exploration mechanism, with subsequent extensions such as Ma et al. (2025). In this same context, numerous other works propose reward signals (often spurious with respect the true task objective) that encourage an agent to explore the state space in ways that eventually uncover genuine rewards (Pathak al., 7017:2 Zhang et al., 2021; Wang et al., 2023; Li et al., 2024a). Spurious rewards have thus played a substantial role in improving exploration. One prominent theoretical foundation is posterior sampling (Russo & Van Roy, 2014), which motivates exploration through uncertainty and has been generalized to broader RL settings (Chen et al., 2024b; Xu et al., 2025).
Spurious reward in RLVR. We now turn to recent works that study spurious rewards in RLVR. Although these works report broadly similar empirical phenomena, their experimental configurations differ important ways. In Shao et al. (2025), the prompt omits the standard Qwen-style instruction to place the final answer in box. As they note, Qwen-Math h is highly sensitive to prompt formatting, and such differences can substantially shift baseline performance. In contrast, our experiments follow the default Owen prompt used in verl (Sheng et al., 2025), which explicitly instructs the model to place the final answer in a boxed expression-mirroriin the RIVR verifier in verl, which extracts the boxed answer for scoring and reward assignment.
Apart from this prompt choice, we match all other hyperparameters in Shao et al. (2025). Oertell al. (2025), however, adopt markedly different configuration: (i) a rollout-length cap of 1024 tokens (well below the 4096-token context window of Qwen Math), (ii) different training dataset (MATH (Hendrycks al., 2021) instead of DeepScaleR (Luo al., 2025)), (iii) significantly smaller learning rate (1 x 10-? versus 5 x 10-7 in Shao et al. (2025)), and (iv) reduced batch size (64 versus 128). The smaller learning rate changes the effective update magnitude, and the smaller batch size yields noisier estimates of the stochastic reward distribution. Given these differences, the empirical results reported across prior works are not directly comparable.
Contamination. We further comment on potential contamination in the Qwen2. 5-Math models. As reported by Wu et al. (2025), contamination in Qwen Twen-Math has been observed only on validation sets (e.g., MATH500). Beyond these findings, the official Qwen2. 5-Math technical report (Yang et al., 2024, Table 1) also acknowledges possible contamination arising from the close similarity between the training and validation sets of the MA TH dataset (Hendrycks et al., 2021). Their training corpus comprises two components: (i) CoT data synthesis, which includes GSM8K, MATH, and NuminaMath (Yang et al., 2024, 1.1, and (ii) a tool-integrated reasoning dataset containing GSMSK, MATH, Moth NuminaMath, MuggleMath, and DotaMath (Yang et al., 2024, $3.1.2). In contrast, experiments employ the DeepScaleD training set, which consists exclusively of selected questions from AMC, AIME, Omni-Math, and Still (Luo et al., 2025). None of these datasets appear in the training sources listed for Qwen2. 5 h- Therefore, we believe that our training data does not overlap with the datasets used to train Qwen2. S-Math and is thus not contaminated.
LLM entropy. Agarwal et al. (2025) demonstrate that token-level entropy minimization can substantially improve LLM reasoning without verifiable feedback, arguing that reduced entropy increases model confidence and reveals latent reasoning capability. This mechanism parallels clipped training under random rewards, where updates primarily modulate entropy rather than exploit informative rewards. However, we show that entropy minimization alone may drive the policy toward
16

---

Published as a conference paper at ICLR 2026
low-entropy yet suboptimal solutions; hence, entropy should be viewed stabilizing regularizer rather than a replacement for genuine RLVR signals.
Related work explores entropy through the lens of self-confidence. In particular, Prabhudesai et al. (2025) use low-entropy rollouts as implicit rewards, achieving gains across diverse benchmarks, while Gao al. (2025) show that even single unlabeled example can improve reasoning via entropy reduction. Methods such as EMPO (Zhang et al., 2025b) and Zhao et al. (2025b) similarly enhance performance in unsupervised settings by amplifying model confidence. van Niekerk al. (2025) further construct preference datasets from confidence scores, achieving RLHF-level improvements without human feedback. In this context, Cui et al. (2025) propose a simple but influential empirical relationship between policy entropy H. and model performance R, fit across extensive experiments:
R aa exp (H) + b, a >> 0.
This relation suggests that performance increases monotonically as entropy decreases but plateaus once entropy collapses too early. Intuitively, when a model overemphasizes certain tokens, its output distribution becomes overconfident and loses exploratory capacity, leading to a performance ceiling.
To mitigate early-stage entropy collapse, several works propose alternative strategies. Shen (2025) analyze why entropy regularization suffers from limited benefit in RLVR training for LLMs by attributing it to the vast response space and the sparsity of optimal outputs, and then introduce an adaptive control method using clamped entropy bonus with automatically tuned coefficients. Song et al. (2025) show that ORM yields induces sharp reductions in output entropy and diversity (as shown by lower passEn scores), and propose outcome-level entropy bonuses to counteract it. Prior works (Wang et al., 2025a; Yao et al., 2025; Zheng et al., 2025; Cheng et al., 2025) also develop additional techniques for controlling entropy during RLVR training.
Reinforcement learning for LLM. Proximal policy optimization (PPO) (Schulman et al., 2017) has become standard for reward-based policy updates in LLM training and remains a core component of RLHF, However, since PPO requires loading and maintaining four separate models during training, it is computationally and memory intensive. This has motivated the development of lighter- and adapted policy-gradient gradient updates (Li et al., 2024b; Ahmadian et al., 2024; Shao et al., 2024; Guo et al., 2025). In parallel, advances verifiable reward construction (Cobbe al., 2021; Uesato et al., 2022; Zelikman et al., 2022; Singh et al., 2023; Hosseii et al., 2024; Lightman et al., 2024; Wang et al., 2024; Luo et al., 2024; Setlur al., 2025; Zhang et al., 2025c) have enabled reinforcement learning substantially improve LLM reasoning, particularly in mathematical problem solving. Beyond training algorithms, recent work also explores post prot-psoc and collaborative strategies to strengthen reasoning performance. Kay et al. (2025) and Zhao al. (2025a) propose consensus-baas based and answer-age -aggeeggggentttoon methods within multi-model frameworks. Chen et al. (2025a) introduce self-questioning paradigm for iterative refinement, while Park et al. (2025) develop an online multi-agent collaborative reinforcement learning framework.
Offiine alignment. Direct alignment methods (Rafailov et al., 2023) provide simple and stable offline alternative to RLHF. Extensions to DPO include broader ranking objectives (Dong et al., 2023; Yuan et al., 2023; Song al., 2024; Chen al., 2024a; Liu al., 2025a) and lightweight variants that remove the reference model (Hong et al., 2024; Meng et al., 2024). Since DPO avoids reward model, limited human preference data becomes a key bottleneck; recent work addresses this by generating additional preference pairs via SFT policies (Zhao et al., 2023; Liu et al., 2024a). The framework has also been generalized to token-level MDPs (Rafailov et al., 2024) and broader RL settings (Azar et al., 2024). Complementary approaches incorporate online human feedback to reduce distribution shift and improve reasoning (Dong et al., 2024; Xiong et al., 2024; Pang et al., 2024). Another line studies unintentional alignment and proposes remedies (Pal al., 2024; Tajwar et al., 2024; Liu et al., 2024b; Xiao et al., 2024; Yuan et al., 2025; Razin et al., 2025; Chen et al., 2025b). For example, Razin et al. (2025) filter noisy preference pairs using CHES similarity, while Chen et al. (2025b) show that combining comparison oracles with DPO mitigates unintended alignment effects.
B ADDITIONAL EXPERIMENTAL RESULTS
We begin with a high-level overview of the experimental setup, followed by a comprehensive presen- tation of results in both the main text and the appendix. Our experiments are organized around two
17

---

Published as a conference paper at ICLR 2026
Clipped 0-1.1
Clipped -y- 0.15 -Qwen2.5 Matt2-5-Math-75
Ratio clipiing activation OwenZ.5 Math
$.20
0,09
0.01
0.000
350
1.50 2:0
200
350
Figure 5: All experiments follow the same setup as Figure 1, varying the threshold with six independent runs for each setting: trials with clipping ratio & = 0.1 (Lefft); trials with clipping ratio E = 0.15 (Middle); and the ratio of clipping activations across E e {0.2, 0.15, 0.13 (Right).
objectives: (i) characterizing the interplay between clipping, policy entropy, and performance under spurious rewards, and (ii) assessing whether the observed benefits of spurious rewards generalize beyond Owen-Math to a broader range of model families.
For the first objective, we focus on Owen-Math- 7B and provide a controlled setting for examining how clipping and policy entropy affect model performance under spurious rewards. This choice is supported by previous empirical findings and practical considerations: -W moderate parameter count and relatively short 4K context window, stabilizing training and reducing exposure to issues such as gradient explosion. This stability is crucial since clipping is commonly used to prevent gradient explosion in larger models with longer chain-of-thon of-thought rollouts. As shown in Figure 2 (Right) and discussed in $4.2, removing clipping on stronger model with longer rollouts can cause catastrophic training collapse, making it difficult to disentangle the core relationship between clipping, entropy, and performance. Indeed, one key motivation for applying the clipping in GRPO originates from the need to stabilize training for the DeepSeek-RR RR1-676677 66-671 671B model. For the second objective, additionally evaluate two contamntaminated model families, Llama and QwQ, for which no contamination has been reported in the community, to demonstrate that the benefits and behaviors of spurious rewards extend beyond Qwen-Mathh and reflect broader RLVR learning.
has a
In $3.2, we examine how clipping affects model performance by comparing training with and without clipping. In $4.2, we validate theoretical findings on the relationship between clipping and policy entropy. For consistency, these experiments use Qwen Wwven-Math-777e 7B trained on the DeepScaleR dataset. Then, in $4.3 and this section, investigate the interaction between entropy and performance on the more challenging AIME training set, again evaluating both clipped and unclipped training. We find that policy entropy is not directly related performance improvements, and that models gain significantly less from random rewards when their baseline performance is reduced by dataset difficulty. This stands in contrast to stronger L.1 ama and QwQ models, which continue to benefit from random rewards even on harder tasks. Finally, in $5, we proceed to a broader spectrum of model strengths, showing that stronger models are more likely to benefit from random reward signals.
Ablation analysis. We ablate the GRPO group size G. Larger groups (G 16) yield more balanced binary rewards, while smaller groups increase the likelihood of extreme reward-misaliggmeet events, such as entire groups receiving reward 0 despite contain- ing correct rollouts, or reward despite containing incorrect ones. Thus, reducing G inherently amplifies instability from the reward- misalignment perspective. As shown in Figure 6, using smaller group size (G- = 8) allows most runs to improve, but leads to higher variance and less stable learning dynamics throughout training.
0.20
0,20-
sau
Figure 6: Smaller group size.
We further analyze the effect of varying the clipping ratio threshold e. Indeed, Figure 1 examined the cases ε = 0.2 and oo (no clipping), showing that relaxing the clipping threshold does not degrade performance. However, this does not yet confirm robustness under stricter clipping. To address this, we report additional results for 5 {0.15, 0.1} in Figure 5. Across these settings, we observe behavior consistent with Figure 1: (i) some runs fail to improve, as predicted by our probabilistic reward- wawaarddmilliigmmee. framework; and (ii) successful runs converge to roughly 70% validation accuracy regardless of the clipping strength. As discussed in $4.1, clipping primarily influences policy entropy. Among the improving trials, stricter clipping tends to reduce variance
18

---

Published as a conference paper at ICLR 2026
across seeds, reflecting more deterministic policy toward convergence. Taken together, these results indicate that our findings remain robust under different choices of
Unclloped Qwen2.5 Mathh-7 AIME Training Set
Unclppped Policy Estropy
Unclipped Single trial example
1s by urioe
Strouge
3.00
8.0
Coodaa
Figure 7: Unclipped Qwen2.5-vv 5-Math- on the hard AIME dataset: independent runs following from the setup in Figure 3 (Left); corresponding policy entropy dynamics during unclipped training (Middle); joint evolution of model performance and policy entropy for an example trial (Right).
Policy skewness comparison between positive and negative @(m)
Nele
6:0-
122
4.00
2200 1220
Wetl 0.0004
Figure 8: Visualization of policy action distributions over 12 prompt x; Each subplot displays the sorted log-probability of T(y x;) for 64 sampled responses from each prompt x;. Columns 1-2 (blue) correspond to prompts x; with (( x;)) > O, while Columns 3-4 (orange) correspond to prompts with ( xx)) 0. As discussed in Theorem 4.2, the entropy increase under unclipped training can occur only for the skewed one shown Columns 3-4.
Unclipped training. In Figure 3, we present clipped training results for Qwen 5 S-Math- h 7B on the AIME dataset, where clipping induces entropy collapse as demonstrated by Theorem 4.3. Although entropy decreases, performance also degrades, indicating that lower entropy is not reliably associated with better model performance. This raises an important question: what happens in the complementary regime where entropy increases under random reward? To answer this, we conduct additional experiments and report the results in Figure Across independent seeds shown in Figure (Left), the behavior remains qualitatively similar the clipped case in Figure 3: some runs improve, others degrade, and overall learning dynamics appear stochastic. In Figure (Middle), we empirically confirm the predicted entropy increase under unclipped training. Among the improving runs, Figure 7 (Right) provides a representative example which performance improves even as entropy increases. These results answers question: there is no direct causal relationship between policy entropy
19

---

Published as a conference paper at ICLR 2026
and model performance. Both clipped and unclipped experiments support that, as the model's initial performance on a dataset decreases, its likelihood benefiting from random rewards diminishes.
Policy skewness. We empirically evaluate the skewness measure (TT) introduced in Remark 4.2 on the actual Owen-Math-7B policy. Recall that under unclipped training, entropy increase after a single update only when ((T)) < 0. Since the policy induces different action distributions (a x) for different input questions x, we estimate skewness across questions by sampling the first 500 examples from the DeepScaleR training set (Luo et al., 2025). For each question x, we generate 64 responses y from the policy using the same sampling and decoding hyperparameters as $3.2 and compute an empirical estimate of ( (- x)). We visualize selected prompts x, along with their corresponding skewness values in Figure 8, providing clearer picture of how Qwen -wennMatth-
behaves across the dataset. Among 500 sampled questions, 358 ones satisfy (( xx)) < O, which consistent with the observed entropy increases for the unclipped training.
C THEORETICAL ANALYSIS
Setup. We model the next-token generation using softmax at each history. Let V be the vocabulary and h, (x,y<t) the history. For each prompt x E and response a (a1, (01,-..,,a) ag) E V
where E V/ we have
Ta x) II no, (at ht), where no, (ax he) En'ey explOr.by..!)
exp(Oe,ha.aa)
where 0 (0 ..... TD), and 0, E RI for all =1,. L.
Given trajectories drawn from Vold. we define the per-token ratio (0) To Ib
For group
Foa by, (0))
{ Ti=1 (- x) and the corresponding outcome-reward advantages {A; I== the empirical per-history advantage used in the policy update is
G
A(h, a) 2/2 Σ
1/b 1/b/9 =by"" Ai. Raal
1=1
t=
This can be derived from Eq. (3) using L = ly for all i = 1,2...., G. Following Williams (1992) and Li et al. (2024b), define the clipped surrogate loss with per-token ratios without adding separate length normalization terms as follow,
min 1 (0)A, clip(r! e, (
(0)
ΣΣen t=1
Without clipping, the surrogate loss reduces to
( = (2
(0)
We derive the closed-form token-level update for optimizing the unclipped surrogate loss with forward KL penalty to Vold as follows. To begin with, notice that
I
j(0) - 1=
Σ3 e (0) A,
ΣΣ h DE
Ta(a
h)
1Σ> J=
1(h
-h. In)
A
Hatata
t=1
Ā(h,a)
Using mirror descent (MD), for each iteration, one solves
j(0) 1 Σ ( h) Tota(- h))
max
0
20

---

Published as a conference paper at ICLR 2026
which is equivalent to solving for each fixed h, b)d(h, a) (
Σx(er
Σ x(a h) log sla/h)
max
maaaaa
=(-b)
Introducing a Lagrangian multiplier Xn for the probability simplex constraint E. <<a h) 1, by first order condition
Å(h, a) (log =(a[h)
+1) + An =
(a/h)
Solving the above equation for we yields
Te (alb) exp(n((haa)
Ta(a h) Lv *eau (a'[b) exp(n
and evaluate at the realized pairs (a, h) = (y/ h in training. Note that the above GRPO update can be analyzed by interpreting it as one natural policy gradient (NPG) step under softmax tabular parametrization (Agarwal et al., 2021).
Algorithm Iterative Group Relative Policy Optimization
1: Input: model parameters 8mni reward models rep prompts X, and hyperparameters E, B,
2: Initialization: < Pinit-
3: for iteration = 1, do
4: Tref TT- M do
5: for j
6: Sample batch X,, from X.
7:
Update the old policy model Tew f-
8:
Sample outputs {o: ~ To x) for each question E Xj.
9:
Compute rewards {r:) for each sampled output o, using the reward model n...
10:
Compute A, for the t-th token of og via group-relative advantage estimation.
Update the policy model Ta using GRPO. 12: end for
11:
Update rip through continuous training using a replay mechanism.
14:
end for Output: .e.
GRPO analysis. Interpreting the GRPO update as natural policy gradient (NPG) step has been widely adopted to study entropy dynamics throughout training (Cui et al., 2025). Here, we summarize the key components of GRPO in Algorithm 1, which motivate our reduction an NPG-style update for analyzing the effect of clipping in GRPO. We note that Algorithm should be viewed as an abstraction of GRPO implementations used practice (Shao et al., 2024).
In the outer loop, reference policy is fixed once per iteration, and the per-step objective may include KL penalty that constrains the updated policy To to remain close to Tee, thereby controlling the effective step size and preventing excessive policy drift. Recent "zero-RL" setups (see Yu et al., 2025), which are also adopted in the empirical evaluation of Shao al. (2025), set the KL coefficient to zero, effectively removing the explicit KL term from the objective. As such, we likewise omit the KL term our analysis. Under this regime, the outer loop does not affect the subsequent analysis.
In the middle loop, which is for GRPO training, the model samples each batch, which is update-style agnostic. The key difference between exact-GRPO- and NGP-style update happens in the inner loop (line 10). First, j is a constant hyperparameter for the number of actual updates per macro batch, used to improve sample efficiency and better optimize the surrogate while clipping limits drift from Tol- Therefore, the statement for GRPO iteration = 1...., A performs μ optimizer steps the same
mini-batch to maximize the clipped GRPO surrogate. At each step, importance ratio r(()
! [x) stan (9)x) [x)
are recomputed and the loss Ei, min(r!" (4) A, clip(r?") (4)
e, 1 + =)A} backpropagated.
In GRPO, the p-step inner loop produces chain of micro-updates whose importance ratios evolve across steps, making the expected contribution of clipping analytically intractable unless one specifies
the per-step clip-activation rate (the expected fraction of tokens/micro-hatteeee with & [1 11-e, 1 +e)).
21

---

Published as a conference paper at ICLR 2026
This rate is model- and dataset-dependent and is only available empirically. Conditioning the empirically measured activation rate, we collapse the I clipped micro-steps into a single NPG-update with actual model-specific token-level expected clipping activation ratio. This surrogate preserves the first-order effect of clipping and enables tractable bounds for our theoretical results. Comparing to recent works that directly used NPG for GRPO analysis, our setup for clipping analysis validly justified, facilitating the later theoretical derivation and without unjustified oversimplification
C.I
MISSING PROOFS IN $2
Proof Theorem 2.2.
Fixing h, we rewrite Eq. (2) as
Too (a h) Tes(a/b exp(nA(h..))
(u(n)
where
(x(1) = Σ Tota h) (xp(nA(h)))) a)) B ( (xxp
aEV
Taking the logarithm of both sides yields
log( (new (a h)) = log(TTaa h)) + n a) - log Zn (n))).
(10)
We define (n) log (og(Zn (n)).. Then, we have wh (0) (0) p(h). and (0) = (h). Fix-
ing (h, a), we define 4 (h, == El=i" 1(h; (0) h.yl a} E {0. 1} and N = Σ (h, Then, we have
G
Å(h, a) Taj
Σ 4(h
By using EG A; and Σ
G Ei=1 A? S G, we have
G
Σ
7, a)4, S - N) 5
NN
This implies [A(h, a)l S mmii for all a E By using Taylor's theorem with Lagrange remainder, we have
(n) (b)n o (b)r) + 8 (e"((E)"" for some E (0,
Since (1) is the third central moment of Å(h, a) under the exponentially tilted distribution and Å(h, a) [-M, M], the sharp bound [E[(( [[X]
(b-a)"
for X [a, b] yields
6V3
(w) 6v3(5miaa)
Therefore, we conclude that
(w.(n) ((h)n - 36 33mmm))
Combining this with Eq. (10) yields the claimed inequality with C' 33mmn for log(r(h, a)) and the standardized case follow immediately.
The statements
Proof of Theorem 2.4. We prove three statements one by one follows.
(i) L 10 + 1) be (rI....., .G) = (1 rı..... 1 - rG). If (ri...., T(r....., rG), then we can also define A and F' similar to A, and With the notation above, we have F' = 1 1 and
r - F' = (1-1,)) - - (r, - F).
Hence S,r S, and A/ AA Since (Ii...., .G) is i..d. Bernoulli(t) its law is invariant under r. Thus, we know (ri,..., r'c) d (ri.... rG) and Ai d A.. Combining the above facts, obtain A, A; and thus [[APK-1] 0.
22

---

Published as a conference paper at ICLR 2026
iii Let K Ej=1 ry, then F
K and S. G
((G-K) G(G-1) Thus,
VG - 1
G-K
ifr; = 1,
[A.1
Gk
VG - VG-1 IV G(G-K)
rf = 0.
Thus, it is easy to see
1 |A. S VG
v0
(iii) Let K = E Dj Binomial(G, 1)) and p = G
k
On (1
S K
< S G 1}, we have
1-P
r=
r
S. vp(1 -pl1-p. p),
A
p
I-pp
=-
r;
Hence for ls E N+
E A K] p (177)
1-p
k/2
+ (1 - p)
p
k/2
k/2
[-pp
14
with = 1-p > 0. Define (z) == x k/2 / + x² 1-2//2
x - 1. Then
(x(x) = \ g 1) 1/2 2 ( 1) x k/2-1 0, <=> 2
and (1) (1) 0. By convexity, h2 (x) 2 for all 0, hence E|| * K] 2
K
whenever 1 < K < G 1. Taking the expectation and using the fact that A, = 0 on
{K {0, G)} yields
E[AI| Σ
(
(
2
G
KJ
Σ
if 2.
K=1
Finally, is trivial to write down B[]A:]] with the above information.
This completes the proof.
Failure of Eq. (5) under random reward. We have
Covy (Iog(Toul( A(x, y))
(x)x log[TTuaayy x))A(x,y) - Ey Ey [log(Tolayy xx)] B (A(xx
Eywxsa(+)))) [log(Tola(y x))] E (y)
0.
In other words, the co-variance between A(x. and (y x) is uninformative under random reward since these two terms are independent from each other in this specific setting. Thus, a more accurate estimation of ([Tnew) H(Tola) beyond Eq. (5) is desirable.
C.2 MISSING PROOFS IN $3
Proof of Theorem 3.2.
By definition, we have City E DJ A. Thus, we have
I.
BI
EDD DD Σ B((D))2A
Σ EED
(II)
1=1
sflt
Recall in the proof of Theorem 2.2, we have shown that [A(h, a)l S 2
We also have
end(bee) Zm(n) Zn(n) Σ Tala (a h)ea)(
r(h, a)
Since A(h,a) 2womn we have (T) 2 7/ (/(TTei))) and enr(ha,)) < (27mmm)))) Thus, we have r(h, a) < e Sa//ainn =: Rmmax This implies r, S Rmax for all In what follows, we bound the diagonal and off-diagonal terms in the right-hand side of Eq. (11).
23

---

Published as a conference paper at ICLR 2026
Diagonal term. On 1T) 1}. we have ri > 1 and 6. + Thus, have
|D [11 + rz) = (r: - e)T - (re 1)1..
By using the fact that (r - 1)2 < 2xδ(r) for all x 2 1, we have
(D & (r: 1) - 2rt
Since |A| S M (c.f. Theorem 2.4), we have
B[(D)) A & ME( ~ 2M
Since á is strictly increasing and rt I R we have M()(1)) S Ru o( ( ) Putting these pieces together yields
B((DT)PA3] s 2LM Rn ( ((R 2LM Re ( p++
(12)
Off-diagonal term. We define Ze := D and have /zl & | S M DJ Then, we define (Rz 1 +)+ and have
A
Df = (rr 1- s)4 & (R
- 1 - -1- = 1,
Putting these pieces together yields |Ze| S MA, 4 and
E Z1 S M (4))
B S (4
Σ4
(13)
Since E/'1 I,+ S L, we have
& L( 1t).
Thus, we have
1
27
I <
E
Σ 1t
LE Σ 1, L'p.
On (1+ = =],, we have ri 2 1 + & and ((r:)) 2 161 + E). This implies 1 S
oirs) x(1+)) and
L E )
1
Lo(Rman) (11+e)
((++))
Putting these pieces together yields
E
Σ
++
< 1 min
p.
(
((Ruua)) all+e)
(14)
Plugging Eq. (14) into Eq. (13) gives
2
Σ sfet
ElIZ,Z,11 S (2 min p.
(15)
Conclusion. Using E|Ct|| S v EllGt| 12] and /I / S V + /u for x, 2 0 together with Eq. (11), Eq, (12), and Eq. (15) yields
ElIGtI M v 2p L LR (amaxx((Ri ((R + MLA min { (Pt, o(I+s)
(16)
Let n//min S so R e" < e. For u E [0, 1]. we have e (e 1)u and "(e") S u² Thus, we have Rimxx o R S eu² and A S e" S (e 1)u. This together with Eq. (16) yields E| I e II + min can L} where C₁ = M 2 min: M(e TT m min.
and C3 (e (1 + - min This completes the proof.
24

---

Published as a conference paper at ICLR 2026
Proof of Theorem 3.4. We recall that |A] S 2
M. By Theorem 2.4, A, is symmetric, and
thus A(h,)) :) is also symmetric. Thus, we have re(n)
d
r (-))
and
E
A
re(
= E
|A
rel (c(-))
1=
This implies
L
| Naw || = E |A
Σ
re(u)tra(nn)
2
We write rn = (h,
where (1) E Ra b) (exp(nA(h, a))] Then, we have
[n(
To
Tnp-n v (n()))n((1)
2
By using the convexity of x: + t on M-M we have nh S M+z
+ M-# - = co + मे sinh(n)), fol x E [-M,M].
2M
2M
Averaging under T. h) yields
(n( M)
w(h) M) Zn(-n) S cosh(n))) p(h
M sinh(n
where phh) = Be (-bb) (A(h, a))] Then, we have
Zn (n))) S cosh
[((b] (MM
(nnh S cosh
Putting these pieces together yields
Intr
M)
cosh(n// (2
Applying this to (h,, Yt), using cosh(v) exp(x² 2/22//// /2), and summing over yields
((()))((())))
L coshi(n (2Tmin))
Le ?² (8= Le Cyy
2
Taking the expectations yields Eq. (7). This together with Theorem 3.2 yields the desired bound.
C.3 MISSING PROOFS IN $4
We first summarize the setup and notations used in this section. We only consider L. = 1 (bandit case) for illustration. Denote new as the new policy obtained from Eq. (2) (unclipped case) and the new policy obtained from
{F(T):= = j(m) 1 DKL (T (Toaaa)
max REAly
where we only consider upper clipping in the surrogate function as follows,
j(T) min (r)(r((())) (in(r(y(0,11e))))
We define ra(a) = aie (a) Tas(a)
rc(a))
= (a) ma) and
G S_ = Σ A.1ly = a, <3.
= a, (a == Σ =
Lemma C.1. Let V. be the vocabulary and À be defined in Eq. (3) with L Then (17)
E[H((T,)) H((Taaa) = =/ (A) + Cov mas (A² log Fora)l + E[R(r)),
Whenever there exists â E (0,1) such that (((a)) S 27 for all E V, the remainder satisfies the explicit bound
(2)(2
R()]] <
(192 + 176 log( 11)
176n *
"1"E Emain [A4]
(18)
24
25

---

Published as a conference paper at ICLR 2026
Proof. Write L(a)= == log (a) and (n)) log Z()). Then log (a)(a) = L(a) +A(a) ((n)
and hence
H(( Bx [log myy -R,, [ nB [A] (n))
(19)
Differentiate (, Told (a)c"A(u)/221) (Z(r):
d mn(aa = (0)(a(((()) - -v(())), (((1) = EET,, [4].
dn
Thus, for any g: V + R.
E,, [و] Σ g(a) dn (a) = Cova, (g, A).
(20)
dn
a
From Eq. (19) and ('(1) En,, [A], we can rewrite
H((T)) -B= [4] m7/5 (n) + v(n).
Differentiate once and use Eq. (20) and (n) = VVrx (A):
H' H'(r) (n) - Covx, (L 17 Vaxx (A).
Differentiate again and evaluate at 17 = 0:
d
(0)
Cov (L, A)
Var (A).
dn
1==0
Now expand Cov,, (L, A) E,,, [LA] ETT, [T, and differentiate each expectation using Eq. (20). A direct calculation gives
d Cov A) = Cov (I, A2) 2R [A] Cov (b, A). dn
At 17 = 0, since E [A] = O, we have
Cov Cov=, (L,
= Cov AZ)
(21)
dr
9=
Therefore
- (A) Cov (A², L),
(22)
which, with expectation, is the coefficient of 1)2 in Eq. (17).
Define f() ;= =B,,(TT,)) - H((TTla)] where the outer E[.] averages over the randomness of A. By symmetry, (+ (-, -A), hence H(T-,i A) H( - A) and f(n) f-(-)). Therefore
f'(0) (3) (0) 0. and Taylor's theorem yields that for any vi 0 there exists & E (0, 11) such that
50(0)
( n
f(1) 2
n²
24
Combining with Eq. (22) gives Eq. (17) with E([(n))] = f( 17a and the bound
24
B[R(n]| < n
sup 15(4)()) (n) < .1
sup E
(23)
24 [e
24
tE(o.n]
Thus, it remains to bound (H(4) by E., [A+]. Let my Ex, [A]. X., := À - m, and L., Ex,,, [Z]. A direct four-times differentiation of Eq. (19) using Eq. (20) yields the explicit identity
(n) -E,, [L. X -E,,, X EE [T,,,,X, X,
6E_,, X BB, X
-3E,,, | + 9 ( 1x, + 17 (10 Ex, X Bn, |X, E,,, (x
(24)
Now ||0, I log(1 (g((1/Tmin)) because I(a) [log uuin 0). Also, by Sauchy-Schwar and Hölder, (B(x)))² < [(X;), E|XX, ||XXX,1((I,111111 < B(X;)
26

---

Published as a conference paper at ICLR 2026
Therefore the first line of Eq. (24) is bounded by 11 log ) [X,]] and the first two terms of the second line by 12 [ For the last term, |m,, - [[[AA|] = 12(2 and hence [Xx,) S |A] m,yl - f pointwise. Thus
| - - [[
1 B[ B[[X,19) # [[
B||x,
which gives
| -| S 1X | $ _ E[
Plugging these bounds into Eq. (24) yields
(7 (n)l - 12 11 log( -
lll *
Bri, [x,]]]
(25)
Finally, (u- - -0) S 8(u + w² and Jensen's 's inequality imply Ex,, [X S 16 [A*) hence
(7 = 199 + 176 Amin +
1764 $
E [A*].
(26)
By Jensen's 's inequality and the fact that Exaid [A] = 0,, we have 2(n) 2 1. Hence for 17 2 0,
E [A+]
Z(n) Eroule ( A+] Peroule" Ae s e/((2 Errea (A+),
Combining with Eq. (26) gives, for t [0, m).
(m)ll S 2/(2#) 192 + 176 log( Aamin
+
176t #
E Er [49
Taking the supremum over t E [0, yields
sup EIIH((() (n) S (25) (192 + 176 log(
+
1767 Fmasal (14)
E
#
tE(O,g]
Plug this into Eq. (23) to obtain Eq. (18).
Corollary C.2. Under the same setting as in Theorem C.I, if we know that conditioned on an event
B, [A(a)) S z then we have a refined bound
[B[R(N)] S P(BB)C(*) (1- (LF()))))
(27)
where
(2n)
C(T) == 24 192 176 log(
-ا Tauin
176g
E E
Bota [A+] 1/²
Proof. Let : P(B). By the tower property, E(R(n)) (B(R()) B] + ()) | B*),
hence by the triangle inequality.
[[((n)]] s (n) B| + (1 ((())) | BT).
Conditioned B we have the envelope [A(a) to the conditional probability space yields E[)(n) and [A(a)) S 27ala(a) S
for all a. applying the proof of Theorem C.1 B]] S C(+). On the other hand, (a)a 2 Tmin
27mmn for all a, so Theorem C.1 also gives [E((()) B I C(Tmin).
Substituting these bounds back proves Eq. (27).
Lemma C.3. Let V be the vocabulary and À be defined in Eq. (3) with L = 1, then
Va (A)] = 1-21- 1- (1VI-1)
E|Cov mas (1,,1o(TTTl))) = 1-21-G 6
Σ L(a) - [21)
E Emaaa (14)
S
E[A') G
(S2 - 7Si + 12
6) + 3E (Si 2|V| + +1,,
G
where S1, S2 > 0 are
S
1--1
Emin
+ -[V[-1)amna
S
(umin) 1 (I-(V-- 11
-
27

---

Published as a conference paper at ICLR 2026
Proof. We first compute E[Var (A)]. Indeed, we have
var (A) = Σ Tola Tala(a)(A(a))
Σ (oossa)A((a))
G
Σ mata(aa
Tacla]G
A A 1y(0=o) Σ
Σ
Roula)G
sau(a)
Σ
A (y(0
i=1
Σ
A,A,
Σ (a) [a) - (yriszss)
=
Using the fact that A, is independent of y(), we have
B[Var (A)]
B[A3]
E
Σ (a)
1t Σ
[[A
E
Σ स I
itj
= &
B[A
+
Σ
B[A,A,]
1=1
Since Σ i=1 A, 0, we have E A3 Eits ArAj. In addition, by Theorem 2.4, we have
G
E[A?] = 21-G Thus, we have
Va (A)] ل (1V1 11.
We then compute IF(CV vas (A², (og()) ] Indeed, we have
Cov (A², (oL,Tog(T))a))
Σ (a) log( Var (A)
Σ Tota(a) (a) log Tata(a))
Then, we have
E(Cov (A?,, (A2, (og(aa)())) 1-21-G
Σ log(TTou(())) -10) - Σ (a) log((oaa(a)))
Finally, we compute EI We define [i(a) 1(y(0) =a]. Thus, we have L((a)
Bernouli(p)) i.dd and it independent of [A,] We write S(a) = = A,4(aa) and
Σ (a)((a) Σ (ala(a)
Aa = a)
(S(o))
TaulaGG
[mal
Conditioning on { and using DE A = 0. a direct fourth-moment momont expansion gives
G
Em [S((a))* [ = (To - 7(TT + (a() (aTaaa
Σ
A
G
2
+ (3(TTol(aa) 12(Too + ((Tooa((a)
Σ
A3
This implics
(
Emy E [Ā+] 11*]|{A [
- Σ Tue(aj
+ 24/1 -6)
A
= t
[T))
G
2
+
G4
1
3
Σ
mau(a)
12 +3
Σ A?
i=1
28

---

Published as a conference paper at ICLR 2026
In addition, we have
E
A
GELAA],
Σ A
GLE[A + G G 1)E[AAAA]
and Σ
1-
V
[a)
Maain
1-(1V|-1)m 1yamio
[a)
*
[ 11)mmn))
Putting these pieces together yields
E E [14]
]|A!!
(S2 7S, + 12 - 6) + SELATH4G-DD(AAAAA)
(Si 1-1--111 -2/21 + 11,
G
where $1.So > are
S
S2
D- (Tain) ) (1-(() 1)
Emin
1 1-([2]
m)min
In what follows, we provide a detailed version of Theorem 4.1 and its proof. Theorem 4.1 (restated). T. = 1, then without clipping. for any 1j > 0, we have
(TTww)))) H((TToaa] 2 + B[R(n)), [R(m) - On
where co
1-21-G
and
2G
1+ Σ (o)(((())== - P Σ T(a) Tia) log(T(a)))
(TT)
NEV
aVE
((Tmi))
m/ii
199 + 176 log(
+
176n
M,
24
Tamin
M (Tmin)
E|1l G3
(Tmin) - 7S1 (Tmin) + 12 V/ - 6)
= +
(S1 (Tmiu)) - 2| + 1),
G
with
S (TTin)
VV1-1
Amin
T-(
1-(
(2(Ti)))=
(Tmmn)) 7
(1- (V) 1))mm))
Proof. If L = 1, there is only one h and every rollout has the same h, so we ignore h for simplicity. In this regard, we abbreviate (aa h) as π (a), (6) as ((),, and
Á(a) = 1/
Σ
10y A
Ram(a)
We rewrite the update rule as follows,
Tyy (a) == Tnew (a) 2(1)
2(n)
Σ 8EV
Total (a) n
We define (n) == log (n) and u(a) == nA((a) (nn) In Theorem 2.2, we have shown [Å(a)) < 1/(2mmiaa for all a, so by taking के Tiin we can apply Theorem C.1 to obtain
(T,) H( = E/ (A) + Cov 22,100 log mota)] + E(R()).
(A"
Finally, apply Theorem C.3 and define
(<x) = VI - 1+ Σ log( - -V Σ ((a) (())lo((T(()))
aEV
to obtain the desired result.
29

---

Published as a conference paper at ICLR 2026
We provide a numerical example in Figure 9 to illustrate the above theoretical result. Building the two-armed setting in Remark 4.2, we conduct additional numerical experiments under unclipped GRPO training. As shown in Figure 9, entropy growth occurs only when the policy is initialized in sufficiently skewed regime. This observation underscores that injecting spurious rewards without clipping can help preserve restore entropy in GRPO training, particularly when the policy entropy has already collapsed or degraded toward a highly skewed distribution.
Fat Policy Intiadiatiinn
Showed Pellcy lettedartion
Figure 9: Simulation of policy entropy evolution unclipped GRPO training. Each panel includes the result with 10 independent trails. Flat (relatively less-skewed) policy т initialization (Left); Skewed policy ı initialization (Right).
Then, we proceed to the entropy dynamics with upper clipping. Before proving Theorem 4.3, we present some useful lemmas.
Lemma C.4. The surrogate objective J(x) with only upper clipping can be rewritten as [(T] = à - (a)r(a) + S+ 1 +
aEV
There exist À E R and (HabaEv with Ma 2 0 and (aro(a) = 0 such that, (S + (+))e)
Iaola) (logro(a) + 1) Ato (a) Pa
for every a,
7
where
1,
(a) < < +1
E. =
0,
if (f) > + E,
10,
otherwise.
In particular, rf > 1 1 E, we have &a = 0 and log(r(((()) = n sal n7
Proof. Since π (a) (a) we have E, (() = 1.r(a) 2 0 and
DTT(T rrl(z) = Σ (())((a)lo((((a))
By definition, we have
IS S4 m()))))r))) min(r()),1+)))) - Σ
क (a)r(()) logr(o)))
A new
arg max
720
TOrEA
Since the objective is concave in r and the constraint qualification holds, the KKT conditions are necessary and sufficient. We define g(r) == min(r, 1 + $} and derive its subdifferential as follows,
{1), ffrl++E,
0g(r) = {0). fr>1+e
1),
otherwise.
We introduce the Lagrangian function with for the equality constraint and μa. 0 for r(a) 2 0:
C(r, x) = 1/4 > (S_ + S+ (a) min(r(a),
Σ (og((1))))(((a) - Σ ((a)r(a)-) -1 -
+ +Euar(a)
30

---

Published as a conference paper at ICLR 2026
For &a @ E}, we have
0 Or(a) (a) (a)))a) z(e) (log(r(a)) + 1) - Am((a) + Ma-
7
We also have (ar(a) 0. This implies Ma for any with re(a) > 0 and
m(a) (log(re(a)) 1) (S) S+ (a){aa Ar( E, we have La = 0. Putting these pieces together yields the desired result.
(Fr(())
Lemma C.S. For any group of samples have
we define U == V (y( "o.u,,..yyyy)) .(G)). Then, we
Told (U) = Σ (a) 2 (1V - T =: Mo-
aEU
Suppose that
(1+5)n
Moo 1- log(1 +8).
(28)
2
Then, we have ra(a)= S for all E.
Proof. First, we show DKI. (Tow Tota) S ( +)). By definition, we have F(Thew) 2 F(TTM)) (
Since and ((Toaa) E3 A; O, we have F(Fou) O. Thus, obtain from F(T
that DKL Feew ( I n (
(T Due to the upper clipping, given the number of samples with
reward +1 in the group S K G 1, we have
G
(Tieew S (1++))
Σ
A11A > 03 = (1++)
L-K/G K/G = (1 $) v (1 5
S
L+s
1=1
Putting these pieces together yields the desired result.
v(++e) Indeed, this can be derived from the Pinsker's inequality
Then, we show neew (U) 2 Mo as follows,
new (U) Told (U) < T /aall TV S / DKI,(TTi (Ticw ToMa)
2 3/v(1 11(1 +
This together with Told (U) 2 Mo yields the desired result.
Finally, we show that ra) S + E all a E V given Eq. (28). Suppose that there are some a* so that re(a*) 1 =. Then, by Theorem C.4, we have Ea- 0 and
(a))
log(rc(a"))
nS-(a") Tad
nd
mnd - 1,
By definition, (b) (b) 0 for all b E U. Thus, have log(rc(b)) -nd and rb( 2 re(a") > + for all E U. This implies
DKL (mew mTaa))
Σ
Neo (a) log(re(a)) > log(1 + c) (U))
Putting these pieces together yields direct contradiction to Eq. (28) as follows,
(1++))
2 n1( + + 2 (mI ( (Taa) > loog1 + e) MMo-
2
This completes the proof.
Theorem 4.3 (restated). Define C, := {A. > O, (.(((()) 1+ Let p P(C1) and
5 (y) (1 e) Ci). Then, for 1 0 satisfying Eq. (28) and any p (Tmin: 1), we have E(H(( - H(TTOla) S
+ E(R(n)] + c(p)G (pden Nmax (G 1)p)
where co, T and R(n) are defined as the same as Theorem 4. and
c(p) = - Tmin (log pe (2 ) Sell : Xmax (6 M(P))
Xmax h((p)
X Xmax == exp(n/ (27Tm//)22mmm)) - +s), M(p) = (exp(n/(2p)) +)]++
31

---

Published as a conference paper at ICLR 2026
Proof. For simplicity, define AH H((Teew)) ((ToT and AH = H((Tow)) H(Too)) Then, we have
B|AH (T - H (T(TTeee)) EE [(A")..
By first part of Theorem C.I and Theorem C.3, we have
[|A 1-2'-6 @ (foo)))² + E(((n))
(29)
2G
suffices to consider [E[H( )ew) H(Twoew)) Indeed, consider
H((Tew)
Σ Nnw (a) log (rog(Trnw ( = Σ * (a) log( (T (a)) DKL (xL((T (T T
Since DKL. (Tew (iow)) 2 O, we have
H((The)) H((TTiiw) S
Σ New (a) log( (he (ew) + Σ Nnew (a)
Σ (T (a) new (a)) log( (a)).
NEV
By Theorem C.5, have Tnew (a) S + e) (a) for every a. If rf) 1 e, have
new (a)
new (a)
= Tala (a)) - (iew(a) 2 mad(a)(ru(a)) - (1+)))
> Tmin (r)(a) (1+
Thus, we have
H (TTOO)) - H(( 1
Σ VEV
Tmin (ri((a) - (1 + log(Thew (a)).
(30)
For any p E (0, 1. on the set (Mota(y(t) S p), we have
log(T ()) log( (y (3) )ru (x s min(0, (pe (2rmin))
Using Eq. (30), restricting to indices i where C2 occurs, have
H( (((Fee)) (eeww)
S min(0, log(pe" (27mm)))
Σ
Tuii (rai(ra(a)) - (1+ +))
cra(a)>1
< min log(pe lo (2
Σ
Tmin (Fa(a)) (1 + +e).
(.)()))> (1)=1++,,a0(4)55
We let X2:= = (Ta(a) (1+ 1))1 and nu := " 11y a)lc, Then, we have
Σ
miin (ra(( -1 =)) = Tiin
(, "y(0)
a:ru arr_()))1+ermmll)5)
Define = My(t) Conditioned on (0)) = a and mata(a) S p. we have
K < ==1
1 = a} Binomial(G - 1, Told (a)).
Thus, we have
(TTaaaaa S spll
= B|E[K' (y (0) S 1) (Toaaa (0) S p]]
E[(G 1 TT (x(1) (Tou <Pl
= (G - 1) Σ (T))a(a)) ( S sp
S (G 1)p.
Since for any K 0, we always have
1 2 1 -
K
R+1
2
x,1(fa((y())5) 2 Totd
< p)
1.
Tode (y S pp((nyon 1).
15(0
32

---

Published as a conference paper at ICLR 2026
Thus, we have
E
X X1a(malyy Sppt
2 (T S p|]
Xmax (G 1)p, 2
(1)
where X; I X max == exp(1/ (2 - (1 + c). This Xmax is derived by recalling that [A(a)] -
1/ 1/(2TTmiu) and noticing that Jensen's 's incquality implies Z(n) == Enau [enA(a]] 2 grE 1. Taking expectation of both sides of Eq. (30) yields
(( H((Tiew)) S c(p)G
()) S pp]] - ميي (G (G-1)p))
1)p)
(31)
where we denote c(p) == min(0, log(pe"/ (2mmmn) ((Mtmmm)))))))) we have another bound X; S (M)) = } Tmin Finally, notice that on {Toid (y > p). we have
- By symmetry, for any i 1,... G,
5_ B[r(y(0) (1+
B[X, C:] = ([ p} + [E|+E[X,, (Fooa (1)) > | Ci]
EmaxPP(TTMl (0) s Ci) + M1p)( P( (y s C.)).
This implies
P(Toou ) s C:) 1 X
which further guarantees
(To, spp) Oi]
8 - E[X,1 f Tota (y > p) C.]
IV 2 8 M(P) (ataa (y
S (
C2)
Xuma Mp) Xmaxx M((p)
Xama (8 M(D)) Xma M(p)
Notice that E[X, (Taia (y())) S p} C:] is nonnegative. Then, we have
[[X;1{T (Tula (y S p} CJ
Xmna (onal6-MMi (6- == 8oft
X max M(p)
By using the symmetry again, have
[Tx, (1) S PPl P(CC) (o (0)) spt C.. 2 pō.n.
Therefore, Eq. (31) becomes
B[H((T H((TTie S c(p)G (pden Xmax (G - 1)p) Combining Eq. (29) and Eq. (32), we obtain the desired result
(32)
B(H((T.)) - [TT.) 1-2 26 ( )1² + B[R(n)] + c(p)G (p5cs Xaux (G- 1)p)
((Foo))) + [[() GTmin (log(pe (2Tmmi))) (( (pdell 2
Xmas (G - 1)p)
2G
(33)
This completes the proof.
Lemma C.6. Let Ve* == {a V: (a) S {} with some स E [Tmin: 1). Define Colle+ := {3 e
Vga+
No 2 2} where Na = Ei
D ((y(0) = a). Then P(Collg*) < V9)), Furthermore, if Tmin then on Coll we have [A(a)] & 24 for all a.
A S (Vv--//V)
G
Proof. Let usa == Tota( For any fixed a E Vs*. we know Na ~ Binomial(G, (()) and P(N. 2 2) < (a)² Union bound over E Vea yields
P(Collg=) < ( Σ (a))
(9)
Σ
T(a)
a:n(a)
and using A** I | gives the final bound P(Collg*) & VIE? On Colls if Tola(a) < n, then the token appears most once in the batch, hence A(a)
Ca(aj or A(a) 0. For
A_
the former one, by Theorem 2.4, we have [A(a)) < -C-I/VE
Gm(a) < va-I/va
By assumption
Grmin
क S 2(V0-1/Vo) Tmin. one concludes that A(a)) < 2
33

---

Published as a conference paper at ICLR 2026
Remark C.7. In this remark, we will show that under practical settings. the assumption, i.e., Eq. (28),
in Theorem 4.3 is indeed satisfied and the extra term ensure the entropy is decreasing in expectation.
Recall the parameters
used observed in experiments: G = 16, 17 5 x 10 |V| x 150000,
Tmin = 10-7, < = 0.2, x 0.001, 5 z 10, and we choose the threshold p it min = 2 x 10-7
By Theorem 4.3, we compute maa 10.98, M(p) 2.29, 8eft 9..74, and c(p) 1.29 10-6
Thus, the clipping correction term (the third term) in the upper bound of [[[A is 2.01 x 10-7 Furthermore, egti² 7.81 10-15 and ((xT) 2 min -223 10 Finally, define Collga
in Theorem C.6, applying Theorem C.2, we have
[[[[()]] S (1 q=) (ft) + 9= (+Tmii))):
where Φ+ 2 7.2 10 by Theorem C.6, C(#) = 3.43 10-s, and ((Tmin) 5.31 x 10 Hence, [E( S 3.43 x 10-5 Combining all numbers, we have
E|AH| < (-2.23 x 10 x (-7.81 x 10 15) + 3.43 x 10 8 2.01 x 10 1.49 x 10
- < O.
Thus, we can conclude B[AH] < O.
C.4 MISSING PROOFS IN $5
Proof of Theorem 5.2. Since f ~~ Binomial ni: () and g ~ Binomial( }). we have E[ʃ] = Rh
and Esg] ns. We rewrite Eq. (8) as A ne ne(s-g)
g and have
E E| = m_(n; ne)
m n=(G-ne)
20 + 2
G
In addition, we have Var(f) n and Var(g) = ". Since f and are independence, we have
Var(A) = (z) Var(J) + (7)² Var(g) = (%) 7 + + 4 This completes the proof.
2
m.
1G
Proof of Theorem 5.3. and
We define X = f and Y nhe 9. Then, we have Y Sinomial(ne.) (Ney 2
A o (n n.u. + o
-
n.Y c
(34)
By definition, Z = X + ~ Binomial (G, )) and
f>g g + / > ne.
f<g + Z < ne.
By definition, we have Pr(r, = 1 7 = 2)
for each j and each j E C. Thus, we have
E|y Z = 2] = n;z Taking the conditional expectation of both sides of Eq. (34) yields
E Z = 2] = nz
G:
G
E Z =]
n. G
+ B
Z z] E[y Z n.n.
By using the tower property, we have
E
f > gl = E(A 9
Dilk.
Note that [/))) = E[A
9) Pr(s g) and E[A1 (,<))
B[A 1 > P) f)-
Thus, it suffices to prove that Pr(f
9) Pr(ff <g). Indeed, write them in wedge form
Pr(f > g) = 1
Σ
T4
ne
Pr(g>
1
Σ
no
n4
(35)
2c
2c
k
E
k>
k>e
Fixing the integers t l 2 O, we define the function as follows,
(n)
for all n 2 k.
We claim that V (n) is strictly increasing n. Indeed, we have V (n) e (n-4) e (IT)=-- (n-E) (n- At (n-k) k)) At
=
Each term the product is strictly increasing in n. Thus, this yields the desired result. Since ne > 2 k we have
(nne) > ((n.)
This implics
(r²))
The n4 b
T4:
noe
for all l.
k
8
This together with Eq. (35) yields the desired result.
34

---

Published as a conference paper at ICLR 2026
Conditional variance analysis. Let f ~ Binomial(m;; 2) and g Binomial (n: 2) be indepen- dent, and let be defined in Eq. (8). We write Binomial {). Then, we have
ne and let Z = X Y
E|A Z
2]
Var A
Z 2)
G:(G-ni z(G-2) G-1
G
We let C G:(G-n and define h(z) (G 2). Then, have 1)
Var(A > 9) CE 2) Z > ne].
Var( 19> f) C)B(( Z < ne].
Th >> ni, we have Var A f > g) < Var s>
Proof. Conditional on Z 2, the positive labels are uniformly scattered among G positions. Then, the count X of positives falling inside the indices of I is
X 7 = Hypergeometric G, m,4)-
Thus, we have
B[X]
Z
Var(X
Z 2) 7 5)
G-n:
G-1
By definition, we have
n.X G
+
m Y)
X nz
+
n.a.
G
6
and
A Z 2| E[X
Z
2]
0 +
and
Var A = 2) = Var( X = 2)
GG-nn) GG-z) G-1
G²
GG) and define I<2)) = xG - z). Since E[A Z] is independent of Z, we have
We let
= G=(G-1)
Var(A A) Va A Z) A] CE(h(Z) A],
for an event A measurable w.r.t.
Since
Z n. and g f Z Re. have
Var A 9) (IE( Z > ne]. Var(A g>> J) CI((((2) Z < ne].
Since Z ~ Binomial(G,, hG 2) h(z) and h(2) is strictly increasing on 1,,,,,,3, ...: 16 111).
we have
E[4(Z) 7 > n.] E[M(Z) Z G ne].
Since ne 2: we have < - n. < nhe S G. Thus, we have
([) Z G no E(h ()) Z nel.
Multiplying both sides of the above inequality by C > 0 yields
Var I > g) < Var A g > f).
This completes the proof.
35


Reviews:
[{"review_id": "2zqvO4ogtT", "reviewer": "Reviewer_9GLK", "summary": "The authors investigate seemingly paradoxical phenomena with RLVR: spurious rewards, which should decrease exploitation, and entropy minimization, which should increase exploitation, seem to both yield improvements for LLM's trained on MATH 500. The authors present a series of theoretical and empirical analyses to explain this phenomena. \n\nFirst, following previous hypotheses that clipping induces a bias towards high probability tokens under the initial model, they derive a bound on the magnitude of the effect of clipping on the gradient and perform empirical experiments to show that improvement still occurs under spurious rewards without clipping.\n\nSecond, they analyze the effect of clipping on policy entropy and present empirical results that Qwen2.5-Math-7B undergoes entropy increase with spurious rewards without clipping and entropy decrease with clipping on MATH 500, while both improve in validation performance. They discuss how entropy minimization may not always be useful, depending on the performance of the initial model, and perform empirical experiments with models of different sizes.\n\nThird, they analyze when spurious rewards can be useful in terms of the success rate of the true correctness rate of the initial policy. The basic intuition is that stronger base models will tend to have correct responses reinforced by spurious rewards, and they show experiments with a fine-tuned Llama3 model to show that improvement under spurious rewards becomes more likely as the model gets stronger.", "questions": "Section 3 questions / comments:\n* what is the main takeaway from the upper bound on the clipping bias? The implication seems to be that the clip correction having a small magnitude makes it insignificant, but the later results show that it has meaningful effects (e.g. on entropy). The empirical results are much more clear to me (clipping not required for learning under spurious rewards)\n* r as the importance ratio is used in remark 3.1 without defining it (r up to this point had been reward?)\n* would be helpful to state why the law of clipping is important\n* would be helpful to state the key conclusion from the figure in the caption\n\nSection 4 questions / comments\n* it would be helpful to lead with the results in figure 2 to motivate the theoretical analysis (these are actually mentioned in the theory section motivation before they are presented, which is a bit confusing)\n* it would be helpful to discuss the implications of the derived expression for entropy change more explicitly (e.g. after theorem 4.2, explain how this indicates that training with clipping leads to entropy decrease). The notation is very dense and proof is in the appendix, so it is not obvious how to interpret this analysis as one reads through. Currently the paragraph after the proof refers to the empirical results, rather than the implications of the theorem\n* Remark 4.3 does a better job of discussing implications, but again it refers to the empirical results that haven't been introduced\n* section 4.3 felt a bit out of place given that the whole section up to this point had been about the relationship between clipping and entropy, not entropy and performance. Useful analysis, but it would be helpful to set it up a bit more for the reader (e.g. mention this question in the intro paragraph for this section)\n\nSection 5\n* Figure could be made more clear. If the key point is that initial model performance explains how likely a model is to improve with spurious rewards, a plot comparing initial model performance to improvement would be helpful", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors carefully investigate open questions related to RLVR and present interesting insights. The combination of empirical and theoretical analysis is compelling.", "weaknesses": "1. The presentation of the motivation for each analysis and key takeaways could be more clear. A few general suggestion would be to explicitly state the goal of each analysis at the beginning of each section, to lead with the empirical results that the theory aims to explain, and to use more descriptive figure captions that discuss the key conclusion from each figure. At a few points reading through the paper, it was not clear to me why an analysis was being conducted and what conclusion the authors were justifying with the theoretical result.\n\n2. Notation is not always clear (e.g. r is used to represent importance ratio before defining in section 3)\n\n3. Overall, could do a better job of guiding the reader through the analysis as a story. Current flow seems to be 1) clipping does not explain learning from spurious rewards 2) clipping leads to entropy reduction and acts as a regularizer 3) entropy decrease from clipping does not always lead to improvements 4) models that have high success rates before RL are more likely to improve with spurious rewards. Motivating each of these points and connecting them back to the initial questions would be helpful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zVEOP7dAvv", "reviewer": "Reviewer_PiCo", "summary": "This paper investigates how policy entropy influences reinforcement learning with verifiable rewards and whether spurious rewards can yield gains through clipping bias and model contamination. Through rigorous mathematical analysis supported by empirical experiments, the authors establish a direct connection between clipping and entropy, showing that clipping primarily reduces policy entropy. They argue that clipping does not provide a genuine learning signal and discuss when reduced entropy is beneficial. The authors introduce a reward-misalignment model to explain under which conditions spurious rewards enhance performance.", "questions": "- Why did you not stick to the same model for the experiments or use them all?\n- How do empirical results change with different clipping thresholds?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important and timely question in reinforcement learning for large language models.\n- Rigorous theoretical analysis linking clipping and entropy, extending prior accounts.\n- The reward-misalignment model offers a probabilistic explanation for the benefits of random rewards.\n- The paper is well motivated, interesting, and clearly presented.", "weaknesses": "- The evaluations are concentrated on MATH500, and ablations on hyperparameters (e.g., clipping ratio, group size) are missing.\n- Some findings, while formalized, may only confirm intuitively expected behaviors (e.g., entropy minimization failing when incorrect trajectories are the peak of the distribution).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zkP9q0hiXE", "reviewer": "Reviewer_1ufZ", "summary": "This paper investigates the exploration–exploitation dynamics in reinforcement learning with verifiable rewards (RLVR), a method used for training reasoning-focused large language models. The authors analyze the roles of clipping, policy entropy, and spurious (random) rewards, arguing that prior explanations of performance gains under random rewards are incomplete. They propose theoretical analyses and empirical studies suggesting that clipping primarily reduces policy entropy, and that observed gains arise from reward–entropy interactions and model–data regime effects rather than contamination or genuine learning.", "questions": "1.\tYou mention that experiments were run across several model families and sizes, but not all settings overlap. Could you clarify how these choices were made and whether comparable setups were used across models?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "•\tInteresting theoretical framing: The paper offers a formal treatment linking clipping bias and policy entropy, contributing to a better conceptual understanding of RLVR optimization dynamics.\n•\tRelevance to ongoing discourse: Given recent debates about “spurious reward” effects and entropy minimization in reasoning LLMs, this paper addresses a timely and relevant topic for the ICLR community.\n•\tClarity of theoretical results: The analytical sections (Theorems 3.3–4.2) are clearly presented and mathematically rigorous, providing useful insight into when and why entropy changes during GRPO updates.", "weaknesses": "•\tExperimental design inconsistency: Although the paper claims “extensive experiments across multiple model families and sizes,” not all experiments are conducted uniformly; rather, different subsets of experiments use different models and data. This fragmented setup makes it difficult to assess the generality of the conclusions.\n•\tLimited empirical depth: The experiments do not convincingly support the claim of “reconciling conflicting reports” in the literature. The evaluation scope remains narrow, and results on random rewards or entropy minimization are mostly qualitative or limited to a few benchmarks (e.g., MATH500, AIME).\n•\tUnclear novelty of findings: The conclusion that entropy minimization acts as a regularizer, or that clipping primarily affects entropy, is intuitive and has been suggested in prior work. The contribution beyond existing studies remains unclear.\n•\tIncomplete closure of central questions: While the paper lists three core research questions, the final sections do not directly or conclusively answer them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fGFD4vqICi", "reviewer": "Reviewer_tUmE", "summary": "This paper investigates the exploration-exploitation trade-off in RLVR for LLMs. It examine the paradoxical finding that random rewards can improve LLM reasoning performance. The core contributions are a theoretical and empirical reframing of the underlying mechanisms. This paper argues that clipping bias, previously thought to be the driver of these gains, provides a negligible learning signal. Instead, it demonstrates that clipping acts as a regularizer that deterministically reduces policy entropy. Through experiments on multiple model series, it shows that these gains are not exclusive to potentially contaminated models. Finally, this paper proposes a reward-misalignment model to explain why stronger models are more likely to benefit from this training regime.", "questions": "see Weaknesses and:\n\n- The explanation for the entropy *increase* in unclipped training (Fig 2 Left) relies on the policy being \"skewed\" (Remark 4.3 , Fig 5). Can this paper provide any empirical measure of the skewness of the initial Qwen2.5-Math-7B policy to confirm it is in the regime described by Remark B.7?\n\n- The paper implies (via Thm 4.2 and Fig 3 Right) that clipping should be detrimental on harder datasets like AIME because it forces entropy minimization. Did this paper tests the performance of *unclipped* training on the AIME dataset? This seems like a critical experiment to connect the paper's theoretical sections.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper provides a  theoretical argument that refutes the prevailing view that upper-clipping bias is the primary driver of performance gains under random rewards.\n- This paper establishes a relationship between the application of clipping and the reduction of policy entropy, effectively reframing clipping as an entropy-minimizing regularizer rather than a source of learning signal.\n- This paper introduces a  probabilistic model  to explain which models are likely to benefit from random reward training. This framework provides an explanation for the empirical observation that stronger models tend to improve more.", "weaknesses": "- Several key figures presenting experimental results are difficult to interpret due to missing legends and insufficient detail in captions. Figures 1, 2, 3 (Middle), and 4 all contain plots with multiple colored lines, but none include a legend to explain what each line represents. The review is left to assume they correspond to independent training runs, but this is not stated. The captions often lack sufficient detail. For example, the caption for Figure 1 does not explain that the faint lines in the Left and Middle panels are likely individual runs.\n- The paper's mathematical exposition is dense, and the notation is not always clear or consistent.  The definitions for the clipping analysis in Theorem 3.3  are imprecise. $N_t^{clip} := \\bar{r}_t \\hat{A}_t$ is defined, but $\\bar{r}_t$ itself is not. The proof of Proposition 3.2  appears to rely on unstated assumptions. The expansion $log~Z(h) = \\frac{1}{2}\\eta^2 + \\mathcal{O}(\\eta^3)$ requires $\\mathbb{E}[\\hat{A}] = 0$ and $\\mathbb{E}[\\hat{A}^2] = 1$ .\n- The paper's refutation of contamination does not address a key alternative. The low clipping activation rate ($\\approx 0.001$) for Qwen2.5-Math-7B (Remark 3.5) is used to show the clipping *signal* is negligible. However, this low rate could *itself* be an *effect* of contamination (i.e., the model is already so confident on the benchmark that $r_t$ rarely exceeds $1+\\epsilon$). This possibility is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)