Entropy-preserving reinforcement learning
Authors: Aleksei Petrenko
Ben Lipkin
Kevin Chen
Erik Wijmans
Marco Francis Cusumano-Towner
Raja Giryes
Philipp Kraehenbuehl
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model
reinforcement learning
entropy
GRPO
PPO
Primary area: reinforcement learning
TL;DR: A study of the entropy dynamics of policy gradient algorithms for LLMs and new algorithms for controlling entropy and preventing entropy collapse.

Abstract: Policy gradient algorithms have driven many recent advancements in language model reasoning.
An appealing property is their ability to learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solutions. As we show in this paper, many policy gradient algorithms naturally reduce the entropy---and thus the diversity of explored trajectories---as part of training, yielding a policy increasingly limited in its ability to explore.
In this paper, we argue that entropy should be actively monitored and controlled throughout training.
We formally analyze the contributions of leading policy gradient objectives on entropy dynamics, identify empirical factors (such as numerical precision) that significantly impact entropy behavior, and propose explicit mechanisms for entropy control.
These include REPO, a family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping approach. Models trained with our entropy-preserving methods maintain diversity throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in new environments.

Published as a conference paper at ICLR 2026
ENTROPY-PRESERVING REINFOR LEARNING
Aleksei Petrenko" * Apple
Ben Lipkin*! MIT
Kevin Chen Apple
Erik Wijmans Apple
Marco Cusumano-Towner Apple
Raja Giryes
Philipp Krähenbühl Apple
Apple
ABSTRACT
Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solu- tions. As show this paper. many policy gradient algorithms naturally reduce the entropy and thus the diversity of explored trajectories as part of training, yielding a policy increasingly limited in its ability to explore. In this paper, we ar- gue that entropy should be actively monitored and controlled throughout training. We formally analyze the contributions of leading policy gradient objectives on en- tropy dynamics, identify empirical factors (such numerical precision) that sig- nificantly impact entropy behavior, and propose explicit mechanisms for entropy control. These include REPO, family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping ap- proach. Models trained with our entropy-presenve throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in environments.
methods maintain diversity
INTRODUCTION
Online policy gradient reinforcement learning (RL) has become the standard for boosting the rea- soning abilities of language models (Jaech et al.] [024] Comanici et al.) [025 Guo et al. (0025). This approach involves sampling trajectories from the current policy within a given environment and reward functionn then using these to estimate gradient that maximizes expected reward. Ef- fective RL optimization requires balancing exploration and exploitation (Thrun] 1992 Sutton et al.) 1998), where a robust learner should generate diverse trajectories to cover the spectrum of potential solutions. Maximum entropy reinforcement learning offers a framework for achieving this balance (Ziebart et al.
2008 Haarnoja et al. 2017 2018 Eysenbach & Levine 2022). While trivially the optimal solution to a finite Markov decision process (MDP) is a deterministic stationary policy, optimization over the intermediate landscape requires a balance of exploration and exploitation.
common issue observed in online algorithms like GRPO (Shao et al. 2024) is entropy collapse. This phenomenon occurs when training excessively narrows the distribution around already high- probability solutions from the base model, neglecting other correct but less probable options. This often leads premature convergence to local optimum, enhancing pass@ relative to base model at the expense of eass@k k (Shao et al. 2024 Dang et al., 2025) Yue et al. 2025). This challenge has spurred innovations performance (Chen et al.] 2025b). Concurrently, research has highlighted GRPO's 's training insta- bility and the complex interplay between off off-policy drift, importance weight clipping, and entropy, inspiring modifications such as DAPO (Yu et al. 2025) and GSPO (Zheng et al. 2025).
policy gradient algorithm design, e.g. directly optimizing for pass@k
In this work, we argue that entropy should be actively monitored and controlled throughout RL training. We analyze entropy preservation recent algorithms and propose explicit mechanisms for entropy control. An important observation from our work is that, while correlation exists between final entropy and performance, more
unifying lens for understanding the successes of
"co-first authorship.
work performed during an internship Apple.

---

Published as a conference paper at ICLR 2026
Qwen3 32B (AppWorld Test-Normal)
Qwen (AppWorld Test-Normal)
Qwen3
0.6
West Sacund
Test accuracy
Test sccuracy
0.0
0.0
0.0
0.0
Partoken entropy
0.0
0.0
Paritkkn entropy
3.3
0.0
Partoken entropy
0.4
0.7
Test sccurary
Test sccuracy
Test securaty
Cumulative entropy
Cumulative entropy
50
Cumulative entropy
Figure 1: Top: Evolution of the average per-token entropy and test accuracy during training for sev- cral basclines (GRPO, LOOP, DAPO, GSPO) and their entropy regularized versions (REPO). Each curve shows the average trajectory over several training runs with different seeds. Bottom: Cumu- lative entropy experienced during training up given checkpoint is positively correlated with the test accuracy. Each point is checkpoint of single training run (best-perfonne run highlighted). Algorithms that collapse the entropy early (see e.g. Qwen 3- 8B on AppWorld; middle column) perform significantly worse than algorithms that maintain steady entropy during training.
checkpoint per
informative measure is the entropy trajectory throughout the optimization process. As the saying goes, "it's not the destination, it's the journey." [igure[[trcke this effect. A trajectory characterized by lower entropy throughout training yields lower performance. Conversely, entropy trajectories are similar for most of the optimization but differ only in the final steps, performance is largely unaffected.
Our contributions span theory and algorithmic development. We analyze how policy gradient ob- jectives modulate entropy dynamics, proving that PPO's clipping bounds entropy change and that DAPO's and GSPO's clipping implicitly preserve entropy. We identify critical implementation fac- tors affecting entropy dynamics, including numerical precision (BF16 vs FP16) and framework be- haviors (FSDP2 output casting), explaining previously observed training instabilities. We propose explicit entropy control mechanisms-Re REPO, which modifies the advantage function, and ADAPO, an adaptive asymmetric clipping approach-bort using adaptive controllers maintain target en- tropy levels. Our numerical fixes alone yield state-of-the-ard on AppWorld (79% Test Normal, 71 % Test Challenge), while formance, closing the gap to on-policy training and retaining trainability for sequential learning.
REPO and ADAPO achieve the strongest off- oof-po per-
PRELIMINARIES
Language modeling. Let x E X' denote the tokens a vocabulary and æ E X the strings expressible via concatenation of those tokens. A language model (LM) parameter- ized by 8 defines a probability distribution over strings that factors autoregressively such that <x())= ( x)) I <e(2; Ie2), where o denotes an end of sequence (EOS) marker. For notational convenience we will use Ta to express probabilities on both tokens and strings.
Language modeling as Markov decision process. Let the policy ne sample actions E A X U {a) (any token or EOS) given a state s E X (a string context). Let state transitions append generated actions to the state Let denote trajectory, sequence of states and actions generated by the policy and environment. Let T ~ Ty denote the trajectory distribution. We consider tasks
State transitions deterministically append the generated action to the context, terminating generation at EOS or upon some other environment condition. In some domains, e.g., those involving tool calls, state transi- tions may also append additional tokens to the state that were generated by some unobservable process such as executing code interpreter.
2

---

Published as a conference paper at ICLR 2026
with terminal rewards R(c, r). Given some task context sampled from some dataset D, the MDP objective is to maximize I MDP def (. (R(e, T)].
Policy gradient reinforcement learning directly computes a gradient through the REINFORCE algorithm (Williams 1992). which is amenable Monte Carlo estimation:
VeJ MDP Be-DD)r=u((e) [A(c, 7) Va log (T c)],
where A(c,7) 7) R(c, 7) b is an advantage function shifting the return R(c, by baseline b.
REINFORCE leave-one-out (RLO0) (Kool et al. 2019: Ahmadian et al. 2024 Chen et al., 2025a) is one of the most popular estimates of advantage for language modeling. generates K independent samples on-policy Ti ..... Tк ~ np(- c) for each task c. The reward for each trajectory may then be baselined against the remaining K - independent samples, yielding an unbiased, low variance advantage estimator:
| 2024 Kazemnejad et al.)
K Σ K K-1
K Σ 2 R(C.T))
K = K-1
ARLOOlC, T.) de R(C,TT)-
R(C,T,)4(A1))
R(c,T.) - K ==1
Policy gradient algorithms are on-policy by nature: They rely on new set of trajectories in each context T (- c) after each gradient update the policy иT.
Proximal policy optimization (PPO) allows the updated policy to deviate slightly from a sampling policy (Schulman et al. 2017). It uses an importance weight to correct the magnitudes of parameter updates such that the expected policy gradient remains unbiased. These importance weights are typically clipped to avoid divergence from local trust region (Schulman al., 2015).
To new (a+ C, a.et)
T JPPO def (omoooo
Σ min (A(c,T) we, A(c, T) well)? a.ET
|
ald C, a<i)
where well will examine PPO with and without clipping. The version studied will be clear from the context.
1tee clips the importance ratio from below 1 1-e and above 1+e. In our theoretical analysis,
LOOP (Chen et al. 2025a) and GRPO (Shao et al. 2024) combine the above PPO objective with RLOO leave-one-out advantage estimates. GRPO rescales advantages by the standard deviation of the sample returns, introducing a small bias (Liu et al. 2025b), while LOOP uses ARLOO directly.
R(e, T;) mean (amaannR(e,)))) (R(e, TI),.. R(C,TK)
AGRPO(C,T) T;)
std(R(c,T:)... R(c, R((,,TT)()..,.(TTx)))
Group Sequence Policy Optimization (GSPO) Zheng et al. 2025 uses a trajectory-Ievel trust region defined by the geometric average of a sequence 's probability ratios
1
JGSPO EewD (ii((A(c,T)) GSPO A(c,T) w GSPO GSPO [+t]]
w GSPO
(T
new
(T
c)
Th
old
T
c)
GSPO yields an equivalent gradient estimator to GRPO, LOOP, and RLOO on-policy, but clips tokens and trajectories differently the updated policy ли new drifts from the sampling policy old
Policy entropy. The inherent uncertainty that a policy places over its generations may be ex- pressed from an information theoretic standpoint as entropy expected surprise: Hr (D)
-Ee-D [Ele, [log Ta (T] c)]]. In addition global entropy, may consider the entropy over actions at any given state (c, acr) as H. (. 8) (Es-r(()ss [log s)).
In this paper, we analyze how state-wise entropy evolves as variants of policy gradient optimize their objectives. We identify which algorithm variants are naturally entropy preserving, and which lead to rapid collapse ($3) We demonstrate that subtle implementation details can distort entropy dynamics, causing unexpected collapse in algorithms that should theoretically preserve entropy ($4). Finally, we propose simple modifications of RL methods that lead to effective entropy regularization and improve downstream task performance 5
3

---

Published as a conference paper at ICLR 2026
3 THEORY: ENTROPY DYNAMICS OF POLICY GRADIENT
The entropy dynamics of policy gradient RL boils down to the relationship between two values: (1) action pr-obbbbbiiiiis, and (2) the advantages yielded by those actions. Intuitively, assigning a positive advantage to some action increases its probability. For high probability actions, this effect sharpens the distribution, and entropy decreases. For low probability actions, this flattens the distribution, increasing entropy. The opposite pattern holds for negative advantages. This effect natural: after all, sharpening an uncertain policy around correct actions directly maximizes the expected return. However, we will see, not all RI. algorithms sharpen the distribution equally.
Formally, consider the policy gradient update with on-policy actions in state s. Under a first-order Taylor approximation to the training dynamics, the expected change in entropy is as follows.
Theorem 1. Given a policy gradient update Ō = 0 + MDD the expected change in entropy is approximately:
AHoo (. (18) ~ aa. Ba mot (8) [A(s,a) I(s,d) a(s,a)" u(s,d').
T 8) Ba [log Te(a s)] denotes mean-centered log- log-rrobabilities and u(s,0) a) del Va log (a 8) is the score function for a policy na evaluated at state 8 and action o.
L(s, a)
[Proof in App. A.2 The entropy change is driven by a multiplicative relationship between action hog-proaabilities and the advantages yielded by those actions. In exact derivation, these weighted by the score vector outer product. With additional independence assumptions or a tabular softmax policy parameterization, this expression can be further simplified, resulting in a weighting by the action probabilities. This yields the following corollary:
Corollary 1. Assuming u(s, a) "((s, a') = for all a + d', the entropy change is proportional to:
AH 8) - -aawno(-le) [A(s,a) L(s.a) To(a s)]
[Proof in App. A.3 This latter form encodes the dominant behavior of entropy dynamics in a manner that is inherent to policy gradient. Using this form, explain the observed behaviors of various RL algorithms. A similar derivation can be shown for tabular softmax policies (Cui et al.) 2025 see Corollary 2 in App. A.44. Thm. [ and Corollary T tell us that the change in entropy is governed by a correlatio between advantages and log- probabilities, weighted by action probability.
Entropy dynamics of PPO. The biggest feature of PPO is its ability train on slightly off- policy trajectories, given that the updated policy does not deviate from a trust region around the current policy. This allows PPO take multiple policy-improvement steps for single set of trajectories. The effect of these repeated updates is much larger policy updates between con- secutive PPO steps, which empirically amplifies entropy collapse. This being said, the clipping on PPO, when appropriately orchestrated, can protect against entropy collapse as well. Clip- ping ensures that no policy gradient update is performed if the policy drifts outside a trust region (1 (lw)) (a 8) S Th (a 8) S (1 1 Etigh) TT (a 8). This bounds the change in entropy:
Theorem 2. Proximal Policy Optimization (PPO) bounds the entropy Hagerr (- 8) of the updated
policy by the original policy entropy Hzgar(: s) such that:
- (1ww))
Hapal (- 8) < Ha (. S (1 Ehigh) Hrp (-|
[Proof in App. A.5 The clipping thresholds directly limit the maximum induced change in entropy per token. Intuitively, the change in entropy per token is stochastic: some actions have a large correlation between advantage and log probability: others do not, or even have an anti anti-correlation. For symmetric clipping regime, this results in an entropy change that largely follows the statistical trends outlined above, but at lower magnitude.
Entropy dynamics of DAPO. Now consider DAPO Yu et al. 2025). with an asymmetric clipping
regime <low < Ghigh- This allows for larger entropy increases, while limiting the entropy decrease. Due to the stochastic nature of the entropy changes, this directly contributes to overall increase in per-token entropy over sufficient samples. Threshold values fno = 0.2 and Ehigh = 0.28 proposed in Yu et al. 2025) stabilize the entropy throughout training, show experimentally.
4

---

Published as a conference paper at ICLR 2026
Entropy dynamics of GSPO. GSPO defines trust region 1
GSPO 'lw
w S 1 +
GSPO or high
equivalently (1 (1- (ow ( IT
ao (Tlc) xa (r|c) I (1
GSPO high
T This induces an equivalent bound to
Thm. however, the bound now depends the trajectory length |T|. Longer trajectories may induce a larger change in entropy, shorter trajectories induce a smaller change in entropy. With parameter values suggested in Zheng et al. (2025), Flow
3 10 and CSPO 4 x 10-4, the Shigh
entropy bound is tighter for trajectories
|T| < dn(1
In(1+))
600 tokens compared to DAPO, Like
DAPO, the clipping range is asymmetric
.GGPO
GSPO
leading stochastic increase in entropy.
*low
Shigh
Summary. The theoretical analysis above reveals that entropy dynamics in policy gradient algo- rithms are governed by the correlation between advantages and nog-probabilities. PPO's multiple off- off-policy updates amplify entropy collapse, while clipping mechanisms bound the entropy change per update. Asymmetric clipping (DAPO) and sequence-level clipping (GSPO) provide implicit entropy preservation by allowing larger entropy increases than decreases. However, these implicit mechanisms may not be sufficient in all settings.
Importantly, even strictly on-policy algorithms like RLOO are subject the entropy dynamics de- scribed in Corollary [ if the base policy is already well-calibrated to the reward function, the correlation between advantages and log-probabilities will be positive, and entropy will decrease. RLOO avoids the amplification of this effect that arises from off-policy drift and repeated updates on recycled advantages, but does not eliminate the underlying dynamic. This explains why RLOO retains more entropy than PPO PPO-based algorithms in most settings, yet can still exhibit meaningful entropy loss when the base model strongly pre-calibrated to the task. Explicit entropy control mechanisms, which we present in $ can therefore be valuable even in on-policy settings.
4 EMPIRICAL FINDINGS: IMPLEMENTATION DETAILS AFFECTING ENTROPY
We identify empirical factors that significantly impact entropy dynamics, discussed in this section.
4.1 16-BIT QUANTIZATION OF MODEL OUTPUTS AFFECTS CLIPPING
In PPO, GRPO, DAPO and similar methods, the clipped objective requires computing the probabil- ity ratio: <o(als)
Tee (a|s) By default, common LLM training stacks (e.g. HF Accelerate & FSDP2; App. B. cast model outputs to the training dtype before downstream computations. As result, the probability ratio is ultimately computed as: Tobwerved = exp(bf16(1))) no (als))) wo (als))) where bfl6(-) denotes bfloat 16 casting with -ourn-to-ncarrssteeen.
Theorem 3. Under bf16 quantization, the observed ratio exhibits a multiplicatiiv upward bias:
E|robserved ['mee] > Time- [Proof in App. A.8
Note that the bias is proportional to Tirue: larger ratios experience proportionally larger absolute bias. This creates effective asymmetric clipping that systematically favors entropy decrease:
Ratio of cilpped tokens
1.4
Lowercclp ratio
Uppen-clpp ratio
Avg. prob. difference
0.006
10.0
Max prob. ratio
7.5
1.2
DAPO b/166
Mr
fp16
1.0
0.8
2.5
Iterations 2400
160
160
240
80
160
240
Iterations
Iterations
(a) 16-bit log quantization effects.
(b) Inference/train discrepancy depending dtype.
Figure 2: (a) Fraction of tokens hitting clip bounds with 16-bit rounding: Chigh is reached more often and Gow less often, hindering the promotion of low-probability actions. (b) Average probability difference and max importance weight ratio between vLLM inference and training forward pass.
5

---

Published as a conference paper at ICLR 2026
For advantageous actions (A > 0, r 1), the upward bias causes Tobserved reach the upper clip earlier, limiting probability increases.
Upper clip:
Lower clip: For disadvantageous actions (A < 0, r 1), the upward bias pulls Tobverved toward 1, making the lower clip less restrictive. Fig. 2ajemirricll shows how Chigh is reached more frequently and <low less frequently with quantization compared to the full-precision calculation.
Chigh the opposite direction from DAPO's entropy-preserving asymmetry! While this bias exists in any finite-precision implementa- tion, very limited precision of BF16 promotes this to a strong entropy-decreasee trivial solution for this problem to simply use full precision for model outputs throughout the computation graph, including any importance ratio calculations.
This behavior equivalent to asymmetric clipping with Glow >
effect (App. B.T
4.2 6 VS BFLOAT TRAINING
bf16
1p16
Clipping fix. bf16
Clipping fp16
Test Challenge Success
0.4
0.32
It is customary use the BF16 floating-poin
0.3
0.24
type in LLM training because of its larger dy- namic range. However, Qi port improved results with fooalóó (FPI6) as
Entropy
(2025) re-
0.2
0.16
its additional mantissa bits enable more ac- curate gradient representation. Using FP16 format significantly reduces the discrepan- cies between the LLM inference (vLLM) and training subsystem, inherent to the modern post-training stacks (Fig. 2b).
0.1
0.08
160
240
160
240
Iterations
Iterations
Figure 3: FP16 training on Owen-3-83 AppWorld and clipping fix lead to a qualitative change: DAPO entropy collapse transitions to entropy increase.
In practice. with appropriate loss and gradient scaling, FP16 training tends mitigate entropy collapse and yield more stable and predictable training. To highlight the importance of these empirical findings: FP16 training in conjunction with the log my rounding fix ($4.1) results in qualitatively different entropy dynamics, allowing DAPO's cntropy-increaste asymmetric clipping to overcome collapse (Fig.
EXPLICIT ENTROPY CONTROL METHODS
The theory in $ and empirical analysis in $ that entropy dynamics are influenced by many factors and that minor implementation details can qualitatively change the algorithm behavior. While implicit mechanisms (asymmetric or sequence-leve clipping) provide some level of control, an explicit entropy regulation technique may be required for stable RL post-training.
A standard approach to entropy preservation is adding an explicit entropy bonus to the objective: I entrroyy JPP + 3 H.g. However, this approach has significant drawbacks:
Fixed coefficient: A fixed /3 does not account for the evolving entropy dynamics over training. Memory cost: Computing the entropy bonus exactly requires materializing all logits, which is memory-intensive for large vocabularies, especially compared to memory meffry-efffcient methods such as Cut Cross-Entropy (CCE, Wijmans al., 2025).
Below, we address both issues by proposing an adaptive entropy controller and paired-sampling estimator that jointly estimates the policy and entropy gradients without materializing full logits.
5.1 REPO: REGULATED ENTROPY POLICY OPTIMIZATION
We propose REPO (Regulated Entropy Policy Optimization), which modifies the advantage function
to include scaled policy log-likelihood term: AREPO(8, a) = A(s, a) 333 L(s, a) for each (c, a. This updated advantage is no longer constant throughout the trajectory like in RLOO and variants, but differs for individual tokens a, E T.
Following Thm. by Prop. 3 the induced change in entropy with AREPO is:
A A.REPO (-
AH., (-1 - [a [[(s a) (u(s,))]])
20

---

Published as a conference paper at ICLR 2026
This provides direct mechanism to control entropy where Bg > 0 increases entropy relative to the default dynamic and Ba < 0 downregulates entropy.
REPO-D (Decorrelate). One natural choice is to counteract entropy collapse on a per-token level
by setting GREPO-D x - -A (- 8) as approximated in Corollary| This neutralizes A AHry, allow- ing A A to approach kg
REPO-R (Rescale). REPO-R is an efficient practical approximation that captures the core in- tuition: increasing policy entropy requires upweighting rare correct solutions, while reducing (on average) the penalty assigned to rare incorrect solutions. This accomplished by rescaling advan- tages based on action probabilities via &ee S = Ç [[(s,a)) The derivation from the general REPO advantage and implementation details are provided in App. D.2
Adaptive controller. The optimal scale of the regularizer depends on learning rate, gradient struc- ture, second-order effects, etc. We use a simple adaptive bidirectional controller:
1. Estimate Hiig the policy entropy over experience collected in the first iteration; set ( +- 10-3
Each iteration: if H.g < Hini update Ç + 2; Hf > Hini update Ç + c 2.
3. Flip the sign of Ç to exert pressure the opposite direction: [<] < Smin set Ç +- -S.
4. Clip magnitude of [C] to [Smin Smax]. our experiments we used [10-3, 10] for REPO-D and [10- 0.05) for REPO-R.
Efficient estimation. Both REPO-D and REPO-R can be effectively estimated using only the log- probability of the sampled token or, which is already available during the forward pass when using CCE (Wijmans al. 2025). This stands in contrast to explicit entropy bonus, which requires materializing the full logit vector over the vocabulary. We show in App.[A. AA.7 REPO-D formally equivalent such entropy bonus, but estimated via REINFORCE using paired samples, incurring zero additional memory cost and acting as a control variate that reduces gradient variance when advantages and bog-probabilities are positively correlated, which is typical.
5.2 ADAPO: ADAPTIVE ASYMMETRIC CLIPPING
An alternative approach is dynamically adjust the asymmetric clipping thresholds, taking advan- tage of DAPO's entropy-preserving properties. ADAPO (Adaptive DAPO) keeps élow = 0.2 fixed and initializes <high 0.28 (as in DAPO), then varies <high in the range [0.2, 0.32] based on the ob- served entropy: (1) If H.m Himit. Ehigh f- min( (1.05 Ehigh, 0.32) (allow more entropy increase) Ehigh; 0.2) (limit entropy increase). This provides bidirectional control over entropy through the clipping mechanism. Note that REPO can be used with many popular RL methods (RLOO, GRPO, DAPO, etc.) while ADAPO utilizes asymmetric clipping and is relevant to methods like DAPO or GSPO. In our experiments, on top of GRPO and ADAPO on top of DAPO (6)). RL methods using adaptive asymmetric clipping were independently proposed in contemporaneous work (Xi et al. (2025).
(2) If Hr > Hyiit, decrease Ehigh <- max (0.95 max(0.95
evaluate REPO-
EXPERIMENTS
We evaluate whether entropy- peoopp--rreeiinnggg training yields improvements to strong models on challeng- ing environments when compared to state-of-the-art learning algorithms. We choose Qwen-3-te 3-8B and Qwen-3-325 as our starting policies (Yang al. 2025).
Environments. Interactive tool-use agent. Training scenarios are drawn from the train split (90 problems) of the AppWorld benchmark (Trivedi et al., 2024). The AppWorld Test Normal (TN, 168 tasks) and Test Challenge (TC, 417 tasks) splits are used for evaluation. Terminal reward is calculated via task- iis--provdded unit-tests that check the final state of the environment against ground truth (additional details in C.I. Competition-
mathematics. Training scenarios are drawn
from a non-overlese -oerollprlppingggg quality-filtern filtered subset of the AMC/AIME section of NuminaMath- 1.5 (563 problems; Li et al. 2024). AIME 2024 (30 problems) and AIME 2025 (30 problems) are used as
7

---

Published as a conference paper at ICLR 2026
evaluation datasets. Terminal reward indicates whether the generated answer matches the reference. We note that recent models are significantly overfit to math benchmarks so we strictly limit token budget to 4096 in AIME to create challenging learning problem.
Algorithms. For each algorithm, we highlight its distinguishing features with otherwise minimal de- viations from the base policy gradient to aid reproducibility (thus, some details and hyperparameter choices may differ slightly from original sources).
RLOO: REINFORCE with the ARLOO advantage estimator. Training is strictly on-policy (1 epoch) with a large minibatch that comprises all collected experience. GRPO: Off-policy extension of RLOO that uses normalized Leave None Out (LNO) estimator AçRPO and symmetric PPO clipping with 0.2. With GRPO and all algorithms below we train on the collected experience for 2 epochs with the minibatch size of 128 World))l or 256 (AIME) trajectories. LOOP: variant of GRPO with non-normalized estimator ARLOO (RL SOTA on AppWorld at the time of writing). DAPO: variant of LOOP with asymmetric clipping (elow 0.2, Ehigh 0.28). GSPO: An adjustment of DAPO using w GSPO importance weighting and trajectory- based clipping with llww 3 10 and GSPO 4 x 10-4 REPO-R: modification of GRPO with the additional entropy control mechanism.
ADAPO: Adaptive DAPO that adjusts Chih based observed entropy dynamics 5.2)
6.1
VARIABLE ENTROPY DYNAMICS ACROSS ALGORITHMS
Test: Challenge IllengeTee TGC
0.32
Test Normal TGC
0.45
0.24
Entropy H
0.15
0.30
0.16
0.10
ADAPO
GSPD
0.00
0.05
60
120
180
60
120
100
120
180
Iterations
Figure 4: catropy-preeeeving methods compared to baselines with Qwen-3- on AppWorld.
Test Challenge TGC
0.6
Test NormaTGG TGC
244
Entropy 11
0.4
0.4
GRPd
1.6
REPO-N
0.2
DAPD
0.2-
0.8
AD4PO
100
150
0.0
150
0.0
150
Figure 5: Entropy-preserving RI. training with Swwe-3-330 on AppWorld.
0.45
0.45-
AIME 2024 pass:
0.3
Entropy
0.30
2204
0.2
ERPG
0.15
REPO
0.1
DAPO
ADAP0
RL00
60
120
160
60
120
100
0
60
120
180
Iterations
Figure 6:
RL training with Owen- on AIME.
We observe consistent patterns across AppWorld (Figs. and 5 and AIME experiments (Fig.
PPO-like algorithms deplete entropy faster than strictly on-policy. GRPO reduces entropy by nearly 90% over training. while RLOO loses considerably less. LOOP behaves very similarly to GRPO and thus omitted for readability. See comprehensive results summary in App. C.3
8

---

Published as a conference paper at ICLR 2026
AppWorid TN NGGC
AepWortd TC TOC
0.10-
0.3
0.12
Entropy
0.2
0.06
0.1
30
60
90
60
90
30
60
AIME 2024 Pass88
0.6
AIME 2025 Pass@1
0.3-
Entropy
0.2
0.2-
GRPO DAPO
0.2
seerations
Figure 7: Sequential learning experiment. Top row: We use an trained model for GRPO, DAPO, REPO-R, and continue training the model AppWorld. The left and middle plots show Task Goal Completion (TGC) on the normal (TN) and challenging (TC) test sets. collapsed model (GRPO) does significantly worse than in which entropy preserved. Bottom We AppWorld-trainee model and continue training on AIME. The same trends hold. All curves reflect the mean across three independent seeds.
Clipping modifications protect entropy. Following the intuition provided $ DAPO and GSPO retain considerably more entropy. Confirming our observations in o DAPO's entropy can uncon- trollably increase in some experiments without an entropy-control mechanism (Fig 5
Entropy-preserving methods outperform baselines. REPO-R and ADAPO score higher than their off-policy baselines (GRPO and DAPO) and maintain steady policy entropy throughout training.
6.2 ENTROPY PRESERVATION AND DOWNSTREAM PERFORMANCE
We evaluate the effect of entropy preservation on downstream performance. See Fig. for pre- view of these results. We find that methods that preserve per-token entropy, maintaining higher cumulative entropy over training, yield higher final test accuracy than those that don't t. These trends are stronger on AprWorld than AIME. We hypothesize that Owen-3 family of models is heavily optimized for AIME, and so this optimization may have primarily involved sharpening around ex- isting solutions. WorWorl,, on the other hand, requires considerable exploration to discover new capabilities.
6.3 ENTROPY PRESERVATION ASSISTS SEQUENTIAL TRAINING
We evaluate how well different algorithms support further RL fine-tuning different task (i.c. sequential training). To that end, we first train Qwen-3-68 on either the AIME or AppWorld. We then take the best checkpoint the starting point for training the opposing environment. Fig. shows that policies trained with GRPO perform poorly in the second training stage; due to entropy collapse, they lose their ability to explore. On the other hand, DAPO, and especially REPO, start re-training with ample entropy and retain their exploration ability over the course of training.
6.4 NUMERICAL PRECISION STABILIZES ENTROPY AND PERFORMANCE
Figure 3 shows that for Qwen -- AppWorld training the numerical fixes have dramatic im- pact: DAPO, which previously exhibited entropy collapse in this setting, now shows rapid entropy increase the analysis of its asymmetric clipping design suggests. This shows that the observed entropy dynamics are highly sensitive to implementation details that may not be immediately ap- parent, and that some previously reported entropy collapse phenomena may have been artifacts of numerical precision rather than fundamental properties algorithms.
RLOO achieves state-of-the-art performance. After switching to FP16 training ($4), purely on- policy RLOO sets the highest score at the time of submission on the AppWorld benchmark: our best checkpoint scored 79 % Test Normal and 71 % Test Challenge with Qwen-3 32B..

---

Published as a conference paper at ICLR 2026
7 RELATED WORK
Reinforcement learning has emerged as the dominant paradigm for aligning pre-trained language
models (Ziegler al. 2019 Stiemnon al. 2020 Ouyang et al. 2022). This approach has been suc-
cessfully scaled in environments yielding verifiable rewards such as programming and mathematics
(Jaech al. 2024 Lambert et al. 2024 Comanici al. 2025 Guo et al. 2025 Team al.. 2025).
Empirically, training in this setting has typically been viewed as sharpening the base policy around existing solutions rather than yielding new ones (Gandhi et al.) [025 Liu et al.) [202bb] Yue et al.) 2025 Zhao et al. 2025). A good pre-trained base policy starts off already calibrated to many rea- sonable reward functions, and post-training can be viewed as tempering this distribution (Kadavath) et al. 2022, Cui et al. (025). In fact, several works directly exploit this calibration to drive accu- racy improvements via unsupervised post-training. AAgawal et al. (2024) simply minimize entropy, Prasad et al. (2024); Zhang et al. (2025); y; Zuo et al. (2025) align to the model's majority vote dis- tribution, Wang et al. (2025) get by with a single labeled sample, and [hao et al. (2025) even use random rewards. All of these works can be explained by simply allowing policy gradient to sharpen an already calibrated base policy. While this type of approach can help pass@1, 1, it harms pass@k
Shao et al. 2024 Dang et al. 2025 Yue al. 2025).
Some works protect against this pathological entropy collapse using modified policy gradient ob- jectives. He al. (2025) add auxiliary rewards solutions function of their probability rank within a batch. Yu et al. (2025) introduce wider PPO clipping to encourage stronger reinforcement of low probability correct actions. ZZheng et al. (2025) propose sequence-level clipping more inde- pendent of individual action probabilities. Chen et al. 2025b) reformulate online policy gradient to optimize pass @ as opposed to pass@l. Most similarly to our work, Cui et al. (2025); Xi et al.) 2025); ;; Wang et al. (2026) derive theoretical results regarding the covariance between advantages and probabilities mediating entropy collapse and then propose different approaches to counter this.
Other works impose a DKL penalty during training as an approach for preserving the base policy (e.g., Diegler et al. 2019 Guo et al. 2025| etc.). However, has been shown that such an approach limits how much the policy can learn (Korbak et al. 2022 Yang et al. 20244] Wu & Choil 20225. For this reason, Chen et al. (2025a): Yu et al. (2025) remove the DKL penalty, (Vassoyan et al.. 2025) ignore it for subset of tokens, and (Liu et al. 2025a) iteratively reset the reference policy.
8 CONCLUSION
In this work, argue that entropy should be actively monitored and controlled throughout rein- forcement learning training for language models. We provide a theoretical analysis showing how policy gradient objectives modulate entropy dynamics, explaining why algorithms like GRPO ex- hibit entropy collapse while DAPO and GSPO provide implicit preservation. We identify critical empirical factors, notably numerical precision (BF16 FP16) and framework behaviors (FSDP2 output casting), that impact entropy dynamics and training instabilities. Building on these insights, we propose explicit mechanisms for entropy control: REPO, which modifies the advantage func- tion, and ADAPO, which adaptively adjusts clipping thresholds. Our entropy-preser preserving methods perform strongly on AIME and AppWorld, outperforming their baseline counterparts (GRPO and DAPO) and improving sequential learning. We also report state-of-the-ar results on AppWorld at the time of submission (79% Test Normal, 71% Test Challenge with RLOO and FP16 training).
We identify distinction between strictly on-policy algorithms like RLOO and weakly on-policy algorithms like GRPO and GSPO. Our results show that, with proper numerical handling, strictly on- policy RLOO achieves the best performance overall. However, strictly on-policy training requires synchronous updates, which creates a bottleneck in distributed systems. Weakly on-policy methods enable asynchronous training pipelines where trajectory collection and policy updates can proceed in parallel, significantly improving throughput. The (REPO, ADAPO) compatible with both paradigms and can help weakly on-policy methods ap- proach the performance of strictly on-policy training while maintaining the throughput benefits of asynchronous execution.
mechanisms we propose
Overall, we highlight that entropy (and the corresponding exploration capability) is crucial for ef- fective policy optimization and should be treated first-class concern in RL training pipelines.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper investigates the properties of policy gradient algorithms for language model reasoning, specifically focusing on the tendency for entropy collapse during training. Our research is primarily theoretical and analytical, involving mathematical analysis and algorithm development. Our work aims to improve entropy during reinforcement learning, which can lead to better exploration and wider diversity generated outputs. We acknowledge the potential for misuse of advanced lan- guage models, including the generation of biased, harmful, misleading content. We believe that responsible research practices, including transparency in model limitations and potential socictal impacts, crucial for mitigating these risks, and we hope that our research contributes to the development of more robust, creative, and beneficial language models.
REPRODUCIBILITY STATEMENT
Complete proofs for all theoretical claims, along with experimental details and hyperparameters, are included in the appendix. All data points presented in this work the result of multiple repetitions of each experiment using independent random seeds.
USE OF LARGE LANGUAGE MODELS FOR WRITING
We acknowledge the use of large language models to assist with typographical corrections, phrasing, and self-review aimed at improving the clarity and structure of this manuscript.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. The Twelfth International Conference on Learning Representations, 2024.
URL htttt::://ppprreeeeeee www.wwwwwwwweeeenneeeee..
net/ forum? id 32Kt aqx
(Cited p. 10)
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-styL optimization for learning from human feedback in L.MMs. In Proceedings of the 62nd Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248-122 12267, 2024.
URL https::///// http:s:://claathhoogo... orte/rrg///0 2024 acll-acl long. 662 (Cited on p. 3
Kevin Chen, Marco Cusumano-Towwe Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive LLM agents. arXiv preprint S.Xii:25022./600 2025a. URL httph:ttp///cc... //axiv../rriiv../a org. abs/ (Cited on p.
2502. 01 610
30)
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508. 10751, 2025b. URL https: wtwpaaiiiiv////a///.aa////aaavva wwtpivv..rrpppp////eee// 10751
(Cited on p.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261. 2025. URL htttps//aarxiv.oorg//b//// 06261
(Cited on
[0)
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint 2tti::22552.2..11. 2025. URL httptttt////xii... (Cited on
erq/abs/ 2205. 22617
Xingyu Dang, Christina Back, Zico Kolter, and Aditi Raghunathan. Assessing diversity collapse in reasoning. In Scaling felf-Improving Foundation Models without Human Supervision, 2025.
L https::///// hittps://openreeee ew n d KsHLjQh (Cited p. 10

---

Published as a conference paper at ICLR 2026
Benjamin Eysenbach and Sengey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022. URL https:
//openzeview.o wttp::/wwwweeee.t..e..eee http://www....e..oo..oooo/aaaaaa
(Cited p. !
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cogni- tive behaviors that enable self-inmro reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307 2025. URL
www. httprrg/ab/2200000 01307 (Cited
on p. 10
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL
http:://eeeddd.rooo http:///enoo...crreeee//////////
(Cited
on p. 25
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-RI: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprini arXiv:2501. 12948, 2025. URL https://
arxiv.org/abs: 2501. 12948 (Cited on p.
10
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017. URL URLhhttps:/ ptttpps:/proceeediiii
ml wtt.m//....eeee/////or/r///////.
html
(Cited on P.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018. URL httpss:/pprocedding (Cited [
mlr. press// w00 h/waarrooo jal 8b
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting GRPO beyond distribu- tion sharpening. arXiv preprint arXiiv25506.02355, 2025. URL /ttttss///axxv....t abs/
2506. 02355 (Cited on p. 10)
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Confer- ence on Learning Representations, 2022. URL http:::/ nZeVKeeFYf9, (Cited on p. (5)
Aaron Jaech. Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. OpenAl ol system card. arXiv preprint arXiv:2412.16722 2024. URL https: atw.aaaxivvv.... httpa/aiw..oo.ab//////aaa 16720 on p.
(Cited
10p
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Seafield-Dodde Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207.0522/, 2022. URL https:
//arxiv.org/abs/ http///as/222..... 05221 (Cited on p. 10)
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Unlocking RI. potential for L.LM reason- ing through refined credit assignment. In International Conference on Learning Representations, 2024. URL https::////. http///ppneeeiiiieeeee
wwtee/fooummicc.m wtaprrummm..-mmmmGGxxx 5mJrGtXVwz (Cited on p.3) 3)
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! Deep RL Meets Structured Prediction Workshop at ICLR, 2019. URL https: (Cited on p.
//openreview.. htw.://wwnentttmmmmmeooolllge-ee
Tomasz Korbak, Ethan Perez, and Christopher Buckley. RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computational A Linguistics: EMNLP 2022, pp. 10981 2022. otl.tttss.//aalaattooooo.oo///// wtwp:://2..iiiiinnnneeee emalp..
77 (Cited on p. 10)
12

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th symposium operating systems princi- ples, pp. 61 1-626,, 2023. URL. https::/ddd acm. org wttgg/dii/ee /atoo//ab//ii /10. 00......11551000000 3613165
(Cited p. 25
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15122 2024. URL
abs/ https://arxiv:: wttp://p//bbb/..a.a.baa
(Cited on p. 10
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang. Kashif Rasul, Longhui Yu, Albert Jiang. Ziju Shen. Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. NuminaMath-I4e 1.5, 2024. URL hetps:
//huggingface. ct/ wwtpsetstaasaaaaaaaeaaaaaaaass MO/ MK/Numinean WuwiiaMMthh-ia
(Cited on p.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged reinforcement learning expands reasoning boundaries large language mod- cls. arXiv preprint arXiv:2505.2448 248644, 2025a. URL nttps://arxivv. org abs///bs////0000 2505. 24864
(Cited on p.
10
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. arXiv preprint arXiv:2503.20788 2025b. URL httph:://aaaate wwt.:/o//g///ao./ee 20783 (Cited on p.
10
Sami Marreed, Alon Oved, Avi Yaeli, Segev Shlomov, Ido Levy, Aviad Sela, Asaf Adi, and Nir Mashkif. Towards enterprise-ready computer using generalist agent. CoRR, 2025. (Cited on p.
26
Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondfej Čertík, Sergey B Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, al. SymPy: symbolic computing in Python. PeerJ Computer Science, 3:e103, 2017. URL https::/peeett com/
articles/cs-103: (Cited on p. 25
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. vances in neural information processing systems, 35:27730-27744, 2022.
Ad- URL
httpsss/pproceeiinggseeee mww.eeiiiiinnneuuuusoere. httpa//////peeeeeeee//////////// LIEfe5aeeea111440000000111-aaaa0
html
(Cited on
p. 10)
Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mo- hit Bansal, Sainbayar Sukhbaatar, Jason E Weston, and Jane Yu. Self-consistency preference optimization. In Forty-second International Conference on Machine Learning. 2024. URL https: /ttp::////eeeeieeeeeee www../ee///// hett/oorummdd=4444eee.
(Cited on 10
Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. De- feating the raining-inference mismatch via fp16. arXiv preprint arXiv:2510.26788 2025. (Cited on 23
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pp. 1889-1897. PMLR, 2015. htttpp////poeeednnnnn.m.....eeee p.
html (Cited on
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv1707.0634/ 2017. URL https: aarxiv..
brg//bs//700 06347
(Cited on p.
13

---

Published as a conference paper at ICLR 2026
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in RLVR. arXiv preprint arXiiv2506.109900 10947, 2025. URthtts:: http://axxivv...gaaa...
2506. 10947
(Cited on p. 10
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. DeepSeekMath: Pushing the limits of mathe- matical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URL
ht httpss://arxa www.aiiii..........o..oooooooore abs /onss/24422 2402. 03300 (Cited on p. 10
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. with human feedback. Advances in neural information processing systems, 33:3008- neur ips. http///ape///////eeeeeeeeeeeeeee Abstract (4986586622299333f83488888888883 at ceattracthhtmll (Cited on 10
Learning to summarize
3021, 2020. URL https::/
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. (Cited on p. [
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599, 2025. URL htttt:://aaattttte org/ hbt.g//bb//5000 12599
(Cited on p. 10)
Sebastian B Thrun. Efficient exploration in reinforcement learning. Carnegie Mellon University, 1992. (Cited on p.
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16022- org. acl-acllll ong. 850/
16076, 2024. URL ptatps://aclanthoooo P.7
(Cited on
Jean Vassoyan, Nathanaël Beau, and Roman Plaud. Ignore the KL. penalty! boosting exploration on critical tokens enhance RL fine-tuning. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 6108-6118 18, 2025. URL pttrp:://aclanttolo.g.oo org/
2025. Findings--aach 340
(Cited on p.
10
Shumin Wang, Yuexiang Xie, Wenhao Zhang, Yuchang Sun, Yanxi Chen, Yaliang Li, and Yanyong Zhang. On the entropy dynamics in reinforcement fine-tuning of large language models. arXiv preprint arXiv:2602.03392. 2026.
(Cited
on p. 10
Yiping Wang, Qing Yang. Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example, arXiv preprint arXiv:2504.20571, 2025. URL https://
Arxiv.org/ abs 2504. 20571 (Cited p. 10
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning near near-perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations, 2020. URL.tttpss /tttttss://ppeeevieeee /pppnrevie......
net. (ttoorummi==l11xeeee
(Cited p. 25
Erik Wijmans, Brody Huval, Alexander Hertzberg, Vladlen Koltun, and Philipp Krachenbuehl.
Cut your losses in large-vocabulary language models. In The Thirleemth International Confer-
ence on Learning Representations, 2025. URL httpt::/t///oo ftttps://opezzevew..oo net forrttffrmmmmmmm
E4FK3YuG56 (Cited on p.
6 7 21 25
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reifoorrmmmniiiinnmmmmmmmt learning. Machine learning, (33):2292556, 1992. URL httpt::ttt/ssitt http::://iiaa...siing..... com/
(Cited
17
14

---

Published as a conference paper at ICLR 2026
Fang Wu and Yejin Choi. On the limits of RLVR: Support, entropy, and the illusion of reasoning. In 2nd Al Math Workshop@ ICML 2025, 2025. URLjttpp:///ppeneviiee net ftr.ruuuuweefffummm forum? id= 1d=Kxt
(Cited on p.
[0P
Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, et al. BAPO: Stabilizing off-policy reinforcement learning for LLMs via balanced policy optimization with adaptive clipping. arXiv preprint arXiv:2510.18927, 2025.
URL https: /tappp://aaxiv... wrwiv. org/ absaabss/5555 2510. 18927 (Cited on
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXhv:2505.093888, 2025. URL https: http://axvvv... org/ 2505 09388 (Cited on p-
Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami. Asymptotics of language model alignment. In 2024 IEEE International Symposium Informa- tion Theory (ISIT), pp. 2027-2032. IEEE, 2024. URL
iew...rrreeee..//..oo..oooo org/
alstractt/dcument//00/
(Cited on p. P.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14472 2025. URL htlhttsss//axix>> wrg.ii.oorrroooooo..ooo /bs/ (Cited p.
2503. 14476
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? arXiv preprint arXiv:2504.1388 13837, 2025. URL https::/ //axiii ott.:////////../33///3///
(Cited on
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. is already half the answer: Fully unsupervised LLM reasoning incentivization. arXiv:2504.058/2, 2025.
Right question arXiv preprint (Cited p. TOp
Ruisi Zhang. Tianyu Liu, Will Feng, Andrew Gu, Sanket Purandare, Wanchao Liang, and Francisco Massa. SimpleFSDP: Simpler fully sharded data parallel with torch. compile. arXiv preprint
arXiv:2411.00284, 2024. URLhhttps / /wwa/aiivvv..// www.ii.orr//oo.. abs 2411. 00284 (Cited on p. 23
25
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RL post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912. 2025. URL aittps://arxiin org httggabb//555t /504 07912 (Cited on p-
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18077, 2025. http:ttts///axxi..orrgggo
1180//.11//2500 18071 (Cited on p.
3
10)
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433 1438. Chicago, IL, USA, 2008. URL 0tt.://.1..555111111 5555/ 62027202222 1620297 (Cited on p.
https: wtt.:/aa.....oooooo////oo
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0859; 2019.
(Cited
on p. 10
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. TTRL: Test-time reinforcement learning. arXiv preprint arXiv:2504. 160.4 2025, URL htthttts::/ arxiv. ort/ 16084 (Cited on p. 10
15

---

Published as a conference paper at ICLR 2026
A PROOFS & DERIVATIONS
A.1 BROADLY USED LEMMAS
Lemma 1. The expected score function of policy The at some state s is:
Bawro(-la) [Vo log Ta(a 8)] = 0
Proof.
(- [Vo To 8)]
Σ To(a s)
Ve log Te(a 8)
Σ V V s)
Vo Σ To (a |s)
(o(1)
Lemma 2. The gradient of a sample estimate Eg, [f(())] of function Jo over distribution Pa is:
VO Po [fo(x)] Ez (Vfo(() fo(x) Vo log Po(x))
Proof.
VO [fo(x)] Σ Vo (
Σ Po(x) Vofo(z) fa(x)
VoPo(z)
Po(z)Vo log Polr
Σ (o(f)(() + fo(z) Vo logog Po(x))
Ep (V.ol() + fo(x) logog Po(x)]
Lemma 3. The gradient of a sample estimate Ex [fo(x)) of function fo over distribution Po can be baselined for any arbitrary independent of x:
VgEr-P, (fo(z) VOEEu [fo())]
Proof.
VOE PpEEP [fo(z) - b] = EpnP, [[fe(z) - b) Vo log [x
Eg [fe(z) Vo (a(x) Eg [b Vo log Po(x))
EswPo [fe(r) V log Pa(x)) b- Ez (Vo log (a())
EawP [fo(z) Vo log Po
VoEx~a [fo
Lemma 4. The gradient of MDP objective J JDP at some state s is: MDJMDD(s) = Ea-za(la) [(R(s, - b) Vo log (a(a 8)]
for any arbitrary baseline b independent of a.
16

---

Published as a conference paper at ICLR 2026
Proof. Largely following (Williams 1992). Lemma 2 and Lemma 3
VoJ MDD(s) = Vo VoBurne((1e) [R(s, a)]
Vo
[R(s, a) -b]]
(a (oonxo((1)]((((s..)))))) [R(s, -b) Ve log ( 8)] Rawao(-le) (R(s, a) b)
(()(s, [()(ss)) b) Va mogna(a s)]
Lemma 5. The gradient of the policy entropy some state is:
VoHla = -Bawnp (+1s) [log To(a s) - b) Vo log To(a s)]
for any arbitrary baseline independent of a.
Proof. Follows directly from Lemma 4 with R(s,a) a) = -olog Ta(a 8).
VoHlno ( = V -VoBa-tullee [log To(a |s]]
= - (- (logmo(a so(a 8) b) Vo To(a) 8)]
Lemma 6. The expected advantage function A(s, a) d R(s,a) b, with baseline V(s) def Eawxo(-1a) (R(s, a)}, at some state is:
E { [)(s,))] =0
Proof.
E Ba (A(s))) a)] = BaNtoo [[(8 a) V(s)
= Bawnro [R(s.a) V(s)
V(8) - ((8)
0
A.2 ENTROPY DYNAMICS UNDER POLICY GRADIENT
Theorem 1. Given a policy gradient update ō = 0 + a VoJ MDD(s), the expected change in entropy is approximately:
AH (: (1)) -a (e) [A(s,a) L(s,d) u(s,,)]_(ssdd))
T(8, a) dd (a 8) Bawwal--le) log Tola s)] denotes mean-centered log-probabilities and u(s.a) Vo log (a 8) is the score function for a policy na evaluated at state s and action a.
s)] denote mean-centered log- probabilities and let u(s, a) Va log Ta(a 8) denote the score function of policy evaluated at action and state s. Let g(s) and h(s) denote the respective mean-baselined policy gradient and entropy gradient evaluated on- on-policy in some state s:
Proof. Let a(s,a) def log (a
s) EEa-To(-)s)loo [log (a
g(s) V MDP Ea [A(s, a) u(s, a)]
h(s) V H- (- 8) R [T(s, a) u(s,a))
Here, each estimator allows for arbitrary baseline that cancels through the parameter gradient Vo. While the baseline does not influence the exact mathematical construction, does influence approximations the change in entropy. Here chose mean baselines to center the policy, mini- mize variance each gradient estimator, and to agree with a tabular softmax approximation of the change in entropy (see Corollary 2)
17

---

Published as a conference paper at ICLR 2026
Using the first-order Taylor approximation: Ha (. 0 9) ~ Hoo s: small learning rate a, the expected change in entropy from policy gradient update in state is:
1 + 9 h, for
A mm (- a. g(s) ((s)
(Ea [A(s. a) a)s) (Balwma((1s) [L(s, u(s)d')))
a Bawno a' [A(s.a) a) L(s, a( (lss,))T(ssdd))
A.3 APPROXIMATE ENTROPY DYNAMICS UNDER POLICY GRADIENT
Corollary 1. Assuming u(s, a) Jo)TTul TuT, a²) = for all a + o', the entropy change is proportional to:
AH 8) -awNTo(-)s) [A(s,a) a) L(s,a) TT(a s)]
Proof. Assuming the score vectors satisfy orthogonality of the off-diagonal terms such that u(s,a) a) T u(s a') = O for a # al, the double expectation can be collapsed, yielding:
AH (- (-1 -a (a [a( 8) A(s,a) L(s,a) [] a)ll
Assuming independence of the squared gradient norm magnitude, such that it can be treated as constant with respect the expectation,
AH (- () x - -E a) L(8, a) To(a) s)]
A.4 ENTROPY DYNAMICS UNDER POLICY GRADIENT FOR TABULAR SOFTMAX POLICIES
Proposition 1. For two functions fxx and g(z) over samples x as of softmax distribution as(x) cxp(S=)/ Σn ex)(Sk), the dot product of expected gradients
(f log (s))
Eywrs (9()) log ns(vil) Ex (ra( g)
where f Buwns [=)] and g Eg [9)))
Proof. First, let's compute V log (x: for the softmax distribution:
exp(S.) = S. - -log Σ exp(S (S&)
log (xx) = log Σx exp(Sx)
exp(S.)
S. log Tx() La=z
= L==x - (2)
k exp(Sx)
where Iz=y is the indicator function if y. otherwise).
Now let's compute the dot product V. log T. (x:) V log (s));
V (og)s(#) T V log (y) = Σ 2= (1== (==)-ms(())(1
Ts(i))
Σ Ly=j 1==) TT((s) 1y= + Ts())²)
1 1 - (T()) TT()) EENEY [zT(
Now we can compute the dot product of expected gradients:
E (z) V log Rs (2) Eywns (g()) V log (sg))())))
E [f (x) ((y) V lolog (x)) V log [y()
E [(() g(v) (1.=y (x) TT( + E (=)())
18

---

Published as a conference paper at ICLR 2026
Let's compute each term separately:
Br-rs ywm [xx) ( 1 l==yl = Bx TT(x) (x) g(x)]
Eu (x) ((v) [T)= Ex [f(z) . sx)=]] Ey [g())l = Ea [ f(x)] g
E Bxwns [xx) g(v) =s(y)] Exrrs [f()] By-rs (g(v) Ts(U)l - flayns (T)(1) g(v)]
Exwnswwrg (x) ((y) Ex [=)]]] Ex [ms(2)] ExENs [f()] Eynor [9()) Exnxs [TS(T)] f-
Therefore:
(E (f()) V log Ts (x) Ey Vs ls Ts)))
Bnwws (x) f(z)- g(x)] En [T>( f(z)] g f- En [==ng|=s(x)) g(>)] + E En [ f.g
Baowy (x) (f(x) s(x) f(x) 99 g()) + +f
Baowy (x) (f(x) f) (0(x) - 5)]
where f RE [fx)] and g Es [x)))
The above proposition holds for simple softmax policies, but involves a much more complex gradient term and inner product for generic transformer-based policies.
Corollary 2.
Under tabular softmax policy, policy gradient update 0 == + VeJ MDP changes the entropy approximately:
AHry (.18) ~ - (awns((la) -C--Eunsstla [s( (8) (1og [log 8) log RS(. 8)) (R(s, R(s) R(())
where (s (s [log Ts(a s)] and R(s) (a [R(s,a).
Proof. Let g(s) and h(s) denote the respective policy gradient and entropy gradient evaluated on- policy in some state
((8) JeJ Ea (-1a) [R(8,a) a) Va log (a 8)] ((s) VoHlo (- s) = - (alrr(())) logg To s) Vo log To(a s)]
Using the first-order Taylor approximation: How ( 8 0 + a g) R H (.1 s 0) + a 9 h, for small learning rate the expected change in entropy from policy gradient update in state is:
AHro (: 8) g(8) h(s)
= (awTTs()))
(T(aa 8) (logTs(a log Rs(- 8))
R(s)
The second line follows Prop. [ Note that gradient interactions through the softmax automatically center the reward function, i.e., A(s, a) R(s, a) V(s) R(s, a) R(s). The log-probabilities, too, are centered as here they reflect R(ss a) = log TT (a s). This yields form equivalent to Corollary
A.5 ENTROPY DYNAMICS UNDER CLIPPED PPO
Proposition 2. Given two distributions ा (æ) and ((z) with constraint relative entropy is bound by
+ e all x, their
H(m) (1 + 6) H(O)
19

---

Published as a conference paper at ICLR 2026
Proof. Let's parametrize <(x) 320(x) with Pg and compute its probability
H()) = - Euns [log π (x)]
Exnr [log ((x)] Exys [log 3x]
log (x))
Exns ((x)] Bx
((x)
Dil.(=110)20
< Exo [lo ((x)]
(z)
Exne x(x)
log (xx)
Exws [3g log vxx)
< Exn [11 + c) - log d(x)]
+c) H())
The second-last line uses log x0x) 2 0 and 3r S (1 +c) by definition, hence Bg log ((x) <
(1 + +) - logg (xx).
Theorem 2. Proximal Policy Optimization (PPO) bounds the entropy Ha (. 8) of the updated
policy by the original policy entropy H Hzzuee: 8) such that:
(1 -low) Harnel (- 8) I Hm 8) S (1 + Chigh) Hrgmel: (s)
Proof. Applying Prop.
Ne sew 2 to
S 1 + Chigh yields the upper bound
To
Hag (. S (1 + Ehigh) mm
Applying Prop.
2
to
Elow
Me sew ua
Te ou (equivalently
Tg tew
I T
yields the lower bound
1
H Hxz (1|
Ho (-1
1
Elow
or equivalently
(1 - flow) H.m ao
(18) S Hor (. 18)
A.6 ENTROPY CHANGE UNDER AREPO ADVANTAGE FUNCTION
Proposition 3. For advantage AREPO(S, a) de A(s,a) - Ba I(8, a). the first ordtr change in entropy induced by
step is: AHREPO (. (1 R AH (. (8) B. [|BaNto((1)) (L(8,a) u(s a)]l?
Proof. Let g(s) Ea--ra(-ls) [A(s, a) u(s, a)] and lu(s) - Ba-av((lss) [L(s a) u(s, a)) denote the respective policy gradient and entropy gradient evaluated on-policy in some state s.
Using AREPO, the policy gradient becomes:
(REPO(s) = (aa-a((-i)) [(A(s, - BaL(s, u(s,a))
The first-order entropy change is:
REPO (: 8) R a (REPO(8) h(s))
no
(Fanns()le) [(A(s, a) (,I((, u(s, h(s)
(Banns((ls) [A(s,a) u(s, a))) h(ss) Bo ...a a (Eawnot)la) [L(s, a)) u(s, h(s)
a g(8) h((8) + Bg h(8)) li())
AHHo( 3, [ [T( a) u(s a)]l
20

---

Published as a conference paper at ICLR 2026
A.7 CONNECTION TO ENTROPY REGULARIZATION
The REPO objective has a direct connection to standard entropy regularization. Consider the REPO- D advantage for given state 8:
AREPO(S, a) = A(s, - 3Ba L(s,a).
The resulting policy gradient is: VoJ REPO (8) Bawno(-1e) [AREPO a) u(s, a)]
( [A(s.a) a) u(s,a) a)] +Bs BanTo((1s) ((s u(s, a)].
(ODAMDe(s) where the second equality follows from Lemma 5 Thus, REPO-D is exactly equivalent to augment-
(-Ho,
ing the MDP objective with explicit entropy bonus:
(REPO(S) T MDP( 3. HTo M (-)s).
Advantage over an explicit entropy bonus. Despite this equivalence, REPO-D offers two prac- tical advantages over directly computing and differentiating the entropy bonus.
Zero additional memory cost. Computing an exact entropy bonus requires materializing the full logit Vector over the vocabulary for every token in every trajectory, the entropy H. 8) Eg (a s) log (a s) is sum over all vocabulary entries. For large vocabularies (c.g., > 10 tokens) and long trajectories, this is unnecessarily memory-intensive.
contrast, REPO-D estimates the entropy gradient via REINFORCE using only the log-probability the sampled token, which is already computed during the forward pass when using Cut Cross-Entropy (Wijmans et al., 2025 REPO-D therefore incurs zero additional memory cost beyond the standard policy gradient computation.
Variance reduction via control variates. Because REPO-D couples the advantage A(s, a) and the centered log- bog-probability L(s, a) at the level of individual sampled actions. it naturally acts as a control variate for the policy gradient estimator. When advantages and log-probabilities are pos- itively correlated-as as is typical during the entropy-collapsing phase of training, per Corollary [ subtracting the 3. term reduces variance in the combined gradient estimate. This coupling would be lost if the policy gradient and entropy bonus were estimated from independent samples.
A.8 PROOF OF THEOREM 3 (BF16 MULTIPPIIATIIV BIAS)
Theorem 3. Under bf16 quantization, the observed ratio exhibits a multiplicative upward bias: B|r [Twe]] > Tune- [Proof in App.
Proof. Let Enow and Eold denote the quantization errors:
log No (a)s) bf16(log (a(ass)) + Enew
(I)
log Te. (als) = bf16 Te.. (a(s)) + Eold
(2)
Assuming the lower 16 mantissa bits are uniformly distributed, Enew Eold Uniform(-uu (-ulp/2, ulp/2) independently, where ulp is the unit-in-last-place exponent.
the relevant
The true ratio can be expressed as:
= exp(bf16(log Te) - bf16(log + Enow Eold)
(3)
Tobserved exp(6)
(4)
where 8 = Enew Sold-
Therefore:
Tabserved
Time exp(-6)
Since 5 is symmetric around zero with Var(ō)
Var(Enew) Var(Eula) (ulp + ulppaa//12
Jensen's inequality for the convex function exp( -..) yields:
E ['oobscreed
Tune]]
Fune
IE [[[exp(-0)]
(5)
Fune cxp(E(-5))
(6)
Fine
21

---

Published as a conference paper at ICLR 2026
For small quantization errors, a Taylor expansion gives:
B/ B
Tobserved
T'muel
RJ Tirue
(1 + Var(o)/2)
(8)
Turue
(1 (ulp + 224)
(9)
The bias is proportional to Ctree: larger ratios experience proportionally larger absolute bias.
22

---

Published as a conference paper at ICLR 2026
B NUMERICAL CONSIDERATIONS: ADDITIONAL DETAILS
The main findings numerical considerations (FSDP2 output casting, FP16 vs BF16 training, and their impact on entropy dynamics) are presented in $ details.
This appendix provides additional technical
B. ADDITIONAL DETAILS ON FSDP2 OUTPUT CASTING
As described in App. GPUs (Zhang et al. 2024). In the HuggingFace Accelerate library, FSDP2 configured to cast all module outputs to the chosen Roating-point type (e.g., BF16), including the final model out- puts, even when the computations involving logits (such as softmax) are performed in full 32-bit precision.
we use the FSDP2 framework for distributed training on multiple
This is the default behavior of the library, and at the time of submission there was no single con- figuration parameter to switch it off. To preserve full-precision log probabilities, the user must explicitly override the output_dtype of the Mi sedPrecissonn wisssooooooloooooooo ciyycyy (MPP) object (see torch/
ssharddffsdpaap py for details).
Naively, this cast should not affect the RL gradients, as the backward pass of such a casting opera- tion is the identity function. Indeed, there appears to be measurable difference for fully on-policy algorithms like RLOO. The half- malf-precision downcast, however, does measurably impact the numer- ical stability of the importance weight and thus affect off off-policy algorithms that use clipping, such as LOOP, GRPO, and DAPO.
Fig. 2a empirically demonstrates the clipping bias introduced by the 16-bit rounding when training with DAPO. We observe that when the rounding is present (before the Mi xedP recis onPol olll icy fix), more tokens get clipped due to exceeding the higher end of the range ehigh preventing probability increase for low probability tokens and thus reducing overall entropy. At the same time, fewer tokens are clipped due Ejow- The overall effect is the tightening of the clipping on the higher end of the range while relaxing it on the lower end, resulting in the reduced effectiveness of entropy preservation from the asymmetric clipping. It can be further noted that the 16-bit rounding changes the clipping outcome only for tiny fraction of tokens, fewer than 0. 1 % of the total number of output tokens. This suggests that a very small number of pivotal tokens play an essential role in learning and warrants further study of this effect.
App. B.3 empirically confirms the significant impact of half-precision rounding on the overall per- formance and entropy dynamics (see Fig. 8).
B.2 6 TRAINING
In our original experiments, the models were trained exclusively in bfloat16 (BF16), which has become common practice in LLM training because of its higher dynamic range. Recent publica- tions (Qi et al.] 2025) reported improved RI. training with withfloat (FP16) floating-point format as its additional 3 mantissa bits enable more accurate gradient representation.
In addition, the choice of floating-point format affects the discrepancy between inference (vLLM) and training policies. These discrepancies are inherent to RL systems with separate inference server and arise from small differences in model-layer implementations as well as from the lack of batch-size invariance in GPU kernels. In our experiments, we find that FP16 training significantly reduces the anference-trainigg discrepancy (see Fig. 2b).
B.3 ABLATION STUDY
Fig. g summarizes the ablation study of the numerical tweaks described in Apps. B.I and B.2 per- formed for DAPO training Qwen-3 BB. We observe that when the MPP fix and FP16 training are used together, the entropy dynamics of DAPO change completely, from collapse and sub-par exploration to rapid increase in entropy the course of training. More generally, we observed improved training across models and algorithm variants when both of the above changes were ap- plied.
23

---

Published as a conference paper at ICLR 2026
br16
1p16
Clipping fix, b/16
Clipping fix, fp16
0.32
0.60
Test Challenge TGC
Test Normal TGC TGC
0.3
0.45
Entropy
0.16
0.1
0.08
0
160
240
80
160
240
80
160
240
Iterationn
Figure 8: Cumulative effect of the MixedPrecisionPoliey (MPP) fix and FP16 training when applied DAPO algorithm with Swen-3-88. Each curve represents the mean of three independent (seeds). This is the expanded version of Fig.
B.4 SOFTMAX GRADIENT NUMERICAL PRECISION
An additional numerical issue arises in the backward pass of the softmax function when computing gradients for high-probability tokens. For a token with probability the gradient computation involves the term- -A (1 (1-p) for the sampled token and for all other tokens in the vocabulary. When p exceeds approximately threshold in half precision), the value 1 p rounds exactly zero, causing the gradient to vanish entirely for high-confidence sampled tokens while the gradient for all other tokens is still non-zero. Essentially this makes the gradient vectors for high-confidence tokens incorrect. Surprisingly, in our experiments this affects on the order of 30-40% of tokens; these high-confident tokens often correspond to chat template formatting or programming language syntax.
2-23 z 0.9999999 in single precision (or the corresponding
This issue is distinct from the BF16 multiplicative bias discussed in App. B. and affects even full- precision training when the model becomes highly confident. The problem manifests as:
Gradients for high-probability tokens become incorrect
Learning dynamics may be distorted for tokens where the model is already very confident
The solution involves an 0(1) per time step recalculation of the problematic terms 64-bit precision. Specifically, when computing the softmax gradient, we identify the highest probability tokens for each time step and simply recalculate their gradient in softmax backward step using 64-bit arithmetic. This targeted approach preserves numerical accuracy for high-confidence tokens without incurring the memory overhead of full 64-bit gradient computation for the entire vocabulary.
Fix.
In practice, this effect is much less noticeable compared to the other numerical improvements (FP16 training and MixxedrrcissonnPoiiyy fix) mostly because the magnitude of affected gradients is tiny compared to tokens with p & The combined effect of all numerical fixes shown in Fig.
24

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENT DETAILS
C.I ENVIRONMENTS
Interactive tool-use agent. For the AppWorld benchmark, rollouts proceed in turns (up to 30 turns during training and during evaluation), in a manner akin to an interactive notebook.
During each turn, the model generations are parsed to extract any Python code blocks, potentially containing calls to Wpr API. These are executed retrieve information or alter the environ- ment state. The outputs of successful API calls or the error trace of incorrect calls appear in the agent's context after each turn. Once done, the agent may mark a task as completed at which point it is assessed whether the task state was updated successfully. Failure to mark the task complete within the turn limit or context limit (32K) results in a failure. Sparse outcome- -based rewards in [0, 1] are assigned during training as the fraction of passing unit-tests. Binary rewards in {0, 1) are used during evaluation requiring complete correctness.
Mathematical reasoning. For the AIME henchmarks, model responses processed and scored using the Fleuther Al Im-eval-harness Minerva math parsing utilities (Gao et al. (024). The final unnormalized answer is first identified and parsed, then the answer is normalized remove units, formatting, etc., and finally equivalence between the model answer and reference answer is deter- mined using Sympy (Meurer al. 2017).
C.2 TRAINING
Experiments are typically executed on 3 NVIDIA H100 8-GPU nodes. One node is used for rollout generation, one for learning. and one for evaluation. Rollouts are generated using two instances of vLLMM (Kwon et al. 2023) servers using 4 GPUs each with tensor parallelism. Custom RI. imple- mentation based on FSDP2 (Zhang et al. 2024) used for training.
account for any discrepancies between sampling and training subsystems, the prg-oobbabilitis of rollout tokens are recalculated on the training node to ensure accurate importance weights for backpropagation. Cut-Cross- Cut-Cross-Entropy (CCE) is used to reduce the memory footprint during training by preventing the materialization of all logits except the target (Wijmans et al. 2025). Models are fine-tuned with LoRA (rank = 16, 32) the self-attention (key, value, query, output) and MLP modules (Hu et al. 2022). We use an Adamä optimizer with a constant learning rate of x 10-5, we wht ght-decayy = 0.01, and gradi- clipping with max-norm
0.1. To speed up rollout collection, criteria. Once at least 4/6 rollouts per task and 90% of total rollouts are collected, we immediately proceed training prevent bottlenecks caused by very few extra long generations (Wijmans al. 2020
introduce early stopping
Chen et al. 2025a).
C.3
SUMMARY OF RESULTS
This section provides additional analysis of the performance of different algorithmic variants, see Tabs. To
Each row the table corresponds three independent training runs. REPO-R is implemented on top of GRPO unless stated otherwise (since GRPO experiences significant exploration collapse without entropy regulation).
Entropy bonus baseline is implemented on top of GRPO with the adaptive coefficient described in $5.1 Due to higher memory usage, these experiments with Qwen-3 3-an-33325 32B required B200 training nodes.
For AppWorld, in each run we find the checkpoint with the highest validation score (based on dev dataset split) and report the mean scores test splits across independent seeds, standard deviation, as well as the best score on particular test split across all seeds. For AIME we simply pick the checkpoint with the highest average test score between AIME '24 and '25 since setup lacks separate validation split. The last column reports the change in entropy (absolute and relative) for the best checkpoint compared to the initial policy.
Tabs. [ and 2 show strong performance of (DAPO and GRPO respectively). as well as more stable entropy dynamics.
methods compared to basclines
25

---

Published as a conference paper at ICLR 2026
On AIME (Tabs. and ADAPO and REPO-R perform competitively with other off-policy meth- ods, however the results are very close between all algorithms and the peak performance is unlocked very early in training (Fig. (6. This suggests that the base models might already be overfit to this benchmark. To make the task harder and to make the training dynamics more interesting, we limit the maximum allowed context length 4096 tokens, requiring the models learn more efficient and compact reasoning.
With numerical improvements described in Apps. B.2 B.4 on-policy RLOO performs remark- ably well in all our tests despite slower progress in early training iterations. We reach 79% success rate Test Normal and 71% on Test Challenge. significantly exceeding the highest reported scores dev/Leaderboard. achieved with an agen-
at the time of submission (https: /ttps:///ppwolddd
tic GPT-4.1 --based system (Marreed et al. 2025).
Algorithm
Test Normal
Best TN
Test Challenge
Best TC
AH
RLOO
0.53 a 0.05
0.64
0.32 0.05
0.40
+0.01 (+4%)
GRPO
0.46 0.01
0.46
0.21 0.02
0.23
-0.09 (-64%)
LOOP
0.40 = 0.02
0.42
0.19 0.02
0.22
-0.08 (-53%)
GSPO
0.56 0.04
0.60
0.32 0.03
0.36
+0.00 (+2%)
DAPO
0.51 = 0.03
0,55
0.28 = 0.01
0.29
+0.06 (+40%)
GRPO + H. bonus
0.48 0.03
0.52
0.26 0.02
0.28
-0,01 (-6%)
ADAPO
0.53 = 0.02
0.57
0.30 0.02
0.33
-0.00 (-2%)
REPO-R
0.49 0.01
0,49
0.25 0.02
0.27
-0.01 (-9%)
Table Task goal completion scores for AppWorld with Qwen-3-8B by training algorithm.
Algorithm
Test Normal
Best TN
Test Challenge
Best TC
AH
RLOO
0.77 0.02
0.79
0.61 0.06
0.71
-0.18 (-36%)
GRPO
0.67 + 0.03
0.71
0.46 + 0.03
0.49
-0.29 (-57%)
LOOP
0.66 0.02
0.68
0.45 0.03
0.47
-0.31 (-60%)
GSPO
0.69 0.01
0.70
0.51 + 0.00
0.51
-0.07 (-14%)
DAPO
0.73 0.04
0.77
0.52 0.02
0.55
+1.52 (+298%)
GRPO + H bonus
0.71 + 0.02
0.74
0.49 + 0.02
0.51
+0.14 (+27%)
ADAPO
0.78 0.02
0.82
0.58 0.03
0.62
+0.52 (+102%)
REPO-R
0.73 0.04
0.78
0.54
0.63
+0.03 (+7%)
Table 2: Task goal completion scores for AppWorld with Qwen-3 3-2BB by training algorithm.
Algorithm
AIME 2024
Best '24
AIME 2025
Best '25
AH
RLOO
0.47 0.01
0.48
0.33 0.01
0.34
-0.04 (-29%)
GRPO
0,45 0.02
0.47
0.32 0.02
0.35
-0.07 (-47%)
GSPO
0.43 0.03
0.47
0.34 #001
0.34
+0.05 (+37%)
DAPO
0.43 0.01
0.44
0.32 0.02
0.33
+0.23 (+158%)
ADAPO
0.43 0.00
0.43
0.34 0.02
0.36
+0.13 (+93%)
REPO-R
0.45 0.01
0.46
0.34 0.01
0.35
+0.13 (+91%)
Table 3: Scores for AIME with Qwen-3-88 by training algorithm.
The model limited to 4K
context length.
26

---

Published as a conference paper at ICLR 2026
Algorithm
AIME 2024
Best '24
AIME 2025
Best '25
AH
RLOO
0.53 0.01
0.55
0.39 0.01
0.40
+0.04 (+28%)
GRPO
0.51
0.52
0.36 0.02
0.39
+0.06 (+40%)
DAPO
0.43 0.01
0.44
0.31 0.03
0.35
+0.75 (+519%)
ADAPO REPO-R
0,46 0.03
0.50
0.34 0.01
0.35
+0,29 (+198%)
0.49 0.02
0.51
0.35 0.02
0.38
+0.17 (+117%)
Table 4: Scores for AIME with Qwen-3-325 by training algorithm. The model is limited 4K context length.
27

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RESULTS
D.1
GEOMETRIC INTERPRETATION OF REPO
REPO-D, Ç=
REPO-R, 0=0.1
1.0
Advantage AKEPO (s,
0.5
0.0
11
-0.5
-1.0
-15.0
-12.5
-10.0
-5.0 Token log Rola
-7.5
-2.5
0.0
-15.0 -12.5 -10.0
-7.5 Token nola (
-5.0
-2.5
0.0
Figure 9: The REPO transformation rotates log a) pairs, promoting low probability actions. Original unmodified advantages shown in blue, advantages ARePO-D shown in green and AREPO-R in red. This plot demonstrates the transformation in a Qwen 3- 323 AIME experiment, showcasing the real distribution of the data: (A, log T) pairs highly concentrated near log 0.0.
The transformation induced by REPO algorithm can be viewed in Fig. REPO-D reflects a con- sistent rotation across the space, boosting the advantages of actions proportional to their surprisals (- log To). REPO-R scales advantages proportionally not only to the surprisal, but to the magni- tude of the advantage. Intuitively, this strongly reinforces low-probabilityy correct actions, especially when they yield outcomes significantly better than average for a given batch of experience.
Fig. uses data from the Qwen - 3-32B AIME experiment. The parameters of the algorithm revealed in the structure of the data: there are 5 distinct positive and negative advantage values, corresponding to 5 unique outcomes of group-basee soup- advantage estimation success failures, successes / 4 failures, etc.). Groups with zero advantages are filtered out.
With the appropriate value of Ç. REPO-D transformation counteracts the covariance-like term in AH approximation, therefore REPO-D is short for REPO-Decorrelate. REPO-R is shorthand for REPO-Rescale, as it simply rescales the advantages by multiplicatiie factor Ç L(s,a).
D.2 REPO-R: DERIVATION AND IMPLEMENTATION DETAILS
We derive the practical REPO-R formula from the general REPO advantage and discuss the simpli- fications made in practice.
Derivation.
Starting from the general REPO advantage ($5.1) AREPO(s, a) = A(s, a) - L(s,a),
PREPO-R REPO-R sets a,a
Ç IA(s, a)l, yielding:
AREPO-R a) = A(s, a) - (() L(s.a).
For positive advantages (A 0), this simplifies to:
A A (8, = A(s, a) (1 - I(s.a))
Since L(s. a) log (a log . s), this boosts rare correct actions (which have more negative L) and attenuates common ones when is positive. Negative value Ç reverses the effect punishing exceedingly rare tokens.
28

---

Published as a conference paper at ICLR 2026
For negative advantages (A < 0), we have:
AREPO-((..) = A(s, (1+ L(s,a)))
Because L(s, a) is negative for rare actions and positive for common ones, the factor (1 + S. L) decreases penalties rare incorrect actions and strengthens penalties common incorrect actions (for >>>>>>>> 0).
Practical simplification: using log mg instead of centered L. The theoretically motivated for- mulation uses the centered log-pree oog-prbbability L(s, a) = log To (a s) E ( flog To s)]- In practice we replace as with the raw log prop-rbbbbblity log T (a s). An implementation us- ing L(s, a) would require recomputing log πe(- 8) over the entire dataset of trajectories after each gradient step, which computationally expensive, and omitting it has negligible effect in practice.
Reference implementation. The following code listing shows the REPO-R advantage rescaling and the adaptive controller used our experiments.
Adaptive controller (once per iteration) if current_entropy > target_entropy: if zeta >= 0:
zeta /= 2.0
if zeta zeta zeta_min:
zeta -zeta_min flip the sign if needed
else:
zeta max (- zeta_max, zeta * 2)
elif current_entrop? < target_entropy:
10
if zeta >= 0:
zeta min (zeta max, zeta 2)
12
else:
zeta /= 2
14
if zeta > -zeta_min: zeta zeta_min flip the sign
15
Bidirectional REPO-R (per token)
loop new_logp detach () # latest model 's logprobs + stop grad
if adv. ite.(tem)) () 0:
adv = adv (1 zeta logp)
adv adv. 1ataammmminnO00
elif adv. ddv.item( () 0:
adv adv adv = adv. (0..cll0p_mxx
+ zeta logp)
29

---

Published as a conference paper at ICLR 2026
E QWEN 2.5 EXPERIMENTS
In previous publications, methods like LOOP performed well with "non-thinking" models such as Qwen 2.5 32B (Chen al. 2025a). In Qwen3 experiments however, LOOP (and very similarly, GRPO) experienced early entropy collapse and underperformed compared to other RL methods.
We conducted additional experiments (see Fig. and Tab. 5 to determine whether this discrepancy arises from differences in model behavior or from implementation details. Key observations:
Qwen 2.5 32B exhibits a significantly higher initial success rate (before the first training iteration) compared to Qwen models. For example, on Test Normal the initial success rate is close to 40% versus under 10% for Qwen 3. We attribute this to Qwen3's tendency to produce excessively verbose thinking blocks which hinders actual progress on the task.
The best results on the hardest test split (Test Challenge) are substantially lower for Qwen 2.5 compared Qwen most likely reflecting the limitations the respective base mod- els.
We were able to replicate and exceed results reported in previous work for Qwen 2.5 32B: the success rate of our best-performing LOOP checkpoints surpasses those in Chen et al. (2025a) by approximately 7% on Test Normal and 9% on Test Challenge. This improve- ment most likely attributable to the numerical changes described in App. B as our setup and hyperparameters for Qwen 2.5 closely match those in Chen et al. (025a in all other respects.
Unlike in our Qwen 3 experiments, LOOP/GRPO do not experience rapid entropy col- lapse, whereas RLOO does, suggesting that base-model characteristics play major role in entropy dynamics during training irrespective of the RL algorithm.
0.60
1.00
Test Normal TGC
0.75
Test Challenge TGC
Entropy N
0.60
RLOO
0.50
LOOP
0.30
DAPO
0.25
0.30
0.00
0
50
100
150
50
100
150
50
100
150
Iterations
Figure 10: Qwen 2.5 32B test performance and token entropy on App vs. training iterations.
Curves show mean across three independent seeds for each algorithm.
Algorithm
Test Normal
Best TN
Test Challenge
Best TC
RLOO
0.72
0.78
0.47
0.50
LOOP
0.75
0.78
0.50
0.54
DAPO
0.74
0.77
0.50
0.56
ADAPO
0.73
0.78
0.51
0.59
Table 5: Task-goal completion scores for AppWorld Qwen 2.. each test split, we report the best average score across three seeds and the highest score among all seeds and training iterations.
5-3BB by training algorithm. For 5-
30


Reviews:
[{"review_id": "3hJY7RwQU5", "reviewer": "Reviewer_KAD6", "summary": "This paper investigates the entropy dynamics of existing policy gradient algorithms for LLM post-training, then proposes a method for preventing entropy collapse (REPO). Figure 1 shows the trajectories of training in terms of Test Accuracy vs entropy, which seemed like an interesting visualization method. The results showed that low entropy trajectories tend to have low final test accuracy. In terms of theoretical results, they present the change in entropy based on 1 policy gradient step, when applying REINFORCE (this seems mostly like an extension of Cui et al. 2025).\nREPO is motivated by their derivation of the change in entropy per step, and adds a weighted term (beta*(logpi - E[logpi])) to the policy gradient advantage to counteract the drop in entropy per step.\nThey add two heuristic variants for tuning beta to allow for greater robustness: (REPO-D) This sets beta to zeta * \\Delta H, where \\Delta H is the computed change in entropy from a gradient step. Also, zeta is clipped between a max and min value, and it is adaptively tuned to reach a target policy entropy, similarly as done in the original PPO paper (they double or halve the zeta constant depending on which direction to move to reach the target entropy). (REPO-R) This variant instead uses \\beta = -\\zeta * max(A, 0), i.e., it multiplies by the positive advantage, and sets negative advantages to 0 (the motivation being to prevent converging to large advantage values while still penalizing wrong actions).\n\nExperimentally, they compare against REINFORCE leave-one-out (RLOO), GRPO (a normalized PPO variant), LOOP (a PPO variant with leave-one out return estimation), DAPO (LOOP with asymmetric weight clipping), GSPO (variant of LOOP with with trajectory based clipping). They test using a Qwen3 (8B and 32B) base models on the AppWorld tasks as well as on AIME 24 and 25 math tasks and achieved good performance (a bit better than RLOO on the 32B App tasks, but maybe slightly worse or similar on 8B tasks). For REPO, they also had a DAPO variant as well as a GSPO variant (the former worked better on 32B tasks, the latter on 8B tasks). REPO-R seemed to always be better than REPO-D.", "questions": "To me, the major conceptual issue is that the method appears very similar to an entropy bonus, but just with the gradients estimated in a noisy way. Perhaps there is some advantage to this as perhaps the change in entropy can be coupled to sampled action, and this may alter the entropy dynamics, which may be beneficial, but this is not discussed in the paper. Can you elaborate on why your method would be beneficial over a simple entropy bonus?\n\nMy experimental concerns listed in the weaknesses are also a major concern for me. One way to mitigate this would be to run experiments using Qwen2.5, as this allows direct comparison with existing published results (in particular, your results for LOOP with Qwen3.0 are weaker than published results with Qwen2.5).\n\nAnother small comment is that your derivations appear closely related to Cui et al. 2025, but it appears cited after you introduce your results. It may be better to cite it before, and make the connection clear.\n\nAnother small comment I have is about the phrase in the abstract: “As we show in this paper, most policy gradient algorithms naturally reduce entropy”. However, this is a well-known issue, so I think it may be better to present it as such.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I liked the entropy trajectory visualization.\n\n- The performance is slightly better than the current published SOTA on these tasks (LOOP)", "weaknesses": "- One major point for me was a discrepancy between the results in the paper and previous published LOOP results (this discrepancy likely comes from the previous results being with Qwen2.5, while the current results being with Qwen3.0).\n\nThe existing LOOP scores can be seen in the below links:\nhttps://appworld.dev/leaderboard\nhttps://arxiv.org/pdf/2502.01600\n\nThese results are with Qwen2.5 (instead of Qwen3 like in the current paper), and it achieves results of 71.3 and 45.7 compared to the results in the current paper (64 and 40). Moreover, their performance is better than the results they present with RLOO. It seems unlikely that the performance drops that much when switching from Qwen2.5 to Qwen3, so I would guess that there may be implementation issues in LOOP. (for reference the RLOO scores in the LOOP paper are 57.2 and 36.7 for the normal and challenge tasks respectively, and switching to the Qwen3 model in the current paper lead to 71 and 52 for RLOO).\n\nOn the AIME tasks, there is also no consistent improvement and RLOO seems to work reasonably well in the current paper.\n\n- Another major point for me was a conceptual point about the method. The REPO method as presented on line 227, adds the log pi term to the advantage, and if we take the expectation pi log pi, this is just the negative entropy, so the method appears to be simply adding an entropy bonus, which is a well-known technique. Different to past implementations is that they do not directly compute the derivative of the entropy, but instead perform a REINFORCE estimate of the gradient of the entropy (the subtraction of the average log pi value can also be interpreted as the baseline); whether this is a good or bad thing remained unclear to me. Perhaps it induces different learning dynamics than simply directly computing the exact entropy and adding that as a bonus, but these points were not examined.\n\nTo make clear what I mean by this, consider the derivation: E_pi[dlogpi/dtheta * (logpi - E(logpi))] = E_pi[dlogpi/dtheta * (logpi)] = d/dtheta E_pi[logpi] = d\\dtheta H(pi), is the derivative of the negative entropy.\n\nThey did actually add a comparison with a pure entropy bonus based method in a similar fashion as their REPO, however, I had some concerns about this. This was done on the Qwen8B task, and they added the entropy to the DAPO method. However, on 8B, GSPO-REPO performed better, so adding an entropy bonus to that would have been more convincing to me. \n\n- The added tuning methods did not seem to have a strong theoretical justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "osFCqB7Hwn", "reviewer": "Reviewer_k7r1", "summary": "This paper tackles the problem of \"entropy collapse\" in reinforcement learning , where policy gradient algorithms, used to train language models, naturally reduce the diversity of explored solutions during training. This premature narrowing of focus can limit performance by causing the model to get stuck in local optima. The authors analyze how different algorithms either amplify this collapse (like PPO) or implicitly mitigate it (like DAPO and GSPO). They then propose a new method, REPO (Regulated Entropy Policy Optimization), which uses an adaptive controller to actively monitor and stabilize entropy throughout training. Experiments show that models trained with REPO preserve entropy, achieve higher average performance, and, crucially, can be effectively re-trained on new tasks in novel environments, unlike models that have suffered entropy collapse.", "questions": "The performance difference of proposed algorithms and RLOO does not seem to be statistically significant. Maybe I am simply missing, but would you tell me what $\\pm$ means in tables?\n\nAs I wrote in Weakness, RLOO does not seem to have any implicit mechanism to avoid entropy collapse and does not really align with the analysis in the paper. Would you explain why?\n\nIn the proof of Theorem 2, I do not really understand why clipping of the importance ratio ensures that no policy gradient update is performed if the policy drifts outside a trust region as in Line 200. Due to function approximation error, $(1+\\epsilon) \\pi_\\theta^{old} (a|s) < \\pi_\\theta^{new} (a|s)$ can occur if one really updates policies according to Line 130.\n\nI do not really understand the description on the entropy dynamics of GSPO. The trajectory length is also dependent on a policy, and the current proof of Theorem 2 cannot be straightforwardly extended. Would you add more explanation on this paragraph?\n\nIn Line 191, I think \"Corollaries 1 and 2\" is a typo, and it must be \"Theorem 1 and Corollary 2\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper proposes an interesting idea to analyze how the entropy evolves during RL finetuning. The writing is clear and easy to follow (especially Section 3).", "weaknesses": "The performance difference of proposed algorithms (REPO-R and REPO-D) and RLOO does not seem to be statistically significant, making me wonder whether the proposed method is useful. (Even if the difference is statistically significant, the magnitude of the difference seems to be small.) Furthermore, while RLOO has no mechanism to keep early collapse according the proposed analysis, its final entropy is somehow high. This also makes me wonder whether the analysis really explains how entropy evolves and consequently allows us to derive a better algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZcKW3BoYT6", "reviewer": "Reviewer_CBf6", "summary": "The paper studies entropy behavior in policy gradient optimization within the context of large language model (LLM) reasoning tasks. It identifies the issue of entropy collapse during training and proposes a regularization approach called Regulated Entropy Policy Optimization (REPO), which adaptively reweights advantages and log-probabilities to maintain stable entropy. The method is evaluated on reasoning benchmarks such as AppWorld and AIME 2024, showing stable entropy dynamics and comparable or slightly improved performance over existing RL algorithms. The paper provides theoretical insights into entropy modulation in policy gradients and discusses which RL variants naturally preserve or collapse entropy during optimization.", "questions": "**Detailed Review:**\n\nThe paper presents a focused and well-motivated analysis of entropy dynamics in reinforcement learning for LLM reasoning. It highlights how entropy often collapses prematurely in standard policy gradient methods, leading to limited exploration and suboptimal convergence. The proposed Regulated Entropy Policy Optimization (REPO) method adaptively scales the advantage term during optimization to counteract entropy collapse and maintain stable training dynamics.\n\nThe theoretical framing is coherent and connects policy gradient updates with entropy evolution. The resulting algorithm is straightforward and practical, which makes it potentially useful for a wide range of RLHF applications. However, it would be beneficial to include a more detailed comparison to standard entropy regularization methods, such as those used in actor-critic frameworks, to clarify conceptual and empirical differences.\n\nExperimental results demonstrate that REPO stabilizes entropy without harming model performance. The reported trends across benchmarks such as AppWorld and AIME 2024 show promising consistency. Nevertheless, the magnitude of improvement is relatively small, and the lack of variance across multiple seeds limits confidence in the statistical strength of the findings. Clarifying how many independent runs were performed and whether results are averaged would help substantiate the claims.\n\nSome figures (2, 3) would benefit from normalization, as differing y-axis ranges obscure direct comparisons. The discussion of entropy per token (Figure 1) is interesting but requires a clearer explanation of its computation and interpretation, especially when entropy appears to collapse to a single value. Likewise, the differing entropy behaviors of RLOO and REPO across Figures 2 and 3 require deeper discussion, particularly regarding model size and task difficulty.\n\nFrom a methodological standpoint, the paper’s simplicity is an advantage. The adaptive scaling via $\\beta$ provides a structured mechanism to control entropy without introducing new objectives. However, a broader range of experiments, including other model families and additional reasoning benchmarks, would improve the generality of the results. The motivation of Section 5.3 could also be clarified, especially regarding whether it demonstrates task transferability or robustness.\n\n\n**Questions:**\n\n1. How many independent runs were conducted for each experiment, and can variance or confidence intervals be reported?\n2. Why are the y-axis scales inconsistent across subplots in Figure 1 (and 2, 3), and could they be standardized for clearer comparison?\n3. How is the per-token entropy computed, and why does it appear to converge to a specific point in certain figures?\n4. Why does RLOO show opposite entropy trends between Figures 2 and 3, and why do the baseline algorithms differ across these figures?\n5. How significant are the reported performance differences, and were any statistical tests conducted?\n6. Can REPO integrate with PPO or other policy gradient methods, and how would that adaptation look in practice?\n7. How is the entropy bonus (lines 372–373) computed, and how is its scale selected?\n8. What is the motivation behind Section 5.3, and how does it relate to the core objective of entropy regulation and generalization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clearly identifies and analyzes the problem of entropy collapse in policy gradient methods for LLM reasoning.\n- Introduces a simple and interpretable approach (REPO) to stabilize entropy during training.\n- Provides theoretical insight into how different RL algorithms modulate entropy.\n- Demonstrates that REPO maintains stability without degrading baseline performance.\n- Presents the paper with strong structure, clear notation, and consistent motivation.", "weaknesses": "- Lack of variance reporting or multiple-seed averaging, making statistical reliability of results unclear.\n- Performance improvements are modest, and their statistical significance is not demonstrated.\n- Evaluation scope is limited, focusing mainly on two model sizes within a single model family.\n- Distinction between REPO and conventional entropy regularization is not well articulated, leaving the novelty somewhat unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1VlPgbn0i", "reviewer": "Reviewer_CecX", "summary": "The authors present REPO, a 'novel' on-policy RL algorithm for LLM fine tuning that adds an adaptive entropy penalty to ensure the policy entropy is maintained at a desired (initial) level. They provide some theoretical analysis of the entropy behaviour of existing RL for LLMs algorithms, and propose an adaptive entropy regulariser that tries to keep the entropy of the policy at an initial desired value. They then present a set of empirical results on the AppWorld and AIME benchmarks, showing how REPO sustains higher entropy than existing algorithms and results in slightly better benchmark performance.", "questions": "1. What is the reason for trying to keep entropy at the same level as the base (before post-training) level? Is there a reason to believe that the models have the 'appropriate amount' of entropy after pre-training?\n2. I would like to further understand the observation the authors make in lines 46-53. In general, entropy in RL policies is good during training to ensure exploration and to avoid collapsing to local minima too fast. It is not clear to me that sustained entropy is desirable in general, all the way through the resulting policy after post-training. In all tasks with verifiable rewards, one could argue that a model that collapses to a zero entropy policy that always achieves maximum reward for any query is optimal in an RL sense. What is the case for learning language models with high entropy policies? \n3. I do not fully agree with the motivation behind REPO-R. Biasing the regularisation towards positive advantage actions seems quite myopic: I would hypothesise that you observe positive results because, in essence, the complexity of the tasks (and thus the amount of exploration required) is low. Another reason for this is that the RL algorithms considered have a big simplifying assumption compared to classic RL algorithms: rewards are episodic and there is no need for constructing value function estimates. In general, an agent may need to explore bad actions for a while (or, equivalently, good actions may seem bad due to bad value estimates) to find good solutions in the long term, so discouraging negative advantage actions early on may cause catastrophic failure in most general RL settings. It also does not seem to add much improvement when compared to REPO-D, so I would appreciate if the authors would further motivate this decision and their assessment that REPO-R is a good idea.\n4. Theorem 1 is not stated in a formally correct way. A\\approx B is not a logically falsifiable claim and is ambiguous, as I could say 1\\approx 2 or 1\\approx 10 depending on the order of magnitude of the approximation error.\n5. Similarly, corollary 1 states 'approximately' and then formulates a 'proportional to' relation. Is it an approximation, or a proportional to?\n6. I don't know if I would formulate Theorem 2 as a theorem, as it suggests that it somehow presents a novel theoretical contribution of significant weight (perhaps a corollary of Gibbs inequality + the bounded ratio assumption). If the distance between two probability distributions is bounded, it follows from standard arguments that the entropy change is bounded (this is a minor point and a matter of personal preference).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problem of encouraging exploration efficiently in RL (and in LLMs) is very relevant and pressing. \n- The analysis the authors carry out on how different algorithms affect entropy is interesting.\n- The paper and its objectives are clear.", "weaknesses": "- I am not convinced of the novelty or the contribution weight of the work. Entropy regularisation in RL is a very well studied problem, with MaxEnt RL being almost a subfield on its own. While the analysis of the effects of each algorithm in entropy are interesting, it is perhaps unsurprising that 1) algorithms push entropy to decrease fast (since policy gradient algorithms push the policy to be deterministic: this is optimal from an MDP perspective) and 2) that entropy regularisation helps with exploration and with preventing collapse to local minima (since this is essentially the main argument behind MaxEnt RL).\n- The idea of having an adaptive coefficient to sustain a desired level of policy entropy is also not novel in itself [1] (which is not cited in the paper).\n- Some of the mathematical statements are superficial or formally imprecise.\n\n\n[1] Haarnoja, Tuomas, et al. \"Soft actor-critic algorithms and applications.\" arXiv preprint arXiv:1812.05905 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)