On Entropy Control in LLM-RL Algorithms
Authors: Han Shen
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
LLM
Primary area: reinforcement learning

Abstract: For RL algorithms, appropriate entropy control is crucial to their effectiveness. To control the policy entropy, a commonly used method is entropy regularization, which is adopted in various popular RL algorithms including PPO, SAC and A3C. Although entropy regularization proves effective in robotic and games RL conventionally, studies found that it gives weak to no gains in LLM-RL training. In this work, we study the issues of entropy bonus in LLM-RL setting. Specifically, we first argue that the conventional entropy regularization suffers from the LLM's extremely large response space and the sparsity of the optimal outputs. As a remedy, we propose AEnt, an entropy control method that utilizes a new clamped entropy bonus with an automatically adjusted coefficient. The clamped entropy is evaluated with the re-normalized policy defined on certain smaller token space, which encourages exploration within a more compact response set. 
In addition, the algorithm automatically adjusts entropy coefficient according to the clamped entropy value, effectively controlling the entropy-induced bias while leveraging the entropy's benefits. AEnt is tested in math-reasoning tasks under different base models and datasets, and it is observed that AEnt outperforms the baselines consistently across multiple benchmarks.

Published as a conference paper at ICLR 2026
ON ENTROPY CONTROL IN LLM-RL ALGORITHMMS
Han Shen Ant Group
shenhanhs@gmiil com
ABSTRACT
For RL algorithms, appropriate entropy control is crucial to their effectiveness. To control the policy entropy, commonly used method is entropy regularization, which is adopted in various popular RL algorithms including PPO, SAC and A3C. Although entropy regularization proves effective in robotic and games RI. conventionally, studies found that it gives weak to no gains in LLM-RL training. In this work, we study the issues of entropy bonus in LLM-RL setting. Specifically, we first argue that the conventional entropy regularization suffers from the LLM's extremely large response space and the sparsity of the optimal outputs. As a remedy, we propose AEnt, entropy control method that utilizes entropy bonus with an automatically adjusted coefficient. The clamped entropy is evaluated with the re-normalized policy defined on certain smaller token space, which encourages exploration within a more compact response set. In addition, the algorithm automatically adjusts entropy coefficient according to the clamped entropy value, efffectively controlling the putropy-induced bias while leveraging the entropy's benefits. AEnt is tested in math-reasoning tasks under different base models and datasets, and it is observed that AEnt outperforms the baselines consistently across multiple benchmarks.
clamped
INTRODUCTION
RL seeks to maximize the reward received by sequential decision making system. In recent years, RL has proven to be an effective tool for training LLMs (Yang al.) [2025] DeepSeeeeS [025] Comanici et al. 2025). The advances of L.LMs in math, coding and planning tasks has been astonishing, with their performance on competitive benchmarks drastically increasing after RI. training.
The methods used in LLM-RL are predominantly policy-gradient based, e.g., the PPO (Schulman jtt al. [201)) family. Policy gradient based methods reinforce the sampled actions that lead to higher rewards compared to other sampled actions. However, when the optimal actions are not sampled, the policy gradient methods can over-reinforce the sampled locally optimal actions, ultimately resulting in the policy stuck at suboptimal points (Agarwal et al. 2021 The sub-optimal actions can be meaningless deep RL and oftentimes have a large performance gap from the optimal ones (Henderson al. 2018), e.g., in L.M tasks, the policy can be stuck producing the correct format but incorrect resu ts. A straightforward remedy for the issue was the so-called entropy-regularize RI. methods (Williams & Peng 1991]. where the policy maximizes sum of rewards and some entropy bonus (policy randomness). This technique was commonly used in policy-gradient methods including A3C (Mnih al. 2016), PPO (Schulman al. 2017) and SAC (Haarnoja al. 2018), providing strong benefits in tasks requiring hierarchical behaviors. Intuitively, the entropy bonus keeps the policy random and explorative, thus prevents the policy from over-reinforcing certain actions and getting stuck, Moreover, entropy regularization is shown to provide strong optimization benefits both empirically (Ahmed al. 2019) and theoretically (Mei et al. 2020 Klein et al. 2023).
However, it is observed that entropy regularization offers little gains in LLM-RL training. Specifically, the experimental results to be shown Section 5 suggest that regulaizre GRPO yields minimal gain compared to basic GRPO. In addition, Cui et al. (2025) observes that the validation accuracy is unchanged under different scaling of the entropy bonus in some LLM- M-tahhh tasks. These results particularly underwhelming compared to those in other deep RL tasks including robotics and games, where the benefit of entropy bonus is significant (see, e.g., (Haarnoja et al.) 2018 Figure 3)). Moreover, such empirical contradiction also indicates theoretical gap between the existing

---

Published as a conference paper at ICLR 2026
analysis which justifies the entropy's 's benefit (Mei et al. 2020) and its effect in LLM-RR. Therefore, a careful study and a remedy for this issue is in dire needs, as the potential gain from entropy bonus is yet to be unlocked for LLM training.
In this work, we first give a theoretical view of the entropy effect in LLM-RL training, which explains the conventional entropy bonus's ' emergent issues in LLM tasks. To that end, we then propose AEnt, an entropy regularization method that uses an adaptive and clamped entropy bonus. Our main contribution is twofold:
A theory on the entropy effect and its issues in LLM-RL. Under no entropy bonus, we show that entropy collapse indicates learning stagnancy and give performance bound. Then we show that entropy regularization can fail to improve this result under LLM's large response space and the task's s sparse optimality.
AEnt, a recipe to enable effective entropy regularization. Inspired by the theoretical analysis, we then propose a recipe for this issue. Instead of using the traditional entropy bonus, AEnt uses clamped entropy defined with the re-normalized LLM policy on size-reduced token space. The clamped entropy only smooths out policy on the reasonable responses set, which enjoys decreased bias compared to the original entropy. Furthermore, the clamped entropy bonus is scaled with a coefficient that gets automatically adjusted to balance its bias and benefits. Empirical evidence suggests that AEnt consistently improves over the baselines across multiple benchmarks.
1.1 RELATED WORKS
Policy-gradien based LLM-RL algorithms. The RL algorithms used in LLM post-training have been predominantly policy-gradient based (Sutton et al., 1999). They are either based on PPO (Schulman al., 2017) (see, e.g. GRPO (Shao al. 2024). DAPO (Yu et al. 2025) and (Fu et al. 2025)), or the more basic REINFORCE algorithms Williams 1992) (see, e.g., (Ahmadian et al. 2024. Chu al. )) Though PPO was initially proposed in actor-critic style, the critic is replaced with Monte-Carlo rollout in resource-limited or outcome-drive -riven LLM training scenarios. Contrary to the practice in robotic and games RL (Mnih al. 2016: Schulman et al. 2017), the fore-mentioned
LLM-RL algorithms do not consider entropy regularization.
Entropy regularization in RL. regloiy- RL was initially introduced (Williams & Peng 1991). It has been commonly used in various popular policy-based deep RL algorithms (Mnih et al.
2016 Schulman et al. 2017; Haarnoja et al. 2018) which have provided ample empirical evidence for its effecttivnesss in robotic and games tasks. Entropy regularization' 's optimization benefits have also been empirically (Ahmed et al. 2019) and theoretically (Mei et al. 2020 Klein et al. 2023) studied. However, it does give notable performance gains for LLMs (see, e.g., Section 5 and (Cui
jet al. 2025)). As a result, alternative entropy control techniques are often adopted. In (Zhang et al.) 2024) reshapes the reward function regulate the policy. Or in a concurrent work (Cui et al. 2025). the algorithm clips or regulates the parts the policy update that decrease entropy too much. To our best knowledge, existing works do not answer the question of why and when entropy regularization can fail in LLM-RL, and have not uncovered the potential benefits of entropy bonus.
2 PRELIMINARIES
In this section, we will first give formal definitions of some RI. concepts, and then introduce several prominent policy optimization algorithms.
Finite-horizon Markoy decision process (MDP). In LLM-RL setting, the learning task can be modeled as finite-horizon MDP defined by a M = (S, A, HJ,, where S is a finite state space (e.g., input token sequence of the LLM), A is finite action space (e.g.. LLM's vocabulary), and the state transits by $t+1 = P(st, a.) where P is a concatenation operation of the input sequence st and the output token a, Function r(s, a) E [o. 1] assigns a reward to (s, a). Horizon H is the max response length. An LLM LLM-ccy parameterized by 9 E IRd is denoted as To (als), which assigns a probability for each token A given input S.
2

---

Published as a conference paper at ICLR 2026
RL objectives. Given the initial time step h and state sn define the cumulative reward as
H-1
(8)
B
=
(2.1)
where no(ss) = no (-)ss, the expectation is taken over the trajectory (at, st+1...., aH=1) where at To (5:) for each t. Given dataset DD containing input queries, the objective of RL
H-1
max V (D) EEND V (s) E
Σ
rrst, a.)
(2.2)
B
where 47 (s) V/0h0 (s), and omit time step subscripts for the value functions of step 0.
Entropy regularized RL. Given D, we can define the entropy of the policy xo as
H-1 log ne (acls²)
H(TT)) = Exp
(2.3)
Entropy regularized RL aims to maximize the entropy regularized objective (D) = VD) XH((ma). Due to space limitation, we defer some definitions to Appendix A.1
PPO-clip family. To solve for 2.2 prominent algorithm is the PPO-clip (Schulman et al. 2017).
Given the sampling policy Tb, the objectives of PPO-clip algorithms can be written as
(sc AG, clip (a)(as))))
(a (st) 1- flow: 1 1+ A.)
LPo (0) Baoon (ooom min
(2.4)
Tb(aalas)"
where A, is an estimate of the advantage function. GRPO uses Monte-Cario estimate of the trajectory-level advantage. DAPO additionally decouples the clip ratio by setting different flow thigh and incorporates extra sampling constraints and overlong response penalty. Given suitable clip range, the PPO-clip algorithm can be viewed as a policy gradient algorithm (Jin et al. 2023).
3 A THEORY ON ENTROPY EFFECT IN POLICY GRADIENT BASED LLM-RL
In this section, we give some theoretical insights into L.M-RL training. We will show performance bounds for RL algorithms without entropy control or with conventional entropy control. We will also draw connections with some concurrent works based on our thcoretical insights. Suppose the LLM is softmax policy, that is
exp(O..a)
To (als) = Σ. cxp(Os.a)
where 0,.a is the logit of token a given input s. The policy gradient based algorithms without entropy regularization, including PPO-clip, are generally guaranteed to converge to an e-stationary point of the RL objective 4TH (D) satisfying (VV*((D) <e (Agarwal et al. 2021 Jin et al., 2023). When doing policy optimization without regularization, (Cui et al. 2025) observes that the policy entropy quickly diminishes as performance increases, and ultimately performance saturates when entropy completely collapses. In the following result, we give some theoretical insights into this observation.
Proposition 1 (Bounds under no entropy control). Assume the policy is softmax. We have:
()) Policy entropy is an upper bound of the policy gradient: || (D)] S 2H((a))
(II) V [D)] < 6, then given any query Sn in dataset DD the policy suboptimality on the query satisfies
1
V= (so) - = (so) S
(3.1)
(su)
where T" Earg max, (T(D()))) C (su)
max max 1) (50) 11:-'' no (a.ls))
D
in which (sp) = (agg Qj,..... 11) E A
3x" 1 (a.ls)) > 0} is the set of
all optimal responses given query 80-
3

---

Published as a conference paper at ICLR 2026
Num of hrt.mo---tt actions-15
Num of octimall actions-10
Yu' of opcinal act 01G-1
Reward gain over noveg basel ne
0.0
reg
Entmpy rng.
entropy
2.6
0.6
0.6
Clan pod entropy rog.
3.44
0.4
0.4
Rewerd
0.2
0.1
3.0
1200
2000
3000
4000
0.0
1000
2000 Step
3000
4000
0.0
1000
2000
3000
4000
0.0
Stop
Stop
Number of oodimal action
Figure 1: Test in a controlled MDP with a large action space of size |A| = 10 and increasingly sparse optimal actions.
The first bullet suggests the policy entropy is indicator of the policy stationarity, that is, small entropy indicates a small policy gradient || (D)= and the convergence of the policy. The second bullet (1I) quantifies the actual performance of the almost stationary policy, where the reward optimality gap on query 80 is bounded by Oce/C*o (so)). The factor (*e((s) can be controlled (bounded away from 0) when the initial LLM and the RL algorithm
sufficiently explore the optimal response to 8p. For example, one can either use a large batch size (Klein et al., 2023) or strong initial model (Weissmann et al. 2024) to control C (sa). In this case, the performance is ultimately bounded by O(e). The eтror e decreases with prolonged RL training, while it usually cannot decrease to 0 due to the presence of sampling noise or the advantage estimation error.
On the other hand, the maximum entropy RL optimizes the regoop--reulrrizedd reward sum V (D) (Williams Peng T99T). non-LLM deep RL tasks, this method has long been popular and can significantly outperform methods without entropy control (Mnih et al.) [2016] [Haannoj et al. 220 8). However, experiments (to be shown in Section (5) show that traditional entropy regularization gives weak to no gains in LLM-RL training. In the next result, we give theoretical insight into this issue.
Proposition 2 (Bound for entropy-regularized methods). Assume the policy is softmax. If VVV [D) S then given any query 80, the policy suboptimality on the query satisfies
2
IAI
V (5o) v~00
AHlog
(3.2)
C (su) 2A
A (5o)
where (5)) will be specified in the proof.
Similar conditions to Propositions 21) have been derived in Mei et al. 2020 for the discounted infinite horizon MDPs, while our results hold for the finite horizon MDPs under concatenation transition. Proposition
2 also provides a more accurate bound for the entropy bias.
Entropy regularization suffers from immense response space with sparse optimality in LLM tasks. As compared to no entropy control case in Proposition|/ (II) the above bound's 's dependence on improves Oce² /2X) when C (sp) is bounded away from O. However, this optimization benefit does not come free as a bias term is introduced. The entropy bias is O(H log(IAI/IA4, (s)))) + )), which increases with the response space size H log [A] and the sparsity of optimal responses log(1/| |A (su)l). The bias is especially ubiquitous in LLM-RL, where the response space is typically extremely large (hundreds of thousands tokens to sample from in each step) as compared to that in, e.g., classic control and games where the action space size and horizon are typically at the hundreds (Brockman et al., 2016; Silver et al., 2017). To better demonstrate this effect, we report a numerical test result in Figure The test is done in synthetic MDP with [A] 10S, and the optimal action is sparse with various numbers in {1, 5, 10, 15) (see Appendix A.5 for details). It is observed that entropy regularization leads gains over no-regularization when number of optimal actions is 10, 15, but offers no gains when the optimal action becomes too sparse with fewer than optimal actions.
To unlock the benefit of entropy regularization, it is crucial to mitigate the negative effect caused by the large response space in LLM tasks. In the following sections, we propose our recipe for this issue and empirically demonstrate its effectiveness.
4

---

Published as a conference paper at ICLR 2026
4 AENT: ADAPTIVE ENTROPY REGULARIZATION WITH TOKEN SPACE CLAMPING
In this section, will first introduce the core components of our method and then the full algorithm.
4.1 ENTROPY WITH TOKEN SPACE CLAMPING
Recall the entropy regularized RL objective is V (D) + XH((T)) where
H
H(mT)) =
Σ
B
Σ
To(a|s;) log (a(ls)))
(Entropy)
t=0
aEA
Entropy is maximized by the uniform policy Tuniform 1/|.Al. Maximizing the entropy pulls =
the LLM policy towards Tuniform (alo))maaaa)))) 1 1/ .A, increasing the likelihood of low- low-probability actions while decreasing those of the nigh-probability ones. Intuitively, this helps when the optimal actions have low probabilities and are thus less likely to be sampled and reinforced. Such mechanism works well in the RL tasks where the discrete action space is small (Brockman et al.) 2016). While is extremely inefficient in LLM-RL setting since A is prohibitively immense with sparse optimal tokens. Specifically, when A/| small, pulling To (a))ss for every a E A towards 1 A/| gives weak gains and produces large bias due to the large amount of non-optimal tokens in the complete token space.
To overcome this issue, we instead use a clamped entropy:
H
H((TO) ==
Σ
Σ
no(als;) log (a(()s)))
(Clamped entropy)
(EA((M)
exp (0s,a)
with (a(s) =
EneA(s) exp (0a,a) and A(8) = {top (1 (1-p) percent tokens in To (-())
The clamped entropy is evaluated by re-normalized policy 7o size-reduced, input-dependent token space A(s). By the insights from Proposition Figure [ regularizing on a smaller response space with denser optimality generally leads to reduced bias and larger gains. With this principle, we set A(s) as the the top-prober oop-robbbbility tokens set of (T)s). The intuition is that since the base models are pra-ted or fine-tuned prior to the RL phase, the bottom probability tokens are unlikely to be optimal. We find leaving them out reduces entropy-induced bias and leads to performance gains. It can be observed from the toy demonstration in Figure | that clamped entropy regularization leads to performance gains when entropy regularization does not (number of optimal actions S 5), and is generally more robust to optimality sparsity increase.
4.2 ADAPTIVE CLAMPED ENTROPY CONTROL
For entropy-reeullrizeed RL, constant entropy coeffi- cient À is often sufficient to properly control the policy entropy in robotic and games RI. (Mnih et al. 2016 Haarnoja et al.) 2018). However, we observe in Fig- ure that this assumption does not necessarily hold in LLM-RL training as the entropy can change dras- tically in the mid of training, and the initially chosen coefficient fails. In the example, the entropy stabilizes in the early period, but starts to drastically fluctuates after step 200 while the policy performance saturates. The entropy coefficient is not adjusted to change such a trend and fails to deliver better performance promised by entropy control.
0.64
0.7
0.43
1est 5core
0.5
Entrpy
0.42
0.43
0.40
0.3
0.39
0.2
0.38
Figure 2: GRPO with a constant entropy bonus coefficient.
To alleviate this issue, we automatically adjust the efficient during training following
X - Proj Drow Nanue) [A B min (H((Te)) Hlow 0) + 3 miiminnHaa (Hrigh =((T))) 0)]
(4.1)
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 AEnt: Adaptive entropy regularization with token space clamping
1: Initialize the algorithm, including choosing Hlow, Hnigh and Atow, Anigh, clamping percentage
2: for global step k: to K do
3: Set the sampling policy Tb-
4: Sample batch of so and for each so, batch of (ao, $1, $1,01,.... : -H--1, a==1 following mb.
5: Optimize for the batch surrogate of CA (0: X) w.r.t. 6.
6: Adjust the clamped entropy coefficient À following scheme
7: end for
where B is the coefficient learning rate, and Hlow Hhigh respectively the lower and upper limit of the (clamped) entropy. The algorithm will try to confine H within [How: Huigh] by increas- ing/decreasing À when H((e) is lower/higher than the limits. The intuition is that when entropy is high, the coefficient should be tuned down to reduce the entropy induced bias and shift weights to reward maximization, which in turn consumes entropy. While when entropy level is too low, the coefficient be tuned up leverage the benefits of entropy regularization. For better training stability, the entropy coefficient is also boxed in the range [Atow, Abigh] so that large fluctuations of entropy do not lead to coefficient over-shoot. Empirically, we find that this scheme helps improve reasoning efficiency by avoiding entropy and response length explosion.
4.3 ALGORITHM
Given the current LLM policy va. we approximately maximize the following objective at each step:
A A) = (0) + (TT)
(4.2)
where (ro(O) is a policy optimization objective, e.g., the GRPO objective is used in tests. At each global step, we set the sampling policy na according to the choice of policy optimization objective as the policy from last global step. Given ma.
CpO. For example, in PPO-type algorithms, m batch of queries 8p 'D are sampled, and for each query, T. rolls out a batch of trajectories up to the maximum time step. With the batched samples, we can then optimize for the batch surrogate of AAE (0; A) for several mini-epoctis At the end of each global step, the entropy coefficient is adjusted according to scheme 4.1 The whole process is summarized in Algorithm T
EXPERIMENTS
In this section, we conduct experiments to verify the effectiveness of our method.
5.1 TRAINING DETAILS
Models, training datasets and baselines. The algorithms are tested in multiple training settings: (a)
we train the -rre2.55-mtt--.55 base model the MATH dataset (Hendrycks et al. 202T), which contains 7500 math problems with various difficulties and covers multiple mathematical areas; (b)
train the DeepSeek-PP -i-tkk-ii--tllledd --distiledd-wen---... Qwen-1.50 (DeepSeek-Al 2025) model on 40k verifiable queries from the OpenR -math RO 2025) dataset, which is derived from Numina-math dataset (Li et al. 2024). In addition,
also train the 0wen2.5-math-7hh model on 6k samples from DeepMath dataset (He et al., 2025), the results of which is deferred to Appendix [A.6] We compare our algorithm
with GRPO and the conventional entropy regularization method which we call EntReg, where the
GRPO objective is augmented with the original entropy bonus used in (Mnih et al. 2016 Schulman
ct al. 2017).
Evaluation. We evaluate models on the AIME 2024, MATH-Hard test split (Hendrycks et al.
202T), MATH-500 (Lightman et al. 2023), AMC23, MinervaMath (Lewkowyez et al.) [2022] and OlympiadBench (He et al. 2024). We estimate the test score by averaging 4 tries per query on all benchmarks. The test-time generation temperature 0.6, top-p 0.95 and top-k is 20.
Hyper-parameter settings. The tests are based on the verl framework (Sheng et al. 2025 When training Qwen2.5-math- 1.5b base model on the MATH dataset, we use Adam optimizer with a
Our code is available at http:://gthhaaccomm/agggggAAaaaa

---

Published as a conference paper at ICLR 2026
0.24
0.99
0.22
Iueroge test score
0.20
AINE74
0.8
0.34
GRPO Entragg
GRPO
0.0
Entfleg
AEnt
0.38
0.88
103
356
200
250
50
100
199 Step
200
258
Step
(a) Training @wen2.5-math-- 1.5b on MATH datasct.
0.64
0.62
AIME24 score
Aparage
0.29
EndReq
EntRegg
0.54
AFr
AFnI
(h) Training DeepSeek-R 1--distilled-Qwenn on a subset OpenR -math dataset.
Figure 3: Test score comparison (see Figure 4 for more training metrics).
Table 1: Test scores by benchmark, where we evaluate the model with the highest average test score trained by each algorithm. Here (a), (b) indicates the two settings described 5.1 Bold numbers
indicate the best performance one on the benchmark.
MATH-Hard
MATH-500
AIME24
Minerva
Olympiad
AMC
Setting
(a)
(b)
(a)
(b)
(a)
(b)
(a)
(b)
(a)
(b)
(a)
(b)
Base
0.368
0.661
0.584
0.792
0.083
0.225
0.179
0.311
0.279
0.432
0.406
0.594
GRPO
0.524
0.773
0.756
0.865
0.192
0.367
0.311
0.347
0.364
0.576
0.550
0.769
EntReg
0.546
0.808
0.752
0.872
0.167
0.342
0.316
0.359
0.370
0.576
0.562
0.794
AEnt
0.552
0.813
0.750
0.882
0.217
0.392
0.330
0.359
0.377
0.591
0.581
0.825
learning rate of 10 We set the max response length as 3072. We use a batch size of 512, and for each query roll out 16 responses with default sampling parameters (top-p and temperature set as 1). For AEnt, we use the GRPO objective as Lro- We use a clamping percentage = 0.33, and set FLow 0.11 and Hhigh = 0.24. We use an initial entropy coefficient of 0.002, and start updating the coefficient from the third epoch with 3 0.002. We clip the coefficient in between 0.0006 and 0.009. For EntReg method, we use the traditional entropy bonus with fixed entropy coefficient of 0.002. When training DeepSeek-R1-eeee 1--paneeee-----iillll------ --it------dstiildd-eeeel model the OpenRI 1-math dataset, a learning rate of 10 max response length of 7 7168, a batch size of 256 and for each query
we roll out responses. We use 0.25, Hlow 0.35 and Hnigh = 0.62, an initial coefficient of
3 x 10 10-4, and start updating the coefficient from the second epoch with 3 10 10-4. We clip the coefficient in between x 10-5 and 0.001.
5.2 PERFORMANCE ANALYSIS
We report the test performance in Table I and Figures It is observed AEnt outperforms the baselines on average, and on 5 out of the benchmarks acTOSs the two different experimental settings.
An observation the test score and the entropy trend. An interesting observation from Figures 4a is that after around 175 steps (collapse time), the policy entropy of GRPO largely depletes and the entropy of EntReg starts drastically fluctuate, while AEnt's policy entropy is kept stable. Then one

---

Published as a conference paper at ICLR 2026
GRPO EntReg AErt
GRPO EntReg
0.3
Responee length
800
AEMt
06
720
Entrany
0.4
600
350
0.2
0
350
159
250
100
150
202
250
Step
Step
(a) Training Qwen2.5-math- 1.5b on MATH datasct.
5500
EntRea
08
Response length
AEnt
5000
GRPO
0.2
1500
400 Stoo
200
400
000
Stop
(b) Training DeepSeek-R1-distileedd 1.5b subset of OpenR I-math dataset..
Figure 4: Entropy and response length trend (see also Figure 3
for test score comparison).
can observe from Figure 3a that the test score of GRPO and EntReg plateaus around the same step, while the score of AEnt continues improve and surpasses the baselines past the collapse time. This observation is consistent with our intuition and theoretical analysis: after GRPO's 's entropy collapse, few paths and no new information process, ultimately leading to the stagnancy of the learning process. This is predicted by Proposition ( that the policy will become stationary after entropy depletion. Additionally, can be observed from Figure that the entropy regularization methods result in slightly longer response in the mid/end of the training period. The potential reason is that the regularization makes the model less certain, and thus the models tend to continue its generation, resulting in longer response. Nonetheless, the increase in response length is relatively mild and we did not observe major drawback in reasoning efficiency.
its policy becomes concentrated
be gained in the sampling
5.3 ABLATION STUDIES
In this section, we conduct ablation studies on our algorithm.
Adaptive coefficient stabilizes training. In Figure 5 compare the performance of adaptive coefficient vs constant coefficient of the regularizer. The test performance similar for the two methods in this particular experiment. However, adaptive coefficient leads a significant advantage on reasoning efficiency by delivering more compact responses while not sacrificing accuracy. In the third plot of Figure constant coefficient fails stabilize policy entropy in the mid of training, which results in the entropy blow up. We observe positive correlation between entropy and response length in this case, where a exploding entropy leads repeated reasoning patterns that do not increase the test scores. On the other hand, the adaptive coefficient successfully prevents the entropy and response length from blowing up.
Analysis of the entropy clamping percentage p. In Figure G we compare the algorithmic perfor- mance under different choice of clamping percentage p. Intuitively, the percentage p decides the size of the clamped space A(s), where larger leads to more aggressive clamping and less tokens taken into account during entropy calculation. This would smooth the LLM policy on a more compact space, reducing the bias induced by entropy maximization while running the risk to leave out valuable tokens. In this sense, is reasonable to try to maximize p until the performance drops, which is also suggested by our reported results. Despite the fact the AEnt's performance is affected by the
8

---

Published as a conference paper at ICLR 2026
0.47
0.4
0.22
0.5
AEnt (constare cooff)
Respo lengtn
0.44
JIME20 score
Esposp 0.2
YMM
0.43
0.42
0.00
AEnt (constart cooff)
AErit (constant coalt)
0.30
150
200
250
Stop
Stop
Stop
Figure 5: AEnt with adaptive entropy coefficient vs with a constant coefficient. The score this test similar. Adaptive coefficient better controls the response length and the policy entropy.
0.24
26
p-0.2
scaee
0.4
Aearage
011
0.4
best score
pm0.3
0-20.2
pro.3
D-002
02-
P.A
Figure 6: Comparison of different clamping percentage
choice of p. its advantage over the baselines is somewhat robust the choice. It be observed from Figure that AEnt outperforms the baselines with different choices of
5.4 TIME COMPLEXITY
We compare the time complexity of each algorithm under different base models and training datasets. The results are reported Table The experiments on 2wen-2.5-math- 1.5h and MATH training dataset are conducted on 4xA1000 and all other experiments are conducted on 8xA 100. The hyper- paremter setting has been described Section 5. where for fair comparison, we keep the batch sizes, the max response length and all computation speed related configures the same for all algorithms.
Table 2: Time complexity comparison under different settings. "Update per step" indicates the GPU
hours of forward/backward process per step; "to GRPO/highest score" indicates the total GPU hours to reach the highest achieved by GRPO/the algorithm itself. The first and second column respectively reports the results for setting (a) and (b) described in Section 5.
1ont-math-1.5--MAATE
update per step to GRPO score to highest score update per step to GRPO score to highest score
GRPO EntReg
0.234 h
57 h
57 h
0.303 h
237
237 h
0.253 h
49 h
52 h 0.316 h
215
238 h
AEnt
0.256 h
35 h
69 h
0.324 h
186
275 h
Overall, AEnt consumes slightly more compute per step due to 1) AEnt-trained model's response length is moderately larger than those of the baselines, indicated in Figure 2) compared to GRPO, it requires the extra forward/backward process of the clamped entropy regularizer, and compared to EntReg, the additional clamping related computation is off-loaded to CPU in our implementation for memory save. However, AEnt reaches a common score threshold faster than the baselines indicated by the "to GRPO score" results, indicating AEnt preserves the acceleration effect of entropy regularization. AEnt reaches the highest test score slower than the baselines since it prevents premature convergence, and converges higher test scores which takes more training steps.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION AND FUTURE DIRECTIONS
In this work, we showed that entropy regularization suffers from large bias LLM-RL training. As remedy of this issue, we propose an entropy control method that utilizes clamped entropy bonus with an automatically adjusted coefficient. We show that AEnt consistently outperforms competitive baselines across multiple benchmarks. We believe AEnt can demonstrate more significant advantages if tested on larger models with more compute. In this work, we did not include theoretical analysis of the clamped entropy. addition, we believe the choice of the clamped space A(s) is crucial the algorithm's s effectiveness, and finding a better choice can potentially yield significant performance gains. For example, one may consider removing actions that are redundant (Baram et al., [02T]] Zhong al. 2024) grouping similar actions in entropy calculation. Alternatively, since computing entropy on token level suffers from the large dimensionality of LLM vocabulary, it can be beneficial to design entropy regularizer in the state action representation space (Tennenholtz & Mannor 2019 Tavakoli et al. 2018). We leave these studies to future works.
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting reinforce style optimization for learning from human feedback in lims. arXiv preprint 2rXiv:2402.147444 2024.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pp- 151-160. PMLR, 2019.
Nir Baram, Guy Tennenholtz, and Shie Mannor. Action redundancy in reinforcement learning. In Uncertainty in Artificial Intelligence, pp. 376-385. PMLR, 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1oX.vv:/066.01555 2016.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02544 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint urXiv:2507.0626// 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22 226177, 2025.
DeepSeek-AL. Deepseek-r1: Incentivizing reasonnng capability in llms via reinforcement learning, 2025.
www.
12948
Wei Fu. Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei. Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24228, 2025.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. Pmlr, 2018.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
10

---

Published as a conference paper at ICLR 2026
Zhiwei He, Tian Liang. Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang. Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath- 103k: A seage,scale, challenging. de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504. / 1456, 2025.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint
2021.
Ruinan Jin, Shuai Li, and Baoxiang Wang. On stationary point convergence of ppo-clip. In The Twelfth International Conference on Learning Representations, 2023.
Sara Klein, Simon Weissmann, and Leif Döring. Beyond stationarity: Convergence analysis of stochastic softmax policy gradient methods. arXiv preprint arXiv:2310.02671, 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong. Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath.
[https: (attps:// http::/aa................//iiiia ctw.:ppreeoeeeettiiiiaa aimo- prmo-pprorres--priieeeee wlwbbblbbbbbee http:///ss.pppapaooooo//oaaaaa/p numtit///uiiiiii na haw.aaaaaa.aaddaaaaaaaaa 2024.
co
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20051 2023.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning, pp. 6820-6829. PMLR, 2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PmLR, 2016.
Open-R1. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https //github. com hww.hhhggggggggooecc /ppen-rl
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:1707.06:: 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou. Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
11

---

Published as a conference paper at ICLR 2026
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep rein- forcement learning. In Proceedings of the aaai conference on artificial intelligence, volume 32, 2018.
Guy Tennenholtz and Shie Mannor. The natural language of actions. In International Conference on Machine Learning, pp. 1295-6205. PMLR, 2019.
Simon Weissmann, Sara Klein, Waiss Azizian, and Leif Döring. Almost sure convergence rates of stochastic gradient methods under gradient domination. arXiv preprint urXiv:2405.13592 2024.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-255 2566, 1992.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241 1991.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. Entropy-regularized process reward model. arXiv preprint arXiv:2412.11006, 2024.
Dianyu Zhong, Yiqin Yang, and Qianchuan Zhao. No prior mask: Eliminate redundant action for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 38, pp. 17078-17088 2024.
A APPENDIX
A. OMITTED NOTATIONS IN SECTION
Value functions. We first define some notations we omitted in Section Given the definition of W (s(),, we can define the Q-function as Q7 (s, a) r(s,a) + V (s') with s' P(s,a). By this definition, we can also cquivalently write
(s, E
Σ r(st, ar) a, =
We can also define the advantage function AA (s) Q7, (s) VT²" (s).
Entropy-regularizeed value functions. For the define the entropy- regular-ee value functions as
setting, we can analogously
#-1
(x) E
Σ
(r(se, at) - A (T(aa =
T (s, a) = rss a) + (s') with s' P(ss
Then the entropy-regularized advantage function is defined as (s, (s,
log (e s)) (s).
Note that we omit the time step subscript in the value functions when = 0, e.g., we write 1 h--0
V and similarly for all the value functions.
12

---

Published as a conference paper at ICLR 2026
A.2 PRELIMINARY LEMMAS
Lemma (Entropy gradient). For the softmax policy, we have
H-1
VH((TT) -Brro
Σ
Vlog (a sh,)
Σ log no To [1r
(A.1)
Proof. Starting from the definition of entropy, expand the expectation and write
H
H((T)) =
Σ
F(so, .....,
.... (H-1(TT))
log ne (se)st)
H
Σ
((o)))((a((su)) (
log ma (a)(s)))
(A.2)
where in the first equality, the expectation only taken over sn and the action sequence since the transition is a deterministic in our LLM setting. Then the gradient of the entropy is given by
VH(T))= =
Σ
PI To (an (p)
log no (acls)))
H
Σ
P(sg)
Tan
log To (atls))
(A.3)
80. 20.00
For the first term in the RHS of equation A.3
have
Σ
log To(celss))
Σ
P(so) V
To
=
Σ P(so))
Σ
HH-1
(anlssh))
dg...Gu-1
Σ
V1=
(A.4)
For the second term in the RHS of equation
A.3
have
#-1
( /=0 mo
log (T((())))
H-1
#
P(so) K=0 To (an 18h)
v
log (o (sh)
log no (acls))
5o 40.40
h=0
H
E
v log (a
log ne (arlse)
H
H-1
H-1
E m
V log To (k))
log T (a
+ Erg
V log no (ax|sh)
Σ ı=A
log To (a.ls:)
1=1
h=0
H
h-1
E
Σ
I (m))
V log Te (anl>)))s) sp
Σ
log To (a(((1)))
H
H
E
log To (ap (sh)
log To (a(((()))
hma
H
E n
V log To (sx)
log कe (4r (st)
(A.5)
M=
t=
13

---

Published as a conference paper at ICLR 2026
where the second last equality follows from the towering property of the expectation, and the last equality follows from the fact that for any s, we have
(a) V log (a(])))
Σ To Vog
(aa(als)
V
Σ
[a(as))= V1 =
(A.6)
Substituting equation A.4 and equation
A.5
into equation
A.3 yields
H H-1
H-1
VH((T)) = -Ei
V log no (an (8))
Σ
log no (a (ce)se)
(A.7)
h=0
This completes the proof.
Lemma Given any h E (0, 10.1,...: H 11-1 13 and some baseline functions b%." S + IR, we have for any policy Te that:
E sTe
v log To (a (5)))) =
(A.8)
where the expectation taken over (so D, ag....... aH-1) generated under policy TT.
Proof. We have
H
Exs
log To (o((ns)))) (sn)
11
= E
Ban
V log (an [sn]] b% (sn)
(A.9)
h=0
which follows from the towering property of the expectation and equation A.6
Lemma (Entropy regularized softmax policy gradient). If the policy is softmax, we have
H-1
Vo... V (D) = Σ
p.o
(3))
(A.10)
t=
where P.T is the shorthand notation P(s = s ) which is the probability of reaching state time step given policy πe.
Proof. By the policy gradient theorem (Sutton et al.
1999) and its adaptation the finite-horizon
setting (see, e.g., (Klein et al. 2023)) we have
H
VV (D) = Ess-Darrsseee
V log no QT oz)
(A.11)
The above equality combined with the entropy gradient given in Lemma yields
VV (D) = VV (D) AVH((T))
H
H-1
Em
V log lag (sr, -
Σ
log No (a,ls)))))
Em
Voog no (s) ai) =log
(A.12)
14

---

Published as a conference paper at ICLR 2026
The above equality gives the policy gradient formula with the Q-function. It can also be rewritten with the advantage functions. By Lemma we have
E
Σ t=
Vog Togals())))) O.
V
(A.13)
Using equation A.13
equation A.12 gives
H
VV (D) E.,
V log (o((aa)s)) 0x) Alogmo(a)ss) - V2 (*)))
V log (13
(A.14)
which follows from the definition of the entropy-regularized advantage function. We can also rewrite the policy gradient formula in equation A.14 with respect to the state marginal distribution as follows:
H-1
VV (D) E
Beg (o
log 47) at)
H
E
P (s)
v (a (s,
pilo
loggls))lo
(A.15)
t=
Under the softmax policy, we have Ves log (als) = 1.=1 (1o=a - (t(3))) Then the element- wise policy gradient is
H
pTa (s) Σ To(als)] (= (10 To(ā|3) A (s,
V Vo
(D)
1=0
To(a|s) Iu-a (1uaa- no(als) (5,0)
Σ Po
(A.16)
where the last inequality is due to (a)no(s) (s, a)] 0 following the definition of the value functions.
Lemma (Performance difference lemma), We have for any E (0.1..... H 1} and state s E S, the performance difference between any two policies x and n' is
H (s) V/ (s) = E
Σ
A (st. as)(sn = s).
(A.17)
15

---

Published as a conference paper at ICLR 2026
Proof. We have
(s)
Vi (s)
E
r(s az) 8h =
V%7((s) (s)
H
H-2
H-2
r(s,, a+)
+
Σ
V
1,
(8++1) - -E
V (8++1)]88 = - Vo (8)
t=h
1=b
H-1
E
Q7 (86,
Σ
(x)())) =
E
AF (1)s =s
(A.18)
This completes the proof.
A.3
PROOF OMITTED IN SECTION 3
A.3.1 PROOF OF PROPOSITION I
Proof. We start with proving the first bullet. Denote the entropy of TT(-|s) as
H(((-1s))
Σ Log
(A.19)
Since x S -log x for < x S 1, we have
H((((1s)) 2 Σ (1
(A.20)
Viewing mo(+ as a vector A Al, is known that the softmax Jacobian can be written as (m Diag(To(-(s)) (=(-)))To(115)
(A.21)
00,,
Then we have
iio (-s()
(To(-|s)
-
20...
00
S Σ (To(als)(1 (T(ass) - To(a|s) Σ To(alls))
(1 - (a(sss))
(A.22)
By equation A.15 in Lemma
we have
#-1
VVT (D)
Σ
Σ
p (s) Σ [olaa))log (s)
#
a
@no(als)
p
A (s,a)
De..
dmo(s
(s,))
(A.23)
00..
1=0
16

---

Published as a conference paper at ICLR 2026
where the third equality is due
8aola a) 202,
Of s' *
Then we have
H
0 (-ss)
[VV
S
P (5)
00,,
1=0
H
< 2
pT
(8) (T) (-18)
t=
(T))
(A.24)
where the last inequality is due the definition of the policy entropy:
H
H(mo) ETTT
log No (aelss))) so D)
H-1
= Σ
Σ
(8) Σ (a|s)) log (als)
(A.25)
This completes the proof of the first bullet.
Next we provide the proof of the second bullet. Let x¹ E arg max, (*(D) be any deterministic optimal policy. Given any 8g ~ D, let 8) oz be state-action pair generated by n² up to time step e.g., ox = n" (67)). We write sg 80.
Given any 80, we have
H
(D)) 0.5
IV (D)I N
V (D)
h=0
(T)) (s)) " of)
(sh
(A.26)
where the second inequality follows from Cauchy- SehuhrySShwartz inequality, and the equality follows from the softmax policy gradient derived in Lemma 3
Given 8o. by the assumption of LLM tasks that 8 P(sr, ae) is concatenation of St, de for
any 0 & < H - 1, we have "
(8h) = 0 for any o h. Using this fact in equation A.26
VV (D)I
Σ
(7 (sh.o7))
(A.27)
Continuing from equation A.27
V/ (I)
#
1
mo (8)
VH
(T
P. (afls))) (sh,
VH
Pr
(a (s;)
p (a))))
P7 (s)) (sh,aa))
(1 (a
(A.28)
17

---

Published as a conference paper at ICLR 2026
where the second last inequality follows from the definition of 17 (5.). and the last inequality follows from the fact that x² is defined as a deterministic optimal policy, yielding
F(su) (a (ails))... .... =aaa = P(so) =
(A.29)
D
Continuing from equation A.28 we have
VV (D)]
[D]
min
p (5))
Ph (s)) (at (7) (s% oñ)
he (0. --1
H
= 4=0 To (8)
P (8) (a (sh.
V
IT
h=0
nd
(
B (a) [0]]
TT < 18)) (V' (so) V (80)
(A.30)
Note that this inequality holds for any trajectory (sus aú. aj, ... , -1) generated by any determinis- tic optimal policy π² Then we have
IV Th (D)I 2 C (so) (v (VT (so) VTa (so)
(A.31)
where (sa)
max og. 1)A (5o) II -1 To (e(a with (i)(s)
((ao, 01, (1,...,(H-1) E AH 3m" E arg max, (D)),, n x (a+ (86) > 0).
A.4 PROOF OF PROPOSITION 2
Proposition 2 can be proven by combining Lemma 5 and Lemma
Lemma 5. It holds that
A
v* (so) - (s)) S VV (so) ( + AH log
(A.32)
A (b0))
where x = arg max V (D), and recall n' E arg max, V (D).
Here (so)
{aa,, 01, -11 E A 3m" II'"" T (acls)) > 0} is the set of all optimal responses given query Sp-
Proof. Define H(T[sp)) as
H-1 log x (o
H(T((s)) = EE
Σ
(A.33)
For any x² max, V (D), by the optimality of क have
( - V (so) 2 V (so) - VVT"(su)
V (3o) (** + X(H(((T()))))) (so) H(To(s0))
(A.34)
where the equality follows from the definition of V (5o). Then have
V (8p) - V (NO)
yaa (5o) v00 +
max
H((T|s5)) H(TTolso)
(A.35)
Earg max, VT
18

---

Published as a conference paper at ICLR 2026
Notice that
H-1
max H(
max E
log x (asl8)))5)))
t=
Σ
x (arlse)
log 11 x= (aels))
max
a 1EAA (en)
Σ
P(7) logg(t))
max
(EA))(00)
E (s)))
log
1
A (80)
TEAN (50)
log [A4z (so)
(A.36)
where in the third incquality, (AA, (50)) denotes the probability simplex on Ais (so). Additionally, it is known that
H(mg|))) (0) max H( H log IA
(A.37)
Substituting equation
A.36 and equation A.37 into equation A.35 yields
A
V (50) (80) 2 V* (so) - (80) AH log
(A.38)
A
(50)
which completes the proof.
Next we present the performance bound under entropy regularization. The derivation is adapted from (Mei et al. 20200, Lemma 15) for the LLM setting modeled as finite-horizon MDPs with deterministic state transition.
Lemma 6. Assume the policy is a softmax. Then it holds that
(50 VT S 2A Cx (su)
VVV (D)122
(A.39)
where and O (so) specified the proof.
Proof. The performance gap can be bounded
m (80) - Vio (80)
1-1
Ex
r(s,, a+) - Alog (arl())))))) + (x)( - VATA (5c))(0)
Vi (so)
Exg
r(st, az) Alog= 1 (84411) V (s))
#-1
Ex
Q (st, (st, 0z) Alog (a(()))))))))))))))))))))))))( V. V. (st)]s0)
1=
H-1
Σ
E
( a) - A Allg - V (s)
(A.40)
pi (-150)
where P (-1s0) = P(s; = 110,, Rÿ) is the probability distribution of st under policy лं given the initial state 8p. The second last equality uses the definition of Q that Q (St, oz) (St,, (t) Th
Ve41A (80+1) with st+1 (8t,a)).
19

---

Published as a conference paper at ICLR 2026
Given any have
Ea (2) (i)((sa) - Alooo mi(a|s)
Σ max Σ
to(als,t)Q (
ATo(als, log ta(a)s,t)
log Σ exp(Q
(A.41)
where oo (a = exp
exp (s. Notice that
(+)
Σ
Al
T033))) Alog (log - A log #o(als,t)
Alog exp (0
(A.42)
Substituting equation
A.41
and equation A.42 into equation
A.40 yields
V (50)
V,X (so)
E
s-
(-100)
[=U
H-1
(r,(s,))
Ea N.
To Q2,(s, - 0s.a
0g.
(A.43)
(110)
[A
where 1 E RR|-| is an all-one vector and the last inequality follows from 27).
(Mei et al. 2020 Lemma
Following the derivation of Lemma it straightforward to verify that equation A.15 holds with (s, Alog T (a|s) in place of the advantage AM
H
VVT (D)
Σ
pao
(8)
Σ 8) Vog
- AlogTo(a(s))
H
n
Vro(als)(O - llo (als))
(ro(cls) 20 + x Σ exp 0s.))
Σ VTo(als) A0s,a)
(A.44)
where third equality follows from the Ta(as)) is softmax function, and the last equality is due to the fact that
Σ VTo(als) Σ 0s. = Σ Exp 0n.a V Σ Ta(als) = Σ exp 0s,a VI = 0.
Then from equation A.44
have
TVT
#-1
Omo(als)
p (s)
(Q (s, - -10ea)
00 s.-
00...
t=0
H-1
dmo(-
Σ
pioe (s)
(Q,2(s,)) -.0..).
(A.45)
00..
t=
20

---

Published as a conference paper at ICLR 2026
where the first equality is due to the fact that ( for any s' * and the last equality follows from matrix- product rewriting.
Define S(so) S as the set of all possible states starting from so, i.c., S( = {oo} U (se E Sit
{I..... 1), a--1 E A, st-1 E S(so), $, = -() Then we have
(V (D)
0.5
VV (D)l N
Σ
00...
+ES(#a)
a
(D)
Σ
v SS(())
08,
sES(#o)
(-s()
Ca
Σ sES(90)
Σ
I
(
(A.46)
00,.
where the second and the third inequalities follow from Cauchy- Schwartz inequality, and the last incquality follows from equation A.45 The constant Ca =
Vis(sa)
Continuing from equation A.46 follows similar the derivations in (533)-(536) in (Mei al. 2020) that
VVV
(D)]
H-1
Pr (s) min min -00,, Ea Q8, a) A0
2 C1 Σ
Σ
(A.47)
AA
#ES(sa)
1=0
Then we have
VV (D)I
H-1
min [(]x)))) -20,, Ea 17.7
10
2
Σ
Σ
(P (s)
AA
(E)()
1=U
H-
(z) min, (a(s))) p (s|sp)
C2 Σ
(s, 1/2 os..
P (s|s0)
SES(sa) t=
Ea 9TA (8, a)/a 0s.a
[A|
Ea (s, a)/A
0
X² (Su)
p (s|so) (T, -Os.
[A
#ES(sa)
t=
80
H- X² (sn)
073 0x, E. (s, a)/a
Σ
(A.48)
p (1100
A
t=0
where
(P (s)
min, To (a(ss))
CT (so) = C2 min
((00)
(P((s))))
Combining equation A.48 and equation
A.43 gives
(T (TTM((O) S 21 CT
(A.49)
(so)
which completes the proof.
A.5 TOY VERIFICATION IN FIGURE
To verify the claim made after Proposition we will need to vary the sparsity of optimal tokens and observe experimental results. However, it is generally difficult control the sparsity of optimal
21

---

Published as a conference paper at ICLR 2026
GRPO
GRPO
EntReg
2222 30300
0.330
030
130
2.43
620
0.42
615
GRPO
0.75
EntReg
Stap
Stap
Stop
Figure 7: Results of training Qwen2.5 1aat2..5-Mthh-BB55 on 6k samples from the DeepMath dataset.
Table 3: Benchmark scores of training Qwen2.5-Math-- on 6k samples from the DeepMath dataset. Bold numbers indicate the best result on the benchmark.
MATH-Hard MATH-500
AIME24
Minerva Olympiad
AMC
Base 0.443
0.626
0.183
0.143
0.290
0.469
GRPO
0.622
0.808
0,250
0.358
0.412
0.631
EntReg
0.620
0.810
0,214
0.365
0.437
0.644
AEnt
0.657
0.828
0.258
0.379
0.493
0.637
responses for real queries. Thus the verification experiments reported in Figure Tare conducted in synthetic task, and we leave the results on non-synthetic tasks to Section 5
Task setting. In the synthetic task, the total number of actions is |AA| 10. where the number of optimal action Ropt can be picked from {15, 10, 5, }} and the number of suboptimal action is fixed at 500. The reward for optimal action is 1, for suboptimal action 0.2 and is for all other actions. For simplicity, we set H = 1. The policy is given by a tabular softmax with parameter e 10 To mimic pre-trained initial policy, initialize the policy parameter corresponding the optimal actions and 500 other random actions from A(1, 1); while we initialize all other logits from ( 1). Results in the figure are averaged over 20 independent runs.
The learning rate is set as 0.02, and the batch size is 64. The syyper-parametee for each algorithm are found through grid search. For Ropt 15, 10, 5, 1: the regularization coefficient for entropy regularization is set 0.0005, 0.0005, 0.0005, 0.0007, and the coefficient for clamped entropy regularization set as 0.0008 uniformly. The clamping percentage p set as 0.98, 0.98, 0.985, 0.997.
A.6 ADDITIONAL EXPERIMENTS
In this section, we report some additional experimental results.
Experimental details. The algorithms tested (wen2.5-math-77 base model 6144 samples from the DeepMath dataset (He et al.) (025). We randomly select queries with over-long filtering (no more than 1024 tokens) from the dataset. The evaluation method is consistent with that in Section 5.1 We use AdamW W optimizer with a learning rate of 1 x 10-5 We set the max response length as 3072. We use batch size of 512, and for each query we roll 8 responses. For AEnt, we use the GRPO objective as LpO. We use a clamping percentage p 0.33, and set Hlow =0 and Hhigh 0.3. We use an initial entropy coefficient of 0.002, and start updating the coefficient from the third epoch with 3 = 0.001. We clip the coefficient in between 0.0006 and 0.005. For EntReg method, we use the traditional entropy bonus with an entropy coefficient of 0.002.
Observations. The results are reported Figure 7 and Table 3 The performance observation of Figure is overall consistent with that in Section 5.2 It can be observed that AEnt is able to outperform the baselines (left plot) potentially by preventing an early entropy collapsing (middle plot). The response length of the algorithms is overall similar towards the late training period. AEnt's 's training reasoning efficiency is thus similar to the baselines in this test.
22


Reviews:
[{"review_id": "vseMMXG13R", "reviewer": "Reviewer_gqdk", "summary": "This paper addresses the well-known problem of standard entropy regularization failing to provide benefits in LLM-RL training, unlike in traditional robotics or game environments. The authors provide a theoretical diagnosis, arguing this failure stems from the combination of the LLM's extremely large response space and the sparsity of optimal outputs, which introduces a significant entropy-induced bias. As a remedy, they propose AEnt, a straightforward entropy control method featuring two key components: 1) a \"clamped entropy\" bonus calculated only over a size-reduced, high-probability token subspace to mitigate this bias , and 2) an adaptive coefficient that dynamically adjusts to keep this clamped entropy within a target range, preventing training instability. The authors present empirical results on math-reasoning tasks to demonstrate that AEnt can stabilize training and outperform baselines like GRPO and traditional entropy regularization.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper studies an important problem that why entropy regularization works for games or robotics but not for llm\n- The motivation is strong; the reason why entropy regularization fails in LLMs—attributing it to the large and sparse action space—is intuitive and well-supported. This claim is convincingly backed by both the theoretical bound in Proposition 2 (which explicitly shows the bias term's dependency on action space size $|\\mathcal{A}|$ and optimal response sparsity) and a compelling, clear toy experiment in the Figure.", "weaknesses": "- The paper's first method is 'clamped entropy' , which means maximizing entropy within the LLM's 'top-p' token range. In fact, this idea is quite straightforward; I have even tried it myself. However, this method is extremely sensitive to the hyperparameter, specifically the coefficient in front of the entropy loss. A good result can be obtained through extensive hyperparameter tuning, but it still requires re-tuning for a new model, and the performance is not very stable.\n- The paper's second method, dynamically adjusting the entropy coefficient to maintain stable entropy, is also not a new concept. This adaptive mechanism is functionally similar to early KL control approaches (e.g., [https://arxiv.org/pdf/1909.08593.pdf]) designed to keep the KL divergence within a stable range, a method also integrated into the VeRL repository ([https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py#L150]). To my knowledge, practitioners today still prefer fixed KL coefficients precisely because these dynamic adjustment mechanisms are notoriously unstable. The method's core premise—that forcing entropy to stabilize within a fixed numerical range is beneficial—is unfounded. It is difficult to determine a priori what a \"good\" range for entropy or KL should be. Furthermore, it is questionable why entropy must be stable; a beneficial training trajectory might just as well involve entropy slowly rising and then slowly falling. The optimal shape of the entropy curve is likely case-by-case, and attempting to manually prescribe its behavior is a questionable engineering choice\n\n- The paper's experimental results are not convincing.\n    - In Figure 3, using the peak of the curve to represent the performance is flawed, as it could very likely be an artifact of training randomness.  \n    - Table 1 only shows the final results for only one model,  but Figure 3 shows the training curves for DeepSeek-R1-distilled-Qwen-1.5b and    Qwen2.5-math-1.5b. Table 1 should be aligned with Figure 3.\n    - I would expect the paper to demonstrate consistent improvements on at least three different models without requiring heavy hyperparameter tuning to truly prove the method's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5tcUXeb8n", "reviewer": "Reviewer_3biZ", "summary": "This paper addresses the limited effectiveness of conventional entropy regularization in LLM-RL training, which contrasts sharply with its success in robotics and games. The authors provide theoretical analysis showing that entropy regularization suffers from bias that scales with response space size and optimal action sparsity. They propose AEnt, which employs two key mechanisms: (1) clamped entropy computed on a re-normalized policy over a reduced token space containing only the top probability tokens, and (2) an adaptive coefficient that automatically adjusts based on whether current entropy falls within target bounds. The method is evaluated on mathematical reasoning tasks using 1.5B parameter models (Qwen2.5-math-1.5b and DeepSeek-R1-distilled-Qwen-1.5b), demonstrating improvements over GRPO and standard entropy regularization baselines on 5 out of 6 benchmarks across two experimental settings.", "questions": "- Have you evaluated the method on larger models (7B, 13B, 70B parameters)? The experiments exclusively use 1.5B models, but larger models may exhibit fundamentally different entropy dynamics due to better pre-training, sharper probability distributions, or different exploration-exploitation trade-offs. Specifically, do larger models already concentrate probability mass appropriately, making clamped entropy redundant? Additionally, how does the computational overhead of computing clamped entropy scale with model size, and at what scale does this become prohibitive relative to the performance gains observed?\n- Why specifically use top-probability tokens for defining the clamped space, and have you systematically compared against alternative criteria? The choice appears to assume pre-trained models already assign high probability to reasonable tokens, which may be circular reasoning if the goal is to improve exploration. Have you experimented with task-relevant vocabularies defined by domain experts or training data or learned token importance via a separate lightweight model? Furthermore, does the top-p criterion work equally well across different input contexts—for instance, does it fail when the model is uncertain and distributes probability broadly, or when pre-training biases are strong but incorrect for the RL task?\n- How does performance vary across fundamentally different task domains beyond mathematical reasoning? Mathematical reasoning has unique properties—verifiable correctness, structured formats, relatively sparse optimal solution sets—that may make it particularly amenable to entropy control. Have you tested on tasks such as open-ended instruction following where response diversity is valued or multi-turn dialogues which require contextual adaptation? More critically, does your theoretical analysis suggesting sparse optimality as problematic even apply to domains where optimal response sets are large or poorly defined? Without broader evaluation, it remains unclear whether you have solved a mathematical reasoning-specific problem or a general LLM-RL challenge.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides clear theoretical motivation through Propositions 1 and 2, explaining why conventional entropy regularization underperforms in LLM settings due to large action spaces and sparse optimality, with rigorous proofs and synthetic validation in Figure 1.\n- The proposed clamped entropy mechanism directly addresses the identified theoretical limitation by restricting entropy maximization to a compact, relevant token space, reducing bias from irrelevant tokens while maintaining exploration benefits.\n- Experimental validation includes relevant ablation studies demonstrating the necessity of both adaptive coefficients (Figure 5) and appropriate clamping percentages (Figure 6), with consistent improvements across multiple mathematical reasoning benchmarks.", "weaknesses": "- Experimental validation is narrowly confined to mathematical reasoning with 1.5B models, providing insufficient evidence for a method claiming to solve a fundamental LLM-RL problem. Mathematical reasoning has unique properties (verifiable correctness, structured outputs) that may not generalize to open-ended generation, instruction following, or creative tasks where optimal response sets are less well-defined.\n- The adaptive coefficient scheme essentially performs online hyperparameter tuning during training, introducing additional hyperparameters (H̃_low, H̃_high, β, λ_low, λ_high) that require careful task-specific tuning. This increases rather than decreases the complexity of deployment, contradicting the implicit promise of a more robust entropy control method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9TixVylLaf", "reviewer": "Reviewer_j9T9", "summary": "The author reveal why entropy regularization  no gains in LLM-RL training.\nThrough theoritic anysis, the author reveal that entropy will improve the convergence to optimal solution but bring bias.\nThe bias is closely related to the scale of action space and the sparsity of the optimal outputs.\nBased on this anaysis, the proposed method calculated entropy only in a smaller(top 1-p) action space.\nexperiments in math-reasoning tasks shows that the proposed method outperforms the baselines.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I believe this is a strong paper with novel contributions and valuable insights for the LLM-RL community, particularly in the context of entropy tricks. I recommend accepting this paper.\n\n1. The motivation for the work is clearly presented.\n2.The theoretical analysis of entropy is well-executed and convincingly explains why the entropy trick cannot be applied directly in LLM-RL.\n3.Simple Yet Efficient Method: The proposed method is both straightforward and effective.", "weaknesses": "I have a few questions regarding some details, and I believe additional clarity would be helpful for the broader application of this work.\n\n1. **Choice of Top (1 - p) Percent Tokens:**  \n   Why do you use the top (1 - p) percent of tokens in \\( \\pi \\), rather than applying a threshold(e.g. \\pi > \\epsilon) to all tokens?\n\n2. **Experimental Results in Figure 3:**  \n   As shown in Figure 3, the experimental results indicate that the performance is similar during the first 90% of the training steps, with significant improvement occurring only in the last few epochs.  \n   How does this happen? Could you provide a more detailed curve with additional steps to better illustrate its stability? An explanation would be greatly appreciated.\n\n3. **Related Work in Entropy Analysis and Action Space Shaping/Pruning:**  \n   Are there existing works that analyze entropy in large or continuous action spaces within traditional RL methods?  \n   Additionally, is there prior work on action space shaping (continuous) or action space pruning (discrete) in RL methods (e.g., [1], [2], [3], [4])?  \n   Could these methods be beneficial for applying the entropy trick in LLM-RL?\n\n**References:**\n\n[1] Zhong D, Yang Y, Zhao Q. No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(15): 17078-17086.\n\n[2] Baram, N.; Tennenholtz, G.; and Mannor, S. 2021. Action redundancy in reinforcement learning. In Uncertainty in Artifcial Intelligence, 376–385. PMLR.\n\n[3] Tennenholtz, G.; and Mannor, S. 2019. The Natural Language of Actions. ArXiv, abs/1902.01119.\n\n[4] Tavakoli, A.; Pardo, F.; and Kormushev, P. 2018. Action branching architectures for deep reinforcement learning. In Proceedings of the aaai conference on artifcial intelligence, volume 32.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rHk2OZPWyF", "reviewer": "Reviewer_oda4", "summary": "This paper presents a technique, AEnt, that makes the PPO's (GROP's) entropy regularization work with LLM-RL setting by clamping action-space, focusing on top-N tokens. Previously, the entropy regularization didn't work well in LLM-RL setup due to the extremely large action-space, compared to standard RL setup. In AEnt, this issue is mitigated by clamping the action-space and additionally introducing an algorithm that automatically tunes the entropy coefficient. In experiments, AEnt shows its strong performance improvements as well as the sane behavior of the policy entropy. Ablation studies show that the policy entropy cannot behave well without the proposed clamping mechanism and the automatic coefficienct tuning.", "questions": "The weakness includes my questions that I want to clarify.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has strengths as follows:\n- The idea introduced in AEnt is simple and generic enough to be applied to general LLM-RL algorithms.\n- In the main experiment, AEnt is shown to be improving performance from the baselines.\n- In ablation studies, the changes introduced in AEnt is vital to gain performance improvements.", "weaknesses": "The paper is well written. Thus, there isn't obvious flaws in the manuscript. However, here is a list of comments to improve the quality of this paper.\n- In the context of automatic entropy cofficient tuning, SAC has also proposed the similar mechanism that is implemented as a dual optimization problem. Authors should explain how the proposed tuning algorithm is selected over SAC's algorithm.\n- In Figure 4, the response length plots aren't really described in the body texts. Since you plot them along with the entropy curves, authors should add some descriptions about correlation between the entropy and the response length at least.\n- Authors should explain how many random seeds each does each experiment run with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)