Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
Authors: Zilin Kang
Chonghua Liao
Tingqiang Xu
Huazhe Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Entropy
Continuous Control
Large Language Models
Image Classification
Primary area: reinforcement learning
TL;DR: We propose ERA, a new paradigm that constrains sampling entropy by applying specially designed activations to models' output. This approach demonstrates broad effectiveness across domains.

Abstract: We propose ERA, a new paradigm for entropy-constrained policy via output activation. It guarantees minimum sampling entropy by transforming the outputs of the last layer. Our approach demonstrates broad effectiveness across different domains: 1) for large language models(LLMs), boosting the average score across six benchmarks for Qwen2.5-Math-7B by 11.6%; 2) for continuous control reinforcement learning agents, improving performance by more than 30% over strong baselines such as SAC on the challenging HumanoidBench; 3) for image classification, enhancing ImageNet top-1 accuracy by 0.69% for ResNet-50. These gains are achieved with a computational overhead of less than 7%. Our work validates output activation as a powerful tool for entropy control, opening a new direction for designing simpler and more robust algorithms. Code available at: https://nothingbutbut.github.io/era

Published as a conference paper at ICLR 2026
ENTROPY REGULARIZING ACTIVATION: BOOSTING CONTINUOUS CONTROL, LARGE LANGUAGE MODELS, AND IMAGE CLASSIFICATION WITH ACTIVATION AS ENTROPY CONSTRAINTS
Zilin Kang!.2 1.ng:.2-Chrr Chongguua Liao 3* Tingqiang Xu Huazhe Xu 1,,4
Shanghai Qi Zhi Institute
2 *Department of Computer Science and Technology, Tsinghua University
3 Institute for Interdisciplinary Information Sciences, Tsinghua University
$ Shanghaa Artificial Intelligence Laboratory
(kz122, 1ch22, xttg3)@mmaaaaaa
cn
ABSTRACT
We propose ERA, new paradigm for entropy-constrained policy via output ac- tivation. It guarantees minimum sampling entropy by transforming the outputs of the last layer. Our approach demonstrates broad effectiveness across differ- ent domains: 1) for large language models (LLMs), boosting the average score across six benchmarks for Qwen2.5-Me 5-tth-----BBB by 11.6%; 2) for continuous control reinforcement learning agents, improving performance by more than 30% over strong baselines such as SAC on the challenging HumanoidBench; 3) for image classification, enhancing ImageNet top-1 accuracy by 0.69% for ResNet-50. These gains are achieved with computational overhead of less than 7%. Our work validates output activation as powerful tool for entropy control, opening a new direction for designing simpler and more robust algorithms. Code available at: https:/hotthnnguubuu www.rrubuuggituuuiio.. github.io/era
INTRODUCTION
Decision-making problems represent broad class of challenges, from robotic control to Large Language Models alignment (Sutton et al., 1998; Ouyang et al., 2022; Kober et al., 2013). In these settings, encouraging exploration and maintaining policy stochasticity, often quantified by entropy, is critical (Ziebart et al., 2008; Schulman et al., 201 In reinforcement learning, the maximum entropy paradigm, exemplified by algorithms like Soft Actor-Critic (SAC) (Haarnoja et al., 2018), has become prevailing approach in control tasks. However, these methods, which add entropy bonus directly to the training objective, inevitably alter the optimization landscape and can interfere with the optimization of the primary objective.
The challenge becomes even more pronounced LLM alignment. Policy gradient methods (Sutton al., 1999) such GRPO (Shao et al., 2024) frequently suffer from entropy collapse (Cui al., 2025b), leading to reduced diversity and performance degradation. Directly incorporating entropy bonuses has been shown to be unstable or ineffective in this setting (Cui et al., 2025b). Moreover, prior works have explored methods that avoid direct modification of the loss function, including clip-higher (Yu et al., 2025) and training exclusively on the high-entropy tokens (Wang et al., 2025). While these methods provide useful insights, they remain ad hoc, lack principled mechanism for entropy regulation, and are narrowly tailored to the LLM domain, limiting their applicability to broader settings such as continuous control and computer vision tasks.
These observations highlight a fundamental gap: existing approaches either distort the primary optimization objective, as in RL algorithms with entropy bonus terms, or provide heuristic, domain- specific fixes with no theoretical guarantees, as in LLM alignment. Therefore, there is a pressing need
"Equal Contribution

---

Published as a conference paper at ICLR 2026
Hose
Kl.cow
Clip cow
CIRP
ISAA
Originai
ERA-A ERA-Augmested
Aug
1420
53.00
77
03.55
312
2341
MUSI
408
Jan
5339
AIME24
AIMEI5
TDMPC2 OBAC
ImageNel
CIFARIO
(a)
Figure 1: ERA Boosts Large Language Models, Continuous Control and Image Classification. (a) Large Language Models: ERA consistently enhances the performance of Qwen-2.5-Math-7H on AIME'24 24.AIME'22 25 and AMC datasets. (b) Continuous Control: ERA significantly improves multi- ple popular RL algorithms, including SAC, PPO, TD-MPC2 and OBAC. (c) Image Classification:
ERA consistently boosts the performance of ResNet-50 on ImageNet and CIFAR-I0 datasets.
for a new cotropy-consttaining paradigm that is universally applicable, non-invasive to the primary objective, and theoretically grounded.
In this work, we introduce Entropy Regularizing Activation (ERA), a novel paradigm for entropy- constrained training. The key insight of ERA lies in realizing entropy-come constraind policy via output activation. Specifically, we impose the constraint through a class of well-designed activation functions applied to the model's final output. This approach completely decouples the optimization of the primary objective from the entropy constraint, allowing the loss function to focus solely on its original goal (c.g., maximizing rewards). We show that ERA only provides provable entropy guarantees in theory, but in practice, it functions as non-invasive module that can be seamlessly integrated with existing algorithms.
The generality and effectiveness of this paradigm are validated across diverse domains, including continuous control, image classification, and large language models. For example, on the DeepMind Control Suite (Tassa et al., 2018), ERA improves the performance of SAC on high-dimensione tasks like Humanoid and Dog by over 25%. Its versatility also demonstrated image elassification, domain where preventing model overconfidence via regularization is critical. Our approach complements established methods, boosting performance on top of strong data augmentation and label smoothing (Szegedy al., 2016). In LLM RL, ERA enables GRPO-trained Qwen-2.5-Maite 7B (Yang et al., 2024b) to achieve a remarkable improvement of 4.7% and 70.7% on the AIME-24 24 and AIME-25 benchmarks, respectively.
Our main contributions are summarized as follows:
We introduce ERA, novel entropy constraint paradigm based on activation functions, and establish a theoretical framework with provable entropy guarantees.
We design effective instantiations of ERA for both continuous (control) and discrete (image classification) domains. For large language models, we propose specialized, adaptive variant of ERA that addresses the unique challenges within this domain.
Our experiments of these instantiations demonstrate significant performance improvements over strong baselines across domains, and reveal their properties such parameter sensitivity.
RELATED WORK
Policy learning control. Entropy maximization is a crucial aspect RL, significantly enhancing exploration and robustness (Ziebart, 2010; Haarnoja et al., 2017). Prior work has explored various methods to incorporate entropy maximization into RL algorithms (O' (O'Donoghue et al., 2016; Nachum et al., 2017; Haarnoja et al., 2017). PPO (Schulman et al., 2017a) introduced an entropy bonus in its clipped surrogate objective. SAC (Haarnoja et al., 2018) later employed maximum-entropp objective with dynamically adjusted temperature parameter, but this can lead to suboptimal performance. More recent approaches have introduced alternative methodologies for implementing maximum entropy RL (Chao al., 2024; Choe & Kim, 2024), while others have shifted the optimization focus directly to state entropy (Zhong et al., 2024). A different line of work avoids modifying the objective function. Akrour et al. (2019); Otto al. (2021) pioneered this direction by projecting the policy
2

---

Published as a conference paper at ICLR 2026
parameters to constrained subspace. However, their instantiation relies isotropic transformations (e.g., uniform mixing), which impose uniform regularization across all dimensions a strategy that scales poorly to high-dimensional action spaces. In contrast, our work intervenes at the output layer with non-linear activation that provides dimension-specific gradient guidance, enabling the network to learn structured exploration strategies rather than being forced into uniform stochasticity.
RL for LLMs. Recent breakthroughs in LLM reasoning, such as OpenAl-ol (Jaech et al., 2024), DeepSeek-R1 (Guo et al., 2025), and Kimi-k1.5 (Team al., 2025), have redirected attention from chain-of-thought prompting (Wei et al., 2022) and supervised fine-tuning (Li et al., 2024a; Yeo et al., 2025) toward RL. Within this paradigm, policy entropy collapse emerges as a fundamental obstacle: the decay of exploratory behavior often leads to performance plateaus. A prevalent approach is reward shaping (Cheng et al., 2025), which augments the reward or advantage with an entropy bonus to maintain a viable exploration-exploittiie exploitation trade-off. Complementary strategies, including loss re-weighting (Wang al., 2025; Cui et al., 2025b) and clip-higher regularization (Yu al., 2025), mitigate the risk of entropy collapse. Unlike these approaches, our method is general and concise paradigm, universally applicable across domains and endowed with rigorous theoretical guarantees.
PRELIMINARIES
Markoy Decision Process. We consider a Markov Decision Process (MDP) (Bellman, 1957) defined by the tuple AS P, R), where S, A are the state and action spaces, P is the transition dynamics, and R is the reward function. A policy TT(ar(s:) parameterized by 0 aims to maximize the expected discounted return:
T
Σ ns R(s
((TO) = EVoro
where E [0, 1) is the discount factor, is the timestep, 8t E S and a E A are the state and action at timestep t, and (So, ao, $1, o1,...) represents full trajectory sampled by following the policy
Policy optimization. Policy gradient (PG) methods optimize J(TT)) via gradient ascent. In the context of large language model (LLM) alignment, this MDP formalism is adapted: the state 8, represents the initial prompt combined with the sequence of tokens generated so far (U<r), and the action at is the next token sampled from the policy ee (yt The full trajectory 7 thus corresponds to the complete generated response, denoted as y = (3/1 ... gr) ). The reward is typically sparse, with single score r(y) (from a reward model) assigned to the entire sequence at the final timestep.
Proximal Policy Optimization (PPO) (Schulman et al., 2017b) is commonly used for this optimization. The GRPO variant estimates the advantage A(y) for a single, complete response y. This advantage is normalized using a set of K responses, y initial prompt:
".....,",, sampled from the policy for the same
A(y) = r(y) mean mean(r(y (x))))
(2)
(td(r()1:k)
The policy is then updated using the clipped surrogate objective, which operates at the token level:
CLIP (0) E, Al,, -,,,
[s() is the probability ratio, and the per-timestep advantage A, is the trajectory- where r0) Tony [er level advantage A(y) from Eq. propagated back to timestep 1.
Policy entropy. Policy entropy. H-(((-ss)) measures the policy's stochasticity. For discrete action spaces, the token-level entropy is given by Eq. 4. For continuous policies, there are several common ways to ensure actions remain within a bounded space. A popular method is to use squashed Gaussian policy, which outputs a bounded action a = tanh(u) by sampling u from a Gaussian distribution To To(:)s) () =N(u((s), Es)s) parameterized by the policy network. The entropy of this policy is given by Eq. 5. Alternatively, another common approach is to directly sample actions from Truncated Gaussian distribution (+{+))) (N(poss), Eo(s), -1, over the bounded hypercube
3

---

Published as a conference paper at ICLR 2026
[|1. Assuming the dimensions independent, its entropy is given by Eq. 6.
H(To) - -E Exap, v)
log no (1e (velvee,))
(4)
B
H(TT)) -esppe No
log log.N(uluo0s)) (o.))) + Σ log(1 tanh(us)3)
(5)
Σ
D
(1g((oo(())((s)222)) (((s))(B((s)) ()(s)((a;(s)
H((T)) = =Esppt
(6)
2Z
where for the truncated Gaussian entropy in Eq. 6, Φ and O are the PDF and CDF of the standard normal distribution, respectively. We define the standardized bounds as (-1 (o,s(s)) /oo,(s))
(((s) = (1 (=e,(()) (8)), and the normalization constant (s) = + (B,
(s))
Maximum entropy reinforcement learning. Building upon policy entropy, the maximum entropy RL framework aims to maximize the standard reward objective subject to minimum entropy constraint Ho:
max J(xo) S.t. Rap,, [H (+1))] 2 Ho-
Practical algorithms like Soft Actor-Critic (SAC) (Haarnoja al., 2018) solve the Lagrangian dual of this problem. SAC is an off-policy actor-critic algorithm that updates soft Q-function Qp and policy no. The Q-function is updated by minimizing the soft Bellman residual Jo(Ø):
JQ(Ø) = Re (Q at) -y)²
(8)
R(8;, (4) (uutrroo((ee))) 12 (804(81 (l+1) log To To((
(9)
with the target y computed using a target Q-network Qy The target network parameters o' are updated via an exponential moving average (EMA): ø' + TO + T)Ø'.
Ja (0) EA,~D.ar (2 a.) log (a, st)]
(10)
The policy is then updated by maximizing the objective in Eq. 10.
4
THE ENTROPY REGULARIZING ACTIVATION
4.1 THE CORE IDEA: ENTROPY- POLICY VIA OUTPUT ACTIVATION
The core of Entropy Regularizing Activation enforce maximum entropy reinforcement learning on the policy, not through a loss penalty, but via integrating the constraint into the network's architecture via a special activation function at the output layer.
Let parameterized policy fa(s) produce distribution parameters fo(s), where belongs to a parameter space Z. The policy corresponding to these parameters is m. (-(8). We introduce an set 2' g(2). The
activation function g - Z, which transforms the initial parameters final policy, which we denote as Te. is thus given by T-(sss) = (()](( (-1s). The function g(.) is designed to ensure that the policy me satisfies constraint on its expected entropy, for given target entropy Ho:
Egpp, (-mo(-1s)) 2 Ho
This formulation enables the policy satisfy the expected entropy condition while leaving the training objective for 0 free of an explicit entropy term, as shown in Eq. 7. This architectural perspective unifies prior projection methods: for instance, the method Akrour et al. (2019) can be viewed as specific, linear instantiation of g(-). ERA generalizes this class of non-linear activations that strictly satisfy the bound while modulating gradients in a dimension-aware manner, avoiding the suboptimal uniform regularization of prior linear methods. For the facts of this argument, refer the reader to the derivations in Appendix B. I and the experimental results in C.I. where detailed theoretical analysis and experimental demonstrations provided.
4

---

Published as a conference paper at ICLR 2026
4.2 INSTANTIATIONS FOR CONTINUOUS AND DISCRETE SPACES
To ground the general framework presented in section 4,1, we now instantiate the entropy regularizing activation g(-) for two canonical policy classes: policies based a bounded Gaussian distribution, such as the Tanb-squashed Gaussian (Haarnjja et al., 2018) 8) or the clipped Gaussian (Fujimoto et al., 2018), commonly used in continuous control; and the softmax policy prevalent in discrete spaces.
4.2. 1 CONTINUOUS CONTROL WITH BOUNDED GAUSSIAN POLICIES
In continuous control, policies often sample actions from Gaussian distribution and then apply bounding function (c.g., a tanh squash or clipping) to ensure outputs lie within a valid range. This bounding operation complicates direct entropy maximization, it introduces state-dependent bias term. Prior methods typically address this by adding an entropy bonus to the learning objective. Our insight is that the entropy of the final bounded policy, Hx. can be seen the entropy of the original unbounded Gaussian, HGmsssinn minus a non-negative bias term introduced by the bounding operation, the final policy
i.e. Hr. HGaussian -[bias). Consequently, minimum entropy constraint can be satisfied by constraining the underlying Gaussian's entropy to a corresponding. higher value. This is achieved by adjusting the Gaussian's standard deviation, a. The entropy of D-dimensional Gaussian with a diagonal covariance matrix is:
HGasssian (s) = Σ log log(2Te
(11)
To maintain training stability, the standard deviation must also be kept within a predefined range miin o maxx Our activation function g(-) simultaneously satisfies both constraints. Given network outputs (before tanh squash or truncation) for the mean and pre-activation standard deviation o, the function g(1,,) à) produces the final parameters (n', oo) where:
e::
v' n. o' = exp max log mmax + (Hg- Dlog V2Te D log o max)
llog min
Σ/=1 e05
(12)
Here, Hy is the target entropy for the final policy Ho plus a compensation term 8 -E [[ias] to account for the bounding bias. We use parameter 8 estimate 8. In practice, 8 can either be set a constant automatically tuned by learning with the loss in Eq. 13.
L()) B.DD [8(H[[(-11>] Hoo]
(13)
We refer the reader to Appendix A.1 for implementation details and Appendix B.1 for a proof of the entropy bound.
By satisfying the entropy constraint architecturally, our method obviates the need for an explicit entropy term in the objective function. Hence, target of the critic and the actor loss of SAC in Eq. and Eq. 10 can be simplified to the form in Eq. 14 and Eq 15
= R(st, ai) + +BBa, 1 Q (sL41, 0041) log T fort
(14)
Ja (f) = EayDD, (Q az) log ma (s(ar|)) [3)]
(15)
4.2.2 DISCRETE CLASSIFICATION WITH SOFTMAX POLICIES
In discrete classification, regularizing the predictive entropy is crucial for preventing the overconfi- dence that leads to overfitting. ERA provides architectural regularization by enforcing minimum entropy level, analogous to how techniques like label smoothing improve generalization by smooth- ing the output distribution. For softmax policy, we enforce this constraint by transforming the pre-activation logits z into succh that the resulting policy's 's entropy is at least Ho:
|logT + (Cro - logt T
])
z'
h
max
(16)
D
Here, -1 denotes the inverse of -re" (-00, 1). approximated by h-((x=
/2(-1- - In(=)) + In.. We also define Cho = exp(Hln - 1). where 7 2 e is a fixed hy- perparameter (e.g., 4). A formal proof is provided in Appendix B.2.
5

---

Published as a conference paper at ICLR 2026
HamernidBeeeehhh
MBCS MPCULERA
Heattt ResF-CERA
HO PO
0.5
0.5
1.3M
3.004
500 DK
E.M
300K
100.0K
1.5M
3.0M
steps
steps
steps
steps
Figure 2: Main Results of ERA in Continuous Control. Aggregate normalized performance on HumanoidBench (6 tasks, with SAC), DMC (Humanoid & Dog) (6 tasks, with TD-MPC2), HumanoidBench (8 tasks, with FastSAC) and Mujoco Gym (4 tasks, with PPO). ERA consistently accelerates learning and achieves superior asymptotic performance.
In contrast to label smoothing, which applies fixed and uniform regularization, ERA offers greater flcxibility. It allows the model to learn a structured, input-dependent uncertainty distribution, tailoring the regularization to each sample and thus offering greater expressive capacity and potential for improved performance.
4.3 INSTANTIATIONS FOR RL IN LARGE LANGUAGE MODELS
In reinforcement learning for L.Ms,, each token is treated as a discrete action, with the policy defined by canonical softmax distribution. Prior approaches addressing entropy collapse in LLMs-such as the traditional entropy bonus, clip-higher, KL-Cov, and Clip-Cov -o not provide provable entropy lower bound, and are incompatible with the on-policy setting, as they often need the importance sampling ratio or the KL loss term that arises only in off-policy training. In contrast, our method introduces ERA, simple and non- non-invasive activation function that offers theoretical guarantee of a minimum entropy level, effectively resolving entropy collapse in on on-policy reinforcement learning.
In contrast to standard RL settings, the action space is extremely large. In the previous ERA instantiation, each token has lower entropy bound. However, due to the intrinsic structure of natural language, most tokens are nearly deterministic; therefore, directly enforcing high entropy across all tokens is impractical: will lead to unintended tokens and can corrupt the entire response. Furthermore, modifying the internal structure of the model also introduces instability in different training environments, leading to unpredictable behavior.
To address these challenges, propose new instantiation of ERA that is applied after the sampling process. Specifically, responses are first generated using the original model output z, and the advantages computed following the GRPO rule. Then, during model updates, the probabilities of the sampled tokens are reinterpreted as 2', obtained by applying our entropy-regue reguopy-eggllrrized activation. This design leaves the sampling policy unchanged while still ensuring effective entropy regularization.
Formally, when updating model parameters, we apply an activation layer to the logits z to obtain a transformed set z², defined
kz
Heepp Wlow: A, >
(wiow < Hresp S Whigh A, < 0) or A, O,
(17)
22
Heepp Whigh A > 0,
where & 1, and Wlow: Whigh are algorithm-specific constants. Here, A, denotes the advantage of the token, and Hresp is the average entropy of the top 20% of tokens with the highest entropy the response. To balance the gradient between modified tokens and unmodified tokens (details are shown in Appendix B.3), we add another scaling factor on the advantages of modified tokens:
Ar
Hreg < Whow, A. O,
A
A
(wlow < Hresp I Whigh, At < 0) or A, > 0,
(18)
k.A,
Hresp > Whigh, A > 0,
The on-policy GRPO objective becomes:
J(0)= E, (. log (a(()))))
(19)

---

Published as a conference paper at ICLR 2026
where To is the original policy from 2 (representing that the inference still follows the original policy), and Ti is the ERA-adjusted policy from z' (representing that the model update relies on the new policy). Intuitively, this activation layer adjusts all positively advantaged responses: when entropy is too low, it sharpens the probability distribution; when entropy is too high, it flattens it. Unlike our instantiation for control tasks, increasing policy entropy here requires sharpening the distribution. The rationale is that sampling has already occurred, and by treating the samples as if they were drawn from sharpened policy, the model perceives itself as overexploiting, thus encouraging additional exploration. The choice of the top 20% tokens is based on the fact that, in natural language, these tokens are considered forking tokens, whose entropy is the target of regularization, and the remaining tokens are allowed to have almost zero entropy (Wang et al., 2025).
We show that, under reasonable assumptions, this ERA instantiation ensures that the policy entropy remains above fixed constant Ho. We refer the reader to Appendix B.3 for formal proof.
5 RESULTS AND ANALYSIS
5.1 EXPERIMENTS ON CONTINUOUS CONTROL
We conduct extensive experiments to validate the effectiveness of ERA continuous control tasks. We demonstrate the broad applicability and performance gains by integrating ERA into five distinct algorithms- SAC, OBAC (Luo et al., 2024), TD-MPC2, PPO, and FastSAC (Seo et al., 2025). The wide range of challenging benchmarks, including the DeepMind Control
evaluation is performed Suite (Humanoid & Dog), HumanoidBench (Sferrazza et al., 2024), and MuJoCo Gym (Todorov al., 2012). Implementation details, environment specifies, and hyperparameter settings available in Appendix 1. Comprehensive results for all tasks can be found in the Appendix C.
Main results. We present our main results in continuous control in Figure 2. Integrating ERA consistently yields significant improvements in both sample efficiency and final performance across diverse algorithms and benchmarks.
ERA consistently improves performance across various entropy targets. We evaluate the perfor- mance of SAC and SAC-ERA under varying entropy targets. The results in Figure 3a, tested on four DMC tasks (dog-run, dog-trot, humanoid- humanoid-walks with 5 seeds on each environment, show that SAC-ERA consistently outperforms original SAC across the entire tested spectrum of entropy values. By bypassing the entropy constraint within the learning objective, ERA allows the policy to focus more directly on reward maximization. While simply removing the entropy term from SAC can also avoid this constraint, its performance is inferior to the ERA-enhanced version due to insufficient exploration. This consistent outperformance suggests that ERA can achieve strong results without precise tuning of the entropy hyperparameter, offering significant practical advantage.
5.2 EXPERIMENTS ON IMAGE CLASSIFICATION
Table 1: Top-1 and Top-5 accuracy (%) on ImageNet and CIFAR-10. We compare ERA against the original ResNet-50 baseline. A denotes the absolute improvement of ERA. All models trained for 200 epochs.
Without Data Augmentation
With Data Augmentation
Dataset
Method
Top-1 Acc.
A
Top-5 Acc.
A
Top-1 Acc.
A
Top-5 Acc.
A
Original
74.75 + 0.38
92.04 + 0.23
76,93 + 0.36
93.37 + 0.21
ImageNet
ERA
75.44 +: 0.37
+0.69
92.15 +: 0.23
+0.11
77.30 4 0.36
+0.37
93.39 : 0.21
+0.02
Original
93.61 + 0.14
99.69 + 0.08
93.53 + 0.03
99.84 + 0.02
CIFAR-10
ERA
93.82 + 0.08
+0.21
99.82 + 0.03
+0.13
93.93 + 0.12
+0.4
99.86 + 0.01
+0.02
We evaluate our method on the ImageNet (Russakovsky et al., 2015) and CIFAR-1O datasets (Krizhevsky et 2009). Our implementation utilizes the ResNet-50 architecture from the Py PyTorch Image Models (timm) library (Wightman, 2019). To ensure a fair comparison, both our method and the baseline were trained for 200 epochs, with all other hyperparameters held constant. Notably, we retain key default settings from timm for all experiments, including a label smoothing factor of 0.1. This demonstrate ERA's complementarity with existing regularizations.
7

---

Published as a conference paper at ICLR 2026
ImageNet
CIFAR-10
1.r
732
-944
0.3
73.0
-912
940
Top-1 Acc
Ade
T Accc
V1.
Top-5 Ace
0.66
72.8
98.7
0.4
12.6
RoNo-ERA
12.6
Tap-l RoNis
0.2
SAC BRA
72.4
Tip-5 RoiN ERA
914
99.5
ennogy neget
Tip-d RoNie
014
0.0
1.5
-1.0
-0.5
0.0 0.5
722 0.7
16
5.0
5.4
0.5
entropy target
entropy target
entropy target
(b)
Figure 3: Sensitivity of ERA to the Minimum Entropy. (a) 1M Steps Performance on DMC
Tasks. Comparison between SAC-ERA and the baseline SÁC on Humanoid and Dogs environments under various minimum entropy constraints. Our method achieves superior performance across all settings. (b) Accuracy ImageNet and C-FAR-10. ResNet-ERA maintains stable Top-1 1 and Top-5 accuracy across a range of minimum entropy values, indicating its robustness to the choice of this hyperparameter.
Main results. Table I summarizes the primary classification results, comparing ERA against the standard ResNet-50 baseline. For these results, we use a minimal entropy of 1.2 for ImageNet and 0.6 for CIFAR-10. 10. The comparison is conducted under two settings: with and without the standard data augmentation provided by the timm library. The results show that ERA consistently outperforms the baseline across both datasets and settings.
Ablation study on minimal entropy. We study our method's robustness to the minimal entropy hy peppraarreeter on ImageNet and CIFAR-IO, 10, using checkpoints from the 100th and 200th epochs, respectively, for efficiency. As shown in Figure 3b, our method exhibits low sensitivity to this parameter. Rather than fine-tuning for peak performance, our intent is to show that competitive accuracy is maintained across reasonable range of values. This demonstrates strong performance is achievable without extensive tuning.
5.3 RESULTS AND ANALYSIS ON LARGE LANGUAGE MODELS
We first present the results of ERA in $5.3.1 Main Results and $5.3.2 Extension More Models and Algorithms. We then use $5.3.3 Analysis on Entropy and Reasoning Capacity Boundary and $5.3.4 Dut-of-Distribution Generalization to illustrate the role of encouraging exploration. Additional ablation studies on method design are provided in the Appendix C.3.
5.3.1 MAIN RESULTS
We evaluate ERA on @wwe2..5MMath- trained with the DAPO-Math-17K (Yu et al., 2025) dataset using codebase adopted from verl (Sheng et al., 2025). We set whow 0.45, Whigh 3.0, and k 2.
We then evaluate the final resulting model on six standard mathematical reasoning tasks: AIME 24, A1ME'25, AMC'23 (Li et al., 2024b), MATH500 (Hendrycks et al., 2021), Minerva (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). Table 2 presents comparisons against base models, approaches. AIME' 24, AIME 25, and AMC'23
classical RL methods, and recent entropy- are conducted with decoding temperature of 0.7, and reported as the average accuracy over 16 sampled responses. MATH500, Minerva, and OlympiadBench The evaluation process is sampled on the original policy z (before ERA). Full implementation details and hyperparameter settings are provided in Appendix A.3. The results show that ERA consistently achieves the best results on most of the benchmarks. Notably, it outperforms strong entropy-based baselines such as KL-Cov and Clip-Cov by significant margins.
conducted with greedy sampling.
5.3.2 EXTENSION TO MORE MODELS AND ALGORITTMS
To demonstrate ERA's effectiveness across different model sizes and algorithms, extend it to the weaker Qwen2.5-Math. 1.5B model and also apply ERA to other algorithms such as GSPO (Zheng et al., 2025) Swwe2.5--Mah-77- showing that ERA is generic approach not tied to any specific model or algorithm. As reported in Table 3, ERA yields significant gains on both the smaller model and GSPO. For instance, Qwen225-Math-1.55 it achieves average improvement of 14.15 %.
8

---

Published as a conference paper at ICLR 2026
Table Main results (%) on five competition-level reasoning benchmarks based on Qwen2.5 1ren2.5-MMth- 7B. For AIME and AMC, the results are avg. @ 16. The best results on each benchmark are highlighted bold.
Model
AIME24
AIME25
AMC
MATHS00
Minerva
Olympiad
Arg.
Bave Model
Seen2 Yang (2024))
8.6
63
52.2
50.8
12.1
17.2
24.5
Ouen2.S-Math-nneee Yang (20244)
13.3
1000
57.1
81.0
32.7
38.8
38.8
Methods
SimpleRL-ZZroo Zeng et (2025)
26.7
9.3
60.0
74.6
27.6
35.8
39.0
OpenReasener-ZZee Zero Hu et (2025)
15.4
13.4
56.5
$1.0
32.7
43.2
40.4
PRIME-Zeno Cui (2025a)
11.7
57.7
79.0
36.4
40.6
40.7
(2025)
Enirapy Control Methous
GRPO Entropy Loss
32.5
14.0
66.9
80.8
36.0
42.5
45.5
GRPO w/ 20% Forking Tokens (Wang al., 2025)
29.0
17.7
63.6
81,8
39.7
44.6
46.1
2025b) Clip-Cov (Cui sl,, 2025b)
13.7
62.5
78.4
40.3
GRPO (Shao et al,, 2024)
34.4
12.3
69.5
80.6
36.8
40.6
45.7
FRA
36,0
21.0
76.6
85.4
40.1
46.8
51.0
+4.7%
+70.7%
+10.4%
+9.0%
+1..5%
+11.6%
Table 3: Accuracy (%) results of different LLMs and different algorithms across six benchmarks. The best results in each box are highlighted in bold.
Method
1
AIME25
AMC
MATH500 1 Minerva
Olympiad
Avg.
Qwen2.5-Mane 1.5B
Yang (2024a)
CoT
4,3
2.3
26.4
59.0
24.3
27.6
24.0
GRPO
11.1
6.0
40,2
66.4
25.0
30.1
29.8
ERA
12.1
6.8
49,5
70.6
30,5
34.7
34.0
A (t)
+9.0%
+13.3%
+23.1%
+6.3%
+22.0%
+15.3%
+14.1%
Qwen2.5 5- Math-7B Yang et (2024a)
CoT
8.6
6.3
52.2
50.8
12.1
17.2
24.5
GSPO
29.8
61.2
85.1
37.1
35.1
43.7
GSPO ERA
33.3
15.2
63.8
84.3
40.8
42.7
46.7
A (t)
+11.7%
+10.9%
+4.2%
-0.9%
+10.0%
+21.7%
+6.9%
5.3.3 ANALYSIS ON ENTROPY AND REASONING CAPACITY BOUNDARY
To better understand the effect of our approach on exploration and reasoning, we examine both the entropy dynamics of the learned policies and their downstream reasoning performance. Figure compares the entropy trajectories of our method (first stage) with the GRPO baseline. While GRPO suffers from entropy collapse, method maintains a stable entropy level throughout training. This stability indicates the existence non-trivial entropy lower bound, as we desired by the definition of ERA, which prevents premature policy concentration and preserves the model's I's ability to explore diverse reasoning paths.
The presence of this entropy floor correlates with improved reasoning performance. To reduce the effect of stochastic fluctuations, we report the pass@ score as the highest value achieved across all evaluation checkpoints (evaluated every 10 steps). As shown in Figure 4, ERA consistently attains higher pass@ k scores than GRPO on AIME'24 and AIME'25. These results indicate that preventing entropy collapse is not merely a statistical artifact, but directly translates into enhanced reasoning capability. In particular, maintaining sufficient entropy enables the model to preserve diverse candidate reasoning trajectories, thereby increasing the probability of producing correct solutions under @ evaluation.
5.3.4 OUT-OO-DDISTRIUUTIO GENERALIZATION
Models trained in specific domain often struggle when applied to other domains (Yuan et al., 2023; Wang et al., 2024a). Since ERA uses entropy constraints to encourage exploration, we hope it can learn more general skills. Therefore want to see if ERA will also do better (OOD) data than standard GRPO. To test this, we evaluate ERA on three hard OOD benchmarks: ARC-C (Clark et al., 2018), GPQA-Diamond (Rein et al., 2024), and MMLU-Pro (Wang al.,

---

Published as a conference paper at ICLR 2026
Eatropy
Pase Performance -AMME22
Pusslök Performance- ATME25
0.40
GRPO-EIA GRPO
DIRA
GRPO ERA
GRPO
GRPO
0.30
0.55
Passik Score
Eattopy
0.25
0.60
0.4
0.20-
0.3
005
0.40
Step
k
Figure 4: Entropy comparison and pass @k l: results for GRPO with ERA (ours) versus GRPO alone. The entropy curves demonstrate that ERA mitigates entropy collapse and establishes clear lower bound. The pass@A k results further indicate that ERA enhances exploration and strengthens the model's 's reasoning ability.
2024b), As shown in Figure 5, ERA outperforms GRPO by 16.9% on average. This confirms our hypothesis that ERA can also enable models learn more generalizable abilities.
LIMITATIONS AND FUTURE WORK
Our work is centered within the maximum entropy reinforce- ment learning (MaxEnt RL) framework, with the primary objec- tive of imposing effective entropy constraints enhance explo- ration. We have demonstrated its effectiveness across diverse tasks, including continuous locomotion, discrete-space image classification, and the reinforcement learning post-training of large language models.
Basc
GRPO
ERA
254
However, this reliance the MaxEnt objective constitutes potential limitation. The goal of maximizing entropy is not uni- versally beneficial and can lead suboptimal policies in certain task scenarios, as highlighted by Zhang et al. (2025). There- fore, the broader applicability of method in domains where maximum entropy may not be the desired objective requires further investigation.
152
ARC-C
GPQA-D MMLLU-Pro
Figure 5: Results three OOD
benchmarks (Qwen2.5-Math-7B)
A promising direction for future research is to adapt and apply our method to a wider range of domains. This includes areas such as diffusion and flow-based generative models, or other tasks that could benefit from structured policy diversity and efficient exploration, even outside the strict MaxEnt RL paradigm.
CONCLUSIONS
In this work, we introduced ERA, novel entropy-constraine paradigm built upon the unique principle of treating output activations as a direct medium for entropy regularization. Our theoretical analysis substantiated by strong empirical results across diverse and challenging domains. In these settings, ERA consistently surpasses prominent baselines without incurring significant computational overhead. Ultimately, this work offers new perspective on entropy regularization for both supervised and unsupervised decision-makinn opening promising research avenue for developing more robust and efficient learning agents.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We are strongly committed to the reproducibility of our work. To this end, we provide detailed derivations and proofs for all theoretical claims the appendix. The appendix also contains compre- hensive experimental details, including hyperparameters, environment setups, and additional results, which are crucial for replicating our findings. Furthermore, the core source code for our proposed method, ERA, instantiated across all domains, is included in the appendix. As our implementations are built upon publicly available codebases and frameworks, we believe provided key source code is sufficient for straightforward reproduction of our results. Our official implementations can be found at hutps://nothinggbuuuuu githuh. iofera
ACKNOWLEDGEMENTS
This work was supported by the Tsinghua University Initiative Scientific Research Program No. 20257020004. We would also like to express gratitude Kaizhe Hu, Ruizhe Shi, and Huanyu Li from the Tsinghua Embodied AI Lab for their invaluable discussions and insightful feedback, which have significantly contributed this work.
REFERENCES
Riad Akrour, Joni Pajarinen, Jan Peters, and Gerhard Neumann. Projections for approximate policy iteration algorithms. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference Learning Research, pp. 181-190. PMLR, 09-15 Jun 2019. URL https: /ttp:///e/oceeeees//eeeeee mlr.
Machine Learning, volume 97 of Proceedings of Machine
press/v97/ akrour 9a html.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://ar //axxiv org/abs//607 06450.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pp. 679-684, 1957.
Vivek S. Borkar. Stochastic approximation with two time scales. Systems & Control Letters, 29(5): 291-294, 1997.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime:diffusion-bartt based maximum entropy reinforcement learning, 2025. URL
https://arxiv. http://w.wwg///////////oo 25w.. 02316.
Chen-Hao Chao, Chien Feng, Wei-Fang Sun, Cheng-Kuang Lee, Simon See, and Chun-Yi Bi Lee. Maximum entropy reinforcement learning via energy-based normalizing flow, 2024. URL https: // //t.xxv.orggaas////// wtt.orgg/ab///5../// 13629.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758, 2025.
Jean Scong Bjorn Choe and Jong-Kool Kook Kim. Maximum entropy on-policy actor-critic via entropy advantage estimation, 2024. URL https::/arxiv.org/bbb////
18143.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: AS03:1803.05455 2018.
R. M. Corless, G. Gonnet, D. E. G. Hare, D. J. Jeffrey, and D. E. Knuth. On the Lambert W function. Adv. Comput. Math, 5:329, 1996.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025a.
11

---

Published as a conference paper at ICLR 2026
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.226611 2025b.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. URL https://arxiv.cr org/ abs/1802. 09477.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12245 2025.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. Pmlr, 2018.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-lever bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3828-3850, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025. URL
24290.
Sergey Ioffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by 03167.
reducing internal covariate shift, 2015. URL http:://arxxii.or////aa//
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey,, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Zilin Kang, Chenyuan Hu, Yu Luo, Zhecheng Yuan, Ruijie Zheng, and Huazhe Xu. A forget- and-grow strategy for deep reinforcement learning scaling in continuous control, 2025. URL https://arxinn wrtp httpss//arxii..orr//b/////00.. 02712.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 32(11):1238-1274, 2013.
Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021. URL https:tt//// /gi thub. com/ wtt.c//o//ii//ee Ktom///kostriko/jarrl..
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning, 2025. URL https://arxiv. org/ /bs/ 2502. 15280.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances neural information processing systems, 35:3843-3857, 2022.
12

---

Published as a conference paper at ICLR 2026
Chen Li, Weiqi Wang, Jingeheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXhv:2403.04706, 2024a.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. /tttp::/huugingee
///datasees//uminaaattt.t 2024b. Hugging Face repository, 13:9.
Zhi Li. DRL-code-pytorch. https: htas:///iiti httgggtthubb.... cww cessed: 2025-09-10.
2022. Ac-
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off aff-policy arXiv preprint 10XYv:2405.18520, 2024.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy, 2025. URL htttp:://aaxiv.orgg/eee//
00361.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances neural information processing systems, 30, 2017.
Michal Nauman, Michal Bortkiewicz, Piotr Mitoš, Tomasz Trzciński, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity actor-critic: the bitter lesson of reinforcement learning, 2024. URL httpps://arxiv aat///////2// 240.0/0544......... 00514.
Brendan O'Donoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning, arXiv preprint arXiv:1611.01660 2016.
Fabian Otto, Philipp Becker, Ngo Anh Vien, Hanna Carolin Ziesche, and Gerhard Neumann. Dif- ferentiable trust region layers for deep reinforcement learning, arXiv preprint arXiv:2101.09200 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. b: Reliable reinforcement learning implementations.. Journal of Machine Learning Research, 22(268):1-8, 2021. URL /ttt:://jmill http://nro.o.r......////.../oo woo./oogggseeee.. httppppprs//v2/220-0ee 1364, html.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https::/ wtwpeenreiieeeeneeofuuuumm net/oormmmdff Ti67584b98.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015. URL
0t///44990000 0575.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (007:0677006777 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (Tt0:::7700077777 2017b.
13

---

Published as a conference paper at ICLR 2026
Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao- Heng Yin, and Pieter Abbeel. Fasttd3; Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505. 2025.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for shole-body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning open language models. arXiv preprint vrXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang. Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradi- ent methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https::/ppooeeedings nwwr wttppppppcpppppppppppppppppeeepp paappe_fileesppper//Seaae pdf.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826. 2016.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. Scaling reinforcement learning with llms. arXiv preprint arXiv: 12599, 2025.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Wei Ye, Haojun Huang, Xiubo Geng, Binxing Jiao, Yue Zhang, and Xing Xie. On the robustness of chatgpt: An adversarial and
perspective. IEEE Data Eng. Bull., 47(1):48-62 -2, 2024a.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, et Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.00 220.::2500.01999, 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint 240.::2406.01744 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- th-houtt prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Ross
Wightman.
Pytorch image models.
com/ http:///c///wwg/////ee// rwiggtman////////
pytorch- imw-immmdeless,...mmmmmmmmmsssse 2019.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvemmnt, 2024a. URL https: org/ 2rgg 12122.
14

---

Published as a conference paper at ICLR 2026
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self self-improvement. arXiv preprint wxxv:2249.112122 2024b.
Edward Yeo, Yuxuan Tong, Xinyao Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of- thought reasoning in LLMs. In ICLR 2025 Workshop on Navigaling and Addressing Data Problems for Foundation Models, 2025. URL https://openrevieew wwttteeetteeooe wowrumrrmm iirrrmi-ggggLhMMO
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fang Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting out-of-distribution robustness in NLP: Benchmarks, analysis, and LLMs evaluations. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503, 18892, 2025.
Ruipeng Zhang. Ya-Chien Chang, and Sicun Gao. When maximum entropy misleads policy opti- mization, 2025. URL
2506 056155.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071 2025.
Dianyu Zhong, Yiqin Yang, Ziyou Zhang, Yuhua Jiang, Bo XU, and Qianchuan Zhao. Maximum next-state entropy for efficient reinforcement learning, 2024. URL http::ttto/t/eeeiiiiiiieee net/ forum?id=0G6rRLYcck
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438 Chicago, IL, USA, 2008.
15

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 IMPLEMENTATION DETAILS OF CONTINUOUS CONTROL TASKS
A.1.1 CODE IMPLEMENTATION OF ERA IN CONTINUOUS CONTROL
Listing 1: Original Implementation
Listing 2: ERA Implementation
Original implementation from the jaxrl codebase, suggested by
# h_0: target entropy, can be a fixed value or learnable parameter
Ilya
log_std_min, sog_std_mas bounds for log standard deviation action_dim: dimension of the action space
# action dim: dimension the action space
k self.action_die *
log_std_max h_e jnp log (jnp
pre_stds: direct output from the actor network
sqrt jnp.pi jnp.
log_stds k nn. softmao (pre_stds,
log_stde = log_std_min + _od_mad_max log_std_min)
axis = -1) + log_std_max
log_stds jax cwwppclllsssppcceocoo
self.
0.5 (1 tanh (pre_stds))
log_std_min, self log stg_std_maxy
Figure 6: Comparison of the activation function at the actor's output.
We provide the following JAX implementation snippet of ERA for the reader's 's reference, where h_0 the target entropy (Hg in Eq. 12), which can be constant (e.g., dectionddim/2) or learnable parameter. The terms log std min and log std max represent the lower and upper bounds of the log standard deviation, respectively; action dim is the dimension of the action space; and pre_stdi refers to the raw output of the actor network.
A.1.2 ENVIRONMENTS
Figure 7: Visualization of some continuous control environments used our experiments. From left to right: dog-run (DMC), h1-hurdle-v0 (HumanoidBenchh), h hand-slide-vo (Humanoidßencch, humanoid (DMC)
Our evaluation of ERA spans a diverse set of continuous control tasks from three established benchmarks: Mujoco Gym (Todorov al., 2012), DeepMind Control Suite (DMC) (Tassa al., 2018), and HumanoidBench (Sferrazza et al., 2024). For the Mujoco Gym and DMC environments, we utilized their standard, unmodified configurations. For HumanoidBench, we introduced specific modifications for certain agents.
For experiments involving SAC and OBAC on HumanoidBench, we implemented an action repeat of 2 and disabled episode termination. These adjustments were necessary because the standard tasks proved exceedingly challenging for baseline SAC agent, as demonstrated Figure 8. Con- versely, for the FastSAC agent, which is capable of solving the original tasks, used the standard HumanoidBench environments without these modifications.
For comparison against TD-MPC2 DMC environments, used the performance data reported in the original manuscript. We therefore adhered to their experimental setup, which includes an action repeat of 2.
16

---

Published as a conference paper at ICLR 2026
For main results and training curves, report results over 10 random seeds for SAC, OBAC, and FastSAC, 5 seeds for PPO, and seeds for TD-MPC2, matching the number provided its original publication.
h1-run-v0
h1 h1-crawl-v0
hl-walk-v0
1000
1000
1000
W/O Modiiiiiiioss
WiB. ModiEoationn
Retum
Return
Retur
0.00
1.5M
3.0M
1.5M stops
3.0M
1.5M
3,0M
stops
steps
Figure 8: Ablation of Environment Modifications for HumanoidBench.. Performance comparison of standard SAC agent on three challenging HumanoidBench tasks with and without our modified settings (action repeat and disabled termination). The significant performance gap justifies using these modified settings for our main SAC-based experiments.
The action, observation spaces and maximal episode length the respective environments are shown in Table and Table 5.
Table 4: List of tasks from DeepMind Control and MetaWorld which the agents were ablated. The table also contains the dimensions of action, observation space and maximal episode length.
Task
Observation dimension
Action dimension
Max episode length
DEEPMIND CONTROL
Dog-Trot
223
38
1000
Dog-Wit Walk
223
38
1000
Dog-Run
223
38
1000
Humanoid-BB
67
24
1000
Humanoid-Wall
67
24
1000
Humanoid-Stan
67
24
1000
MUJOCo GYM
HalfCheetab-v4
17
6
1000
Ant-v4
27
8
1000
Hopper-v4
11
3
1000
Walker2d-v4
17
6
1000
A.1.3 PSEUDO CODE OF SAC-ERA
To better illustrate the role of our method within the algorithmic framework, we present the pseu- docode for a representative example, the Soft Actor-Critic (SAC) algorithm, adapted with ERA in Algorithm I.
A.1.4 HYPERPARAMETERS
We present the hyperparameters used in experiments with SAC and PPO in Table 6
Our implementations of SAC and OBAC are heavily inspired by the official jaxrl reposi- tory (Kostrikov, 2021). For the network design, follow the insights from Nauman et al. (2024) and incorporate LayerNorm (Ba et al., 2016) into the neural networks.
Our OBAC implementation is built upon the codebase provided by Kang al. (2025). It shares the same fundamental hyperparameters as our SAC implementation, with the behavior cloning weight set to 1 10-3
17

---

Published as a conference paper at ICLR 2026
Table 5: List of tasks from HumanoidBench on which the agents were ablated. The table also contains the dimensions of action, observation space and maximal episode length.
Task
Observation dimension
Action dimension
Max episode length
hl-walk-v0
51
19
500
h1-run-v0
51
19
500
h1-stand-v0
51
19
500
h1-hurdle-v0
51
19
500
hl-stair-vO
51
19
500
h1-crawl-vO
51
19
500
hhand-balance
suunnd-allane_simplleeee simple-v0
164
61
1000
h hhanddhhrddlee
151
61
1000
h hhandpole-vo
151
61
1000
h hand-push-vo
163
61
1000
h hhand-stair-vO
151
61
1000
h I
151
61
1000
hand-walk-vvv
151
61
1000
hlhand-run-we
151
61
1000
Table 6: Comparison of hyperparameters for SAC and PPO.
Hyperparameter
SAC
PPO
Optimizer Settings
Actor optimizer
Adam
Actor learning rate
3 x 10-4
Critic optimizer
AdamW W
Adam x 10-4
Critic learning rate
Temperature learning rate
3 10-4
Adam epsilon
x 10 0.5
Gradient clipping
Network Architecture
Actor/Critic network
3-layer MLP (64, 64)
Hidden layer dimensions
(512, 512) ReLU
Activation function
Tanh
LayerNorm
True
False
Algorithm Hyperparameters
Discount factor (7)
0.99
Replay buffer size
1 106
Polyak averaging coefficient (T)
0.005
Initial temperature (a)
1.0
Target entropy (Ho)
dim((A)2 (A)/2
Gradient steps per step
2
Random exploration steps
5,000
GAE parameter (A)
0.95
PPO clip ratio
0.2
Entropy coefficient
0.01
Batch size
256
2048
Mini-batch size
64
Log std Interval [Amin, maxx]
[-8.0] for ERA, [-10,2] for baseline
For the PPO and PPO-ERA experiments, our implementation is based on the publicly available codebase Li (2022). We use target entropy of -0.3.4 for main experiments on PPO-ERA.
For the TD-MPC2 baseline, utilize the official implementation provided by the original authors. The results for comparison are also directly sourced from those reported the official repository. We target entropy of - for main experiments TD-MPC2-ERA.
18

---

Published as a conference paper at ICLR 2026
Similarly, implementations of FastTD3 and FastSAC are based on the official codebases provided by their respective authors. We note that our construction of FastSAC-ERA differs from the method described in the original paper; these differences detailed in Section A.1.5.
A.1.5 FASTSAC-ERA
The FastTD3 (Seo et al., 2025) framework demonstrated the potential of applying off-policy RL methods massively parallel RL scenarios, achieving excellent performance on HumanoidBench.
Authors of FastTD3 also provided FastSAC implementation, which replaced the mixed noise mechanism in FastTD3 with the standard entropy maximization objective from Soft Actor-Critic (SAC). However, they noted that this approach yielded unstable results, and hypothesized that maximizing action entropy in high-dimensional action spaces might be inherently challenging.
To address this issue, we investigated a solution based on minimal modification to the original FastTD3. Our approach, named FastSAC-ERA, is derived from FastTD33 by retaining its noise mechanism while removing the Delayed Policy Updates and incorporating an entropy constraint via ERA implementation. This method achieved performance superior to that FastTD3
In practice, our implementation was built directly upon the official FasITD3 codebase. The only mod- ifications were the removal of Delayed Policy Updates and the addition of the ERA implementation the actor's 's output. All other hyperparameters and implementation details were kept identical the original FastTD3 configuration.
A.1.6 IMPLEMENTATION DETAILS: NORMALIZED SCORE COMPUTATION
In this work, we use normalized scores to evaluate and compare algorithm performance across multiple environments. The rationale for this is that when aggregating results across a benchmark, raw scores can allow environments with disparate score ranges to have a disproportionate influence on the final result. Normalized scores mitigate this by mapping all results onto uniform scale, enabling a more equitable comparison and aggregation.
When calculating the normalized score, we uniformly use the minimum and maximum scores achieved among all tested algorithms in that specific environment as the normalization bounds, rather than relying on the environment's theoretical minimum maximum scores. This approach avoids distortions that can arise from theoretical score ranges being exceptionally large or small, thereby providing more accurate reflection of the algorithms'' relative performance in practice.
Specifically, the normalized score is calculated using the following formula:
Normalized Score Algorithm Score Min Score
Max Score Min Score
Where Algorithm Score the score of a given algorithm in a specific environment at a particular time, and Min Score and Max Score are the minimum and maximum scores, respectively, achieved among all participating algorithms in that same environment.
To compute aggregate normalized normalized score for each algorithm within each environment. We then average these scores. This method ensures that each environment contributes equally the final metric, providing a more comprehensive and fair assessment of overall algorithm performance.
across multiple environments, first calculate the
A.1.7 IMPLEMENTATION DETAILS: SHADING AREAS PLOTS
For aggregated performance plots 2, we use 25% and 75% percentiles to create shaded areas around the mean performance curves. This choice corresponds common practices in RL community.
For training curves of individual environments the appendix, we use 95% confidence intervals to create shaded areas around the mean performance curves. This choice provides clearer depiction of variability in individual environment results, which can be more pronounced than in aggregated plots.
A.2
IMPLEMENTATION DETAILS OF IMAGE CLASSIHICATION
A.2.1 CODE IMPLEMENTATION OF ERA IN IMAGE CLASSIFICATION
19

---

Published as a conference paper at ICLR 2026
Algorithm 1 Soft Actor-Critic (SAC) with ERA
1: Initialize: actor parameters 9, critic parameters Ø1, Ø2.
2: Initialize: target network parameters &1 +- Φ1, 0/2 + 92.
3: Initialize: replay buffer D.
4: Hyperparameters: learning rates X Aq, Aas target entropy Ho, Polyak coefficient T.
5: for each training step do
6: Sample action from the policy: a₁ ~ (T(+)))).
7: Execute action d,, observe reward r; and next state 8t+1-
8: Store transition (St, Qt, Ft, St41) in replay buffer D.
9:
Sample random minibatch of transitions B {(s, a,,, from D.
10:
// Update the Q-functions (critics)
Sample next actions: a' <T(--s').
12:
Compute the target Q-value by taking the minimum of the two target critics:
Quagar (s' d') +- min Qel (s',
1=1,2
13:
Compute the Q-target y (matches Eq. 14):
- r + 7 Vunget (s', d')
14:
Update both crities by step of gradient descent using the loss from Eq. 8:
Σ (s,0,))BB
2 (Qos a) - 8)² for i = 1,2
B
// Update the policy (actor)
16:
Sample new actions for the policy update (using reparameterization trick): à ~ me(-ss). Compute Q-values for the new actions using the minimum of the two crities:
min (s, a)
min Qa. (s,
1,2
18:
Update the policy by one step of gradient ascent to maximize the objective from Eq. 15:
V
Qmin (8, a))
B
SEB
19:
1/ Update target networks using Polyak averaging
20: 81 + TØ: + (1 - -) for i = 1,2
21: end for
We provide the implementation of ERA for image classification tasks in Listing 3. In the code, C_H corresponds Cho defined in Eq. 16, and n_dims denotes the number of classes. We set 4 in our implementation without performing any tuning for this parameter.
A.2.2 TRAINING SETUP
Our training for ImageNet was completed on A100 GPUs, and we report the 95% confidence interval calculated from the dataset. For CIFAR-10 -00 which requires less computation, we trained three separate runs on machines, each with A40 GPUs, and report the confidence interval computed from these three results to ensure maximum reproducibility.
A.2.3 COMMANDS USED FOR EXPERIMENTS
We provide two main commands used for training in image classification. The two commands delineate the training procedures for models under two distinct settings: one incorporating data augmentation and the other without it. The training commands were sourced directly from the reference ImageNet training script within the timm library. We employed this identical set of
20

---

Published as a conference paper at ICLR 2026
Listing 3: ERA Implementation in Image Classification
class ERA(nn, Module):
def __initt__seef C_H: float, n_dims: int):
()
self._tau = 4.
self. = C_H
self n_dims n_dims
self. upper wwp.uupppuuuete bound math. log (self._tay) / self._tau
assert C_H >= self upper_ppp__ou self slope = (self.. upper_bound C_H n_dims) / (1 - n_dims) self.b (C_H self.slopp / n_dims
def forward x: torch. Tensor) -> torch. Tensor:
x: logits before softmax, shape (..., n_dims)
return: adjusted logits before softmax, shape (..., n_dims)
**n
self. slope softmax (dim=-1) + self.b
-1 torch.log(h)
new_logits (-1 torch. (2 u) 3/4 * u) to dtypepe))
max_values torch. (x dim=-1, keepdim=Trus values detach () x nax_values
min_values = torch. min (new_logits, dim=-1, keepdim=True) values. ()
new_logits new_logits min_values
return new logits
commands for training on both the ImageNet and CIFAR-10 datasets without any dataset-specific hyperparameter tuning to ensure a consistent experimental setup.
Listing 4: Command launch training with data augmentation.
/ddistribute__train sh 4 --da --data -iir .afrr../daaaaa /ata -- --dataset torch cifrr10 -- dataset-download -b 64 --model re --sched cosine --epochs 200
--lr 0.05 aamp --remode pixel --reprob 0.6 --aug -ppitss
rand ms tde 5-inc1 resplit --split bnn --jsd --dist-ba reduce
Listing 5: Command to launch training without data augmentation (baseline).
idistributed_train sh --data -iir /ata -- --dataset torch/ parch/cifar@@ + dataset -b 64 -- --model resnet5e -- --sched cosine --epochs 200 lr 0.05 amamp --dist bn reduce
A.3 IMPLEMENTATION DETAILS LLM TRAINING
A.3.1 CODE IMPLEMENTATION OF ERA IN LLM
We provide the core implementation of ERA in LLM in Listing 6. In the code, era_lb, era_ub and era_k corresponds to Whow, whigh & defined in Eq. 17, respectively. In the first training stage, we further apply top-k filter (retaining the 20 largest logits) within the logprobs_ fro___ro______ from_logits function to enhance training stability. In addition, provide advantage scaling as option, as removing such scaling would reduce the update to pure logit shift. The impact of advantage scaling is discussed in C.3.1.
21

---

Published as a conference paper at ICLR 2026
Listing 6: ERA Implementation in LLM
length response_mask _makks
mtt.mmassk.uuus. sum (dim=-1)
k_per_ sample (0.2 length) long clamp (min=1)
mean = []
(rreryy..astddiillrrrsooooosnoe masked_entropy entropy masked_fill (*response_mask bool O, float ("-inf")
for in (ew.eeettoosssi(ee(""
top_entrapy_b, = torch. topk (masked k)
mean_top_entropy. append (topottttroppp____ mean
mean_ rean__to_enttre top_entropy torch. stack waannnttopeeerrrpe unsqueeze (-1)
cond_A (mean_top top_too__eeeee top_entrop era_lb) (advantages > 8)
cond_B [mean_top_entropp > era_ub) & (advantages > 0)
[ogits[cond_A] logits[cond_A) era_k
logits[cond_B] Logits[cond_]] / era_k
if adv adv_scale:
advantages [con_AL advantages [cond_AA] / era_k
advantages advantages [cond_B] era_k
log_prob = logprobs_from_log from_logitss (lagits)
A.3.2 HYPERPARAMETERS
For GRPO, GRPO w/ 20% Forking Tokens, ERA, we use a training batch size of 256 and mini batch size of 256 in the verl configuration, which results in on-policy setting. For KL-Cov and Clip-Cov, we use training batch size of 256 and a mini batch size of 32, and other hyperparameters are consistent with their original paper. GRPO + Entropy Loss coefficient 0.002. The learning rate is 10-6 and no learning rate warm-up or scheduling is applied. We also utilize dynamic sampling to enhance training efficiency. Since our setting on-policy, the clip ratio is irrelevant. The maximum response length is 8192 with no overlong reward shaping, For Owen2.5-Mart 1.5B, we use MATH problems of levels 3-5 as the training set in this experiment since DAPO-Math-177 17K is too difficult.
entropy regularization term with
The hyperparameters ERA are fixed to whw 0.45, Whigh 3.0, and 2 across all settings, without any tuning. These values are chosen with reference to the initial entropy of the model, Hresp Rs 1.5, such that wlow and whigh lie below and above this value, respectively. We believe results are not sensitive these ERA hyperparameters.
B PROOFS AND DERIVATIONS
B. PROOF OF ENTROPY BOUND IN CONTINUOUS SPACE
In this section, we provide rigorous analysis of the Entropy Regularizing Activation (ERA) for continuous control. We proceed three steps:
1. Static Guarantee: We prove that the ERA functional form structurally guarantees the entropy lower bound, provided the parameter is sufficiently large.
2. Dynamic Convergence: We prove that the learnable parameter converges to the required value under coupled policy updates, using two-timescale stochastic approximation theory.
3. Non-negativity of Bias: We prove that the entropy compensation term 8(s) is neg-negative for both Squashed and Truncated Gaussian distributions.
22

---

Published as a conference paper at ICLR 2026
B.1.1 SETTING AND DEFINITIONS
Recall the continuous form ERA. For state and network outputs (u(s; (1,;;),((ss) (s; 0)), the activation maps to the final standard deviation o':
e.,(0;0) Σ (
]
of 8; exp max log log Fng (Ho + C)
log min
(20)
where C = D log v 2ne + D log max- Here, Hn is the target entropy, and å is a learnable parameter intended compensate for the entropy bias dnias()) induced by the bounding function (e.g.. Tanh or Truncation).
The actual entropy the final policy T0,8 is given by:
(1 = Hausssia (m(8; 0), diag(o'(0,); òpias (8),
(21)
where HGanssiam
2 log(2 EE= log d.
B.1.2 STATIC ENTROPY BOUND
Proposition 1. Given a target entropy Ho and residual entropy parameter 8 2 Smaa(s), the policy defined by Eq. equation 20 satisfies ((T) 2 Ho, and o' is strictly bounded within [Tminn maax)
Proof. The entropy constraint H(T) 2 Ho is equivalent to HGamssian Enias (8) 2 Ho. Substituting Gaussian entropy formula, we require:
D
Σ log of 2 Ho + Spias (s)
2 (og(2Tte)
(22)
From Eq. equation 20, noting that the max operator ensures of 2 Omin+ consider the term inside the exponent:
D
D
Σ
log
of
log max + (No + -C)
(23)
D
j=1
e
D log max + (Ho 8-
(24)
D
e
j=1
D log max + Ho + 8 (D log V2Te D Dlog maxx)
(25)
D
Ho 8 llog(2m))
(26)
Comparing this result with Eq. equation 22, that if 8 (s). the condition is satisfied. Furthermore, the functional form explicitly constrains outputs via max(-, log amin) and log Fmax (in the softmax upper bound), ensuring o' E [omin max]]
B.3. CONVERGENCE UNDER COUPLED UPDATES
We now prove that automatically converges to the necessary value to satisfy the constraint, even when the policy parameters are updating simultaneously, We utilize the framework of two-timescale stochastic approximation (Borkar, 1997).
Update Rule. The parameter is updated to minimize the loss L(8) ((H((T)) Ho). leading to the gradient update:
8 + 8, + 3,Ho - H(TT..
(27)
Assumptions.
23

---

Published as a conference paper at ICLR 2026
(A1) Regularity: The mappings p(0) and o()) are continuously differentiable with bounded gradients.
(A2) Non-saturation: The optimization operates in regime where the ERA activation is not fully saturated at the lower bound mmin for all dimensions. This ensures UN > 0.
(A3) Timescale Separation: Let {au} and { be the step sizes for 0 and ê respectively. We assume å updates on a faster timescale: lime-+o0 = 0, alongside standard Robbins-Monro conditions (E ar = 00, D f < oo, etc.).
Lemma (Monotonicity). Under (A2), for fixed 0, H(T 8) is strictly monotonically increasing with
respect to 8.
Proof.
B Ing d
O. Since H & log d/ it follows that 2H 0.
86
Σe
86
Proposition 2 (Global Asymptotic Stability). Under the stated assumptions, the coupled iteration
(0,, 8t) converges such that & asymptotically tracks the equilibrium 8* (0.) satisfying H H(To,,5-) Hy-
Proof. We analyze the system dynamics in two timescales:
Fast Timescale (ô-update): Since az + 0, 0 is viewed quasi-static. The dynamics of å
follow the ODE: 8(1) Ho H( Define the Lyapunov function V ) 8 (0))
where 8" (0) is the unique root of H (TT.5))) Ha. The time derivative is V (ō 8* 5 H())).
By monotonicity, 8 > 8*, then > Ho. implying V < Thus, converges globally to 8* (0).
2. Slow Timescale (0-update): By the theory of two-timescale stochastic approximation, tracks 8* (0c) almost surely. The policy update O, effectively proceeds along the manifold M = [0, H( = Ho}, solving the constrained optimization problem.
3. Robustness (Finite Step Sizes): practice, if a Be is bounded but non-zero, the system is Input-to-Stan State Stable (ISS). The policy update 9 acts as a bounded disturbance. The entropy error is bounded by the ratio of the disturbance magnitude to the controller gain:
lim sup [M((T)))
Hol S C suppar
This guarantees that the entropy remains bounded within a small neighborhood of Ho.
B.1.4 NON-NEGATIVITY OF THE BIAS TERM
Finally, we show that the bias term Ósins()) (5) in Eq. equation 20 is non-negative, iustifgi the form of our compensation.
Case 1: Tanh-squashed Gaussian. The bias is given by 6ranh -B log(1 tanh?(u.) (u.)) Since - tanh² (w) (0, 1], its logarithm is non-positive. Therefore, the negative expectation is non-negative: Seanh 2 O.
Case 2: Truncated Gaussian (TN), Let Torig = Nu,, o²) be the original Gaussian distribution and TIN be the truncated distribution restricted to the interval [-1, 1]. The bias is defined as the entropy difference: 8IN h(TTagg) h(WIN).
To rigorously prove ôTN 2 0, we introduce a moment-matcher Gaussian distribution मे = A (MTT, UTN), where MyN and vrn denote the true mean and variance of the truncated distribu- tion TIN. The proof proceeds in two steps:
1. Maximum Entropy of Gaussians (h(R)) ((TTT)))))))) )): Among all continuous probability distributions with a fixed variance, the Gaussian distribution maximizes differential entropy. Since the constructed distribution स is Gaussian and shares the exact same variance eTN as TYN, its entropy must be greater than or equal to that of TTx:
h(T) > h(TTN)-
(28)
24

---

Published as a conference paper at ICLR 2026
2. Variance Reduction by Truncation (h([Torig) h(*)): The entropy of Gaussian dis- tribution is monotonically increasing with respect to its variance, given by h(N xo)) = 2 (o Therefore, showing ((
h(=) ) is equivalent to proving that truncation
reduces variance, i.e., UTN S o².
We prove this inequality analytically by examining the sensitivity of the truncated mean MTTN with respect the original location parameter μ..
First, it known result in truncated statisties that the derivative of the truncated mean with respect to the location parameter is exactly the ratio of the truncated variance to the original variance:
OTITN 0μ
TTN o²
(29)
Second, we bound this derivative using properties of log log-concave functions. The truncated mean can be expressed terms of the normalization constant Z(u) = 2(3) - 2(a) as:
TN / 01n In (())
(30)
Differentiating this expression with respect to yields:
θMTNN + o² 82 In 2(n)
(31)
The normalization term Z(p) can viewed as the convolution the standard normal PDF <(-) and the indicator function of the interval [-,, Since both the Gaussian PDF and the indicator function of a convex set are log-concave functions, and the convolution of log-concave functions preserves log-concavity (Boyd & Vandenberghe, 2004), Z Z(u) is log-concave in μ..
By definition, the second derivative of the logarithm of concave function is non-positive. Thus:
8 In (p))
S 0.
(32)
0n2
Substituting this inequality back into Eq. equation 31, we obtain the upper bound:
OMTN <1.
(33)
0y
Finally, combining this bound with Eq. equation 29, arrive at:
UTN 02
< 1
-
UTN S 0².
(34)
This strictly implies (TTTrr))) 2 h(T)).
Conclusion:
Summing the inequalities established in steps 1 and 2, we have: h((aaig) 2 h h())) h(TIN).
(35)
Consequently, the bias term 6TN = h((Tori)) - (TTN) is guaranteed to be non-negative.
B.2 PROOF OF ENTROPY BOUND IN DISCRETE SPACE
Recall the discrete form of ERA:
logg
+ (CHo - n log T
0 )
max
T
T D D-1
1
D Σド=
Before we delve into the proof of its entropy bound, we first provide some insights into the design of ERA in the context of vision tasks. To adapt the entropy constraint function from continuous spaces for discrete domains,
initial idea was to have the network output the entropy of individual components rather than their logits. However, this direct approach is problematic because the function H(p) lp p is non-monotonic the interval [0, 1]. This ambiguity means given entropy
25

---

Published as a conference paper at ICLR 2026
h(x)
xe* and its Inverse Approximation
-xex
(x)(xvevers))e)
635
Plot of h(xx) -c and its Inverse Approximation (x). We reverse the x and values for h-1 to visualize the inverse relationship(The more two curves overlap. the closer the approximation). We can see that the approximation very close to the true inverse function.
Figure 9:
value cannot be uniquely mapped back to its corresponding probability; for instance, an entropy of 0 could correspond probability of either 1.
To resolve this ambiguity. we introduce scaling factor T > and consider a "r-divided distribution," where each probability scaled down by T (We note that the 7-divided distribution is not actually valid probability distribution, but tool for analysis). By selecting T >> E, we ensure that the function -plpp p is strictly monotonically increasing on the interval [0. 1/r]- This establishes one-to-one mapping. allowing for the unique recovery of probability value from its entropy within this restricted range. Therefore, our network is designed to output the entropy of this r-divided distribution. We then map these entropy values back to logits using an inverse function, / h-1 Note that an entropy value is H = -p In for some p E [0, 1/r]. From logits to entropy, we have the following mapping function:
h(x) = In r exp(x) exp(xx
(36)
Therefore, the inverse function h-1 maps entropy values back to logits is exactly the inverse of -I -xexp(x), we have h (x) = (-r).. W is known as the Lambert W function (Corless et al., 1996). Since there no closed-form solution for the Lambert W function, we utilize numerical approximation h h-1(x) = - / 2 In(x)) + 3 In We derive this approximation from (1 x + In(-x)) = -1 v 2r 3 normalization step required because the resulting probabilities from this inverse mapping do not inherently sum to one.
1 x. Here "-1" denotes the inverse function. A final
Crucially, we have proven that the entropy loss during this normalization process is bounded. By leveraging the continuous-space entropy constraint function to ensure the initial output entropy is above threshold Cha, we can guarantee that the entropy of the final discrete distribution will also exceed Cho- This constitutes the core mechanism behind the implementation of ERA in discrete spaces.
Proposition 3. Given a target entropy Ho and hyperparameter T 2 e, the policy defined by Eq. 16 has entropy H(T)) 2 Ho.
Proof. We denote = max log + (CHo log7
D-I
D 55 ), 0). Similar to the continuous
case, we have bounded within [0, log and Eb Ki 2 Cho.
We denote the probability of the final
softmax policy p softmax (2')
Then have:
H(T)) Σ
p. log ps
1=1
Σ
D
(me) h- (KB))
+ logl c (Ns))
D
Σド= ch 1(xx)
D
j=1
> 1 + log(
*((x)) (o4i)/111 ((mz))-((ks))
(37)
1=1
26

---

Published as a conference paper at ICLR 2026
Recall that h -x oonzoee so h h- =Ino(-xlna)
Hence have:
H(T) 1+ + log(
D
+ log
N1) 2 1 log(CHoo) = Ho
(38)
B.3 PROOF OF ENTROPY BOUND IN LLMS
Recall the definition of the ERA instantiation for LLMs:
kz
Hresp < Whows As > 0, (wlow S. Hresp S Whigh; A. 0) A, 0,
Hresp > Whigh: At > 0,
and
E Ae
Hrep Whow, Ar >
A
A
(wrow Heep S Whigh A < 0) or A, 0,
AAA,
Hresp > Whigh: A >> 0,
where are the logits, A, the advantages, and Hresp is the average entropy of the top 20% of tokens with the highest entropy the response.
These transformations are applied after sampling. The modified policy-gradient objective is therefore
J(0) Ee(Baaraoolla log (ar (8x) A1]
Intuitively, when the entropy too low, ERA sharpens the policy: when it is too high, ERA flattens By rescaling the advantages of modified tokens, we show below that ERA is equivalent augmenting the vanilla policy-gradient gradient objective with an adaptive KL regularizer. This KL term guarantees that the entropy of responses remains in the interval [Wiow, whigh). preventing entropy collapse. Under mild assumptions, we derive a positive entropy lower bound.
Fixing the state St, denote na (a)ss)) Hh Tt (a|st), and A, the advantage of action a. The entropy is H Ea log We first derive the gradient of the entropy.
Lemma 2.
0H
Ölog Ta
(Ta' log Tar + FTa
0=
Ōza
Σ (la = a'l Too log Taa + Ta.)
- -loggg n. H).
(39)
We also define the w-weighted covariance that will be used later: Definition 1. Define the n-weighted covariance for two vectors (za). y (Va) by
Cov+(x.V) Σ TaPaya
Σ TaFa
(Σ Kayaa
Now we show our main result:
Proposition 4. Let Ty be the base policy and vi the ERA-adjusted policy from Eq. equation 17. Suppose that:
27

---

Published as a conference paper at ICLR 2026
(i) (Logit approximation) The change in entropy be approximated by treating logits the effective policy parameters and using first-order (infinitesimal) sensitivity of entropy w.r.t 2.
(ii) (Positive advantage mass) The aggregated positive advantage restricted to the tokens con- sidered in Heep
C(s:)
Σ 0,Ag20
Aa
satisfies C(ss) 2 7 for some > O.
(iii) (Bounded response entropy) In some intermediate point of the training process, Hresp has lower bound min and upper bound Whigh-
(iv) (Bounded PG-induced entropy decrease) We assume the vanilla policy-gradient term's 's expected effect on entropy is bounded as
ElCovx(TToAn log Ro)l S aHH,
for some ar 0 and any fixed H, where H denotes the entropy of the current policy n.
(v) (Bounded KL-induced entropy decrease) We assume there exists constant By (thar depends on k and Hmin) such that
Cov Tas log Ta) 2 BgH,
If 7By >> for then there exists a constant Ho 0 such that the response entropy satisfies
H[H Ho
under ERA updates using gradient flow approximization.
Proof. When Hrep Whow: ERA sharpens positively advantaged actions. Following the derivation, the ERA-adjusted gradient satisfies
En'ra log T':
aza
θ aza
Balns (Aa' > log Ttas T. Aa' + [A < <0] Te Aui
Ea'ns [Aa' > O] Blog TT Oz
8 2og Taa
Aa' + [An' <
Aa')
0= Iz.
Izo
En'ns ([Aw O]([a' = a] ) Aa' + [Aa' < 20[[a' a) ne )ar)
A, Σ
Tha An
Σ
An'
(40)
30000
Since the expectation of advantage is zero, and we have defined C(st) = Ea' Ag Taa Aa' yielding
0z. Banna log The A'. = nA ((s))(TT -Ta)-
(41)
For the vanilla loss, this reduces
Oza Ew log Ta Ta Ad = TgAa
(42)
Meanwhile, by similar derivation, the gradient of the KI. divergence is
8
[L[
End log na. na T
(43)
az
3za
28

---

Published as a conference paper at ICLR 2026
Thus, by combining equation 41, equation 42 and equation 43, the ERA-adjusted objective can be written as
J' = I,, [B log To (apls))) +sg(C((s)) (- To [-(((s))]]
(44)
Jea
Jka
where the sg(-) denotes the stop gradient operator. For the other case wlow S (we have assumed that Hresp S Whigh- the same structure holds; only the definition of T/ changes. Hence, ERA is equivalent policy gradient objective augmented with an adaptive KL regularizer that sharpens flattens the distribution depending on Hresp and also the value of C(ss).
We will evaluate the instantaneous directional derivative of entropy along these gradient directions (this corresponds to the first-order change in entropy under an infinitesimal step in the indicated direction).
Using equation 39, the first-order change of entropy caused by JPG is
AHG
Σ
OH
na Aa
Ozo
Σ na (log na + H) na Ao
Σ Aa Ta H)
-ov (To Aa, log To)-
(45)
By assumption (iv) this term is bounded below by -all:
E[AHpa] 2 -aH.
Thus the vanilla policy-gradient component can decrease entropy, but by no more than afl in magnitude. Similarly, the KL-term directional derivative
AHKL
Σ
0H Oza
(To -Ta)
Σ Ta (log Ta + H) (Ta - T'a)
Σ Ta - Ta)(log To + H)
-Co= - Ta, log na)
(46)
By assumption (v) we have Cov=(TT Tay log TT) 2 BgH. Using assumption (ii) C(st) 2 therefore yields
((ss)HHkL 2 B..
Combining the two contributions,
[[ EEAHpG + C(s)) 2 -all + 7BBlll (y a.
By the hypothesis BB 3 we have AH >> 3H whenever H > 0 and H is in the sharpening regime. Thus, if Hresp drops below Whos the FRA-induced update produces positive first-order increase in entropy proportional to Hresp- Consequently the dynamics push Hresp upward until it leaves the sharpening regime (i.c., until Heesp 2 whow or the KL-term no longer sharpens).
Formally, when Hresp < Wlow have E[AH] 3.Hrspp and when Hresp 2 Wlow we have BAH >
-alles Therefore, the overall expected change entropy is at least
3 Summ [Erepp]
a 11 Zama (Hresp)
(47)
29

---

Published as a conference paper at ICLR 2026
Applying Markov's inequality gives Pr Pr(Hresp 2 Wlow) S p//howw where μa [espp) Further, by
assumption (ii)): Hmin S Heesp I Whigh we obtain the sufficient condition to make the expected
entropy change non-negative:
Hihhigh
B M a
(Wlow
11) min
The entropy expected to increase (R[AH) 0) whenever the term in this inequality holds. Solving for μa, we find the condition:
Burow H Hmin
p. S
OWWhigh + 3.mmi
Then we set Ho as
Bukow Hmin
Ho
OtWhigh + H
Under the gradient-flow approximation, we have
d [essppp 2 0 whenever Ei S Ho. di
By assumption (iii), there exists a time to such that Heggl 2 Hy at to- Then, by the principle of differential inequalities, the ERA objective ensures that E H respl stays above this threshold for all
t lo.
We now justify the assumptions made in Proposition 4.
(i) The first assumption, namely approximating entropy differences by treating logits as policy parameters, is standard and also adopted by (Cui et al., 2025b). This simplification is essential for analytical tractability; without it, the theoretical analysis the model's behavior becomes prohibitively complex.
(ii) Recall that C(st)= = Ee,,, An measures the aggregated positive advantage, which reflects the "importance" of a token. Intuitively, C(s,) indicates whether a token should remain explorative and thus be subject to entropy regularization. We assume that for important tokens, C(s;) is uniformly bounded below by some constant y > 0.
(iii) Empirically, our training curves show that responses with Hresp > Whigh vanish rapidly, and such cases contribute negligibly to the average entropy. This supports the assumption Hresp < Whigh- Moreover, in the early stage of training, the highest entropy tokens (top 20%) contain lot of exploratory tokens, exhibiting large average entropy, motivating the assumption of positive lower bound Hresp 2 H
(iv) It is provable that
Cov (a log na) S H,
where H denotes the entropy. In practice this upper bound rarely tight, and we assume instead looser bound with a small constant É (0, 1).
(v) In our regime, the entropy is low enough that the token with the largest probability dominates (with probability 2 0.6). In this setting, the covariance large enough and proportional to the entropy H.
In practice the observed entropy lower bound is higher than the theoretical bound derived in Proposition 4, owing both the looseness of the Markov inequality used in the derivation and the fact that the tokens outside Hresp (bottom 80%) also get an entropy boost.
C ADDITIONAL RESULTS
C.I ADDITIONAL RESULTS ON CONTINUOUS CONTROL TASKS
In this subsection, provide additional experimental results continuous control tasks to further validate the effectiveness our proposed method, ERA, and to find more insights regarding entropy regularization in reinforcement learning.
30

---

Published as a conference paper at ICLR 2026
C.1.1
TRUNCATED GAUSSIAN IS MORE STABLE THAN TANH GAUSSIAN
humanoid-run
humanoid-run
humanoid-run
6.00
Action Noma
Europy
-1
0.00
300,0K steps
1004
302.6 stops
1.04
5010K
1X04
autrog target
steps
(b)
(c)
CrusetteNormoo(ooss
TonhNorma(Const)
TrunstedNormmllAuut))
TanhNormal(Auto)
Figure 10: Analysis Policy Distributions. Comparison of Truncated and Tanh Gaussian policies with varying 5 DMC tasks. Target entropy represents the desired average entropy per action dimension. (a) The Truncated Gaussian exhibits greater stability across four DMC tasks. (b) For the Tanh Gaussian with a learned 8, instability arises action norms approach the boundary, causing training to collapse.
(c) The Truncated Normal distribution's 's entropy remains stable and well- controlled in both modes, shown here for target entropy of -0.75.
We study the choice of policy distribution and the handling of its standard deviation, 6. We compare a Truncated Gaussian against -quu-squeshed Gaussian, each with a constant 8 (set to 0 in our experiments) and a learned 6, using SAC on four hardest tasks from the DMC Dog Humanoid suites(dog- dog-frot, humanoid-run, humanoid-wall) with 5 seeds and IM environmental steps. As shown in Figure 10, the Truncated Gaussian is significantly more stable. The Tanh Gaussian experiences catastrophic training failures when ó is learned. Our analysis reveals that with the Tanh Gaussian, the action norm often approaches the distribution's boundaries. This causes the learned to grow explosively, creating vicious cycle of instability as the policy attempts to output actions near the boundary while satisfying the entropy objective. This issue is absent the Truncated Gaussian, which yields stable 5 values. Given that the performance difference between learned and constant 5 is minimal under the Truncated Gaussian, we adopt the truncated gaussian distribution with constant 5 of 0 setting for its simplicity in main results.
C.1.2 BATCH-LEVEL ENTROPY REGULARIZATION V.S. STATE-LEVEL ENTROPY REGULARIZATION
08
IQM
0.4
ERA-Batch
ERA-Single
62
SAC
0.0
-1.0
-0.5
entropy target
Figure 11: Comparison between state-level and batch-level entropy regularization methods on DMC Dog & Humanoid suites. Both methods outperform the SAC baseline.
In addition to the state-level entropy regularization method presented the main paper, we also investigate batch-level entropy regularization method, which directly constrains the expected entropy of the action distribution over Pr. Specifically, we modify the activation form of ERA in Eq. 12 to the form in Eq. 48.
(H/ (Hi)
d' exp
max
log (log max
log V2Te
llog Fmax
log amin
(48)
Where の N over the sampled batch, and during evaluation, we can use running average of e over the training
Σi=1 a is the average of e over the batch. During training, we can calculate e
31

---

Published as a conference paper at ICLR 2026
process, which is similar the running statistics in BatchNorm (loffe & Szegedy, 2015). We conduct an ablation study to compare the performance state-level and batch-level entropy regularization methods on DMC Dog & Humanoid suites(dog- dog-trot, humanoid run, Homanoid-wall, As shown in Figure 11, both methods achieve similar performance, outperforming the SAC baseline. This indicates that in Decomotion-dominated control tasks, which require high exploration due to the need for randomness but do not demand high precision, the difference between state-level and batch-level entropy regularization is minimal.
C.1.3 SAC-ERA ON MUJOCO GYM ENVIRONMENTS
We also evaluate the performance of SAC-ERA on the classic Mujoco Gym environments, including HalfCheetah-v4, Hopper-v4, Walker2d-v4, Ant-v4, Humanoid-v4, Swimmer-v4, and compare it with the SAC baseline. Figure 12 shows the learning curves of SAC-ERA and SAC on these environments. Despite their massive performance gap on HumanoidBench, SAC-ERA demonstrates only slight advantages over SAC on Mujoco Gym environments. This may be due to the relatively low action space dimensionality
Mujoco environments, which reduces the impact of different constraint schemes. This finding suggests that modern algorithm design should shift focus from considering Mujoco to higher-dimensional action spaces, which can better evaluate algorithm performance in complex environments.
HalfCheetah-vd
Hopper-v4
Swimmer-v4
SAC-BRA
Return
Return
Return
6500
2000
100
0.00
S0O OK
1.0
500 OK
1.0
0
S0ILIK
1.0M
steps
steps
steps
Humanoid-v4
Ant-v4
Walker2d-v4
7000
6000
6000
Return
Return
Return
3000
2500
2000
0.00
0.00
500.0K
1.0M
500,0K
1.0M
500.0K
LOM
steps
stops
Figure 12: Learning curves of SAC-ERA and SAC on Mujoco Gym environments. SAC-ERA demonstrates very slight advantages over SAC.
C.1.4 APPLICABILITY OF LLM RL TECHNIQUES TO CONTINUOUS CONTROL
We investigated the applicability of recent techniques from Reinforcement Learning for Large Language Models (LLM RL), designed to prevent entropy collapse, to the domain of continuous control. Specifically, we trained PPO agent the HalfCheetah-v4 benchmark for 10 random seeds, incorporating two distinct methods: Selective High-Entre Entropy Training, which trains the agent only on certain proportion of high-entropy samples, and Clip- Cligerr which applies larger clip ratio for advantages greater than one. Recognizing the significant disparities between LIM RL and continuous control tasks, we evaluated a range of parameters for each technique to ensure that any ineffectiveness was not due to improper parameter selection.
The results, presented Figure 13, show that these techniques struggle to provide higher policy entropy compared to the standard PPO algorithm in the control task. Furthermore, they yield no significant or only marginal performance improvements, we suspect such minor gains may not even stem from better entropy regularization. Consequently, the performance of these methods is not comparable to our proposed approach, ERA. These findings lead to two main conclusions. First, they highlight the substantial differences between L.M RL and continuous control, demonstrating
32

---

Published as a conference paper at ICLR 2026
that techniques effective in one domain do not necessarily transfer to the other, even when using the same algorithmic framework. Second, they underscore the superior performance of our proposed ERA method.
HolfCheetah-v4
HaltCheetab-vd
HallCheetah--d
HalfCheetabb-v4
30%
Rolum
Entrupy
4000
Eattogg
-1000
-0000
2.00
15K steps
1.0
1.5M
35M
0.00
LSM
1/0M
1.M4 stops
3.M
steps
stcpa
Figure 13: Results of Selective High-Entropy Training and a Clip-Higher Strategy in Continuous Control. (a) Performance when training the agent exclusively on a top percentage of high-entropy
samples. (b) Performance of the clip-higher strategy with varying clipping ratios.
C.1.5 COMPARING ERA WITH OTHER MAXIMUM ENTROPY RL APPROACHES
A key baseline for our entropy regularization approach (ERA) is the projection- based method from Akrour et al. (2019), which we term Scale Std. This method scales the standard deviation of a Gaussian policy by a factor if its entropy falls below a threshold, conceptually similar rejection sampling. While this appears similar to our mapping concept, the mechanism is fundamentally different. Scale Std merely translates the policy's 's output log standard deviations by uniform factor. This does not incentivize the policy to learn an optimal allocation of entropy across different action dimensions, as the constraint borne uniformly.
This difference is evident in the gradient signal. For the Scale Std method, the mapped standard deviation o¹ is calculated as:
= exp (Hungel H(T))/d
(49)
where d is the action dimension, Hurret is the target entropy, and H(6ō) aे In (2Tr) + Σi= In
the current policy entropy. The action is then sampled as NNu,, diag(o")))
)). The resulting
gradient with respect to the policy's original log standard deviation In (which the network outputs) is:
da
e; exp (Hurget - H(()))/d 00 (o:-2)
(50)
I σ.
Uniform Scalar C
For comparison, the gradient for SAC without an entropy penalty is simply:
aa
c di
(51)
In d.
Thus, the Scale Std gradient is merely the standard SAC gradient scaled by a uniform constant C and offset by another uniform constant (C'- (2d)). This adjustment provides no differential signal to incentivize entropy allocation between dimensions. This post-processing of the policy output does not truly make the policy learn to allocate entropy among dimensions.
In contrast, while the gradient with respect to ERA's final log std is also & Gg this gradient is backpropagated through the ERA activation to the policy's original output (whiih is not actually standard deviation). This process multiplies the gradient by the derivative of the ERA function, which is dimension-specific due to the softmax mechanism. This provides the necessary differential signal, compelling the policy to learn an optimal entropy allocation, which is the fundamental reason for its success.
We validated this theoretical analysis by comparing SAC-ERA and SAC (Scale Std) the DMC dog-trot task. We used a target entropy of -A (where = 38 is the action dimension) and a compensated truncated distribution for both to ensure fair comparison. To visualize the learned exploration strategy, we generated density heatmaps of the policy's 's log standard deviations over training (10 seeds), shown in Figure 14. For Scale Std, we plot the pre-translation log stds (as
33

---

Published as a conference paper at ICLR 2026
uniform translation only alters the distribution's location, not its shape), and for ERA, plot the final, post-mapping log stds.
The results stark. SAC (Scale Std) exhibits highly uneven distribution: most dimensions collapse to the lower bound -8, while a few saturate at the upper bound (using the default range [-8, OD.. The mean log std around -7, indicating that most dimensions excessively. Conversely, SAC-ERA shows a clear diffusion from a uniform start, as the policy learns allocate entropy across dimensions in targeted manner. The final distribution is well-spread, clustered at the bounds, indicating all dimensions participate meaningfully in exploration.
exploration, while few to explore
This strategic difference directly impacts performance, as shown in Figure 15. We tested four complex tasks: DMC dog-trot, humanoid-walk, and HumanoidBench h1-walk and hl-run. SAC (Scale Std) shows mild improvement on dog-trot and is significantly worse than the baseline SAC on the other three, suggesting its naive exploration strategy hinders learning. In contrast, SAC-ERA significantly outperforms both SAC and SAC (Scale Std) in all environments, confirming that ERA effectively guides the policy to rationally allocate entropy across dimensions, failure point for Scale Std method.
dog-trot
Scale
ERA
-36
Suile Scl
Actor Log Sid
Actor Log Sid
Entropy
HGG
MISInG
3.5
388
500K
LM
steps
Training
Stops
Training Steps
Figure 14: Entropy curves (left), evolution of log standard deviation distributions for SAC (Scale Std) (middle) and SAC-ERA (right) on the dog-trot task. Scale Std leads to polarized, uneven distribution, while ERA learns balanced, diffusive allocation.
dog-trot
humanoid
bl-walk-vD
1000
1000
SActtaal:
R.tun
Retum
598
500
steps
Figure 15: Performance comparison of SAC-ERA against SAC (Scale Std) and baseline SAC on complex locomotion tasks (DMC drot humanoid-wale and HumanoidBench hl-walk, hl-run).
ERA
Scale Std
SAC
0.81
Normalized Score
0.8
0.6
0.40
0.42
0,0
ERA
Scale Std
SAC
Figure 16: Normalized scores across 4 tasks. SAC-ERA significantly outperforms both SAC and
SAC (Scale Std) all environments
34

---

Published as a conference paper at ICLR 2026
In addition to the projection- based method, several other approaches have been explored to imple- ment maximum entropy reinforcement learning, including recent diffusion- based and flow-based methods (Celik et al., 2025; Chao et al., 2024; Ma et al., 2025). However, these methods often require significantly more computational resources; for instance, the MEow algorithm (Chao et al., 2024) requires least 2.3 times the training time of SAC. We therefore focus comparison methods that also adopt Gaussian policies:
recent
EAPO (Choe & Kim, 2024): The core innovation of Entropy Advantage Policy Optimisation (EAPO) is decomposing the max-entropy objective into cumulative reward and trajectory entropy, then independently estimating advantage functions for each. It introduces a dedicated "entropy critic" to separately learn the value of future uncertainty, combining with the traditional value of future rewards.
MNSE (Zhong et al., 2024): The Maximum Next-State Entropy (MNSE) paper argues for the direct maximization of next-state entropy. positing that this more directly measures the diversity of states induced by the policy and can lead to more efficient exploration.
Since public code repositories were available, compare against the curves reported in the original papers. The experimental setups are as follows:
EAPO utilizes the PPO algorithm as its base and was trained for million timesteps (more than the million timesteps used in PPO-ERA).
MNSE is built upon the SAC algorithm and was trained for million timesteps (the same as SAC-ERA).
We compare PPO-ERA with EAPO, and SAC-ERA with MNSE on the Mujoco Gym benchmark. The results are presented in Figure 17 and Figure 18. shown, ERA demonstrates superior performance over EAPO when both are built on PPO, and it also outperforms MNSE when SAC is used as the base algorithm. Although Mujoco Gym is a relatively low-difficllyy benchmark, we are limited to as neither of the other papers presented results in more complex environments like DMC Suite or HumanoidBench. These findings suggest that ERA is a more effective implementation of maximum entropy reinforcement learning.
HalfCheetah-ve
Walker2d-v4
Ant-v4
5000
ERA
Return
Return
Return
4500
3000
3000
0
1.5M
3.0M
1.5M steps
3.0M
1.5M steps
3.0M
steps
Figure 17: Performance comparison of PPO-ERA against EAPO on MuloCo benchmark tasks.
HaltCTheechh.. wll
Hopper
Woller22
Humazoid
$300
2008
3000
SAC-ERA
anrpe
Figure 18: Performance comparison of SAC-ERA against MNSE on MuJoCo benchmark tasks.
Furthermore, both EAPO and MNSE require additional network architectures and computational resources. EAPO necessitates an extra entropy critic network, while MNSE requires an additional inverse dynamics model network. In contrast, ERA does not require any additional networks. leading to negligible increase in computational overhead. This makes ERA a more advantageous choice for practical applications.
35

---

Published as a conference paper at ICLR 2026
C.1.6 SENSITIVITY ANALYSIS ON THE INTERVAL
The hyperparameters max and min frequently employed in algorithms such as SAC. Standard settings for min typically include -20, -10, and -8, whereas max is commonly set to 0 or 2. We evaluated the performance of SAC-ERA on the dog-run and humanoid-walk environments using three distinct sets of these values, as illustrated in Fig. 19.
dog-rum
humanoid-walk
1000
1000
[-8.0]
[-20,0]
[-8.2]
Return
Return
500
500
500K
1M
500K
1M
steps
steps
Figure 19; Ablation study on the interval [i max for SAC-ERA in dog-run and humanoid- walk, with 5 seeds. We compare performance across three different interval settings derived from Fmin E { f-20, -8} and Fmax 23. The results show that the choice of these bounds has no significant impact on performance, highlighting the robustness of our method.
The experimental results demonstrate that all three settings exhibit nearly identical learning curves on the dog-run environment, On the bumanoid-walk environment, the performance differences are also not significant, although the [-8, 0J setting yields slightly better performance compared to the other two configurations. Overall, method exhibits strong robustness the choice of the interval. In practice, we recommend prioritizing the [-8, O) interval, which we use as the default in our experiments, and considering other settings only when further fine-tuning is required.
C.1.7 ON THE CHOICE OF THE ENTROPY TARGET
The selection of the entropy target is a key hyperparameter when employing ERA. As discussed in the main paper and prior appendices, the optimal value for SAC-ERA depends on the use of compensation factor ò. For the truncated normal policy, we recommend higher target (e.g., 0.25.4) if the compensation factor is set to zero. If the compensation factor is used, recommend target of AA, which aligns with the empirical values used in standard SAC implementations (c.g.. in stablebasclines and other prior work).
For PPO-ERA, we conducted an ablation study on the entropy target value in the HalfCheetah- and Ant-v4 MuJoCo environments. The results presented in Figure 20. Overall, these results indicate that PPO-ERA is not highly sensitive to the choice the entropy target in these environments. outperforms the PPO baseline by a significant margin across broad range of target values, with optimal performance observed around a target of -0.25A.
Our experiments also involved TD-MPC2. Due the extensive training time required for this algorithm, we only tested and reported the results for a target of -A. This value was selected based on the empirical standard commonly adopted SAC implementations.
C. COMPARISON WITH SMALL INITIAL TEMPERATURE SAC
Recent studies (Lee et al., 2025) based on SAC have adopted smaller initial temperature (e.g.. 0.006) to mitigate the impact of fluctuations in the entropy constraint term during training. We compared the performance of SAC initialized with small temperature (0.006) against the baseline SAC used this work (initialized at 1.0) on 4 tasks, including the DMC dog-run, humanoid and HumanoidBench h/-walk and h/-run The results are presented in Fig. 21.
36

---

Published as a conference paper at ICLR 2026
1QM
PPO(Origiza)
entropy target
Figure 20: Ablation study on the entropy target for PPO-ERA in HalfCheetah-v4 and Ant-v4 environments.
dog-Tun
bumanoid-walk
bl-walk-vv0
hl-rum-v0
EAC
Rethmn
Rctumm
Retum
530
508
0.30
033
500
000
0.00
Figure 21: Performance comparison between SAC with a small initial temperature (0.006), the baseline SAC (initial temperature 1.0), and SAC-ERA 4 tasks. SAC-ERA outperforms both baselines, demonstrating its superiority in complex control environments.
The results indicate that SAC with small initial temperature outperforms the baseline SAC (initial temperature 1.0) in two the four tested environments, while performing comparably or slightly worse in the other two. This suggests that using small initial temperature may mitigate the impact of entropy constraint fluctuations in certain scenarios, but it is not effective all environments, and its efficacy likely depends specific environmental characteristics. Fundamentally, this approach does not resolve the underlying issue: while a small initial temperature can partially mitigate the fluctuations caused by the entropy constraint, the continued presence of the entropy term the loss function may still hinder the optimization of cumulative returns, particularly when environmental rewards are sparse. Moreover, SAC-ERA significantly outperforms SAC with small initial tempera- ture across all four environments, further demonstrating the superiority of ERA in complex control environments.
Furthermore, many existing SAC implementations widely adopted by the community, such as stablebaselines3 (Raffin al., 2021) and jaxrl (Kostrikov, 2021), 1) still default to an initial temperature of 1.0. We argue that our use of this more common 1.0 initial temperature as a baseline is reasonable, given that the optimal initial temperature possibly requires citroommnntsppecffe tuning. contrast, employing ERA completely obviates this issue.
C.1.9 VALIDATION AGAINST STABLE- BASELINES3 (SB3) IMPLEMENTATIONS
To validate the reliability and generalizability of our experimental findings, we benchmarked the performance of our SAC and PPO baseline implementations against the standard implementations provided by the Baole-BBslline33 (SB3) library.
SAC Comparison. For the Soft Actor-Critie (SAC) agent, we precisely aligned the network architecture and hyperparameter configurations with the SB3 implementation. We then conducted comparative experiments on four tasks from the DeepMind Control (DMC) Suite (specifically, the Dog and Humanoid domains). The results indicate that the SB3 SAC implementation performs slightly better than our JAX RL-based implementation on the Dog tasks, but conversely, underperforms on the Humanoid tasks. Ultimately, the final normalized scores for our baseline and the SB3 baseline were nearly identical. However, the SB3 implementation demonstrated greater stability (i.e., lower variance across seeds). Overall, the SB3 baseline still exhibits suboptimal performance on these complex control tasks, showing a significant performance gap compared to our proposed SAC(ERA) agent.
37

---

Published as a conference paper at ICLR 2026
190
.008
deg-trod
180 brmmmedd--
Rctaun
049
08 0.0
10:.5
800
50.56
0.00
20.06
176
0.0
10.00
(a)
(b)
Figure 22: Performance comparison of Staale (SB3), JAX RL SAC(our baseline), and SAC(ERA) on DMC Dog and Humanoid tasks, averaged over seeds. (a) Normalized scores across four tasks. SB3 SAC demonstrates greater stability (lower variance) compared JAX RL SAC, although their average scores are comparable. (b) Learning curves for the four tasks. SB3 SAC excels on the Dog tasks, while JAX RL SAC performs better on the Humanoid tasks. SAC(ERA) consistently outperforms both baselines across all environments, while also exhibiting comparable or superior stability.
PPO Comparison. Similarly, for the Proximal Policy Optimization (PPO) agent, we utilized hyper- parameter settings identical to those in our primary experimental setup. The evaluation reveals that the SB3 PPO implementation achieved slightly inferior results compared to our PPO implementation on both the HalfCheetah and Ant environments. Consistent with the SAC results, the SB3 PPO agent again exhibited superior stability.
HalfChectah-v4
Ant-v4
9000
1.
PPO
0,42
FPO/SB3) PPO(ERA]
Retur
Return
Normalized
4500
3000
04-
0.32
0.33
02
0.00
0.00
3.0M
PPO
steps
(a)
(b)
Figure 23: Performance comparison of StableBaselines3 (SB3), PPO(our baseline), and PPO(ERA) on HalfCheetah and Ant, averaged over 5 seeds. (a) Normalized scores across the two tasks. SB3 PPO shows enhanced stability (lower variance) relative our PPO baseline, though their average scores are similar. (b) Learning curves for the two tasks. Our PPO baseline and SB3 PPO achieve similar performance both environments. PPO(ERA) consistently surpasses both baselines across all tested environments, while also demonstrating comparable or superior stability.
Conclusion. In summary, the baselines used in our study and their SB3 counterparts demonstrate highly comparable performance. This suggests that substituting baselines with the SB3 im- plementations would not substantively alter the main conclusions of this work. While the SB3 baselines exhibited greater stability, this difference is significant enough affect our conclusions, which are based on aggregates over at least 5 random seeds. Furthermore, it is noteworthy that our ERA-enhanced agent significantly outperforms the SB3 baselines across all tested environments, while also demonstrating comparable or superior stability. This underscores the effectiveness of the ERA method robustly boosting both agent performance and stability.
C.1.10 COMPARISON ENTROPY DYNAMICS WITH SAC VARIANTS
We conducted a comparative analysis of three methods on the dog-trot task: standard SAC (using the Tanh-Gaussian policy with std range of [-10, 2)). SAC with truncated normal distribution (SAC-TN), and SAC-ERA (using a truncated normal distribution with an auto-tuning compensation term). For all methods, the target entropy was set AA. Following the same visualization protocol used our Scale Std analysis, we plotted both the entropy curves and the log std density heatmaps for all three approaches, shown in Figure 24
38

---

Published as a conference paper at ICLR 2026
The results indicate that, given the same entropy target, SAC-ERA maintains the most stable entropy curve. SAC-TN exhibits slightly smaller oscillations than the standard SAC. The log std density heatmaps reveal further distinctions. Both standard SAC and SAC-TN undergo rapid, abrupt shift in the log std distribution during the early stages of training; this corresponds to the dynamic adjustment of the entropy temperature parameter it converges to the target. Concurrently, their log std distributions diffuse both faster and more broadly compared to SAC-ERA. In contrast, the log std distribution for SAC-ERA is markedly more stable, exhibiting a gradual and controlled diffusion process over time. This highlights significant divergence in training dynamics, distinguishing SAC's extrinsic adjustment via an entropy term from ERA's intrinsic regulation via its activation function.
In terms of final performance. SAC-ERA also outperforms both SAC and SAC-TN. The performance of SAC-TN approximately on par with the standard SAC. This finding suggests that merely replacing the Yanh-Gaussian policy with truncated normal distribution does not, by itself, yield significant performance gains. Instead, the critical factor appears to the ERA entropy constraint mechanism, which provides more stable entropy regulation process and, consequently, more stable training dynamics.
dog-trot
10
1.0
8.5
Sy
Amoott Acaen Lagg Std b
5X0
1NC
steps
Tranting Stegs
Toitiing Segs
Toining Stape
Figure 24: Comparison of entropy curves (left) and log standard deviation heatmaps (middle, right) for standard SAC, SAC-TN, and SAC-ERA on the dog-trol task.
dog-trot
1000
SAC
SAC(TN)
ERA
Return
500
500K
1M
steps
Figure 25: Performance comparison of SAC, SAC-TN, and SAC-ERA on the dog-trot task.
C.1.11 A DEMONSTRATIVE EXPERIMENT ON GRADIENT CONFLICTS IN SAC
We do simple experiment to demonstrate the gradient conflict between reward maximization and entropy maximization in SAC on DMC humanoid-run task. We compute the gradients of the reward objective and the entropy objective on distribution parameters μ, and the final action a. We then compute the cosine similarity between the two gradients to measure their alignment. A cosine similarity greater than 0 indicates that the two gradients are aligned(í.e. their angle is less than 90 degrees), while cosine similarity less than O indicates that the two gradients are conflicting(i.e. their angle is greater than 90 degrees). We plot the cosine similarity over training steps in 26. Our results show that for all three parameters, among all 5 seeds tested, the cosine similarity is all negative for the majority of training time, indicating that the reward and entropy objectives, for the most part, has conflicting gradients. This supports claim that in SAC, the reward maximization and entropy maximization objectives are often at odds, leading to inefficient policy optimization paths. In contrast, with ERA, the mean and standard deviation only receive gradients from the reward objective, while
39

---

Published as a conference paper at ICLR 2026
the entropy constraint is handled internally by the policy itself, allowing for more direct and efficient
optimization towards the reward goal.
Grad Cos (mean)
Grad Cos (log std)
Grad Cos (action)
Grad Cos (mean)
0.0
Grad Cos (log std)
0.0
Grad Cos (action)
0.0
seed
0.1
-0.2-
seed
-0.2
sead 3
-02
seed 4
0.5
1.0
0.5
1.0
0.5
1.0
Training Steps
1
Training Steps
Training Steps
Figure 26: Cosine similarity between reward and entropy gradients on a, over training steps in SAC on DMC humanoid-rut run task. Negative values indicate conflicting gradients.
C.1.12 TIME COST OF ERA IN CONTINUOUS CONTROL
A potential concern might be the additional time overhead introduced by using ERA. To evaluate this, we recorded the training times of FastTD3 and FastSAC-ERA on HumanoidBench, as shown in Figure 27. It can be observed that using ERA does introduce some time overhead due to the more complex activation function applied to the output. However, this overhead accounts for only about 6% of the total training time average. Considering the improved exploration performance and higher sample efficiency brought by ERA, we believe this a worthwhile trade-off.
The scenario for comparing training speed against FastTD3 is particularly stringent. This is because FastSAC-ERA must additionally output per-dimension policy standard deviations, which introduces computational overhead not present in FastTD3. To quantify the specific overhead of method, measured the training time of baseline SAC versus SAC-ERA in the dog-trot environment. When trained on a single A10 GPU, the additional time cost of SAC-ERA, averaged over five seeds, was approximately 3%.
Algorithm Time Comparison
500
Time (minutes)
400
300
200
100
FastTD3 FastSAC-ERA
Figure 27: Time comparison on hlhand-hurdle-rd, We compare the training time of FastTD3 and FastSAC-ERA on HumanoidBench. The results show that using ERAintroduces a modest time overhead, averaging around 6% of the total training time, which is reasonable trade-off for the improved exploration performance and sample efficiency provides.
C.2 ADDITIONAL RESULTS ON IMAGE CLASSIFICATION
C.2.1 COMPARING ERA WITH COMMON REGULARIZATION TECHNIQUES
plethora of regularization methods have been proposed and utilized in the field of image clas- sification. To further investigate the comparative effectiveness of ERA against commonly used regularization methods like dropout and label smoothing in the vision domain, we conducted series of straightforward comparative experiments on the CIFAR- dataset. In our main experiment, we adopted the default settings from the timm library, which include label smoothing factor of 0. and no dropout. For the sake of comparison, and 0.3, and the dropout rate to 0.1, 0.2, and 0.3. The results were then compared against the baseline algorithm from our main experiment and ERA.
respectively adjusted the label smoothing factor to 0.2
40

---

Published as a conference paper at ICLR 2026
The experimental results are presented in Figure 28. The findings indicate that increasing the intensity of label smoothing adversely affects model performance, while the improvement from employing dropout is marginal (the top-1 accuracy may decrease, whereas the top-5 accuracy shows an improvement). In contrast, ERA effectively and consistently enhances model performance, with margin of improvement significantly superior that of both dropout and label smoothing. This outcome further validates the advantage of ERA over conventional regularization methods. While constraining the model's entropy, ERA permits the model to freely allocate uncertainty among dimensions, thereby better adapting to the intrinsic structure of the data. This enables ERA to more effectively boost the model's generalization capability.
Top-I Accuracy
Top-5 Accumcy
Smootting
99.9
Dmpport
No Aug Out
Accuncy (%)
99.8
22.75
$3.50
99.7
83
93.00
99.0
0.16
0.13
3.30
0.25
0.30
0.10
0.15
0.20
025
0330
Regularization Strength
Regulatization Steenghh
Figure 28: Comparison of different regularization methods on the CIFAR-10 dataset. The left subplot shows the T accuracy, and the right subplot shows the Top-5 accuracy. Our method, ERA, is compared against varying intensities of Label Smoothing and Dropout.
Furthermore, we extended our comparison to two other entropy constraint methods, which are common in reinforcement learning but rare in image classification. The first is the Entropy Term, which adds an entropy penalty directly to the loss. The second is the projection-based method from Akrour et al. (2019), which term Linear Interpolation. Similar to its continuous space counterpart, this method acts when the policy's 's output entropy falls below a target: it increases the entropy by interpolating the distribution with uniform distribution.
Analogous to the continuous case, we provide a theoretical analysis of the gradient back-prop pock-propagation mechanism under the cross-entropy loss elucidate the fundamental difference between ERA and projection- baseed based methods. Consider the policy output distribution T (a]s)) with corresponding logits ī. For target class k, the cross-entropy loss is L = - log π (ax.ls).
In the Linear Interpolation method, the adjusted probability a mixture of the original policy Torig and uniform distribution, governed by the entropy constraint:
log N- -Ho
Ho H(Torii)
π (ap = log N H ((Trig))
Torig Torig (axls)
(52)
log N H (Torig)
N
Applying the chain rule, we derive the gradient of the loss with respect to the original probability Tarig (0x|s). Note that the mixing coefficient À depends on the global entropy H (Targg), which in turn depends on worig:
0C OrT(ax (8)
(53)
T(ak 1s)
OT (ak) (8)
ax rron (arls)
0(1 2)
A + Torig (arls)
(54)
@raiig (ax
N rroni (ax
Substituting the partial derivatives of the entropy term the complex sensitivity term:
(H(( Drane (px
log Torig (ax(s) - 1, we obtain
Om (n(ag))) X + (Torig (Trii(aals) 1
log N Ho
(log N H((Torig)) ))z (log Tunig (akls) + (55)
T (ax
41

---

Published as a conference paper at ICLR 2026
Critically, although mathematically involved, this sensitivity term depends principally on the target class and the global entropy state. When propagating to the logits 4, the total gradient becomes:
ac
aC
(r(())))
Omong (axls)
04,
=
@T(akls) Toorig (ax
04
(56)
((T,k))
Here, @Hag(as]=) is the standard softmax gradient. The term ( k)) acts effectively a scalar 8t;
coefficient common to the gradient flow. This indicates that the projection method primarily acts as uniform gradient scaler: creates gradient signal that pushes the distribution towards uniformity globally, without providing dimension-specifer guidance beyond what the original softmax offers. This limitation stems from its nature as a post-proessiing step.
In stark contrast, ERA modifies the logits directly within the architecture before the softmax. The gradient flow for ERA is defined as:
Oc
Oc Ôt (T(aals)
d Oa(as 8)
0y
(57)
al,
o 04
where 11 1 (g(44)) approximating g(4)) as shifted softmax g(4) a el can be expressed as:
"E,es 5
+ b, the gradient
Oc x (dix - T(alss)) a. 0l,
δ 04 ES
ele
oi Og(
(58)
ets
ar The crucial differentiator is the term ōgih) Since this derivative depends on the value of gl,), which varies across dimensions according to their individual contribution to the entropy, it acts as dimension-speccfic scaling factor. Unlike the post- post-processing projection which applies uniform scalar V to all gradients, ERA generates structured gradient field that adapts individually to each logit 4,, enabling the model learn an optimal entropy allocation strategy.
We tested both methods on CIFAR-10 using the same experimental setup as ERA (without data augmentation, as in our ablation studies). We tested the Entropy Term with coefficients of le-4, Ie-3, and le-2. For Linear Interpolation, the target entropy was set to 0.6, identical to that used in ERA.
The results are shown Figure 29, Both of these entropy constraint methods underperform ERA in both t and t accuracy. This suggests that the utility of these RL-centric entropy methods may be limited in image classification, which could explain their infrequent use in the CV domain.
Top-1 Accuracy
Top-5 Accumey
94.25
Ermmayy Jerm
99.9
Linese Iniopplation Original
99.8
93.75
Assumicy
83.50
99.7
93.25
Entropy Weight
Regulatization Strength
Figure 29: Comparison of different regularization methods on the CIFAR-10 dataset. The left subplot shows the Top-1 accuracy, and the right subplot shows the Top-5 accuracy. Our method, ERA, compared against varying entropy term weights and the Linear Inerpolation method.
Furthermore, we evaluated the efficacy of the SAC-style automatic temperature adjustment mechanism on the CIFAR-I0 dataset. It's worth noting that while ERA and Linear Interpolation regulate the lower bound of entropy in image classification, SAC-style automatic temperature adjustment regulates the expectation of entropy. We have to choose higher target entropy to keep the entropy level aligned with ERA. We experimented with three target entropy values: 1.2, 1.25, and 1.5. These values were selected based on prior experimental findings, where the final training loss typically converged around 1.21 and increased to approximately 1.23 with the addition of ERA.
42

---

Published as a conference paper at ICLR 2026
Consequently, target entropy of 1.2, the entropy constraint term remains largely inactive; at 1.25, the target entropy aligns with the ERA baseline; and at 1.5, the target imposes a higher entropy requirement. We initialized the temperature coefficient at 10 with a learning rate of 10 3. The results, depicted in Fig. 31, reveal a distinct trade-off between Top-1 and Top-5 accuracy when using the Sty-style adjustment. Specifically, while Top-5 accuracy exhibits slight improvement as the target entropy increases, Top-1 accuracy declines. Ultimately, this approach fails to achieve the performance levels attained by ERA.
Entropy On CIFAR-10
Entropy On CIFAR-10
Linea=0.6
1.3
SAC-12
ERA-006
SAC-1.25
SAC-1.5
Entropy
Entropy
ERA-06
1.25
1.25
1.2
1.2
100
200
Epochs
Epochs
(a)
(b)
Figure 30: Entropy curves for CIFAR-10 classification. (a) Entropy curves for ERA and Linear Interpolation with entropy target both set to 0.6. They both regulate the entropy on all samples to be above the target. So entropy curves are quite similar. While in practice they demonstrate different top-1 and top-5 accuracies. (b) Entropy curves for SAC style entropy adjustment and ERA on CIFAR-10 10 classification. We test three different entropy targets (1.2, 1.25, 1.5) for SAC style adjustment. It's worth noting that SAC style adjustment only regulates the expected entropy to be close to the target, so we must raise the target to achieve similar entropy levels as ERA-0.6. Even when SAC style adjustment achieves similar entropy levels (target=1.25) 1,2)), ERA still outperforms it in terms of top-1 I and top-5 accuracies.
In the context of image classification tasks, we further observed that the SAC-style constraint mechanism is largely ineffective. This is primarily because the number of gradient steps is significantly fewer than in control tasks. Moreover, the initial entropy is substantially higher than the loss, causing the entropy term (temperature coefficient) to decrease initially; it only begins to increase gradually once the entropy approaches the threshold. Consequently, it is difficult to effectively satisfy the target entropy constraint within the limited training duration, resulting in final performance that remains close to the baseline.
Top-1 I Accuncy
Accumey
Estroy Tom Critinal ERA
99.9
94.00
Accuraccy (50
-3.75
91.0
99.7
93
99.6
1.2
1.25 Entropy Torget
1.5
12
1.35 Entropy Target
1.5
Figure 31: Comparison of SAC-style automatic temperature adjustment on the CIFAR-10 dataset. The left subplot shows the Top-1 accuracy, and the right subplot shows the Top-5 accuracy. Our method, ERA, is compared against varying target entropy values using SAC-style temperature adjustment. Here "entropy target" refers to targets of the SAC-style method. fixed entropy target of 0.6 is used for ERA in this experiment.
C.2.2 TIME COST OF ERA IN IMAGE CLASSIFICATION
We compared the training time of the 50 model the CIFAR-10 dataset, with and without using ERA, under the data augmentation supported by the timm library. Consistent with our main results, the experiments were conducted on three machines, each equipped with four NVIDIA A40
43

---

Published as a conference paper at ICLR 2026
GPUs, and report the average training time. The results are presented in Figure 32. As shown in the figure, since the data is already well-parallelized, there is almost no difference training time between the algorithm using ERA and the original version.
Algorithm Time Comparison
Time (hours)
3
2
ResNet
ResNet-ERA
Figure 32: Time comparison C-FAR-10. We compare the training time of ResNet and ResNet-
ERAon CIFAR-10. The results show that using ERA introduces almost no time overhead.
C.3 ADDITIONAL RESULTS ON LLMS
C.3.1 ABLATION STUDY ON ADVANTAGE SCALING
In this section, we explore the use of advantage scaling:
1 Al
Hresp Wlow, At >
A
(whow Hresp I Whigh: A, < 0) or A, >00,
kAg Hresp > Whigh: A > 0.
For ERA with advantage scaling, train it for 1400 steps, with hyperparameter whow 0.45, Whigh 3.0, l: = 2; and for ERA without advantage scaling, we train it in two stages for 1100 steps in total, where the first stage lasts 600 steps with 4how 0.45, Whigh 3.0, 2, and the last 500 steps trained with Wlow 0.2, Whigh +00, 2.
As shown in Table 7, both variants-trainiee with or without advantage improvements over the GRPO baseline. Although adding advantage scaling results in higher score, the advantage estimates in GRPO are already noisy, so we expect both options to work similarly well and the performance gap remain relatively small.
substantial
Table 7: Ablation study on advantage scaling based on Qwen2.5-Math-77 For AIME and AMC, the results are avg. 16.
Model
AIME24
AIME25
AMC1
MATH5500
Minerva
Olympiad
Avg.
Models
Qwen2.5- 5-Math Yang et al. (2024a)
8.6
52.2
12.1
17.2
24.5
Owen225-Mattllaann Matti-llstt Yang (2024a)
13.3
10.0
57.1
81.0
32.7
38.8
38.8
(Shao 2024)
45.7
ERA (wlo advantage scaling)
37.5
16.9
72.8
84.6
42,6
46.5
50.2
ERA (w/ advantage scaling)
36.0
21.0
76.6
85.4
40.1
46.8
51.0
C.3.2 DETAILED ENTROPY ANALYSIS
We present the complete entropy curve of our two-stage training without advantage scaling in Figure 33, After decreasing Wlow, the entropy rapidly drops and stabilizes the second-level entropy lower bound. This confirms that our ERA method successfully enforces non-trivial entropy floor for the model.
We further analyze the entropy distribution across tokens by plotting the average entropy of the top 20% tokens (H and the bottom 80% tokens Figure 34. This experiment is carried out with Wlow 0.45, Whigh 3.0, & 2 without topk. Following Wang et al. (2025), observe
44

---

Published as a conference paper at ICLR 2026
Combined Entropy Plot
Firet Stage (dlepe 9-500)
Sccond Stngo 6000-100)
0.25
0.15
200
400
800
800
1000
Step
Figure 33: Entropy curve during two-stage training. After decreasing Wlww, entropy rapidly drops and stabilizes at the second-level entropy lower bound, showing that ERA enforces non-trivial entropy floor.
that the bottom 80% tokens exhibit nearly zero entropy, consistent with our theoretical prediction. Additionally, we plot the proportion of responses with Hresp < Whw. Hrep > Whigh in Figure 34. The fraction of responses with yesp > Whigh quickly drops zero, while the fraction with Heap < whow remains stable at the interval [0, 0.06]. This demonstrates that whenever overly low-entropy responses appear, ERA adaptively raises their entropy to a moderate level.
Entropy Means (First 500 Steps)
Response Entropy Fractions (Finst 500 Steps)
tup 20% tokens
low Hesup fruction
High Heary fracion
1.4
Response Entropy Fraction
0.04
Eatropy Mean
0.03
0.8
0.02
0.01
0.2
0.
0.00
100
250
100
400
500
300
400
500
Step
Step
Figure 34: Detailed entropy analysis. Left: average entropy of the top 20% tokens (Hres)) and the bottom 80% tokens. Right: proportion of responses (running average with window size 20) with Heepp < Wiow or Hresp > Whigh demonstrating ERA's ability to prevent both entropy collapse and overly high entropy.
C.3.3 ABLATION STUDY ON ENTROPY BOUND
Since the purpose of Wjow to set a lower bound on entropy, we explore the role of Whigh in the ERA. As can be in Figure 35, without the constraint of Whigh, the model's entropy explodes in very short time. This indicates that adding an upper bound constraint during training is essential for controlling the entropy of the training process.
45

---

Published as a conference paper at ICLR 2026
1.0
GRRHHERA
CRPO-ERA w/o 6hgs
0.8
Emitroy
0.6
0.4
0.2
100
200
300
400
500
Step
Figure 35: Comparison of ERA with and without Whigh- The entropy of ERA without whigh tends to explode within a very short number of steps, leading to the collapse of model training.
C.3.4 ABLATION STUDY ON THE PROPORTION OF HIGH-ENTROPY TOKENS
In this section, we explore the use of different proportions of tokens to calculate Heesp for rollout samples. We select the top 10% of tokens with the highest entropy from each rollout to represent the entropy resp of that sample. For other parameters such as Whow and Whigh. we kept them unchanged from the original settings.
As shown in Table 8, modifying the calculation of Hresp still achieves significant improvements compared GRPO. However, the improvement smaller compared to ERA. This is because the Heep calculated from the top 10% tokens is naturally higher than that from the top 20%. As result, fewer samples meet the condition Heesp < Whow compared to the version using 20%. Therefore, the constraining power of entropy is limited, and the results lie between ordinary GRPO and ERA.
Table Ablation study on the proportion high- high-entropy tokens based on Qwen2.5-Math- For AIME and AMC, the results are avg. @ 16.
Model
AIME24
AIME25
AMC
MATH500
Minerva
Olymplad
Avg.
Rase Madels
Qwen2. 5-Mthh Yang al. (2024a)
8.6
6.3
52.2
50.8
12.1
17.2
24.5
Owen2.5-Malb --Matb-Instruct Yang (2024a)
GRPO (Shao et al., 2024)
34.4
12.3
69.5
80.6
36.8
40.6
45.7
ERA 10% takens
36.6
15.8
71.8
82.4
38.9
43.1
48.1
ERA
37.5
16.9
72.8
84.6
46.5
50.2
C.3.5 TIME COST OF ERA IN LLM
ERA is applied when computing the log_probs of tokens in the responses. To evaluate its efficiency. we compare the value of timing- s/old_log_prrdd prob at the first step in verl's implementation. The experiments were conducted on 32 NVIDIA H20 GPUs, consistent with our main results. The outcomes are shown in Figure 36. As illustrated, since the sampled response is identical the first step, ERA introduces only about 5.6% overhead in time cost. When considering entire training step, the overhead ERA is even smaller, since its implementation does not affect other components of training (e.g., generation, model update, advantage calculation).
C.4 TRAINING CURVES OF CONTINUOUS CONTROL TASKS
D THE USE OF LARGE LANGUAGE MODELS IN THIS PAPER
In the preparation of this paper, we utilized LLMs as a general-purpost purpose writing assistance tool. Specifically, LLMs were employed for proofreading and polishing the language of certain sections to improve clarity and readability. The final title this paper was also partially inspired by suggestions from an L.M.
46

---

Published as a conference paper at ICLR 2026
A Algorithm Time Comparison
12
Time (seconds)
8
4
2
GRPO
ERA
Figure 36: Comparison of computation time between GRPO and ERA, measured by timing_s Baainnss/olll at the first step. ERA introduces only about 5.6% overhead.
I 11--stand-v0
h1-run-vO
hJ b1-staie-v0
hi-burdle-vo
Rethan
Retum
200
900
300
0.50
0.8
steps
h1 -valk-vO
1000
3902
Sotum
0.30
1.45
3.0M
0.08
1.94
3.45
strps
stops
Figure 37: Training curves of OBAC and OBAC-ERA on HumanoidBench environments.
However, we clarify that the core contributions of this work were conceived and developed entirely by the human authors. The design of the methodology. the execution of experiments, and the interpretation of the results did not involve the use of LLMs. All content, including text, figures, and tables, was carefully reviewed, edited, and verified by the authors to ensure scientific accuracy and integrity.
Finally, we would like to express our gratitude for the occasional sparks of inspiration and the assistance in debugging code provided by LLM friends. Their contribution, while not qualifying for co-authorship. was nonetheless appreciated.
47

---

Published as a conference paper at ICLR 2026
bumuneid-rum
humaneid-walk
hurmmmiid-stamd
deg-rum
1000
1938
1900
1000
Racan
Rocun
Retum
530
902
538.6K
1.0M
593
1.5M
500.0K stops
1.M
5000K stops
LOM
stops
stops
dog-wwalk
dog-lrot
R.thu
Ritum
530
598
0.00
0.08
Figure 38: Training curves of TD-MPC2 and TD-MPC2-ERS on DMC environments.
HalfChoetab-e4
Hopper-e4
Walker2d-v4
Ant-v4
1000
4008
8000
5000
Sterum
Retum
2300
2000
1.3M
1.304 stops
steps
steps
steps
Figure 39: Training curves of PPO and PPO-ERA on Mujoco Gym environments.
hi hand-silide--
b1land-run-vl
hlland-walk-st
bi blhund-pole-vo
Tuctio
Sclun
-300
53.8k
1028K
-300
S00K
1008K
-200
S00K slaps
-300
380K stops
HOOK
steps
steps
h1/hand-stair-vo
hand-push-vv
-haand-hlllnce simple-vei
hthand-hurdle.vd
300
Rocann
300
-303
stops
Figure 40: Training curves of FastTD3 and FastSAC-ERA on HlumanoidBench environments.
48

---

Published as a conference paper at ICLR 2026
1-standdvvO
M1-stair-vD
--hurdle-vdi
SACIRRA
M
Retura
Rtoura
Dotrrn
300
202
500
200
stops
11 -walk--v
11-crawl-vV
dog-cun
dog-walk
1800
1932
1000
84tmm
Retum
Retur
0.58
1.554
3.094
0.2
1.5M
3.54
000
506.2
1:M
610
300.00
1.M
stepp
steps
steps
sleps
dog-trot
humanoid-rum
humanoid wwalk
humamoid-stand
1800
2008
1900
1000
Rehmm
Retum
400
902
009
SK1.KK stpps
SKVEK steps
1.0M
EK stops
s3pps
Figure 41: Training curves of SAC and SAC-ERA on HumanoidBench and DMC environments.
49


Reviews:
[{"review_id": "7vDYgJ3Ka9", "reviewer": "Reviewer_m4ks", "summary": "The paper introduces Entropy Regularizing Activation (ERA), a framework that enforces entropy constraints via custom output activations rather than through explicit loss terms. This allows entropy control without disturbing the main optimization objective. The authors demonstrate ERA in three settings, continuous control, image classification, and reinforcement learning for large language models, showing consistent performance improvements and theoretical guarantees for minimum entropy. The method is lightweight, general, and easy to integrate.", "questions": "Some suggestions following from Weakness\n\n**1. Clarify the naming and framing**: Consider revising the term “entropy as an activation” to something like “entropy-constrained policy via activation” to better capture the actual mechanism. The current phrasing may be conceptually misleading.\n\n**2. Acknowledge limitations of fixed entropy:** In the limitations section, explicitly discuss when ERA might underperform compared to standard SAC or entropy-maximized RL, especially in cases where adaptive, state-dependent entropy is important.\n\n**3. Evaluate under stronger SAC setups:** Re-run or compare against more standardized, high-performing SAC configurations (e.g., small initial entropy coefficients or temperature scheduling) to demonstrate that ERA’s improvements persist under robust, modern baselines.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1. Simple yet general idea:** Treating entropy regularization as an architectural operation rather than a loss modification is elegant and broadly applicable. It bridges reinforcement learning, LLM training, and supervised classification under one principle.\n\n**2. Strong empirical validation:** The experiments are extensive and diverse, showing consistent improvements with minimal tuning or overhead (<7%). This cross-domain effectiveness strengthens the claim of universality.", "weaknesses": "**1. Naming / framing:** Calling it “entropy as an activation” feels misleading; the object under control is the policy entropy. “Entropy-constrained policy\" or \"Entropy-constrained policy via output activation” would better reflect what’s actually enforced (and avoid implying entropy is a property of the activation itself).\n\n**2.Control-side doubts:** SAC’s state-conditional entropy is a feature, not a bug: high entropy in uncritical states, low in critical ones. A fixed or globally enforced entropy can bias learning toward suboptimal policies and blunt per-state adaptivity (e.g., see recent discussions on fixed-entropy pitfalls) [1]. This deserves explicit treatment in limitations and experiments probing failure modes when the target is mis-set.\n\n**3. Baseline realism in SAC:** Recent work/practice often uses very small or carefully scheduled entropy coefficients in SAC to avoid instability. If ERA is compared to SAC with suboptimal temperature tuning, improvements may be overstated [2],[3]. The paper should verify that gains hold under strong, modern SAC setups (small-α inits, robust target-entropy tuning/schedules).\n\n[1] When Maximum Entropy Misleads Policy Optimization, ICML'25.\n\n[2] https://araffin.github.io/post/sac-massive-sim/\n\n[3] Hyperspherical Normalization for Scalable Deep Reinforcement Learning, ICML'25.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cJTQUk3wow", "reviewer": "Reviewer_x7W9", "summary": "The authors propose in this paper a set of parameter transformations that map input distribution parameters (e.g. logits of a discrete distributions or mean/logstd of a Gaussian distribution) to parameters that are above an entropy threshold. The main motivation of the paper is that entropy is a key factor in RL as it helps with exploration, yet optimizing an entropy regularized objective can be challenging as it requires to carefully choose the entropy weight. Instead of using soft entropy regularization, the authors propose to introduce entropy regularization directly in parameter space, effectively restricting the search space to policies having an entropy above a given threshold. \n\nIn details, the authors propose three parameter transformations for i) discrete distributions ii) squashed (Tanh) Gaussian distributions and iii) truncated Gaussian distributions. The authors integrated the latter two transformations for continuous control RL tasks with bounded actions showing improved performance over baselines such as SAC or TD-MPC2, and the first transformation for the fine-tuning of LLMs on math problem datasets or for training a ResNet-50 on image classification tasks; in all cases showing improvements over baselines.", "questions": "- What is the difference between the proposed projection for discrete distributions and the one proposed by [2]? How do you expect conceptual differences to manifest in practice?\n- What is your evidence that SAC is unstable when tuning its entropy temperature dynamically and why do you expect the online tuning of $\\delta$ to be more stable?\n- What are the benefits of the parameter transformations compared to enforcing an entropy constraint by tuning the temperature online (as in SAC) for image classification or LLM fine-tuning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper extends prior work on differentiable parameter projections to new distributions and has an extensive empirical evaluation spanning an impressively large number of applications and baseline algorithms, with promising empirical results.", "weaknesses": "- The paper does not discuss prior work even though there is existing literature extremely related to the submission. Namely, the paper [1] (*F. Otto et al.; Differentiable Trust Region Layers for Deep Reinforcement Learning; ICLR 2021*) proposes a set of parameter projections that enforce among other things KL and Wasserstein based trust regions. The paper [2] (*R. Akrour et al.; Projections for Approximate Policy Iteration Algorithms; ICML 2019*) proposes parameter transformations for KL and entropy constraints much like the current paper. [2] proposes a projection for Gaussian distributions and the current submission uses the same projection as a basis to extend the projection to Squashed and Truncated Gaussians. The contribution compared to [2] is thus clear (extending prior parameter transformations to new distributions) but I believe [1,2] should still be discussed in the related work. However, [2] also proposed a projection to enforce an entropy constraint for discrete distributions much like this submission, although the mechanisms seem at a glance a bit different: this paper scales the input logits while [2] mixes the input distribution with a uniform distribution. Since these two transformations serve exactly the same purpose (map an input discrete distribution to a distribution satisfying an entropy constraint), it would be good to at least discuss the conceptual differences between the two transformations, but ideally some additional empirical comparisons to gain insights on their practical differences would also be appreciated.   \n\n- The paper contains a few overclaims, i) first regarding SAC in l96 authors state that the variant of SAC that uses an entropy constraint with a dynamically adjusted temperature can lead to instability. I would appreciate some evidence supporting this claim. To the best of my knowledge this second version of SAC with a hard entropy constraint is the default SAC implementation in many deep RL code bases such as *stablebaselines*, and I have not heard or witnessed such instabilities before. ii) In a few places (e.g. l68 or l111) the authors state that their parameter transformations have provable guarantees, but from what I can see for squashed and truncated Guassians, the authors still need to adjust a parameter online, much like SAC as discussed in l1049, and the satisfaction of the entropy constraint is not guaranteed in for these distributions. \n\n-  For a paper whose main contribution is to provide a better control of the entropy, I was surprised to not see comparisons between the entropy control resulting from the author's method and from the dynamic tuning performed by SAC. Especially in light of the previous two (over)claims, it would be interesting to see how both algorithm manage to comply with their entropy constraints.  \n\n- The proof in appendix B.1 feel incomplete to me. The authors state that the $\\delta$ is a positive constant and although it seems trivially true for $\\delta_{\\text{tanh}}$, I would appreciate a proof for $\\delta_{\\text{TN}}$ \n\n- Wording regarding SAC-ERA's pseudocode is a bit confusing, authors talk of a soft Q-function (l933), but it seems to me that the algorithm is not in the entropy regularized RL paradigm and an unregularized Q-function is learned, making the implementation closer to TD3 than SAC.\n\n\n\nOverall, while I'm currently leaning towards a reject, I am willing to increase my score if the authors can better contextualize their work and provide some empirical insights regarding the entropy control their method proposed compared to existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DVg9Yg0ZGC", "reviewer": "Reviewer_8uvP", "summary": "This paper presents a way to control of policy entropies by introducing ERA, a dedicated activation function for the final layer to clip entropies within bounds. The proposed framework can be applied to generic policy gradient settings and is experimented with continuous control, discrete classification and LLM-RL training although each setting requires dedicated instantiation to integrate the idea into its training. In experimental results, models trained with ERA outperform baselines across those three settings.", "questions": "I understand that LLM-RL training takes time and consumes resources. So I leave some comments as questions, not requests.\n- How the number of `top 20%` was selected in the LLM-RL instantiation? Have you tested different values to see how it affects performance?\n- Have you tested ERA in LLM-RL without the advantage modification? If this works out, ERA for LLM-RL can be just the activation layer change.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- ERA is a generic concept although this requires dedicated instantiations for each task and model.\n- Authors show strong performance improvements of each instantiation in through extensive experiments.\n-  For continuous control experiments, the authors show that ERA improves performance over different RL algorithms, which shows its generality.", "weaknesses": "- It's not clear why SAC without the entropy term in targets is outperformed by ERA since Q-function is basically trained through the same objective between two algorithms in this case. Is this because ERA provides a hard way to constrain the entropy range? I believe that an analysis on behavior of entropy helps to clarify this.\n- This is a minor point. In the LLM-RL setting, ERA needs to modify not only the logits of tokens, but also advantages. This seems beyond just an activation layer although the paper claims it proposes activation functions.\n- Authors should provide how many trials they conducted each benchmark to get statistics not only for continuous control benchmarks, but also for the rest of benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hSe40MUge9", "reviewer": "Reviewer_1XEa", "summary": "The paper presents a novel way to enforce a minimal entropy for the final distribution returned by a neural network. This is typically required for reinforcement learning tasks: you want to learn policies that are not overly deterministic and preserve some stochasticity, which is both good for exploration and robustness. In prior works, such constraint was softly imposed by adding an entropy term to rewards or Q-functions, and thus by maximising a modified Q-function: you maximize both cumulative rewards and an entropy of the policy. Indeed, that was one of the features of the soft-actor-critic algorithm considered as a baseline in this paper. ERA proposed by the paper introduces a new post-activation function that imposes a minimal entropy in the hard way. This is achieved by modifying the final std of the Gaussian, variable $\\sigma$. This approach is further extended to classification and LLM fine-tuning with GRPO tasks. The final results show consistent improvements over many baselines over many continuous control tasks, math solving benchmarks and classic image classification.\n\nI suggest a weak accept for the paper as the idea of imposing entropy through an activation is original and very useful in practice, as shown in the experiments, however the paper can benefit from improved writing, more clear problem formalism and rigourous proofs.", "questions": "**What needs clarification**\n- Section 4.1: It should be more explicit at which level of the neural network ERA is applied. For example, in case of tanh transformation to get the final policy, is ERA applied before tanh or after?\n- Section 4.2.1: In the main paper, it is not clear what is the exact definition of $\\delta$ and how it is different from $\\hat \\delta$. Also it is not clear whether $\\mathcal{H}’_0 = \\mathcal{H}_0 + \\delta$ or $\\mathcal{H}’_0 = \\mathcal{H}_0 + \\hat \\delta$. It becomes more clear after reading the proof, but it should be more transparent in the main paper as well.\n- Figure 3: explain what IQM stands for.\n- Proposition 2, proof: the last inequality in eq.25 and first equality in eq.26 are not clear. I think eq.26 might be wrong for the current choice of $h(x) = \\ln(-x\\ln x)$, while it is trivial for $h(x) = -x \\exp(x)$.\n- Proposition 3, proof: I don’t understand how do you get eq.35, also the reasoning after eq.35 for obtaining sufficient condition. Furthermore, eq.33 and eq.34 might not be completely true: $\\Delta H = \\sum_a \\frac{\\partial H}{\\partial z_a} \\Delta z_a$, but as updates are performed with respect to $\\theta$ using new policy gradient formula, $\\Delta z_a$ may not be equal to $\\frac{\\partial J}{\\partial z_a}$. Check https://arxiv.org/pdf/2405.19816, Eq. 2 for further understanding.\n\n\n**Technical questions**\n- page 4: Preactivation mean $\\mu$ and std $\\hat \\sigma$ are denoted in different styles, can it be homogenized?\n- page 5: why traditional entropy methods are incompatible with the on-policy setting (because of entropy collapse?)\n- Eq. 16: what transformation do you perform when $A_t$ is negative, but $H_{resp}< \\omega_{low}$ or  $H_{resp}> \\omega_{high}$?\n- Figure 2: why do you compare FastTD3 vs Fast Sac-ERA in the third plot? With respect to what do you normalise the performances? \n- What final transformation do you use? Tanh or truncated Gaussian?\n- Table 2,3: I see some discrepancies in reported performances of base Qwen2.5-Math and Qwen2.5-Math-Instruct on Olympiad and Minerva benchmarks with respect to the original paper https://arxiv.org/pdf/2409.12122, can you please comment on why there are some differences?\n- Figure 15: On which tasks did you do time measurements? I expect time complexity to be $O(D)$, while humanoid bench has tasks with D=19 and D=61.\n- What kind of distributions are $\\tau$-divided distributions? Can you provide an example?\n\n\n**General questions**\n- Do you follow any principle when choosing target entropy? In the appendix, the choice seems to be problem-dependent.\n- Is ERA sensitive to the choice of the interval $[\\sigma_{min}, \\sigma_{max}]$? In general, how do you define these intervals?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The idea of designing a special activation function that would enforce a minimal entropy is interesting and very useful in practice. With the help of many experiments, the authors showed that this activation function is sufficient to 1) keep entropy at a required level, 2) achieve superiour results compared to the trainings without ERA. I have appreciated that authors made a lot of experiments in different settings: for continuous control tasks they checked many difficult problems from Humanoidbench, DMC and Mujoco; for LLM  — finetuning 3 different sets of math competitions; for image classification — Cifar 10 and Imagenet. While improving results for many fixed architectures, ERA introduces only a slight time overhead.", "weaknesses": "The paper sometimes lacks mathematical rigour and details. It is the most noticeable when the authors try to formalise the RL for both classic control tasks and LLM. The authors do not introduce what is $s$, $a$, $x$ and $y$, therefore it is also not clear what is $y^{1:K}$. While it is indeed clear what are states and actions in context of RL, for readers that are not familiar with RL for LLM, which is relatively recent, understanding what are $x$ and $y$ and how they are connected to state-action formalism can be challenging. Some equations are written in complicated way, while can be simplified: for example $\\exp(\\max(…, \\log(\\sigma))$ can be simplified by writing  $\\max(\\exp(…), \\sigma)$. Finally, while most theoretical results look sound, the proofs in some parts are not clear and maybe erroneous (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)