When Maximum Entropy Misleads Policy Optimization
Authors: Ruipeng Zhang
Ya-Chien Chang
Sicun Gao
Venue: ICML 2025 poster (2025)
Keywords: MaxEnt RL
Primary area: reinforcement_learning

Abstract: The Maximum Entropy Reinforcement Learning (MaxEnt RL) framework is a leading approach for achieving efficient learning and robust performance across many RL tasks. However, MaxEnt methods have also been shown to struggle with performance-critical control problems in practice, where non-MaxEnt algorithms can successfully learn. In this work, we analyze how the trade-off between robustness and optimality affects the performance of MaxEnt algorithms in complex control tasks: while entropy maximization enhances exploration and robustness, it can also mislead policy optimization, leading to failure in tasks that require precise, low-entropy policies. Through experiments on a variety of control problems, we concretely demonstrate this misleading effect. Our analysis leads to better understanding of how to balance reward design and entropy maximization in challenging control problems.

When Maximum Entropy Misleads Policy Optimization
Ruipeng Zhang' Ya-Chien Chang
Sicun Gao
Abstract
SAC-Simplifedd
SAC-Realistic
PPO-Realistic
The Maximum Entropy Reinforcement Learning (MaxEnt RL) framework is a leading approach for achieving efficient learning and robust perfor- mance across many RL tasks. However, MaxEnt methods have also been shown to struggle with performance-critie control problems in practice, where non-Man Mat algorithms can successfully learn. In this work, we analyze how the trade- off between robustness and optimality affects the performance of MaxEnt algorithms in complex control tasks: while entropy maximization en- hances exploration and robustness, it can also mislead policy optimization, leading to failure in tasks that require precise, low-entropy policies. Through experiments on a variety of control prob- lems, we concretely demonstrate this misleading effect. Our analysis leads to better understand- ing of how to balance reward design and entropy maximization in challenging control problems.
a(mi
e(n)
Yfo
pim
Low-entrops policy is required on critical states ensure success
Success
High-entropy policy mediocre states leads to irrecoverable failure Failure
MaxEnt gets misled
Figure I. (Upper) In the quadrotor control environment, SAC learns well under simplified dynamics, but fails to learn when under realistic dynamics models. PPO can learn well despite the use the latter. (Lower) Intuitive illustration hard control prob- lems, where critical states naturally require low-entropy policies, while MaxEnt RL can favor mediocre states with robust policies low returns that branch towards failure and are not recoverable.
1. Introduction
Interestingly, the well-motivated benefits of MaxEnt and SAC have not led to its dominance in RL for real-world continuous control problems in practice (Shengren et al., 2022; Tan & Karaköse, 2023; Xu et al., 2021; Radwan et al., 2021). Most recent RL-based robotic control work (Kauf- mann et al., 2023; Miki et al., 2022; Zhuang et al., 2024) mostly still uses a combination of imitation learning and fine-tuning with non-MaxEnt methods such as PPO (Schul- man al., 2017). The typical factors of consideration that favor PPO over SAC in practice include computational cost, sensitivity to hyperparameters, and ease of customization. Often the performance by SAC is indeed shown to be infe- rior to PPO despite efforts in tuning (Muzahid et al., 2021; Lee & Moon, 2021; ;; Nair et al., 2024). In fact, it is easy to reproduce such behaviors. Figure/ shows the comparison of SAC and PPO for learning to control a quadrotor to fol- low path. When the underlying model for the quadrotor is a simplified dynamics model, SAC can quickly learn stable controller. When a more realistic dynamics model for the quadrotor is used, then SAC always fails, while PPO can succeed under the same initialization and dynamics.
The Maximum Entropy Reinforcement Learning (MaxEnt RL) framework (Ziebart et al., 2008; Abdolmaleki et al., 2018; Haarnoja et al., 2018a; Han & Sung, 2021) augments the standard objective of maximizing return with the ad- ditional objective of maximizing policy entropy. MaxEnt methods such as Soft-Actor Critic (SAC) (Haarnoja al., 2018a) have shown superior performance than other on- policy or off- off-policy methods (Schulman, 2015; Schulman et al., 2017; Lillicrap, 2015; Fujimoto et al., 2018) in many standard continuous control benchmarks (Achiam, 2018: Raffin et al., 2021; Weng et al., 2021; Huang et al., 2024). Explanations of their performance include better explo- ration, smoothing of optimization landscape, and enhanced robustness disturbances (Hazan et al., 2019; Ahmed et al., 2019; Eysenbach & Levine, 2021).
'Computer Seience and Engineering, UC San Diego Corre- spondence to: Ruipeng Zhang <<uu019@ucsd.cdux>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we show how the conflict between maximizing

---

When Maximum Entropy Mislcads Policy Optimization
entropy and maximizing overall return can be magnified and hinder learning in performance-criticr control problems that naturally require precise, low-entropy policies to solve.
We will first give a toy example showcase the core mis- leading effect of entropy maximization in Section 4, and then generalize the construction to the technique of entropy bifurcation extension in Section 5. We then show experi- mental results of how the misleading effects affect learning in practice, and how the adaptive tuning of entropy further validates our analysis Section 6.
The example of quadcopter control in Figure highlights common structure in complex control tasks: achieving de- sired performance often requires executing precise actions at a sequence of critical states. At these states, only a narrow (often sero-dimensional) subset of the action space fea- sible, hence the ground-truth optimal policy has inherently low entropy. (In aerodynamic terms, this is often referred to as flying on the edge of instability.) Conversely, actions that deviate from this narrow feasible set often lead to states that are not recoverable: once the system enters these states, all available actions tend to produce similarly poor outcomes but accumulate short-term "entropy benefits" that can be favored by MaxEnt. Over time, this drift can compound, ultimately pushing the system into irrecoverable failure.
2. Related Work
MaxEnt RL and Analysis. The MaxEnt RL. framework incorporates an entropy term in the RL objective and per- forms probability matching, such that the policy distribu- tion aligns with the soft-value landscape (Ziebart et al., 2008; Toussaint, 2009; Rawlik et al., 2013; Fox et al., 2015; O' D'Donoguue et al., 2016; Abdolmaleki al., 2018; Haarooja et al., 2018a; Mazoure et al., 2020; Han & Sung, 2021). MaxEnt RL has strong theoretical connections to probabilistic inference (Toussaint, 2009; Rawlik et 2013; Levine, 2018) and well-motivated for ensuring robustness from a stochastic inference (Ziebart, 2010; Eysenbach & Levine, 2021) and game-theoretic perspective (Grünwald & Dawid, 2004; Ziebart et al., 2010; Han & Sung, 2021; Kim & Sung, 2023). SAC and algorithms such as SAC- NF (Mazoure et al., 2020), MME (Han & Sung, 2021) and MEow (Chao et al., 2024) have outperformed most non- MaxEnt methods in many standard benchmarking environ- ments (Brockman, 2016; Todorov et al., 2012; Towers et al., 2024). A common explanation of the benefits of MaxEnt that it enhances exploration (Haarnoja et al., 201 8a; Hazan et al., 2019), smoothes the optimization landscape (Ahmed et al., 2019), and solves robust versions of the control prob- lems (Eysenbach & Levine, 2021). Despite the benefits, we show that they can also mislcad MaxEnt to converge to suboptimal policies in complex control problems.
Consequently, MaxEnt RL may bias the agent toward subop- timal behaviors rather than the precise low-entropy optimal policies that are key to solving hard control problems.
Formalizing this intuition, we give an in-depth analysis of the trade-off. Our main result is that for arbitrary MDP, there exists an explicit way of introducing entropy traps that we define as Entropy Bifurcation Extension, such that MaxEnt methods can be misled to consider an arbitrary pol- icy distribution as MaxEnt-optimal, while the ground-truth optimal policy is not affected by the extension. Importantly, this is not matter of sample efficiency or exploration bias during training, but is the end result at the convergence of MaxEnt algorithms. Consequently, the misleading effect of entropy can occur easily where standard policy optimization methods are not affected.
We then demonstrate that this concern is not theoretical, and can in fact explain key failures of MaxEnt algorithms in practical control problems. We analyze the behavior of SAC in several realistic control environments, including controlling wheeled vehicles at high speeds, quadrotors for trajectory tracking, quadruped robot control that directly corresponds to hardware platforms. We show that the gap between the value landscapes under MaxEnt and regular policy optimization explains the difficulty of SAC for con- verging to feasible control policies on these environments.
Practical Difficulties with MaxEnt in Control Problems. RL algorithms are well-known to be sensitive to parameter tuning (Wang & Ni, 2020; Muzahid et al., 2021: Nair et al., 2024). Various recent learning- based robotics control work reported that SAC delivers suboptimal solutions compared to PPO in complex control problems (Tan Karaköse, 2023; Xu al., 2021: Radwan et al., 2021). We aim to understand the discrepancy between such results and the generally good performance of MaxEnt algorithms (Haarnoja et al., 20188 8b; Achiam, 2018: Raffin et al., 2021).
Our analysis does not imply that MaxEnt is inherently un- suitable for control problems. In fact, following the same analysis, we can now concretely understand why MaxEnt leads to successful learning on certain environments that benefit from robust exploration, including common bench- marking OpenAl Gym environments where SAC generally performs well. Overall, the analysis aims to guide reward design and hyperparameter tuning of MaxEnt algorithms for complex control problems.
Trade-off between robustness and optimality. There is a long line of work studying the trade-off between robustness and performance in supervised deep learning (Su et al., 2018; Zhang et al., 2019; Tsipras et al., 201 8; Raghunathan et al., 2019; 2020; Yang et al., 2020). This trade-off in the RL context often overlaps with exploration issues mentioned above. Note that instead of sample efficiency or exploration issues, in this work we focus on pointing out the deeper issue
2

---

When Maximum Entropy Mislcads Policy Optimization
of misguiding policy optimization results al convergence.
MDP
SAC
3. Preliminaries
So
4
A Markov Decision Process (MDP) is defined by the tuple: M = (S, A, P. 7) where: S is the state space, A is the action space, which can be discrete or continuous, P(s"|s, a) the transition probability distribution, defining the prob- ability of transitioning to state s' after taking action at state r(s,a) a) is the reward function, which specifies the immediate scalar reward received for taking action at state s. y E [0, 1) is the discount factor. A policy is a mapping T : S + AA, where AA is the probability simplex over the action space A, defines probability distribution over actions given any state in S. We often write the distribution determined by policy π at state as (-|ss). The goal of standard RL is to find an optimal policy n² maximizes the expected return over the trajectory of states and actions to achieve the best cumulative reward.
Garle
Pt/ (s.a)
Qualttc!
Sg
0
Sb
PPO
AM4
-45
S+
20
-0:0 -031
10
Figure 2. (Left) MDP in the Toy Example: The MDP consists of an initial state so and two subsequent states Sy (good) and so (bad). It is clear that an optimal policy for so should be centered in the left half of the action interval, since only sy can transit to the terminal state sit with positive reward. (Right) Learning results of SAC and PPO at so at convergence. In the SAC plot, the soft Q-values (Sa, a) is higher for actions leading to sb, results in an incorrect policy centered in A2, the wrong action region (J:: dashed green line, a: green area). We also show the learned Q-values without entropy term with separate networks (red line), showing higher values for actions leading to 8g- the PPO plot, it learns the correct optimal policy.
Maximum Entropy RL extends the standard framework by incorporating an entropy term into the objective, encourag- ing stochasticity in the optimal policy. This modification ensures that the agent not only maximizes reward but also maintains exploration. Instead of maximizing only the ex- pected sum of rewards, the agent maximizes the entropy- augmented objective:
that the MaxEnt- policy is misled at an upstream state. We show simple example to illustrate this effect. Consider an MDP (depicted in Figure 2) defined as follows:
EE Σ (r( + (T( ((T)
(1)
1=
State space = {so. Sg. 86: si: gg sT. Here sp is the critical state for selecting actions. Sg is the "good" next state for sg, and sú is the "bad" next state, in the sense that only 8y can transit to the terminal state of with positive reward (under some subset of actions), while sy always transits to terminal state $6,7 with negative reward.
where H=)(s1s)) = - Ba (log T [a|s)] is the entropy of the policy at state s. The coefficient a 2 controls weight on entropy. The Bellman backup operator T is:
T"d QT a.) r(s), + (V....
(2)
where Vss:)= = (Q(:, a.) + aH (T(-)s:) is the soft state value at 8t- Treating Q-values as energy, the Boltzmann distribution induced then at state s is defined as:
Action space A = [-1, 1]. a continuous interval in R.
The transitions on so are defined as follows, reflecting the intuition mentioned above in the state definition:
[p((ss) exp(a exp(a- ((.)((())()))))((((1)
At state 8o, any action in A = [-1,0) leads to the good state deterministically, i.e., Va Ai, P(s while any action in A₂ = [0, 1] leads to the bad state, i.c., Va E A2, P(s, so. a) = 1. Note that allowing zero- dimensional overlap between the A₁ and Ag with random transitions at overlapping points will not change the results.
with (()(- = f (Q= (P da as the normaliza- tion factor. Policy update in MaxEnt RL at each state aims to minimize the KL. divergence between the policy T( T-1ss) and & (-Is). Naturally, π = मेे itself is an optimal policy at state 8 in the MaxEnt sense, since DKr.(T ( = O.
1,
The transitions from sgi P sg. 0-1, 0.1]) 1 and P(s p( 83. [1,, 0.11 U (0.1, 1]) 1. That is, for any ac- tion in small fraction of the action space, a' E 1-0.1,0.1) we can transit to the pesitive-reward terminal state sT-
4. A Toy Example
From the soft value definitions in MaxEnt, it is reasonable to expect some trade-off between return and entropy. But the key to understanding how it can affect the learning at convergence in major ways is by introducing intermediate states, where entropy shapes the soft values differently so
The transitions from 8, from any action deterministi- cally lead to the negative-reward terminal state, i.c. Va e A, (15,, d') =
3

---

When Maximum Entropy Mislcads Policy Optimization
The rewards are collected only at the terminal states, with r(s)) r(5g -20 and r() -1. The dis- count factor is set to 7 0.99 and is set to 1.
M
M
Backward
Given the MDP definition, it is clear that the ground truth policy at &p should allocate as much probability mass as possible (within the policy class being considered) for ac- tions in the A1 interval, because A1 is the only range of actions that leads to Sg. and then has a chance of collecting positive rewards on the terminal state s/. if the action on 8y correctly taken.
8 S
Compatibility
S
a(s
S'
S
A
A2
We can calculate analytically the soft Q values and the policy distributions that are MaxEnt-optimal (shown in Ap- pendix A.1). We can observe that the soft values of 8y and 96 will force MaxEnt to favor st at Sp- Indeed, as shown in Figure 2, the SAC algorithm with Gaussian policy has its mean converged to the center of Ag- The black curve in the SAC plot shows the soft Q-values of the actions, showing how entropy affects the bias. On the other hand, PPO cor- rectly captured the policy that favors the A, range. More detailed explanations are in Appendix A.I.
P = k
S'
(Terminal)
Forward Compatibility
Figure 3. MDP M and its entropic bifurcation extension M. The extension captures the intuition in the toy example, by using ad- ditional intermediate states which specifically designed reward to mislead MaxEnt-optimal policies that match the soft-Q landscapes.
We will illustrate how the misleading effect of entropy cap- tured in the toy example can arise in realistic control prob- lems through experimental results in Section 6.
1. We write (s()) = {s'|P(s' to denote the set of next states with non-zero transition probability from s. We introduce new states as follows:
(s > 0 for some a E Am
5. Entropic Bifurcation Extension
Building on the intuition from the toy example, we introduce a general method for manipulating MaxEnt policy optimiza- tion. The method can target arbitrary states in any MDP and "inject" special states with a special configuration of the soft Q-value landscape to mislead the MaxEnt-optimal policy. Importantly, the newly introduced states do not change the optimal policy on any non-targeted original states (in either the MaxEnt or non-MaxEnt sense), but can arbitrarily shape the MaxEnttoptimall policy on the targeted state. Thus, the technique can be applied on any number of states, and in the extreme case to change the entire MaxEnt-optimal policy to mirror the behavior of the worst possible policy on all states, thereby creating an arbitrarily large gap between the MaxEnt-optimal policy and the true optimal policy.
For any s' E (s), introduce a new state s that has not appeared in the state space Sy and let the corre- spondence between s' and s be marked as s pus'). We can then write S ((N(s)) for the set of all new states that are introduced in this way for state s.
At the same time, for each s", we introduce fresh state st that is a new terminal state, and write the set of such newly introduced terminal states as S4.
We now let the state space of MT be S SM US U S!. Note that S", S4 and SM are always disjoint.
2. At the original state s, for each s' E -s(s). we now set Par (s'1s, a) = O and (xx()())) ((((ss)=== [s, a) = PMr (s'ss, a). That is, we delay the transition to and let the newly introduced state s" = =s(s') take over the same transitions. For all other states in SM \ {s), the transitions in M and M are the same.
The key to our construction is to introduce new states that create a bifurcating soft Q-value landscape, such that the MaxEnt objective of probability matching biases the policy to favor states with low return and also can not transit back to desired trajectories, thus sabotaging learning.
3. For all the newly introduced states in S" their action space is a new A# C R. At each 8 E SK we will define transitions on two disjoint intervals, i.c., Ai" U As" A" and A n45" = 6. This partitioning is s -deeeeddnnnt (they will be used to tune the soft-Q landscapes). but for notational simplicity we will just write Al and AB when possible. Overall, the new MDP M has action space A = AM UA".
Definition 5.1 (Entropic Bifurcation Extension). Consider an arbitrary MDP M = (SM,An) PM, TM, 7) with contin- uous action space, the entropic bifurcation extension on M at state s is set (M,s) s) of new MDPs AM of form:
M xt. ir PMr, T ir 7) E {(M,s).
constructed with the following steps (illustrated in Figure 3);
4. At each sh the transitions are defined to produce bifurca-
4

---

When Maximum Entropy Mislcads Policy Optimization
tion behaviors, as follows. For any a E AK, P(8'|s", a) = 1. That is, such actions deterministically lead back to the s' in the original MDP. On the other side, any E A leads the the new terminal state, P(s P(st s* = 1. That is, the two intervals of action introduce bifurcation into two next states, both deterministically. This design generalizes the construction in the toy example, splitting the action space into one part that recovers the original MDP, and second part that leads non-recoverable suboptimal behaviors.
an arbitrarity fixed value for the original next state s', and r(s", a) an arbitrary reward for the newly introduced state st, Let U E R be arbitrary target value. Then, there exist choices of AK, Al. and r(sf)) such that V V(s*)= = U is the optimal soft value for the bifureating state st.
The lemmas ensure that for any transition in the original state, (s, a, s'), and for any value MM (s) and M (s') in M under some policy TM. there exist parameter choices in the bifurcation extension that maintains the same values of Var (s) and (s'),), while shaping the MaxEnt-optimal pol- icy arbitrarily. Consequently, the bifurcation extension can create an arbitrary value gap between the MaxEnttoptimal policy and the ground truth optimal policy at the targeted state 8. This leads to the main theorem:
5. The reward function 'M is the same as TM on all the original states and actions, i.e., "ny (s, a) = TM((s,) a) for all (s, a) E SM x AM- The reward on the new state is Txi(( (s", a) = 0 for any action a E AZ. On the new terminal state sy., we can choose r(sf)) to shape the soft-Q values as needed. The same discount y is shared between M and M.
Theorem 5.5 (Bifurcation Extension Misleads MaxEnt RL). Let M be an MDP with optimal MaxEnt policy n², and s an arbitrary state in Sy- Let (-|ss) be arbitrary distribution over the action space Am at state 8. We can construct an entropy bifurcation extension M of M such that M is equivalent to M restricted to SM \ \(s} and does not change its optimal policy on those states, while the MaxEnt-optimal policy after entropy bifureation extension can follow an arbitrary distribution ग T-|s)) over the actions.
Notation 5.2. The construction above defines the set of all possible bifurcation extensions (M). For any specific instance, the only tunable parameters are the size of |,4rl and |AW], as well as rewards on the newly introduced states s" and syy. We will show that these parameters already give enough degrees of freedom shape the soft Q-value landscapes and policy at the target state
Our main theorem relies on two important lemmas, which guarantee that we can use the newly introduced states to arbitrarily shape the policy at the targeted state s, without affecting the policy at another state in the original MDP.
Proposition 5.6 (Bifurcation Extension Preserves Optimal Policies). By setting r(s", ,a) = (1 (s') for every newly introduced bifurcating state slt = pus') and a E Ati the optimal policy is preserved under bifurcation extension.
Backward compatibility ensures that any fixed value of V(s) can remain invariant, by finding an appro- priate soft Q-value landscape at s. Importantly, this Q-landscape can be shaped to match an arbitrary policy distribution at 8.
Now, since this construction can alter the policy at any state without affecting other states, it can be independently used at any number of states simultaneously, and alter the entire policy of the MDP. In particular, the bifurcation extensions can force the MaxEnt-optimal policy to match the worst policy in the original MDP.
Forward compatibility ensures that by choosing ap- propriate r(s |441, AA the MaxEnt optimal value on V(s** can match arbitrary target values, using the original values on the original next states s' E (s)..
Corollary 5.7. Let M be an MDP whose optimal policy has value J and its worst policy (minimizing return) has value J= By applying entropy bifurcation extension on M on all states in M, we can obtain an MDP M whose MaxFintoptiiall policy has value J- while its ground-truth optimal policy still has value J+.
These two properties ensure the feasibility of shaping the MaxEnt-optimal policy at the targeted state s without af- fecting the policy on any other states of the original MDP. Formally, the lemmas can be stated as follows and the proofs are in the Appendix:
Remark 5.8. Our theoretical analysis does not need to use properties of function approximators or other components of practical MaxEnt algorithms, because the Maxtnna policies can be directly characterized and manipulated, as they must align with the soft-Q landscape. In the next section we show how this theoretical analysis explains the practical behaviors of SAC in realistic control environments.
Lemma 5.3 (Backward Compatibility). Let (-1s)) AM 10 be an arbitrary policy distribution over the action space at the targeted state Let U. E IR be arbitrary desired value for state s. There exists a value function V : S" + R on all the newly introduced states 8 such that U. is the optimal soft value of s under the MaxEnt-optimal policy at 8 (Definition 4).
6. Experiments
Lemma 5.4 (Forward Compatibility). Let s = (u(s)) be the newly introduced state for s' E (s).). Let V(s') be
We now show empirical evidence of how the misleading ef- fect of entropy can play crucial role in the performance of
5

---

When Maximum Entropy Misleads Policy Optimization
Vehicle
Quadrotor
OpenCat
Acrobot
Obstacle2D
3000
000
-000
-000
-000
-100
-000
200
300
-005
1201
-100
50K 14
261
3200
30ce
1.5M
1M
1.54
203
4226
116
16x
504
421
Step
Slep
Step
PPO
SAC
Figure+ 4. Reward performance of SAC and PPO across five environments with five random seeds. Note that we choose to show SAC and PPO because they are the best representatives of MaxEnt and non non-MaxEmn algorithms.
MaxEnt algorithms, both when they fail in complex control tasks and when they outperform non-M MaxEnt methods.
State
We first analyze the performance of the algorithms on con- tinuous control environments that involve realistic complex dynamics, including quadrotor control (direct actuation the propellers), wheeled vehicle path tracking (nonlinear dynamic model at high speed), and quadruped robot con- trol (high-fidelity dynamics simulation from commercial project (PetoiCamp)). We show how the soft value land- scapes mislead policy learning at critical states that led to the failure of the control task, while non-MaxEnt algorithms such PPO can successfully acquire high-return actions.
noeer
Quaic
Nytner
Track
Target
Vahicle Path
Figure 5. Soft and Plain Q-value landscapes in Vehicle. (Left) Q landscapes with Qun a) and without Qptin a) entropy. Introduced Entropy in SAC elevates the true Q values to encour- age exploration, risking missing the only feasible optimal actions. (Right) Rendering of the queried state. The grey rectangle denotes the vehicle with the black arrow as its heading direction. The cur- rent SAC policy steers the vehicle left and moves forward, while the PPO policy reasonably steers it back on track with braking, aligning with the optimal region indicated by Qplain.
We also revisit some common benchmark environments to show how the superior performance of MaxEnt can be attributed to the same "misleading" effect that prevents it from getting stuck as non-MaxEnt methods. It reinforces more well-known advantages of MaxEnt with grounded explanations supported by our theoretical understanding.
To further validate our theory, we add new adaptive entropy tuning in SAC, enabling it to switch from soft-Q to normal Q values when their landscapes have much discrepancy. We then observe that the performance of SAC is improved in the environments where was failing. In particular, the newly learned policy acquires visibly-better control actions on critical states. This form of adaptive entropy tuning is not intended new algorithm it relics global estimation of Q values that is hard to scale. Instead, the goal is to show the importance of understanding the effect of entropy, as directions for future improvement of MaxEnt algorithms.
quadruped robot (PetoiCamp). The action space is the 8 joint angles of the robot. Acrobot is a two-link planar robot arm with one end fixed at the shoulder and the only actuated joint at the elbow (Spong, 1995). The action is changed to continuous torque values instead of simpler discrete values in OpenAl Gym (Brockman, 2016). In Obstacle2D, the goal is to navigate an agent the goal behind wall, which creates a clear suboptimal local policy that the agent should learn to avoid. Hopper is the standard MuJoCo environ- ment (Todorov et al., 2012) where SAC typically learns faster and more stably than PPO.
Environments. In the Vehicle environment, the task is to control a wheeled vehicle under the nonlinear dynamic bicycle model (Kong et al., 2015) move at a constant high speed along path. Effective control critical for steering the vehicle onto the path. In the Quadrotor environment, the task is to control quadrotor to track a simple path un- der small perturbations. The actions are the independent speeds of its four rotors (Rubí et al., 2020), which makes the learning task harder than simpler models. The Open- cat environment simulates an open-source Arduino-based
Overall Performance. Fig. 4 shows the overall perfor- mance comparison of the learning curves of SAC and PPO across environments. We notice that SAC performs worse than PPO in the first three environments that are harder to control under complex dynamics, while significantly outper- forming PPO in Acrobot, Obstacle2D, and Hopper (shown in Appendix D.3). Our goal is to understand how these be- haviors are affected by entropy in the soft value landscapes.
6

---

When Maximum Entropy Mislcads Policy Optimization
6.1. Misleading Soft-Q Landscapes
S (s, a)
Q plain (s, a)
In Figure we show the Q values on critical state in the Vehicle control environment, where the vehicle is about to deviate much from the path to be tracked. Because of the high entropy of the policy on the next states where the vehicle further deviates, the soft-Q value landscape at this state is as shown in the top layer on the left in Figure 5. fails to understand that critical actions are needed, but instead encourages the agent stay in the generally high soft-Q value region, where the action the center is shown as the green arrow the right-hand side plot in the Figure 5. It is clear that the action leads the vehicle to aggressively deviate more from the target path. On the other hand, the plain Q value landscape, as shown in the bottom layer on the left, uses exactly the same states that the SAC agent collected in the buffer train, and can realize that only small region in the action space corresponds to greater plain Q values. The blue dot in the plot, mapped to the blue action vector on the vehicle, shows a correct action direction that steers the vehicle back onto the desired path. Notably, this the action learned by PPO at this state, and it generally explains the success PPO this environment.
T
Torque #22
SAC
T
500
400
69
Torque
Torque
Current state
TSCC
No
Roz
Figure 7. (Upper) Comparison soft-Q (left) and plain-Q (right) value landscapes the current state shown below. (Lower) The second fourth snapshots show Hopper's state after taking actions at the circled position of corresponding colors in the action space shown above. The SAC policy benefits from entropy by 'leaning forward', risky move despite this action being suboptimal in the current ground truth value.
State
6.2. Benefits of Misleading Landscapes
Target
It is commonly accepted that the main benefit of MaxEnt is the exploratory behavior that it naturally encourages, es- pecially in continuous control problems where exploration is harder to achieve than in discrete action spaces. That is, entropy is designed "intentionally mislead" to avoid getting stuck local solutions. We focused showing how this design can unintentionally cause failure in learn- ing, but the same perspective allows us to more concretely understand the benefits of MaxFnt in control problems. We brietly discuss here and more details are in the Appendix.
PPQ
Pubret
Vokoet #1
Figure 6. Q landscapes in Quadrotor. (Left) The current state is at the end of the black trajectory. The Red dashed line is the target track. (Right) Que and Quaiin at this state. SAC fails to push upward with minimal action at this state, leading to failure against gravity. PPO succossfully applics greater thrust to the back motor (#3), flying the quadrotor towards the path.
In Figure 6, we observe similar behaviors in the Quadrotor control environment. The quadrotor should track hori- zontal path. The controller should apply forward-leaning thrusts to the two rotors parallel to the path, while balancing the other two rotors. The middle and right plots in Figure 6 show the MaxEnt and plain Q values in the action space for the two rotors aligned with the forward direction. Again, because of the high entropy of soft-Q values on mediocre next states that will lead failure, the value landscape of SAC favors center at the green dot in the plots, which corresponds to actions of the green arrow in the plot on the left. In contrast, the plain Q value landscape shows that actions high quality are centered differently. In particular, the blue dot indicates a good action that can be acquired by PPO, controlling the quadrotor towards the right direction.
Figure shows the comparison between the soft-Q and plain Q landscapes for the training process in Hopper, plotted particular state shown in the snapshots in the figure. The action learned by SAC is in fact not high-return action at this point of the learning process. According to the plain Q values, the agent should take actions that lead to more stable poses. However, after this risky move of SAC, the na- ture of the environment makes possible to achieve higher rewards, which led successful learning. Figure 8 shows the similar positive outcome of MaxEnt encouraged by the soft value landscape. Overall, it is important to note that the effectiveness of MaxEnt learning depends crucially on the nature of the underlying environment, and our analysis aims to give a framework for understanding how reward design and entropy- exploration should be carefully balanced.

---

When Maximum Entropy Mislcads Policy Optimization
Environment
Soft valuee
Advantage valuss
and prioritizing exploitation in states where the misleading effects may result in failure. Note that SAC-AdaEnt is different from SAC with an auto-tuned entropy coefficient with uniform entropy adjustment across all states.
Figure 9 shows how the adaptive tuning of entropy in SAC- AdaEnt affects learning. both the Vehicle and Quadrotor enviromments, the policy learned by SAC-AdaEnt mostly corrects the behavior of the SAC policy, as illustrated in their overall trajectories and the critical shown in the plots.
Ob
Acrobot
PPO
Note that the simple change of SAC-MaxEnt is not intended as a new efficient algorithm, because measuring the discrep- ancy of the Q landscapes requires global understanding at each state, which is unrealistic high dimensions. It does confirm the misleading effects of entropy in control environ- ments where the MaxEnt approach was originally failing, More details of the algorithm are in Appendix E.
Figure 8. Q/Advantage landscapes SAC and PPO in Obstacle2D and Acrobot. (Upper) In the Obstacle2D environment, SAC successfully bypasses the wall while PPO fails, explained by the Soft-Q/ -/Avat//aaat landscapes. (Lower) In Acrobot, SAC learns more stable control policy (applying the right torque to neutralize momentum to prevent failing) while PPO updates are stuck at local solution that fails robustly stabilize.
7. Conclusion
We analyzed a fundamental trade-off of the MaxEnt RL framework for solving challenging control problems. While entropy maximization improves exploration and robustness, it can also mislead policy optimization towards failure.
Vehicle
59
300
We introduced Entropy Bifurcation Extension to show how the ground-true policy distribution at certain states can be- come adversarial to the overall learning process in MaxEnt RL. Such effects can naturally occur in real-world control tasks, where states with precise low-entropy policies are essential for success.
-000
Quadrotor
200
Our experiments validated the theoretical analysis in practi- cal environments such as high-speed vehicle control, quadro- tor trajectory tracking, and quadruped control. We also showed that adaptive tuning of entropy can alleviate the misleading effects of entropy, but may offset its benefits too. Overall, our analysis provides concrete guidelines for under- standing and tuning the trade-off between reward design and entropy maximization in RL for complex control problems. We believe the results also have implications for potential adversarial attacks in RL from human feedback scenarios.
Figure 9. Performance of SAC-AdaEnt v.s. SAC. (Left) Learning curves. (Middle) Full trajectory rendering. (Right) Behavior of policy critic states. In Vehicle, SAC-AdaEnt successfully steers and brakes to bring the vehicle back on track. In Quadrotor, it effectively lifts the quadrotor to follow the designated path.
6.3. Importance of Adaptive Entropy Scaling
To further test the theory that the use of soft values in Max- Ent can negatively affect learning, we modify the SAC algo- rithm by actively monitoring the discrepancy between the soft Q-value landscape and the plain Q-values.
A Acknowledgment
We thank the anonymous reviewers for their helpful com- ments in revising the paper. This material is based work supported by NSF Career CCF 2047034, NSF CCF DASS 2217723, and NSF AI Institute CCF 2112665.
We simultaneously train two networks for Quei and Qplain- During policy updates, we sample from action space each state under the current policy and evaluate their Quon and Qplain values. If Quo deviates significantly from Qplain it indicates that entropy could mislead the policy, and we rely on Pplain as the target Q-value for the policy update instead. This adaptive approach, named AAC-AdaEnt, ensures a balance between promoting exploration in less critical states
Impact Statement
Our paper offers both theoretical and experimental insights without immediate negative impacts. However, it con- tributes to deeper understanding of entropy maximization
8

---

When Maximum Entropy Mislcads Policy Optimization
principles and their role in reinforcement learning, laying the groundwork for future advancements in MaxEnt RL that may also involve new models of adversarial attacks and defense on RL-based engineering of critical Al systems.
Han, S. and Sung, Y. A max-min entropy framework for reinforcement learning. Advances in Neural Information Processing Systems, 34:25732-25745, 2021.
Hazan, E., Kakade, S., Singh, K., and Van Soest, A. Prov- ably efficient maximum entropy exploration. In Interna- tional Conference on Machine Learning, pp. 2681-2691. PMLR, 2019.
References
Abdolmaleki, A., Springenberg. J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. Maximum a posteriori policy optimisation. arXiv preprint arXiv: 1006.v8800.066 1806.06920, 2018.
Huang. S., Gallouédec, Q. Felten, F., Raffin, A., Dossa, R. F. J.. Zhao, Y., Sullivan, R., Makoviychuk, V., Makovi- ichuk, D., Danesh, M. H., Roumégous, C., Weng, J., Chen, C., Rahman, M. M., M. Araújo, J. G., Quan, G., Tan, D., Klein, T., Charakorn, R., Towers, M., Berth- elot, Y., Mchta, K., Chakraborty, D., KG, A., Charraut, V., Ye, C., Liu, Z., Alegre, L. N., Nikulin, A., Hu, X., Liu, T., Choi, J., and Yi, B. Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning, arXiv preprint arXiv:24022.03046 2024, URL https: / aaapaaxvvvv...////.a///aa org. abs abtaaaaa/2242 240. 03046.
Achiam, J. Spinning Up in Deep Reinforcement Learn- ing. 2018. URL https: /ttp::///ghhbbbb// com oppnaa//e///////////// spinningup.
Ahmed, Z., Le Roux, N., Norouzi, M., and Schuurmans, D. Understanding the impact of entropy on policy optimiza- tion. In International conference on machine learning, pp. 151-160. PMLR, 2019.
Brockman, G. Openai gym. 1606,00100 2016.
arXiv preprint
Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V., and Scaramuzza, D. Champion-level drone racing using deep reinforcement learning. Nature, 620 (7976)992-996) 2023.
Chao, C.-H., Feng, C., Sun, W.-F., Lee, C.-.,, See, S., and Lee, C.-Y. Maximum entropy reinforcement learn- ing via energy-based normalizing flow. arXiv preprint arXiv:2405. 13629, 2024.
Kim, W. and Sung, Y. An adaptive entropy-regularizzate framework for multi-agent reinforcement learning. In In- ternational Conference on Machine Learning, pp. 16829- 16852. PMLR, 2023.
Eysenbach, B. and Levine, S. Maximum entropy rl (prov- ably) solves some robust rl problems. arXiv preprini arXiv:2103.06257, 2021.
Kong, J., Pfeiffer, M., Schildbach, G., and Borrelli, F. Kine- matic and dynamic vehicle models for autonomous driv- ing control design. In 2015 IEEE intelligent vehicles symposium (IV), pp. 1094-1099, IEEE, 2015.
Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562. 2015.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- lional conference on machine learning, pp. 1587-1596. PMLR, 2018.
Lee, M. H. and Moon, J. Deep reinforcement learning- based uav navigation and control: A soft actor-critic with hindsight experience replay approach. arXiv preprint arXiv:2106.01016 2021.
Grünwald, P. D. and Dawid, A. Game theory, maximum entropy, minimum discrepancy and robust bayesian deci- sion theory. 2004.
Levine, S. Reinforcement learning and control as proba- bilistic inference: Tutorial and review. arXiv preprint TrXiv:/805.000 2018.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018a. PMLR.
Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint
2015. Mazoure, B., Doan, T., Durand, A., Pincau, J., and Hjelm, D. Leveraging exploration in off off-policy algorithms via normalizing flows. In Conference on Robot Learning, pp. 430-444, PMLR, 2020.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint /rxiv: 2,5905, 2018b.
Miki, T., Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., and Hutter, M. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science robotics. 7 (62)eeabk2822, 2022.

---

When Maximum Entropy Mislcads Policy Optimization
Muzahid, A. J. M., Kamarulzaman, S. F., and Rahman, M. A. Comparison of ppo and sac algorithms towards decision making strategies for collision avoidance among multiple autonomous vehicles. In 2021 International Conference on Software Engineering & Computer Systems and 4th International Conference on Computational Science and Information Management (ICSECS-ICOCSIM), pp. 200- 205. IEEE, 2021.
Shengren, H., Salazar, E. M., Vergara, P. P., and Palensky, P. Performance comparison of deep rl algorithms for energy systems optimal scheduling. In 2022 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Furope), ) pp. 1-6. IEEE, 2022.
Spong, M. W. The swing up control problem for the acrobot. IEEE control systems magazine, 15(1):49-555 1995.
Nair, V. G., D' O'Suzza, J. M., Asha, C., and Rafikh, M. A scoping review on unmanned aerial vehicles in disaster management: Challenges and opportunities. Journal of Robotics and Control (JRC), 1799-1869, 2024.
Su, D., Zhang, H.,, Chen, H., Yi, J., Chen, P.-Y., and Gao, comprehensive
Y. Is robustness the cost of accuracy? study on the robustness of 18 deep image classification models. In Computer Vision - ECCV 2018, pp. 644-661. Springer International Publishing, 2018.
O'Donoghue, B., Munos, R., Kavukcuoglu, K., and Mnih, Combining policy gradient and q-learning. arXiv preprint arXiv:1611.01626, 2016.
Tan, and Karaköse, M. A new approach for drone track- ing with drone using proximal policy optimization based distributed deep reinforcement learning. SoftwareX, 23: 101497, 2023.
PetoiCamp. Opencat: Open-source quadruped robot. URL https://git thub. com/ Petoi Camp / OpenCat? tab-readme ov- file.
Todorov, E.. Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ inter- national conference on intelligent robots and systems, pp. 5026-5033 IEEE, 2012.
Radwan, M. O., Sedky, A. A. H., and Mahar, K. M. Ob- stacles avoidance of self- driving vehicle using deep rein- forcement learning. In 2021 31st International Confer- on Computer Theory and Applications (ICCTA), pp- 215-222, IEEE, 2021.
Toussaint, M. Robot trajectory optimization using approxi- mate inference. In Proceedings of the 26th annual inter- national conference on machine learning, pp. 1049-1056, 2009.
Raffin, A., Hill. A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines3: Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021.
Towers, M., Kwiatkowski, A., Terry, J., Balis, U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 2024.
Raghunathan, A., Xie, S. M., Yang. F., Duchi, J. C.,, and Liang, P. Adversarial training hurt generalization. arXiv preprint arXiv: 0006699066032 2019.
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. Robustness may be at odds with accuracy.
Raghunathan, A., Xie, S. M., Yang, F., Duchi, J., and Liang, P. Understanding and mitigating the tradeoff between ro- bustness and accuracy. arXiv preprint arXiv:2002.10711 2020.
arXiv preprint arXiv: 2018.
Wang. Y. and Ni, Meta-sac: Auto-tune the entropy temperature of soft actor-eritic via metagradient. arXiv preprint arXiv:2007.01932, 2020.
Rawlik, K., Toussaint, M., and Vijayakumar, S. On stochas- tic optimal control and reinforcement learning by approx- imate inference. 2013.
Weng, J., Chen, H., Yan, D., You, K., Dubureq, A., Zhang, M., Su, H., and Zhu, J. Tianshou: A highly modular- ized deep reinforcement learning library. arXiv preprint arXiv:2107 14171, 2021.
Rubí, B., Pérez, R., and Morcego, B. A survey of path fol- lowing control strategies for uavs focused on quadrotors. Journal of Intelligent Robotic Systems, 98(2):441226,, 2020.
Xu, C., Zhu, R., and Yang, D. Karting racing: A revisit to ppo and algorithm. In 2021 International Con- ference on Computer Information Science and Artificial Intelligence (CISAI), pp. 310-316. IEEE, 2021.
Schulman, J. Trust region policy optimization. arXiv preprint arXiv:/502.0553 2015.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.0000 2017.
Yang, Y.-Y,, Rashtchian, C., Zhang, H., Salakhutdinov, R., and Chaudhuri, K. closer look at accuracy vs. robust- ness. arXiv preprint arXiv:2003. 024400 2020.
10

---

When Maximum Entropy Mislcads Policy Optimization
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L.,, and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning, pp. 7472- 7482. PMLR, 2019.
Zhuang, Z., Yao, S., and Zhao, H. Humanoid parkour learning, arXiv preprint 200..2990..1759, 2024.
Ziebart, B. Modeling purposeful adoptive behavior with the principle of maximum causal entropy. Carnegie Mel- lon University, 2010.
Ziebart, B. D., Maas, A. L.,, Bagnell, A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. Aaai, volume pp. 1433-1438. Chicago, IL, USA, 2008.
Ziebart, B. D., Bagnell, D., and Dey, A. K. Maximum causal entropy correlated equilibria for markov games. In Workshops at the Twenty- Fourth AAAI Conference on Artificial Intelligence, 2010.

---

When Maximum Entropy Mislcads Policy Optimization
A. Details on the Toy Example
A.I. Calculation of soft Q(s0, a) for SAC
PPO
SAC
00
Sg
-40
2--4.375.--0.110
pm-0216 en8877
725000/ o d40 Roden 1.71 -
055y 0'
Soft
Sb
610.6% a11l
AdVis #1
-4.00
-8.
-016
-2.0
200
100
-0.3
6.06
Figure Toy Example Results SAC and PPO at states 8g and 8b
In the MaxEnt framework, the policy at so is iteratively updated towards the Boltzmann distribution T2 (-1o)) Given the simple transitions in the MDP, we can easily calculate the Q values for any action. We use a = for the entropy coefficient.
A.1.I. DIRECT CALCULATION WITHOUT PARAMETRIZATION
Since transitions from so 8g and 8, deterministic and yields reward, have
Q(so. a) = (sg///)) NLE ~ (//g//b/C./ d') alog al'og(o'lss,) = 7 T(a'ls//h)) log Z(5 7 log (s/ph))
under the optimal policy क = x expla - Sappla-Q((g//,//o [((t)/0,'))//(((p
For [-1,0) which transits 5g*
@(50. a) 7 log Z(sg) = 7 log( / (')) = log( L.
_o.1 el da'+ frem 0.1
0.1
e-20 da't I' da2 00600,
0.1
and [0, 1] which transits to Sb.
(Ss -log Z(su) y log( / explQ(st, = log( /" c- da') -0.304,
As Q0.0 (-1,0) < Q(00, [0, 1)). SAC theoretically expected incorrectly select a E [0, 1] as the optimal policy.
A.1.2. CALCULATION WITH EMPIRICAL PARAMETERIZATION
Empirically with Gaussian policy, we also compute Q(so, a) given the policies on 8g: 8b, shown in Figure 10. In practice, TT,)) Squash[[(u(s,, ar (s,, ]}, (ss)) = Squash(N(((s where ((sg) 0.013, (s,)) 0.027, ((st) = 0.016, (st)) = 0.877 specifically, thus
( a) (s,,/)) > EE ( (9(8//.. a alog ((a /s//)) log(1 tanh t (a')))
We can compute this numerically as
Q(s0.
= a) >V((,) ~ -1.696 for E [-1,0)
Q(s0, a) - VV()) ~ 00.88 for a E [0
Those values are consistent with the results in Figure 2. Consequently, as expected, MaxEnt algorithms such as SAC quickly converge to the MaxEnt-optimal policy that leads almost all trajectories the terminal state šØ. with negative rewards.
12

---

When Maximum Entropy Mislcads Policy Optimization
A.2. Results of PPO policies
The advantage landscapes at so, 8q: 56 are shown in Figure and Figure 10, From those, PPO is observed converge the correct optimal policy.
A.3. Remarks on arbitrary a
Although we set a in the toy example for simplicity, it can be an arbitrary non-negative value.
Remark A.1. If MaxEnt policy is mislead at so i.c. Q(50, ala E [[1,,0) < Q(50. ala E [0, 1J) when a 1, for arbitrary
+ I, we can keep misleading MaxEnt policy through reward scaling
f(s = ar(s)), f(5,,r) = ar(s,,T) f(5,,T) = ar(s)Tr)
Proof. For arbitrary
((s0,a) a) =Bau' (Q09g/00000000 a log (a' <(aass//b)]] = 7aa yalog ($//b)
where x matches the optimal softmax policy exp(Q(s and Zoza (s//b) = I exp 2ltorniad
For f(*7), f(5 f(, ((5,r)) we have Q( a') = (5/// a'), and Zq/al5//1b) - exp Q( d')a. Therefore the ordering of Zoja(s,) and (o)a(s))) is the same as the original Zo (s,)) and (s())).
Remark A.2. For arbitrary a, we can always find ri so that the optimal policy for standard RI. (e.g. PPO) will favor $g while MaxEnt policy will favor sb, i.c. Q(so, ala ()) < Q(so, ala [0,
Proof. Let (50. a/a E (-1,0)) < Q(so, ala E [0, 1)), we can get
log ZQ/a(19) < log (s/a(ss))
Q(sg: a') da' loglexp( (b) |A]] a log exp (0) dá' - a a A < rn <,
log
exp
a
Given a log exp Qlsynaa) da' S max, Q(sg: a') + a alog/A1, we have left-hand side is upper bounded by max, Q(8g, rt. As long sup, a() rt, the open interval I(sg) aa log exp Q(sgaa) da' log [AI, r+ is non-empty. Therefore, always pick ry e I(sg) that Q( ala e [-1,0) Q(so. E [0,
expp
ie. MaxEnt favors Sn-
B. Full Proofs
Lemma B.1 (Lemma 5.3, Backward Compatibility). Let (-1s)) Au + 10, 1] be an arbitrary policy distribution over the action space at the targeted state s. Let v, E R be an arbitrary desired value for state s. There exists a value function V SH > R on all the newly introduced states 8 such that v, is the optimal soft value of s under the MaxEnt-optimal policy at s (Definition 4).
Proof. Based on the definition of soft value,
V(st) Ea (Q(r, a:)l + add (T((s))
(4)
need to show that there exists function Q (s) Am - IR such that
U/ V(s) =E,, EE IQ(s, a) + (H((s)))))
13

---

When Maximum Entropy Mislcads Policy Optimization
which minimizes the KL-divergence between and the Boltzmann distribution induced by Q. i.e.,
PRI((T() T
To ensure ( O, we can directly construct Q(s, a) such that it matches the optimal policy
mo(a|s) = exp (a (a.)Q((s
Z(Q)
with normalization term Z(Q) = SAN, exp (a- (o(ssa)) a)) da.
Taking logarithms on both sides and rearranging for ((s,
Q(s.a) a log T(a s) log ((s),
(5)
where the normalization factor constant that we can arbitrarily choose without changing the KL divergence. Let a log (s).
Next, taking expectation
E (1)x((a)(((saa))) - x(a ()Q({
Substituting in ((s, a) from Eq. (5), we have
1elat = T (a + c) da
/ (als) log + e / (als)da - +c.
Now, to match the soft value function (ss), we can set:
(zl((())ss)) -cl((((()) + Eawn Q(s a)] + (Il((()))) = all((((1s))
(6)
Thus, solving for c, we obtain Va-
Substituting back into Eq. (5), we get
((s,a) a) = a + v.. (H(((-)))) This ensures that both v, V(8) = Ba B() +
and (RL = 0 are satisfied.
Lemma B.2 (Lemma 5.4. Forward Compatibility). Let s = p(s') be the newly introduced state for s' N()s.. Let V(s') be arbitrarily fixed value for the original next state s', and r(s" a) an arbitrary reward defined for the newly introduced state sH, Let u E R. be arbitrarily chosen target value. Then, there exist choices of Ali Ab, and r r(st)) such that (ss)) = is the optimal soft value for the bifurcating state st.
Proof. Following the definition of the MaxEnt value (Definition 4), we need to show:
V(s = Ee-xx(-e=)) (0)
(8" a)] + a (T((ss"))
where (-(ss)) is the policy distribution at s that exactly matches the Boltzmann distribution induced by some Q-function Qss a), i.e., (πll@T =
With the bifurcating action space A AW U Ali and deterministic transitions, we define:
Q r(s" a) + (Vs',
(8)
Q2 (T(#
(9)
14

---

When Maximum Entropy Misleads Policy Optimization
where r(s", a) is an arbitrarily fixed reward for any a A4 and for any Ab2 set r(s ,)) 0. Since r(s", a) and (is') are fixed, only Q2 is tunable via the choice of r()))-
A policy that minimizes the KL-divergence with TQ at s is:
Af /a
E A
(als"))
292/0
A/.
and we define the normalization term as:
((Q) = IAgle + LAble
The probabilities over action subspaces are:
P1 14//e
P2
Z(Q)
Z(Q)
The expected value is:
Baw (0(", ,a)] p1Qi + P2Q2.
(10)
The entropy of x(-ls") is:
H(T(+Is*)) +: log Z(Q),
so
aH(T(-1s"))
+ log Z(Q).
(11)
Substituting Eqs. 10 and 11 into Eq. 7:
V(s") = pi Q1 + p2Q2 + +HH(T(-))"))
-1 + P242 - (p1Q: + +2222) + a log Z(Q)
log Z(Q).
(12)
Thus, in this corrected version, we observe that:
v(st) = a log (1AtleRula +1ffeeula+|A//ce2l4 (Ab1)
Solving for r( we get:
1
(60/a 14/1 )
r(s7)
7 a log
A
which is valid for all a > 0, and the function
V(s") = a log Z(Q)
a surjection onto It when varying over [AH).. Ag and r(* we conclude that for any v E IR, a valid construction exists such that ((s v.
Theorem B.3 (Theorem 5.5, Bifurcation Extension Misleads MaxEnt RL). Let M be an MDP with optimal MaxEnt policy n², and s an arbitrary state in SM1- Let T-(ss) be an arbitrary distribution over the action space Ast at state s. We can construct an entropy bisurcation extension M of M such that M is equivalent to M restricted to SM \ {8} and does not change its optimal policy on those states, while the MaxEnt-optimal policy at after entropy bifurcation extension can follow an arbitrary distribution T(-(s) over the actions.
In other words, without affecting the rest of the MDP, we can introduce bifurcation extension at an arbitrary state such that the MaxEnt optimal policy becomes arbitrarily bad at the affected state.
15

---

When Maximum Entropy Mislcads Policy Optimization
Proof. Following Lemma 5.4, we introduce the bifurcation extension Definition 5.1 and obtain Q-values on all the newly introduced states Q(s, a) such that V(s) remains unchanged, while the MaxEnt optimal policy becomes x ((ss). Given such target Q(s, a),, which now impose target values on the introduced bifurcation states, i.e., V(s*) = Q(s, a) P(s a), because by construction P(s*|s,a) a) = P(s'|s, a) > O. We then use the forward compatibility Lemma 5,4 to set the parameters in the bifurcation extension, such that (s*t) is attained by the MaxEnt policy at s without changing the existing values on the original next states (is') for any s' E (s).. Since we have not changed the values on or any s' E (s), the bifurcation extension does not affect the policy on any other state in SM \ {t). At the same time, the target arbitrary policy π T-|ss) is now a MaxEnt optimal policy at s.
Proposition B.4 (Proposition 5.6, Bifurcation Extension Preserves Optimal Policies). By setting r(s". .a) (1 (1-)) for every newly introduced bifureating state s pus') and E Ai. the optimal policy is preserved under bifurcation extension.
Proof. In the Mon-MaxEnt setting, the state value of s* maximizes the Q-value, and the optimal policy chooses the actions in Ar. Since r(s", a) = (1 - (s'), the additional reward on (st a) ensures that V (s*)) (s')). Note that Lemma 5.4 holds for arbitrary r(s". a). Consequently, there is no change in Q(s, a) and the optimal policy remains the same between M and M.
C. Environments
C.1. Vehicle
The task is control wheeled vehicle to maintain constant high speed while following designated path. Practically, the vehicle chases moving goal, which travels a constant speed, by giving negative rewards to penalize the distance difference. Also the vehicle receives a penalty for deviating from the track. The overall reward is
ggaa + Tirack -llveehiele-
Paoaill 1a + 3, Reetiice
R
where Pvehiele: Pgaul are the positions for vehicle and the goal respectively, 8, = 0.3 is the scaling factor, Runck 10 is the radius of the quarter-circle track and Ryehiele v / + Mvehicle is the vehicle's 's radial distance from the origin. The initial state is set to make steering critical for aligning the vehicle with the path, given initial forward speed of 3. The action space is steering and throttle. The vehicle follows dynamic bicycle model (Kong et al., 2015), where throttle and steering affect speed, direction, and lateral dynamics. It introduces slip, acceleration, and braking, requiring the agent to manage stability and traction for precise path tracking.
C.2. Quadrotor
The task is to control quadrotor to track a simple path while handling small initial perturbations. The quadrotor also chases a target moving at a constant speed. The reward is given by the distance between the quadrotor and the target, combined with a penalty on the quadrotor's three Euler angles to encourage stable orientation and prevent excessive tilting.
The overall reward function is:
rgoal + Sstability
= - Pquudrotor Ptarger l/ 10| |0|
|+)
where Pqundrotors Puaget are the positions for quadrotor and the target respectively, 0, d, vb are the Euler angles, | 5 is a scaling factor. Simplified: Since the track aligns with one of the rotor axes, we fix the thrust of the orthogonal rotors to zero, providing additional stabilization. The agent controls only the thrust and pitch torque, simplifying the task. Realistic: The agent must fully control all four rotors, with the action space consisting of four independent rotor speeds, making stabilization and trajectory tracking more challenging.
C.3. Opencat
The Opencat environment simulates an open-source Arduino-based quadruped robot, which is based on Petoi's OpenCat project (PetoiCamp). The task focuses on controlling the quadruped's d's joint torques to achieve stable locomotion while adapting to perturbations. The action space consists of 8 continuous joint torques, corresponding to the two actuated joints for each of the four legs. The agent must learn to coordinate leg movements efficiently to maintain balance and move toward.
16

---

When Maximum Entropy Mislcads Policy Optimization
C.4. Acrobot
Acrobot a two-link planar robot arm with one end fixed at the shoulder (01) and an actuated joint at the elbow (02) (Spong, 1995). The control action for this underactuated system involves applying continuous torque the free joint to swing the arm to the upright position and stabilize it. The task is minimize the deviation between the joint angle (01) and the target upright position (01 T), while maintaining zero angular velocity when the arm is upright. The reward function is defined as follows:
- (00 - x)² + (02)²
C.5. Obstacle2D
The goal is to navigate an 2D agent to the goal (3, 0) while avoiding a wall spanning y = [, starting from (0,0). 0). The action range is [-3,3], 3). which makes sufficient for the agent to avoid the wall in one step. The reward function based on progress toward the goal, measured as the difference in distance before and after taking a step. For special cases, receives +500 for reaching the goal, -200 for hitting the wall.
C.6. Hopper
Hopper is from OpenAl Gym based on mujoco engine, which aims to hop forward by applying action torques on the joints.
D. Details on Experiments
D.I. Training without entropy in target Q values
In Sec. 4 and Sec. 6, we simultaneously train Q networks with (soft) and without (plain) the entropy term in the target Q values, in order to illustrate the effect of entropy on policy optimization. Specifically,
Q'Quan (Sc, a+) r(st, a+) + ses (Q (s111, log iT (a [s++)]]]
T Qpaainn (Se, (4) = r(81, ae) + BB (Qpain (++11]]
with the rest of the SAC algorithm unchanged. We still update the policy based the target Q with entropy, i.e. Quaa ( at) as original SAC and training Qplain (Bt, ae) is just for better understanding for entropy's role in the policy updating dynamics.
D.2. Experiment Hyperparameters
The hyperparameters for training the algorithms are in Table 1 and Table 2.
D.3. Performance of DDPG and SAC with Auto-tuned Entropy Coefficient
We also run SAC with auto-tuned a and DDPG across six environments, as shown in Fig. 11. The first row includes environments where SAC fails due critical control requirements, while the second row shows cases where SAC performs better. Notably, auto-tuning the entropy temperature in SAC improves performance in some critical environments but not all, and it still fails to surpass PPO.
D.4. Benefits of Misleading Landscapes in SAC
Nonetheless, entropy in target is beneficial as designed because of necessary exploration. In Gym Hopper, investigate the state shown in Fig. 12. Entropy smooths the Q landscape in regions that may not produce optimal actions at the current training stage, encouraging exploration and enabling the policy to achieve robustness rather than clinging solely to the current optima.
D.5. PPO Trapped by Advantage Zero-Level Sets.
Without extra entropy to encourage exploration, PPO as an on-policy RL algorithm can be trapped in the zero-level set of advantages. In Obstacle2D (Fig. 13 first row), we plot the policy of the initial state, where the optimal action is to move to either the upper or lower corner of the wall, avoiding it in one step. The reward is designed to encourage the agent to approach the goal while penalizing collisions with large negative reward. The region front of the wall is higher-reward
17

---

When Maximum Entropy Mislcads Policy Optimization
Table I. Hyperparametens for SAC((AC--aut--appna) PPO, and DDPG
Hyperparameter
SAC/SAC-auto-a
PPO
DDPG
Discount factor (7)
0.99
0.99
0.99
Entropy coefficient (a)
0.2/ N/A
0
/
Exploration noise
0
0.1
Target smoothing coefficient (T)
0.005
/
0.005
Batch size
256
64
256
Replay buffer size
IM
2048
1M
Hidden layers
2
Hidden units per layer
256(64 for Toy Example)
256(64 for Toy Example)
256
Activation function
ReLU
Tanh
ReLU
Optimizer
Adam
Adam
Adam
Number updates per environment step
10
Clipping parameter (E)
0.2
/
GAE parameter (X)
0.95
Table 2. Learning Rates for SAC and PPO Across Different Enviromments
Algorithm
Learning Rate
Vehicle
Quadrotor
Opencat
Acrobot
Obstacle2D
Hopper
Actor
1e-3
3e-4
le-3
1e-3
1e-3
le-3
SAC
Q-function
le-3
3e-4
le-3
le-3
le-3
le-3
SAC auto-o
a
le-3
3e-4
le-3
le-3
le-3
le-3
Actor
3e-4
3e-4
le-4
3e-4
3e-4
3e-4
PPO
Value function
3e-4
3e-4
le-4
3e-4
3e-4
3e-4
Actor
le-3
3e-4
3e-4
le-3
1e-3
1e-3
DDPG
Q-function
le-3
3e-4
3c-4
le-3
le-3
le-3
Vehicle
Quadrotor
OpenCat
11000
Benward
2000
3000
1500
1000
-7500
500
-5000
500k
1.M
1.5M
2M
3000
500k
1.5M
0
1.5M
Step
Stop
Step
Acrobot
Obstacle2D
Hopper
600
-200
500
-400
400
5000
Rewerd
-600
200
300
2500
2000
1200
-1000
1000
11200
1.0
500
200k
400k
600k
000K
3M
300K
1MM
1.3M
2M
Step
Stop
PPO
SAC
SAC auC-auto-alpha
DDPG
Figure 11. Performance of All Algorithms across six environments
area because of the instant approaching reward. The advantage landscape reveals that PPO's policy moves to the center of the positive advantage region but remains confined by the zero-level set. Notably, although the optimal regions (upper and lower corners) have positive advantages, PPO remains trapped due to its local behavior. The coupling of exploration and actual policy worsens this issue- if PPO fails to explore actions to bypass the wall, its policy's a shrinks, further reducing
18

---

When Maximum Entropy Mislcads Policy Optimization
sooft (s, a)
Q plain (s, a
1.0
T d
6.
000
Torque #2
6.
TT SAC
5.0
π
02
+030
63)
35
1000
-32
-01
obn
EM
Torque #1
Torque #1
Current state
TTSCC
To
π 1z
Figure 12. Q landscapes in Hopper. Upper:
We set torque #0 (top torso) as the current 148 SAC to plot Qua and Qptan for torque #1 (middle thigh) and #2 (bottom leg) in the state shown in the bottom-left figure. Lower: Rendered hopper's gestures result from the corresponding policies. SAC's policy benefits from entropy by 'leaning further forward', taking a risky move despite this action being suboptimal in the current true Investigating the peaks on and og Qpliin reveals that the hopper tends to 'bend its knee' and 'jump up' when following the corresponding policies, demonstrating less exploration.
exploration and leading to entrapment. We can also observe this across training stages, as shown in Fig. 14.
A similar phenomenon is observed in Acrobot (Fig. 13 second row), where PPO's policy shrinks prematurely, leading to insufficient exploration.
However, this phenomenon can also be viewed strength of PPO, as it builds on the current optimal policy and makes incremental improvements step by step, thus not misled by suboptimal actions introduced by entropy. As result, PPO performs better in environments where the feasible action regions small and narrow in the action space, such in Vehicle, Quadrotor, and OpenCat, which closely resemble real-world control settings.
E. Details on SAC-AdaEnt
E.1. Pseudocode
We provide the detailed algorithm in Algorithm 1.
E.2. SAC-AdaEnt improves performance environments that SAC fails
To further validate the misleading entropy claim and enhance SAC's performance critical environments, we propose SAC with Adaptive Entropy (SAC-AdaEnt) and test it on Vehicle and Quadrotor environments, showing improvements in Fig. 15. In these environments, SAC relies excessively on entropy it dominates the soft Q values. To address this, SAC-AdaEnt adaptively combines target Q values with and without entropy. Specifically, we simultaneously train Qsoft and plain in Appendix D. During policy updates, we sample multiple actions per state under the current policy and evaluate their Quoit and Qplain values. By comparing these values, we compute the similarity of the two landscapes. If Q deviates significantly from Qplain indicating that entropy could mislead the policy, we rely on Qplain as the target Q value instead. Otherwise, entropy is retained to encourage exploration. This adaptive approach ensures a balance between safe exploration and exploitation, promoting exploration in less critical states and prioritizing exploitation states where errors could result failure. Note that SAC-AdaEnt is fundamentally different from SAC with an auto-tuned entropy coefficient,
19

---

When Maximum Entropy Mislcads Policy Optimization
Environment
Soft Q values
Advantage values
54.1
73.7
Obstacle2D
53.3
204.00
015.00
200
atuuntage
-80
U2 30
Acrobot
SAC
PPO
Figure 13. Q/Advantage landscapes of SAC and PPO in Obstacle2D and Acrobot. Upper: In Obstacle2D with start (0, 0). goal (3, 0). and wall 2 spanning y = 2- 2]. SAC succeeds in bypassing the wall whereas PPO fails. We plot the Q/Advantage landscape of the initial state. For SAC, entropy encourages exploration, guiding updates toward the upper and lower ends of the wall via soft Q. In contrast, PPO remains trapped despite the presence of positive advantage regions near the wall's ends. Lower: Acrobor, both algorithms achieve swing-up, but near the stabilization height, SAC applics the right torque to neutralize momentum, preventing it from falling again. In contrast, PPO remains stuck in local optimum, leading repeated failures.
(n)
(b)
(c)
0.03
204.33
(4)
Figure 14. Advantage landscapes in Obstacle2D for PPO. (a) to (f) show the advantage landscapes at different training stages for the initial state, where PPO's policy center remains trapped front of the wall while a gradually shrinks.
20

---

When Maximum Entropy Mislcads Policy Optimization
Algorithm 1 SAC with Adaptive Entropy (SAC-AdaEnt)
Initialize: Actor network тø. Q networks and their paired target networks for Q-target with/without entropy
Φ1, Φ2 0targ.1: 0targ.2 (w/ entropy), & $2 Hearg, Ptarg.2 (w/ entropy), replay buffer D, similarity threshold for each training step do
Sample action de (T) and observe St41,Ty
Store (st, de, Yt, $1+- 1) in replay buffer D
for each gradient update step do
Sample minibatch of transitions (s, (,,, from 'D
Compute target value:
y v min (o. - (olgooo(())))) y =r= 1-1,2
(min 7 Roc 1m1.2
(s',)''))
Update Q networks:
N
1 N a) y)² 0, - - no Vol N
Φ f 0:
no
VoL N
1
Σ (o, a) -1
For each 8, sample actions using current policy A {axla, x0{-18}} Compute similarity score:
sim(Q. Q') (s) Q'(s) [I where Q(s)
min Qo (s)
Q'(s)
min Q0(
1m1,2 commo(alss)
111.2
aammaola 8)
Update actor policy using reparameterization trick:
(alogro(ass) Qo, (s, if sismm, (Q, Q')
- Th VoBoN
a molass)(ass) eos (8, otherwise
Update target networks:
Qo. - -Q0. + ()-r) Qs TQ;; + +(1 (1-7)
end for end for
which applies uniform entropy adjustment across all states. In contrast, SAC-AdaEnt adaptively adjusts entropy for each state, making it particularly effective in environments requiring precise control and careful exploration.
E.3. SAC-AdaEnt preserves performance in environments that SAC succeeds
Not only SAC-AdaEnt improves performance in environments where SAC struggles, but it also retains SAC's strengths in those where SAC already excels. We report SAC-AdaEnt's results on Hopper, Obstacle2D, and Acrobot as in Table. 3:
Algorithm
Hopper
Obstacle2D
Acrobot
SAC
3484.46 + 323.87 87.87
501.98 + 0.62
-45.25 + 7.94
SAC-AdaEnt
3285.17 + 958. 43
501.50 + 0.57
-36.31 + 16.42
Table 3. Performance (mean std) of SAC and SAC-AdaEnt across tasks.
21

---

When Maximum Entropy Mislcads Policy Optimization
Vehicle
-500
-1000
-1500
Reward
-200
2200
-3000
-3500
-4000
500k
1M
1.5M
2M
Quadrotor
-500
Reward
-1000
-1500
2000
-200
500K
1.5M
2M
SAC
SAC-AdaEnt
Figure 15. Performance of SAC-AdaEnt SAC. Left: Reward Improvement. Middle: Full trajectory rendering. Right: Behavior of policy on critic states. In Vehicle, SAC-AdaEnt successfully steers and brakes to bring the vehicle back on track, while in Quadrotor, it effectively lifts the quadrotor follow the designated path.
F. Additional Experiments on Other MaxEnt algorithm
Although SAC is a powerful MaxEnt algorithm, to ensure our findings generalize beyond SAC's particular implementation of entropy regularization, we also evaluate Soft Q-Learning (SQL), an alternative MaxEnt method. SQL extends traditional Q-learning by incorporating an entropy bonus into its Bellman backup, resulting in policy that maximizes both expected return and action therrpyy fitting within the maximum-entropy RL framework. It can extend continuous actions by parameterizing both the soft Q-function and policy with neural networks and using the reparameterization trick for efficient, regularreee updates. We compare the performance of all algorithms Vehicle, Quadrotor and Hopper. The results in Table. 4 show that SQL also suffers from the entropy-misleane misleading issue, but its AdaEnt variant effectively mitigates this weakness.
Algorithm
Vehicle
Quadrotor
Hopper
SAC (a = 0.2)
-2003.85 = 867.82
-47.229 = 244.96
3484.46 + 323.87
SAC (auto-a)
-1551.96 = 636.88
-666.62 = 233.19
2572.00 901.35
SAC-AdaEnt
-1250.45 = 725.40
-247.58 45.15
3285.17 958.43
SQL
-2715.48 455.00
-6082.51
1632.35
2998.21
158.19
SQL-AdaEnt
-2077.59 = 266688
-4499.35 + 863.73
3115.94 + 25.19
Table 4, Performance (mean + std) across Vehicle, Quadrotor, and Hopper tasks for various algorithms.
22


Reviews:
[{"review_id": "xVludWczI0", "reviewer": "Reviewer_rvMR", "summary": "This paper posits that the entropy maximization objective in SAC can lead to failure in tasks that require precise, low-entropy policies, essentially \"misleading\" policy optimization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and questions. We will reorganize the writing to make the core claims clear from the beginning of the paper. The toy example seems to have obscured the core results, which we further explain here.  \n\n**Q: Large $\\alpha$ value?**\n\n> - In fact, the entire argument of the paper relies on a large enough value of $\\alpha$. If $\\alpha$ was not large, then there would not be any misleading effect at all.\n>\n> - The analysis in Appendix A is meaningless if the authors assume because they assume a fixed and large value of alpha.\n>\n> - SAC already solves by having a tunable entropy coefficient. The analysis with a fixed high value does not offer any useful insights.\n\n**A:**\n\nWe do not assume large fixed $\\alpha$ values in the core results.  \n\nWe agree with the reviewer that under a large $\\alpha$ value it seems obvious that the learning problem is changed. This is why in Section 3 we derive the general result: the misleading effect can be shown with *arbitrarily small* positive $\\alpha$. All theorems in Section 5 treat $\\alpha$ as a variable in the definitions and proofs to construct the bifurcation extension for misleading effects. \n\nTo see how: for an arbitrary $\\alpha$, the optimal MaxEnt policy follows $\\pi^*(a|s)=\\exp(\\alpha^{-1}Q(s,a))/Z(s))$ with normalizing factor $Z(s)=\\int \\exp(\\alpha^{-1}Q(s,a))da$ (see [Haarnoja, 2018b] Eq (4)). The Q terms are thus canceled out in the state values:\n\\begin{align*}\nV(s)&=\\mathbb{E}_{a\\sim\\pi^*}[Q(s,a)-\\alpha \\log(\\pi^*(\\cdot|s))]\\\\\\\\\n&=\\mathbb{E}_a[\\cancel{Q(s,a)}-\\alpha\\cdot\\bigg(\\cancel{\\log(\\exp}(\\alpha^{-1}\\cancel{Q(s,a)}))-\\log Z\\bigg)] \\\\\\\\\n&=\\mathbb{E}_a[\\alpha \\log Z]\\\\\\\\\n&=\\alpha \\log(Z)\n\\end{align*}\nConsequently, the misleading effect preserves as long as the value ordering between good and bad states incorporates the scaling effect of $\\alpha$. Moreover, with our definition of the bifurcation extension, if MaxEnt is misled for any small $\\alpha$, then with any $\\alpha'\\geq \\alpha$ the MaxEnt is misled as well. Thus the results apply to auto-tuning by choosing $\\alpha$ to be smaller than the final weight after the decay schedule in SAC with autotuning. \n\nSpecifically, the toy example is designed to allow the use of any positive $\\alpha$. For instance, say $\\alpha=0.00001$, then we would multiply with $\\alpha^{-1}$ in the exponents on Line 638 and 642 in Appendix A.1.1. Thus, to maintain the misleading effect we can scale all rewards by $\\alpha$ to cancel out $\\alpha^{-1}$, which preserves the same integrals and the ordering of $\\log Z(s_g)<\\log Z(s_b)$. Thus, in the example $Q(s_0,a\\in A_1)=\\gamma V(s_g)=\\gamma\\alpha \\log(Z(s_g))<\\gamma\\alpha \\log(Z(s_b))=Q(s_0,a\\in A_2)$. Namely, the MaxEnt-optimal policy still prefers the wrong action range $a\\in A_2$. Note that this scaling is why in theoretical analysis of MaxEnt the temperature $\\alpha$ is typically omitted (original SAC paper [Haarnoja, 2018a], remark after Eq(1)).\n\nAll experiments in Section 6 are run with SAC with small $\\alpha$ and autotuning (Line 923 Table 2).\n\nWe will update the paper to emphasize these points. Thank you for pointing out the concern.\n\n**Q: Complex Dynamics?**\n\n> - When a more realistic dynamics model for the quadrotor is used... — the term more realistic is vague and hand-wavy.\n> - The selected benchmarks seem to be arbitrarily selected and complex dynamics is used without any justification of what is complex.\n\n**A:** \n\nThe difference in the dynamics of quadcopters between easy and complex is described in Appendix C.2 and we will further add the full equations. We agree with the reviewer that it is hard to quantify complex dynamics, so we should refer to them as environments that require low-entropy control policies to succeed, which can be observed more directly by plotting the advantage landscapes as in Section 6.1.\n\nThe benchmarks are selected for visualizing concretely *when* entropy affects policy updates, as shown in Section 6. We also showed that the misleading effect can also benefit learning (Section 6.2). Our goal is to understand when MaxEnt helps and when it hinders policy optimization.\n\n\n**Q: No performance issues in SAC?**\n\n> - The works cited in this paper do not give conclusive evidence of MaxEnt algorithms always struggling in important control problems.\n> - Tan \\& Karakose (2023) are cited to claim that SAC delivers suboptimal solutions in comparison to PPO in complex control problems, which is not shown in those papers.\n\n**A:**\n\nWe do not aim to show that MaxEnt algorithm *always* struggles, but to understand when the entropy terms *may* hinder learning.\n\nWe definitely agree that SAC is widely observed to outperform policy gradient, which makes it important to understand in what cases they may underperform. The cited practical robotics papers are examples of such cases. In particular, Tan \\& Karakose (2023) shows in Figure 6 that SAC is outperformed by other methods. We can remove the reference if it is not a convincing example.", "claims_and_evidence": "No. There are several problematic claims.\n\n1. The central claim about the misleading effect relies on $\\alpha = 1$, whereas it is well-known that the entropy coefficient needs to be tuned according to the environment.\n\nL627 says \"without loss of generality, we use $\\alpha = 1$ for the entropy coefficient\", but this assumption does break generality. Naturally, with a high value of $\\alpha$, SAC is not expected to work, so this \"theoretical\" calculation for the toy example does not do anything meaningful.\n\nCrucially, for the special case of $\\alpha = 0$, the MaxEnt effect of SAC would turn off, and the optimal policy would not suffer from the misleading effect. So, trivially, one cannot assume a value of $\\alpha$ without losing generality.\n\nIn fact, the entire argument of the paper relies on a large enough value of $\\alpha$. If $\\alpha$ was not large, then there would not be any misleading effect at all.\n\n2. \"However, MaxEnt methods have also been shown to struggle with performance-critical control problems in practice,\nwhere non-MaxEnt algorithms can successfully\nlearn\" is a very vague claim in the abstract, and the works cited in this paper do not give a conclusive evidence of MaxEnt algorithms always struggling in important control problems. As a matter of fact, SAC is indeed applied to many useful real-world control problems.\n\n3. \"When a more realistic dynamics model for the quadrotor is used, then SAC always fails, while PPO can succeed under the same initialization and dynamics.\" — the term more realistic is vague and hand-wavy.\n\n4. \"In both the Vehicle and Quadrotor environments, the policy learned by SAC-AdaEnt mostly corrects the behavior of the SAC policy, as illustrated in their overall trajectories and the critical shown in the plots.\" is not a valid claim given the negligible improvement shown with AdaEnt in Figure 9.", "methods_and_evaluation_criteria": "- Method: No new method is proposed and without understanding the role of $\\alpha$, the SAC and PPO methods tested are not enough to understand the claims behind the role of entropy.\n- The selected benchmarks seem to be arbitrarily selected and \"complex dynamics\" is used without any justification of what is complex.", "theoretical_claims": "The analysis in Appendix A is meaningless if the authors assume $\\alpha=1$ because they assume a fixed and large value of $\\alpha$. Naturally, if one changes the RL objective from reward optimization too much, then the policy optimization would fail at convergence.", "experimental_designs_or_analyses": "- The experiments only analyze role of SAC at a fixed $\\alpha$, which does not really represent the importance or drawbacks of the entropy term in SAC, because the entropy coefficient plays a key part.\n- The comparison between SAC and PPO is not exactly fair because one is off-policy and the other is on-policy.", "relation_to_broader_scientific_literature": "Many claims are not justified well, and papers like Tan & Karakose (2023) are cited to claim that SAC delivers suboptimal solutions in comparison to PPO in complex control problems, which is not shown in those papers.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The central claim that high entropy regularization leads to poor policy optimization is trivial, which SAC already solves by having a tunable entropy coefficient $\\alpha$. The analysis with a fixed high value of $\\alpha$ in this paper does not offer any useful insights.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PuSPQuEB9U", "reviewer": "Reviewer_gm3Q", "summary": "The authors analyze the trade-off between exploration/robustness and exploitation in Maximum Entropy Reinforcement Learning though a variety of control tasks. The paper demonstrate that in performance-critical control tasks requiring precise, low-entropy actions, Maximum Entropy approaches can become misguided by their entropy-driven exploration, which help readers to understand how entropy maximization affects policy optimization.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and questions. We agree that more discussion and analysis can be done for SAC-AdaEnt, and we provide some more details below. Indeed, we formulated SAC-MaxEnt less as an algorithm that is intended to replace SAC, but more as an ablation mechanism that tests the validity of the theoretical analysis in practical learning environments. We believe the principled understanding of the misleading effect of MaxEnt algorithms (and how it can both positively and negatively affect policy learning) opens up the possibility of developing more customized policy optimization procedures for challenging environments that exploit their specific structures and value landscapes. \n\n**Q: (SAC-AdaEnt)** \n\n> Will SAC-AdaEnt decrease performance in tasks where SAC generally succeeds?\n\n**A:**\n\nFollowing the question, we performed additional experiments on the environments in the paper where SAC generally succeeds: Hopper, Obstacle2D, Acrobot (also attached results for Vehicle and Quadrotor, in which SAC fails). The summary of performance comparison is shown in the table below, and we will add the plots and more discussion in the paper. \n\nWe observe that SAC-AdaEnt led to less mean return compared SAC as well as higher variance. Note that our design of SAC-AdaEnt maintains the entropy-based exploration component in SAC (only changing the policy optimization objective). Thus the overall performance still benefits from exploration, and the higher variance is the result of the advantage landscape being less dominated by entropy. The obstacle2D and acrobot environments are lower-dimensional, and thus exploration itself is likely sufficient for discovering successful trajectories, and thus the performance has not degraded in SAC-AdaEnt. \n\n\n| **Algorithm**       | **Hopper**          | **Obstacle2D**      | **Acrobot**       | **Vehicle**        | **Quadrotor**      |\n|---------------------|--------------------|--------------------|---------------------|---------------------|---------------------|\n| SAC   | 3484.46 ± 323.87    | 501.98 ± 0.62       | -45.25 ± 7.94       | -2003.85 ± 867.82  | -475.29 ± 244.96   |\n| SAC-AdaEnt          | 3285.17 ± 958.43     | 501.50 ± 0.57       | -36.31 ± 16.42      | -1250.45 ± 725.40  | -247.58 ± 45.15   |\n\n\nOverall, we believe the experiments show that in environments where SAC performs well, SAC-AdaEnt can still rely on entropy to support exploration, while the performance may benefit less from the tendency of the standard SAC in escaping action regions with raw high Q-values. \n\n**Q: (High-dimensional RL)**\n> Were entropy-traps also expected or observed in high-dimensional image-based RL tasks?\n\n**A:** \n\nEntropy-traps are fundamentally dependent on the reward structure and the environment dynamics in the MDP, in terms of whether the success of learning relies on low-entropy policies. Thus the misleading effects can definitely be observed in high-dimensional RL problems. We believe the key to observing entropy traps may not always be about the dimensionality of the state space, but more crucially determined by the dimensionality of the action space -- when the action space is high-dimensional, it is more likely that the feasible policies form a low-entropy distribution over the space, thus making it easier to observe the misleading effect of MaxEnt. We believe such problems can arise frequently in high-dimensional RL problems in general, such as fine-tuning of language models, foundation Vision-Language-Action models, as well as diffusion policies.", "claims_and_evidence": "Yes, main claims made in the submission are generally supported by clear evidence.", "methods_and_evaluation_criteria": "The analysis of the misleading effect of maximum entropy RL is conducted on different robotics control environments with 5 random seeds. Results are compared to PPO to highlight performance gaps and benefits of entropy maximization at different situations. Authors also proposed SAC with Adative Entropy Scaling, SAC-AdaEnt to further show the negative effective of soft Q values in some situations.", "theoretical_claims": "Theoretical part looks ok to me.", "experimental_designs_or_analyses": "The authors demonstrate empirically with a toy example and experiments on complex control tasks to show explicit situations where maximizing entropy leads to convergence towards suboptimal, high-entropy solutions, which I found is very clear and sound. \nIssues: the authors also propose a possible solution, SAC-AdaEnt. SAC-AdaEnt is then compared with standard SAC, however, limited discussion/analysis is on SAC-AdaEnt, for example, will SAC-AdaEnt decreases performance in tasks where SAC generally succeed (e.g. Hopper)?", "relation_to_broader_scientific_literature": "The paper is connected to important literature areas of maximum entropy RL and exploration-exploitation trade-off.\nBy clearly illustrating situations where maximizing entropy can degrade performance, this paper significantly extends the understanding of limitations inherent in widely used maximum entropy methods such as SAC.\nThis paper also explicitly compares non-maximum entropy algorithm PPO against SAC, highlighting trade-offs between entropy-driven exploration and precise exploitation on practical tasks.\nWith these discussions, I think the paper can help people improve current maximum-entropy algorithms int he future.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "See above sections", "other_comments_or_suggestions": "See above sections", "questions_for_authors": "Were entropy-traps also expected or observed in high-dimensional image-based RL tasks?\nWill SAC-AdaEnt decreases performance in tasks where SAC generally succeed (e.g. Hopper, or other Mujoco envs)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nexgJXNk5j", "reviewer": "Reviewer_Rq6M", "summary": "The paper examines the trade-off in MaxEnt RL: while the entropy term fosters exploration and robustness, it can mislead policy optimization in tasks requiring precise, low-entropy actions, often causing MaxEnt methods like SAC to converge to suboptimal policies.\n\nThe authors formalize this phenomenon with a toy example and introduce the Entropy Bifurcation Extension, showing that auxiliary states can create misleading soft Q-value landscapes that divert the MaxEnt-optimal policy from the true optimal one.\n\nTo address this, the paper proposes SAC-AdaEnt, an adaptive tuning mechanism that monitors discrepancies between soft and plain Q-values, dynamically shifting policy updates to rely more on plain Q-values when needed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and questions. We agree with the reviewer that comparisons with wider range of baselines will enhance the empirical analysis, and we will expand the paper accordingly. Our current formulation of SAC-AdaEnt is mainly intended for empirically testing the theoretical analysis, to show how the misleading effect practically affects policy optimization (in both negative and positive ways). As our theoretical analysis shows, the effect is rooted in the use of the MaxEnt objective, and thus we believe the principles are applicable to MaxEnt algorithms in general. We discuss additional experiments with Soft Q-Learning below as an example that reflects this. \n\n**Q: (PPO with entropy)** \n\n> (Entropy in PPO) is treated as a secondary regularizer rather than a core objective. This distinction highlights why PPO may avoid some of the pitfalls of overemphasizing entropy.\n\n**A:**\n\nThank you for pointing this out: it exactly aligns with our understanding. Our core construction of entropic bifurcation relies on the fact that the objective of policy optimization in MaxEnt algorithms is to match the Boltzmann distribution, which fundamentally changes the goal of the optimal policy and enables the misleading effect of entropy. In policy gradient methods using entropy as exploration strategies, the objective for policy optimization is unaffected, and thus entropy regularization does not lead to misleading effects as how we formulated in the paper. \n\n**Q: (SAC-auto-alpha v.s. SAC-AdaEnt)**\n\n> In Fig.11, the SAC-auto-alpha refers to automated adjustment? Could you compare the SAC-AdaEnt to the SAC-automated-alpha-tuning?\n\n**A:** \n\nYes, in Fig.11, SAC-auto-alpha refers to automated entropy adjustment. We show more results in the next question, and will update the plots in the paper along with AdaEnt applied to other MaxEnt baselines, as you suggested. Note that SAC-auto-alpha applies a uniform entropy adjustment across all states while SAC-AdaEnt adapts the entropy individually per state.\n\n**Q: (Baselines)** \n> The choice of baselines is narrow; It would enhance the paper if extending the adaptive mechanism to other MaxEnt algorithms.\"\n\n**A:** \n\nWe will follow the reviewer's suggestion and expand the experiment section with more baseline MaxEnt algorithms. Here we provide results from additional experiments comparing with Soft Q-learning (SQL) and with adaptive entropy (SQL-AdaEnt). We observe that the overall performance of SQLs generally underperforms compared to SAC, and SQL is affected by the misleading effect in the Vehicle and Quadrotor environments as well, in which case SQL-AdaEnt also improves the performance. \n\nIn the table below, we summarize the mean and variance of the policy return across Vehicle, Quadrotor (as examples of negative misleading effect) and Hopper (positive misleading effect). We see that SAC generally performs better than SQL across all environments, and the effect of AdaEnt is similar on SQL and SAC. That is, in Vehicle and Quadrotor, where SAC struggled, the AdaEnt component led to better performance. The improved performance of SQL-AdaEnt on Hopper may be due to SQL being a relatively weak baseline. \n\n| **Algorithm**| **Vehicle** | **Quadrotor** | **Hopper** |\n|-|-|-|-|\n| SAC ($\\alpha=0.2$)  | -2003.85 ± 867.82  | -475.29 ± 244.96   | 3484.46 ± 323.87    |\n| SAC auto-$\\alpha$   | -1551.96 ± 636.88  | -666.62 ± 233.19   | 2572.00 ± 901.35    |\n| SAC-AdaEnt| -1250.45 ± 725.40  | -247.58 ± 45.15    | 3285.17 ± 958.43    |\n| SQL| -2715.48 ± 453.00  | -6082.51 ± 1632.35 | 2998.21 ± 158.19    |\n| SQL-AdaEnt | -2077.59 ± 266.84  | -4499.35 ± 863.73  | 3115.94 ± 25.19     |\n\nThe results align with the predictions of our theoretical analysis, which showed that the misleading effect arises from the MaxEnt objective itself. As long as an algorithm aims to reduce divergence from the Boltzmann policy, the performance can be negatively affected as analyzed in the paper.\n\n**Q: (Scalability)** \n> While the adaptive mechanism is promising, measuring discrepancies requires a “global” understanding at each state. In high-dimensional environments, ... computationally expensive.\n\n**A:**\n\nYes, we formulated SAC-AdaEnt to test the theory in practice, acknowledging that it is not intended as a scalable replacement for SAC. As we shown in Section 6.2, the misleading effects of entropy may explain some of the key benefits of SAC, and thus we do not always need to eliminate the effect. We believe a promising direction in challenging environments is to use baseline SAC and PPO algorithms to understand the gap between Q-plain and Q-soft landscapes first, and then modulate the use of entropy on specific regions in the state space to minimize the need for global probing. Consequently, we focused mainly on analyzing the principles of how entropy affects policy optimization, in the hope that the understanding leads to more nuanced policy optimization strategies in challenging environments.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes,", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "I examined the experimental designs and analyses in the paper, particularly those illustrated in Figures 4–11, which involve continuous control tasks such as Vehicle control, Quadrotor control, and standard benchmarks like Acrobot, Obstacle2D, and Hopper.\n\nThe paper compares SAC (with automated entropy adjustment) against PPO and the proposed SAC-AdaEnt. This setup is generally appropriate to highlight the effect of entropy on policy optimization. However, the choice of baselines is somewhat narrow. Including additional variants or state-of-the-art Max-Ent algorithms could have provided a more comprehensive comparison.", "relation_to_broader_scientific_literature": "The paper builds on and extends several streams in the RL literature - MaxEnt RL, PPO, exploration versus exploitation.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "While the adaptive mechanism (SAC-AdaEnt) is promising, the paper itself notes that measuring discrepancies between soft and plain Q-value landscapes requires a “global” understanding at each state. In high-dimensional environments, this might become computationally expensive or infeasible.", "other_comments_or_suggestions": "See Questions.", "questions_for_authors": "1. The paper does not specifically emphasize that PPO also uses an entropy term. While PPO does incorporate an entropy bonus to encourage exploration, this is treated as a secondary regularizer rather than a core objective. In contrast, the MaxEnt framework—exemplified by SAC—integrates entropy maximization directly into the learning objective, which can lead to the misleading effects discussed in the paper. This distinction is critical, as it highlights why PPO may avoid some of the pitfalls of overemphasizing entropy, thereby achieving more precise control in performance-critical tasks.\n2. In Figure 11， the SAC-auto-alpha refers to the popular automated entropy adjustment trick or the AdaEnt mechanism proposed in this paper? Could you compare the SAC-AdaEnt to the SAC-automated-alpha-tuning?\n3. The paper introduces an adaptive entropy tuning mechanism, SAC-AdaEnt, which effectively mitigates the misleading effects of high entropy in the SAC framework. However, it is noteworthy that the proposed tuning mechanism is applied exclusively to SAC. It would enhance the paper if the authors could discuss the potential for extending this adaptive mechanism to other MaxEnt-based algorithms. Such a discussion could broaden the applicability of their approach and provide insights into whether similar benefits could be realized across a wider range of RL methods.\n\n\n\n---\n\n**Updated Review:** Thanks for your response — I think the problem you’re tackling is important, and the added experiments are helpful. I remain positive about the paper. Just a suggestion: it would be stronger if the main comparison focused on SAC with and without the entropy term. I know you included this in Figure 6, but putting it earlier might help clarify the main point. Since SAC and PPO differ in many ways, it’s hard to tell if the gap in Figure 1 is really due to entropy.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2l9hrUZujs", "reviewer": "Reviewer_xbLf", "summary": "The paper examines failure cases of Maximum Entropy (MaxEnt) RL, showing how entropy maximization can mislead policy optimization. It introduces the Entropy Bifurcation Extension, a theoretical construct proving that MaxEnt RL can drive policies toward suboptimal actions (Theorem 5.5, Proposition 5.6). Empirical results demonstrate SAC failures in vehicle control, quadruped locomotion, and drone tracking due to misleading entropy landscapes. The authors explore adaptive entropy tuning as a mitigation strategy but acknowledge its limitations.\n\n## update after rebuttal\nI appreciate the authors’ detailed response and additional experimental results. The clarification about the theoretical setting and the empirical motivation for SAC-AdaEnt helped address my main concerns. The new comparisons with Soft Q-Learning and discussions of performance across more environments (e.g., Hopper, Vehicle, Quadrotor) improved the empirical grounding of the claims.\n\nI still believe that broader algorithmic comparisons and more extensive ablations (e.g., on entropy scaling) would further strengthen the paper, but I find the core contributions valuable. I maintain my score, and I look forward to seeing the suggested revisions incorporated into the final version.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and questions. Indeed, we focused on understanding the principles behind potential limitations of MaxEnt (SAC being its best performing and most widely-used version), complementing most existing results on the benefits of entropy regularization. Although we phrase the effect as \"misleading\", we showed that it can be either beneficial or harmful in different environments (Section 6.2 discusses its benefits). Consequently, the formulation of the algorithm SAC-AdaEnt is intended more as an ablation study mechanism to empirically test the validity of our theoretical analysis in practice, rather than as a new candidate of MaxEnt algorithms. At the same time, we definitely agree with the reviewer that evaluating the algorithm on more baselines and hyperparameter changes can strengthen the empirical understanding and contribution of the paper. Here we respond to the questions directly, and will extend our paper along these lines accordingly. \n\n**Q: (More MaxEnt baselines)** \n\n> How does the failure mode extend to other entropy-based RL methods? A more detailed comparison between SAC and other entropy-based RL methods (e.g., MPO, Soft Q-Learning) would provide a broader picture.\n\n**A:** \n\nWe will follow your suggestion and expand the experiment section with more baseline MaxEnt algorithms. Here we provide results from additional experiments comparing with Soft Q-learning (SQL) and with adaptive entropy (SQL-AdaEnt). We observe that the overall performance of SQLs generally underperforms compared to SAC, and SQL is affected by the misleading effect in the Vehicle and Quadrotor environments as well, in which case SQL-AdaEnt also improves the performance.\n\nIn the table below, we summarize the mean and variance of the policy return in the environments of Vehicle, Quadrotor (as examples of negative misleading effect) and Hopper (positive misleading effect). We see that SAC generally performs better than SQL across all environments, and the effect of AdaEnt tends to be similar on SQL and SAC. That is, in Vehicle and Quadrotor, where SQL and SAC struggled, the AdaEnt component led to better performance. The improved performance of SQL-AdaEnt on Hopper may be due to SQL being a relatively weak baseline.\n\n| **Algorithm**| **Vehicle** | **Quadrotor** | **Hopper** |\n|-|-|-|-|\n| SQL| -2715.48 ± 453.00  | -6082.51 ± 1632.35 | 2998.21 ± 158.19    |\n| SQL-AdaEnt | -2077.59 ± 266.84  | -4499.35 ± 863.73  | 3115.94 ± 25.19     |\n| SAC ($\\alpha=0.2$)  | -2003.85 ± 867.82  | -475.29 ± 244.96   | 3484.46 ± 323.87    |\n| SAC auto-$\\alpha$   | -1551.96 ± 636.88  | -666.62 ± 233.19   | 2572.00 ± 901.35    |\n| SAC-AdaEnt| -1250.45 ± 725.40  | -247.58 ± 45.15    | 3285.17 ± 958.43    |\n\nThe results align with the predictions of our theoretical analysis, which showed that the misleading effect arises from the maximum entropy objective itself. As long as an algorithm aims to optimize policies to reduce divergence from the Boltzmann policy which is optimal under the MaxEnt objective, its performance can be negatively affected in situations that we analyzed in the paper.\n\n**Q: (More comparisons and ablation study)**\n \n> Clarifying how adaptive entropy tuning compares to alternative exploration techniques (e.g., intrinsic motivation, Bayesian exploration) would add value. How sensitive is the failure mode to the choice of reward scaling and environment structure? Could benefit from conducting ablation studies on entropy-related hyperparameters, such as temperature scaling, to analyze their role in failure cases.\n\n**A:** \n\nThese are very important suggestions that we will follow to strengthen the empirical analysis in the paper. In general, our experience is that policy optimization methods that encourage more exploration generally underperform in control environments where low-entropy policies are the key to success, compared to conservative and more greedy approaches such as PPO. On the other hand, similar to our discussion in Section 6.2, in many environments the \"misleading\" effect of MaxEnt and enhanced exploration strategies can be crucial to performance. Based on the theoretical analysis framework that we proposed in this paper, we believe it is promising to develop fine-grained analysis methods that take into account of both the specific structure of an environment to inform the hyperparameter choices and exploration strategies. \n\n**Q: (Related Work)**\n\n> The paper does not cite Ahmed et al., \"Understanding the Impact of Entropy on Policy Optimization\" (ICML 2019)\n\n**A:** \n\nThe paper was cited in the Related Work section on Line 90 as (Ahmed et al. 2019). We will add more discussion in the paper for this and similar analysis on the benefits of entropy regularization.", "claims_and_evidence": "1. The claim that entropy can mislead policy optimization is well-supported by both theoretical analysis and empirical results.\n2. Theoretical constructs such as the Entropy Bifurcation Extension and its proofs are rigorously presented and demonstrate the misalignment between MaxEnt RL and true optimal policies.\n3. The empirical experiments clearly show cases where SAC fails due to entropy distortion.", "methods_and_evaluation_criteria": "1. The evaluation includes diverse continuous control tasks, which are relevant benchmarks for MaxEnt RL.\n2. The paper does not propose a new RL algorithm but instead provides an analysis of existing methods, which may limit its impact as a technical contribution.", "theoretical_claims": "1. The theoretical analysis correctly identifies failure modes of MaxEnt RL and proves that entropy can mislead policies into suboptimal behaviors.\n2. The Entropy Bifurcation Extension (Theorem 5.5, Proposition 5.6) is a novel and valid mathematical construct that highlights the downsides of entropy maximization.\n3. The theory does not offer a general solution to these failures but instead highlights when and why they occur.", "experimental_designs_or_analyses": "Advantages\n1. The experiments effectively demonstrate failure cases of MaxEnt RL in vehicle control, quadruped locomotion, and drone tracking tasks.\n2. The empirical results support the theory, showing how SAC fails due to misleading entropy landscapes.\n\nLimitations\n1. The study focuses on SAC but does not evaluate whether other entropy-based methods (e.g., MPO, Soft Q-Learning) experience similar failures.\n2. Could benefit from conducting ablation studies on entropy-related hyperparameters, such as temperature scaling, to analyze their role in failure cases.", "relation_to_broader_scientific_literature": "The paper contributes to the understanding of when entropy regularization fails in RL, contrasting with prior work that highlights its benefits (e.g., Ahmed et al.,2019)​. By providing a complementary perspective, it challenges the assumption that entropy always improves policy optimization and explores conditions where it can instead mislead learning.", "essential_references_not_discussed": "The paper does not cite Ahmed et al., \"Understanding the Impact of Entropy on Policy Optimization\" (ICML 2019), which analyzes how entropy smooths optimization landscapes and improves convergence. Since this submission focuses on failure cases of entropy regularization, discussing Ahmed et al. would provide a more balanced perspective on when entropy is beneficial versus when it may mislead policy optimization.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper provides new theoretical insights into the failure cases of MaxEnt RL.\n2. The experiments are well-designed and effectively demonstrate the theoretical claims.\n3. The Entropy Bifurcation Extension is a novel construct for analyzing RL policies.\n\nMinor Issue:\n1. The paper does not propose a new method, making its contribution primarily analytical rather than algorithmic.", "other_comments_or_suggestions": "1. Clarifying how adaptive entropy tuning compares to alternative exploration techniques (e.g., intrinsic motivation, Bayesian exploration) would add value.\n2. A more detailed comparison between SAC and other entropy-based RL methods (e.g., MPO, Soft Q-Learning) would provide a broader picture.", "questions_for_authors": "1. How does the failure mode extend to other entropy-based RL methods?\n2. How sensitive is the failure mode to the choice of reward scaling and environment structure?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper addresses a highly relevant challenge in reinforcement learning: improving our understanding of entropy regularization in relation to exploration, robustness, and the optimization landscape. They also recognize the proposed "Entropic Bifurcation Extension" as a novel and valuable contribution. The correctness of the proofs was also checked, including the appendix to a large extent.

After the rebuttal and discussion phases, there was broad consensus among the reviewers (despite one differing view) that the main claims of the paper are supported by clear and convincing evidence.

Three reviewers raised a concern that I share. It can be summarized as an apparent misalignment between the stated goals of the paper and its actual focus. Specifically, the authors place strong emphasis on comparing SAC versus PPO from the beginning, which raises the following questions: (1) why not center the analysis more directly on the presence or absence of entropy regularization, regardless of the specific algorithmic choice? and (2) why exclude other algorithms, e.g., MPO or Soft Q-Learning as examples of regularized methods, or  e.g., DDPG, as an unregularized one?

The rebuttal helped clarify misunderstandings related to this issue, as reflected in the updated reviews.
Overall, this is a well-written paper, and its contribution remains significant and valuable. I recommend acceptance.
I strongly encourage the authors to incorporate the following improvements in the revised version:

- Give greater emphasis to the comparison between SAC with and without the entropy term, rather than presenting it as a subsection of the experiments.
- Include the comparison with Soft Q-Learning and the additional discussions of performance across more environments (e.g., Hopper, Vehicle, Quadrotor) that were provided during the rebuttal phase.
- Re-check all references in the introduction. For example, Tan & Karakose (2023) is not appropriate. Also, two of the three references cited after the sentence "Often the performance by SAC is indeed shown to be inferior to PPO despite efforts in tuning" do not support the claim being made.
Decision: Accept (poster)