State Entropy Regularization for Robust Reinforcement Learning
Authors: Yonatan Ashlag
Uri Koren
Mirco Mutti
Esther Derman
Pierre-Luc Bacon
Shie Mannor
Venue: NeurIPS 2025 oral ()
Keywords: Robust Reinforcement Learning
Risk-Averse Reinforcement Learning
Regularized Reinforcement Learning
Primary area: reinforcement_learning

Abstract: State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its theoretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation.

State Entropy Regularization for Robust Reinforcement Learning
Yonatan Ashlag" Technion
Uri Koren"
Mirco Mutti
Technion
Technion
Esther Derman MILA Institute
Pierre-Luc Bacon
Shie Mannor Technion, NVIDIA Research
MILA Institute
Abstract
State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its theoretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy more sensitive the number of rollouts used for policy evaluation.
Introduction
Despite the impressive success of RL across various synthetic domains 29 37 51 6,, challenging issues still need to be addressed before RI. methods can be deployed in the real world. Some of these challenges involve imperfect models, noisy observations, and limited data. In such settings, policies trained on nominal environments may perform poorly when faced with deviations from the assumed dynamics or reward structure at deployment [34]. This issue has led growing interest in robust RL, which aims to ensure reliable performance despite model misspecification or uncertainty [38] 46
59 31 47 Over the years, the well-established connection between regularization and robustness in machine learning has been extended to the RL setting [1] 21 [1] where regularization has been shown to induce robustness to adversarial perturbations
reward function [12 5 and
transition kernel
[0] This duality has particularly been studied for policy entropy regularization
13
proposed to regularize the standard RL objective with the entropy of the distribution over states induced by a policy (18), either alone in combination with policy entropy. This has empirically shown better exploration and consequently, improved sample efficiency [50]. However, our knowledge, formal studies on the effect of state entropy regularization and its connection with robustness do not exist in the literature yet. Thus, an open question is whether robustness is by by-product of state entropy regularization just like it for policy entropy, and it is, what type of robustness results from state entropy regularization?
Recent works 23
50
65 25
Equual contribution.
Correspondence
to:
yonatan. ashl aahlag@@mail.com
uri www.keeanessss...coooooaoaaoa... technion. i1
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Nominal Environment
Local Perturbation
Standard
Policy Repularization
Reguaaization
Figure I: State entropy regularization leads to diverse behavior and improved robustness. We plot 50 trajectories from standard RL. policy--nttooy-rrgulaaaee RL and state-entropy-reguaarzze RL on manipulation task (left). Policies are trained the nominal environment and evaluated in perturbed version with red obstacle along the optimal trajectory. The centrpyyrrgullarice agent often bypasses the obstacle and solves the perturbed task.
Intuitively, policy entropy encourages stochasticity in action selection but typically spreads random- ness along single dominant trajectory. This makes it effective in smoothing out small or uniform perturbations- but also fragile when that trajectory disrupted. For instance, a large obstacle blocking single high-reward path can cause policy entropy- regularzeed agents to fail catastroph- ically, as illustrated in Fig. T contrast, state entropy incentivizes broader coverage of the state space, potentially distributing visitation across multiple high-reward paths. If policy entropy can be understood methodical way of injecting noise along an optimal path, state entropy can be viewed as encouraging uniformity across optimal, or nearly optimal, trajectories. This distinction motivates our investigation into whether state entropy regularization provably ensures stronger robustness, particularly in settings with spatially localized perturbations.
The contributions provided in the paper are summarized follows.
Section 3 provides a theoretical analysis of the power of state entropy regularization for robust RL: In Section 3.1] prove that state entropy regularization exactly solves reward-robust RL problem, then characterize the induced uncertainty set and adversarial reward. Our theoretical result confirms the informal intuition described above: policy entropy protects against locally informed adversary, while state entropy is robust to globally informed perturbations. We further show that the regularization strength monotonically controls the conservativeness of the induced uncertainty set, interpolating between & -like uncertainty in the www-regularization limit and likke uncertainty regularization increases.
In Section 3.2 we show that state entropy regularization provides a non-trivial lower bound on performance under transition (kernel) uncertainty. We compare this to the combined use of state and policy entropy, and prove that adding policy entropy weakens the bound bound-highlighting highlighting
structural advantage of using state entropy alone in this setting.
In Section 4 we study both theoretical and practical limitations of entropy regularization:
We establish that entropy cnnllaization-whetheee over the policy, the state distribution, or both coth-cannot solve any kernel robust RI. problem (Section 1) Building upon this result, we show that entropy regularization can hurt risk-averse performance arbitrarily (Section 4.2);
We show that the robustness benefits of state entropy regularization can be more sensitive the number rollouts used to evaluate the policy w.r.t. other forms regularization (Section 4.3).
In Section 5 we empirically evaluate the robustness properties of state entropy regularization across discrete and continuous control tasks. We show that it improves performance under spatially correlated perturbations specifically obstacle placement- while not degrading performance under smaller, more uniform perturbations. We also demonstrate how the robustness benefits of state entropy depend on the rollout budget, with diminished gains in low-sample regimes.
Related works are provided in Section 6 All proofs are provided in Appendix
Preliminaries
In this section, we describe the setting and notations will use in the remainder of the paper.

---

Notation. We denote by Az the set of probabiliiy distributions over finite set Z. The entropy of a distribution μ E Az given by Hz (z) == - EEezz lt(2) log((())) The log-sum-exp operator over 2 is defined as LSEz (w) := log (Ezez e)(s)) u E R For any u, E R their inner product is denoted by (u, 4)2 := Eesz ((2) We shall often omit the set subscript if there is no ambiguity. Finally, for distribution μ E Ag and function Z R, we denote the expectation f on μ as [x]f(z)] = 2 (z)..
Standard RL. Markov Decision Processes (MDPs) constitute the formal framework for RI. [54] MDP is a tuple (S, P.r,)). where S is a state space of size S, A an action space of size A, P: x A + ASS) transition kemel, S x A + I reward function, and 7 E [0, 1 discount factor. The MDP evolves as follows. First, an initial state is sampled as so ~ P(.) Then, for every & 2 0, the current state is observed, action d, ~ - (-]ss)) is performed according some policy π : S + AA. prompting a state transition 8t+1 ~ P(st, ax) while a reward ri = r(s at) is collected. The goal is to find policy maximizing specific objective function within a policy space II. As we shall see below, different objectives can be considered.
First, define expected transition kernel and reward under policy as Pd (.1s) Eaea [(als) a) and (s) == (()))rr) a), sometimes denoted as Pa, r in vec- tor form. Also define the discounted state distribution induced by policy on an MDP with kernel P as (s) = o PP( = s/T, and the state-action distribution, a.k.a. occu- pancy pp == (p(s))(aas). The standard RL objective function to maximize over II is [(T, P,r) Epp. [r(s, a)), which may be rewritten as I [TT, P.r) = (07, r) in vector form.
We note that, while the above formulation assumes finite MDPs, the main results extend to continuous state or action spaces under mild regularity assumptions. In that case, sums over states and actions are replaced by integrals with respect to the corresponding densities, and the discrete entropy is replaced by the differential entropy of the occupancy measure.
Robust RI. The robust RL formulation [59,31] 31 aims to tackle uncertainty in the MDP parameters, such that the kernel P and reward are not known but assumed lie in uncertainty set P R. The objective function is defined with the worst case on the uncertainty set
I (T,, P, R)
min (P,r)PxxR
P,r).
By construction, this so-called robust MDP formulation leads to policies that are less sensitive to model misspecification and perform reliably under a range of plausible environments. Throughout this paper, we will often consider reward uncertainty sets {P} x R of the form
R = (i x D(r, F) S eJ,
(I)
where 0, D is interpreted dissimilarity measure between reward functions defining the notion of proximity within the uncertainty set and 7 > r means that there exists some (s, a) E S x A such that F(s, a) S r(s, a) (to ensure the adversary can degrade performance rather than only improve it). Another perspective on robust RI. is to interpret the minimization in J(x, P, R. as the action of an adversary that perturbs the nominal MDP within the uncertainty set so to minimize the policy's expected return. Accordingly, e quantifies the adversary's s budget for allowable perturbations.
catropy-rreularizze RL. A broad range of literature considers augmenting the standard RL objective with some regularization function. The most popular is policy entropy regularization [17. which has been linked to stability of the policy optimization (2), improved exploration [45] and
robustness
The resulting objective is
Ja H.x) == Epp (r(s, + (T)ss(-)
where >> 0 is hyperparameter controlling the regularization strength. In recent works 23 50 65 25 4, regularization with the entropy of the state distribution has been considered, either standalone or in combination with policy entropy, yielding
[T, Hs) Epp (r(s,a)) + aHs(d))
Ja HSxA)) = Epp [r(s, a)) + aFsx)(PF))
In this work, we will focus on analyzing the latter types of regularization.
'Wit a slight overload of notation, we denote the initial state distribution with the same symbol of the transition kernel with void input.
3

---

5. The Regularizer Uncertainty Set
Robust Return
Worst Case Reward
Policy
Beg L (r) s log(4)
Epplij c (Eaz [[.]]]]]] log(A))
r(s,a) a log(sr( s))
State
LSES Ar $ log(S)
Epphi - c (Hs(e)) log(S))
r(s,a) a Amjalay
losidhial)
State-action
LSR5xA Ar) _ log(S4)
- (ogg())
r(s,a) a logíe" (s,
Table 1: Robustness guarantees for each regularization type including the form of the uncertainty
set, the robust return, and the corresponding adversarial reward. We denote Ar == r - 7 and rs = r(,,)) )) ns .(- | )) for brevity.
3 Robustness and State Entropy Regularization
In this section, we analyze the robustness properties of regularization with the state distribution entropy, First, we show that this regularization exactly solves certain reward-robust RL problems. and we compare its theoretical guarantees to those of policy entropy regularization. We then extend the analysis transition (kernel) uncertainty, deriving non-trivial performance bounds in this setting.
3.1
In this section, we show that state entropy regularization amounts to solving a robust RI. problem with an explicit uncertainty set over rewards.
Theorem 3.1. For any II, the following duality holds:
min p [i] = E [r] + ((Hs(dp)) - log(()) -,
(5)
FER
a S < + log(S)). Moreover, the where the reward uncertainty set R (c, {i LSEE(""-")) worst-case reward chosen by the adversary is given by f(s, r(s, log(d)(s)) =(a|s)
The proof is presented in Appx. A.I It follows a similar path 5 and relies on convex conju- gate argument: The entropy regularization term can be interpreted as introducing a soft constraint, corresponding to an uncertainty set over the reward function. The uncertainty set can be written as { r oLSEg -== a (S) S e]. highlighting dissimilarity constraint between reward
vectors as in Eq.
a=0.1
R((a) For Varying a
We compare the obtained uncertainty sets and robustness guarantees with policy entropy and state-action entropy regularization in Table [ The comparison extends previous formulations that neglect either the temperature parameter a or the adversary's 's budget [13,5]. More insight into the proofs of these derivations is included in Appx. A.2 As shown in the adversaria re- ward column, policy entropy regularization de- pends on the policy π (-|ss) only. Thus, it tackles adversaries with access to local information of each state. In contrast, state entropy regulariza- tion depends on the state occupancy distribution di., enabling robustness against adversaries that exploit global properties of the agent's ' behav- ior across the state space. Consequently, poli- cies regularized via state entropy are inherently more resistant to spatially-structure perturba- tions, while policy entropy provides protection primarily against local, state-specific variations. This difference analytically confirms our initial intuition.
a=1.0
a=10.0
2-
a=100.0
(rf E, rg
(e((a=
-2
Figure 2: Uncertainty sets R5(a) (a) induced by state entropy regularization in two-state MDP with r" (s1) = 1, r" (s2) O, and and y-axes correspond to the perturbed rewards pat (s1) and fr (s2), respectively. As increases, the uncertainty set expands, interpolating between l and f1-type constraints in the limits - 0 and a + oc.
We next analyze how the uncertainty set induced by state entropy regularization evolves as function of the temperature parameter While our formal analysis focuses on the state entropy, similar

---

limiting behaviors appear to hold for policy entropy and state-action entropy as well. This formalizes insights that were observed empirically in [3], where varying the regularization strength was found to interpolate between different robustness behaviors. We characterize the limiting uncertainty sets as -> oC and a -> O, corresponding to different adversarial capabilities. The proof is in Appx. AA.3 We visualize this behavior in Fig. 2
Theorem 3.2 (Limit uncertainty a). Consider the uncertainty sets associated with state entropy
regularization as a function of a:
( {c)
R₂, (a(a)==
ISSS
a
Then, the mapping a + R2 (a) is non-increasing in the sense that az 2 an implies R(E, a1) E 02). Furthermore, the limit sets satisfy
lim (a) = JaxoR (a) a0000
= =
SES (-) S
and
lim R₂ R, (a) { max((* -)) - se).
- sec.
a0
3.2 Lower Bound under Kernel Uncertainty
As a first step in our analysis of kernel robustmess, we show that state entropy regularization yields a non-trivial lower bound worst-case performance under transition uncertainty.
Theorem 3.3. For any policy и E IT and a > 0, the following weak duality holds:
min Epp [r] 2 exp [og(r)) + aHH(d)) (c)
a(H((())) - -log(S)
(6)
where
ns (=((aa) = { a
(s)ss
dp
Σ SES
SE
The proof follows similar structure in 13 where transition uncertainty is effectively reduced to reward uncertainty via importance sampling argument and logarithmic transformation. This reduction in turn enables to leverage our earlier result on reward-robustness to derive the lower bound. The full proof available in Appx. A.4
An appealing property of this formulation is that the uncertainty set is expressed explicitly in terms of the induced state distribution. This allows us reason about the distributional effects of transition noise directly, rather than working with abstract perturbations to the transition kernel itself.
Interestingly, using the same derivation, can show that adding policy entropy regularization with the same temperature a leads to the same uncertainty set, but with a strictly looser lower bound. A more formal discussion of this comparison, including the explicit form of the resulting bound under policy entropy regularization, is provided in Appx. A.4.I This result is particularly relevant in light of the next section, where we show that entropy regularization alone-regardless of the type-cannot solve kernel robust RL problems in general. In that sense, the bound derived here is the most we can expect from this approach. That said, we do not claim this lower bound is tight, and we cannot conclusively rule out that policy entropy may help in some settings. In fact, as we show in our experiments, often does. Nonetheless, this result provides insight into the structural advantages of state entropy regularization when viewed through the lens of kernel uncertainty.
4 Limitations of State Entropy Regularization
In this section, we study the limitations of entropy regularization in robust reinforcement learn- ing. First, we show that entropy insularization-whetee over the policy, state distribution, or both-cannot solve any kernel robust RL problems. Building on this result, we demonstrate that in the risk-averse setting, entropy regularization can significantly degrade performance. Finally, we take a practical perspective and show that the robustness benefits of state entropy regularization are highly sensitive to the number of rollouts used during policy evaluation.
5

---

4.1 Entropy Regularization Cannot Fully Solve Kernel Robustness
A kernel-robust RI. problem is defined by a function P(MDP, π) which maps the MDP and policy parameters to an uncertainty set-that is, the set of possible transition kernels from which the adversary may choose. We consider a class of kernel robust RL. problems in which the uncertainty set function depends only on the nominal transition kernel P and the policy и. This captures wide range of uncertainty models, including en-constrained and rectangular uncertainty sets commonly studied in robust RL [27 [10]. We now state more general claim: Any regularization function that depends only on the nominal kernel and the policy cannot, in general, characterize the robust return over such uncertainty set functions. The following impossibility result formalizes this observation.
Theorem 4.1. Let Q(x P) & be a regularization function. There does not exist an uncertainty set function P. (P, πt) such that for every MDP holds:
min E [r] = Boglr] + S(T, P). PEP
The proof, which can be found in Appx. A.5 when the reward is constant, any regularization must be degenerate. A key takeaway is that any regularization capable of capturing kernel robustness must depend on the reward. not just on the nominal kernel P the policy π. This observation consistent with prior results (IO]. As direct corollary, entropy regularization-whettee applied to the policy or the state distribution-caanee fully solve kernel robustness either.
is very simple and uses a symmetrization argument-
4.2 Entropy Regularization and Risk-Averse Performance
While entropy regularization provides non-trivial lower bounds under kernel uncertainty, have shown it cannot fully solve kernel robustness. Our experiments offer further insight into when regu- larization may improve robustness in practice. We now show analytically that for an important type of robustness-risk aversion-entrog, regularization can, in fact, arbitrarily degrade performance.
While robust RL primarily addresses epistemic uncertainty uncertainty about the parameters of the MDP risk-averse RL. [15] instead targets aleatoric uncertainty, which arises from the inherent randomness of the MDP itself. To quantify risk-averse performance, use the common Conditional Value at Risk (CV)R) objective [48 8), denoted CVaRg, which measures the expected return in the worst 3-fraction of outcomes. We denote the optimal policy for the unregularized objective by n² and the optimal policy for the entropy-rreulariiee objective (whether state or policy entropy) by Th
Theorem 4.2. For every temperature a: > 0, constant M E TR, and 0 < B < 1, there exists an MDP M such that:
CVaR
[G] CVaR th [G)] [G]= M
(8)
where G is the total discounted return.
The intuition behind the result is that entropy regularization does not prioritize low-variance trajecto- ries, which are critical for improving risk-averse performance. The proof is provided in Appx. A.6
4.3 On the Sample Sensitivity of State Entropy Regularization
We now investigate how the robustification benefits of state entropy regularization depend on the number of rollouts used to estimate a policy's ' state entropy. While using state entropy regularization does not incur significant computational overhead [50]. we observe that the number of rollouts required to realize the robustness benefits is higher than what is typically sufficient for standard RL. We provide both theoretical and empirical evidence for this phenomenon, and offer two complementary explanations. One explanation is statistical- state entropy is harder to estimate accurately than other quantities like return or policy entropy. The other is behavioral-hi in low-rollout regimes, entropy maximization can incentivize an agent to be less stochastic.
For the statistical explanation, we assume that the state entropy is estimated using k-nearest neighbor estimator [52) in a continuous state space RRD as common in the related literature 42 33 50 For fixed k under mild regularity conditions on the MDP- MDP-such such as the smoothness of d;. for all the

---

policies The estimation eтror of the state entropy decays at rate O(n O(n-1/2) where denotes the number of sampled trajectories [53]. In contrast, both the policy entropy and expected return can be estimated with error decaying at the standard rate On 1/2) [35.. As a result, in high-dimensional settings, state entropy estimation introduces a dominant source of statistical error, contributing to the increased sample sensitivity of antropy-regularee dimensionality D is often reduced by computing entropy over subset of state features (as in Section 5) or by estimating entropy over latent state representations, as demonstrated by [50].
methods. In practice, we note that the effective
For the behavioral explanation, we consider the MDP illustrated in Figure 3 and the empirical objective Ex- [r] + adf (T)) where (xx) is the state entropy estimated on a single rollout. The policy that maxi- mizes the empirical entropy on a single rollout will tend to be deterministic, taking the longest possi- ble route to the goal in order to visit more states. This leads to a counterproductive outcome: Rather than encouraging stochasticity across multiple near- optimal waats--roomtting robustness- the regular- ization yields a deterministic behavior along a subop- timal path, ultimately degrading robustness.
Figure An illustrative toy MDP. The grid is 2 x 6 environment with the initial state marked with "T" and the goal state with "G". The red path shows the longest deterministic trajectory that visits the maximum number of states before reaching the goal. Reward is given only at the goal state.
Experiments
We evaluate the robustness properties of state entropy regularization using a of representative envi-
ronments and perturbation types. These are designed to test three core hypotheses: (1) State entropy regularization offers protection against spatially correlated severe perturbations to the dynamics; (2) State entropy regularization induces robustness to f reward uncertainty; (3) The robustification of state entropy regularization is sensitive to the number of rollouts used during training. We validate these claims in both discrete (MiniGrid (7]) and continuous (Mujoco [57]) domains. We compare policies trained without regularization, with policy entropy regularization, and with state entropy regularization. Note that the state-entropy regularized policies are trained with some policy entropy regularization to favor training stability 2 For all the types, we select the largest regularization coefficients that degrade nominal performance less than 5%. To ensure fair comparison, the methods are trained with the same base algorithm- A2C (36] for MiniGrid and PPO for Mujoco.
To implement state entropy regularization, we largely follow (50) by using a k-nearest neighbor (k-NN) entropy estimator (52). As state entropy regularization is incorporated as an intrinsic reward,
it can be paired with any RL algorithm. Full implementation details are in Appx. B.I
Transition kernel robustness via entropy regularization. To evaluate how well each regularization method handles spatially correlated perturbations to the transition dynamics, we construct environ- ments where specifie regions of the state space blocked off or obstructed. These changes considered "catastroohic" because they remove critical paths or create bottlenecks that fundamentally alter the agent's 's ability reach the goal using its original policy. In MiniGrid, a large wall segment that blocks a connected area of the maze (see Figure (b) for the perturbed environment). This constitutes major spatially-structuree disturbance, between nearby states and introduces sharp structural change in the environment. As shown in Figure 4b), the agent trained with state entropy regularization significantly outperforms both policy entropy and unregularized baselines. We note that we report results for all horizon lengths to avoid misleading conclusions based on a single fixed task length. In cases where one method clearly dominates across the full horizon range, we can draw decisive conclusions. When the curves intersect or remain close, indicates that the relative performance is more sensitive to task structure. To verify that this effect extends to continuous control, we run analogous experiments in Mujoco's 's Pusher and Ant environments by placing obstacles along the optimal trajectories, fully blocking them. Again, as shown in Figs (1a). G[c) state entropy regularization yields substantially more robust performance as its induced behavioral diversity enables bypassing these local catastrophes.
randomly place
it affects transitions
3 For simplicity. here we consider the i.i.d. case in which entropy estimate computed with a single data point coming from each trajectory, sampled with a geometric distribution with parameter y A discussion of the interplay between state entropy and policy entropy regularization is in Appx. B.5
7

---

(b)
(d)
Nomiinal Performance
Success
Rate
0.
0.6
103
156
20
150
300
5.0
155
150
260
unrogularized
policy entropy
Stato enir
trained empty Figure 4: MiniGrid evaluation under different perturbations. All agents
grid where the agent starts at the top left and must reach the goal at the bottom right (a). Top: Visualizations the three perturbation types used of the maze, (c) obstacles are distributed uniformly, and (d) the goal location is randomized. Bottom: Nominal return (left) and success rate under perturbations (right). We evaluate each policy for 200 episodes and report 96% confidence bounds across 25 seeds.
time-(b) large walls randomly block parts
To highlight the challenge of learning policies that are robust to spatially localized perturbations, we compare against the PRMDP and NRMDP frameworks [6], two popular robustness baselines in continuous control. Both methods formulate robustness as two-player zero-sum game: the agent selects actions while an adversary perturbs the actions. shown Figure ([aa), the agent trained with state entropy regularization significantly outperforms this baseline. While we do not expect state entropy to consistently outperform robust RL methods across all types of perturbations, this result-alone with our analysis in the main paper-suggesis that it is particularly effective against spatially correlated perturbations, which are not the primary focus most standard approaches.
To test robustness under more uniformly distributed, small-scale perturbations, we augment MiniGrid with uniformly scattered obstacle tiles. These mild disturbances are spread across the entire state space, simulating the kind of noise that policy entropy regularization is expected to help with [33] In Figure we see that the unregularized agent performs poorly, while both policy and state entropy regularization improve robustness, with policy entropy showing slight advantage. This result aligns with our initial intuition: Policy entropy particularly effective against diffuse perturbations, and importantly, observe that adding state entropy regularization does not hurt this effect.
Reward robustness. To assess how different entropy regularization methods handle reward uncertainty, we randomize the reward location at evaluation time in both MiniGrid and Pusher. In MiniGrid, new goal location is sampled uniformly from the entire grid. In Pusher, the goal is relocated to a random position at distance from its nominal location. This setup induces structured variation in the reward function, formally resembling an é bouuudde uncertainty- where the total change in reward mass is limited but can be concentrated arbitrarily across the state space. The key intuition that if the reward happens to lie along the agent's typical trajectory, then policy entropy regularization- by injecting stochasticity at the action level-may suffice to recover it. However, when the reward is placed less frequently visited areas, broader state coverage becomes necessary. In such cases, state entropy regularization offers natural advantage by encouraging the agent to visit a wider range of states. As shown in Figures dd) ().5 5 (b) policy entropy improves performance relative to the unregularized baseline, but adding state entropy yields more substantial gain, particularly when the reward far from the nominal goal location.
Sensitivity to rollout budget. We now examine how the number of rollouts used per state entropy estimation affects robustness benefits state entropy regularization. As discussed in Section the estimation of state entropy is statistically more demanding than that of policy entropy or expected return. In particular, with limited rollouts, entropy maximization can encourage overly deterministic behavior, especially in sparse reward settings. To study this, vary the number of

---

Returted Dytanes
Sowó 3/Pt
Per Esropy Stivariin Adlattn
Varying Pry toaiiinnn Strepths
14
1/1
FRAC7
HHMP
ronelo euiverect
ronina annareas
5.3
6.15
pataroed anrrovas
number of diriburspe enioy extraatnn
stait entrayy regubatzation deeffoer.
(d) Figure 5: Pusher experiments and ablations. Like the discrete experiment, we train all agents on nominal task, then introduce a localized catastrophe by blocking off the optimal path, visualized in Fig. D All plots report success rate defined when the pucks final distance from the goal is smaller than 0.1. We evaluate each policy for 200 episodes and show 96% confidence bounds across 25 seeds.
(b)
(c)
rollouts used to evaluate the policy and monitor how this affects downstream robustness. As shown in Sigure/jjb), we find that when only small number of rollouts are used, state entropy regularization can actually lead to worse performance than policy entropy. This is supported by the fact that with rollouts, the confidence bounds of state entropy intersect the mean robust performance of policy entropy. Interestingly, while robustness is sensitive to the number of rollouts, nominal performance remains largely unchanged- making it difficult to detect this gap without explicitly evaluating under perturbations. These observations suggests that while state entropy can provide significant robustness benefits, realizing those benefits requires a sufficient amount of rollouts untiissorndn a practical tradeoff between regularization and sample complexity.
Regularization temperature. To study how the temperature parameter influences robustness, we frain state-entropy regularized agents with increasing regularization strength and evaluate their performance in both the nominal and perturbed versions of the Pusher environment. We observe in Figure ([c) that as regularization increases, robust performance improves, illustrating the role of temperature in expanding the effective uncertainty set and promoting diverse robust behavior. Notably, robust performance eventually platcaus, suggesting diminishing returns from further increasing the regularization strength. This empirical trend aligns with our theoretical result in Theorem 3.2 which characterizes the limiting behavior of the uncertainty set. These findings highlight that while tuning entropy regularization can enhance robustness, its benefits saturate beyond a certain point.
Nominal Environment
Perturbed Dynamics
1.0
800
Average Return
0.6
400
Average
200
0.2
Unregulartaed
Poticy entrapy
Stare entroyy
Unregularied
Policy entropy
Stare entropy
(a)
(b)
(c)
Figure 6: Ant experiment. (a) Visualization of the vertical wall perturbation. (b) All methods perform similarly on the nominal task. (c) Success rate in the perturbed environment, i.c., the fraction of evaluation runs in which the ant successfully crosses the obstacle. We evaluate each policy for 200 episodes and show 96% confidence bounds across 25 seeds.
Related Works
Robust RL and regularization. The connection between regularization and robustness is well- established in RL. The work [13] demonstrates an equivalence between policy-entropy regularized objectives and reward robustness and a lower bound for kernel robustness. We extend their analysis state entropy regularization, providing more direct and general proofs, which also accounts for the regularization coefficient (for both policy and state entropy). State-action entropy regularization is

---

analyzed by (2), which as we show in Appx. A.4.1 equals the sum of state entropy plus policy entropy. Note that action entropy encodes the joint information provided by state entropy and policy entropy, but does not encode the conditional information each one gives to the other. Neither of these works provide algorithmic methods implement state-action entropy regularization, nor studies the kernel robustness induced by entropy regularization. Complementary to ours, [TO] derives a regularizer that depends on the value function since they study the primal optimization of RL and robust RL problems. However, connection between regularized dual and primal objectives has yet to be established. The motivation in [TO] [4] also different from ours, as they study tractable robust RL methods, whereas we analyze robustness granted by state entropy regularization.
State entropy objectives. Hazan et al. [88 have first proposed learning objective predicated on maximizing the entropy of the state distribution in MDPs, followed by a considerable stream of works studying various aspects of the problem, including practical algorithms for challenging domains [42]
33 55) 64 [3)], alternative formulations [43] 69 16
40
65 1. statistical complexity [56]] how to
deal with partial observability [66 67]. and many others 22 23
55 63
62
39
24
30
4
2 688 Most of the works focus on the entropy of the state distribution although the latter is
often paired with standard policy entropy regularization in practical methods [33 50
64
32 While
the state entropy has been often considered a standalone objective for pure exploration [e.g., 55 data collection [63,, or policy pre-training (most of the others), some works have employed occupancy entropy regularization for RL [23 50, 65 25 A common challenge for all of the previous is that the feedback on the state entropy is not directly available to the agent and must be estimated from data, leading to scalability issues. A crucial advancement has been made by incorporating
non-parametric entropy estimators 52 in practical algorithms [42 33 50 64
7 Conclusion
This work provides a comprehensive analysis of state entropy regularization through the lens of robustness. We show that it exactly solves certain reward-robust RL problems and induces stronger worst-case guarantees than policy entropy in structured perturbation settings. In the case of kernel un- certainty, we establish non-trivial performance lower bound and highlight that adding policy entropy weakens this guarantee. At the same time, we prove fundamental limitations: entropy regularization cannot solve general kernel-robust problems and may degrade performance in risk- scenarios. On the practical side, we show that while state entropy improves robustness to changes, it does not harm performance under smaller uniform perturbations--tooteee is sensitive to the number of policy evaluation rollouts. Together, our findings clarify when and how state entropy regularization can contribute to robust RL, and where its boundaries lie.
its effectiveness
Acknowledgment
The work of Yonatan Ashlag is supported by the Israel Science Foundation (ISF), grant No. 447/20. The work of Mirco Mutti is funded by the European Union (ERC, grant agreement No. 101041250).
References
[1] Siddhant Agarwal, Ishan Durugkar, Peter Stone, and Amy Zhang. f-policy gradients: A general framework for goal-conditioned RL using f-divergences. In Advances in Neural Information Processing Systems, 2023.
[2] Zafarali Ahmed, Nicolas Le Roux. Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning. 2019.
[3] Marcus Ang, Jie Sun, and Qiang Yao. On the dual representation of coherent risk measures. Annals of Operations Research, 262(1):29-46, February 2017.
[4] Adrien Bolland, Gaspard I Lambrechts, and Damien Ernst. Off-policy maximum entropy rl with future state and action visitation measures. arXiv preprint 2rX6v:2412.06655, 2024.
10

---

[5] Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Gregoire Detetang, Markus Kunesch, Shane Legg, and Pedro A Ortega. Your policy regularizer is secretly an adversary. Transactions on Machine Learning Research, 2022. Expert Certification.
[6] Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H. Chi. Top-k ofo-po correction for reinforce recommender system. In ACM International Conference on Web Search and Data Mining, 2019.
[7] Maxime Chevalier-Boisverr, Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for openai gym. https: // github. com/ Farama- F Fwwndotooooooo ion/ Minigrid, 2018.
[8] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. Advances in neural information processing systems, 2015.
[9] Riccardo De Santi, Marin Vlastelica, Ya-Ping Hsich, Zebang Shen, Niao He, and Andreas Krause. Provable maximum entropy manifold exploration via diffusion models. In ICLR 2025 Workshop Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025.
[10] Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDPs and the equivalence between robustness and regularization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[11] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46(3):946- 969, 2021.
[12] Benjamin Eysenbach and Sergey Levine. If maxent rl is the answer, what is the question? In International Conference on Learning Representations, 2020.
[13] Benjamin Eysenbach and Sergey Levine. Maximum entropy RL. (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022.
[14] Uri Gadot, Esther Derman, Navdeep Kumar, Maxence Mohamed Elfatihi, Kfir Levy, and Shie Mannor. Solving non-rectangular reward-robust mdps via frequency regularization. AAAI Conference on Artificial Intelligence, 2024.
[15] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[16] Zhaohan Daniel Guo, Mohammad Gheshlagi Azar, Alaa Saade, Shantanu Thakoor, Bilal Piot, Bernardo Avila Pires, Michal Valko, Thomas Mesnard, Tor Lattimore, and Rémi Munos. Geometric entropic exploration. arXiv preprint wrXiv:2/01.0005: 2021.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 2018
[18] Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, 2018.
[19] Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. arXiv preprint arXiv: 1w...02789..999 2019.
[20] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, (274):1-18, 2022.
[21] Hisham Husain. Distributional robustness with ipms and links to regularization and gans. Advances Neural Information Processing Systems, 2020.
[22] Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference Artificial Intelligence and Statistics, 2021.

---

[23] Riashat Islam, Zafarali Ahmed, and Doina Precup. Marginalized state distribution entropy regularization policy optimization. arXiv preprint urXiv:1912.05//2 2019.
[24] Arnav Kumar Jain, Lucas Lehnert, Irina Rish, and Glen Berseth. Maximum state entropy exploration using predecessor and successor representations. In Advances in Neural Information Processing Systems, 2023.
[25] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. In Advances in Neural Information Processing Systems, 2023.
[26] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. Advances in Neural Information Processing Systems, 36, 2024.
[27] Navdeep Kumar, Adarsh Gupta, Maxence Mohamed Elfatihi, Giorgia Ramponi, Kfir Yehuda Levy, and Shie Mannor. Dual formulation for rettrecta ip robust markov decision processes, 2025.
[28] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine. and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 1rXiv:1906.05274, 2019.
[29] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(1):1334-177) 2016.
[30] Hongming Li, Shujian Yu, Bin Liu, and Jose C Principe. Element: Episodic and lifelong exploration via maximum entropy. arXiv preprint arXiv:2412.08
2024.
[31] Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust markov decision processes, Advances in neural information processing systems, 26, 2013.
[32] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, 2021.
[33] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances Neural Information Processing Systems, 2021.
[34] Shic Mannor, Duncan Simester, Peng Sun, and John N Tsitsiklis. Bias and variance approxima- tion in value function estimates. Management Science, 53(2):308-322, 2007.
[35] Alberto Maria Metelli, Mirco Mutti, and Marcello Restelli. A tale of sampling and estimation in discounted reinforcement learning. In International Conference Artificial Intelligence and Statistics, 2023.
[36] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lilli- crap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, 2016.
[37] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human- control through deep reinforcement learning. Nature, 518(7540)55995333 February 2015.
[38] Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335- 359, 2005.
[39] Mirco Mutti. Unsupervised reinforcement learning via state entropy maximization. PhD Thesis, Università di Bologna, 2023.
[40] Mirco Mutti, Riccardo Santi, and Marcello Restelli. The importance of non-Markovianity in maximum state entropy exploration. In International Conference Machine Learning, 2022.
12

---

[41] Mirco Mutti, Mattia Mancassola, and Marcello Restelli. Unsupervised reinforcement learning multiple enviromments. In AAAI Conference on Artificial Intelligence, 2022.
[42] Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gra- dient of non-parametric state entropy estimate. In AAAI Conference on Artificial Intelligence, 2021.
arrinsicallyymotivateedd [43] Mirco Mutti and Marcello Restelli. An intrinsically motivated approach for learning highly exploring and fast mixing policies. In AAAI Conference on Artificial Intelligence, 2020.
[44] Alexander Nedergaard and Matthew Cook. k-means maximum entropy exploration. arXiv preprint arXiv:2205. 15623, 2022.
[45] Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv: 2017.
[46] Takayuki Osogami. Robustness and risk-sensitivity in markov decision processes. Advances in neural information processing systems, 25, 2012.
[47] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, 2017.
[48] Tyrrell Rockafellar. Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
[49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.066 2017.
[50] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State en- tropy maximization with random encoders for efficient exploration. International Conference on Machine Learning, 2021.
[51] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, January 2016.
[52] Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowiez, and Eugene Demchuk. Nearest neighbor estimates of entropy. American Journal of Mathematical and Management Sciences, 23(3-4):301-32) 2003.
[53] Shashank Singh and Barnabás Póczos. Analysis of k-nearest neighbor distances with application entropy estimation, 2016.
[54] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018
[55] Jean Tarbouriech, Shubhanshu Shekhar, Matteo Pirotta, Mohammad Ghavamzadeh, and Alessan- dro Lazaric. Active model estimation in Markov decision processes. In Conference Uncer- tainty in Artificial Intelligence, 2020.
[56] Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Pierre Perrault. Yunhao Tang, Michal Valko, and Pierre Menard. Fast rates for maximum entropy exploration. In International Conference on Machine Learning, 2023.
[57] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
[58] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the deviation of the empirical distribution. Hewleut Neeleu--Pckarre Labs, Tech. Rep, page 125, 2003,
13

---

[59] Wolfram Wiesemann. Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research, 38(1):153-183, 2013.
[60] Andreas Winter. Tight uniform continuity bounds for quantum entropies: conditional entropy, relative entropy distance and energy constraints. Communications in Mathematical Physics, 347(1):291-313, 2016.
[61] Huan Xu, Constantine Caramanis, and Shie Mannor. Robustness and regularization of support vector machines. Journal of Machine Learning Research, 10(7), 2009.
[62] Qisong Yang and Matthijs TJ Spaan. CEM: Constrained entropy maximization for task-agnostic safe exploration. In AAAI Conference on Artificial Intelligence, 2023.
[63] Denis Yarats, David Brandfonbrener, Hao Liu. Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201. 0tti::2001.334445 13425, 2022.
[64] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, 2021.
[65] Mingqi Yuan, Man-On Pun, and Dong Wang. Rényi state entropy maximization for exploration acceleration in reinforcement learning. IEEE transactions on artificial intelligence, 4(5):1154- 1 1164, 2022.
[66] Riccardo Zamboni. Duilio Cirino, Marcello Restelli, and Mirco Mutti. How to explore with belief: State entropy maximization in pomdps. In International Conference on Machine Learning, 2024.
[67] Riccardo Zamboni, Duilio Cirino, Marcello Restelli, and Mirco Mutti The limits of pure exploration in pomdps: When the observation entropy is enough. Reinforcement Learning Journal, 2024.
[68] Riccardo Zamboni, Mirco Mutti, and Marcello Restelli. Towards unsupervised multi-agent reinforcement learning via task-agnostic exploration. arXiv preprint arXiv:2502.08365, 2025.
[69] Chuheng Zhang, Yuanying Cai, Longbo Huang, and Jian Li. Exploration by maximizing Rényi entropy for reward-free rl framework. In AAAI Conference Artificial Intelligence, 2021.
[70] Ev Zisselman, Itai Lavie, Daniel Soudry, and Aviv Tamar. Explore generalize zero-shot In Advances in Neural Information Processing Systems, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See sections
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

See appendix A for all proofs. All theorems appearing in the main paper include proof sketches.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See section 5 and appendix
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We will share a link to a github with the code used to run the experiments in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cmm ec/
public/guidess https//ooooeSuuassssooooooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp wwwppbblccgguiicuceccccccooooope wetgggddddeeoeeebbiniinn htt.SSbbbiiiiooooiiiiiooooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See section 5 and appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
All plots include error bars indicating the confidence bounds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See appendix B.6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httppp////aarrro.cccc//////////a
Answer: [Yes]
Justification:
We do not identify any potential deviations from the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: In this work we conduct foundational research with no direct path to a particular application.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release any data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See appendix sections: 2,3
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: will release link to a github with the code to recreate the experiments in the supplementary material. The github will include documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsoureing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
21

---

A Proofs
A.1 Proof of Theorem 3.1
Proof. Since this proof does not involve adversarial perturbations to the transition kernel, we omit the subseript and write d" instead of d' for clarity. We will first show a result for the uncertainty
R(c, TT) = (i LSEs(r* )*)) f c), and then get the result from our main theorem using change of variables.
We fix л. First, we reformulate the optimization problem in terms of Ar =
min E [7] Ep- [r] + min -Ep- [Ar] FER
Areil
Note that R(e, x) is strietly feasible and that the objective and constraints are convex, therefore the optimal point satisfies the KKT conditions. Furthermore, the objective is linear in i, thus the minimum is achieved at the edge of the constraint with À > 0. The Lagrangian is:
L(Ar, A) (p", + X(LSE(Ar)) -e) + Aa s (issottaa(AA"))GG -0
(10)
V Var(s,)) L =
(11)
-12 + Asoftmax (Ar*) =0
(12)
Where we get that last line by taking the sum over E A of both sides. By taking the sum over E S we can note that the last line implies that X (softmax and d are probability vectors). Therefore, original optimization objective from becomes:
min (p",, Ar
"p",p'") Ar) + A.SEs(Ar)) - -c min (d Ar") + +SSEs(Ar))
(13)
Art
Finally, noting that entropy is the convex conjugate LSE we get:
min Ep
[F] = pm [r] min (d ArT) LSEs(Ar")] Eg [r] Hs(dd) - (14)
FER
We also note that substituting Ar(s, a) = log(d"(()) (a)) *(als) + for every constant satisfies equation 11 and
thus the adversarial reward. From the constraint LSE(Ar*) we can deduce that c
Finally, we can now substitute r + a + & + log(S) and get the required result for different temperatures.
A.2 Proof of results in Table
Proof. We proved the result for state entropy in the previous appendix. For both policy entropy and state-action entropy we'll start from equation Note that for both we can imply the KKT conditions since is strictly feasible and the objective and constraints are convex. For policy entropy the gradient of the Lagrangian becomes:
Var(an) = -s" a) + Ad
(s)softmax(Ar:) ( =
(15)
By summing over (s, a) we can again infer that X = 1. We continue by substituting in the identity (s, a) d' (s)=(a s) and get x(a and taking the inner product with p² we get:
softmax(Ar.(aa). By taking the log of both sides
Log(Ta) s))) = Σ dd (s)
=
Σ T(a s) log(=(a $))) Eạ- [HAA(T( s))]
(16)
= - + (T.)) (p" Ar) + E. LSE
(17)
Which is the required result for the case where a = We use the same change of variables as A. to get the result for general temperature. We can verify with through explicit substitution that the worst case reward described the table satisfies 15 and the requirement that EE,,- (Ar.)) = c.
The proof for state-action entropy regularization very similar and is thus omitted.
22

---

A.3 Proof of Theorem 3.2
Proof. First, by taking the derivative of the divergence according alpha we can see that it is always negative, which implies the required containment relation:
0 da
a log
1 S
Σ A.
))
log
S
Σ
& A
a
E
Art
(18)
EE-ulle 4.-
Art
log
sousuftmax And
(19)
log Bssule _0
log soootttn Art
Ar
(20)
0
(21)
The first inequality follows from Jensen, the second inequality follows from propricties of softmax (gives higher weight to higher Ar" values) and from the monotonicity of the exponent and log function.
Next, prove the limiting when oc. Using L' L'Hôpital's 's rule get:
E,
lim alog 40000
3
Σ
c
4.
lim a-+00
Σ,
(22)
a²
lim E sawooftmaxe ArT
Arr
(23)
0-100
1 (s)
Σ Ar)ss
(24)
S
Where the last equality follows from the fact that as a goes to infinity, the softmax term approaches the uniform distribution.
We finally prove the limiting case for a 0. Again using L' L'Hôpital's rule:
lim a log
S
Σ A
=
lim
E sweoftmax
A
AAr max Ar
(25)
Where the final equality follows from the fact that as a goes to zero the softmax term is equivalent to taking a hard maximum over Ar.
A.4 Proof of Theorem 3.3
Proof. First, we want to transform kernel uncertainty into reward uncertainty:
P3
(()(s,a)
log BepgBpprsss (r,, a))] log Epg
(Importance sampling)
(26)
pp
log P p7
2 E, [log(r(s, a))l + Epp
(Jensen)
(27)
= E,g [log(r(s, a)) + Hss (pF) + Ep [logp]]
(28)
We think of = =log o3. as adversarial reward and r' = loog p as nominal reward. We thus have:
P
d d%
Ar' -'r -' loog
log
(29)
pp:
We further observe that:
aLSEs Ar"" )
(7((s)
1/2
a logo(S) = a log
Σ
(30)
S
d
(s))
SES
23

---

Thus F' (c, where (e, a) is the reward uncertainty set described in Theorem containment relation implies:
3.1 This
min PEPT(e,a)
pp [ogp%]] min MER (e,a)
Blr = 1op -lggld (31)
= + log(S)) -6 (32)
Plugging this back to we get:
min
log p [[(,
min PEPa(e.)
Eøp
-log(S)) -e (33)
PEP (e.0r)
We finish the proof by taking the exponent of both sides.
A.4.1 Policy entropy lower bound
We aim to use the same methods as the in the proof of the above theorem, except this time for the regularization aHHs (d=) + alEa [HATT(- |)]] First, note that from the chain rule have HsxxA(p") Hss (d") + -x [HAT s)]. Therefore, we are actually using state-action entropy regularization with temperature a.
We consider the same lower bound as in equation 28 and the same definition of Ar' as in 29 The uncertainty set R. (e, a) corresponding to state-action entropy regularization is (see Table It:
(Ar) a
dp(s)
8 : Log(SA)
aLSEsxA
(og(SS)) a aog
Σ
(34)
dp((s)
s,aESxA d
d7(s) (s)s)
= a log
a log(S) Se
(35)
SES
Which is the exact same uncertainty set PD (c, a) as the one considered for state entropy alone. That being said, using the bounds in table we get:
min log Bog (rg,r((,,)) E,g. [log(r(s, + a(HssA((F)) log(SA)) - (ePa(e.))
(36)
This is worse lower bound than in the state entropy case, since:
HsxA(pp) log(SAA) (Hs(dī)) log(()) (Eaz (Hl [s]] -log(A) (37)
< Hs(dp)) log(S)
(38)
A.5 Proof of theorem 4.1
Let Ω * 0 function. Assume towards contradiction that exists function P(P) that satisfies
minper Epp, [r] = E2 [r] + 12(T P). We consider an MDP with reward vector c where E R is some constant. For all n, P we have:
min E [r] = =c= CTT M)T,P)
(39)
PEP
+ [TT P) =
(40)
But the last line implies Q = 0, a contradiction.
A.6 Proof of Theorem
Before the proof, we state the formal definition of the Conditional Value-at-Risk (CVaR). Let return be defined as R(T) := -xe ''r, where the trajectory (so, ag, .o....) is sampled from

---

80
01
02
R1(121
81
82
p
p
Ra(1-2)
Ra (1(1~)) 7
Figure 7: Counterexample used in the proof of Theorem 4.2 The initial state so- All actions are deterministic except for action taken from state $2. which leads stochastically to 83 with probability 1 p and to 84 with probability p. Self-loops indicate absorbing states with associated returns.
(T( == P(su. ao: Po.... ... Pl. Then, for given confidence level 3 E (0,1) the CVaR of the return under policy ı defined as:
CVaRz[M(7)] = min 7 1- B ((B(T) ((((T) -n>+ vER
1
(41)
where (x)+ = max(x, 0).
Proof. Fix some M > 0, 0, and confidence level < 3 1. In this proof we'll denote the total discounted return by G. We construct an MDP (illustrated in Figure ( that induces arbitrarily poor risk-sensitive performance under occupancy entropy regularization.
The key idea to design an MDP in which two actions from the initial state, d1 and d2. lead to branches that nearly identical in terms of expected return, but differ drastically in their return distributions. Specifically, we construct the branch associated with dg to be extremely risky, such that small degree of stochasticity-inen induced by entropy regularization -itiaaasiinneeposssssseses exposes the agent to severe
downside risk.
More formally, we'll first require that:
[ (80) = ai) - c -0 - (80) = 02] = =pR3= + (1 (1-p) R4
(42)
some > 0, implying x (a1 sp) = and Tt (a1 50) = g Rs (the last claim true because for every temperature
make arbitrarily small).
Second, we'll 11 fix R₁ R and since the branch associated with aj is deterministic we'll have CVaR [G] = J(x) ) = R > We'll require that CVaR [G] < -M which clearly implies that the result from the proof holds.
We now show explicitly that exists Rg. R4, P.,P& s.t. the two conditions above hold. We focus on the 2 [G] minqeu EG GG]
second condition, and use the dual representation of CVaR B: CVaR
where 41 (Q P VG Q(G) P(G) (0,1/3) and P the return distribution for म
Since >> 3 0, we can choose > p > s.t. (1 - 4)p 2 3. We construct probability measure Q u s.t. Q(R3) p(1-9) and Q(R)) = Q(Ra). Note that from the construction of p have that Q(Rs) and 1 - Q(Rs) = Q(Ri) < P(R,) 4 and indeed Q E U. We can now
25

---

analyze the risk averse performance of TR:
p(1 9)
1 p(1 4)
[ S R3
8 + R1
S -M
(43)
Rs - B - R)
+ R
(44)
p(1-9)
We can choose an R3 R' that satisfies the inequality above. Finally, we are only left with satisfying the first condition, and we can fix R4 = 1-p
R-c-pll' and we are done.
A.7 Robustness under estimation errors
In Section 3 we provided robustness guarantees for the reward uncertainty (Theorem 3.1 and kernel uncertainty (Theorem 3.3) of policy maximizing the state-entropy regularized objective. In practice, however, the regularization term not computed with the true state entropy but a sample-based estimate. It is then not obvious than similar guarantees hold under estimation error of the state entropy. In the following, we show how the guarantees generalize to the sample-based regime following straight straightforward concentration argument the state distribution. In the spirit of Section 3 we consider the entropy of finite state space, while we refer to Section 4.3 for an informal discussion on sample sensitivity in continuous settings, in which more sophisticated estimators are employed.
Corollary A.1. Let d an empirical estimate of dip obtained with n sampled trajectories, for which holds I|d dp s 1/2. For (0. and any policy n, the following weak duality holds with probability 5
2S log(2/0)
Sn log(2/8) log(8) 6,
min Ep [
p [r] + Hs(d)-
log
FER
n
where the reward uncertainty set is RD (e, a)) = (F LSES
1 + log(S)}
Proof. To prove the result, sufficient to show that the estimated entropy Hss(d) concentrates around the true entropy Hs(dp). Let R: == lld ddpla. 11, we write
1
+ (1 x) log 1-K (d)(d) Hsl S ni log + x log k
(45)
< 2x log
(46)
25 log(2/0)
Sn log(2/8)
log
[with probability 1 - 5] (47)
where the first inequality follows from continuity bounds of the entropy [60 see], the second inequality follows from the assumption & S 1/2, the last inequality invokes concentration of the empirical distribution (58) Finally, the result is straightforward by plugging equation 47 into the
result of Theorem 3
An analogous result for kernel uncertainty is as follows.
Corollary A.2. Let d an empirical estimate of dip obtained with sampled trajectories, for which it
holds ||d- dz. 1 < 1/2. For any 8 (0, 1), policy n, and 0, the following weak duality holds
with probability 6
/
Sn 2S 1og(2/8) log V log(2/5)
log($) ).)
min Epg (r) exp Epg(log(r)) +a Hs(d) (r(e)
n
where
{ aogo
d (s()s (s)
px (=((a))==
Proof. The proof is straightforward by plugging equation
47
into the result of Theorem 3.3
26

---

B Experiments
The code for our experiments available at |tttpe///gthhu..ccmm/m/baaaaaaa/ entropy_robust_zz
B.1 State entropy algorithm
k-nearest neighbor entropy estimator. Let be a random variable with a probability density function whose support is set X' C R2 Then its differential entropy is given H(X))
-Ez=p(u) (og((((() When the distribution is not available, this quantity can be estimated given N i=1 However, since it is difficult to estimate with high-dimensionn data,
ii.d realizations of { particle-based k-nearest neighbors (k-NN) entropy estimator [22
can be employed:
(x)
1 N
Σ 1=1
N
log
N
2.
NN g/2
29/2
+ Ck
k r
N
1 (X) & N
log 2| - A-NN 112
Where x A-N-NN is the k-NN of 2, within a set {a = log(k) (k) 1 (A:) is bias correction term, v/ the digamma function,I the gamma function, q the dimension of x, â Rs 3.14159. and the transition from (1) to (2) always holds for > O.
Using (2) the state entropy can be modeled into an intrinsic reward in the following way:
(s,)) log(Il y --=nn| 1)
(3)
where ys can be either 8,, latent encoding of s; or selected features of s; The total reward is:
total
3 = r(8j, 0g) + 3 "/ r))))
Where 3 is the regularization temperature, "t is the intrinsic reward discount factor, it's usually the same the extrinsic reward discount factor; empirically find that in environments where the episode doesn't end upon task completion, a stronger intrinsic reward discount helps prevent over-exploration.
Temperature warm-up. We find that it helps performance to start training with a high temperature Batart and anneal it through training. This is common in prior work [50], notably instead of exponential decay we use cosine decay from Sstart to >
Staat
B (1+
+1+cos(= cost a 4)) -
Be 3+
t= L......
2
B.2 Minigrid experiments
The experiment is built upon (26] open code base, which is released under the MIT License Base algorithm is A2C. As the point of the expirement isnt handling partial observability, we remove potential bias by instead of using random encoder, the features used for entropy estimation were the x.y position of the agent one hot encoding of its direction. The nominal environment is an 8 by 8 empty grid with a sparse reward, if goal is reached, 0 otherwise. action space is (1) take a step in current direction, (2) turn right (3) turn left.
Wall perturbation, Fig. 4bb). We randomly sample a point zy along the x-axis (excluding the first and last columns), and add a vertical wall spanning positions (zu, 1) through (Iv, 6).
Random walls, Fig. c). Walls added to 7 blocks randomly sampled from the grid, excluding the start and goal positions.
Goal shift, Fig. dd. The goal position is randomly shifted to any block in the grid, excluding the starting position.
Available http://ghhhbbboom//iiggg////o
27

---

B.3 Pusher experiment
The code for our experiments is based on the CleanRL codebase 20)) which released under the MIT License _ We use PPO with recurrent network for all baselines. Note that it is known that to maximize state entropy over finite number of rollouts, a non-Markovian policy necessary. State entropy is calculated over the puck's x.y positions.
Transition kernel perturbation (wall): The obstacle is composed of three axis-aligned blocks with width 3cm, centered at (0.52, -0.24), (0.55, -0.21) and (0.58, -0.18). These positions are roughly along the perpendicular bisector of the line between the puck's initial position and the goal. We used 200 episodes of length 100 for evaluating each method. As in [3,, to decrease the variance in the results, we fixed the initial state of each episode:
qpos [0., 0., 0., O., 0.3,0,2,
quel = [0., 0., 0., 0., 0., 0,,,0,,,,,,,,,,,,,,,,,,,,,,,
Action robust baseline 66] We ran as baseline both PRMDP and NRMDP. To ensure a fair comparison we implemented the following modifications:
Larger networks: We increased the width of all neural networks to 256 units.
hyperparameter sweep: we swept for exploration noise E [0.2, 0.6, 1], learning rate [e - 3, 5e - 4, le - 4] and batch size E [256, 512, 1024]
Check-pointing: we saved both the best performing checkpoint during training (evaluated in the nominal environment) and the last iteration checkpoint. We evaluated both of them in the perturbed environment and showed the better of the two.
The original goal was located (0.66, -0.35). In the reward shift experiment, new goal is randomly sampled uniformly from the circumference of a circle with radius centered at the original goal location. For each radius [0. 1,0. 15] we randomly sample 200 goals, evaluate once per goal, and report the average performance across all goals.
Reward shift:
Number of rollouts ablation, Fig. b). To ensure fair comparison, we fix the total batch size to 32 across all variants. For a variant using x rollouts per entropy estimate, we divide the 32 rollouts into 32 non-overlapping groups of size x, and compute the intrinsic reward separately for each group. As the number of rollouts affects the scale of the intrinsic reward, we tune the temperature individually for each variant.
B.4 Ant experiment
We train All policies on Mujoco's "ant-v5". To emphasize the affects of each regularization we augment the environment to be less reward shaped in the following way: The episode does not truncate when the ant's in an unhealthy position. Furthermore, there is no reward for being in healthy position and the penalties on enduring contact forces and control cost are turned off. The reward in our setup is only the forward progress reward. State entropy is calculated over the height and orientations of the ant, encouraging the agent to learn to walk in diverse ways.
Transition kernel perturbation (vertical wall) During evaluation, we add an unobservable wall of size of [1, 0o, 0.3]; the only way to pass the wall is to hurdle above it. Evaluation protocol the same as in the pusher experiment.
B.5 Interplay between State and Policy Entropy Regularization
In the experiments reported in the main text, we include policy entropy regularization alongside state entropy to ensure stable training, following common practice. Here, we empirically assess whether policy entropy contributes to robustness beyond its stabilizing role.
As shown in Figure g omitting policy entropy leads to unstable training, reflected in high variance in nominal performance. To isolate the effect of state entropy, take policy trained with both regularizations and continue training it for 2 million steps using only state entropy. As shown in
hunps//githbb.ccm//wwwyyaadddd
28

---

state policy entropy vairations
1.0
state entropy
state & policy entropy
8.8
state entropy warm start
Average success
0.6
6.4
nominal environment
pertubed environment
Figure 8: State entropy regularization unstable training and warm start experiment
0.9
Men1: STD dbts trough cadtet
0.8
3m
330
425
4.50
3.70
6.50
2.1
6M
(a) episodic return
(b) state entropy
(c) policy entropy
Figure 9: State and policy entropy only state entropy distillation
Figure 9 the policy maintains both its return and state entropy, despite significant drop in policy entropy. The resulting policy's robust performance closely matches that of the state-and audle-nndpplicyy entropy variant.
We hypothesize that with more careful algorithmic tuning, state entropy alone could suffice for both robustness and stability. However, we leave a systematic investigation of its role in training stability future work.
B.6 Technical details
All experiments were conducted on a machine equipped with an NVIDIA RTX 4090 GPU. Recreating experiments B.2 takes about 12 hours to run 25 seeds for the baselines, sweeping regularization temperature took an additional day. Recreating experiments B.3 takes day for 25 seeds for the baselines. rollouts, temperature ablations and regularization temperature searching were another days. Other experiments which didn't make it to the final version took about a week of GPU hours.
29


Reviews:
[{"review_id": "jgUdAVRz3i", "reviewer": "Reviewer_rqBU", "summary": "The paper studies the use of state entropy regularization in robust RL. The problem setup is about robust RL where the environment parameters such as the transition kernel and the reward function are not fully known. In such situations, robust RL agents are trained to adapt to any slight changes in these quantities. The authors discuss why policy entropy regularization is suboptimal. They derive uncertain reward sets for state entropy regularization and also show that state-entropy regularization can produce a lower bound for transition kernel uncertainty as well. The authors test their regularization on Minigrid and Mujoco Pusher environments.", "questions": "(1) Is there a better way to estimate the visitation probabilities/entropy?\n\n(2) How would you actually maximize reward + visitation entropy? As this reward now is non-markovian (policy dependent)?", "limitations": "The major limitation is the applicability. The authors experiment on very simple domains with a specialized way of estimating state-visitation distribution which questions where this method can be applied. A thorough experimental evaluation with several baselines is also missing.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n\n(1) Changes in reward functions at test time is a very practical problem so coming up with a simple robust RL method that is mathematically motivated makes sense.\n\n(2) The authors derive the reward uncertainty sets for state-entropy regularized RL.\n\n(3) The authors show that state-entropy regularized RL produce a lower bound for transition kernel uncertainty.\n\n(4) The authors also show that using entropy based regularizations alone, transition kernel based uncertainty cannot be completely solved.\n\nWeakness\n\n(1) Discussions on the physical examples of the reward uncertainty set is missing. There should be some discussion about physically what changes in the reward function are allowed.\n\n(2) The method for computing state-visitation distribution (using k-nearest neighbours) is limiting.\n\n(3) Experiments are also limiting. No real baseline is used apart from policy entropy regularization. Are there other robust RL algorithms?\n\n(4) Missing references:\n\n[1]: Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., & Salakhutdinov, R. (2019). Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274.\n\n[2]: Agarwal, S., Durugkar, I., Stone, P., & Zhang, A. (2023). f-policy gradients: A general framework for goal-conditioned rl using f-divergences. Advances in Neural Information Processing Systems, 36, 12100-12123.", "was_revised": true, "final_justification": "The authors clarified my major concerns.", "rebuttal": "Thank you for taking the time to carefully review our paper and provide constructive feedback. We appreciate the reviewer's comments and will incorporate them into the revised version.\n\n## Questions\n\nBefore answering the reviewer's questions below, we emphasize that these aspects have been thoroughly studied in prior works on state entropy maximization [e.g., 1, 2, 3, 4, 5, 6, 7] and fall outside the scope of our work.\n\n**1. Is there a better way to estimate the visitation probabilities/entropy?**\n\nPossibly, but this is not the focus of our work. In the RL setting, k-NN entropy estimation [1, 2, 3, 4, 5] outperforms alternative methods such as computing the entropy from KDE [6] or learned density models such as VAEs (e.g., [7] mentioned by the reviewer). It has become a standard approach for state entropy maximization, thus justifying our choice.\n\n**2. How would you actually maximize reward + visitation entropy? As this reward now is non-markovian (policy dependent)?**\n\nTo maximize the reward + visitation entropy, we follow state-of-the-art approaches from [1, 3, 4, 5] by combining the external reward with an \"entropy\"-reward derived from the k-NN estimator (implementation details are reported in Appx. B.1). We understand the reviewer's concern: the entropy-reward is technically non-Markovian. However, [1, 3, 4, 5] have shown that it can practically be optimized using standard RL techniques.\n\n## Limitations\n\n**1. Applicability as a major limitation**\n\nThe method used for optimizing the state entropy regularization as analyzed in this paper has been successfully applied in challenging tasks, including high-dimensional continuous control [2] and visual domains [1, 3, 5].\nTherefore, we do not think applicability is a concern and we focus our study on other aspects (robustification induced by the regularization). \nIn principle, our findings should apply to any domain in which state entropy maximization has been successfully deployed.\n\n## Weaknesses\n\n**1. Physical examples of the reward uncertainty set**\n\nThank you for raising this point. Since the reviewer's comment may reflect two distinct concerns, we address both interpretations below—one regarding the interpretation of the uncertainty set, the other regarding reward uncertainty experiments on a physical domain.\n \n **1.1. Bridging Theory and Intuition for the Reward Uncertainty Set**\n\nWe acknowledge that the uncertainty set may initially seem abstract and we will clarify its intuition in the revised version. Specifically, as suggested, we will add a worked example on a simple two-state, two-action MDP illustrating that the method is robust against an adversary with a soft exponential budget (the paper already includes a basic illustrative example in Figure 1). In addition, we believe that highlighting how the uncertainty set interpolates between more familiar $\\ell_\\infty$-like (localized, uniformly bounded) and $\\ell_1$-like (global, budgeted) perturbations can provide further interpretability. Finally, while the set itself couples rewards through a log-sum-exp form, we would like to emphasize that the worst-case perturbations remain intuitive: policy entropy allows only local statewise attacks, whereas state entropy enables global attacks concentrated on frequently visited states.\n\n**1.2. Empirical Illustration of Reward Uncertainty in Pusher**\n\nIf the reviewer’s comment refers to the absence of physical examples involving reward perturbations, we have prepared an additional experiment in the Pusher domain that we will include in the revised version. In this experiment, we simulate reward uncertainty by randomizing the puck’s goal location at test time, uniformly sampling from a circle around the nominal target. This setup captures a natural class of perturbations arising from slight spatial shifts in task objectives. As shown in the table below, the agent trained with state entropy regularization achieves better performance under this variation compared to other baselines.\n\n| agent\\reward shift radius | no shift | 0.1 |0.15|\n| -------- | -------- | -------- | -------- |\n| Unregularized | **0.98+-0.02**|0.53+-0.05 | 0.30+-0.03   |\n| Policy entropy| 0.94+-0.017|0.65+-0.041|0.31+-0.04     |\n| State entropy | 0.95+-0.011|**0.71+-0.039**  |**0.35+-0.03**   |\n\n\n**2. Regarding the Use of k-NN State-Occupancy Entropy Estimation**\n\nPlease see our answer to Question 1.\n\n\n**3. Additional Robust Baselines and Domains**\n\nWe thank the reviewer for raising this important point.\n\nFirst, we extended the original Pusher experiment by adding a standard robust RL baseline [8]. As shown in the table below, the agent trained with state entropy regularization significantly outperforms this baseline. While we do not expect state entropy to consistently outperform robust RL methods across all types of perturbations, this result—along with our analysis in the main paper—suggests that it is particularly effective against spatially correlated perturbations, which are not the primary focus of most standard approaches.\n\n| agent\\task | nominal env |perturbed env - wall|\n| -------- | -------- | -------- |\n| Unregularized | 0.98+-0.02|0.005+-0.01| \n| Policy entropy| 0.94+-0.017|0.03+-0.03|\n| State entropy | 0.95+-0.011|**0.37+-0.12** |\n| PRMDP[8] | 0.98+-0.015|0.02+-0.03 |\n| NRMDP[8]| **1+-0**|0.03+-0.06 |\n\nSecond, we added an additional experiment on the standard Ant benchmark in MuJoCo. In this setting, the agent is trained to progress along the $y$-axis. At test time, we introduce a hurdle perpendicular to this axis that the agent must jump over. As shown in the table below, the agent trained with state entropy regularization significantly outperforms both the unregularized agent and the one regularized with only policy entropy.\n\n| agent\\performance | nominal env -reward | perturbed env -obstacle pass rate |\n| -------- | -------- | -------- |\n| Unregularized | 802.65+-97.96|0.13+-0.04 |\n| Policy entropy| **827.25+-85.72**|0.21+-0.07|\n| State entropy | 795.01+-122.32|**0.36+-0.07** |\n\n**4. Missing References**\n\nWe thank the reviewer for pointing out these relevant works. We will include both in the revised related work section.\n\n---\n\n\n**References:**\n\n[1] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021.\n\n[2] Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate. In AAAI Conference on Artificial Intelligence, 2021.\n\n[3] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, 2021.\n\n[4] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, Younggyo Seo. Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration. In Advances in Neural Information Processing Systems (NeurIPS) 2023\n\n[5] Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto. Reinforcement Learning with Prototypical Representations. In International Conference on Machine Learning, 2021.\n\n[6] Elad Hazan, Sham Kakade, Karan Singh, Abby van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning 2019.\n\n[7] Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., & Salakhutdinov, R. (2019). Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274.\n\n[8] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in\ncontinuous control. In International Conference on Machine Learning, pp. 6215–6224. PMLR, 2019."}, {"review_id": "r6esDVWTQQ", "reviewer": "Reviewer_i4aV", "summary": "The paper investigates the theoretical guarantees and practical implications of using state entropy regularization in Reinforcement Learning (RL) to improve robustness. While state entropy regularization has recently been empirically shown to outperform policy entropy measures (i.e. action conditional entropy), its theoretical underpinnings regarding robustness have not received much attention.\n\nThe main contributions are:\n- The paper characterizes robustness succinctly, following prior work, stating that state entropy regularization enhances robustness against spatially correlated perturbations (i.e. a local adversary). \n- The analysis contrasts state entropy with the widely used policy entropy regularization. It shows that policy entropy regularization protects only against local adversaries, whereas state entropy provides robustness against globally informed perturbations (by encouraging state space coverage).\n- The paper derives theoretical guarantees, largely following prior work.", "questions": "Please discuss the weaknesses outlined above. In particular:\n- It would be good for me to confirm my sense of how the authors think their work fits into the landscape of theoretical results and how practically relevant the derived results actually are.\n- It would be good to understand if the authors have had the chance to experiment with less toy-ish domains.\n- The \"fully kernel robust\" setting doesn't strike me as very practical (and isn't achievable as mentioned in the paper). This isn't a strong weakness in the analysis but I wonder if there are more practical robustness settings one should worry about and analyse?", "limitations": "The paper does a good job at not only selling one method but highlighting limitations throughout and discussing them.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- I worked my way through the theoretical derivations once and while I am not an expert RL theory the derivations seem sound (albeit somewhat close to prior work, see comment below).\n- (Policy) entropy regularization is highly used in current SOTA RL algorithms and a paper characterizing it further and explaining differences between different regularization schemes is relevant to the community.\n- The paper includes some insightful findings that I would not have guessed before reading it. E.g. the paper illustrates that the robustness benefits of state entropy are more sensitive to the number of rollouts used for policy evaluation compared to policy entropy. This sensitivity is attributed to the statistical difficulty of accurately estimating state entropy and behavioral aspects in low-rollout regimes. Another surprising find (at least without thinking carefully about it) is that state and action entropy regularization together can be worse than state entropy regularization by itself in some cases.\n- The paper is generally well written and easy to follow despite it's heavily theoretical content.\n- The presented didactic (toy) examples (e.g. Fig.1 and Fig.3) do a good job at explaining the issues with policy entropy regularization and the sensitivity of state entropy regularization.  \n- I like that the paper doesn't try to sell their analysis or experiments of one method but rather tries to give a balanced view of strengths and weaknesses. \n\nWeaknesses:\n- The derivation of the theoretical results seems like a relatively straightforward extension to known results for policy entropy regularization (which is already known to solve a robust RL problem). The relevant works are cited and the extensions cover a case that's clearly relevant to the community. Nonetheless it does dampen the strict \"novelty\" of this part a bit.\n- To the best of my understanding the analysis largely covers the discrete/tabular case and it's applicability to more practical settings is somewhat limited. The authors also acknowledge that in practice some positive effects of policy entropy regularization are observed that aren't fully explainable by the theory.\n- Minor there is some notation that is a bit loose in the main paper (e.g. Dissimilarity of reward functions isn't formally defined, definition of adversary is a bit vague etc.) I think this is ok as it aids the flow of the paper not having too much notation, but I could imagine some readers with a more theoretical background searching for some definitions.\n- State entropy is notoriously hard to estimate and work with (this is also somewhat covered by the theoretical results re. sensitivity to rollouts). So perhaps a strong guarantee on the case where it can be estimated nearly exactly is not as impactful in practice.\n- The experiments are on fairly toyish domains. The MiniGrid example clearly is more didactic than practically relevant. The control domain (pusher) is good but the only reasonably high dimensional domain considered. It would have been nice to see results on some standard domains or some more practically relevant domains.", "was_revised": true, "final_justification": "The authors addressed my concerns adequately and thus I am raising my score to an accept.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. Below, we respond to the main points raised below, and will incorporate relevant clarifications and improvements into the revised version.\n\n## Questions\n\n**1. Clarifying the Positioning and Significance of our Analysis**\n\nWe thank the reviewer for the question and appreciate the opportunity to clarify.\n\nWhile previous studies have focused on the connection between robustness and policy entropy, our work is the first to thoroughly characterize the robustness properties of state entropy regularization—including its benefits, limitations, and interaction with policy entropy—both theoretically and empirically. In particular, we show that state entropy naturally induces robustness to spatially correlated perturbations, a class of real-world deviations (e.g., in transfer learning) that remains underexplored in the robust RL literature—not only in the context of entropy regularization.\n\nMoreover, maximum state entropy is commonly used for practical purposes such as exploration and coverage [1,2,3,4,5], not explicitly for robustness. By analyzing the type of robustness it provides and its practical challenges—especially its reliance on rollout budgets—we aim to inform both the theoretical understanding and empirical use of this regularizer.\n\n**2. Additional Experiments**\n\nWe indeed had the opportunity to experiment with additional domains beyond those presented in the main paper and plan to incorporate these results in the revision.\n\n- We first ran an additional experiment on Ant in MuJoCo, where the goal is to progress as far as possible along the $y$-axis within an episode. At test time, we introduce a hurdle perpendicular to this axis. To continue progressing, the agent must jump over the obstacle. As shown in the table below, the agent trained with state entropy regularization significantly outperforms the unregularized agent and the one regularized with policy entropy.\n\n\n| agent\\performance | nominal env -reward | perturbed env -obstacle pass rate |\n| -------- | -------- | -------- |\n| Unregularized | 802.65+-97.96|0.13+-0.04 |\n| Policy entropy| **827.25+-85.72**|0.21+-0.07|\n| State entropy | 795.01+-122.32|**0.36+-0.07** |\n\n- We also ran an experiment illustrating reward robustness in continuous control. In this experiment conducted in Pusher, we randomized the puck's target location at test time by uniformly sampling it within a circle around the nominal goal. As shown in the table below, the agent trained with state entropy regularization achieves better performance under this variation compared to other baselines.\n\n| agent\\reward shift radius | no shift | 0.1 |0.15|\n| -------- | -------- | -------- | -------- |\n| Unregularized | **0.98+-0.02**|0.53+-0.05 | 0.30+-0.03   |\n| Policy entropy| 0.94+-0.017|0.65+-0.041|0.31+-0.04     |\n| State entropy | 0.95+-0.011|**0.71+-0.039**  |**0.35+-0.03**   |\n\n\n- We finally extended the original Pusher experiment by adding action-robust baselines, which seem to be standard in robust RL for continuous control [6]. As shown in the table below, state entropy regularization significantly outperforms these baselines. Nonetheless, we must emphasize that we do not expect state entropy to consistently outperform robust RL methods across all types of perturbation. However, this result—along with our accompanying analysis—suggests that it can be particularly effective in scenarios involving spatially correlated perturbations, something that standard methods are not designed to address.\n\n| agent\\task | nominal env |perturbed env - wall|\n| -------- | -------- | -------- |\n| Unregularized | 0.98+-0.02|0.005+-0.01| \n| Policy entropy| 0.94+-0.017|0.03+-0.03|\n| State entropy | 0.95+-0.011|**0.37+-0.12** |\n| PRMDP[6] | 0.98+-0.015|0.02+-0.03 |\n| NRMDP[6]| **1+-0**|0.03+-0.06 |\n\n\n**3. On the Practicality of Kernel-Robust RL**\n\nThank you for raising this point. Generalizing RL agents to environments that differ from training is a fundamental challenge with many important applications (e.g., sim-to-real transfer). While multiple formulations and approaches address this challenge (e.g., [9]), one prominent and well-studied framework is robust RL, which explicitly models uncertainty and optimizes for the worst-case performance (e.g., [10, 6, 11]). Within this framework, kernel robustness remains a widely used and practical formulation—particularly in settings where test-time shifts are hard to anticipate and performance must remain reliable under a range of plausible deviations.\n\nWe also clarify that our negative result regarding kernel-robust MDPs does not imply that this setting is intractable in practice. Rather, it shows that no regularizer depending only on the policy and transition kernel can capture the robust return across all MDPs. In practice, effective solutions exist for practically useful uncertainty sets (e.g., [6] presents an effective solution for a rectangular uncertainty set).\n\n## Weaknesses\n\n**1. On the Novelty of the Results**\n\nWe acknowledge that our analysis builds on similar tools used in prior work on policy entropy and robust RL—a connection the reviewer rightly notes is explicitly acknowledged in the main paper. That said, while we adopt similar techniques, the results themselves are novel. Furthermore, we believe the insights and perspectives developed throughout the paper—summarized concisely in our response to Question 1—are themselves original and offer meaningful value to the community.\n\n**2. On Tabular/Discrete Assumptions**\n\nWe thank the reviewer for the thoughtful comment. Analyzing the tabular case is a standard approach in the literature, as it allows us to characterize robustness in a tractable setting while still offering insights applicable to more complex domains. We also note that most of the results can be extended to the continuous case with relatively minor adjustments, and we will add a note on this in the revision.\n\n**3. Minor Comments**\n\nWe thank the reviewer for the suggestion and will revise the text to clarify key terms while maintaining readability.\n\n**4. State Entropy for Robustness Under Noisy Estimation**\n\nThe reviewer is making a great point that the estimation error of the entropy may degrade the robustness guarantees in practice. We can quantify the degradation by specializing the results of Th. 3.1 (reward robustness) and Th. 3.3 (kernel uncertainity) for the estimated entropy regime. Let $\\hat d$ be the empirical estimate of $d^\\pi_P$ obtained with $n$ sampled trajectories. We can prove that\n$$\n\\mathcal{H} (d^\\pi_P) \\geq \\mathcal{H} (\\hat d) - \\sqrt{\\frac{2S \\log (2 / \\delta)}{n}} \\log \\sqrt{\\frac{S n}{\\log (2 /\\delta)}}\n$$\nholds with probability $1 - \\delta$ for $\\delta \\in (0, 1)$. Then, we can plug the right-hand side in place of $\\mathcal{H} (d^\\pi_P)$ into Eq. 6 and 7 to obtain sample-based high porbability lower bounds on the robustness. The sample-based bounds approximate the exact entropy results at a $O(n^{-1/2})$. We will add these results to the paper together with their proof, which is based on centration of the empirical disitribution [7] and continuity bounds of the entropy [8].\n\n**5. Regarding Experimental Domains**\n\nWe thank the reviewer for the comment. Please see our response to Question 2, where we present additional experiments we plan to include in the revised version.\n\n---\n\nReferences:\n\n[1] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021.\n\n[2] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, 2021.\n\n[3] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement\nlearning with value-conditional state entropy exploration. In Advances in Neural Information\nProcessing Systems, 2023.\n\n[4] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, Younggyo Seo. Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration. In Advances in Neural Information Processing Systems (NeurIPS) 2023\n\n[5] Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto. Reinforcement Learning with Prototypical Representations. In International Conference on Machine Learning, 2021.\n\n[6] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pp. 6215–6224. PMLR, 2019.\n\n[7] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, Marcelo J. Weinberger. Inequalities for the L1 Deviation of the Empirical Distribution. 2003.\n[8] Andreas Winter. Tight uniform continuity bounds for quantum entropies. 2015.\n\n[9] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. 2023. A Survey of Zero-shot Generalisation in Deep Reinforcement Learning. J. Artif. Int. Res. 76 (May 2023).\n\n[10] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817–2826. PMLR, 2017.\n\n[11] Uri Gadot, Kaixin Wang, Navdeep Kumar, Kfir Y. Levy, and Shie Mannor. 2024. Bring your own (non-robust) algorithm to solve robust MDPs by estimating the worst kernel. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 576, 14408–14432."}, {"review_id": "sxH26N1Q8X", "reviewer": "Reviewer_6Gze", "summary": "+ Summary & Contributions\n\t- This paper offers a mixture of theoretical and empirical contributions concerning the use of state entropy regularization in RL.\n\t- Various theoretical results are provided to elucidate properties of state entropy regularization and draw explicit connections between classic RL with state entropy regularization and standard robust RL within an uncertainty set over the reward function.\n\t- Insights into the practical aspects of employing state entropy regularization are also provided with regards to the suitable sample size needed to obtain reliable entropy estimates. \n\t- Empirical results are provided across both MiniGrid and Mujoco environments illustrating that, while nominal performance is comparable and minimally affected by the presence of state entropy regularization, there can be statistically significant improvements in policy robustness. Ablations are included to assess the impact of sample size and the free temperature parameter used in characterizing the reward uncertainty sets.", "questions": "I believe my questions for the authors should be apparent in my review above.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ Quality\n\t- Strengths\n\t\t- Aside from the comment below concerning Theorem 4.1, all the technical proofs seem to be in order. \n\t\t- The main theorems in Section 3 do a good job of walking the reader through the novel connection between state entropy regularization and robust RL with reward uncertainty sets.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- Theorem 4.1 seems rather strange. It would be one thing to assert that there exists a MDP where a certain property does or does not hold. It seems like quite another thing to assert that there does not exist any MDP where a property holds by illustrating that the property does not hold for a single, specific MDP. Either I've misunderstood something about the goals of Section 4.1 or Theorem 4.1 is nowhere near as general as the authors claim it is. The verbatim of Theorem 4.2 (that is, a \"there exists a MDP\" claim) seems more in line with what is shown in the proof.\n\t\t* Minor\n\t\t\t- In Appendix A.3, there is no such thing as \"l’hospital’s rule\" but there is a result known as L'Hôpital's rule.\n\t\t\t- The CVaR result, while clearly important towards risk-averse RL, seems like a bit of an \"odd man out\" relative to the other results. It would have been nice to see an empirical result on risk-averse RL to make CVaR seem less like an after thought.\n\n+ Clarity\n\t- Strengths\n\t\t- The paper is well-written and reasonably structured.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- N/A\n\t\t\n\n+ Originality\n\t- Strengths\n\t\t- While I am not intimately familiar with the existing literature on robust RL, the provided theoretical results are novel, to the best of my knowledge, while appropriately extending existing results (credited by citation) as needed.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- N/A\n\n+ Significance\n\t- Strengths\n\t\t- Improving the robustness of deep RL algorithms continues to be an important area of research. While the scale of the empirical results reported in this work is modest, this seems perfectly acceptable for a paper leaning much more heavily on clean theoretical analysis and insights.\n\t\t- Empirical results demonstrate clear robustness benefits of state entropy regularization and the experiments have been conducted with an ample number of seeds to draw meaningful conclusions.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- As a RL researcher who does not spend much time thinking about robust RL, I can only truly confirm that the theoretical results provided appear to be correct without having much insight into their broader significance and impact on this sub-area of the RL literature. Furthermore, I don't get the impression that any one of the theorems proven in this paper constitute a significant standalone advance; however, it may be the case that the collective insight of the theorems is the valuable theoretical contribution upon which this paper rests. I would be curious to hear the authors' thoughts on the impact of their theoretical contributions and how they perceive the advancement stemming from their contributions to existing work on robust RL.\n\t\t* Minor\n\t\t\t- N/A\n\t\t\t\n\t\t\n+ Final Remarks\n\t- Overall, I think this paper scores nicely on all axes and the issues I've raised can be resolved/clarified quite quickly from the authors via the rebuttal process. My main hestiation is on the perceived impact of this work for the robust RL literature, which is already a difficult and subjective thing to forecast and measure in any paper. With that in mind, I'll recommend acceptance for the paper and look forward to hearing from the authors during the rebuttal process on the few points I've raised.", "was_revised": true, "final_justification": "During the rebuttal period, the authors addressed my major concerns regarding quality and significance. They have also productively engaged with and resolved concerns raised by other reviewers. With that, I maintain my original score for acceptance.", "rebuttal": "We thank the reviewer for the thoughtful and encouraging review. We appreciate the time and care dedicated to evaluating our work, and will make sure to incorporate all suggestions and clarifications into the last version.\n\n## Weaknesses\n\n**1. Quality: Clarifying the Scope of Theorem 4.1**\n\nThank you for raising this point, we understand how the scope of Theorem 4.1 may have been unclear. To clarify, we say that a kernel-uncertainty problem defined by $\\tilde{\\mathcal{P}}(\\mathrm{MDP}, \\pi)$ is equivalent to a regularized objective defined by $\\Omega(\\mathrm{MDP}, \\pi)$ if for every MDP and policy $\\pi$ it holds that $\\min\\_{\\tilde{P} \\in \\tilde{\\mathcal{P}}}\\mathbb{E}\\_{\\rho\\_\\tilde{P}^\\pi}[r] = \\mathbb{E}\\_{\\rho\\_P^\\pi}[r] + \\Omega$. This notion of equivalence is the same as the one used earlier in the paper when showing the relationship between reward robustness and entropy regularization, and aligns with definitions in prior robustness literature (e.g., [1]). Here, we focus on problems where both the uncertainty set $\\tilde{\\mathcal{P}}$ and the regularization $\\Omega$ depend only on the nominal kernel $P$ and the policy $\\pi$. In this setting, exhibiting a single MDP where the equality fails for all $\\tilde{\\mathcal{P}}$ and $\\Omega$ is sufficient to disprove universal equivalence.\n\nThe intended takeaway of Theorem 4.1 is not that equivalence fails for all MDPs, but rather that for every uncertainty which is a function only of $P$ and $\\pi$, there does not exist a regularizer—also depending only on $P$ and $\\pi$—that can represent the robust return for all MDPs. We will rephrase the theorem in the revision to better reflect this intended level of generality.\n\n**2. Quality: Minor Comments**\n\nThank you for pointing this out—we will correct the spelling to L’Hôpital’s rule.\n\nRegarding the CVaR result, we included it as a theoretical complement to highlight an important limitation of state entropy in risk-averse settings. We agree that it may feel out of the main paper's thread and will reduce its emphasis in the revision to keep the narrative more focused (e.g., remove the preliminary mention in the main text).\n\n**3. Significance: Theoretical Contributions**\n\nWhile we do provide novel individual results, we agree that a primary contribution lies in the collective insight of our analysis.\n\nFor robust RL, we show state entropy regularization induces robustness to spatially correlated perturbations—a class of perturbations that is highly relevant in settings like transfer learning, yet remains sparsely addressed in the robust RL literature.\n\nAdditionally, maximum state entropy is well-studied for exploration and coverage, but never explicitly intended for robust RL. Here we highlight the robustness benefits and practical considerations associated with this regularizer, which we believe sheds new light on this important line of work. Our analysis clarifies the type of robustness it implicitly provides and emphasizes key practical aspects—namely, its sensitivity to rollout budgets for reliably realizing these robustness benefits.\n\n---\n\nReferences:\n\n[1] Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022."}, {"review_id": "bSxLaLeAPm", "reviewer": "Reviewer_2pKs", "summary": "This paper contrasts State Entropy Regularization (SER) against the more common Policy Entropy Regularization (PER). The claim given is that SER is robust to more global perturbations whereas PER protects against a locally informed adversary. Theoretical results are provided to support these claims. This is followed by a discussion of the limitations of SER, namely that it can hurt performance in risk-averse settings, and is also sensitive to the number of rollouts used to compute the state entropy and evaluate the policy. Empirical results are provided on a discrete and a continuous RL task.", "questions": "1. Given your analysis and results, can you provide a structured way to decide on when to use PER, SER, and SEAR? Would SER suffice in every case if we have sufficient compute and parallelization to generate the number of rollouts required for it to work?\n2. How scalable will SER be for problems with high-dimensional state-space?", "limitations": "The authors adequately address the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The theoretical results comparing SER, PER, and State-Action Entropy Regularization (SEAR) is really valuable from a practical standpoint because depending on the exploration behavior and robustness properties of the regularization, it may or may not be suitable for a particular task.\n2. The paper is mathematically rigorous and the notation is consistent and generally easy to follow (see below for some exceptions and suggestions).\n3. Given the more theoretical focus of this paper, the empirical results are adequate and substantiate the claims made by the authors (see below for some exceptions and suggestions).\n\n## Weaknesses\n\n1. The authors claim SER is more robust to global perturbations and PER to local. This needs to be made more precise, perhaps with more numerical results. For example, in Table 1, from the column on worst case reward, intuition is provided on why PER tackles adversaries with local information and why SER tackles adversaries with global state distribution information. However, in the example of the MiniGrid environment showing kernel robustness, a single obstruction in the optimal path is considered as a global change, whereas obstructions spread out in state space is considered local. Perhaps a paragraph making it precise what it means to be local and global in this context would be helpful.\n2. The derivations in the Appendix skip over quite a bit of steps, which makes it hard to follow in some parts. For example, how do you get to Equation 14 in Appendix A.1, given that (negative) entropy is the convex conjugate of LSE? Given that the space in the Appendix is not limited, giving more explanation on the jumps between different equations will enhance readability quite a bit.", "was_revised": true, "final_justification": "The authors have addressed all of my questions, I have kept my score, and I still recommend acceptance.", "rebuttal": "Thank you very much for the careful and constructive review. We will make sure to reflect the reviewer's suggestions and clarifications in the revision.\n\n## Questions\n\n**1. Guidelines for Choosing Between PER, SER, and SEAR**\n\nWe thank the reviewer for this interesting and practical question. \n\nThese regularizers have additional benefits beyond robustness (e.g., stability and exploration), but these are outside the scope of this paper. From a robustness perspective, the choice of regularization should mainly depend on prior belief about the deployment environment: policy entropy (PER) suits better to small, uniform perturbation, as it spreads stochasticity along a dominant trajectory, whereas for larger, spatially correlated disruptions, state entropy (SER) promotes broader coverage across multiple well\\-performing trajectories.\n\nIn the absence of clear prior knowledge, our results suggest that, when properly tuned, SER does not harm nominal performance and does not diminish the robustness benefits of PER, so it can be incorporated.\n\n**2. Scalability of SER in High-Dimensional State-Space**\n\nThank you for raising this important remark. As discussed in the paper e.g., when using the k\\-NN estimate, state entropy estimation indeed becomes more statistically demanding as the dimensionality of the state space increases. In practice, this can be mitigated by computing entropy over a subspace of relevant state features selected based on prior knowledge of the test environment, or by estimating entropy over latent state representations, as demonstrated in prior work [1,2,3,4,5]. These approaches help scaling SER to high-dimensional problems.\n\n## Weaknesses\n\n**1. Clarifying the Notion of Local vs. Global Perturbations**\n\nWe appreciate the reviewer’s comment and recognize that the distinction between “local” and “global” could benefit from more consistent wording, particularly when using these terms to describe types of perturbations and behaviors. We will keep this in mind and use clearer, more precise language in the revision.\n\nA more accurate way to phrase our results is that SER improves robustness to spatially correlated perturbations—or equivalently, against adversaries that exploit global properties of the agent’s behavior across the state space. In contrast, PER primarily improves robustness against perturbations that are not strongly state-specific or spatially structured.\n\nFor example, in MiniGrid, the wall-type perturbation places obstacles in highly correlated locations, creating a major local blockage. In contrast, in the spread\\-out perturbation, each cell is obstructed independently with a small uniform probability, leading to more scattered, uncorrelated disruptions.\n\nEach regularizer induces a behavior that aligns with the type of robustness it provides: to handle spatially correlated perturbations, an SER\\-agent learns to take more diverse trajectories, exhibiting more “global” behavior, as expected when maximizing state entropy. Conversely, a PER agent spreads randomness only along a single dominant trajectory, exhibiting more “local” behavior that yields robustness primarily against smaller, more uniform perturbations.\n\n**2. Improving Clarity and Detail in Appendix Derivations**\n\nWe thank the reviewer for the helpful suggestion. We will expand the appendix derivations by adding the missing intermediate steps.\n\nSpecifically, for the proof in Appendix A.1 \\- equation 13 implies that $\\min_{\\tilde r \\in \\tilde R}\\mathbb{E}\\_{\\rho^\\pi}[\\tilde r] = \\mathbb{E}\\_{\\rho^\\pi}[r]-\\epsilon + \\min_{\\Delta r} (-\\langle d^\\pi,\\Delta r^\\pi \\rangle +\\text{LSE}_{\\mathcal{S}}(\\Delta r^\\pi))$. The fact that negative entropy is the convex conjugate of LSE implies that $\\min\\_{\\Delta r} (-\\langle d^\\pi,\\Delta r^\\pi \\rangle +\\text{LSE}\\_{\\mathcal{S}} (\\Delta r^\\pi)) = \\mathcal{H}\\_{\\mathcal{S}} (d^\\pi)$. By substituting this in we get our desired result $\\min\\_{\\tilde r \\in \\tilde R} \\mathbb{E}\\_{\\rho^\\pi}[\\tilde r] = \\mathbb{E}\\_{\\rho^\\pi}[r]-\\epsilon + \\mathcal{H}\\_{\\mathcal{S}} (d^\\pi)$.\n\n---\n\nReferences:\n\n[1] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021.\n\n[2] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, 2021.\n\n[3] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement\nlearning with value-conditional state entropy exploration. In Advances in Neural Information\nProcessing Systems, 2023.\n\n[4] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, Younggyo Seo. Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration. In Advances in Neural Information Processing Systems (NeurIPS) 2023\n\n[5] Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto. Reinforcement Learning with Prototypical Representations. In International Conference on Machine Learning, 2021."}]

Author rebuttal:

Meta-review: (a) Scientific Claims and Findings

The paper's central scientific contribution is the formal characterization of state entropy regularization's effect on policy robustness. The key finding is that maximizing the entropy of the state visitation distribution is equivalent to solving a robust RL problem against a "globally informed" adversary that can introduce structured, spatially correlated perturbations to the reward function. This is contrasted with policy entropy, which is shown to protect against "locally informed" adversaries that make small, state-specific changes. The paper also provides a performance lower bound under transition kernel uncertainty and formally proves the limitations of the approach, namely that entropy regularization alone cannot solve general kernel-robust problems and can be detrimental in risk-averse scenarios.

(b) Strengths

The paper has several strengths. First, it addresses a clear and important gap in the literature, providing the first formal analysis of a widely used technique (state entropy maximization) through the lens of robustness. Second, the analysis is rigorous and comprehensive, presenting a balanced view that includes not only positive results (duality theorems, performance bounds) but also important limitations (impossibility results, risk-averse degradation, sample sensitivity). Third, the paper is very clear and well-written, making complex theoretical concepts accessible through intuition and illustrative examples. Finally, the empirical results, especially after being augmented during the rebuttal, provide convincing support for the theoretical claims across different domains and perturbation types.

(c) Weaknesses

The initial submission had some weaknesses, primarily concerning the limited scope of the empirical validation and the lack of comparison to standard robust RL baselines. Please note that the theoretical techniques are mainly trivial extensions of prior work, though applied in a novel context and well explained. However, these weaknesses were almost entirely addressed by the authors' thorough rebuttal, which included new experiments on more challenging tasks (Ant) and against relevant robust RL methods, thereby substantiating their claims more forcefully. 

(d) Reasons for Decision

The recommendation to accept is based on the paper's significant contribution, technical quality, and potential impact. The paper provides novel and valuable insights into the connection between exploration and robustness, clarifying what kind of robustness state entropy provides. This is of high practical importance, as it offers guidance on when this form of regularization is most appropriate. 

(e) Discussion and Rebuttal

The initial concerns of the reviewers were thoughtfully and comprehensively addressed with new results that substantially strengthened the paper, leading to a unanimous and enthusiastic consensus for acceptance. Strong recommendation to include the new experiments and the clarification on the theory in the final version
Decision: Accept (oral)