Title: State Entropy Regularization for Robust Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Robust Reinforcement Learning

Risk-Averse Reinforcement Learning

Regularized Reinforcement Learning
Abstract: State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its theoretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contrasts State Entropy Regularization (SER) against the more common Policy Entropy Regularization (PER). The claim given is that SER is robust to more global perturbations whereas PER protects against a locally informed adversary. Theoretical results are provided to support these claims. This is followed by a discussion of the limitations of SER, namely that it can hurt performance in risk-averse settings, and is also sensitive to the number of rollouts used to compute the state entropy and evaluate the policy. Empirical results are provided on a discrete and a continuous RL task.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The theoretical results comparing SER, PER, and State-Action Entropy Regularization (SEAR) is really valuable from a practical standpoint because depending on the exploration behavior and robustness properties of the regularization, it may or may not be suitable for a particular task.\n2. The paper is mathematically rigorous and the notation is consistent and generally easy to follow (see below for some exceptions and suggestions).\n3. Given the more theoretical focus of this paper, the empirical results are adequate and substantiate the claims made by the authors (see below for some exceptions and suggestions).\n\n## Weaknesses\n\n1. The authors claim SER is more robust to global perturbations and PER to local. This needs to be made more precise, perhaps with more numerical results. For example, in Table 1, from the column on worst case reward, intuition is provided on why PER tackles adversaries with local information and why SER tackles adversaries with global state distribution information. However, in the example of the MiniGrid environment showing kernel robustness, a single obstruction in the optimal path is considered as a global change, whereas obstructions spread out in state space is considered local. Perhaps a paragraph making it precise what it means to be local and global in this context would be helpful.\n2. The derivations in the Appendix skip over quite a bit of steps, which makes it hard to follow in some parts. For example, how do you get to Equation 14 in Appendix A.1, given that (negative) entropy is the convex conjugate of LSE? Given that the space in the Appendix is not limited, giving more explanation on the jumps between different equations will enhance readability quite a bit.\n\nQUESTIONS:\n1. Given your analysis and results, can you provide a structured way to decide on when to use PER, SER, and SEAR? Would SER suffice in every case if we have sufficient compute and parallelization to generate the number of rollouts required for it to work?\n2. How scalable will SER be for problems with high-dimensional state-space?\n\nLIMITATIONS:\nThe authors adequately address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n+ Summary & Contributions\n\t- This paper offers a mixture of theoretical and empirical contributions concerning the use of state entropy regularization in RL.\n\t- Various theoretical results are provided to elucidate properties of state entropy regularization and draw explicit connections between classic RL with state entropy regularization and standard robust RL within an uncertainty set over the reward function.\n\t- Insights into the practical aspects of employing state entropy regularization are also provided with regards to the suitable sample size needed to obtain reliable entropy estimates. \n\t- Empirical results are provided across both MiniGrid and Mujoco environments illustrating that, while nominal performance is comparable and minimally affected by the presence of state entropy regularization, there can be statistically significant improvements in policy robustness. Ablations are included to assess the impact of sample size and the free temperature parameter used in characterizing the reward uncertainty sets.\n\nSTRENGTHS AND WEAKNESSES:\n+ Quality\n\t- Strengths\n\t\t- Aside from the comment below concerning Theorem 4.1, all the technical proofs seem to be in order. \n\t\t- The main theorems in Section 3 do a good job of walking the reader through the novel connection between state entropy regularization and robust RL with reward uncertainty sets.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- Theorem 4.1 seems rather strange. It would be one thing to assert that there exists a MDP where a certain property does or does not hold. It seems like quite another thing to assert that there does not exist any MDP where a property holds by illustrating that the property does not hold for a single, specific MDP. Either I've misunderstood something about the goals of Section 4.1 or Theorem 4.1 is nowhere near as general as the authors claim it is. The verbatim of Theorem 4.2 (that is, a \"there exists a MDP\" claim) seems more in line with what is shown in the proof.\n\t\t* Minor\n\t\t\t- In Appendix A.3, there is no such thing as \"l’hospital’s rule\" but there is a result known as L'Hôpital's rule.\n\t\t\t- The CVaR result, while clearly important towards risk-averse RL, seems like a bit of an \"odd man out\" relative to the other results. It would have been nice to see an empirical result on risk-averse RL to make CVaR seem less like an after thought.\n\n+ Clarity\n\t- Strengths\n\t\t- The paper is well-written and reasonably structured.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- N/A\n\t\t\n\n+ Originality\n\t- Strengths\n\t\t- While I am not intimately familiar with the existing literature on robust RL, the provided theoretical results are novel, to the best of my knowledge, while appropriately extending existing results (credited by citation) as needed.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- N/A\n\n+ Significance\n\t- Strengths\n\t\t- Improving the robustness of deep RL algorithms continues to be an important area of research. While the scale of the empirical results reported in this work is modest, this seems perfectly acceptable for a paper leaning much more heavily on clean theoretical analysis and insights.\n\t\t- Empirical results demonstrate clear robustness benefits of state entropy regularization and the experiments have been conducted with an ample number of seeds to draw meaningful conclusions.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- As a RL researcher who does not spend much time thinking about robust RL, I can only truly confirm that the theoretical results provided appear to be correct without having much insight into their broader significance and impact on this sub-area of the RL literature. Furthermore, I don't get the impression that any one of the theorems proven in this paper constitute a significant standalone advance; however, it may be the case that the collective insight of the theorems is the valuable theoretical contribution upon which this paper rests. I would be curious to hear the authors' thoughts on the impact of their theoretical contributions and how they perceive the advancement stemming from their contributions to existing work on robust RL.\n\t\t* Minor\n\t\t\t- N/A\n\t\t\t\n\t\t\n+ Final Remarks\n\t- Overall, I think this paper scores nicely on all axes and the issues I've raised can be resolved/clarified quite quickly from the authors via the rebuttal process. My main hestiation is on the perceived impact of this work for the robust RL literature, which is already a difficult and subjective thing to forecast and measure in any paper. With that in mind, I'll recommend acceptance for the paper and look forward to hearing from the authors during the rebuttal process on the few points I've raised.\n\nQUESTIONS:\nI believe my questions for the authors should be apparent in my review above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the theoretical guarantees and practical implications of using state entropy regularization in Reinforcement Learning (RL) to improve robustness. While state entropy regularization has recently been empirically shown to outperform policy entropy measures (i.e. action conditional entropy), its theoretical underpinnings regarding robustness have not received much attention.\n\nThe main contributions are:\n- The paper characterizes robustness succinctly, following prior work, stating that state entropy regularization enhances robustness against spatially correlated perturbations (i.e. a local adversary). \n- The analysis contrasts state entropy with the widely used policy entropy regularization. It shows that policy entropy regularization protects only against local adversaries, whereas state entropy provides robustness against globally informed perturbations (by encouraging state space coverage).\n- The paper derives theoretical guarantees, largely following prior work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- I worked my way through the theoretical derivations once and while I am not an expert RL theory the derivations seem sound (albeit somewhat close to prior work, see comment below).\n- (Policy) entropy regularization is highly used in current SOTA RL algorithms and a paper characterizing it further and explaining differences between different regularization schemes is relevant to the community.\n- The paper includes some insightful findings that I would not have guessed before reading it. E.g. the paper illustrates that the robustness benefits of state entropy are more sensitive to the number of rollouts used for policy evaluation compared to policy entropy. This sensitivity is attributed to the statistical difficulty of accurately estimating state entropy and behavioral aspects in low-rollout regimes. Another surprising find (at least without thinking carefully about it) is that state and action entropy regularization together can be worse than state entropy regularization by itself in some cases.\n- The paper is generally well written and easy to follow despite it's heavily theoretical content.\n- The presented didactic (toy) examples (e.g. Fig.1 and Fig.3) do a good job at explaining the issues with policy entropy regularization and the sensitivity of state entropy regularization.  \n- I like that the paper doesn't try to sell their analysis or experiments of one method but rather tries to give a balanced view of strengths and weaknesses. \n\nWeaknesses:\n- The derivation of the theoretical results seems like a relatively straightforward extension to known results for policy entropy regularization (which is already known to solve a robust RL problem). The relevant works are cited and the extensions cover a case that's clearly relevant to the community. Nonetheless it does dampen the strict \"novelty\" of this part a bit.\n- To the best of my understanding the analysis largely covers the discrete/tabular case and it's applicability to more practical settings is somewhat limited. The authors also acknowledge that in practice some positive effects of policy entropy regularization are observed that aren't fully explainable by the theory.\n- Minor there is some notation that is a bit loose in the main paper (e.g. Dissimilarity of reward functions isn't formally defined, definition of adversary is a bit vague etc.) I think this is ok as it aids the flow of the paper not having too much notation, but I could imagine some readers with a more theoretical background searching for some definitions.\n- State entropy is notoriously hard to estimate and work with (this is also somewhat covered by the theoretical results re. sensitivity to rollouts). So perhaps a strong guarantee on the case where it can be estimated nearly exactly is not as impactful in practice.\n- The experiments are on fairly toyish domains. The MiniGrid example clearly is more didactic than practically relevant. The control domain (pusher) is good but the only reasonably high dimensional domain considered. It would have been nice to see results on some standard domains or some more practically relevant domains.\n\nQUESTIONS:\nPlease discuss the weaknesses outlined above. In particular:\n- It would be good for me to confirm my sense of how the authors think their work fits into the landscape of theoretical results and how practically relevant the derived results actually are.\n- It would be good to understand if the authors have had the chance to experiment with less toy-ish domains.\n- The \"fully kernel robust\" setting doesn't strike me as very practical (and isn't achievable as mentioned in the paper). This isn't a strong weakness in the analysis but I wonder if there are more practical robustness settings one should worry about and analyse?\n\nLIMITATIONS:\nThe paper does a good job at not only selling one method but highlighting limitations throughout and discussing them.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the use of state entropy regularization in robust RL. The problem setup is about robust RL where the environment parameters such as the transition kernel and the reward function are not fully known. In such situations, robust RL agents are trained to adapt to any slight changes in these quantities. The authors discuss why policy entropy regularization is suboptimal. They derive uncertain reward sets for state entropy regularization and also show that state-entropy regularization can produce a lower bound for transition kernel uncertainty as well. The authors test their regularization on Minigrid and Mujoco Pusher environments.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n(1) Changes in reward functions at test time is a very practical problem so coming up with a simple robust RL method that is mathematically motivated makes sense.\n\n(2) The authors derive the reward uncertainty sets for state-entropy regularized RL.\n\n(3) The authors show that state-entropy regularized RL produce a lower bound for transition kernel uncertainty.\n\n(4) The authors also show that using entropy based regularizations alone, transition kernel based uncertainty cannot be completely solved.\n\nWeakness\n\n(1) Discussions on the physical examples of the reward uncertainty set is missing. There should be some discussion about physically what changes in the reward function are allowed.\n\n(2) The method for computing state-visitation distribution (using k-nearest neighbours) is limiting.\n\n(3) Experiments are also limiting. No real baseline is used apart from policy entropy regularization. Are there other robust RL algorithms?\n\n(4) Missing references:\n\n[1]: Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., & Salakhutdinov, R. (2019). Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274.\n\n[2]: Agarwal, S., Durugkar, I., Stone, P., & Zhang, A. (2023). f-policy gradients: A general framework for goal-conditioned rl using f-divergences. Advances in Neural Information Processing Systems, 36, 12100-12123.\n\nQUESTIONS:\n(1) Is there a better way to estimate the visitation probabilities/entropy?\n\n(2) How would you actually maximize reward + visitation entropy? As this reward now is non-markovian (policy dependent)?\n\nLIMITATIONS:\nThe major limitation is the applicability. The authors experiment on very simple domains with a specialized way of estimating state-visitation distribution which questions where this method can be applied. A thorough experimental evaluation with several baselines is also missing.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}