Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses limitation in max-entropy RL frameworks. These frameworks usually add a regularization term to the reward function to increase policy entropy to encourage exploration, but the effect can be limited in states where the Q-function dominates the expression.
The authors propose a new objective that better encourages RL agents to visit rare states with low entropy, and show empirically that their method is able to outperform the baselines used for comparison.
main_review: I very much enjoyed reading this paper. The writing was clear, there was a nice flow of arguments, there was some interesting insights on commonly used MaxEnt RL frameworks, and a concise change to address the limitations found.

I think RL practicioners might find this paper useful, specially when dealing with sparse reward settings.

I only have a few suggestions that I think might help the reader.

- Line 98: why is the first term "the sum of future entropy"? It's the sum of Q, so wouldn it be more like the sum of "entropy-regularized reward"?

line 120-121: From the plot I do see that SAC behaves very differently than Uniform, however I don't know if you could definitely say that SAC visits fewer states than uniform, as the variance is huge and sometimes it visits a lot more states than uniform.

- Eq. 7 defines the new Q function for MME but the difference to Eq. 2 is very subtle, it took me a bit of scrolling back and forth to find the difference. It would be useful for the reader if as the new objective is derived either the previous one is re-stated on the same page, or the differences being explained as they are introduced.

- why is it called "reversed" soft Q-function?

- I'm a bit confused with the notation alpha_{Q}. Does this mean that the value of alpha_{Q} depends on the current Q value?

- Just for clarification, Q_{R,R} and Q_{R,E}, are two separate functions maintained by two separate approximators (neural nets), right?


- Typo line 47: (approximiate) -> (approximate)
limitations_and_societal_impact: Not much has been said about limitations, the authors could expand a bit on this point.
No negative societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a conjecture that the agent should visit the state with low entropy and then do the exploration of this low entropy state with the maximum-entropy principle. To prove this conjecture, the author does a simple experiment on the 4-room maze and illustrates the empirical entropy of the action at different states. There is an interesting observation that the rarely visited states have low entropy maybe due to the function approximation error or off-policy learning explained by the author. Given this observation, the author proposes an algorithm which penalizes the reward with negative entropy （eq7) . Based on this modified Q function, the agent does the policy improvement as the original soft-actor-critic algorithm. The empirical result shows that the proposed algorithm outperforms the baselines such as SAC and many others.
main_review: pros:
1. The authors provide a simple experiment to demonstrate their observations which leads to the easy understanding of the core idea.
2. The empirical performance of the proposed algorithm is strong.

cons:

1. The contribution of this paper is somewhat incremental, since the negative entropy can be thought a special case of the intrinsic reward.
2. In some experiment it seems true that the rarely visited state have low entropy. The author explained that in some heuristic way in page 5 and 6. Does this conclusion hold for most RL problems? I hope the author can provide more rigorous analysis, since it is the core of this paper.   
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors describe a positive feedback loop, that arises in SAC-based implementation of the maximum entropy reinforcement learning and hinders efficient exploration. Q values of frequently visited states are updated more, because these states are stored more into the replay buffer. The policy update increases the entropy of the visited states, i.e. if a high entropy is rewarded in the policy the frequently visited states will be selected even more. In the paper, Max-Min Entropy Rl is proposed that is supposed to break the unwanted feedback loop. It considerably improved the empirical performance in common Maze tasks and Mujoco tasks.
main_review: Originality:
* as far as I know the described limitation was not studied before. The proposed method is new, too. 

Quality:
* The paper does not contain a theoretical analysis, but the experiments are analyzed in quite a detail. I like that the authors plotted the magnitude of the Q-values and the entropy. 

Clarity:
* In the first sentence of the abstract you write "the limitation of ...", but the abstract never says what this limitation is. If you can summarizes the positive feedback loop in one sentence, I would add that. Otherwise I would change "the limitation" to "a limitation".
* the described problem is rather a problem with the SAC implementation of the maximum entropy framework than with the maximum entropy framework itself. The problem is that the sampling based approximation in SAC results in a positive feedback loop, that keeps SAC from converging to the optimal solution of the maximum entropy objective $J_{MaxEnt}$. While this is described later on, reading the abstract and the introduction I misunderstood it as a limitation of the maximum entropy framework itself. 
* The paper is well written in general. Just one suggestion for improvement of the reading flow: The caption of Section 4 mentions that Section 4 considers pure exploration only, but the text itself does not mention it. This made me stumble across the sentence starting in line 97, that only holds for pure exploration. 

Significance:
* The method is widely applicable. However, I am wondering if the situation would be different in MDPs with stochastic transitions $P$. The described example and the experiments are all with deterministic transitions, right? But even if not, I would still consider it significant. 

Further comments:
* Is it easy to see what $ J_{MaxMin}$ would look like?
* In Section 5.1, in line 164, it says: "For Fig. 2(b), every 1000 time steps, we sampled ...", but the x-axis label in Figure 2(b) is "Time Steps (1e5)". 

limitations_and_societal_impact: Societal Impact (last two sentences of the paper): I perceive the formulations "only a risk" and "not very relevant" slightly unlucky in the context of ethics, but contentwise I agree with the authors that the paper does not pose a risk beyond the general dual use problematic in RL.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates the undesired behavior of soft actor-critic (SAC) algorithm that implementing maximum entropy reinforcement learning (RL). 

One of the main contributions of this paper is the empirical investigation of the undesired behavior of SAC. By running SAC with function approximation on no-reward environment, the authors illustrate that the policy obtained by SAC failed to maximize the entropy, while maximizing the entropy is the optimal situation in no-reward environment and the theoretical study of SAC in finite MDP setups guarantees its optimal convergence. The authors pointed out a possible defect in SAC with function approximation: Q value of already visited states tend to be greater than the other states because the policy update increases the entropy of the visited states. 

The second contribution is the proposal of a solution to the above mentioned defect. The authors propose Max-Min Entropy RL (MME). To improve the exploration, the soft Q-function has been revised. Instead of adding the entropy term to the Q-function, the revised soft Q-function subtracts the entropy, if I borrow the authors statement, to enhance the chance to visit rare states with low entropy. This is, however, counter intuitive to me as I will describe in the main review part.

The proposed approach has been tested on sparse mujoco tasks and delayed mujoco tasks, where the environments have sparse-reward shapes, as well as on some standard mujoco environments with dense rewards. The algorithm has been compared to different SOTA approaches. Promising performance has been observed.


main_review: The first contribution mentioned above is a very nice finding of a possible defect of SAC. It may open a new direction of future works. One small comment: This behavior of over-estimation of Q-value for already visited states may be related to the over-estimation bias of Q-value reported in TD3. The authors might want to comment on the similarity between them.

My concern is in the second contribution, i.e., the revised soft Q-function (7). This definition follows the authors idea of "requiring the policy to visit rare states with lower entropy more frequently." However, the rare states do not necessarily have lower entropy. Based on the authors empirical investigation on the defect of SAC, I understand that SAC tends to over-estimate the entropy of already visited states. Therefore, unvisited states have relatively smaller estimated value of the entropy. However, this is the matter of estimation, and the entropy itself is not necessarily lower for unvisited states. If we run the approach sufficiently long, Q-function is expected to tend to the right value, which is the right entropy value in case of no-reward situation. Therefore, I couldn't understand why this is the right modification to do. I appreciate if the authors give more justification of (7). 

Although I am uncertain about the validness of the proposed modification, the empirical results presented in this paper are promising. One minor comment: What happens if the proposed approach is applied to swimmer environment? It is known to be deceptive and more exploration tends to lead to better performance, as is often shown in evolutionary RL and related approaches.

limitations_and_societal_impact: The limitation is the lack of theoretical guarantee, which the original SAC has. This is stated in Appendix. However, since the statement of the limitation is a very important part of a research paper that may promote a future investigation, I would like to suggest to put it in the main text.
review_title: null

Meta-review: This paper proposes an interesting and novel view on MaxEnt RL: It observes that  SAC may suffer from a feedback loop since the Q values of already visited states tend to be greater than that of other states because the policy update increase the entropy. This in turn encourages the policy to visit these states even more. This defect is confirmed in a no-reward environment and a method to address this is presented with encouraging experimental results. While there could have been more rigorous investigation into the claim "rarely visited state have low entropy", the overall contribution of the paper is appreciated by most of the reviewers and the interesting take on MaxEnt RL warrants acceptance.
Decision: Accept (Poster)