Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors address the problem of exploration in reinforcement learning. They suggest applying the maximum entropy principle over the state space and try to maximize the make the occupancy of the policy as entropic as possible while still optimizing the original goal of maximizing cumulative reward. They claims include an algorithm that does this using a replay buffer and off-policy learning, which improves sampling efficiency, in sparse and dense reward settings, in classic motor control benchmarks.
main_review: First sentence "to deal" -> "dealing" . "Open-ended learning" is not defined, I don't know what this means. What does it mean to "carry out" a model?  Page  " including the curiosity drives" -> "including curiosity drives". Some citations are missing parentheses. The next sentence doesn't make sense. The authors here cite the "E3D" algorithm and "MaxEnt" with which they compare in the empirical section, but for somebody not familiar with these algorithms, the paper should at least briefly describe how these are different from their proposal for these comparisons to make sense. "the dynamics of the environment is Markovian" -> "the dynamics of the environment are Markovian", "The dynamic programming objective is described like:", is that a definition rather? Page 3, what is a "set point"?  "to seek for" -> "to seek"? "so as to match" -> "to match".  The paper uses its own language which is not formal or universal, and quite confusing, and introduces many shorthands on top of shorthands. It uses capital letters and lower case letters arbitrary at times, although I assume the regular rule of capital meaning random variables is the intention. What is a "value sample"? Waht does the "future of s_t" mean? What is K? it is not introduced, I assume the log sum exp/partition/normalization function? In section 2.4 equations do not easily from one from the other, leaving the readers confused at times, and having to do these substitutions themselves.  Why is the $\rho(s)$ appearing in equation 3. Where does it come from? When you say something contains "a guess" about something else, what does that mean in a standard concrete mathematical way? What does $\bar$ over a function mean? What does $\tilde$ over a function mean? At the end of page 4., the author say the next objective function is the one considered by Soft-actor critic, but that is not obvious, SAC works in policy space, not over the space of occupancy maps. What is "a complement term". Using again $q_\pi$ not to mean the action-value function, but a stationary distribution, is very confusing. The paper jumps to the variational perspective, but does not explain this interpretation and why it is helpful or provably useful.  Above section 2.7, "at the light" -> "in light", "the progress of the training" -> "training progress". Section 2.7, "goal pathways as modes of the objective occupancy", what does that mean mathematically, what is the \emph{objective} occupancy. What is "a through" (noun)? What does the "width" at which it is influential on the number of pathways" mean? Next sentence confuses upper case S, A with lower case s,a and reverses them.  What is the "objective credit assignment"? it's not clear what this means. On page 6, there is a claim that exploration in this way of entropy maximization helps the policy converge faster to the optimum, but it is not supported by anything (citation, or results inside the paper), "for the implies to concurrently" -> "for this implies to concurrently". What does "overfitting" mean here? we have results in policy optimization explaining slow convergence due to flat regions of the optimization landscape. "state transition vanish during the optimization...", what vanishes? Page 7, Sutton(1988) is missing citation parentheses. The last sentence of that paragraph, is probably a hypothesis explored in this paper, but do we know for certain this is the case? The exact algorithm is not described in the paper, the authors pointing to the appendix for the most important aspect of the paper, such as how does one compute the log-occupancy? It also mentions the use of a "kernel-based density estimation method" which appears to be fundamental, but this is not explained or discussed. The algorithm implementation or pseudo-code is also not given in the paper. "Results" section, "several benchmark environments", which? and why those? "expressed as average return" -> ""expressed as average return". Page 8, "comparing with the SAC" -> ""comparing with SAC". Page 9, contains an interesting description of the games, showing insight. It ends with "this ripple pattern is not contain in the reward signal", what does that mean? 
summary_of_the_review: The paper addresses the exploration-exploitation trade-off proposing the maximum entropy principle over the state-space as a solution to this problem. Although a promising avenue, many parts of the paper are confusing, using inside jargon, and decreasing the overall clarity. Some parts of the paper are not supported by results, and the main methods used are referred to the appendix.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: # Summary & Contributions
* This paper examines a variational approach to reinforcement learning, leveraging occupancy measures over previously visited and future state-action pairs in order to address the exploration challenge.
* The author propose a variational approximation to the so-called "conditional occupancy" of the current behavior policy denoting the distribution over future state-action pairs. The approximation is used to induce broader state visitation from the behavior policy, coupled with a standard actor-critic algorithm.
* Empirical results show considerable performance gains in simpler continuous control problems and matching performance against baseline methods in more high-dimensional control tasks.
main_review: ### Quality
* #### Weaknesses
    * ##### Major
        * While numerous mathematical statements are made throughout the paper, some are already well-established in the reinforcement learning literature (the chain of statements ending Section 2.2 should all hold with equality and dates back to the successor representation[1]) and others are simply stated as fact without proof, although surrounding text would suggest that the authors are claiming them as technical contributions of this work. While the introduction of variational approximations leads to lower bounds that resemble those shown in the paper, the various notations and distributions introduced make it difficult to confirm their validity right away. The authors should (at a minimum) include proofs for novel technical contributions of this work and further include proofs for supporting results that may help the reader understand the nature of their contributions.
        * In Equations 9 and 10, (my understanding is that) the included log-likelihood ratio is between the variational approximation of the behavior policy visitation and that of the optimal policy. It seems rather likely that this ratio may suffer from numerical errors (namely, division by zero) when the exploratory behavior policy visits a region of the state-action space not covered by the optimal policy. Could the authors comment on why this is not an issue? Or are the authors making some kind of absolute continuity assumption?
        * Ultimately, this paper is deeply concerned with addressing the challenge of exploration. Can the authors comment on why this formulation under probabilistic inference is reasonable given recent work which shows how the approximations needed for these methods to be applied in practice fail to pass even basic sanity checks? [2]
	* ##### Minor
		* 

### Clarity
* #### Strengths
    * The clearest element in the paper is Algorithm 1, which at least conveys the generic structure of what the authors are trying to achieve. Unfortunately, everything leading up to Section 3 is incredibly unclear stemming from a variety of sources including non-standard mathematical notation, undefined jargon/terminology, and (to a lesser extent) grammatical errors. The authors have wasted a page describing details of widely-known Mujoco environments that could instead be allocated to adding clarity to Sections 2 and 3, which seem to articulate the bulk of the contribution. 
* #### Weaknesses
    * ##### Major
        * The mathematical notation used throughout the paper is terribly difficult to parse and inconsistent with the broader reinforcement-learning literature. As a concrete example, Section 2.3 contains undefined notation (\mathcal{T}\pi(s)), ambiguous notation (s... notation, s+ appears to denote a state and yet is later said to denote a sequence of future states but then used as a normal state in Equation 3), and nonsensical notation (the second paragraph of Section 2.5 has Q(s,a) \sim \frac{r(s+,a+)}{(1-\gamma)}, which is entirely vacuous without being defined beforehand). The authors seem to jump back and forth between states indexed by time vs. not. 
        * The authors make a bad habit of introducing assumptions unnecessarily. In Section 2.4, if the first assumption on Boltzmann policies is made, then why is the second assumption on the existence of K(s) needed as well? The statement should hold under the first assumption and K(s) can be written out explicitly. Why is the first assumption of Section 2.5 needed as if it re-defines the optimal policy? Unless the policy class over which the argmax is taken is a subset of all policies, this is the definition of the optimal policy for a MDP. 
        * The authors introduce their own terminology without any formal definitions or visualizations to assist the reader in understanding what the terms are meant to convey. As a result, the first paragraph of Section 2.7 reads vacuously without knowing what is meant by "pathways", "conducts", "adverse", through", or "width."
        * The paper has several grammatical errors throughout. While a small handful of these would not constitute a major weakness (and, in fact, could be easily enumerated and corrected), there are enough of them to impact readability of the paper.
	* ##### Minor
		* The use of the phrase "probability matching" in this paper is misleading since it is widely associated with Thompson sampling. Distribution matching would be a more accurate phrase for what the author is trying to convey.
		
 
### Originality
* #### Weaknesses
    * ##### Major
        * The paper lacks a section dedicated to providing readers with an overview of related prior work. Consequently, it is difficult to precisely identify how the authors' variational formulation extends or expands beyond some of the prior work mentioned in brief at the top of page 2.  
	* ##### Minor
		* 

### Significance
* #### Weaknesses
    * ##### Major
        * The authors haven't included details of how many random seeds are used to generate the results in Figure 1; consequently, I'm skeptical of their significance and reproducibility. That said, just examining the average return column (which is the standard metric), it seems that the proposed approach only leads to meaningful performance improvements in simpler control problems, while results in Swimmer and Humanoid are only on par with SAC.
        * Simpler experiments in domains where things can be concisely and cleanly visualized would go a long way towards driving home the central claim that this approach facilitates improved exploration in practice.
	* ##### Minor
		* 

# References
1. Dayan, Peter. "Improving generalization for temporal difference learning: The successor representation." Neural Computation 5, no. 4 (1993): 613-624.
2. O'Donoghue, Brendan, Ian Osband, and Catalin Ionescu. "Making Sense of Reinforcement Learning and Probabilistic Inference." In International Conference on Learning Representations. 2019.

summary_of_the_review: # Final Remarks
* As evidenced by my main review, I have struggled to identify strengths of this paper, in large part because the precise nature of the technical contribution and how it improves upon prior work is unclear to me. 
* Still, the empirical results are at the level of a preliminary workshop paper and more experimentation seems warranted across a broader set of environments to confirm the utility of the proposed approach. 
* The clarity issues raised above suggest that the paper requires substantial revision in order to more clearly convey the contributions and algorithms to readers.
* For now, my score is quite low though I would be happy to raise it in light of suitable clarifications from the authors.

======= Post Rebuttal =======

I thank the authors for their response but the lack of clarity around the approach itself and originality as well as issues with the writing style point to substantial revisions that are needed before the submission is ready for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors introduce a RL method for optimizing entropy over states, as opposed to actions. The algorithm is tested on three MuJoCo domains and one classic control environment. 
main_review: Overall, I found the organization and clarity of the paper to be poor. Consequently, it wasn't clear to me what the theoretical contributions of the paper were. As far as I could tell, 2.1-2.6 were background and a refactoring of notation/existing ideas, and it wasn't clear to me how novel 2.7 is compared to similar methods in the literature. I provide some more detailed comments on how the clarity can be improved at the bottom.

The empirical evaluation is also lacking. In short:
- Missing baselines. The authors don't compare against other methods which aim to maximize state entropy.
- Poor reproducibility. Missing # of trials/seeds. Visually it seems there is some shading which suggests multiple seeds, but this information is absent. Weirdly, this error term disappears over average reward and cumulative rewards. There's per environment hyperparameters and the graphs seem to finish at arbitrary time steps, suggesting these results have been overly tuned to outperform the baselines. Code is not provided (although promised), and the method is not reproducible from only the details given in the paper. 
- Missing analysis. Does the method improve state entropy or add to exploration? There's also no ablation study or any addition insight about the empirical properties of the algorithm.

I've scored my confidence as 4 because I am confident in my score, given the current state of the paper, however, I am not confident I have fully understood all the details of the paper. 

Suggestions for clarity:
- $q_\pi$ notation is never defined. $K$ in section 2.4 is never defined. 
- Section 2 should be split into background and a section which defines contributions. I'm not sure Section 2 needs so many subsections, and there is no explanation besides from the title, of what each subsection is meant to demonstrate.
- Too many recursive definitions. For example Eqn (3) is used solely to define $\beta \bar{R(s^+,a^+)}$, which is then replaced with $\tilde{log} p(s,a)$.
- It would be easier if any unconventional notation was defined in one place rather than scattered throughout section 2. 
- ~ is used weirdly, for example Q(s,a) is "sampled" from the reward function above equation 4. Presumably this is meant to replace expectation but should be defined. 
- Inconsistent notation. S,A are often used but are never defined. Are these meant to be the set of all possible state/actions, as defined by $\mathcal{S}$ and $\mathcal{A}$ in 2.1? 
- Problem statement really needs to be split into at least 2, if not more, paragraphs.
- The full algorithm description should be moved from the appendix into section 3. The current algorithm description is also unclear in details. For example, "update the critic over all losses" is unclear which losses are being referred to. 

Minor comments on clarity
- Given average return is undiscounted, I'm not sure average reward is an interesting metric to look at. 
- Ordering environments by difficulty is subjective. Describing the environments in detail can be delayed to the appendix. 
- Typo in 2.1 [0,1[ (should also be [0,1) if we're considering a discounted objective). 
- It would be easier to follow if $S^+$ notation was introduced above the equation at the end of page 3.

summary_of_the_review: Poor clarity makes the contributions hard to follow, along with weak empirical analysis puts the paper below the bar for acceptance.
review_title: null

Meta-review: While the main idea of the paper (using a Max-Ent objective on the states of an MDP) was considered interesting, all reviewers raised the problem of clarity of the paper which needs to be drastically improved. While the writing could be improved by the revsion, these concerns could also not be fully alleviated by the rebuttal of the authors. The reviewers agreed that the paper needs rewritting in order to clarify the contribution before the paper can be published.
Decision: Reject