Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduced a very interesting idea to facilitate exploration in goal-conditioned reinforcement learning. The key idea is to learn a generative model of goal distribution to match the weighted empirical distribution, where the rare states receive larger weights. This encourages the model to generate more diverse and novel goals for goal-conditioned RL policies to reach.

Pros:
The Skew-Fit exploration technique is independent of the goal-conditioned reinforcement learning algorithm and can be plugged in with any goal-conditioned methods. The experiments offer a comparison to several prior exploration techniques and demonstrate a clear advantage of the proposed Skew-Fit method. It is evaluated in a variety of continuous control tasks in simulation and a door opening task on a real robot. A formal analysis of the algorithm is provided under certain assumptions.

Cons:
The weakest part of this work is the task setup. The method has only been evaluated on simplistic short-horizon control tasks. It’d be interesting to see how this method is applied to longer-horizon multi-stage control tasks, where exploration is a more severe challenge. It is especially when the agent has no access to task reward and only explores the environment to maximize state coverage. It is unclear to me how many constraints are enforced in the task design in order for the robot to actually complete the full tasks through such exploration.

I would also like to see how Skew-Fit works with different goal-conditioned RL algorithms, and how the performances of the RL policy in reaching the goals would affect the effectiveness of this method in exploring a larger set of states.

Section E: it seems that there’s a logic jump before the conclusion “goal-conditioned RL methods effectively minimize H(G|S)”. More elaboration on this point is necessary.

Minor:
Appendix has several broken references.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper introduces SKEW-FIT, an exploration approach that maximizes the entropy of a distribution of goals such that the agent maximizes state coverage. 

The paper is well-written and provides an interesting combination of reinforcement learning with imagined goals (RIG) and entropy maximization. The approach is well motivated and simulations are performed on several simulated and real robotics tasks.

Some elements were unclear to me:
- "We also assume that the entropy of the resulting state distribution H(p(S | pφ)) is no less than the entropy of the goal distribution H(pφ(S)). Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are." How do you ensure this in practice?
- In the second paragraph of 2.2, it is written "Note that this assumption does not require that the entropy of p(S | pφ) is strictly larger than the entropy of the goal distribution, pφ." Could you please clarify?


The experiments are interesting, yet some interpretations might be too strong (see below):
- In the first experiment, "Does Skew-Fit Maximize Entropy?", it is empirically illustrated that the method does result in a high-entropy state exploration. However, it is only compared to one very naive way of exploring and it is not discussed whether other techniques also achieve the same entropy maximization. The last sentences seems to imply that only this technique ends up optimizing the entropy of the state coverage, while I believe that the claim (given the experiment) should only be about the fact it does so faster.
- On the comments of Figure 6, the paper mentions that "The other methods only rely on the randomness of the initial policy to occasionally pick up the object, resulting in a near-constant rate of object lifts." I'm unsure about the interpretation of this sentence given Figure 6 because other methods do not seem to fail entirely when given enough time.
- In the experiment "Real-World Vision-Based Robotic Manipulation", It is written that "a near-perfect success rate [is reached] after five and a half hours of interaction time", while on the plot it is written 60% cumulative success after 5.5 hours and it is thus not clear where this "5.5 hours" comes from.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Summary : 

The paper proposes an exploratory objective that can maximize state coverage in RL. They show that a formal objective for maximizing state coverage is equivalent to maximizing the entropy of a goal distribution. The core idea is to propose a method to maximize entropy of a goal distribution, or a state distribution since goals are full states. They show that the proposed method to maximize the state or goal distribution can lead to diverse exploration behaviour sufficient for solving complex image based manipulation tasks. 


Comments and Questions : 

	- The core idea is to maximize the entropy of the state visitation frequency H(s). It is not clear from the paper whether the authors talk about the normalized discounted weighting of states (a distribution) or the stationary distribution? The entropy of the state visitation distribution only deals with valid states - but I am not sure what it means to maximize the entropy of this term exactly in terms of exploration, since it is neither the discounted weighting of states or the stationary distribution for an infinite horizon task? 
	- The authors do mention that maximizing the entropy of H(s) is not sufficient - so instead suggests for maxmizing entropy of H(s|g). But why is this even sufficient for exploration - if I do not consider new tasks at test time but only the training task? How is this a sufficient exploration objective? Furthermore, since it is the conditional entropy given goal states, the fundamental idea of this is not clear from the paper. 
	- Overall, I am not convinced that an objective based on H(s|g) is equivalent to an maximizing H(s), and why is this even a good objective for exploration? The meaning of H(s) to me is a bit vague from the text (due to reasons above) and therefore H(s|g) does not convince to be a good exploration objective either?
	- The paper then talks about the MI(S;G) to be maximized for exploration - what does this MI formally mean? I understand the breakdown from equation 1, but why is this a sufficient exploration objective? There are multiple ideas introduced at the same time - the MI(s;g) and talking about test time and training time exploration - but the idea itself is not convincing for a sufficient exploration objective. In light of this, I am not sure whether the core idea of the paper is convincing enough to me. 
	- I think the paper needs more theoretical insights and details to show why this form of objective based on the MI(s;g) is good enough for exploration. Theoretically, there are a lot of details missing from the paper, and the paper simply proposes the idea of MI(s;g) and talks about formal or computationally tractable ways of computing this term. While the proposed solutuon to compute MI(s;g) seems reasonable, I don't think there is enough contribution or details as to why is maximizing H(s) good for exploration in the first place.
	- Experimentally, few tasks are proposed comparing skew-fit with other baselines like HER and AutoGoal GAN - but the differences in all the results seem negligible (example : Figure 5). 
	- I am not sure why the discussion of goal conditioned policies is introduced rightaway. To me, a more convincing approach would have been to first discuss why H(s) and the entropy of this is good for exploration (discounted weighting or stationary state distribution and considering episodic and  infinite horizon tasks). If H(s) is indeed a difficult or not sufficient term to maximize the entropy for, then it might make sense to introduce goal conditioned policies? Following then, it might be convincing to discuss why goal conditioned policies are indeed required, and then tractable ways of computing MI(s;g). 
	- Experimentally, I think the paper needs significantly more work - especially considering hard exploration tasks (it might be simple setups too like mazes to begin with), and then to propose a set of new experimental results, without jumping directly to image based tasks as discussed here and then comparing to all the goal conditioned policy baselines. 

Overall, I would recommend to reject this paper, as I am not convinced by the proposed solution, and there are lot of theoretical details missing from the paper. It skips a lot of theoretical insights required to propose a new exploration based objective, and the paper proposes a very specific solution for a set a very specific set of experimental setups. 




review_title: Official Blind Review #1

Meta-review: This paper tackles the problem of exploration in RL. In order to maximize coverage of the state space, the authors introduce an approach where the agent attempts to reach some self-set goals. The empirically show that agents using this method uniformly visit all valid states under certain conditions. They also show that these agents are able to learn behaviours without providing a manually-defined reward function.

The drawback of this work is the combined lack of theoretical justification and limited (marginal) algorithmic novelty given other existing goal-directed techniques. Although they highlight the performance of the proposed approach, the current experiments do not convey a good enough understanding of why this approach works where other existing goal-directed techniques do not, which would be expected from a purely empirical paper. This dampers the contribution, hence I recommend to reject this paper.
Decision: Reject