Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Summary
This paper proposes a new solution to the problem of learning goal-conditioned policies without hand-crafted rewards. Prior work in this domain learn an embedding space to compute reward between current state and goal. In contrast, this paper utilizes unsupervised skill discovery from [1] to obtain a discriminator that identifies which states belong to a particular skill. Then, the final state of a given skill's execution is used as a goal input to a goal-conditioned policy, which is rewarded if it generates states that the discriminator identifies with this skill. The paper aims to validate the benefit of such a reward over other embedding-distance based reward functions on a variety of environments.
​
​
----------
​
# Strengths
- The paper solves an important problem of unsupervised learning of goal-conditioned policies. To the best of my knowledge, the proposed way of training a goal-conditioned policy with skill-based discriminator is novel.
- The training framework combines unsupervised exploration (skill-discovery) and learning of goal-conditioned policy elegantly through the discriminator. I find the insight very interesting that the learned skill's end-point can be treated as a target goal and the skill discriminator can be used as a reward to identify the states that are going towards the goal (since these states fall on the skill's induced trajectory).
- The experiments are conducted on a large set of environments, which is good to justify the claims made by the paper.
- I appreciate the video results and the visualizations provided on the project website.
​
​
----------
​
# Weaknesses
- The paper is missing the ablation study for the primary contributions. The paper argues that approaches taking reward as distance in embedding space can limit the repertoires of behaviors. One such example of unsupervised reward function is DISCERN [2]. To validate this point, and to make a fair comparison with DISCERN, the states used to train DISCERN's reward predictor should be generated from the same abstract-level policy (from [1]) as used by this paper. The current DISCERN baseline seems to be training the reward function on random exploration in the environment. I think a stronger version of this baseline is DIAYN+DISCERN.
​
- The section about disentanglement using information bottleneck seems orthogonal to the main contributions of the paper - about an unsupervised method to train goal-conditioned policies. While the information bottleneck can always be used to improve generalization, it is unclear why its use is emphasized. If I understand this correctly, then the ablation study in Section 5 can be moved to Appendix as it seems an extra detail. It is rather more important to ablate the role of the intrinsic reward function proposed in this paper (as discussed in the above point).
​
- The writing is understandable, but can be improved. It is not always clear what the main contributions of this paper are.
(a) The "abstract-level policy" is precisely the unsupervised skill-discovery framework from [1], and it should be stated as such.
(b) The paper emphasizes the view that the exploration and goal-conditioned policies share the same reward function, but a more relevant viewpoint is that the proposed training framework enables the discriminator learned for exploration to also be useful as a reward for the goal-conditioned policy.
(c) Some parts of writing are incoherent and seem out-of-place at first. For instance, paragraph 3 of introduction should be written in a way that contrasts this paper's contributions to the prior work.
(d) One clarification question I have is about where the goals for evaluation (such as in videos) come from? Are these goals just what were observed in training or are these manually-crafted goal-locations for evalution? It would be nice if a principled evaluation procedure is used, where a dataset of testing goals (maybe procedurally generated) are used to report the performance of the goal-conditioned policy.
​
​
----------
​
# Reason for decision
My primary concern is the missing baseline where alternate unsupervised rewards are combined with better exploration methods (just like this paper uses [1]). In my understanding, the primary contribution is an alternate reward function, which does not depend on explicit embedding distances. To validate this contribution, this baselines is important. I would be happy to increase my score if this concern is addressed.
​
​
----------
​
# Suggestions for improvement
- It was a little difficult to understand the high-level intuition of the method. It can be more succinctly summarized in the text and/or Figure 1 caption, such as: "Skill w is a way to reach the goal g, and the goal-conditioned policy with goal g is rewarded to imitate the trajectory induced by w."
​
​
----------
​
# References
[1] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.
[2] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through parametric discriminative rewards. arXiv preprint arXiv:1811.11359, 2018.
review_title: Weak evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The paper proposes a novel method for learning goal-conditioned policy with images/text goals. 

Quality: The overall quality of the paper is good. 
    Strong side: Extensive evaluation on various environments and tasks showcases the advantage and generalizability of the method. The authors uses figures and algorithm boxes to make their method very clear.
    Weak side: Several clarification on the motivation and details of the method is needed. See below.

Clarity: (1) The main argument is not strong: 1. Image/text goals is not intractable for current goal-conditioned policies [1] 2. What do you mean by ‘extrinsic’ reward? If you mean task-specific reward, many methods learn goal-conditioned policy without extrinsic reward, like hindsight experience replay.
(2) Therefore, I guess the authors wanted to claim they use ‘intrinsic’ rewards, which is a mutual-information-based reward. Now I have two questions for Section 3.2: 1. Why do we use this loss function Eq. (1)? It comes out of nowhere without intuition. (2) Why do you decompose the optimization into ($\mu, \Phi$) and $\theta$? I feel like decomposing into $\Phi$ and ($\mu, \theta$) is more reasonable in that one optimizes the reward first then policies.
Another question on algorithm box: In step two, $g_t$ is changing with time steps. This is kind of strange because in typical goal-conditioned policy learning, in one epoch, people use a fixed goal to train their policy, then change the goal in the next epoch. What is the fundamental difference?
(3) The whole disentanglement thing needs more clarification. I don’t understand the reason why you disentangle your policy. In computer vision, disentanglement has clear physical meanings like disentangling shape and color, but here I don’t have such intuition. In experiments, the effect of disentanglement is only demonstrated in a simple 2D-task, which seems not enough. 
(4) Details: 1. How is p(w) defined? It seems super important, but I was not able to find its details in the paper. Maybe I overlooked something.

Originality: As far as I know, the method is new.

Significance: Learning goal-conditioned policy with high-dimensional goals is an important problem. I think if the authors could clarify the above questions, the solid experiments will make the paper a good contribution. However, I don’t think its current version passes the bar of ICLR.

Reference: [1] https://papers.nips.cc/paper/9623-planning-with-goal-conditioned-policies.pdf
review_title: Like the Extensive Experiments, But The Paper Requires Plenty of Clarification

final_score: rating: 7: Good paper, accept
review_content: Summary
--
This paper proposes an unsupervised learning objective for learning perceptual goal-conditioned policies. The goal is to enable unsupervised discovery of high-level behaviors in tandem with a perceptual-goal conditioned policy that can achieve these behaviors. The learning proceeds by training one policy to exhibit diverse behaviors; the states induced by these behaviors are then rendered and used as target goal states for a separate goal-conditioned policy.

The learning objective is to maximize a lower-bound on sum of two terms -- (1) the mutual information between the behavior variable and states with a behavior-conditioned policy; states from this behavior-conditioned policy are transformed into perceptual goals that serve as input to a separate policy that forms (2) the mutual information between perceptual goals and the states it induces.

The learning algorithm operates via alternating optimization, in which the skill-conditioned exploration policy is learned jointly with a skill 'discriminator' (inference network), and then the perceptual-goal conditioned policy is learned using the discriminator as a reward signal, which essentially estimates the extent to which a robot is achieving a skill given a perceptual goal along the skill-policy trajectory.

A glut of experiments are used to investigate whether the method learns a meaningful skill reward function, whether it can achieve goals in various environments, how the method compares to related methods, and whether the specific 'disentanglement' inductive bias for constructing the policy is useful. The experiments demonstrate favorable performance over existing methods for these tasks.

Quality
--
The goal, method, and experiments are likely high quality, given my understanding. However, there are significant gaps in clarity that reduce my certainty in this assessment, and relatedly, there is some important missing discussion on the specific differences between the proposed method and prior work.

Clarity
--
- The learning algorithm is ambiguous: which goal(s), specifically, are used in the second learning stage? The current algorithm reused the 't' time index, which makes this unclear. I suspect it's just the goal corresponding to the last timestep from the first stage of the algorithm, but I'm not sure.
- The state representation for the archery task is unclear
- The 'fast imitation' procedure is unclear. The paper says the goals are the rendered states induced by the abstract policy, so how do the expert demonstrations get factored in? Is the learning algorithm different (is the abstract policy trained with the expert demonstrations somehow)? This is a significant ambiguity that makes it difficult to interpret the results of the imitation experiments. It's hard to guess at, because the conceptual difference between the standard imitation learning problem and a prototypical unsupervised "RL" algorithm is quite large.
- The presence of two environments in Figure 1 is quite confusing. It's hard to tell from the context of the figure alone whether the policies are operating simultaneously in the same environment, simultaneously in separate equivalent environments, or at separate times in equivalent environments.
- The \tilde notation and relationship between the \tilde an not-\tilde variables needs to be discussed in 3.1

Originality
--
The learning objective seems to be quite similar to the DIAYN and DISCERN objectives, and the task of learning to condition on general perceptual goal in unsupervised setting is shared by RIG and DISCERN. Discussion of specific algorithmic and assumption differences between the proposed method and these approaches is quite necessary, but unfortunately missing. 

Significance
--
In absence of knowing more about the specific differences between the proposed work and related works and the imitation learning experiments, the only thing that is clear is that the method seems relatively performant on an existing well-motivated task across a large range of settings, and compares favorably to existing methods on this task.

Other points
--
Derivation is needed (e.g. in appendix) to show that (3) is a lower bound of the last line of (2). I don't think this aspect of the derivation follows from Jensen's inequality, as stated in the paper.
- In the intro (para 3) "an unsupervised method in RL" is unclear to an RL reader unfamiliar with "unsupervised RL". I think this paper should explicitly define "unsupervised RL" to mean "RL with an intrinsic reward function" or something equivalent, and "intrinsic reward function" to mean a learning objective that can be applied in place of an alternative reward function across different MDPs.
- Intro para 3 "maximize an information theoretic objective" is vague, because all learning objectives for probabilistic models are technically "information theoretic" -- all probabilistic models have a relationship to information.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method for combining intrinsic motivation on a state space with goal-conditioned reinforcement learning (GCRL), where goals are defined in some “perceptual space,” such as text or images, which describe the current state. The authors assume access to a renderer that maps states to perceptual goals, but do not assume that the renderer is differentiable. The authors propose to train an intrinsically motivated latent-conditioned policy, using similar techniques as past work in which a policy maximizes the mutual information between a latent variable and the current state. The goal-conditioned policy is then trained to effectively imitate the latent-conditioned policy by maximizing the same reward as the latent-conditioned policy, conditioned on only the rendered version of the final state reached by the latent-conditioned policy. The authors demonstrate that the overall method outperforms past GCRL methods on a variety of tasks (Atari, MuJoCo manipulation and locomotion, and toy tasks).

The idea of using intrinsic motivation rewards to train a GCRL policy is an interesting and novel idea, and the experiments indicate that this is a promising approach. The writing is a bit verbose and there are some details that rushed through, but overall I find the idea novel and the results compelling.

I would recommend that the authors clarify that this method assumes access to a renderer. Currently, the writing does not make it clear that the comparison to RIG and DISCERN is not an apples-to-apples comparison: RIG and DISCERN do not assume access to the ground-truth state, but rather operate directly in the perceptual space. However, I think this comparison is reasonable since I do not know of any other method that makes the same assumption as this paper.

The authors state that, “the goals for πθ in mujoco tasks are the rendered trajectories induced by π...” In that case, how did the authors condition a policy on entire trajectories? This seems like an important detail, and it’s also unclear how the baselines could have been implemented in those cases.

The analysis of Figure 6 experiment seems to overstate the result. While it is true that, “during training the agent is required to imitate a large number of simple behaviors” it does not seem to be the case that it “has never seen such complex goals.” Unless I am mistaken, the policy at test time is not conditioned on a single complex goal at test time. Instead it’s simply conditioned sequentially on simple tasks, which is effectively the same as during training but just done sequentially.

--- Post Rebuttal ---

I've read the author response and do not intend to increase my score. Thank you for answering my questions. It would be good to clarify in the main paper that conditioning on the trajectory is implemented in the way described in the author response.
review_title: Novel combination of intrinsic motivation and goal-conditioned RL, assuming access to a "goal renderer"

Meta-review: This work extends previous work on unsupervised learning of goal-conditioned policies: an abstract skill policy, which drives exploration of the state space, is used to propose goals as well as derive rewards for a goal conditioned policy. 

Reviewers agreed the approach was novel and interesting. All reviewers raised significant concerns about clarity and/or lack of details, as well as a lack of comparison to DIAYN/DISCERN, though these points were adequately addressed in revisions. One remaining issue raised by two reviewers are that the content related to the information bottleneck/disentangled representation learning seems out of place and ill-justified. Detailed discussion of this aspect of the work has been relegated to the appendix.

This is an important problem and a growing area of study, and while the submission has potential, improvements needed are not minor, and given the short process, we can only accept papers as is, rather than expecting certain changes. We urge the authors to further improve the focus of the work and perhaps plan to investigate the role and importance of disentanglement with IB in this setting in follow up work wherein they have the space to properly do justice to the topic in its own right.
Decision: Reject