Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method to learn goal-conditioned policies by combining ideas from goal-conditioned RL and model-based exploration. It operates by training an exploration policy within the model to maximize model disagreement (as proposed in previous works), as well as a goal-conditioned policy to reach states in the replay buffer. These are then both deployed to grow the replay buffer, and the process is repeated. The approach is evaluated on a newly-defined set of benchmark tasks based on the Deepmind Control suite, as well as simulated robotic manipulation. They report better performance compared to previous methods such as Skew-Fit, DIAYN and two others. They also investigate two different choices of distance functions for the goal-conditioned RL component, and show that the optimal one depends on the setting. 
main_review: Update after the rebuttal: I thank the authors for having added an extensive comparison to prior work on previous environments. Since this was in my view the main limitation of this work, I have raise my score from a 4 to a 6. 


============================================================================

Methods-wise, this is essentially a combination of model-based exploration methods which use the disagreement between dynamics models to incentivize exploration, and goal-conditioned RL (which is performed inside the dynamics model here). Incidentally, a piece of related work which should be mentioned is the following https://arxiv.org/pdf/1911.00617.pdf which proposed model-based exploration with multiple dynamics models and also provided some theoretical results for this class of algorithm.

The novelty of this paper is somewhat limited, in that it is a fairly straightforward combination of existing techniques. However I am not aware of this specific combination being investigated before. 

A big issue with the experimental setup is that the tasks are new and defined in this paper. It’s ok to define new tasks to see where existing methods fail, but it’s also important to know how the proposed method performs on tasks where existing methods are known to succeed. Also, checking that the baselines work in settings where they are known to succeed confirms the implementation, hyperparameter tuning etc is correct. Please include results for tasks from the papers where the baseline methods were proposed. 

For example:
- the simple grid-world environment from the Skew-Fit paper and/or the 2d navigation one from DIAYN. Both of those methods should work there. How do they compare to the proposed method in terms of sample complexity?
- DIAYN uses the learned policy to initialize a policy optimizing task reward on some Mujoco tasks. Please include a comparison of the proposed method with the same setup. 
- etc

It’s also a bit unsatisfying that you need different distance metrics for different types of task (i.e. cosine for RoboYoga and temporal for the robotic manipulation). 

It is unclear from the plot in Figure 4 if the other methods (DIAYN, DDL, SkewFit, GCSL) fail to learn well or if they are slower to converge. Please include all their final performance after longer training (could be a dashed line). 

It is unclear why step 8 in the algorithm is necessary, since in principle (according to MAX, Plan2Explore etc) the disagreement should be sufficient for exploring the different regions of the state space. The way \pi^g is trained is similar to Skew-Fit without the skewing step (and also training it in the model rather than the real environment), which that paper reports does not explore very fast. A useful ablation would be to try removing step 8 in Algorithm 1 and see how well it works. Does \pi^g discover different types of states than \pi^e? It seems like both step 7 and 8 are meant to add novel states to the buffer, but it’s not clear if they are exploring differently and are both needed.

Minor:
Lines 7-8: “Unlike prior methods that explore by reaching previously visited states, our explorer…” this is not true, there are several methods (MAX, Neural-E3, Plan2Explore) which explore by planning/training a policy inside the model. 
Line 32: “A traditional RL would” -> a traditional RL agent would
Line 66: The introduce -> We introduce 
Line 234: nvolves -> involves
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to jointly train two agents in a differentiable world model: an *explorer* that is rewarded to explore uncertain states and an *achiever* whose task is to achieve a given goal state. After training, the *achiever* can be deployed as a goal-reaching policy without further tuning. Authors also spent time investigating how to design the loss function for the *achiever*. In the experiments, three new benchmarks are introduced and the proposed method is shown to outperform baselines.

main_review: **Originality**: The main idea is novel and incorporates many sound ideas in the field including Dreamer, temporal distance, and disagreement for exploration. The related works are adequately cited and described so the contribution is clear to me. The questions asked from line 75-79 also help me understand how this work is different from previous contributions.

**Quality**: Overall, the main idea kinda makes sense. However, some details still bother me:

- Line 176-177, “... to incorporate learning signals from far-away goals, we include them by sampling images from a different trajectory.” -> I am not sure whether images from different trajectories are necessary *far away*. Theoretically, they can be close to each other in terms of “the number of time steps between them”. Does this implementation cause any training unstability?

The technical contributions are supported by the ablation study shown in Figure 8. Thank the authors for running this experiment, I find it pretty informative.

**Clarity**: The high-level idea is presented clearly in the Introduction, Figure 1, and Figure 2. The low-level implementation details are described thoroughly in the supplementary material (which contains code). However, the mathematical notations in Section 2 are confusing and I hope authors can clean them up. I have the following questions:

- What is the capital $V$ in Figure 2? Is it the same as the value $v$ in the text?
- In section 2.1, $e_t$ shows up in equation 1's Posterior term but is not described in the text so its appearance hurts the readability.
- In section 2.2, I think the reward function (equation 3 & 4) of the explorer should appear before equation 2 for readers to have a better understanding of what $v^e$ should look like. Currently, $v^e$ shows up abruptly in equation 2 and its relationship with $r^e$ is only described near the end of section 2.2.
- Line 113 "produced by the encoder $q_{\phi}$", so both $q_{\phi}$ and $enc_{\phi}$ in equation 1 are called "encoder"? I think authors are probably referring to the posterior estimator as an "encoder" following the VAE literature but it would be nice if authors can clarify its relationship with another "encoder" $enc_{\phi}$ in the revised manuscript.
- In equation 3, what’s the relationship between $\hat{z}$ and $z$ defined at line 97?
- Line 146 "... to the goal $e_g$". It would be nice if authors describe it as "the embedding of the goal $e_g$” otherwise it confuses me as authors also denote g as goal at line 11 & 12 of Algorithm 1.
- In equation 7, it's not clear why \hat notation is introduced. How are variables with \hat different from their original meaning? Also, why is $\approx$ used instead of $=$?

Some other wordings that confuse me:

- Line 272-273, “real world data from the replay buffer”, it would be more clear if “real world data” is substituted with something like observed data as all the results in this paper come from the simulation.

Many obvious typos appear in the text:

- Line 31, "a traditional RL would have to" -> "a traditional RL agent would have to"
- Line 61, "between an the states" -> "between the states"
- Line 71-72, "and thus enormous computational ..." -> "and thus consume enormous computational ..."
Line 120, “seek out situations are …” -> “seek out situations that are”.
Line 156, “We address this is by …” -> “We address this by ...”.
Line 250, “train a train a “ -> “train a”.
Line 265, “i.e whether ..,” -> “i.e., whether”
Line 267, “that is need” -> “that is needed”

**Significance**: Since all the experiments are run on three *new* benchmarks introduced by authors, it’s really hard to tell whether the results are important. As the proposed idea is not limited to vision-based observations, I wonder whether authors can show results on state-based environments used by [1] and compare the performance. Experiments on state-based environments will help us understand 1) whether imagination is needed for low-dimensional observation space and 2) whether the proposed method, being more complicated, accidentally fails for simpler domains. 

[1] Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery, Hartikainen et al.

---

I am willing to raise my scores if authors addressed my concerns and above.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary: This paper proposes a model-based method with an ensemble disagreement-based exploration bonus for learning universal goal-conditioned policies. The world model and policy learning methods are adapted from DREAMER. In addition to that, the paper first introduces an explorer to first explore the state with large uncertainties, and then train the achiever to reach the proposed goals. The use of the world model enables data-efficient learning and exploration. The paper makes solid experiments and evaluates their approach on various domains. The paper also discusses two different distance metrics for learning goal-conditioned policy on images.
main_review: - The paper is well-written with convincing experiments. The proposed approach is simple yet effective. The separation of the exploration policy for goal generation and goal-conditioned policy learning looks like a general approach. The experimental results are not perfect. I notice that most tasks can't reach a 100 percent success rate. But considering the challenging image-input setup, it's great to see that it works in practice.
- What concerns me most is the novelty of the paper. The paper's technique is a combination of image-based RL, uncertainty-based exploration, and goal-generation tasks, but I am still happy to see that one can assemble these techniques to solve challenging problems. I would be happy to see it at the Neurips conference.
limitations_and_societal_impact: The authors claim that "Potential societal impacts of robot learning are at this stage largely hypothetical since the technology is not yet in widespread production." and I agree.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a model based RL setting where an agent learns a task agnostic model, learns two policies to explore and reach input goal states (specified as images) respectively and is able to operate in a goal conditional mode during evaluation time. 

The key thesis is to use the world model to incentivize the agent to explore the environment by maximizing expected information gain and reach input goal states by learning to reach states from its past experiences. 

There have been other model free and model based approaches studying the same problem but the claim is that its the first to combine model based exploration and reachability on new benchmark environments. 
main_review: Unsupervised RL is an interesting and important problem in deep and reinforcement learning. There have been several papers and baselines that have tackled this problem before. Perhaps the biggest limitation of this paper is lack of comparisons to past benchmarks. For instance the Sekar et al. paper (cited in the main paper) considers several environments and tasks that are not covered in this paper. The DISCERN (cited) model also provided benchmark results on Atari and continuous control tasks. The benchmark in this paper is new and therefore it is difficult to understand and compare the approach with past results. 

The main argument that the authors present regarding past benchmarks is -- "In contrast to common RL benchmarks such as Atari [5] that require training over 50 different agents and thus enormous computational resources, our unsupervised RL benchmark only requires training 4 agents, which are then evaluated across many tasks, allowing for faster iteration time and making the research more accessible". It seems practical to at least try a handful of the same environments to set the right experimental standards. Otherwise it is very difficult to validate the claims and understand the effect of choices proposed in this paper. I believe the paper would become much stronger if this is addressed. 
limitations_and_societal_impact: Seems adequate
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a framework to train policies for visual goal oriented tasks. The framework comprises two policies, the first, an explorer, which is trained to seek out novel states (where novelty is defined as the level of agreement between an ensemble of future state predictors - an approximate measure of epistemic uncertainty), while the second, an achiever, is trained to reach these states. A model based approach to train the achiever, where the achiever is trained using trajectories generated by a learned dynamics model. The paper also introduces a new benchmark for testing goal oriented polices of this form, comprising a kitchen manipulation task, a block handling task, and a yoga task. 
main_review: Originality:

The primary contribution of this paper is to show that an architecture explicitly balancing exploration and exploitation is also effective in high dimensional settings for goal oriented robot tasks. Existing work has focused on lower dimensional observations. I like the idea, and think its a good one, although the originality seems to be limited to scaling existing ideas to harder tasks.

The second contribution of the paper is a new suite of goal oriented tasks for robot goal tasks. These tasks seem to be more complex (in terms of manipulation, mapping of actions to goals) than existing benchmark suites, although I think some additional discussion on this may be warranted, eg.  benchmarks like robothor https://ai2thor.allenai.org/robothor/ are already widely used for goal oriented visual navigation, and have physical counterparts that are better suited for real world transfer.

Quality and clarity:

The idea is solid, and experiments are clear and concise. Ablations highlight the importance of the explorer network. A number of baselines and comparisons to prior work are included.

Significance:

The results presented here are unsurprising, but solid. I think the benchmark of tasks is a nice contribution, and that the  demonstration that policy training balancing exploration and exploitation is effective in high dimensions is a useful.

Detailed questions/comments:

Reward engineering. The introduction notes that a challenge with RL lies in reward engineering, where it can be difficult to choose effective rewards for tasks, and hence the need for unsupervised goal oriented learning. While I agree with this in principle, the results in this paper seem to me to contradict this, as it is clear that different cost functions eg. the cosine vs temporal distance measure, are still needed to train policies for different tasks. It seems to me that all that has happened here is that reward engineering has shifted to a different latent space. 

Could you clarify how the explorer and achiever were trained. Is this jointly, or is the explorer trained first, with the achiever then trained, by bootstrapping on data gathered by the explorer? From the text I gather the latter, but I would appreciate a bit more clarity here, eg. how much data was seen by each, etc.

Results: How is success determined for each of the tasks? Is this distance to a particular simulator state? A perceptual visual metric? Some human label? 

In some of the example videos, it seems that camera perspective effects is affecting the type of goals reached. How robust is this sort of approach to small changes in camera angle, etc.

Minor: 

Typos: line 66, the introduce the
line 129-230, grammar

limitations_and_societal_impact: I don't see immediate negative societal impact for this work, as it is extremely early stage research. 

However, a number of limitations were not discussed, and should be addressed in the paper.

A major challenge preventing the deployment and practicality of exploratory policies in real world settings is safety. Looking at the motions followed by the policy, and given the ease with which torque controlled robots like the panda can be broken, it is clear that these policies are completely infeasible for real-world robots. I think it is important to make this clear, and reflect on the gaps between sim and real here, and the need for stringent safety constraints if this was attempted in real world settings.

Moreover, the proposed approach will rely significantly on environment resetting, which is also infeasible in real world settings. Additional detail about the number of times and conditions under which these environments were reset, episode length etc. and being explicit about this would be valuable.
review_title: null

Meta-review: This paper proposes to learn a useful goal-conditioned policy in the absence of reward by jointly training an explorer that learns to visit uncertain states and an achiever that learns to reach a given goal state. All of the reviewers agreed that this is a novel combination of the prior work on learning a goal-conditioned policy in self-supervised learning and model-based RL, though each component is not entirely new. The only initial concern was that the proposed method was not evaluated on domains where the baselines were evaluated. However, the authors provided an additional result during the rebuttal period, and the majority of the reviewers are satisfied with it. Therefore, I recommend accepting this paper and suggest the authors to include the new result in the camera-ready version.
Decision: Accept (Poster)