Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose an approach for learning hierarchical policies from demonstration data. TACO-RL first learns a goal-conditioned, imitation learned policy from play data, similarly to [13]. Then TACO-RL uses offline reinforcement learning to train a high-level policy to propose goals for the low level, facilitated by a goal relabeling and negative mining approach to improve performance. This is then shown on a maze and robot desk environment in real and sim to outperform LMP and CQL baselines in both short-horizon and long-horizon tasks.
strengths_and_weaknesses: he paper is generally clear, as are the figures. The approach is interesting and developing scalable policies that leverage unstructered demonstrations is important. The results show it performs well, particularly for image space, interestingly it also seems to stabilize short-horizon policies as well as long. Including the code is appreciated.

My primary concern with TACO-RL is its framing with HRL and prior work that makes it difficult to judge the novelty. 
- The authors minimally discuss other HRL work and frame this approach as a “planner” which I do not believe it fits into the class of. Particularly many recent works have focused on offline HRL and goal-conditioning. See:
  - [1] Data-Efficient Hierarchical Reinforcement Learning (https://proceedings.neurips.cc/paper/2018/file/e6384711491713d29bc63fc5eeb5ba4f-Paper.pdf) 
  - [2] Actionable goal-conditioned policies (https://openreview.net/pdf?id=Hye9lnCct7)
  - and many more related to these works
- The authors claim “To our knowledge, our method is the first learning system for a generalist-purpose robot that uses the benefits of hierarchical policies skill-chaining latent behaviors to reason over long-horizon tasks.” This is a significant overclaim to my understanding, based on prior work such as below (but also many others in HRL):
  - [3] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (https://arxiv.org/pdf/1910.11956.pdf) which chains goals
  - [4] Value function spaces (https://arxiv.org/pdf/2111.03189.pdf) which chains skills
  - [5] Neural Task Programming (https://arxiv.org/pdf/1710.01813.pdf)
  - [6] GTI (https://arxiv.org/pdf/2003.06085.pdf)
  - Even the original play paper does this to an extent.
- These approaches do skill chaining particularly for robots, the first of which proposes a very similar methodology to TACO-RL.
- The authors need to discuss and compare to works such as above, as the comparison to LMP and CQL is nice, but not informative as they are not comparable methods in this class.
- With a more HRL framing the authors should compare to some of these prior works and possibly on some standard benchmarks.

A few major questions/clarifications:
- The authors of [3] discuss issues around training a high-level policy on demonstration data, when the low-level policy may not mirror it. Is this an issue at all here?
- The authors should clarify in Section 4.1 what is the standard LMP formulation and what is theirs. And highlight any changes.
- The authors should ablate over their algorithmic choices, such as the negative mining.
- It would be useful to see long-horizon tasks in real.

There are significant typos and grammatical errors. The paper should be reviewed for clarity:
- L18- reinforcement learning (RL) achieved -> reinforcement learning (RL) has achieved 
- L20- fixed datasets yield a strong current in -> rephrase
- Split up paragraph 2, its hard to get through
- “generalist-purpose robot” -> “general purpose
- Many more

Minor:
- Images of every environment would be useful
- Figure 1 could be clearer
- Define LH-VisMTLC
- It would be useful to have one video with the an overview and then rollouts from each environment. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The approach is reasonable and the performance is good compared to baselines of LMP and CQL. But these are not the baselines I would have expected, as this is a type of HRL planner. The framing of TACO-RL as a planner versus HRL, leads to missing these ablations and baselines, and generally discussing and framing the paper within existing literature.

After rebuttals I have raised my score to weak accept.
issues: See weaknesses above.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a hierarchical policy and skill learning approach that can learn from human collected play data. The goal is to be able to achieve longer horizon tasks while training the hierarchical policy in a task agnostic manner. Firstly, a latent action model is learnt to encode play data. Specifically, this consists of learning a mapping sub trajectories: $s_t, s_{t+1}..., s_{t + k}$ to a latent $z_t$ which encodes the "skill". A low level policy is used to decode this latent into actions $a_t$. A policy prior is also trained to output the right $z$ to reach between state $s_t$ and goal state $s_g$. 

Using the learnt encoder and policy prior, an offline RL goal-conditioned policy is learnt to produce the right $z$ given current state, achieved goal state and desired goal state. This is trained via a standard offline RL procedure and a designed goal state relabeling scheme. Experiments show good performance on CALVIN benchmark and standard offline RL benchmarks. 
strengths_and_weaknesses: Strengths: 

I think this paper tackles an important problem of learning a hierarchical policy from play data. Play data can provide useful signal and it is non-trivial to try to build a policy using this. This method, to my knowledge, is a novel way to learn a goal conditioned policy based on play data -- but the novelty could be made cleared (see weaknesses paragraph). In general, this method is technically sound and well motivated, and the presentation quality of the paper is good. Overall,  I appreciate experiments on both the offline RL benchmark, CALVIN benchmark and real world settings. In all of these, the method shows a good performance. 

Weaknesses: 

I think the main weakness for this paper is that it is unclear how it situates with previous work done in this area, such as LMP [13]. I think there could be more baseline comparisons, especially with other skill learning and hierarchical policy learning approaches. It is currently now quite clear how the CQL + HER baseline works - in my understanding this is just doing relabeling on the raw data. Overall the experiments could be strengthened with a more rigorous ablation section, for example understanding the role of the relabeling scheme (and comparison to others), they type of imitation learning done on the play data, which skills are actually represented by $z_t$. The real world experiments are interesting, but it would be good to see chaining of the skills for long horizon tasks -- as this is one the stated benefits of this approach. This could include for example opening the drawer, picking up the object, putting it inside the drawer and closing it. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: My recommendation for this paper is a "weak reject" but I am willing to increase my rating if the authors can clarify exactly where this work falls with respect to prior work such as LMP and others. Overall, I think the 
issues: The main issues that should be addressed: 

- More comparisons to hierarchical policy learning methods 
- More ablations analyzing the work
- Possible long horizon tasks, especially combining the skills seen in the real world (open drawer, grasp object etc)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach that learns a hierarchical policy from offline play data. On the low level, the propose approach imitates a latent-conditioned policy using a CVAE. On the high level, a goal-conditioned policy is trained using CQL. The proposed approach is evaluated in both simulated and real-world table manipulation tasks and achieves better performance than LMP and CQL baselines. 
strengths_and_weaknesses: Strengths: 

- The proposed method consistently achieves a noticeable performance improvement compared to LMP and CQL+HER. 

- The real-world task seems to be challenging and the proposed method achieves good success rates in these tasks.


Weaknesses:

- The paper does not clearly justify why the proposed hierarchical model is necessary. Usually, hierarchical reinforcement learning enables sharing knowledge about low-level skills across multiple tasks. The alternative and more common hierarchical RL frameworks either learn a goal-conditioned policy on the low-level and use a planner on the high level (e.g. Planning with Goal-Conditioned Policy (https://arxiv.org/abs/1911.08453), Planning-to-Practice (https://arxiv.org/pdf/2205.08129.pdf)), or learns a latent-conditioned policy on the low-level and use a non-goal-conditioned high-level policy. The proposed method is a hybrid of both and seems to be redundant to me. Could the authors explain why this design is better and include comparisons with these approaches in the experiments?

- The proposed low-level learning algorithm is reasonable but similar approaches have been proposed in prior work, e.g. Pertsch et al. (https://arxiv.org/abs/2010.11944), Fang et al. (https://arxiv.org/pdf/1910.13395.pdf), Singh at al. (https://arxiv.org/abs/2011.10024). I would recommend the authors discuss the differences with these works and include comparisons in the experiment section. 

- It is not clear to me why the authors regularize q_{\phi} to a prior conditioned on both the s_c and s_g instead of only the current state. The sentence on line 143 (“Since our goal is to obtain a latent plan……” is a bit confusing to me.

- Only “three different random seeds” are used in each experiment, which is a bit insufficient.

- Is there any specific reason for using CQL instead of AWAC or IQL, which demonstrates better performance in many experiments. 


- The precisions in each of Table 1-4 are not unified. Sometimes the number is rounded to the first digit sometimes the second. 

- s_c is not defined in the paper. 

- The limitations discussed in the paper is about goal-conditioned RL and behavioral cloning in general instead of being about the proposed method.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I am leaning towards rejection for now since the paper needs to better justify its design choices and include more thorough comparisons with prior work. I would consider raising my scores if the authors can address the above issues. 
issues: - The design choices need to be better explained and fully justified. 

- More prior work needs to be cited and discussed.

- More baselines need to be compared in the experiment section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a method (TACO-RL), to tackle self-supervised, goal-conditioned RL from offline datasets with just visual observations. Specifically, given "play" data (unstructured human demonstrations), the method learns a policy to reach goal images, with an emphasis on reaching long horizon goals. 

The proposed method learns two models:
- First, a low-level skill conditioned model trained with imitation learning. This part of the model is same as the LMP method (Lynch et al) where relabeled goals are mapped to a latent skill space which the policy is conditioned on.
- Second, a high level goal conditioned policy whose actions are the space of skills which gets fed to the low level policy. This policy is trained with offline RL with relabeled goals and a binary relabeled reward. 

The proposed method outperforms each of the parts of the method (i.e. LMP and CQL+HER individually). Results look strong on a D4RL ant task, CALVIN manipulation, and a real robot setup. 

strengths_and_weaknesses: *Strengths*
- The problem tackled is well motivated, learning to reach long-horizon goals from reward-free offline dataset is important in getting robots to learn from large datasets.
- The proposed method is straightforward and well motivated. It makes sense that the goal-conditioned imitation is reasonable on shorter horizons, but wont reach distant goals. Meanwhile, offline RL with goal-relabeling can stitch together trajectories and makes sense as a choice for a high level policy enabling reaching distant goals. Also unlike many HRL algorithms, since the low-level policy is simple imitation and decoupled from the training of the high level policy, the method seems stable and scalable.
- The simulated experiments on CALVIN show pretty solid improvements over LMP and CQL+HER individually. 
- The paper has robot results.

*Weaknesses*

- First, the coverage of related work could be improved. There are several prior works [1,2,3,4,5] which also aim to reach long-horizon goal images in a self-supervised way via subgoals/hierarchy that aren't mentioned. [5] could also be good include as an additional point of comparison. 

[1] Jayaraman et al. Time-Agnostic Prediction: Predicting Predictable Video Frames. 2018.
[2] Nair et al. Hierarchical Foresight. Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation. 2019.
[3] Pertsch et al. Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning. 2019.
[4] Pertsch et al. Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors. 2020. 
[5] Rybkin et al. Model-Based Reinforcement Learning via Latent-Space Collocation. 2021.

- Second, I'd be interested in seeing some more exhaustive ablations in the experiments. While LMP and CQL+HER are effectively the main two ablations, I'm also curious what would happen if the hard negative mining trick from Tian et al. is removed from TACO-RL. From what I can tell from the text, the baseline CQL+HER does not get this same trick, in which case the comparison isn't totally fair. 

- Third, while it is great the paper has real robot results, the real world tasks look pretty short horizon, especially compared the skill chaining highlighted in the simulation results. Some longer-horizon real world tasks would make the results more compelling. 

Finally, I had a few more questions about the method:
- From Figure 2 it seems like the high level policy is executed once for every 1000 steps of the low level policy. I'm curious how this parameter was chosen, and if performance varies depending on the choice of how frequently to query the high level policy. 
- How important is the LMP latent plan/skill learning component of the low-level policy? Could the low-level policy be replaced with a simpler GCBC policy? 

Some other minor comments:
- Table 4 caption has some formatting issues.
- It would be nice to see videos of the task execution (especially the simulated skill chaining). An anonymized website which highlights the sim and real rollouts of TACO-RL and baselines would be a nice addition to the paper. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall a solid paper which combines LMP and CQL+HER for good results on long-horizon goal reaching from offline datasets and visual observations. Method is clear, has strong improvement over baselines in sim, and has real robot results. 
issues: See Strengths and Weaknesses.
review_title: null

Meta-review: metareview: The paper presents a framework for offline RL. The key idea is to train a high-level goal-generator trained via offline RL to guide a low-level goal-conditional policy trained via imitation learning. The method is tested on the CALVIN benchmark and standard offline RL domains, as well as a real-world robot setup. 

Strengths: The reviewers agree on the importance of the problem setup and the presentation quality of the paper. They also acknowledge the value of the real-world evaluation. 

Weaknesses: The main criticism lies in the technical novelty of the method and its relationship to prior works. As reviewers nPLZ, jnvs, and mMCb pointed out, there are quite a few missing reference to relevant methods. And the lack of discussion and empirical & conceptual comparison to prior works in hierarchical decision making makes it difficult to judge the novelty of the work. Reviewers 3aE6 and mMCb also pointed out that the tasks in real-world evaluation are rather short-horizon and may not demonstrate the value of the hierarchical policy framework.

Post response period: The authors have addressed most concerns raised by the reviewers. In particular, the authors extended related works discussion, included additional HRL baselines, increased number of real-world task evaluations. The scale of the real-world evaluation is in particular laudable. During the post-response period, the reviewers have all increased their ratings (3 WR->WA, 1 WA -> SA). However, some doubts in technical novelty and empirical comparisons with HRL baselines still remain. Given the author response, the updated ratings and their justifications, the AC recommends the paper to be accepted with a poster presentation. 
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)