Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Paper proposes a new method for meta-learning hierarchical imitation learning, provides theoretical analysis of the algorithm showing its convergence, and demonstrates SOTA results on the meta-world benchmark and competitive results on the D4RL kitchen environments.
strengths_and_weaknesses: Strengths
- Nicely written and easy to understand
- Provides both theoretical analysis of the algorithm showing its convergence, and empirical results showing its benefits to prior methods
- Includes exhaustive ablation studies showing the importance of each component of the algorithm

Weaknesses
- The proposed method is rather complex, which may make it difficult to be deployed on real systems.
- It would be nice to include some intuitive discussion on why *both* the high level and low level components need to adapt to a new task? Since many different subskills can be represented with the low level component, it seems like maybe only the high level may need to adapt given a new task, by picking the right sequence of subskills? 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper provides an interesting new method and demonstrated superior empirical results compared to prior work. 
issues: See weaknesses section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Use MAML-style gradient-based meta-learning to learn a hierarchical structure from imitation learning. This requires an inner loop that fine-tunes the controller and the sub-skills and an outer loop that updates the meta-gradients. The order is from inner to outer, where the first step decides the assignment of sub-skills and assigns loss based on the behavior cloning loss of that sub-skill (log-likelihood). Then the second inner loop updates the sub-skills based on the assignments given by the controller to minimize behavior cloning. The outer loop then updates the meta-gradients. This work also demonstrates that the inner and outer gradients are equivalent to E and M steps for coordinate descent, implying that the algorithm will coverge to a local minima.
strengths_and_weaknesses: Strengths:
This work provided an interesting algorithm, clever implementation and good experiments

Weaknesses:
The writing is difficult to follow, using ill-defined terms like 23 "plenty", or 32 "come from that the" or "may conflict with descrpant situations". As it is, the second paragraph in the intriduction is difficult to understand and after several readings, it is still hard to determine exactly what it is saying. The third paragraph has the same issue, with statements like "the same question applies to the high-level network", when referring to what high-level network that low level sub-skills should choose, which does not appear to make sense. It was difficult to gain more insight from the introduction than what was stated in the abstract.

It is not entirely clear from the methods section how the meta-gradients actually work, and considering that is the main point of the algorithm (the rest is similar in concept to the EM algorithms that generally get used), that is somewhat disappointing. The proof is difficult to follow and thus to verify, though it appears intuitive enough: by making it equivalent to the EM algorithm, the algorithm will converge.

Overall, the choice of meta-world for tasks is good and the performance appears to be quite effective, especially with 45 tasks. However, it does seem a little bit suspect that 2 sub-skill performed best, considering that it seems like that implies that two general behavior patterns solve 45 different tasks. Would 1 task have performed even better? Either way, it seems like the MAML component may be where the performance comes from, rather than the skill learning, and some non-hierarchical change in architecture resulted in better training.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: This paper provides an insightful algorithm that seems like it has the potential to be built upon and evaluated in a classic benchmark with good results. While some of the specifics of the algorithm, such as the way that the number of sub-skills appears not to scale, cast some doubt on the algorithm or the evaluation, for the most part, it appears to be a valuable contribution. I was not particularly impressed by the proof, but that might be more related to my background than to any limitations with it. 
issues: Most of the issues are in the weaknesses section, but mostly the paper could benefit from having some more extensive experiments related to determining why such little variation is needed in the number of skills. In addition, the writing was overall difficult to follow and could be improved by some editing.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper tackles the following two problems (simultaneously): 
- Given a dataset of demonstrations from many different tasks, learn a set of hierarchical policies that solve these tasks well.  
- Learn the hierarchical policy above in such a way that when given a new task, you can quickly adapt both the high-level and low-level policies to this task. 

The former problem has been explored in the hierarchical imitation learning (HIL) literature, while the latter problem has been well-explored in the meta-imitation learning (MIL) literature. The main contribution of this paper is to address both of these problems at the same time. The proposed method combines an EM-style algorithm (popular in HIL) with MAML (popular in MIL). Experiments are performed on simulated robotics tasks, and show results that are decent (method works), but not stellar (comparable performance to competing methods on 2 out of the 3 benchmarks). 

strengths_and_weaknesses: Strengths
- The paper tackles an important problem (transferring subskills across tasks), that I believe is somewhat under-studied in the hierarchical imitation learning literature. 
- The paper is well-written. 
- Authors demonstrate results on manipulation tasks, which is nice to see (a lot of hierarchical learning papers end up focusing on locomotion / navigation tasks, where the hierarchical structures are easier find). 

Weaknesses
- The link provided for code and demonstrations points to an expired repository: https://anonymous.4open.science/r/DMIL
- “DMIL achieves higher rewards on two out of four tasks and comparable results on the other two tasks, which exhibits the effectiveness of the bi-level meta-training procedure.” – I am not sure if this is a fair assessment. The performance for DMIL is only higher for one task (the third task), and the results are comparable for the rest of the three tasks. The confidence intervals appear to be quite large for both methods. Similarly, for ML10, DMIL results are comparable to vanilla MIL. Overall, I would say that the empirical performance is decent (the method works), but not stellar (it’s only significantly better for ML45, one out of the 3 benchmarks studied in the paper). 
- Most meta-world tasks are fairly short (pick + place tasks, or turning a lever), so it’s unclear to me if a hierarchical method is actually needed for these tasks. As shown in the results K=2 works best for meta-world tasks. When more options are assigned, they typically go unused (Figure 2). The Kitchen tasks are longer, but the performance when compared against competing methods is not particularly stellar.
- The meta-imitation problem setting is well-suited to real world learning (since it can be done entirely offline), so it's a bit disappointing that the paper does not have any real robot results. Perhaps the proposed meta-learning algorithm has stability issues when working under real world constraints such as visual observations? If that is the case, it should be mentioned in the limitation sections. 

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, while the paper has weaknesses, I think it's technically solid, and targets an important problem, so I am leaning towards acceptance. Note that I did not check the theoretical convergence results (the intuitive derivation combining EM and MAML made sense to me). 
issues: Please take a look at the weaknesses, and use them to either update the paper or expand the limitations section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a new hierarchical imitation learning structure based on MAML to learn modular sub-skills and then transfer them to new tasks. Experiments on meta-world tasks and the Kitchen environment show the proposed method outperforms previous skill-based imitation learning methods.
strengths_and_weaknesses: Strengths:

1) This paper is well-written and easy to follow.
2) The proposed method is verified both theoretically and empirically.
3) Extensive experimental results and ablation studies show the advantage of the proposed method and the contribution of each component.

Weaknesses:

1) The structure of the paper could be improved to cover more technical details, like the details of auxiliary loss and corresponding experimental results.

2) Some references to the appendix are not clear enough, like in line 193, ‘Detailed information can be found in C.’ -> ‘Detailed information can be found in Appendix C.’ Similar in Line 205.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper clearly presents the proposed method and provides theoretical analysis as well as empirical analysis. I did not check the technical details of the theoretical analysis carefully. But based on my current understanding and if the authors address my concern, I would vote for acceptance of this paper.
issues: Please refer to the pros and cons part.

I have one major concern about Equation 2 which matters whether the low level can learn meaningful sub-skills. Since all networks are initialized randomly, so why low-level policies could predict different actions given a state $s$? Why should the closest action be the supervised signal (ground truth?) to the high-level network? 

Minors,
Line 173 ‘freezed’->’frozen’
review_title: null

Meta-review: metareview: Phase 1:

Strengths:
The paper is overall well and intuitively written. It includes convincing theoretical and empirical results and detailed ablation studies. Most reviewers appreciate the sound technical contributions and simulation-based experiments.

Weaknesses:
A big limitation is the absence of real robot experiments and some of the reviewers doubt future real-world application given complexity. Further, best results on metaworld 45 point towards the minimal number of sub-skills working best (k=2) and the reviewers would like to see this point discussed in further detail. Particular in light of the discussion of limitations which is called out across all reviewers as a point of required improvements. Further minor points target the slight overemphasis on potential gains and limited success in domains other than ML45.

Phase 2:

The feedback has been overall positive with 4 weak accepts but also includes some criticism (limitations, specific gains with smallest number of skills). I agree with the reviewers and recommend acceptance. Please take the remaining points from the review process seriously and follow up with improvements on open points and promised changes.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)