Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes Dual Meta Imitation Learning (DMIL) which is a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning (MAML). The DMIL is a hierarchical extension of MAML-based imitation learning (IL) and is a meta-learning extension of Hierarchical Imitation learning (HIL).  The authors provide theoretical proof of the convergence based on the connection with the Expectation-Maximization (EM) algorithm. They showed the state-of-the-art few-shot imitation learning performance on the meta-world benchmark.
main_review: The strength of the paper is that they proposed a natural extension of meta-IL and HIL. The total algorithm is reasonable, and its convergence is proved based on the theory of the EM algorithm. The connection to the EM algorithm is natural and understandable. The experiment showed the usefulness of the method as well. 
The weakness of the paper is the limitation of evaluations. They tested their methods on two pre-existing simulation environments. The evaluation using a task meta and hierarchical nature is more explicitly required is expected. The strength of the method would be more clearly shown with such a task.

summary_of_the_review: The paper proposed a new meta HIL method called DMIL. The method is clearly explained. The background is described in a proper manner and is informative. The novelty is clear. Also, the empirical evaluation is appropriate though the number of environments can be increased. 
As a whole, I admit that the paper is an excellent paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose to learn a high level policy network that picks a subskill policy to predict actions with meta imitation learning (DMIL). Both the high and low level policy networks are fine-tuned at meta-test time. They show that DMIL converges and evaluate their approach against several ablations on the ML10 and ML45 setting of the meta-world benchmark.
main_review: **Pros**
- The authors show that DMIL is competitive against a thorough set of ablations on both ML10 and ML45 settings from meta-world.

- The convergence analysis hints at the algorithms training stability.

**Cons**
- The method mostly borrows components from existing works without explaining how their setting adds more complexity to the approach.

- From the results, it's difficult to compare this method against hierarchical imitation learning approaches that they cite (Yu et al., 2018b; Finn et al., 2017b; Yu et al., 2018a)

- The authors do not show that the results are robust across different tasks/environments (e.g., "Hierarchical Few-Shot Imitation with Skill Transition Models" looks at maze and kitchen environments) or number of subskills. I believe this is important because the manipulation tasks in meta-world may have compositional task structure that can't be assumed in other environments or tasks (such as navigation).

**Questions**
- How would you contrast DMIL against "Hierarchical Few-Shot Imitation with Skill Transition Models"?

**Feedback**
- It would aid understanding to include a algorithm box that outlines the order of inner and outer updates in one place.
summary_of_the_review: I vote to marginally reject this submission because the experimental results are not thorough enough to demonstrate robustness across environments and number of subskills and the novelty of DMIL mostly comes from combining known approaches. If more environments and subskills could be tested, I think the paper could be a candidate for submission.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a meta imitation learning framework aimed at learning long-horizon robot control tasks with fast adaptation capabilities. Specifically, the proposed approach adapts the model-agnostic meta learning framework for learning a hierarchical policy, where both levels of the hierarchy are meta-trained and fine-tuned at test time to learn new tasks. On the metaworld benchmark, the proposed method outperforms prior work, and qualitative analysis shows that the method discovers meaningful skills.
main_review: This paper has a number of strengths:
1. This paper studies an important research problem  — the ability to discover meaningful abstractions of behavior from prior experience and to transfer these abstractions to efficiently learn new tasks
2. The related work section lays out a detailed overview of relevant work
3. While the proposed approach integrates existing elements from MAML and hierarchical imitation learning, it does so in a unique and novel manner
4. The theoretical analysis complements the work well (the AC should note however — I did not scrutinize this section sufficiently to verify its claims)
5. The experiments consider a thorough set of baselines

Alongside these strengths, I also have numerous concerns, all regarding the quality and clarity of presentation:
1. In my view, the introduction of the paper sells the work short. Much of the discussion is on centered “what” approaches previously exist and “what” the technical components of this paper are. Relatively little discussion in centered on “why” this proposed research problem is important, “why” previous approaches falls short, and “why” addressing the limitations of prior approaches is challenging. Adding this context can help better motivate this work.
2. The figures are difficult to follow. In particular, figure 1 and 2 have a lot of visual details, but are accompanied with sparse textual descriptions. Ideally, figures should be self-contained, where the caption can describe the figure in full detail.
3. While I appreciate the motivation in the introduction to learn long-horizon tasks, I do not think that the experiments necessarily study long-horizon tasks. The metaworld environments entail relatively short sequences of simple primitives, and furthermore this paper only uses 3 sub-skills which is not reflective of the diversity that comes with long-horizon tasks. 
4. The writing currently suffers from a significant number of grammatical issues. There are a number of passive sentences throughout, and some sentences are simply difficult to follow, such as the following: “there is no enough ability in sub-skills to quickly adapt to new tasks, thus no certain sub-skill exceeds others to give out dominant supervision for the high-level network.” Other sentences such as “DMIL is a brand new iterative hierarchical meta-learning procedure” use informal language (“brand new”) and should be revised accordingly.
5. The discussion section mentions future avenues of research, but does not quite touch upon the core limitations of the work. Here are a set of potential limitations to warrant discussion: what are the failure modes encountered during experiments? was there mode collapse among the skills? how sensitive is the method to the number of skills K?
6. I’d like to know more about the MIL baseline — why was a transformer used here as opposed to a simple MLP, and why does DMIL not use a transformer?

summary_of_the_review: While I think the technical contributions of the paper are sound and the experiments are generally thorough, this paper at its current form is limited by the writing quality. A signifiant amount of work needs to go in re-writing the introduction, improving the figures, and fixing grammatical issues.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach for few-shot imitation learning that jointly meta-learns a high-level policy and a set of low-level policies from a diverse set of demonstrations (multi-task). It then finetunes both high-level and low-level policies on few demonstrations of the target task for imitating them. In experiments on the MetaWorld50 task suite the proposed approach outperforms few-shot imitation methods that either don't use hierarchy or don't use meta-learning.

main_review: ## Strengths
- few-shot imitation is an interesting and impactful problem

- the paper solves a hard problem: jointly learning HL and LL policy + quickly adapting them on unseen tasks

- the method is clearly described, the figure help understanding the approach

- the method is evaluated on a representative benchmark and compared to meaningful baselines

- the qualitative results show that the approach learns meaningful skills in the MetaWorld benchmark


## Weaknesses

- My main concern with the method is that it is trying to solve too many things at once: (1) inferring skills from unstructured datasets, (2) jointly learning high-level and low-level policy and (3) meta-learning both these policies to have them adapt quickly on new tasks. Intuitively, the resulting iterated meta-optimization scheme seems unstable to train / sensitive to choices like the policy initialization, number of subskills, switching regularization coefficient, ... (the theoretical convergence results do not elucidate how hard it is to train agents with this approach *in practice*). The paper could add an analysis on hyperparameter robustness of training to show the stability of the approach.

- While the paper cites many works on hierarchical imitation and meta-imitation learning, it does not cite the relevant recent works on learning skills from large, multi-task datasets (SPiRL, Pertsch et al 2020, OPAL, Ajay et al 2021) and using them for imitation learning (SkiLD, Pertsch et al 2021, FIST, Hakhamaneshi et al 2021). Especially FIST seems like an interesting method to compare to since it is made for the few-shot imitation setting. That being said: this comparison is not mandatory since FIST is concurrent work.

- I think the paper could be strengthened by including a comparison to a method that is hierarchical but *does not* learn HL and LL policies jointly. All of the abovementioned works first learn a set of low-level skills and then learn a high-level policy over them. It would improve the comparisons of this paper to include an approach that pre-trains temporally extended low-level skills, then freezes them and meta-trains a high-level policy over these skills. Such a training scheme would presumably be much easier to tune (since it does not require EM-style iterated training) -- so it would be good to see whether the performance is much worse than the proposed, more complicated method. (the already included DIML-High ablation still trains HL and LL jointly even though it keeps the LL fixed during downstream finetuning from what I understand)

- The paper does not include an ablation on the chosen number of discrete learned skills. It is possible that the DIML-High ablation could perform better if it had a larger number of low-level skills to choose from (since it does not adapt the low-level skills to the downstream task), so including such an experiment would strengthen the paper.


## Further Suggestions
- I wonder whether the approach could apply an idea similar to label smoothing where instead of using a hard max for the inner training loop of the HL policy we would set the targets for the HL cross entropy loss to be the softmax of the action prediction errors of the low-level. Maybe such a more nuanced training signal could improve the training stability?
summary_of_the_review: The paper proposes an interesting approach for jointly learning adaptable high and low-level skills. While the method is well-explained it seems to more complex than alternative approaches and the current experiments don't fully convince me that it would be easy to train on new tasks without a significant amount of tuning to get the iterated learning loop to converge. If the authors can provide experimental evidence that the method is no harder to train / tune than alternative approaches for few-shot imitation I am leaning towards accepting the submission, but I am open to changing my opinion based on the other reviewer's feedback.
review_title: null

Meta-review: The paper proposes a hierarchical meta imitation learning framework for few-shot transfer in the context of long-horizon control tasks. Underlying the framework is a hierarchical adaptation of model-agnostic meta learning (MAML) that jointly learns the high-level policy together with the set of modular low-level policies (sub-skills), both of which are fine-tuned at test time based on a small number of demonstrations. Experimental evaluations on the meta-world benchmark as well as a kitchen environment benchmark compare the proposed framework with recent baselines.

As several reviewers note, the problem of jointly learning modular policies together with the high-level policy for composing these sub-skills is both challenging and interesting to the robotics and learning communities. The manner by which the paper extends existing work in meta-learning (MAML) and hierarchical imitation learning is novel and technically sound. The reviewers raised some concerns, notably those regarding (1) the the framework's sensitivity to various hyperparameter settings and its ability to generalize to other domains; (2) the merits of joint optimization over decoupled optimization of the sub-skills and high-level policy; and (3) the need for experiments/evaluations on different domains. The authors provided a detailed response to each of the reviewers that includes the addition of a different benchmark evaluation (the kitchen environment), new ablation studies, and updates to the text. After a thorough review, however, concerns remain regarding the reproducibility of the results, which call into question some of the key contributions that the paper claims to provide over the existing state-of-the-art. The authors are encouraged to provide a more balanced discussion of the contributions along with evidence to support reproducibility in any future version of the paper.
Decision: Reject