Transferring Hierarchical Structures with Dual Meta Imitation Learning
Authors: Chongkai Gao
Yizhou Jiang
Feng Chen
Venue: CoRL 2022 Poster (2022)
Keywords: Hierarchical Imitation Learning
Meta Learning

Abstract: Hierarchical Imitation Learning (HIL) is an effective way for robots to learn sub-skills from long-horizon unsegmented demonstrations. However, the learned hierarchical structure lacks the mechanism to transfer across multi-tasks or to new tasks, which makes them have to learn from scratch when facing a new situation. Transferring and reorganizing modular sub-skills require fast adaptation ability of the whole hierarchical structure. In this work, we propose Dual Meta Imitation Learning (DMIL), a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning. DMIL uses the likelihood of state-action pairs from each sub-skill as the supervision for the high-level network adaptation and uses the adapted high-level network to determine different data set for each sub-skill adaptation. We theoretically prove the convergence of the iterative training process of DMIL and establish the connection between DMIL and Expectation-Maximization algorithm. Empirically, we achieve state-of-the-art few-shot imitation learning performance on the Meta-world benchmark and competitive results on long-horizon tasks in Kitchen environments.

Transferring Hierarchical Structure with Dual Meta Imitation Learning
Chongkai Gao Department of Automation Tsinghua University gck208mails tsinghua. edu. cn
Yizhou Jiang Department of Automation
Tsinghua University
jyz20Cmails. tsinghua, edu. cn
Feng Chen
Department of Automation, Tsinghua University
LSBDPA Beijing Key Laboratory
chenfengdmail htt.:/mmma..siinuuaaaaaaa tsinghua. edu.
Hierarchical Imitation Learning (HIIL) is an effective way for robots acquire sub-skills from long- oonghooizoon unsegmented demonstrations. However, the learned hierarchical structure lacks the mechanism to transfer across multi-tasks or adapt to new tasks, which makes them have to learn from scratch when facing a new situation. Transferring and reorganizing modular sub-skills require fast adaptation abilities of both the high-level network and sub-skills to reschedule new forms of sub-skills in new tasks. this work, we propose Dual Meta Imitation Learning (DMIL), hierarchical meta imitation learning method where the high- level network and sub-skills are iteratively meta-learned with model-agnostic meta- learning (MAML [1]). DMIL uses the likelihood of state-action pairs from each sub- skill as the supervision for the high-level network adaptation, and uses the adapted high-level network to determine different data set for each sub-skill adaptation. We theoretically prove the convergence of the iterative training process of DMIL and establish the connection between DMIL and Expectation-Ne Expectation-Maximmzzttoo algorithm. Empirically, we achieve tfate-of-the-ar few-shot imitation learning performance the Meta-world [2] benchmark and competitive results long-horizon tasks of Kitchen environments.
Abstract:
Keywords: Hierarchical Imitation Learning, Meta Learning
Introduction
Imitation learning (II) has shown promising results for intelligent robots to conveniently acquire skills from expert demonstrations [3, 4]. Nevertheless, imitating long-horizon unsegmented demonstrations has been a challenge for IL algorithms, because the well-known issue of compounding errors [5]. This is one of the crucial problems for the application of IL methods to robots since plenty of practical manipulation tasks are long-horizon. Hierarchical Imitation Learning (HIL) aims to tackle this problem by decomposing long-horizon tasks with hierarchical model, in which set of sub-skills are employed to accomplish specific parts the long- morizoo task, and high-level network responsible for determining the switching of sub-skills along with the task. Such hierarchical algorithm [6. 7, 8] goal-conditioned IL paradigms [9]. HILL expresses the nature of how humans solve complex tasks, and has been considered to a valuable direction for II. algorithms [10].
structure is usually modeled with Options trained with Expectation-Maximin
However, most current HILL methods have explicit mechanism to transfer previously learned sub-skills to new tasks with few-shot demonstrations. This requirement comes from that the learned hierarchical structure may conflict with discrepant situations in new tasks. As shown in fig. 1(a), both the high-level network and sub-skills need to be transferred to new forms to satisfy new requirements: the high-level network needs new manners schedule sub-skills in tasks (for example, calling different sub-skills pulling or pushing at the same state), and each sub-skill needs to adapt to new specific forms in new tasks (for example, grasping different kinds of objects). This drives to develop new methods to endow HIL with the ability to simultaneously transfer both the high-level network and sub-skills with few-shot task demonstrations.
6th Conference on Robot I Learning (CoRI. 2022), Auckland, New Zealand.

---

trained high-level policy: "Call sub-sall pull in this staate"
requirement of high-level scheduling "Call sub-skill push in this state"
A dispiable Monolithic Network
Demognttiioo
Multi-tak Demonstratons
transfer
META
drawer close
Tew-sshot demonstrations
A Adaptable Hierarchical Structure
T One Shot
Fine- tune
demonssrattinn
Mulli-lask Demorstration
Pla
trained sub skill
pickppace
new forms of sub skill placr
DMILL
transfer
plck
(a) Illustration of the bi-level transfer problem of HIL in new tasks.
(b) Comparison of MIL and DMIL.
Figure I: (a) Both the high-level network and sub-skills need to be transferred to new tasks. Above: when the robot arm is over half-open drawer, the task can be either opening or closing the drawer, which requires the high-level network call different sub-skills. Below: the same sub-skill pick-place may exhibit different specific forms in new tasks. (b) DMIL aims to integrate MAML into HIL with a novel iterative optimization procedure that meta-learns both the high-level network and sub-skills.
Recently, meta imitation learning (MIL) [11, 12, 13] employs model- meta-learning (MAML) [1] into the imitation learning procedure to enable the learned policy to quickly adapt to new tasks with few shot demonstrations. MAML first fine-tunes the policy network in the inner loop, then evaluates the fine-tuned network to update its initial parameters with end-to-end gradient descent at the outer loop. The success of MII. inspires hierarchical structure in new tasks. However, this is not straightforward. HIL is a bi-level structure that is trained in an iterative and self-supervised paradigm [6]. thus both the high-level network and sub-skills need to be meta-learned by MAML. Intuitively, the question is: should sub-skills choose the fine-tuned high-level network or the original high-level network for their inner loops and outer loops? The same question applies to the high-level network. This is dual meta learning problem, and we need explore appropriate methods for MAML to schedule the fine-tune steps (inner loops) and meta-updatt steps (outer loops) of the bi-level network in HIL to ensure convergence.
integrate MAML into HII. to transfer the
In this work, we propose novel hierarchical meta imitation learning framework called Dual Meta Imitation Learning (DMIL) to incorporate MAML into the iterative training process of HII,, as shown in fig. ((b). Based on the common EM-like HILL structure [6. 7] we design elaborate bi-level MAML procedure for this hierarchical structure to make it can be fully meta-learned. In this procedure, we first fine-tune the high-level network and sub-skills in sequence at inner loops, then meta-update them simultaneously at outer loops. We theoretically prove the convergence of this special training procedure by leveraging previous results from [14, 15, 16] to reframe both MAML and DMIL as hierarchical Bayes inference processes and get the convergence of DMIL according to the convergence of MAML from previous results [17]. We test our method on the challenging meta- meta-world benchmark environments [2] and the Kitchen environment of D4RL. benchmarks [18]. In our experiments, we successfully acquire a set of meaningful sub-skills from a large scale of manipulation tasks, and achieve state-of-the-arr few-shot imitation learning abilities in the MI.45 suite. In summary, the main contributions of this paper are:
We propose DMIL, a novel hierarchical meta imitation learning framework that meta-learns both the high-level network and sub-skills from unsegmented multi-task demonstrations in general EM-like fashion, and theoretically prove its convergence.
We achieve state-of -f-the-arr-- fhw-shot imitation learning performance on meta-world bench- mark environments and competitive results in the Kitchen environment.
2
Related Work
2.1
Hierarchical Imitation Learning
Recovering inherent sub-skills contained in expert demonstrations and reusing them with hierarchical structures has long been an interesting topic in the hierarchical imitation learning (HIL.) domain. According to whether there are pretraining tasks, we can divide HII. methods into two categories. The first one employs set of manually designed pretraining tasks that encourage distinct skills

---

or primitives, then learn high-level network to master the switching of primitives to accomplish complex tasks [19, 20, 21, 22, 9]. However, for unsegmented demonstrations where no pretraining tasks are provided, which is more common in the real world, these methods can not be applied.
The second kind of methods aim to learn sub-skills with unsupervised learning methods. Daniel et al. [6], Krishnan et al. [7] acquire Options [23] from demonstrations with an
like procedure and use the Baum-Welch algorithm to estimate the parameters of different options. Henderson et al. [24]. Jing et al. [8] integrate generative adversarial networks into the option discovery process. Li al. [25]. Sharma et al. [26]. Lee and Seo [27] incorporate generative aadveraaaala imitation learning [28] framework and an information theooretic metric [29] to simultaneously imitate the expert and maximize the nutuull-nnormaan
between latent sub-skill categories and corresponding trajectories to acquire decoupled sub-skills. There are also some methods called mixture- mixture-of-expert of-expert (MoE) that compute the weighted sum of all primitives to get the action rather than only using one of them at each time step [30, 31, 32]. Other methods aim to seek an appropriate latent space that can map sub-skills into it and then condition policy on the latent variable generated from the latent space to reuse sub-skills [33, 34, 35, 36, 37]. In this work, we use the EM-like HIL methods used in [6, 7] as they can acquire semantic and separated sub-skill networks which have shown better results in recent works [38], and compare other kinds of HIL methods as baselines to show the superiority of our method in experiments. None of above HIL methods do not take the fast adaptation ability into consideration: they assume testing tasks are in the same distribution of training tasks. Although some work fine-tune tune the whole structure in new tasks [36], the performance of fine-tuning all depends on the generalization of deep networks, which may vary among different tasks and network designs. Thus we aim to enhance HII. methods with meta learning abilities, as described below.
2.2 Meta Imitation Learning
Meta imitation learning, or one-shot imitation learning, leverages various meta-learning methods and multi-task demonstrations to meta-learn a policy that can be quickly adapted to a new task with few-shot new task demonstrations. Duan et al. [39], Cachet et al. [40] employ self-attention modules to process the whole demonstration and the current observation to predict the current action. Yu et al. [13], Finn et al. [12]. Yu al. [11] use -agoos-g meta-learning (MAML) [1] to achieve one-shot imitation learning ability for various manipulation tasks with robot or human visual demonstrations. Xu et al. [41], Yu et al. [42] propose to meta-learn a robust reward function that can be quickly adapted to new tasks and then use it to perform IRL in new tasks. However, they need downstream inverse reinforcement learning after the adaptation of reward functions, thus conflicts with our goal of few-shot adaptation. Most above methods only learn one monolithic policy, lacking the ability to model multiple sub-skills in long-horizon tasks. Some works aim to tackle the multi-modal data problem in meta-learning by avoiding single parameters initialization across all tasks [43, 44, 45, 46), but they lack the mechanism to schedule the switching of different sub-skills over time. There are some works that also meta-learn a set of sub-skills in hicrarchical structure [13, 45], but they either use manually designed pretraining tasks or relearn the high-level network in new tasks, which is not appropriate in few-shot imitation learning settings.
3 Method
3.1
Meta Imitation Learning
We denote a discrete-time finite-horizon Markov decision process (MDP) as tuple (S, A,T, po). where S is the state space, A is the action space, T is the time horizon, P: S x A [0. 1] is the transition probability distribution, r S x A + IR is the reward function, and po is the distribution of the initial state sg- The goal of meta imitation learning to extract some common knowledge from a set of robot manipulation tasks {T.} that come from the same task distribution P(T), and adapt it to new tasks quickly with few shot new task demonstrations, As in model-agnostic meta-learning algorithm (MAML) [1], we formalize the common knowledge as the initial parameter 0 of the policy network Te that can be efficiently adapted with new task gradients.
For each task To ~ P(T), a set of demonstrations D; is provided, where D; consists of N demonstration trajectories: D; {rap), and Tij consists of sequence of state-action pairs: Ti {(s,, at) 22) where Tij is the length of Tıj. Each D, is randomly split into support set D and query set D.aa for meta-training and meta-testing respectively. During the training phase, we sample m tasks from p(T). and in each task Tú, we use D to tne-tune-nn Ty to get the adapted task-specific parameters X; with gradient descent, and then evaluate with Deal to get the meta-gradient of T., and optimize the initial parameters 0 with the average of meta-gradient from all tasks. The policy
3

---

High-love
High-love network
St
S2
Muthpty with hund sellecting
network
On
a
Sabb deil cadegrey probibility
c
Subs coligpar polibiiity
d11
d
dz2
Forwart grocsss
Sub-skill 0
Sublskill
Sub-skill 2
Subl 0
skil 1
Sub 2
Medi-wwrring
Figure 2: The iterative meta-learning process of DMIL at each iteration. Left: the supervision of high-level network (sub-skill categories) comes from the most accurate sub-skill (the green one, sub-skill here). Right: the sub-skill updated at current step (the green one, sub-skill 0 here) is determined by the fine-tuned high-level network.
To S -> AA is trained to maximize the likelihood such that 0" arg max, Ex=1 log To (a. where N the number of provided state-action pairs. We denote the loss function of this optimization problem as (0, D), and the general objective of meta imitation learning problem is:
min Σ
LBC (N,, D.
(1)
where ); 0 (oLBC D.T,, and a is a hyper-parameter which represents the inner-update learning rate.
3.2 Dual Meta Imitation Learning (DMIL)
In this work we assume at each time step t, the robot may switch to different sub-skills to accomplish the task. We define the sub-skill category at each time step t as 2 1, 1..... K, where K is the maximum number of sub-skills. We assume a successful trajectory Tij of task T; is generated from several (at least one) sub-skill policies, i.e., Tr Σ [s,, (x)))==))) where TE represents the expert policy. Our goal is to learn a hierarchical structure from multi-task demonstrations (Di,... } in an unsupervised fashion. The high-level network Te, that parameterized by On determines the sub-skill category & at each time step 4, and the 2-th sub-skill among K different sub-skills Te,, ..... Teix will be called predict the corresponding action de of state st. where the hat symbol denotes the predicted result. We use Ao and \y1. ... Au to represent the adapted parameters of 0n and f...... 0ik respectively. We condition the high-level network only on states, i.e., & = To, (s:). to fit the actual situation at the testing phase.
DMIL aims first fine-tune both no, and To.: ... Toix and then meta-update them. In task, Te, may not provide correct sub-skill categories as stated in the introduction. However, sub-skills still retain the ability to give out supervision for the high-level network with knowledge learned from previously learned tasks and few-shot demonstrations. This is because most robot manipulation tasks made up of a set of shared basis skills like reach, push and pick-place. As shown on the left side of fig. 2, the sub-skill that gives out the closet de to ae can be seen as supervision for the high-level network classify s, into this sub-skill. On the other hand, the adapted high-level network can classify each data point in provided demonstrations to different sub-skills for them to perform fine- tuning, as shown on the right side of fig. In summary, DMIL contains four steps for one training iteration. We call them Ingg--nner-Updatee (HI), Low-Inner-Update (LI), High-OOterrUppee (HO), and Low-Outer-Updatt (LO), which represents the fine-tuning and meta- updating process of the li-le networks respectively. The key problem is how to arrange these optimization orders to ensure convergence. We first introduce these steps formally here, then discuss why they can achieve convergence in the next section. The whole procedure is summarized in algorithm in appendix.
HI: For each sampled task Tú sample the first batch of trajectories {Tu} from DJr The principle of this step is to use sub-skill that can predict the closest action to the expert action to provide self-supervised category ground truths for the training of high-level network, which is classifier in form. We make every state-action pair passed directly to each sub-skill and compute LBC Ts1), k = 1,.... K, and choose the ground truth at each time step as the sub-skill category k that minimizes EBc(Au. (st, (t))
1, if h: = arg min, LBC(O (8t, 0t))
p(z,1e= = k) =
(2)
0, else

---

Then we get predicted sub-skill categories from the high-level network: 2/10 Teh, (st), and use cross-entropy loss to train the high-level network:
Ta
K
Ln Tan
p(zi log pogip( = 1t).
(3)
Finally we perform gradient descent on the high-level network and get A₁ = On - aVo, Cn Ta).
Note f...... 8b are freezed here.
LI: We sample the second batch of trajectories {ri)} from D The adapted high level network TA will process each state in T22 to get ukbls category 2i2: x, (s,) each time step. thus get K separate data sets for different sub-skills: 'D2 = {812t, 4:2: ( 2/2: = k). k = 1,... K. Then we compute the adaptation loss for each sub-skill with the corresponding dataset. In case of continuous action space, we assume that actions belong to Gaussian distributions, so we have:
N2
(Bc( D2k) = No
1
Σ 1=1
(e - (st))²
(4)
where N& is the number of state-action pairs in D2R- Finally we perform gradient descent on sub-skills and get Ay+ = 8ik a 0.2 BBc D2x). Note SAA is frozen in this process.
HO: We sample the third batch of trajectories { } from D and get [(A,, Taa) as in the HI process. Then it to compute the meta-gradient Vo, ((),; Tas) which equals to:
VAn [(A,: ria) A 00, -a (()) Va,dn.
(5)
LO: We sample Tia and get C(Au, Dak), ko = 1. K as in the LI process, then we use it to compute the meta-grrdieett Von, [(Ax, Das) which equals to:
Exi.f(Au DAr Dat) Ixu==il-- Di)) Vo, Avk-
Note after the training of m tasks, we average all meta-gradients from m tasks and perform gra- dient descents the initial parameters together to update high-level parameters 6% On
B2 m7 -- V 0y L(),, Tra) and sub-skill policies parameters 8% = 6t, 3 3 =i=1 Voa C(vx, TM)).
K, i.e., we do not update them at step 5 and 6. This is crucial ensure convergence.
For testing, although our method needs two batches of trajectories for one round of adaptation, in practice find only using one trajectory to perform HI and LI also works well in new tasks, thus DMIL can satisfy the one-shot imitation learning requirement. Besides the above process, we also add auxiliary loss to better drive out meaningful sub-skills to avoid the excessively frequently switching between different sub-skills along with time. Detailed information can be found in C.
4 Theoretical Analysis
In this section, we show the above algorithm can converge by rewriting both MAML and DMIL as hierarchical variational Bayes problems to establish the equivalence between them since the convergence of MAML can be proved in Fallah et al. [17]. Proofs of all theorems in Appendix D.
4.1 Hierarchical Variational Bayes Formulation of MAMI.
According to [14], MAML is a hierarchical variational Bayes inference process. The general meta- learning objective (1), which can be rewritten C, L(O, (o, (mm) log II p (P) can be formulated as follows:
Cgz
(x)lllo 0) E (og
log ((0;; A.)]}.
where á,,i 1.... m represent the local latent variables for task Tr, and A.. AM are the variational parameters of the approximate posteriors over .1, , &MM- We denote X; as (D, 0) and p(0; (D;, 0) as p( 6) to mean that X; and di are determined with prior parameters @ and support data D First we need minimize (L(g(06 (p) 8)) w.n.t X, According to D.2, we have:
(D 0) arg max (q(des) (D) [d+) KL(g(0n (=)lp(((:)0),
X;
We can establish the connection between 8 and the inner loop in MAML by the following Lemma:
Lemma In case ((;;; A;) is Dirac-delta function and choosing Gaussian prior for p(d: (0), equation 8 equals to the inner-update step of MAML, that is, maximizing log p(D!)) w.r.t. X; by carly-stopping gradient-asee asscett with choosing μi initial point:
Ax 0) μr a V logp (D/ le=we
(9)
5

---

Then we need to optimize [(0, A1. . w.r.t. 8. Since we evaluate (D, 0)) with only Di we assume pD, A, 0)) = p(D,al) (D, 0)). We give out the following theorem to establish the connection between the meta-update process and the optimization of Egg
Theorem In case that 2ig - 0t, i.e., the uncertainty in the global latent variables 0 is small, the following equation holds:
M
VoL = -E Xx log lD (A) Vo) 0).
(10)
A general EM algorithm will first compute the distribution of latent variables (E-step), then optimize the joint distribution of latent variables and trainable parameters (M-step), and the likelihood of data can be proved to be monotone increasing to guarantee the convergence since the evidence lower bound of likelihood is monotone increasing. Here oi, i = I,.... .M are the latent variables, and 0 corresponds to the trainable parameters. Lemma and Theorem I correspond to the E-step and M-step respectively. In the following part we establish the equivalence between with and 4, and between 10 with 5 and 6 to prove the equivalence between DMIL and MAML.
4.2 Modeling DMIL with Hierarchical Variational Bayes Framework
For simplicity. here only derive in one specific task T,, since derivatives of parameters from multi-task can directly add up. We first establish the connection between the maximization of log p ( (0,, O1,,.... .8ix) with the particular loss functions in DMIL:
Theorem In case of p(a,) se, f1x) ~ N (191 we have:
Ve, log p (D."" 10,, Ou...... 0ix) = Vo, Coh,, D(-),
(1I)
and
Von. log (D 9m, 8ix) = Voie Cnc( D2s),
(12)
where k = 1,... K. Note in 12, D2s corresponds to data sets determined by the adapted high level network Ap. and this connects with and 4 in DMIL. According 8, finding A equals to
maximize log (D!" (0) in specific conditions, and here in Theorem 2, we prove that maximize and in DMIL. Thus theorem corresponds to the E- step of DMIL, where we take Ta and 722 as DI',, and optimize arg max (e [log p( (D [D)] (L(((0;: (=) with coordinate descent method, which can be proved to be equal to 9 in D.5.
log p(DA (0,, An,,"' (ix) corresponds
For the M-step, we take T33 and T14 as Drai, According to Theorem 1, we can take the derivative of Ash 'an.... Aak to maximize the joint distribution of latent variables and trainable parameters to maximize the likelihood of dataset, so we have:
V Ven,, 8. log (DD
Paan Aa) = V log p(D."al (1a) * Ve, Au (D. ,0)),
(13)
AA log p p(Drol Da) Ve 0,)
This is exactly the gradients computed in HO and LO steps. Note this computation process can be automatically accomplished with standard deep learning libraries such as PyTorch [51]. To this end, we establish the equivalence between DMIL and MAML, and the convergence of DMIL can be proved.
where 0n = 10.1. 0sk]" and Au [Aii, A \
For clearer comparison, MAML is iterative process of > 0', and DMIL is iterative process of On, 0, - Ao. O - An, A - 0/2 0r. where the posterior estimation stages On, 02 - An, 0 An, > has no effect on parameters 0,, O,, thus can be divided to steps in DMIL. This decoupled fine-tuning fashion is exactly what we need to first adapt the high-level network and then adapt sub-skills. If we end-to-end fine-tune parameters like 0n 6 - An Ai. sub-skills will receive supervisions from an unadapted high-level network, which may provide incorrect classifications. Different this, the meta- meta-updating process An, A - *8,, 6f must be done the same time, since if we update f,, and 6 successively, the later one will receive different derivative (for example, Vo, log p(D) [8% Aa) from derivatives in MAML (Vo, log ((D)al) Dih Aa)), and the equivalence would not be proved.
5 Experiments
In experiments we aim to answer the following questions: (a) Can DMIL successfully transfer the learned hicrarchical structure to new tasks with few-shot new task demonstrations? (b) Can DMIL achieve higher performance compared to other few-shot imitation learning methods? (c) What are the
effects of different parts in DMIL, such as the skill number K, the bi-level meta- meta-lcarning procedure, and the continuity constraint? Codes and video results are provided in supplementary materials.

---

Table 1: Success rates of different methods on Meta-world environments with K 3. Each data point comes from 20 random seeds.
ML10
ML45
Meta-training
Mets- Meta-testing
Meta training
Meta-testingg testiitt
Methods
I-shot
3-shot
I-shot
3 3-shot
1-shot
3-shot
I-shot
3-shot
OptioGAAlI.
0.015
0.952:0.016
0.241
0.640::0.025
0.506--0.008
0.715:000066
0.220:10.013
0.481=000100
MIL
0.776
0.86920.029
0.36110.040
0.689100032
0.58410.011
0.74510.017
0.205510.024
0.510:10.005
PEMIRL
0.598=0.023 0,10000.077
0.162:0.003
0.256:20.009
0.289=0.051
0.39620.024
0.105=000005
0.126=000088
MLSH
0.506_10.134
0.725
0.106.10.032
0.135.10.009
0.235:10.091
0.2951000211
0.050.1.000
0.050.10.000
DMIL
0.775=0.010
0.949:0.009
0.396:10.016
0.710:00021
0.590:0.011
0.859
0.37620.004
0.640=0.009
Table 2: Cumulative rewards of different methods on four unseen tasks in Kitchen environment with K 4. Boldface indicates excluded objects during training.
Task (Unseen)
FIST-no-FT
SPiRL
(Mul((uurs
Microwave, Kettle. Top Burner, Light Switch
2.0 + 0.0
2.1 + 0.48
1.510.48
Microwave, Bottom Burner, Light Switch, Slide Cabinet
0.0 0.0
2.3 0.49
2.35
Microwave, Kettle, Hinge Cabinet, Slide Cubinet Microwave, Kettle, Hinge Cabinet, Slide Cabinet
1.0 0.0
1.9 : 0.29
3.15
2.0
3.3 + 0.38
2.95=10:44
5.1 Environments and Baselines
We choose to evaluate DMIL on two representative robot manipulation environments, The first one is Meta-world benchmark environments [2], which contains 50 diverse robot manipulation tasks, as shown in fig. 6 and fig. 7. We use both the ML10 suite (10 meta- veea-ttainnng tasks and 5 meta- meta-testing tasks) and ML45 suite (45 meta-training tasks and testing tasks) to evaluate our method, and collect 2K demonstrations for each task. We choose Meta-world since we think large scale of diverse manipulation tasks can drive semantic skills. We use the following approaches for comparison in this environment: OAlion-GAlL: hierarchical generative adversarial imitation learning method to discover options from unsegmented demonstrations [8]. We use Option-GAIL to evaluate the effect of meta- meta-lcarning in DMIL. MIL: transformer-basee meta imitation learning method [40]. We use MIL to evaluate the effect of hierarchical structures. MLSH: the meta-learning shared hicrarchies method [45] that relearns the high-level network in every new task. We use MLSH to evaluate the effect of fine-tuning (rather than relearning) the high-level network in new tasks. PEMIRL: a contextual meta inverse RL method which transfers the reward function in the new tasks [42]. We use PEMIRL to show DMIL can transfer to new tasks that have significantly different reward functions.
The second one is the Kitchen environment of the D4RL benchmark [18), which contains five different sub-tasks in the same kitchen environment. The accomplishment of each episode requires sequentially completions of four specific sub-tasks, as shown in fig. 9. We use an open demonstration dataset [48] to train our method. During training. we exclude interactions with selected objects and at test time we use demonstrations that involve manipulating the excluded object to make them unseen tasks. We choose this environment to show DMIL can be used in long-horizon tasks. We use two approaches for comparison in this experiment: SPiRL: extension of the skill extraction methods to imitation learning over skill space [37]; FIST: an algorithm that extracts skills from offline data with inverse dynamics model and distance function [36].
We use fonlecconnected neuron networks for both the high-level network and sub-skills. More details of experiments can be found in appendix F.
5.2 Results
Table 1 shows success rates of different methods in ML and ML45 suites with sub-skill number K 3. We perform -shoo and 3-shot experiments respectively to show the progressive few-shot performance of different methods. DMIL achieves the best results in ML10 testing suite and MI training and testing suites. OptionGAIL achieves high success rates in both ML10 and ML45 training suites. These results show the adequate capacity of hierarchical structures to fit potential multi-moda behaviors in multi-task demonstrations. MIL achieves comparable results for all meta-testing tasks but is worse than DMIL. This shows the necessity of meta-learning processes. Compared to them, PEMIRL and MLSH are mediocre among all suites. This comes from that the reward functions across different tasks are difficult transfer with few shot demonstrations, and the releareed high-level network of MLSH damages previously learned knowledge. We also illustrate t-sne results of these methods in fig. 4(a) to further analyze them in appendix E.2.
Table shows the rewards of different methods on four unseen tasks in the Kitchen environment. FIST-no- no--no-FF FT refers to a variant of FIST that does not use future state conditioning, which makes the
7

---

0.50
0.35
0.00
120
s4o
Figure 3: The iterative meta-learning process DMIL at each iteration. Left: the supervision of high-level network (sub-skill category) comes from the most accurate sub-skill. Right: the sub-skill updated at current step is determined by the fine-tuned high-level network.
comparison fairer. DMIL achieves higher rewards on two out of four tasks and comparable results on the other two tasks, which exhibits the effectiveness of the bi-level meta-training procedure. The poor performance of DMIL on the first task may come from the choice of skill number K or from low-quality demonstrations. We perform ablation studies on K in the next section.
Fig. shows curves of sub-skill probabilities along time of two tasks window-close and peg-insert- side of Meta-world and the werrowave-kette--eee burner-light task in Kitchen environment. We can see the activation of sub-skills shows a strong correlation to different stages of tasks. In first two tasks, Thp activates when the robot is closing to something, Te., activates when the robot is picking up something, and 2 activates when the robot is manipulating something. In the third task, T activates when the robot is manipulating the microwave, Th activates when the robot is manipulating the kettle or the light switch, and Thi activates when the robot is manipulating the burner switch. This shows that DMIL learns semantic sub-skills from unsegmented multi-task demonstrations.
5.3 Ablation Studies
Table 3: Success rates of different sub-skill number Meta-world environments.
In this part, we perform ablation studies on dif- ferent skill numbers K. Due to the limited space, we put ablations on different meta-training processes, continuity constraints, and hard/soft EM choices in appendix E.
MI.10
MI.4
steps,
training 1-shot
Meta testing
craining 3-thot
Meta testing 3-chot
3-shot
1-shor
3-shot
1-shor
0.76
0.955
0.32
0.72
0.563
0.818
01.4
0.67
0,775
0.939
0.396
0.71
0.59
0.859
0.376
0.64
0.795
0.94
0.52
0.57
0.713
0.92
0121
0.48
Effect of different skill number K: Table
0,8
shows the effect of different sub-skill number K in Meta-world experiments. We can that larger K can lead to higher success rates on meta-training tasks, but a smaller K can lead to better results on meta-testing tasks. This tells us that an excessive number of sub-skills may result over-fitting on training data, and a smaller can play the role of regularization. In Kitchen experiments, we can see similar phenomenons in Table 5, is worth noting both environments, we did not encounter collapse problems, i.e., every sub-skill gets well-trained even when K = 8 in kitchen environment or K 10 in Meta-wort world environments. This is because more sub-skills can help the whole structure get lower loss in the meta- meta-training stage. However, in our supplementary videos, we can see that sub-skills trained with a large K (for instance, K = 10 Meta-world environments) are not as semantic as sub-skills trained by a small K (for instance, K = 3 in Meta-world environments) during the execution of task.
Limitations
The limitations of DMIL come from several aspects, and future works can seek meaningful extensions in these perspectives. Firstly, DMIL models all tasks as bi-level structures. However, in real-world situations, tasks may multi-level structures. One can extend DMIL multi-level hierarchical structures like done in recent works [49]. Secondly. DMIL does not capture temporal information in demonstrations. Future state conditioning Hakhamaneshi al. [36] seems an effective tool to improve few-shot imitation learning performance in long-horizon tasks such as in the Kitchen environments. Future works can employ temporal modules such as transformer [50] as the high-level network of DMIL to improve its performance.

---

Acknowledgments
We would like to thank Tianren Zhang and Haichuan Gao for their insightful comments of the whole work, and Qualcomm China WRD UR Program. This work was supported by the National Natural Science Foundation of China 62176133.
References
[1] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning, 2017.
[2] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In 3rd Annual Conference Robot Learning, 2019.
[3] Y. Zhu, Z. Wang. J. Merel, A. A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Robolics: Science and Systems XIV, 2018.
[4] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: example-guided deep reinforcement learning of physics- based character skills. ACM Trans. Graph., 2018.
[5] S. Ross, G. J. Gordon, and D). Bagnell, A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011.
[6] C. Daniel, H. van Hoof, J. Peters, and G. Neumann. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 2016.
[7] S. Krishnan, R. Fox, I. Stoica, and K. Goldberg. DDCO: discovery of deep continuous options for robot learning from demonstrations, In 1st Annual Conference Robot Learning, 2017.
[8] M. Jing, W. Huang, F. Sun, X. Ma, T. Kong, C. Gan, and I.. Li. Adversarial option-aware hierarchical imitation learning. Proceedings of the 38th International Conference on Machine Learning, 2021.
[9] H. M. Le, N. Jiang, A. Agarwal, M. Dudik, Y. Yue, and H. D. III. Hierarchical imitation and reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, 2018.
[10] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. 2018.
[11] T. Yu, C. Finn, S. Dasari, A. Xie, T. Zhang, P. Abbeel, and S. Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. In Robotics: Science and Systems XIV, 2018.
[12] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-lcarning. 1st Annual Conference on Robot Learning. 2017.
[13] Yu, P. Abbeel, S. Levine, and C. Finn. One-shot hierarchical imitation learning of compound visuomotor tasks. arXiv preprint 1800: 1043, 2018.
[14] S. Ravi and A. Beatson. Amortized bayesian meta-lcarning. In 7th International Conference on Learning Representations, 2019.
[15] C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, 2018
[16] Y. Zou and X. Lu. Gradient-em bayesian meta-learning. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems, 2020.
[17] A. Fallah, A. Mokhtari, and A. E. Ozdaglar. On the convergence theory of gradient- -baed model-agnostic meta-learning algorithms. The 23rd International Conference on Artificial Intelligence and Statistics, 2020.

---

[18] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driver reinforcement learning. arXiv preprint arXiv:2004.00 07219, 2020.
[19] C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In 5th International Conference on Learning Representations, 2017.
[20] X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine. MCP: learning composable hierar- chical control with multiplicative compositional policies. In Advances in Neural Information Processing Systems, 2019.
[21] L. Liu and J. K. Hodgins. Learning to schedule control fragments for physics-based characters using deep q-learning. ACM Trans. Graph., 2017.
[22] Merel, A. Ahuja, Pham, Tunyasuvunakool. S. Liu, D. Tirumala, N. Heess, and G. Wayne. Hierarchical visuomotor control of humanoids. In 7th International Conference on Learning Representations, 2019.
[23] R. S. Sutton, D. Precup, and P. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 1999.
[24] P. Henderson, W. Chang, P. Bacon, D. Meger, J. Pineau, and D. Precup. Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[25] Y. Li, J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. In Advances Neural Information Processing Systems, 2017.
[26] M. Sharma, A. Sharma, N. Rhinehart, and M. Kitani. Directed-infe GAIL: learning hierarchi- cal policies from unsegmented demonstrations using directed information. In 7th International Conference on Leurning Representations, 2019.
[27] S. Lee and S. Seo. Learning compound tasks without task-specific knowledge via imitation and self- learning. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[28] Ho and Ermon. Generative adversarial imitation learning. Advances in Neural Informa- tion Processing Systems, 2016
[29] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, 2016.
[30] M. J. Hausknecht and P. Stone. Deep reinforcement learning in parameterized action space. In 4th International Conference on Learning Representationss 2016.
[31] G. Neumann, W. Maass, and J. Peters. Learning complex motions by sequencing simpler motion templates. In A. P. Danyluk, L. Bottou, and M. L. Littman, editors, Proceedings of the 26th Annual International Conference on Machine Learning, 2009.
[32] R. A. Jacobs, M. Jordan, J. Nowlan, and G. F. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.
[33] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In 3rd Annual Conference on Robot Learning, 2019.
[34] T. Haarnoja, K. Hartikainen, P. Abbeel, and S. Levine. Latent space policies for hierarchical reinforcement learning. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, 2018
[35] K. Hausman, J. Springenberg. Z. Wang, N. Heess, and M. A. Riedmiller. Learning an embedding space for transferable robot skills. In 6th International Conference on Learning Representations, 2018
[36] K. Hakhamaneshi, R. Zhao, A. Zhan, P. Abbeel, and M. Laskin. Hierarchical few-shot imitation with skill transition models. 2021.
10

---

[37] B. Price and C. Boutilier. Accelerating reinforcement learning through implicit imitation. Journal of Artificial Intelligence Research, 19:569-629 2003.
[38] S. Nasiriany, H. Liu, and Y. Zhu, Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. arXiv preprint arXiv:2110003655, 2021.
[39] Y. Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and
W. Zaremba. One-shot imitation learning. In Advances in Neural Information Processing Systems, 2017.
[40] Т. Cachet, J. Perez, and S. Kim. Transformer- meta-imitation learning for robotic manipu- lation. In Neural Information Processing Systems, Workshop on Robot Learning, 2020.
[41] K. Xu, E. Ratner, A. D. Dragan, S. Levine, and C. Finn. Learning a prior over intent via meta- veeta-inverse reinforcement learning. In Proceedings of the 36th International Conference Machine Learning, 2019.
[42] L. Yu, T. Yu, C. Finn, and S. Ermon. Meta-inverse reinforcement learning with probabilistic context variables. In Advances in Neural Information Processing Systems, 2019.
[43] R. Vuorio, S. Sun, H. Hu, and J. Lim. Multimodal model-agnostie meta-learning via task- aware modulation. In Advances in Neural Information Processing Systems, 2019.
[44] F. Alet, T. Lozano-Pérez, and L. P. Kaelbling, Modular meta-learning In 2nd Annual Conference on Robot Learning, 2018.
[45] K. Frans, J. Ho, X. Chen, Abbeel, and J. Schulman. Meta learning shared hierarchies. In 6th International Conference on Learning Representations, 2018.
[46] H. Yao, Y. Wei, J. Huang, and Z. Li. Hierarchically structured meta-learning. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[47] C. G. Atkeson and S. Schaal. Robot learning from demonstration. In Proceedings of the Fourteenth International Conference on Machine Learning, 1997.
[48] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: 1910. 1956. 2019.
[49] T. Shu, C. Xiong, and R. Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In 6th International Conference on Learning Representations, 2018.
[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[51] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, gegh-performance deep learning library. Advances in neural information processing systems, 2019.
[52] E. Grant, C. Finn, S. Levine, T. Darrell, and T. L. Griffiths. Recasting gradient-based meta- learning as hierarchical bayes. In 6th International Conference on Learning Representations, 2018.
[53] R. J. Santos. Equivalence of regularization and truncated iteration for general ill-posed problems. Linear algebra and its applications, 1996.
[54] D. Duvenaud, D. Maclaurin, and Adams. Early stopping as nonparametric variational inference. In Artificial Intelligence and Statistics. PMLR, 2016.
[55] R. Müller, S. Kornblith, and G. Hinton. When does label smoothing help? arXiv preprint arXiv: /906. /906..069. 2019.
[56] Kostrikov, K. K. Agrawal, Levine, and J. Tompson. Addressing sample inefficiency and reward bias in inverse reinforeement learning, arXiv preprint (0X.:::809900923.. 2018.
11

---

[57] S. K. S. Ghasemipour, S. Gu, and R. S. Zemel. Smile: Scalable meta inverse reinforcement learning through context-conditional policies. In Advances in Neural Information Processing Systems 32:, 2019.
12


Reviews:
[{"review_id": "YZImRrt8M4c", "reviewer": "Reviewer_dgTJ", "summary": "This paper tackles the following two problems (simultaneously): \n- Given a dataset of demonstrations from many different tasks, learn a set of hierarchical policies that solve these tasks well.  \n- Learn the hierarchical policy above in such a way that when given a new task, you can quickly adapt both the high-level and low-level policies to this task. \n\nThe former problem has been explored in the hierarchical imitation learning (HIL) literature, while the latter problem has been well-explored in the meta-imitation learning (MIL) literature. The main contribution of this paper is to address both of these problems at the same time. The proposed method combines an EM-style algorithm (popular in HIL) with MAML (popular in MIL). Experiments are performed on simulated robotics tasks, and show results that are decent (method works), but not stellar (comparable performance to competing methods on 2 out of the 3 benchmarks). \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper tackles an important problem (transferring subskills across tasks), that I believe is somewhat under-studied in the hierarchical imitation learning literature. \n- The paper is well-written. \n- Authors demonstrate results on manipulation tasks, which is nice to see (a lot of hierarchical learning papers end up focusing on locomotion / navigation tasks, where the hierarchical structures are easier find). \n\nWeaknesses\n- The link provided for code and demonstrations points to an expired repository: https://anonymous.4open.science/r/DMIL\n- “DMIL achieves higher rewards on two out of four tasks and comparable results on the other two tasks, which exhibits the effectiveness of the bi-level meta-training procedure.” – I am not sure if this is a fair assessment. The performance for DMIL is only higher for one task (the third task), and the results are comparable for the rest of the three tasks. The confidence intervals appear to be quite large for both methods. Similarly, for ML10, DMIL results are comparable to vanilla MIL. Overall, I would say that the empirical performance is decent (the method works), but not stellar (it’s only significantly better for ML45, one out of the 3 benchmarks studied in the paper). \n- Most meta-world tasks are fairly short (pick + place tasks, or turning a lever), so it’s unclear to me if a hierarchical method is actually needed for these tasks. As shown in the results K=2 works best for meta-world tasks. When more options are assigned, they typically go unused (Figure 2). The Kitchen tasks are longer, but the performance when compared against competing methods is not particularly stellar.\n- The meta-imitation problem setting is well-suited to real world learning (since it can be done entirely offline), so it's a bit disappointing that the paper does not have any real robot results. Perhaps the proposed meta-learning algorithm has stability issues when working under real world constraints such as visual observations? If that is the case, it should be mentioned in the limitation sections. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, while the paper has weaknesses, I think it's technically solid, and targets an important problem, so I am leaning towards acceptance. Note that I did not check the theoretical convergence results (the intuitive derivation combining EM and MAML made sense to me). ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please take a look at the weaknesses, and use them to either update the paper or expand the limitations section. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "WuOL_SjetG6", "reviewer": "Reviewer_iytY", "summary": "Paper proposes a new method for meta-learning hierarchical imitation learning, provides theoretical analysis of the algorithm showing its convergence, and demonstrates SOTA results on the meta-world benchmark and competitive results on the D4RL kitchen environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Nicely written and easy to understand\n- Provides both theoretical analysis of the algorithm showing its convergence, and empirical results showing its benefits to prior methods\n- Includes exhaustive ablation studies showing the importance of each component of the algorithm\n\nWeaknesses\n- The proposed method is rather complex, which may make it difficult to be deployed on real systems.\n- It would be nice to include some intuitive discussion on why *both* the high level and low level components need to adapt to a new task? Since many different subskills can be represented with the low level component, it seems like maybe only the high level may need to adapt given a new task, by picking the right sequence of subskills? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper provides an interesting new method and demonstrated superior empirical results compared to prior work. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See weaknesses section.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "OVVQZqYSM7w", "reviewer": "Reviewer_ZhJq", "summary": "Use MAML-style gradient-based meta-learning to learn a hierarchical structure from imitation learning. This requires an inner loop that fine-tunes the controller and the sub-skills and an outer loop that updates the meta-gradients. The order is from inner to outer, where the first step decides the assignment of sub-skills and assigns loss based on the behavior cloning loss of that sub-skill (log-likelihood). Then the second inner loop updates the sub-skills based on the assignments given by the controller to minimize behavior cloning. The outer loop then updates the meta-gradients. This work also demonstrates that the inner and outer gradients are equivalent to E and M steps for coordinate descent, implying that the algorithm will coverge to a local minima.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis work provided an interesting algorithm, clever implementation and good experiments\n\nWeaknesses:\nThe writing is difficult to follow, using ill-defined terms like 23 \"plenty\", or 32 \"come from that the\" or \"may conflict with descrpant situations\". As it is, the second paragraph in the intriduction is difficult to understand and after several readings, it is still hard to determine exactly what it is saying. The third paragraph has the same issue, with statements like \"the same question applies to the high-level network\", when referring to what high-level network that low level sub-skills should choose, which does not appear to make sense. It was difficult to gain more insight from the introduction than what was stated in the abstract.\n\nIt is not entirely clear from the methods section how the meta-gradients actually work, and considering that is the main point of the algorithm (the rest is similar in concept to the EM algorithms that generally get used), that is somewhat disappointing. The proof is difficult to follow and thus to verify, though it appears intuitive enough: by making it equivalent to the EM algorithm, the algorithm will converge.\n\nOverall, the choice of meta-world for tasks is good and the performance appears to be quite effective, especially with 45 tasks. However, it does seem a little bit suspect that 2 sub-skill performed best, considering that it seems like that implies that two general behavior patterns solve 45 different tasks. Would 1 task have performed even better? Either way, it seems like the MAML component may be where the performance comes from, rather than the skill learning, and some non-hierarchical change in architecture resulted in better training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper provides an insightful algorithm that seems like it has the potential to be built upon and evaluated in a classic benchmark with good results. While some of the specifics of the algorithm, such as the way that the number of sub-skills appears not to scale, cast some doubt on the algorithm or the evaluation, for the most part, it appears to be a valuable contribution. I was not particularly impressed by the proof, but that might be more related to my background than to any limitations with it. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Most of the issues are in the weaknesses section, but mostly the paper could benefit from having some more extensive experiments related to determining why such little variation is needed in the number of skills. In addition, the writing was overall difficult to follow and could be improved by some editing."}}, {"review_id": "2TYjX4OdRhY", "reviewer": "Reviewer_aty5", "summary": "This paper proposes a new hierarchical imitation learning structure based on MAML to learn modular sub-skills and then transfer them to new tasks. Experiments on meta-world tasks and the Kitchen environment show the proposed method outperforms previous skill-based imitation learning methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) This paper is well-written and easy to follow.\n2) The proposed method is verified both theoretically and empirically.\n3) Extensive experimental results and ablation studies show the advantage of the proposed method and the contribution of each component.\n\nWeaknesses:\n\n1) The structure of the paper could be improved to cover more technical details, like the details of auxiliary loss and corresponding experimental results.\n\n2) Some references to the appendix are not clear enough, like in line 193, ‘Detailed information can be found in C.’ -> ‘Detailed information can be found in Appendix C.’ Similar in Line 205.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper clearly presents the proposed method and provides theoretical analysis as well as empirical analysis. I did not check the technical details of the theoretical analysis carefully. But based on my current understanding and if the authors address my concern, I would vote for acceptance of this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Please refer to the pros and cons part.\n\nI have one major concern about Equation 2 which matters whether the low level can learn meaningful sub-skills. Since all networks are initialized randomly, so why low-level policies could predict different actions given a state $s$? Why should the closest action be the supervised signal (ground truth?) to the high-level network? \n\nMinors,\nLine 173 ‘freezed’->’frozen’\n"}}]

Author rebuttal:

Decision: Accept (poster)