Transferring Hierarchical Structure with Dual Meta Imitation Learning
Authors: Chongkai Gao
Yizhou Jiang
Feng Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: imitation learning
meta learning
hierarchical structure
robot learning

Abstract: Hierarchical Imitation learning (HIL) is an effective way for robots to learn sub-skills from long-horizon unsegmented demonstrations. However, the learned hierarchical structure lacks the mechanism to transfer across multi-tasks or to new tasks, which makes them have to learn from scratch when facing a new situation. Transferring and reorganizing modular sub-skills require fast adaptation ability of the whole hierarchical structure. In this work, we propose Dual Meta Imitation Learning (DMIL), a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning. DMIL uses the likelihood of state-action pairs from each sub-skill as the supervision for the high-level network adaptation, and use the adapted high-level network to determine different data set for each sub-skill adaptation. We theoretically prove the convergence of the iterative training process of DMIL and establish the connection between DMIL and the Expectation-Maximization algorithm. Empirically, we achieve state-of-the-art few-shot imitation learning performance on the meta-world benchmark.

Under review as a conference paper at ICLR 2022
TRANSFERRING HIERARCHICAL STRUCTURE WITH DUAL META IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Hierarchical Imitation learning (HILL) an effective way for robots to learn sub- skills from long-howe long-horizon unsegmented demonstrations. However, the learned hi- erarchical structure lacks the mechanism to transfer across multi-tasks or to new tasks, which makes them have learn from scratch when facing Transferring and reorganizing modular sub-skills require fast adaptation ability of the whole hierarchical structure. In this work, we propose Dual Meta Imita- tion Learning (DMIL), a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning. DMIL uses the likelihood of acatin pairs from each sub-skill as the supervision for the high-level network adaptation, and use the adapted high- level network to determine different data set for each sub-skill adaptation. We theoretically prove the convergence of the iterative training process of DMIL, and establish the connection between DMIL and the Expectation-Maxiiizaatee Maximization algo- rithm. Empirically, we achieve state-of-the-arr few-shot imitation learning perfor- mance the meta-world Yuu et al. 2019b) benchmark and comparable results the Kitchen environment.
situation.
INTRODUCTION
Imitation learning (IL) has shown promising results for intelligent robots to conveniently acquire skills from expert demonstrations (Zhu et a0. et al., 2018). Nevertheless, acquiring skills from long-horizon unsegmented demonstrations has been a challenge for IL algorithms, because of the well-known issue of compounding errors (Ross et al. 201T). This one of the crucial problems for the application of IL methods to robots since plenty of manipulation tasks are long-horizon. Hicrarchical Imitation Learning (HIL) aims to tackle this problem by decomposing long-horizon of sub-skills employed accomplish specific
tasks with hierarchical model, in which parts of the long-horizon task, and a high-level network is responsible for determining the switching of sub-skills along with the task. Such hierarchical structure is usually modeled with Options (Daniel et al. 2016 Krishnan et al. 20117] Zing et al. [2221) or goal-conditioned structure (Le et al., 2018). HIL expresses the nature of how humans solve complex tasks, and has been considered be a valuable direction for IL algorithms (Osa et al. 2018).
However, most current HIIL methods have no explicit mechanism to transfer previously learned skills to new tasks with few-shot demonstrations. This requirement comes from that the learned hierarchi- cal structure may conflict with discrepant situations in new tasks. As shown figure [(a] both the high-level network and sub-skills need to be transferred to new forms to satisfy new requirements: the high-level network needs new manners to schedule sub-skills in new tasks (for example, calling different sub-skills at the same state), and each sub-skill needs to adapt to new specific forms in new tasks (for example, grasping different kinds of objects). Quickly adapting to new tasks can signifi- cantly increase the generalization ability of HILL methods and make them be applied to wider range of scenarios, thus an appropriate approach needs to be exploited to endow HIL with such kind of ability with sho- new task demonstrations.

---

Under review as a conference paper at ICLR 2022
A daptable Monolithic Network
trained high-level policy Call pull this state"
new manners of high-level seigh-uuulshheulligg 'ail push in this state
One Stagl
transfer
Demonstiaion
Mulsi task
Demonddraions
Fine tine
META
drower open
Tew-shot
HIL
task
A daptable Hierarchical Structure
demonss
Qne-Shot Demonstraton
old task
demonstrations
Multi
trained skill
new forms of sub skill pice plane
D ecsonstrattoos
en
pickeplace
DMIL
transfer
8
012
M
pick piace
hommer
shelf pioce
basketboll
(a) Illustration the transfer problem of HIL in new tasks.
(b) Comparison of MIL and DMIL.
Figure I: (a) Both the high-level network and sub-skills need to be transferred to new tasks. Above: when the robot arm over half-open drawer, the task can be either opening or closing the drawer, which requires the high-level network to call different sub-skills. Below: same sublskill pick-place may exhibit different specific forms in new tasks. (b) DMIL aims to integrate MAML into HIL with novel iterative optimization procedure that meta-learns both the high-level network and sub-skills.
Recently, meta imitation learning (MIL) Yu et al. 2018b Finn et al. 2017b; Yu et al. 201 20) employ model-agnostic meta-learning (MAML) (Finn al. 2017a) to imitation learning procedure to enable the learned policy to quickly adapt to new tasks with few shot demonstrations. MAML first fine-tunes the policy network in the inner loop, then evaluates the fine-tuned network to update its initial parameters with end-to-end gradient descent at the outer loop. This inspires us to integrate MAML into HIL to transfer the hierarchical structure. However, most MIL methods meta-learn a monolithic policy in an end-to-end fashion, which does not conform to the iterative training of the bi-level structure in HIL, where the training supervision of each level comes from the other level. Integrating MAML into such a self-supervised bi-level optimization procedure is novel problem. The main challenge comes from how to schedule the inner loops and outer loops of the meta-learning process of bi-level modules in HIL to ensure convergence.
In this work, we propose a novel hierarchical meta imitation learning framework called dual meta imitation learning (DMIL) to successfully incorporate MAML into the iterative training process of HIL, as shown in figure [(b]] We make the high-level network and sub-skills as mutual supervision during the bi-level MAML process: the likelihood of each state-action pair in few-shot demonstra- tions of each sub-skill can provide the supervisions for the meta-training of high-level network, and the fine-tuned high-level network in turn determines the data sets for the meta-training of each sub- skill, like a general EM process. We further design an claborate training procedure for DMIL that first fine-tunes the high-level network and sub-skills in sequence inner loops, then meta-updates them simultaneously at outer loops. We theoretically prove the convergence of this special training procedure by leveraging previous results from Ravi & Beatson (2019); [Finn et al.) (2018); Zou & Lu (2020) to reframe both MAML and DMIL hierarchical Bayes inference processes and get the convergence of DMIL according to the convergence MAML Fallah et al. (2020).
We test our method on the challenging meta-world benchmark environments (Yu et al. 2019b) and the Kitchen environment of D4RL benchmarks (Fu et al. 2020). In our experiments, we successfully acquire set of meaningful sub-skills from a large scale of manipulation tasks, and achieve state-of- the-art few-shot imitation learning abilities in the MLA5 suite. In summary, the main contributions of this paper
follows:
We propose DMIL., a novel hierarchical meta imitation learning framework that meta-leams both the high-level network and sub-skills from unsegmented multi-task demonstrations.
We propose a special EM-like training algorithm to schedule the meta-learning and hierar- chical imitation learning processes in DMIL and theoretically guarantee its convergence.
We achieve state-of-the -rrt---rr- few-shot imitation learning performance on meta- world bench- mark environments and comparable results in the Kitchen environment.
2

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
2.1 HIERARCHICAL IMITATION LEARNING
Recovering inherent sub-skills contained in expert demonstrations and then reuse them with hier- archical structures has long been an interesting topic in the hierarchical imitation learning (HIL.) domain. According to whether there are pretraining tasks, we can divide HII. methods into two cat- egories. The first kind aims to manually design set of simple pretraining tasks that could encourage distinct skills or primitives, and then they learn a high-level network to master the switching of prim- itives to accomplish complex tasks (Florensa et al. 2017 Peng al. 2019) Liu & Hodgins 2017 Merel et al., 2019 Le et al., 2018). However, for unsegmented demonstrations where no pretraining tasks provided, which is the situation in our paper, these methods can not be applied.
The second kind of methods aim to learn sub-skills with unsupervised learning methods. Daniel et al. (016);; Krishnan et al. (017) acquire Options (Sutton et al. 1999) from demonstrations with an
procedure and use the Baum- Welch algorithm to estimate the parameters of different options. [enderson et al. (2018); Jing et al. (2021) integrate generative adversarial networks into option discovery process. Li al. 20 2017); et al. (2019) [Le & Sco 2020) incorporate generaatie-aavvessaiaal imitation learning (Ho & Ermon 2016) Tramework and an
coommation-theoreee metric (Chen et al. (2016) to simultaneously imitate the expert and maximize the mutual- inftraliifformationn between latent ski-skill categories and corresponding trajectories to acquire decoupled sub-skills. There are some methods called mixture-of-expe
(MoE) that compute the weighted sum of all primitives to get the action rather than only using one of them at each time step (Hausknecht & Stone 2016; Neumann al. |009) Jacobs et al. 1991). Other methods aim to seek an appropriate latent space that can map sub-skills into and then condition policy on the latent variable to reuse sub-skills (Lynch al. 2019: Haarnoja al. 2018 Hausman et al. 2018
Hakhamaneshi et al.. 2021 Price & Boutiller] 2003).
For transferring learned sub-skills, some work fine-tune the whole structure in new tasks (Hakhamaneshi et al.] (022). However, the performance of fine-tuning all depends on the gener- alization of deep networks, which may vary among different tasks and network designs.
2.2 META IMITATION LEARNING
Meta imitation learning, or one-shot imitation learning, leverages various meta-learning methods and multi-task demonstrations to meta-learn a policy that can be quickly adapted to a new task with few-shot new task demonstrations. [Dunn et al.] (2017); Cachet et al. (2020) employ self-attention modules to process the whole demonstration and the current observation to predict current action. Yu et al. 80): Finn et al. (017b)); Yu et al. (2018b) use mndee--ggostic meta-lcarning (MAML) (Finn et al. 2017a) to achieve one-shot imitation learning ability for various manipulation tasks with robot or human visual demonstrations, Xu et al. (011);; Yu et al. (2019a) propose to meta-learn a tasks and then use it to perform IRL in new tasks. However, they need downstream inverse reinforcement learning after the adaptation of reward functions, thus conflicts with goal of few-shot adaptation. Most of above methods only learn one monolithic policy, lacking the ability to model multiple sub-skills in long-horizon tasks. Some works aim tackle the multi-modal data problem in meta-lcarning by avoiding single parameters initialization across all tasks (Vuorio et al. 2019 Alet et al. 2018 Frans et al. 2018) Yao al. 2019), but they lack the mechanism to schedule the switching of different sub-skills over time. There are some works that also meta-learn a set of sub-skills in hierarchical structure (Yu jet al., 22188a] |Frans al., 2018), but they either use manually designed pretraining tasks or relearn the high-level network in new tasks, which is not appropriate in few-shot imitation learning settings.
robust reward function that be quickly adapted
3 METHOD
3.1 PRELIMINARIES
We denote a discrete-time finite-horizon Markov decision process (MDP) as a tuple (S, A, T, P, r, Po), where is the state space, A is the action space, T the time horizon,
3

---

Under review as a conference paper at ICLR 2022
High-level network
High-leve network
$2
Milliply with land salocting
En
Aro
ag
Sub deil colopcery with layfand protibiiity
C
Lx
Sub seill calocoay weth Law pobiiiiity
dea
dr1
r2
a
dg 1
a+2
4
Fieward
Sub
skill
Sub 2
Suil 0
Sub skill
Sub 2
Mota Louranng
Figure 2: The iterative meta-learning process of DMIL at each iteration. Left: the supervision of high-level network (sub-skill categories) comes from the most accurate sub-skill (the green one, sub-skill here). Right: the sub-skill updated at current step (the green one, sub-skill 0 here) is determined by the fine-tuned high-level network.
P S x A 8 - [0. 1] is the transition probability distribution, r : 888 A - R is the reward function, and Pu is the distribution of the initial state Su-
3.2 FORMULATION OF META IMITATION LEARNING PROBLEM
We firstly introduce the general setting of the meta imitation learning problem. The goal of meta imitation learning is to extract some common knowledge from a set of robot manipulation tasks (Ti) that come from the same task distribution p(T), and adapt new tasks quickly with few shot new task demonstrations. As in model-agnostic meta-learning algorithm (MAML) (Finn et al. 70) formalize the common knowledge as the initial parameter 0 of the policy network Ty that can be efficiently adapted with new task gradients.
For each task Ti p(T), a set of demonstrations D; is provided, where D; consists of N demonstration trajectories: D. = {Tıy)}= and Tay consists of sequence of state-action pairs:
Tr = {(E,, at) ]e), where Tiş is the length of Tij- Each D; is randomly split into support set D and query set Deai for meta-training and meta-testing respectively. During the training phase, we sample m tasks from p(T), and in each task Tú we use D to fine-tune лy get the adapted task- specific parameters A; with gradient descent, and then evaluate it with Diaa to get the meta- veta-graddeet of Tr and we optimize the initial parameters with the average of meta- meta-gradients from all m tasks. As in Finn al. (2017b) Duan al. (2017), we use behavior cloning (Atkeson & Schaal 1997) loss as our metrics for meta-training and meta-testing. It aims to train policy my : S - A that maximizes the likelihood such that 0 = arg max, E log (o(a; where N is the number of provided state-action pairs. We denote the loss function of this optimization problem as LB D), and the general objective of meta imitation learning problem is:
min 0
LBC (Ni, D.
(I)
where A; 0 aVoLBc(O, D. and is hyper-parameter which represents the inner-update learning rate.
3.3 DUAL META IMITATION LEARNING (DMIL)
In this work we assume each time step f, the robot may switch to different sub-skills accom- plish the task. We define the sub-skill category at each time step i as 1,..., K, where K the maximum number of sub-skills. We assume a successful trajectory Tij of task T gen- erated from several (at least onc) sub-skill policies, i.c., Tú = E' {( ((ss,,T(ss) {=)+)=)))} where represents the expert policy. Our goal is to learn a hierarchical structure from multi-task demon- strations {D,,... ... } in an unsupervised fashion. In our model, high-level network wo, that parameterized by 0n determines the sub-skill category & at each time step t, and the z-th sub-skill among K different sub-skills : Te.x will be called to predict the corresponding action de of state st, where the hat symbol denotes the predicted result. We use X and \t1, ... Au to represent the adapted parameters of 0 and f..... Oix respectively. In order to achieve fhw-shot learning ability in new tasks, we condition the high-level network only on states, i.c., & = xe, (st), since at the testing phase only have access to states and have access action information.
4

---

Under review as a conference paper at ICLR 2022
DMIL aims to first fine-tune both ne, and Téio.". R and then meta-update them. In a new task, To, may not provide correct sub-skill categories as stated in the introduction. However, sub-skills still retain the ability to give out supervision for the high-level network with knowledge learned from previously learned tasks and few-shot demonstrations. This is because most robot manipulation tasks made up of a set of shared basis skills like reach, push and pick-place. As shown in the left side of the sub-skill that gives out the closet di to Or can be seen as supervision for the high-level network to classify 8t into this sub-skill. On the other hand, the adapted high-level network can classify each data point in provided demonstrations to different sub-skills for them to perform fine- tuning, shown in the right side of figure 2 summary, DMIL contains four steps for one training iteration. We call them Brigh-Inner-Udaate (HI). Low-Inner- Update (LI), High-Outer- Update (HO), and Low-Outer- Uuw--uur--Updteee (L.O), which represents the fine-tuning and meta-updating process of the bi-level networks respectively. The key problem is how to arrange these optimization orders to ensure convergence. We first introduce these four steps formally here, then discuss how to schedule them in the next section. The whole procedure is summarized in algorithm I
HI: For each sampled task Tr, we sample the first batch of trajectories {Ta) from D The principle of this step is to use sub-skill that can predict the closest action to the expert action to provide self- supervised category ground truths for the training of high-level network, which is a classifier in form. We make every state-action pair passed directly each sub-skill and compute Lec Ta ) 1..... K, and choose the ground truth at each time step as the sub-skill category that minimizes CBc(0 (s,, ae):
p(z = k) = {
1, if k = arg min (Bc( (81;0t)) 0z))
(2)
0, else
Then we get predicted sub-skill categories from the high-level network: 2iie = Te, (st), and use cross-entropy loss to train the high-level network:
Ta
K plani
Ln (Oh: ra)
Σ
= -k) log = 1:).
(3)
Ts
I=1
Finally we perform gradient descent on the high-level network and get An. - Ca (Oh: Ta)) Note .r,, 0i are freezed here.
LI: We sample the second batch of trajectories {T1²] from D." The adapted high level network TX will process each state in T22 to get sublskill category 2/20 = TA, (s:) at each time step. thus we get K separate data sets for different sub-skills: D2x {S a.2:) 2:2: k]. k L. ... K. Then we compute the adaptation loss for each sub-skill with the corresponding dataset. In case of continuous action space, we assume that actions belong to Gaussian distributions, so we have:
N₂.
1
Lno(O D2x)
Σ
(a. To, (8:))
(4)
N
where N& is the number of state-action pairs in D2x. Finally we perform gradient descent on sub- skills and get Aue Oi a aVo,, 0z CBc(On. D22). Note TA, is frozen in this process.
HO: We sample the third batch of trajectories {TT)} from D.r and get L(),, Tis) as in the HI process.
Then we compute meta- gradient for fn with L(),, Tis) as follows:
Vg, L(A,. Tas) = V [()r, Tis (as))l,= =0, (( Vo, An-
(5)
LO: we sample Tua and get L()i DaK). k = I,... K as in the LI process, then we compute meta-gradis eeta-gradient for 6ik with L(Au, DaK) as follows:
Vo,, C(Aik, Dak)
Dak) Ix= Dik) Von Aue-
(6)
Note after the training of m tasks, we average all meta-gradients from m tasks and perform gra- dient descents on the initial parameters together to update high-level parameters 0' = On -- 3 8 Ve,, C(Xn, Tis) and sub-skill policies parameters 0/ Ou B Em Vel C(Au, Tu),
=, K, i.c., we do not update them at step and This is crucial to ensure convergence.
For testing, although our method needs totally two batches of trajectories for one round of adapta- tion, in practice we find only using one trajectory to perform HI and LI also works well in new tasks, thus DMIL can satisfy the one-shot imitation learning requirement. Besides the above process, also add an auxiliary loss to better drive out meaningful sub-skills to avoid the excessively frequently switching between different sub-skills along with time. Detailed information can be found in
5

---

Under review as a conference paper at ICLR 2022
4 THEORETICAL ANALYSIS
DMIL is a novel iterative hierarchical meta-learning procedure, and its convergence of DMIL needs be proved to ensure feasibility. As stated in the above section, what makes DMIL special that in HI and LI, we update parameters of each module immediately, but in LO and HO, we store this section, we show this can make
the gradients each part and update them simultaneously. DMIL converge by rewriting both MAML and DMIL as hierarchical variational Bayes problems to establish the equivalence between them, since the convergence MAML can be proved in Fallah ct al. (2020). Proofs of all theorems are in Appendix C
4.1 HIERARCHICAL VARIATIONAL. BAYES FORMULATION OF MAMI.
According to (Ravi & Beatson 2019),
MAML is hierarchical variational Bayes inference process. can be formulated as follows:
The general meta-learning objective
[(,, A1, -. (m)) = log
(D.
(KL(g(o;Aa) p(o: 0))
+ Eo(deid.) flog Dj, di(0) - log ( {:)]},
where i,,i 1,... represent the local latent variables for task To, and λ1. AM are the variational parameters the approximate posteriors over $1,-.. .bs- We denote X, as (D; 0) and p( (D; 0) as pDo, 0) to mean that X; and o; are determined with prior parameters @ and support data D First we need to minimize KL(g(0;; ( ()r,)) w.r.t. X According to
we have:
(D! 0) arg max (ogg [d)) (T ()lp(()0)))))
(8)
x
and we can establish the connection between and the fine-tuning process MAML by the follow- ing Lemma:
Lemma case ((,;; A²) is a Dirac-delta function and choosing Gaussian prior for p( (0), equa- tion 8 equals to the inner-update step of MAML, that is, maximizing log p(D(") w.r.t. X; by early- stopping gradient-ascent with choosing μsg as initial point:
(D 0) = μi + o logg (D! 10=p--
Then we need to optimize L(,, \1, .AM) w.r.t. O. Since we evaluate (DD, 0)) with only
Deal, we assume p(D; ( (D 0)) p(D:aal (D 0)). We give out the following theorem to es- tablish the connection between the meta-update process and the optimization of L(e, A1, Aas)
Theorem In case that Eg + 0+,, i.e., the uncertainty in the global latent variables 0 is small, the following equation holds:
M VeL(O,A1. ... AM) Σ
VA, log (Dg val (A)) VoA,
(10)
A general EM algorithm will first compute the distribution of latent variables (E-step), then optimize the joint distribution of latent variable and trainable parameters (M-step), and the likelihood of data can be proved to be monotone increasing to guarantee the convergence since the evidence lower bound of likelihood monotone increasing. Here ò;, 1.... M are the latent variables, and 0 corresponds to the trainable parameters. Lemma and Theorem I correspond to the E-step and M-step respectively. In the following part we establish the equivalence between with and and between 10 with and tto prove the equivalence between DMIL and MAML.
4.2 MODELING DMIL WITH HIERARCHICAL VARIATIONAL BAYES FRAMEWORK
For simplicity, here only derive in specific task Tr since derivatives of parameters from multi-task can directly add up. We first establish the connection between the maximization of log p(D/" (0,, ..,.... 0ix) with the particular loss functions in DMIL:

---

Under review as a conference paper at ICLR 2022
Theorem In case of p(arlse. B(x)) (ponu(se), g²), we have:
Ve, log (D." " (0h, Ou..-.. 0ik) = Vol Co
(1I)
and
01. log [0,, [n,, On, ,0k) - Va VoyhCn Ln D2x), -,, 1,... K.
(12)
Note in 12 D2x corresponds to data sets determined by the adapted high level network An. and
this connects with 3 and 4 in DMIL. According to g finding X equals maximize log (D (0) in specifie conditions, and here in Theorem 2, we prove that maximize log p(D (0, f1,..........................f.. Buk corresponds to 3 and in DMIL. Thus theorem 2 corresponds to the E-step of DMIL, where take
Ti and Ti2 as D.r. and optimize arg m ) (log p (D(r) (x)) (L((( (x) with coordinate descent method, which can be proved to be equal to
9 in
For the M-step, we take Taa and T3 as Diai According to Theorem 1, we can take the derivative of Ask: Aai.-.. ARK to maximize the joint distribution of latent variables and trainable parameters to maximize the likelihood of dataset, so we have:
V 0o log Der Aa) Xun log (DDi (1ah) * Vo, Asn (DF", Bx),
(13)
XA log p p(Dr Aa) (o, 0))
where O2 10..... Oxx]" and Ay [Acr.... ; A This is exactly the gradients computed in HO and LO steps. Note this computation process can be automatically accomplished with stan- dard deep learning libraries such as PyTorch (Paszke et al. 2019). To this end, we establish the equivalence between DMIL and MAML, and the convergence of DMIL can be proved.
For a clearer comparison, MAML is an iterative process of 9 - + 6', and DMIL is an it- erative process of On, Ot + An. O2 + An, 0n.6r Or - 0n + An. A has no effect on parameters f/ 0,, thus can be divided to two steps as in DMIL. This decoupled fine-tuning fashion is exactly what we need to first adapt the high-level network and then adapt sub-skills. If we end-to-end fine-tune parameters like 0n, 0r + An. A,, sub- skills will receive supervisions from an unadapted high-level network, which may provide incorrect classifications. Different to this, the meta- uprating process λp. A - 6/ 8y must be done at the same time, since if we update fj and 0/ successively. the later one will receive different derivative (for example, Vo, log pf (DDpal 16'p- Xa) ) from derivatives in MAMI. (Vo, log p'Dpaaaaaaaa Aa)). and the equivalence would not proved.
0,,0%, where the posterior estimation stages
EXPERIMENTS
In experiments we aim to answer the following questions: (a) Can DMIL successfully transfer the learned hierarchical structure new tasks with few-shot new task demonstrations? (b) Can DMIL achieve higher performance compared to other few-shot imitation learning methods? (c) What are
effects of different parts in DMIL, such as the skill number K, the bi-level meta-learning procedure, and the continuity constraint? Video results are provided in supplementary materials.
5.1 ENVIRONMENTS AND EXPERIMENT SETUPS
We choose evaluate DMIL two representative robot manipulation environments. The first one Mera-world benchmark environments Yu et al. 2019b), which contains 50 diverse robot manip- ulation tasks, as shown in figure 6 and 7 We use both ML10 suite (10 meta-training tasks and 5 meta-testing tasks) and ML45 suite (45 meta-training tasks and 5 meta-testing tasks) to evaluate our method, and collect 2K demonstrations for each task. We choose Meta-world since we think a large scale of diverse manipulation tasks can help our method to learn semantic skills. We use the following approaches for comparison in this environment: Option- hierarchical gener- ative adversarial imitation learning method to discover options from unsegmented demonstrations Ting et al.) 2021). We use Option-GAlL to evaluate the effect of meta-learning
DMIL. MIL: a transformer-basee meta imitation learning method (Cachet et al. 2020). We use MIL. to evaluate the effect of the hierarchical structure in DMIL. MLSH: the meta-learning shared hierarchies method (Frans et al. 2018) that relearns the high-level network in every new task. We use MLSH to eval- uate the effect of tune (rather than relearn) the high-level network in new tasks. PEMIRL: a contextual meta inverse RL method which transfers the reward function in the new tasks (Yu et al.) 2019a). We use PEMIRL to show DMIL can transfer to new tasks that have significantly different reward functions.
7

---

Under review as a conference paper at ICLR 2022
Table 1: Success rates of different methods on Meta-world environments with K 3. Each data
point comes from the success rate of 20 tests.
ML.10
ML45
Meta training
Meta testing
Meta training
Meta-testing
Methods
I-shot
3-chox
I-shot
3-shot
I-shot
3-shor
I-shot
3-shot
OpticnGAILL
0.455x0:011
0,95220.016
0,24110.042
0.640=0.022
0.50620.008
0.220z0:013 0.71520.006
0.48110.010
MIL
0.776+0.025
0.869=0.022
0.36 040
0.689#0.032
0,584+0.011
0.745=0.017
0, 0.205+0.024
0,510#0.0000
PEMIRI.
0,59840.023
0.810=0.007
0.162+0.003
0.256+0.009
0,289+0.051
0.396+0.024
0.105+0.005
0.126+40.008
MLSH
0,506+0.134
0.725=0.021
0.106+0.032
0.135=0.009
0,235+0.095
0.295=0.021
0.00000.000
0.050#0.00 000
DMIL.
0.77540.010
0.949=0.009
0.396+0.016
0.710=0.021
0,590+0.010
0.859*0.008
0.376+00.004
0.640=00.009
Table 2: Rewards of different methods on four unseen tasks in Kitchen environment with 4. Boldface indicates excluded objects during training.
Task (Unseen)
FIST-no-FT
SPiRL
DMIL(ours)
Microwave, Kettle, Top Burner, Light Switch
2.0 + 0.0
2.1 0.48
1.5+0.48
Microwave, Bottom Burner, Light Switch, Slide Cabinet
0.0
2.3 # 0.49
2.35=0.39
Microwave, Kettle, Hinge Cabinet, Slide Cabinet Microwave, Kettle, Hinge Cabinet, Slide Cabinet
1.0
3.15+0.22 0.29
0.0
3.3 0.38
2.95+0.44
The second one is the Kitchen environment of the D4RL benchmark (Fu et al. 2020), which con- tains five different sub-tasks in the same kitchen environment. The accomplishment of each episode requires sequentially completions of four specific sub-tasks, shown in figure 9
We use open demonstration data set (Gupta et al. 2019) to train our method. During training, we exclude inter- actions with selected objects and at test-time we provide demonstrations that involve manipulating the excluded object to make them as unseen tasks. We choose this environment to show DMIL can be used long-hore hong-horizon tasks and is robust across different environments. We use two approaches for comparison in this experiment: SPiRL: an extension of the skill extraction methods to imita- tion learning over skill space (Price & Boutilier 2003); FIST: an algorithm that extracts skills from offline data with an inverse dynamics model and distance function (Hakhamaneshi et al. 2021).
We use fully-connected neuron networks for both the high-level network and sub-skills. Detailed de- scriptions the environment setup, demonstration collection procedure, dlper-parameee setting, training process, and descriptions of different methods can be found in appendix
5.2 RESULTS
Table 1 shows success rates of different methods in MI.10 and MI45 suites with sub-skill number K 3. We perform 1-shot and 3-shot experiments respectively to show the progressive few-shot performance of different methods. DMIL achieves the best results in ML10 testing suite and MLL45 training and testing suites. This shows the superiority of our method for transferring across a large scale of manipulation tasks. OptionGAIL achieves high success rates in both ML10 and ML45 train- ing suites. These results come from their hierarchical structures that have adequate capacity to fit potential multi-modal behaviors in multi-task demonstrations. MIL achieves comparable results for all meta-testing tasks but is worse than DMIL. This shows the necessity of meta-learning processes. Compared to them, PEMIRL and MLSH are mediocre among all suites. This comes from that the reward functions across different tasks are difficult to transfer with few shot demonstrations, and the relearned high-level network of MLSH damages previously learned knowledge. We also illustrate the t-sne results of these methods in figure 4(a) to further analyze the methods in appendix D.2
Table 2 shows the rewards of different methods on four unseen tasks in the Kitchen environment. FIST-no-FT refers to variant of FIST that does not use future conditioning, which makes the comparison fairer. DMIL achieves higher rewards on two out of four tasks and comparable results on the other two tasks, which exhibits the effectiveness of the bi-level meta-training procedure. The poor performance of DMIL on the first task may come from the choice of skill number K or from low-quality demonstrations. We perform ablation studies on K in the next section.
Figure 3 shows curves of sub-skill probabilities along time of two tasks window-close and peg- insert-side of Meta-world and the microwave-kettle-top burner-light task in Kitchen environment. We can see the activation of sub-skills shows a strong correlation to different stages of tasks. In first two tasks, Th activates when the robot is closing something, Te, activates when the robot is picking up something, and Th activates when the robot is manipulating something. In the third task, na:2 activates when the robot is manipulating the microwave, Th activates when the robot is
8

---

Under review as a conference paper at ICLR 2022
0.25
0.00
60 10e 120 140
Figure 3: The iterative meta-learning process of DMIL at each iteration. Left: the supervision of high-level network (sub-skill category) comes from the most accurate sub-skill. Right: the sub-skill updated at current step is determined by the fine-tuned high-level network.
manipulating the kettle or the light switch, and Th, activates when the robot is manipulating the burner switch. This shows that DMIL has the ability to learn semantic sub-skills from unsegmented multi-task demonstrations.
Table 3: Success rates of different sub-skill number in Meta-was world environments.
5.3 ABLATION STUDIES
ML10
ML4S
In this section we perform ablation stud- ies on different components of DMIL to provide effects of different parts. Due to limited space, we put ablations on fine- tuning steps, bi-level meta-learning pro- cesses, continuity constraint and hard/soft EM choices in appendix D
Mets-training s-shot
Meta-testing
Meta-training 3 3-shot
Meta-testing 3-shot
K
3-shot
1-shot
3-shot
1-shot
I-shot
0.76
0.955
0.32
0.72
0.563
0.818
0.44
0.67
3
0.775
0.949
0.396
0.71
0.59
0.859
0.376
0.64
0.795
0.94
0.52
0.57
0.713
0.92
0.21
0.48
10
0.8
0.975
0.38
0.62
0.736
0.931
0.34
0.64
Effect of different skill number K: Table 3 shows the effect of different sub-skill number K in Meta-world experiments. We can see that larger can lead to higher success rates on meta- training tasks, but a smaller K can lead better results meta-testing tasks. This tells us that an excessive number of sub-skills may result in over-fitting on training data, and a smaller K can play the role of regularization. In Kitchen experiments,
can see similar phenomenons in table 5) It is worth noting in both environments, we did not encounter collapse problems, i.e., every sub-skill gets well-trained even when K 8 in kitchen environment or K 10 in Meta-world environments. This may come from that, in the meta-training stage, adding more sub-skills can help the whole structure get lower loss, thus DMIL will use all of them for training. However, in our supplementary videos, we can see sub-skills trained with large K (for instance, K = 10 in Meta- world environments) are not as semantic as sub-skills trained by a small K (for instance, = 3 in Meta-world environments) during the execution of a task.
DISCUSSION AND FUTURE WORKS
In this work, we propose DMIL to meta-learn a hicrarchical structure from unsegmented multi-task demonstrations endow with fast adaptation ability to transfer to new tasks. We theoretically proved its convergence by reframing MAML and DMIL as hierarchical Bayes inference processes to get their equivalence. Empirically, we successfully acquire transferable hierarchical structures in both Meta-world and Kitchen Environments.
The limitations of DMIL comes from several aspects, and future works can seek meaningful ex- tensions in these perspectives. Firstly, DMIL models all tasks as bi-level structures. However, in real-world situations, tasks may be multi-level structures. One can extend DMIL to multi-level hi- erarchical structures like done in recent works (Shu et al. 2018). Secondly, DMIL does not capture temporal information in demonstrations. Future state conditioning in Hakhamaneshi et al. (202T) seems an effective tool to improve few-shot imitation learning performance in morizoo tasks such as in the Kitchen environments. Future works can employ temporal module such as trans- former (Vaswani et al. 2017) as the high-level network of DMIL to improve its performance.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
We provide video results in supplementary materials. We provide some basic codes to run few-shot testing in weta-world environments in anonymous repository | The complete training code will
be provided later. Details of experiments such as environments, hyper-parameter setting, model setup, training procedure and data preprocessing can be found at B
REFERENCES
Ferran Alet, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Modular meta-learning. In 2nd Anmual Conference on Robot Learning, 2018.
Christopher G. Atkeson and Stefan Schaal. Robot learning from demonstration. In Proceedings of the Fourteenth International Conference on Machine Learning, 1997.
T Cachet, J Perez, and S Kim. Transformer-basee meta-imitation learning for robotic manipulation. In Neural Information Processing Systems, Workshop on Robot Learning, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, 2016.
Christian Daniel, Herke van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference for determining options reinforcement learning. Machine Learning, 2016.
Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abheel, and Wojciech 7aremba. One-shot imitation learning. Advances in Neural
Information Processing Systems, 2017.
David Duvenaud, Dougal Maclaurin, and Ryan Adams. Early stopping nonparametric variational inference. In Artificial Intelligence and Statistics. PMLR, 2016.
Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. On the convergence theory of gradient- based model-agnostic meta-learning algorithms. In The 23rd International Conference Artifi- cial Intelligence and Statistics, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning, 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning. In /st Annual Conference Robot Learning, 20175
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Ad- vances in Neural Information Processing Systems, 2018.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. In Sth International Conference on Learning Representations, 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In 6th International Conference on Learning Representations, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219 2020.
Seyed Kamyar Seyed Ghasemipour, Shixiang Gu, and Richard S. Zemel. Smile: Scalable meta
inverse reinforcement learning through context-conditional policies. Advances Neural In- formation Processing Systems 32:, 2019.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas L. Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. In 6th International Conference on Learn- ing Representations, 2018.
10

---

Under review as a conference paper at ICLR 2022
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving ong-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956. 2019.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, 2018.
Kourosh Hakhamaneshi, Ruihan Zhao, Albert Zhan, Pieter Abbeel, and Michael Laskin. Hierarchi- cal few-shot imitation with skill transition models. 2021.
Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. In 4th International Conference on Learning Reprexentations, 2016.
Karol Hausman, Jost Tobias Springenberg. Ziyu Wang, Nicolas Heess, and Martin A. Riedmiller. Learning an embedding space for transferable robot skills. In 6th International Conference on Learning Representations, 2018.
Peter Henderson, Wei-Di Chang, Pierre-I.ue Bacon, David Meger, Joelle Pineau, and Doina Precup- Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforce- ment learning. In Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton, Adaptive mixtures of local experts. Neural Computing, 1991.
Mingxuan Jing, Wenbing Huang, Fuchun Sun, Xiaojian Ma, Tao Kong, Chuang Gan, and Lei Li. Adversarial option-aware hierarchical imitation learning. Proceedings of the 38th International Conference on Machine Learning, 2021.
Ilya Kostrikov, Kumar Krishna Agrawal, Sergey Levine, and Jonathan Tompson. Addressing sample inefficiency and reward bias in inverse reinforcement learning. arXiv preprint arXiv: arXiv:1809.02925, 2018.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. DDCO: discovery of deep continuous options for robot learning from demonstrations. 1st Annual Conference on Robot Learning, 2017.
Hoang Minh Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III. Hierarchical imitation and reinforcement learning. In Proceedings of the 35th International Con- ference on Machine Leurning, 2018
Sang-Hyun Lee and Seung-Woo Seo. Learning compound tasks without task-specific knowledge via imitation and self-supervised learning. Proceedings of the 37th International Conference on Machine Learning. 2020.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, 2017.
Libin Liu and Jessica K. Hodgins. Learning to schedule control fragments for physics-based char- acters using deep gelaamnin. ACM Trans. Graph., 2017.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In 3rd Annual Conference on Robot Learning, 2019.
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. In 7th International Conference Learning Representations, 2019.

---

Under review as a conference paper at ICLR 2022
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint D.Xiv:1906.026293 2019.
Gerhard Neumann, Wolfgang Maass, and Jan Peters. Learning complex motions by sequencing simpler motion templates. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman (eds.), Proceedings of the 26th Anmual International Conference on Machine Learning, 2009.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Picter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 2019.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: example- guided deep reinforcement learning of physics- based character skills. ACM Trans. Graph., 2018.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. MCP: learning composable hierarchical control with multiplicative compositional policies. In Advances in Neu- ral Information Processing Systems, 2019.
Bob Price and Craig Boutilier. Accelerating reinforcement learning through implicit imitation. Jour- nal of Artificial Intelligence Research, 19:569-624 629, 2003.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning, In 7th International Conference on Learning Representations, 2019.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction imitation learning and structured prediction no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011.
Reginaldo J Santos. Equivalence of regularization and truncated iteration for general ill ill-posed prob- lems. Linear algebra and its applications, 1996.
Mohit Sharma, Arjun Sharma, Nicholas Rhinchart, and Kris M. Kitani. Directed-infe GAIL: learn- ing hicrarchical policies from unsegmented demonstrations using directed information. In 7th International Conference Learning Representations, 2019.
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In 6th International Conference on Learning Representations, 2018.
Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps. A frame- work for temporal abstraction in reinforcement learning. Artif. Intell., 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Multimodal model-agnostic meta- learning via task-aware modulation. In Advances in Neural Information Processing Systems, 2019.
Kelvin Xu, Ellis Ratner, Anca D. Dragan, Sergey Levine, and Chelsea Finn. Learning a prior over intent via meta-inverse reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. In Advances in Neural Information Processing Systems, 201 9a.
Tianhe Yu. Pieter Abbeel, Sergey Levine, and Chelsea Finn. One-shot hierarchical imitation learning of compound visuomotor tasks. arXiv preprint arXiv: 1810. / 1043, 2018 8a.
12

---

Under review as a conference paper at ICLR 2022
Tianhe Yu, Chelsea Finn, Sudeep Dasari, Annie Xie, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. In Robotics: Science and Systems XIV, 2018b.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In 3rd Annual Conference Robot Learning. 2019b.
Yuke Zhu, Ziyu Wang. Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvu- nakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. Robotics: Science and Systems XIV, 2018.
Yayi Zou and Xiaoqi Lu. Gradient-em bayesian meta-learning. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, 2020.
13

---

Under review as a conference paper at ICLR 2022
A ALGORITHM
Algorithm Dual Meta Imitation Learning
Require: task distribution p(T), multi-task demonstrations {Di}, = 1, m, initial param-
eters of high-level network 0% and sub-skill policies " ,0K, inner and outer learning rate a, ,3.
while not done do
Sample batch of tasks Ti p(T)
for all Ti do
Sample {Ta)) (T2) (Tis), {TEa} from [D:}
Evaluate Cn Cx Tai) according to and Ti
Compute adapted parameters of high-level network: 'o 0₁ aVo, Ln (On, T31)
Evaluate Ve,, LBc BB D2s) according to and Ti2. k = 11, 1,
Compute adapted parameters of sub-skills: Auk = 0t aa Au BC ( (0kk, D24), K 1,..,,
Evaluate Vo, CT, T() and Vau CT, (Aik. DaN,, k = 1, 1. K
end for Update 8j t 0 Von ΣTOP(TT LT (A),
Update O1k Gik or ETEWpTT Lr,, (Nux; Dat), 1, K end while
B AUXILIARY LOSS
We adopt an auxiliary loss for DMIL to better drive out meaningful sub-skills by punishing excessive switching of sub-skills along the trajectory. This comes from intuitive idea: each sub-skill should
be -eteedelextended macro-action, and the high-level policy only needs to switch to different
skills few times along a task, as the same idea of macro-action in MLSH Frans et al. (2018). We denote:
{
1, if x True
sign(x)
(14)
0, ifr = False
and the auxiliary loss is:
Lauv (T) sign(21+1 # 2x) len(T).
(15)
Although this operation seems discrete, in practice we can use the operations in modern deep learn- ing framework such as PyTorch (Paszke et al. 2019) to make differentiable. We add this loss function to the L, (On, ra) and Ve, ((),, Tis) with a coefficient À = 1. We also perform ablation studies Louz and results are in table
C PROOFS
C.1 PROOF OF LEMMA
Lemma In case q Ax) is Dirac-delta function and choosing Gaussian prior for p(o, equa- tion 8 equals to the inner-update step of MAML, that is, maximizing log p(D(r) w.r.t. X by early- stopping gradient-ascent with choosing μs as initial point:
A:( 0) μ4 Va logp (D! la-we-
(16)
Proof: in case of the conditions of Lemma 1, we have:
> (D arg max[log (p)))) 1| noll3 /2522)
(17)
As stated in Grant et al. (008),, firstly in the case of linear models, early stopping of an iterative gradient descent process of À equals the maximum posterior estimation (MAP) (Santos 1996).
14

---

Under review as a conference paper at ICLR 2022
In case the posterior distribution refers q(d, and MAML is Bayes process to find the MAP estimate as the point estimate of A, (D f). In the nonlinear case, this point estimate is not necessarily the global mode of the posterior, and refer to Duvenaud et al. (2016) for another implicit posterior distribution over di and making the early stopping procedure of MAMI. acting as priors to get the similar result.
C.2 PROOF EQUATION
Equation can be written as:
AD 0) arg min () (D) 0) (og]((1;))))
A;
arg max B (lgg Bo
(18)
A
arg max F (o [b.)] (T ())p(())))))))))
A
where in MAML we assume p(D) (.)) = p(D , 0). and use the joint distribution p D 0: replace PDo, 0) since we assume that p(D)") subjects to uniform distribution. Thus 8 can be proved.
C.3 PROOF OF THEOREM 1
Theorem In case that Mg - 0+, i.c., the uncertainty in the global latent variables 0 is small, the following equation holds:
M
VeL(0,\1," (,M)) _
Xx log p (D:aa Di (.) (oA(DD)
(19)
Proof:
M
V VoL(O, A1,
Am) ~
Ve Vo Lole - log ((); Ac)]
Ve Pollogp(D!att (D ( 0)) (D,,,
(20)
M
Ve loggp(( (D.vll (r 0))
M
Vx, log (D Ar) VoAr(D!
where the first approximate equal holds because the VI approximation enror is small enough, and the second approximate equal holds because that in case Σ. - 0+ and assuming X; be neuron network, log (D!," ()) ss holds almost everywhere, so Vo log 0)(0) 0. Note the condition of theorem is usually satisfied since we are using MAML, and the initial parameters are assumed to be deterministic.
From another perspective, the right side of above equation is the widely used meta-gradient in MAML, and is equal to m (I - VVCCne (0, D(rr)
Vx LBC (0 - VoLB (0, , which is proved to be converged by Fallah et al. 2020
C.4 PROOF THEOREM 2
Theorem In case of p(aclst, O1k) N (HHeaeee)) 02), we have:
Vo, log (D." (0n, Ou,, Oik) Vo, Conn, D(T),
(21)
15

---

Under review as a conference paper at ICLR 2022
and
Vou log ((D On, ,0k) = Von LBc D2k), 1,,.. , K.
(22)
Proof: since p(D( 1,, i,,... 8ik) = m p(a st. 0r ......... OLK 1p((s110 10,, On, .. Oix) and
the second term is independent of 0, consider the first conditional probability:
plaelse, On 01,, 8ix) Σ plzulse, 0x)p(ae Bue)
(23)
In the HI step, placlse. 0()) is fixed, thus 23 becomes convex optimization problem:
Σ
p(zal8e 0n)p(aels,,
(24)
8h
arg max, ploelst, Ouk )- This means that тH, needs to predict the sub-skill category at time step as k, in which case The can maximize p(ac St, 0lx). In case choose Te be a classifier that employs Softmax layer at the end, minimizing the cross entropy losuls to maximize 24, thus 21 can be proved.
The solution of this problem is X which satisfies p(=x|st, 0e, AI
In the LI step, p(z* (s,, An.) is fixed, and the data sets for optimizing 01.. 0ik are also fixed as D2k- = {(Sije, a:jt)) = N) Thus we need maximize each p(ay|s,, 01x) with D2k- In case p(ar|s;, 01k) ~ N (Pooa(s..) o²) x expl- (a. (a4- (s,)) we have maxeu plar|se, Onk) +
20
ming, (a: Ton (s,)), which leads to the loss function thus 22 can be proved, which finishes the prove of Theorem 2.
C.5 PROOF OF THE E-STEP OF DMIL
According to Theorem 1, we aim to maximize 17 w.r.t X from the initial point 0g with coordinate gradient ascent. We here need to prove that in DMIL, we could also achieve the global maximum point of X as in MAML. We first give out the following Lemma:
Lemma 2 Let z be the solution found by coordinate gradient descent of fzr). Let 2,, i = 1,...,. be the n coordinate directions used the optimization process. If (x) can be decomposed as:
f(x) = g(x)
h:(w).
(25)
where g(x) is differentiable convex function, and each hx(x)) is convex function of the coordinate
direction F;, then is the global minimum of fx:).
Proof: Let be another arbitrary point, we have:
( (()) + h(y) - (g(x) + M(x))
2 V (y) xx) Σ ha(ys) hx(2s)
(26)
> (Vi(())(1 -x,)= - h((ws) - h2s (x()))
2 0.
16

---

Under review as a conference paper at ICLR 2022
Now let's consider problem. Consider
log (D. 0.) log
p(a se, Ssh (s)p(ss)
K
log
((()
plz ae Osnppaalss 0x)
1=1
T
((s)
+ log
plza|se, ar (aappaalsse 0n)
(27)
Σ Dlg ((a)
log plzzp(/ ae Oen ) 1sc, 0w)l
T
K
K
[og p(s))
+
log p(=x at 0th) +
Σ k=1
log pl p(aels,,
In our case, two coordinate directions are 8th and 8a- Let's consider the terms inside the brackets. According to Lemma 2, we think lo,)(s;) as ( it equals to constant), Σk= logp(ze 22 (ec ae, Bsh)) as h1()) and Σk logplas Bu) as hz(x). Thus the optimum can be proved.
D ADDITIONAL ABLATION STUDIES
D.1 EFFECTS OF THE BI-LEVEL META-LEARNING PROCESS
We use two variants of DMIL to see the effectiveness of bi-level meta-earning process. DMIL- High: variant that only meta-learns the high-level network, and DMIL-LLw: variant that only meta-learns sub-skills. We use Option-GAll. as a comparison that does not meta-learn any level of the hicrarchical structure.
Table the results of this ablation study, DMiL-High achieves close results with OptionGAIL in all meta-training suites and better results all meta-testing suites, but worse than DMIL in all cases. This shows that meta-learming the high-level network can help the hierarchical structure adapt new tasks, but only transferring the high-level network not enough for accomplish all kinds of new tasks. DMIL--Low achieves poor results in all suites except in MI.10 meta-training suites. This shows that transferring the high-level network is necessary when training on large scale of tasks or testing in new tasks.
Table 4: Success rates of DMIL-High DMIL-Low, DMIL and OptionGAIL on Mela-world envi- ronments with K = 3. Each data point comes from the success rate 20 tests.
ML45
Meta training
Meta testing
Meta training
Meta-testing
Methods
3-shot
I-shot
3-shot
OptionGAlL
0.75540.011
0.952:0.016
0.241+0.042
0.64020.025
0,50620.008
0.71520.006
0.220e0.013
00081
Digh
0.63420.001
0.91420.011
0.298#0.012
0.670=0.015
1.49500.007
0.73520.006
0,280=0.016
0.551
DMIL-Low
0.74640.011
0.94320.006
0.29110.0224
0.666200021
0.51120.005
0.76520.009
0.266100.010
0.492a0.014
DMIL
0.775:0.010
0.94920.009
0.39620.016
0.710=0.021
0.590=0.010
0.859=0.008
0.376:0.004
0.640=0.009
For better understand the effect of bi-level meta-learning process, we perform ablation study for three DMIL variants in an danuallyddesinned new task push-around (figure [8. In this task, the robot needs to grasp a cube and circle it around the wall. This brand new skill that is not in the meta-world suite. The accomplishment of this new task requires quickly adapting abilities of both the high-level network and sub-skills. We sample two demonstrations and use the first one few-shot data, and illustrate the sub-skill categories of the second demonstration given by the high-level network at figure 4bb) Before adaptation, all variants give out approximately random results. However, after one-shot adaptation, DMIL classifies almost every state into sub-skill 0 and
17

---

Under review as a conference paper at ICLR 2022
Before Adapiation
Adptdie
DMIL
Aft Adptere
DMIL-LOW
After Adaptation
MISH
OptionGAIL
DMIL
DMLL-High
T-sne results of demonstrations and sub-skill categories of several (b) Sub-skill categories of task push- hierarchical models for meta-testing task hand-insert.
around-wall for ablation study.
Figure 4: T-sne results and ablation studies about the bi-level meta-learning process.
sub-skill which indicates these two sub-skills DMIL have been adapted the new task, and the high-level network has also been adapted with the supervision from adapted sub-skills. Compared DMIL., DMIL.-Low still can not give out reasonable results after adaptation, since its high-level network lacks the ability to quickly transfer to new tasks. Instead, DMIL-High gives out plausible results after adaptation. This shows the high-level network has adapted to the new task according to the supervision from adapted sub-skills, but sub-skill can dominate for long time period since all sub-skills lack the quickly adaptation ability.
D.2 T-SNE RESULTS OF DIFFERENT METHONS
For comparison of different methods, illustrate the t-sne results of states of each sub-skill in
MI meta-testing task hand-insert in figure [4(a) We use demonstrations for adaptation, and
draw the t-sne results another 16 demonstrations. This task is meta-testing task, no method has ever been trained on this task before.
MLSH shows almost random clustering results no matter before and after adaptation, since its high- level network is relearned in new tasks. OptionGAIL clusters to three sub-skills after adaptation. Compared to them, DMIL clusters the data only two sub-skills after adaptation. We believe fewer categories reflect more meaningful sub-skills are developed in DMIL.
D.3 EFFECTS OF SUB-SKIL.L NUMBER K IN THE KITCHEN ENVIRONMENT
We perform ablation studies of sub-skill number K on the Kitchen environments and choose 2, 4, 8 respectively. Table 5 shows the results. We can that a smaller number of sub-skills can achieve better results on such four unseen results that a large number sub-skills. This may indicate that the sub-skill number K can work as a "bottleneck" like the middle layer in an auto-encoder.
Table 5: Ablations of sub-skill number K in Kitchen environments.
Task (Unseen)
K=2
K=4
K=8
Microwave, Kettle, Top Burner, Light Switch
1.940.43
1.5+0.48
1.740.22
Microwave, Bottom Burner, Light Switch, Slide Cabinet
2.15+0.19
2.35+0.39
2.0+0.37
Microwave, Kettle, Hinge Cabinet, Slide Cabinet
2.45+0.25
3.15+0.22
1.85+0.23
Microwave, Kettle, Hinge Cabinet, Slide Cabinet
2.01+0.24
2.95+0.44
2.44+0.47
18

---

Under review as a conference paper at ICLR 2022
D.4 EFFECTS OF FINE-TUNING STEPS
As all performance of the trained model. It controls the balance between under-fitting and over-fitting, We perform ablation studies of fine-tune steps in Meta-world benchmarks with K 5 and Ir=le-2 in O Results are as follows:
few-shot learning problems, the fine-tuning steps in new tasks to some extent determine the
Table 6: Ablation studies of the fine- tuning steps in world experiments with K 5.
ML10
ML45
Meta-training 3-shot
Meta-testing
Meta-training
Meta-testing
fine-tune steps
1-shot
1-shot
3-shot
1-shot
3-shot
1-shot
3-shot
10
0.5
0.575
0.28
0.24
0.374
0.49
0.05
0.17
30
0.69
0.915
0.25
0.31
0.602
0.85
0.13
0.32
50
0.695
0.895
0.27
0.39
0.583
0.87
0.12
0.32
100
0.665
0.905
0.23
0.41
0.614
0.872
0.07
0.43
300
0.66
0.845
0.28
0.44
0.605
0.876
0.12
0.44
500
0.63
0.91
0.25
0.39
0.584
0.867
0.13
0,42
Range
0.195
0.34
0.05
0.2
0.231
0.386
0.08
0.27
D.5 EFFECTS OF CONTINUITY REGULARIZATION
We perform ablation studies of the effect of continuity regularization following table DMIL_ne means no continuity regularization. We can see that the continuity constraint would damage meta- training performance slightly, but increase the meta-testing performance greatly.
Table K
10
MI.10
MI.45
Meta-training 1-shot
Meta-testing 1-shot
Meta-training 3-shot
Meta-testing 3-shot
variants
3-shot
3-shot
1-shot
1-shot
DMIL
0.795
0.94
0.52
0.57
0.713
0.92
0.21
0.48
DMIL.nc
0.788
0.96
0.32
0.56
0.703
0.927
0.17
0.35
Gap
0.007
-0.02
0.2
0.01
0.01
-0.007
0.04
0.13
D.6 EFFECTS OF HARD/SOFT EM CHOICES
In DMIL, we use hard EM algorithm train the high-level network. One may think about to use soft cross entropy loss to train the high-level network to get better results. We perform this ablation study in the following table We can see that soft cross entropy training won't help increase the whole success rates. This may comes from that, usually we use soft cross entropy (such as label smoothing) prevent fittingg However. in our situation, this may cause under-fitting, since training on such a large scale of diverse manipulation tasks is already very difficult. Future works can seek more comparisons about this choice.
Additionally, we found an interesting phenomena that the training loss of the high-level network with a softmax shows trend of rising first and then falling, as shown in In our experiments, softmax loss may regularize the optimization process to make the high-level network be under-fitting on training data. This may come from that, the experiment environment (Meta-wone Morld) contains large scale of manipulation tasks, in which the training of the high-leve network be difficult and unstable. Thus a soft-max cross entropy loss cannot help that much here like how it works as a regularizer prevent over-fitting in the label smoothing (Müller et al. 2019).
19

---

Under review as a conference paper at ICLR 2022
Table 8: Ablation about hard/soft EM choices with K in the Meta- Meta-world environments.
ML10
ML45
Meta-training 1-shot
Meta-testing 1-shot
Meta-training
Meta-testing
variants
3-shot
3-shot
1-shot
3-shot
1 s-shot
3-shot
DMIL
0.795
0.94
0.52
0.57
0.713333
0.92
0.21
0.48
DMII..soft
0.37
0.65
0.33
0.46
0.235556
0.43
0.1
0.32
Gap
0.425
0.29
0.19
0.11
0.477778
0.49
0.11
0.16
342
245
244
Figure 5: The training loss of the high-level network with softmax shows trend of rising first and then falling.
E EXPERIMENT DETAILS
E. ENVIRONMENTS
See figure 6 and
Train
Test
cosembiy
bosHetbali
button preso
button pross
button button
coffee button
coffee
coffee
bin picking
dil turn
dioassemble
door open
coor untock
drower GlODe
crower open
Toucet ocon
Towcet Gl00e
hommer
box ciise
oide
BION
pull
TIPE
umpkig
hole
plok piace
piote side
plate sice
piote siide
piote side
push back
push
reach
door unlock
reoch wCil
shell place
5occer
stick push
Slick pull
sweep Into
Sweep
WRncW open
window closs
hand Insert
Figure 6: The ML45 environment.
E.2
MODEL SETUP
DMIL: The high-level network and each sub-skill modeled with a 4-layer fully-connected neuron network, with 512 ReLU units in each layer. We use Adam the meta-optimizer. DMIL--High and
20

---

Under review as a conference paper at ICLR 2022
Train
Test
boeketbali
button
dial
drower
Insert
6:00
cicse
lever
sheif
pick
sweep into
window
Figure The ML10 environment.
Figure 8: Task push-arounddwall
Table 9: DMIL hyper-parametens.
Parameter
Value
B 5e-4
le-4
fine-tune iterations
3
batch size (in trajectory)
16
0.1
Table 10: MIL Parameter
Value
Thhnnd
8
Blayer
3
dmodel
512
dy
64
d,
64
Ppposiiinn
250
dropout
0.1
batch size (in state-action pair)
512
DMIL-Low use the same architecture with DMIL. Hyper-parameter settings are available in table
We use transformer Vaswani et al. 2017) the policy to perform MAML. The input of the encoder the whole one-shot demonstration or 3-shot demonstrations with concatenated state and action. The input of the decoder is current state. The output is the predicted action. Hyper-parameter settings are available table 10
MIL:
We use the same settings of network with DMIL here. The macro step of high-level network Since our problem is not a reinforcement learning process, we use the behavior cloning variant of MLSH. The pseudo reward is defined by the negative mean square loss of the predicted action and the ground truth, and we perform pseudo reinforcement learning process with off-policy demon- stration data. We PPO importance sampling weights that required by replacing the sampling process in the environments with the demonstrations in the replay buffer, which has been shown be effective in practice in
MLSH:
reinforcement learning algorithm. Note in this way ignore the
21

---

Under review as a conference paper at ICLR 2022
Figure 9: Kitchen environments.
Table 11: MLSH
Parameter
Value
high-level learning rate
le-3
sub-skill learning rate
le-4
PPO clip threshold
0.02
high-level warmup step
500
joint update step
1000
batch size (in state-action pair)
900
Table 12: PEMIRL hyper- parameters.
Parameter
Value
learning rate of all models
le-4
PPO clip threshold
0.02
coefficient (r) of hep
0.1
batch size (in trajectory)
16
Kostrikov et al. (018): Ghasemipour et al. 2019)) Hyper-parameter settings are available in table
PEMIRL: We use the same setting of the high-level network as the policy ww and the inference model qo PEMIRL. We use PPO our reinforcement learning algorithm. We 3-layer fully- connected neuron network as the context-dependen disentangled reward estimator re(s, m) and the context-dependent potential function hy m). Here we also use the behavior cloning variant of PEMIRL to only train models on the off- off-policy data.
settings are available in table
12
E.3 TRAINING DETAILS
For fine-tuning, OptIonGAIL. DMIL-Low and DMIL-High have meta-learning mechanism for (some parts of) the trained model. In the few-shot adaptation process, we have different fine-tune method for these baselines:
OptionGAIL:
We train OptionGAIL models on the provided few-shot demonstrations for a few
epochs.
We fix the high-level network and only tne-tune sub-skills with few-shot demonstra- tions.
DMIL-Low:
22

---

Under review as a conference paper at ICLR 2022
DMIL-High: We fix sub-skills and only fine-tune the high-level network with few-shot demonstra-
tions.
E.4 DETAILED RESULTS
We provided detailed success rates of each environments of different methods in figure T0 and with K 3.
23

---

Under review as a conference paper at ICLR 2022
ML-10 Maximum Per-Task Success Rates (N 200)
DMIL
basketball-v2
MIL
MLSH
OptionGAIL
button-press-v2
PERMIRL
dial-turn-v2
Train Envrionment
drawer-close-v2
peg-insert-side-v2
reach-v2
push-v2
pick-place-v2
sweep-into-v2
window-open-v2
door-close-v2 v2
Test Environment
drawer-open-v2
lever-pull-ye
shelf-place-v2
sweep-v2
20
40
60
80
100
Success Rate
Figure 10: The ML10 results of different methods after 3-shot adaptation.
24

---

Under review as a conference paper at ICLR 2022
ML-45 Maximum Per-Task Success Rates (N=20)
assembly-v2
DMIL
basketball-r2
MIL
button- press--opooow---222
MLSH
button-press-tooddowwwlll-
OptionGAIL
button-press-v2
PERMIRL
button-
preton--ress-wll-22
coffee-button-v2
coffee-pull-v2
coffee-push-v2
dial-turn-v2
disassemble-v2
door-close-v2
door-open-v2
drawer-close-v2
Train Envrionment
drawer-open-v2
faucet-open-v2
faucet-close-v2
hammer-v2
handie-press-side-22
handie-press-v2
pandie-pull-side-22
handle-pull-v2
lever-pull-v2
peg-insert-side-v2
pick-place-wall--2
pick-out-of-hole-v2
push-back-v2
push-v2
pick-place-v2
plate-slide-v2
plate-slide-side-22
plate-slide-backe22
2ate-slide-back-side-22
peg-unplug-side-22
soccer-v2
stick-push-v2
stick-pull--2
push-wall-v2
reach-wall-v2
shelf-place-v2
sweep-into-v2
sweep-v2
Test Environment
window-open-v2
window close-v2
bin-picking-v2
box-close-v2
door-lock-v2
door-unlock-v2
hand-insert-v2
0
20
40
60
80
100
Success Rate
Figure 11: The ML45 results different methods after 3-shot adaptation.
25


Reviews:
[{"review_id": "cHApdJfJXZi", "reviewer": "Reviewer_ouLD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has a number of strengths:\n1. This paper studies an important research problem  — the ability to discover meaningful abstractions of behavior from prior experience and to transfer these abstractions to efficiently learn new tasks\n2. The related work section lays out a detailed overview of relevant work\n3. While the proposed approach integrates existing elements from MAML and hierarchical imitation learning, it does so in a unique and novel manner\n4. The theoretical analysis complements the work well (the AC should note however — I did not scrutinize this section sufficiently to verify its claims)\n5. The experiments consider a thorough set of baselines\n\nAlongside these strengths, I also have numerous concerns, all regarding the quality and clarity of presentation:\n1. In my view, the introduction of the paper sells the work short. Much of the discussion is on centered “what” approaches previously exist and “what” the technical components of this paper are. Relatively little discussion in centered on “why” this proposed research problem is important, “why” previous approaches falls short, and “why” addressing the limitations of prior approaches is challenging. Adding this context can help better motivate this work.\n2. The figures are difficult to follow. In particular, figure 1 and 2 have a lot of visual details, but are accompanied with sparse textual descriptions. Ideally, figures should be self-contained, where the caption can describe the figure in full detail.\n3. While I appreciate the motivation in the introduction to learn long-horizon tasks, I do not think that the experiments necessarily study long-horizon tasks. The metaworld environments entail relatively short sequences of simple primitives, and furthermore this paper only uses 3 sub-skills which is not reflective of the diversity that comes with long-horizon tasks. \n4. The writing currently suffers from a significant number of grammatical issues. There are a number of passive sentences throughout, and some sentences are simply difficult to follow, such as the following: “there is no enough ability in sub-skills to quickly adapt to new tasks, thus no certain sub-skill exceeds others to give out dominant supervision for the high-level network.” Other sentences such as “DMIL is a brand new iterative hierarchical meta-learning procedure” use informal language (“brand new”) and should be revised accordingly.\n5. The discussion section mentions future avenues of research, but does not quite touch upon the core limitations of the work. Here are a set of potential limitations to warrant discussion: what are the failure modes encountered during experiments? was there mode collapse among the skills? how sensitive is the method to the number of skills K?\n6. I’d like to know more about the MIL baseline — why was a transformer used here as opposed to a simple MLP, and why does DMIL not use a transformer?\n", "summary_of_the_paper": "This paper proposes a meta imitation learning framework aimed at learning long-horizon robot control tasks with fast adaptation capabilities. Specifically, the proposed approach adapts the model-agnostic meta learning framework for learning a hierarchical policy, where both levels of the hierarchy are meta-trained and fine-tuned at test time to learn new tasks. On the metaworld benchmark, the proposed method outperforms prior work, and qualitative analysis shows that the method discovers meaningful skills.", "summary_of_the_review": "While I think the technical contributions of the paper are sound and the experiments are generally thorough, this paper at its current form is limited by the writing quality. A signifiant amount of work needs to go in re-writing the introduction, improving the figures, and fixing grammatical issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UExpdrT58qw", "reviewer": "Reviewer_ZyiJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of the paper is that they proposed a natural extension of meta-IL and HIL. The total algorithm is reasonable, and its convergence is proved based on the theory of the EM algorithm. The connection to the EM algorithm is natural and understandable. The experiment showed the usefulness of the method as well. \nThe weakness of the paper is the limitation of evaluations. They tested their methods on two pre-existing simulation environments. The evaluation using a task meta and hierarchical nature is more explicitly required is expected. The strength of the method would be more clearly shown with such a task.\n", "summary_of_the_paper": "The paper describes Dual Meta Imitation Learning (DMIL) which is a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning (MAML). The DMIL is a hierarchical extension of MAML-based imitation learning (IL) and is a meta-learning extension of Hierarchical Imitation learning (HIL).  The authors provide theoretical proof of the convergence based on the connection with the Expectation-Maximization (EM) algorithm. They showed the state-of-the-art few-shot imitation learning performance on the meta-world benchmark.", "summary_of_the_review": "The paper proposed a new meta HIL method called DMIL. The method is clearly explained. The background is described in a proper manner and is informative. The novelty is clear. Also, the empirical evaluation is appropriate though the number of environments can be increased. \nAs a whole, I admit that the paper is an excellent paper. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J23oEgmD948", "reviewer": "Reviewer_2AqY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n- The authors show that DMIL is competitive against a thorough set of ablations on both ML10 and ML45 settings from meta-world.\n\n- The convergence analysis hints at the algorithms training stability.\n\n**Cons**\n- The method mostly borrows components from existing works without explaining how their setting adds more complexity to the approach.\n\n- From the results, it's difficult to compare this method against hierarchical imitation learning approaches that they cite (Yu et al., 2018b; Finn et al., 2017b; Yu et al., 2018a)\n\n- The authors do not show that the results are robust across different tasks/environments (e.g., \"Hierarchical Few-Shot Imitation with Skill Transition Models\" looks at maze and kitchen environments) or number of subskills. I believe this is important because the manipulation tasks in meta-world may have compositional task structure that can't be assumed in other environments or tasks (such as navigation).\n\n**Questions**\n- How would you contrast DMIL against \"Hierarchical Few-Shot Imitation with Skill Transition Models\"?\n\n**Feedback**\n- It would aid understanding to include a algorithm box that outlines the order of inner and outer updates in one place.", "summary_of_the_paper": "The authors propose to learn a high level policy network that picks a subskill policy to predict actions with meta imitation learning (DMIL). Both the high and low level policy networks are fine-tuned at meta-test time. They show that DMIL converges and evaluate their approach against several ablations on the ML10 and ML45 setting of the meta-world benchmark.", "summary_of_the_review": "I vote to marginally reject this submission because the experimental results are not thorough enough to demonstrate robustness across environments and number of subskills and the novelty of DMIL mostly comes from combining known approaches. If more environments and subskills could be tested, I think the paper could be a candidate for submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G9oPjxX887R", "reviewer": "Reviewer_wfsx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- few-shot imitation is an interesting and impactful problem\n\n- the paper solves a hard problem: jointly learning HL and LL policy + quickly adapting them on unseen tasks\n\n- the method is clearly described, the figure help understanding the approach\n\n- the method is evaluated on a representative benchmark and compared to meaningful baselines\n\n- the qualitative results show that the approach learns meaningful skills in the MetaWorld benchmark\n\n\n## Weaknesses\n\n- My main concern with the method is that it is trying to solve too many things at once: (1) inferring skills from unstructured datasets, (2) jointly learning high-level and low-level policy and (3) meta-learning both these policies to have them adapt quickly on new tasks. Intuitively, the resulting iterated meta-optimization scheme seems unstable to train / sensitive to choices like the policy initialization, number of subskills, switching regularization coefficient, ... (the theoretical convergence results do not elucidate how hard it is to train agents with this approach *in practice*). The paper could add an analysis on hyperparameter robustness of training to show the stability of the approach.\n\n- While the paper cites many works on hierarchical imitation and meta-imitation learning, it does not cite the relevant recent works on learning skills from large, multi-task datasets (SPiRL, Pertsch et al 2020, OPAL, Ajay et al 2021) and using them for imitation learning (SkiLD, Pertsch et al 2021, FIST, Hakhamaneshi et al 2021). Especially FIST seems like an interesting method to compare to since it is made for the few-shot imitation setting. That being said: this comparison is not mandatory since FIST is concurrent work.\n\n- I think the paper could be strengthened by including a comparison to a method that is hierarchical but *does not* learn HL and LL policies jointly. All of the abovementioned works first learn a set of low-level skills and then learn a high-level policy over them. It would improve the comparisons of this paper to include an approach that pre-trains temporally extended low-level skills, then freezes them and meta-trains a high-level policy over these skills. Such a training scheme would presumably be much easier to tune (since it does not require EM-style iterated training) -- so it would be good to see whether the performance is much worse than the proposed, more complicated method. (the already included DIML-High ablation still trains HL and LL jointly even though it keeps the LL fixed during downstream finetuning from what I understand)\n\n- The paper does not include an ablation on the chosen number of discrete learned skills. It is possible that the DIML-High ablation could perform better if it had a larger number of low-level skills to choose from (since it does not adapt the low-level skills to the downstream task), so including such an experiment would strengthen the paper.\n\n\n## Further Suggestions\n- I wonder whether the approach could apply an idea similar to label smoothing where instead of using a hard max for the inner training loop of the HL policy we would set the targets for the HL cross entropy loss to be the softmax of the action prediction errors of the low-level. Maybe such a more nuanced training signal could improve the training stability?", "summary_of_the_paper": "The paper proposes an approach for few-shot imitation learning that jointly meta-learns a high-level policy and a set of low-level policies from a diverse set of demonstrations (multi-task). It then finetunes both high-level and low-level policies on few demonstrations of the target task for imitating them. In experiments on the MetaWorld50 task suite the proposed approach outperforms few-shot imitation methods that either don't use hierarchy or don't use meta-learning.\n", "summary_of_the_review": "The paper proposes an interesting approach for jointly learning adaptable high and low-level skills. While the method is well-explained it seems to more complex than alternative approaches and the current experiments don't fully convince me that it would be easy to train on new tasks without a significant amount of tuning to get the iterated learning loop to converge. If the authors can provide experimental evidence that the method is no harder to train / tune than alternative approaches for few-shot imitation I am leaning towards accepting the submission, but I am open to changing my opinion based on the other reviewer's feedback.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a hierarchical meta imitation learning framework for few-shot transfer in the context of long-horizon control tasks. Underlying the framework is a hierarchical adaptation of model-agnostic meta learning (MAML) that jointly learns the high-level policy together with the set of modular low-level policies (sub-skills), both of which are fine-tuned at test time based on a small number of demonstrations. Experimental evaluations on the meta-world benchmark as well as a kitchen environment benchmark compare the proposed framework with recent baselines.

As several reviewers note, the problem of jointly learning modular policies together with the high-level policy for composing these sub-skills is both challenging and interesting to the robotics and learning communities. The manner by which the paper extends existing work in meta-learning (MAML) and hierarchical imitation learning is novel and technically sound. The reviewers raised some concerns, notably those regarding (1) the the framework's sensitivity to various hyperparameter settings and its ability to generalize to other domains; (2) the merits of joint optimization over decoupled optimization of the sub-skills and high-level policy; and (3) the need for experiments/evaluations on different domains. The authors provided a detailed response to each of the reviewers that includes the addition of a different benchmark evaluation (the kitchen environment), new ablation studies, and updates to the text. After a thorough review, however, concerns remain regarding the reproducibility of the results, which call into question some of the key contributions that the paper claims to provide over the existing state-of-the-art. The authors are encouraged to provide a more balanced discussion of the contributions along with evidence to support reproducibility in any future version of the paper.
Decision: Reject