Year: 2018

Abstract:
A noisy and diverse demonstration set may hinder the performances of an agent aiming to acquire certain skills via imitation learning. However, state-of-the-art imitation learning algorithms often assume the optimality of the given demonstration set. In this paper, we address such optimal assumption by learning only from the most suitable demonstrations in a given set. Suitability of a demonstration is estimated by whether imitating it produce desirable outcomes for achieving the goals of the tasks. For more efficient demonstration suitability assessments, the learning agent should be capable of imitating a demonstration as quick as possible, which shares similar spirit with fast adaptation in the meta-learning regime. Our framework, thus built on top of Model-Agnostic Meta-Learning, evaluates how desirable the imitated outcomes are, after adaptation to each demonstration in the set. The resulting assessments hence enable us to select suitable demonstration subsets for acquiring better imitated skills. The videos related to our experiments are available at https://sites.google.com/view/deepdj.

heading: 1 INTRODUCTION
text: Imagine that you intend to learn how to make a free throw in basketball, which requires you to throw the ball into the basket from a fixed position. Without the proper knowledge, one may observe professional players perform a free throw on YouTube by obtaining numerous exemplary videos. However, learning from every demonstration videos might lead to worse performance, as they may contain unsuitable or even irrelevant content.
The challenge of learning from noisy demonstration sets is as well crucial in the robot imitation learning regime, as demonstrations which are not aligned with achieving the intended goal deteriorate the learning process. The assumptions of optimality (or at least sub-optimality) of the demonstrations are often made in state-of-the-art imitation learning algorithms (Ross & Bagnell, 2010; Ross et al., 2011; Ho & Ermon, 2016; Sermanet et al., 2018). As a result, they are vulnerable to demonstrations that are potentially detrimental to the learning outcomes in the given set. To address assumptions of optimality, in this paper, we aim for a generic framework capable of learning from a noisy demonstration set, via evaluating the suitability of imitated skills judged by task specific heuristics.
Prior works have handled deteriorated imitated outcomes due to noisy demonstration sets by utilizing expected Q-values provided in the demonstrations to avoid learning from bad demonstrated actions (Kim et al., 2013; Nair et al., 2017; Gao et al., 2018). However, these works require demonstrations to have a rich representation such as incorporating the aforementioned Q-values, which may neither be available in other cases nor directly applicable to the agent training environment. Our framework, on the other hand, does not require demonstrations to contain specific information, and hence is able to cope with any forms of expert demonstrations. To be specific, we propose to first assess which demonstrations in the given set might be more suitable by learning from them, and then train the agent imitating only a selected subset.
In order to achieve selectively learning from suitable demonstrations, we examine if the learning outcomes are favorable after imitating each demonstration in a set. Typically in robot learning regime, we are able to receive designed feedbacks in the target training environment to evaluate how well the agent performs. Thus, in each task, we predefine specific heuristics for assessing if the agent exhibits imitation learning outcomes desirable for reaching the goals of the tasks.
The key challenges for the feasibility of assessing learned outcomes from each demonstration are the efficiency and generalization ability. A framework should be capable of producing these assessable outcomes as quick as possible, and generalizing to unseen demonstrations. To this end, we propose a framework with the demonstration suitability assessor leveraging meta-learning, where we train adaptive parameters via meta-imitation-learning. The meta-imitation-learned parameters can thus: (1) produce assessable imitated outcomes at testing time quicker than both imitation learning from scratch and fine-tune a pretrained initialization (Finn et al., 2017a), and (2) adapt to imitating newly sampled unseen demonstrations. Overall, the imitated outcomes after adaptation will be judged by task heuristics to indicate the suitability of certain demonstrations. We then train agents using imitation learning from the selected suitable demonstration subsets for obtaining better policies. We demonstrate two empirical approaches to utilize the resulting suitability assessments. One composes new subsets of demonstrations from the top ranked, the other iteratively fine-tunes the meta-learned parameters by strengthening or weakening certain demonstrations in a set according to current suitability judgments, producing a selected suitable subset at convergence.
In some cases, the distribution of demonstration sets can be imbalanced or multi-modal. To prevent over-fitting to certain subsets of such demonstration distributions, we augment the meta-imitationtraining with a regularization objective-maximization of mutual information between the demonstration and the induced behavioral differences from imitating it. This additional regularization term aims to make the meta-trained parameters more responsive to the demonstrations being imitated, and as a result, help differentiate better the adapted behaviors from a noisy and diverse set.
We test our framework on four different simulation sports environments in MuJuCo. Our results, both qualitative and quantitative, show that the proposed method outperforms various baselines, including vanilla MAML and fine-tuning a pretrained initialization, on learning better policies from noisy demonstration sets.

heading: 2 RELATED WORKS
text: Imitation Learning and Learning Diverse Behaviors In imitation learning, an agent tries to acquire skills by learning from demonstrated expert behaviors (Schaal, 1997; Argall et al., 2009; Ziebart et al., 2008; Kuniyoshi, 1989; Hussein et al., 2017; Ross et al., 2011). Ho & Ermon (2016) discover that imitation learning can be formulated as an adversarial training (Goodfellow et al., 2014) between demonstrations and imitated behaviors of the agent (GAIL), which is adopted in this work as the main imitation learning framework. Extension works of GAIL (Wang et al., 2017; Hausman et al., 2017; Li et al., 2017) augment the original algorithm with a latent code or embedding for imitating a diverse set of demonstrations, increasing diversity of the imitated outcomes. Duan et al. (2017) learn an encoder-decoder framework for digesting multiple demonstrations and achieving one-shot imitation learning during testing time. Hester et al. (2018) train a deep q-network by initializing the replay buffer with expert demonstrations. The demonstration assessing process of our work can also be viewed as leveraging the results of few-shot imitation learning from diverse demonstrations. However, the emphasis of our goal is centered on utilizing the assessments to acquire better skills learning from selected suitable demonstrations.
Learning from Imperfect Demonstrations Despite remarkable performances in imitative learning of complex control behaviors from prior works, most of the algorithms assume the optimality (or at least sub-optimality) of the demonstrations. The given demonstration set is usually presumed to contain only helpful demonstrations: it is not noisy. Kaiser et al. (1995) presents several types of experts violating the optimality assumption and basic ways to cope with them. Tomasello (2016) provides psychological evidences that infants imitate rationally, and that they are more prone to certain demonstrations that do not only fundamentally make sense but effectively help the development of their skills. Grollman & Billard (2011) utilize failure demonstrations as negative constraint on the exploration to prevent learning agent from approaching behaviors likewise. Kim et al. (2013) casts learning from limited demonstrations as a relaxed large-margin constrained optimization problem. Several recent works (Gao et al., 2018; Nair et al., 2017) also assume the expert representation contains expected Q-values and utilize them to filter bad demonstrated actions. However, these additional information may not be available in other cases such as learning from raw video demonstrations (Sermanet et al., 2016; 2018). Moreover, specifically for Q-values utilized in aforementioned works, the assumption of these information being applicable to the agent training
environment does not necessarily hold. We thus aim for a more general framework capable of coping with any forms of demonstrations, while this work mainly based off state-action pairs.
Meta Learning in Imitation Learning Our work is also closely related to a series of recent advancements in meta learning, that we as well seek fast adaptive parameters which should learn how to learn from certain demonstrations. Finn et al. (2017a) propose a gradient-based model-agnostic framework (MAML) for training a policy prior that can quickly adapt to newly sampled tasks during testing (termed as meta-testing) time by small number of gradient update steps. In another work (Finn et al., 2017b) MAML is adopted to train a set of adaptive parameters achieving gradient-based one-shot imitation learning during meta-testing time. As an extension of MAML with regularization in an inner imitative learning objective, we are, to our best knowledge, the first to utilize the meta learning framework to evaluate a demonstration set for learning a better policy from selected demonstrations.

heading: 3 PRELIMINARIES AND PROBLEM SETUP
text: A finite-horizon Markov Decision Process (MDP) can be defined asM = (S,A, P,R, γ, p0, T ), where S and A denotes state and action spaces, P : S × A× S → R+ denotes the state transition probability, R : S × A → R is a reward function, p0 an initial state distribution, γ is the reward discount factor, and T is the horizon of the MDP. Let τ = (s0, a0, ..., sT , aT ) be a trajectory of state and action pairs, the associated reward can be written as R(τ) = ∑T t=0 γ
tR(st, at). Reinforcement learning seeks a policy πθ(a|s) parameterized by θ that maximizes the expected total return: Eat∼πθ,st∼S [R(τ)]. In imitation learning regime, demonstrated expert behaviors are given while the reward function is missing, and the agent acquire desired skills learning from these demonstrations.
We now formulate our problem setup: given a noisy demonstration set V = {v1, ..., vn}, where each vi ∈ V can be represented as state-action pairs: {(si,t, ai,t)}, and several environment variants E = {e1, ..., ek}, where each ei denotes different goal in a multi-goal learning task. We require a suitability assessment score Svi |ej ∈ R of a demonstration vi under ej for judging and selecting suitable demonstrations. And hence, the assessment scores {Svi |ej} can be the reference for selecting the most suitable demonstration as arg maxi Svi |ej , vi ∈ V , or a subset {vopt} for imitation learning.

heading: 4 METHOD
text: Our goal is to learn a good policy from a noisy demonstration set by imitating the selected suitable demonstrations. In order to produce imitated outcomes efficiently for suitability evaluation, we utilize the adaptation process in meta-learning framework, Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017a), to meta-imitation-learn a set of fast adaptive parameters from the given demonstration set. At meta-testing time, the parameters should adapt to each sampled demonstration to produce assessable learning outcomes, leading to selected suitable subsets for the agent to learn from. To alleviate insensitivity imitating a particular demonstration to an unbalanced set distribution, motivated by (Chen et al., 2016), we regularize the inner imitation objective of MAML with maximization of mutual information between the demonstration and the induced behavioral differences from imitating it. Fig.1 depicts the overview of our framework, and the pseudo codes are provided in the appendix.

heading: 4.1 META-TRAINING FOR FAST ADAPTIVE PARAMETERS
text: Following the conventions as described in MAML (Finn et al., 2017a;b), meta-imitation-learning has two objectives, where the inner objective tries to optimize for adapting to a sampled individual demonstration, while the outer objective (meta objective) tries to optimize for fast adaptation across multiple newly sampled demonstrations. Denote fθ as the policy model mapping observations to actions parameterized by θ, the desired adaptive meta-parameters. The inner objective in the MAMLbased meta-imitation-learning is an imitation objective, where θ is updated to θ ′
i after learning from demonstration vi by gradient descent with learning rate α, which can be written as:
θ ′
i = θ − α∇θLvi(fθ), vi ∈ V (1)
Here Lvi denotes the imitation loss imitating the sampled demonstration vi. And hence the meta objective and the corresponding meta-imitation-update can be derived as (with meta learning rate β):
min θ ∑ vi∈V Lvi(fθ′i ), where θ ← θ − β∇θ ∑ vi∈V Lvi(fθ′i ) (2)

heading: 4.1.1 INNER IMITATION LEARNING UPDATE
text: The inner imitative update in Eq.1 is not limited to certain methods, we hence adopt two major imitation learning algorithms: generative adversarial imitation learning and behavioral cloning.
Generative Adversarial Imitation Learning (GAIL): Our primary adopted imitation learning method (Ho & Ermon, 2016). For each demonstration vi ∈ V , we maintain a discriminator Di dedicated to treating demonstration vi as real data in the GAIL setting contributing to Lvi in Eq. 1. Specifically, each inner update from θ to θ ′
i is computed with reward − log(Di(s, a)) via TRPO (Schulman et al., 2015) update rule. In this paper, we denote this reward as RIL and the sparse task reward as Rtask. In a standard imitation learning scheme, both RIL, Rtask should be taken into account. However, Rtask would bias the training of the adaptive parameters θ, that θ should be optimized for quick adaptation to any given demonstration rather than succeeding the task. Therefore, we drop the Rtask term and denote our reward definition as R(st, at) = RIL, 0 ≤ t ≤ T . In order to adapt to unseen demonstrations, we train a meta-discriminator Dmeta along with training the adaptive policy parameters θ. Similar to Eq.2, Dmeta is meta-updated as:
Dmeta ← Dmeta − β ∑ i [Êτi [∇log(D ′ meta,i(s, a))] + Êvi [∇log(1−D ′ meta,i(s, a))]], (3)
where τi denotes the trajectories sampled from fθ′i (after learning from demonstration vi), vi is the trajectory of demonstration as one episode of state-action pairs (expert trajectory in GAIL setting). Each Dmeta,i is the updated discriminator parameters during the inner update for demonstration vi, which follows the GAIL update rule for discriminator.
Behavioral Cloning: In behavioral cloning, the inner update in Eq.1 is simply a supervised training, where Lvi = 1T ∑ t ‖fθ(si,t)− ˆai,t‖2 using a mean squared error with ˆai,t denoting the expert action from demonstration vi at time-step t.

heading: 4.2 MUTUAL INFORMATION REGULARIZER
text: In order to overcome the potential challenge of learning from an unbalanced demonstration distribution, we augment the inner update in meta-training phase with a regularization loss: maximization of the mutual information I(vi,∆πvi) between the induced behavioral differences and the demonstration being imitated. ∆πvi here denotes the differences of behaviors induced after an inner update learns from a particular demonstration vi, which we estimate using a sample-based method. Maximizing I(vi,∆πvi) can be approximated by optimizing the following variational lower bound:
I(vi,∆πvi) ≥ E[logQ(vi|∆πvi)] +H(vi) = RMI(πvi , Q), (4)
where Q denotes the posterior network jointly trained with the adaptive policy parameters. Since vi is drawn from a fixed given demonstration distribution (a set), H(vi) can be treated as a constant, we thus only need to consider optimizing Q. Q can be trained by estimating ∆πvi through sampling
trajectories before and after the inner imitation update, denoted as τold and τnew. We implement Q using a siamese LSTM network taking as inputs both the old and new trajectories. The output of the posterior network is an encoded feature of the demonstration vi. We sample two sets of trajectories for both τold and τnew as one used for training Q, and the other (test set) for estimating E[logQ(vi|∆πvi)] to be used as the regularization reward. We then augment the original reward function in GAIL with RMI = −E[logQ(vi|∆πvi)] as follows:
R(st, at) = { RIL +RMI if t = 0 RIL if 0 < t ≤ T , RMI = −E[logQ(vi|∆πvi)] (5)
However, this regularization term should be applied to the parameters before the imitation update, since it is interpreted as the probability of such behavioral differences induced by a particular demonstration vi. Denote the parameters for inner imitation update from demonstration vi as θi, our inner objective thus becomes:
Lvi(fθi) = −Eτold∼fθi,vi [ T∑ t=0 Rvi(st,old, at,old)]
Rvi = RMIi +RILi ⇒ Lvi(fθi) = LMIi(fθi) + LILi(fθi)
(6)
and hence the update where we revert θ ′
i back to θi with the additional regularization term can be written as:
θi − α∇θiLvi,old(fθi) = θi − α∇θiLILi,old(fθi)− α∇θiLMIi,old(fθi)
= θ ′ i − α∇θiLMIi,old(fθi) (7)
Which implies we can compute the gradients with respect to the old trajectories but update on the new parameters θ ′
i. In practice, we implement an additional value network dedicated for this additional regularization reward. As for behavioral cloning, we update the policy parameters θ utilizing policy gradients computed from this RMI , while other parts remained as standard supervised training.

heading: 4.3 META-TESTING: JUDGING
text: During meta-testing, we evaluate each demonstration vi ∈ V and select the most suitable one(s) to imitate judged by some task heuristics K under environment variant ej . Each adaptation at metatesting time will be run for a few iterations. K is essentially a scoring function which takes as input trajectories generated from a policy π and outputs a real-valued number, ie. scoreπ = K(τπ) ∈ R. Denote πθ as the meta-trained policy, and πθi is the policy adapted to demonstration vi, the demonstration suitability assessment score is then computed as:
Svi |ej = K(τπθi |ej )−K(τπθ |ej ) (8) The most suitable demonstration viopt|ej is then selected by:
iopt = arg max i Svi |ej = arg max i K(τπθi |ej )−K(τπθ |ej ), ∀vi ∈ V (9)

heading: 4.4 APPROXIMATE OPTIMAL SUBSET SELECTION
text: To select a subset of suitable demonstrations to learn from, we design two empirical approaches that utilize assessment scores obtained in the meta-testing phase. One approach, which is further described in Sec. 5.2 as one of our evaluation metrics, takes the top-K ranked demonstrations as the selected subset. The other approach is an iterative algorithm that searches for a suitable subset using a slightly adjusted version of our meta-imitation-learning framework. The intuition for this approach is described as follows: we introduce a weight ci, initialized as 1.0, for each vi in the demonstration set. During meta-update, the meta-prior θ is updated by a weighted sum of gradients computed when adapting to each demonstration vi, ie. θ ← θ − β ∑ vi∈V ci · ∇θLvi(fθ′i ). After each meta-testing, ci is adjusted based on the difference between current assessment score Svi |ej and the baseline score obtained from the meta-parameters Sθ|ej . At convergence, we select a subset {vi|ci > }, where is a predefined threshold. We include the pseudo code for obtaining such subset in the appendix B.

heading: 5 EXPERIMENTS
text: The goals of our experiments are to: (1) Compare different methods on assessing and ranking a given demonstration set according to the suitability of each demonstration. (2) Examine if our meta-learning-based frameworks can generalize such suitability assessment to unseen demonstrations. (3) Provide empirical approaches to learn a better policy from noisy demonstration set. For brevity, we will term the demonstration suitability assessments as DSA. We evaluate the effectiveness of the following methods on learning from a noisy demonstration set in four experimental environments:
• Avg Fine-Tune: A framework that fine-tunes towards a particular demonstration with a set of parameters pre-trained using the entire demonstration set.
• MAML: The standard MAML framework as described in (Finn et al., 2017a). • MAML + MI: A MAML framework regularized with mutual information maximization of the

heading: 5.1 ENVIRONMENTS AND SETUPS
text: We built four different simulation sports environments as illustrated in Fig.2. These environments aim to resemble real sports activities and their task heuristics are intuitively designed to fit human preferences. We sample generated state-action pairs {(st, at)} from RL pre-trained agents during different stages of their RL training to compose the demonstration set. Details in the appendix A.
• Free Throw: As shown in Fig.2a, the thrower needs to make a basketball free-throw from a fixed position, where it scores if the ball is successfully thrown into the basket. Two datasets are tested containing 8 and 20 demonstrations respectively. The latter one with 20 demonstrations includes demonstrations from a taller thrower agent for verifying our concept that successful demonstrations are not necessarily the suitable ones, it has to depend on the learning agent.
• Penalty Save: As shown in Fig.2b, an ant agent is required to jump at the right timing with the proper force to block an automated incoming penalty shot. This is a multi-goal task since the incoming shots can have different directions and speed. There are 12 demonstrations in this dataset.
• Handstand: As shown in Fig.2c, a humanoid is required to withstand an upside-down handstand position. The appropriate body-pose is defined as: the hands must touch the ground while feet are raised over a threshold height. There are 15 demonstrations in this dataset.
• Martial Arts: As shown in Fig.2d, a humanoid is required to make a jump kick to kick a target placed higher than the height of the humanoid. There are 10 demonstrations in this dataset.

heading: 5.2 EVALUATION METRICS
text: Here we explain the metrics used for evaluating and comparing our method and the baselines:
• Top-K combinations: After obtaining an empirical rank of the given set sorted by the adapted heuristics scores, we can combine the top-K ranked demonstrations to compose some potentially better datasets. Eg. top-3 means such dataset consists of 3 demonstrations ranked as the top 3 in the set according to the heuristics scores generated during adaptation using one of the methods. The empirical rank generated by our method and the baselines are thus compared by the performances of agents learning from different top-K demonstration sets using imitation learning from scratch.
• Adapted Behaviors Distance (ABD): We compare the state-action pairs generated from the adapted parameters and the optimal behaviors (experts) by computing the following euclidean
distance: 1T ∑ i
∥∥s′i,t − ˆsi,t∥∥+ ∥∥a′i,t − ˆai,t∥∥, where (ŝi, âi) denotes the trajectories from demon-
stration vi, and (s′i, a ′ i) denotes the generated trajectories from parameters adapted to vi.

heading: 5.3 EXPERIMENTAL RESULTS
text: Top-K Combinations: As shown in Fig.3, each row consists of learning reward curves generated from imitating several top-K demonstration subsets composed by each method. For empirical studies, suppose there are N demonstrations in the given noisy set, we select subsets from top-1, top-2, top-N/2, and top-N , which is simply learning from the entire original noisy set.
• Free Throw (FT): as shown in Fig.3 top row, MAML + MI outperforms other baselines by resulting in better composition of top-K demonstration sets and successfully selects the most suitable demonstration from the set. The learning curves for the 20 demonstration set consisting behaviors from a taller expert agent can be found in the appendix C.2.
• Penalty Save (PS): as shown in Fig.3 second row, MAML + MI also outperforms other baselines. Notice that since this environment is a multi-goal learning task, the DSA should depend on the environment variants as Eq.9. The curves we show here are trained under a fixed environment variant, which are better illustrated in the videos. • Handstand (HS): as shown in Fig.3 third row, while learning from the top-K subsets selected
by MAML + MI achieves the best results, MAML also selects a considerably suitable top-1 demonstration. However, top-2 of MAML performs even worse than the baseline, implying that the second highest ranked demonstration negatively affect the agent outweighing the positiveness of the top-1 ranked demonstration. All of the top-K combinations composed by Avg Fine-Tune perform worse than the baseline supposedly due to wrongly selecting detrimental demonstrations.
• Martial Arts (MA): as shown in Fig.3 fourth row, it can be noticed that MAML + MI successfully achieves the task goal using the top-1 demonstration within 2,000 iterations, but neither MAML nor
Avg Fine-Tune succeeded even being trained further. For this environment, we further conducted experiments on generalization to unseen demonstrations and our approximate optimal subset selection algorithm, denoted as OSS. The golden thick curves are the learning reward curves imitating the selected subset by the OSS for each method. It can be seen that the performances of learning from our selected subset is significantly better in MAML + MI case, while other baselines fail to utilize this algorithm to select better suitable demonstration subset. Learning curves generated adapting to unseen demonstrations can be found in appendix C.3.
Qualitative Results The videos showing how the meta-trained parameters can adapt to certain demonstrations and their comparisons with different methods are shown on the project page.
Behavioral Cloning: The results on handstand and martial arts can be found in appendix C.4.
ABD: Table 1 shows the computed behavioral distances between the adapted behaviors and the expert behaviors, across the entire demonstration set for a mean score. It can be shown that MAML + MI outperforms other baselines by adapting better to the sampled demonstrations. MAML works the best in the free throw environment, potentially due to posterior network Q was not well-trained using the small size of dataset.

heading: 5.4 EFFICIENCY ANALYSIS
text: In order to verify the efficiency as compared to demonstration judging via training from scratch, We set a limited quota be the total number of iterations required for meta-training the adaptive parameters θ plus the number of gradient updates during adaptation to each demonstration, which sums up to around 6,000 and 7,500 for our martial arts and handstand environments respectively. We then evenly distribute this quota to train different agents imitating each of the demonstration from the noisy set, and output their top choices. In Fig.4, the heuristics curves generated up to 500 iterations produce the required assessment scores, we verified that the top choices from both environments are of non-suitable demonstrations, the associated reward curves are presented in the appendix C.6.

heading: 6 CONCLUSION
text: We propose a framework to tackle the challenging problem - learning a good policy through imitation learning from a noisy demonstration set. Our framework, built on top of MAML with a mutual information maximized regularization, learns a set of adaptive parameters from the given noisy set. The agent should exhibit significant learning outcomes after fast adaptation to certain demonstrations where these outcomes can be evaluated via predefined task heuristics. By being a learning framework, the system learns to discover the most suitable demonstrations for the agent from the expert rather than selecting based on hand crafted judging rules. For future research direction, we hope this work can serve as the first trial to lure more advanced research on tackling imitation learning from noisy demonstration sets.

heading: B PSEUDO CODES FOR OUR ALGORITHMS
text: Algorithm1 summarizes the proposed method for a training the fast adaptive set of parameters. During meta-testing, we adapt the meta-learned parameters to each demonstration in the set one at a time. Algorithm2 summarizes the proposed algorithm for meta-testing as the suitability assessing phase of our framework. We will use the same terminologies as in Sec. 3 and Sec. 4 in the main paper. For
example, we will denote the horizon of a trajectory as T to differentiate it from the entropy term H . Algorithm3 describes the proposed subset selection algorithm.
Algorithm 1 Info Meta Imitation Learning (Blued lines are GAIL ver.) Require: V = {v1, v2, ..., vn}: A set of demonstrations Require: E = {e1, e2, ..., ek}: An environment with k goals (k = 1 for 3 of our environments) Require: α, β: step size hyperparameters
1: randomly initialize θ, Q, Dmeta 2: Sample a batch of Demonstrations vi ∈ V 3: while not done do 4: for all vi do 5: Sample a batch of environments ej ∈ E 6: for all ej do 7: Sample K trajectories τ = {(s1, a1, ..., sT )} using fθ for updating Dmeta to D ′
meta,i
8: Evaluate∇LILi(fθ) using τ and vi, and update θ ′
i = θ − α∇LILi(fθ) with TRPO 9: Update Dmeta to D ′
meta,i with the gradient: Êτi [∇log(Dmeta(s, a))] + Êvi [∇log(1−Dmeta(s, a))]]
10: Sample K trajectories τ ′ ij = {(s′1, a′1, ..., s′T )} using fθ′i after imitating vi under ej 11: Update posterior Q with τ , τ ′
ij , and encoded vi 12: Sample new trajectories using fθ and fθ′i as τnew and τ ′ ij,new respectively 13: Compute RMIi with Q, using newly sampled τnew and τ ′
ij,new as (5) 14: Evaluate ∇LMIi(fθ) with RMIi , update θ ′′ i = θ ′
i − α∇LMIi(fθ′i ) with TRPO as (7) 15: Sample K trajectories τ ′′
ij = {(s′′1 , a′′1 , ..., s′′T )} using fθ′′i 16: end for 17: end for 18: Meta Update the θ using all the τ ′′
ij with TRPO update rules as (2) 19: Meta Update the Dmeta using all the gradients collected from D ′
meta,i as (3) 20: end while

heading: C MORE RESULTS
text: C.1 ADAPTATION CURVES
In Fig.5, we show the adaptation curves from each of the method in every environments we tested on.
C.2 FREE THROW WITH TALLER AGENT
As shown in Fig.6, both top-1 and top-2 obtained by MAML + MI are suitable demonstration subsets, with demonstration 1 and 5 as top-2 as depicted in Fig.5 top row. Demonstration 2 in this set is actually a demonstration recorded from a much taller agent, which we show in this top-K curves it is not selected as a suitable one despite being a successful demonstration achieving the free throw task. This phenomenon is better illustrated in the videos.
Algorithm 2 Meta-Testing as Judging (Blued lines are GAIL ver.) Require: V = {v1, v2, ..., vn}: A set of demonstrations Require: E = {e1, e2, ..., ek}: An environment with k goals (k = 1 for 3 of our environments) Require: α, β: step size hyperparameters Require: max_iter: maximum iterations for the meta-testing Require: θ: the meta-trained parameters Require: Pre-defined Task Heuristics K Require: Meta-Trained Discriminator Dmeta
1: for all ej ∈ E do 2: for all vi ∈ V do 3: Initialize θij as meta parameters θ 4: Initialize Di ← Dmeta 5: for iter = 0 to max_iter do 6: Sample K trajectories τ = {(s1, a1, ..., sT )} using fθij 7: Update Di with the gradient: Êτ [∇log(Di(s, a))] + Êvi [∇log(1−Di(s, a))]] 8: Compute RIL imitating vi with Di 9: Evaluate∇LIL(fθij ) using τ and vi, and update θij ← θij − α∇LIL(fθij ) with TRPO
10: end for 11: Evaluate θij for M random seeds 12: Compute DSA score Svi |ej = K(τπθij |ej )−K(τπθ |ej ) 13: end for 14: Record the index iopt for the most suitable demonstration under ej with iopt = arg maxi Svi |ej , ∀vi ∈ V 15: end for
Algorithm 3 Optimal Subset Selection Algorithm Require: V = {v1, v2, ..., vn}: a set of demonstrations Require: N : predefined iterations Require: β: constant for multiplying and dividing weights of demonstrations Require: : threshold for selecting final subset of demonstrations Require: W = {w1, w2, ..., wn}: weights for meta-learning each demonstration.
1: Initialize all wi to 1 2: while not done do 3: Update θ with Imitation Learning using current set V for N iterations, applying weights W for the demonstrations during meta-update. 4: Get heuristic score for meta-trained parameters θ, store in S 5: for all vi in V do 6: Set ∆K,ij = Svi |ej − S 7: if ∆K,ij > 0 then 8: Set wi = wi · β exp(∆K,ij) 9: else
10: Set wi = wi · β−1 exp(−∆K,ij) 11: end if 12: end for 13: Sprev = S 14: end while 15: Set V = {vi | wi > } 16: return V.
C.3 UNSEEN DEMONSTRATION IN MARTIAL ARTS
In Fig.7 we show that our framework can also work on selecting suitable demonstrations given an unseen demonstration set. We meta-imitation-learned on a large set of demonstrations consisting of 100 martial arts demonstrations and tested it on a disjoint random sampled demonstration set. From the reward curves, although requiring more number of iterations to train, MAML + MI selects a subset that eventually succeeds achieving the goal of the task, while other two baselines struggle finding a suitable top-K subset.
C.4 IMITATION LEARNING VIA BEHAVIORAL CLONING
In Fig.8, we show learning reward curves using behavioral cloning as the inner imitative update in Eq.1 on two of our harder environments, handstand and martial arts. For handstand environment, the top-2 and top-7 produced by MAML + MI are significantly consistently better than those from other two baselines, implying the top 7 ranked demonstration selected from MAMl + MI are of the more suitable subset. Interestingly, MAML produce reasonably good top-2 as well, which is almost on par with the MAML + MI. We hypothesize that behavioral cloning does not introduce too much randomness requiring further regularizing vanilla MAML. However, behavioral cloning can not produce perfect handstand results due to the complexity of such task, while our primarily adopted imitation learning algorithm GAIL successfully produced perfect imitative outcomes. For martial arts, both MAML + MI and MAML produce reasonable results, while MAML + MI produce better top-1 achieving better overall performances. The Avg Fine-Tune method surprisingly works in this environment with behavioral cloning, hypothetically due to the steadiness of supervised training.
C.5 SUBSET SELECTION METHOD
The trends of each subset weight ci and how they change throughout the subset selection training is as shown in Fig.9. We again sub-sample only the best 3 evaluated weights and the worst 3. It is obvious that MAML + MI diverges in a more consistent fashion and produce more reasonable final results.
C.6 RESULTS FROM EFFICIENT ANALYSIS
In Fig.10, we show the learning reward curves imitating the top-K demonstration combinations generated by training from scratch and snapshot at 500 iterations as described in Sec.5.4. It is obvious that these learning outcomes are significantly worse than those presented in Fig.3 generated
by MAML + MI and MAML. This verifies that in most cases, training from scratch is not only inefficient but also prone to more randomness if reinforcement learning is involved (in our case, GAIL is involved).

Review summaries:
The paper makes its intent plainly clear, it wants to remove the assumption that demonstrations are optimal.  Thus it should show that in a case that some demonstrations are bad, it outperforms other methods which assume they are all good. The method proposed, while interesting, well-conceived and potentially novel, is not convincingly tested to this end.