Watch, Try, Learn: Meta-Learning from Demonstrations and Rewards
Authors: Allan Zhou
Eric Jang
Daniel Kappler
Alex Herzog
Mohi Khansari
Paul Wohlhart
Yunfei Bai
Mrinal Kalakrishnan
Sergey Levine
Chelsea Finn
Venue:  (2020)
Keywords: meta-learning
reinforcement learning
imitation learning

Abstract: Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging experience from learning similar tasks. In the presence of task ambiguity or unobserved dynamics, demonstrations alone may not provide enough information; an agent must also try the task to successfully infer a policy. In this work, we propose a method that can learn to learn from both demonstrations and trial-and-error experience with sparse reward feedback. In comparison to meta-imitation, this approach enables the agent to effectively and efficiently improve itself autonomously beyond the demonstration data. In comparison to meta-reinforcement learning, we can scale to substantially broader distributions of tasks, as the demonstration reduces the burden of exploration. Our experiments show that our method significantly outperforms prior approaches on a set of challenging, vision-based control tasks.

Published as a conference paper at ICLR 2020
WATCH, TRY, LEARN: META-LEARNING FROM DEMONSTRATIONS AND REWARDS
Allan Zhou' & Eric Jang
Daniel Kappler & Alex Herzog
Google Brain
X
{allanz, j ggoogle.com
(kappler, exexherzoggex.tee
Mohi Khansari, Paul Wohlart, Yunfei Bai & Mrinal Kalakrishnan
(khansari, wohlhart, punfebbai,kalakris// team
Sergey Levine
Chelsea Finn
Google Brain, UC Berkeley
Google Brain, Stanford
slevine@google.com com
chelseaff@ooogle... com
ABSTRACT
Imitation learning allows agents learn complex behaviors from demonstrations. However, learning complex vision-based task may require an impractical num- ber of demonstrations. Meta-imitation learning enabling agents to learn new task from one or a few demonstrations by lever- aging experience from learning similar tasks. In the presence of task ambiguity unobserved dynamies, demonstrations alone may not provide enough informa- tion; an agent must also try the task to successfully infer a policy. this work, we propose method that can learn learn from both demonstrations and trial-and- error experience with sparse reward feedback. In comparison to meta-imitation, this approach enables the agent to effectively and efficiently improve itself tonomously beyond the demonstration data. In comparison to meta-reinforcement learning, we can scale to substantially broader distributions of tasks, the demon- stration reduces the burden of exploration. Our experiments show that our method significantly outperforms prior approaches on set of challenging, assion-based control tasks.
promising approach towards
INTRODUCTION
Imitation learning enables autonomous agents to learn complex behaviors from demonstrations. which are often easy and intuitive for users to provide. However, learning expressive neural net- work policies from imitation requires a large number of demonstrations, particularly when learning from high-dimensional inputs such images. deta-imitation learning has emerged approach for allowing an agent to leverage data from previous tasks in order to learn a new task from only handful of demonstrations (Duan al. 2017 Finn et al. 2017b James et al.) 2018). However, in many practical few-shot imitation settings, there is an identifiability problem: it may be possible precisely determine a policy from one or few demonstrations, especially in new situation. And even if a demonstration precisely communicates what the task entails, it might not precisely communicate how accomplish it in new situations. For example, it may be difficult to discern from single demonstration where to grasp an object when it is in new position or how much force to apply in order to slide an object without knocking it over. It may be expensive to col- lect more demonstrations to resolve such ambiguities, and even when we can, it may not be obvious to a human demonstrator where the agent's difficulty is arising from. Alternatively, it is easy for the user to provide success-or-failure feedback, while exploratory interaction is useful for learning how to perform the task. To this end, our goal is to build an agent that can first infer policy from
promising
Work done Google Al Resident.

---

Published as a conference paper at ICLR 2020
demonstration
trial
retrial
Figure 1: Each column displays the first and last frame of episode top-to-bottom. After watching one demonstration (Ieft), the seene is re-arranged. With one trial episode (middle), our method can learn solve the task (right) by leveraging both the demo and trial-and- and-crror experience.
one demonstration, then attempt the task using that policy while receiving binary user feedback, and finally use the feedback to improve its policy such that it can consistently solve the task.
This vision of learning new tasks from a few demonstrations and trials inherently requires some amount of prior knowledge or experience, which we can acquire through meta-learning across range of previous tasks. To this end, we develop a new meta-learning algorithm that incorporates elements of imitation learning with trial-and-error reinforcement learning. In contrast to previous meta-imitation learning approaches that learn one-shot imitation learning procedures through imita- tion (Duan et al. 2017 Finn et al. 2017b), our approach enables the agent to improve the test task through trial-and-error: Further, from the perspective meta-RL. algorithms that aim to learn effi- cient RL procedures (Duan et al. 2016; Wang et al.) 2016: Finn al., (017a). approach also has significant appeal: as we aim to scale meta-RL towards broader task distributions and learn increas- ingly general RL procedures, exploration and efficiency becomes exceedingly difficult. However, a demonstration can significantly narrow down the search space while also providing practical means for user communicate the goal, enabling the agent to achieve few-shot learning of be- havior. While the combination of demonstrations and reinforcement has been studied extensively in single task problems (Kober et al., 2013; Sun et al., 2018 Rajeswaran et al. 2018 Le et al., 2018), this combination is particularly important in meta-learning contexts where few-shot learning of new tasks is simply not possible without demonstrations. Further, we can even significantly improve upon prior methods that study this combination using meta-learning more effectively integrate the information coming from both sources.
The primary contribution of this paper is meta-learning algorithm that enables learning of new behaviors with a single demonstration and trial experience. After receiving a demonstration illus- trating goal, the meta-trained agent can learn accomplish that goal through small amount of trial-and-error with only binary success-or-faile prior methods challenging, vision-based control problem involving manipulation tasks from four distinct families of tasks: button putton-pressing, grasping, pushing, and pick and place. We find that our approach can effectively learn tasks with new, held-out objects using one demonstration and single trial, while significantly outperforming meta- vett-imitation learning, meta-reinforcement learning, and prior methods that combine demonstrations and reward feedback. To our knowledge, our exper- iments are the first to show that meta-learning can enable an agent to adapt new tasks with binary reinforcement signals from raw pixel observations, which we show with a single meta-model for variety of distinct manipulation tasks. We have published videos of our experimental results] and the experiment model codd
labels. We evaluate our algorithm and several
2 RELATED WORK
Learning to learn, or meta-learning, has a long-standing history in the machine learning litera- ture (Thrun & Pratl 1998. Schmidhuber 1987 Bengio al. 1992, Hochreiter et al. (001). We particularly focus on meta-learning in the context of control. Our approach builds on and signifi-
orgripper
2

---

Published as a conference paper at ICLR 2020
cantly improves upon meta-imitation learning (Duan al. 2017 Finn et al. 201 James al.
2018 Paine et al. 2018) and meta- reitforeiioor learning Duan et al. 2016; Wang et al.
2016
Mishra et al.) 2018: Rakelly et al. 2019). extending contextual meta- veeaalearning approaches.
Unlike
prior work in few-shot imitation learning (Duan et al. 2017] Finn et al.) 2017b; Yu et al.,
2018
James al. 2018 Paine et al. 2018),
method enables the agent to additionally improve upon
trial-and-crror experience. In contrast to work in multi-task and meta-reinforcement learning (Duan
al.
2016 Wang et al. 2016 Finn et al. 2017a; Mishra al. 2018 [Houthooft et al. 2018 Sung
et al. 2017] Nagabandi et al.) 2019| SSemundsson et al.) 2018 Hausman et al. (017)) our approach
learns to use one demonstration to address the meta-exploration problem (Gupta et al., 2018: Stadie
et al. 2018). Our work also requires only one round of on-policy data collection, collecting only
1, 1,500 trials for the vision-based manipulation tasks, while nearly all prior meta-learning works re-
quire thousands of iterations of on-policy data collection, amounting to hundreds of thousands of
trials (Duan et al. 2016; Wang et al. 2016: Finn et al. 2017a Mishra et al., 2018).
Combining demonstrations and trial-and-error experience has long been explored in the machine learning and robotics literature (Kober et al.) 2013). This ranges from simple techniques such as demonstration- based pre-training and initialization (Peters & Schaal, 2006 Kober & Peters 2009 Kormushev et al., 2010; Kober et al.) 2013) [Silver et (016) to more complex methods that incor- porate both demonstration data and reward information in the loop of training (Taylor et al.] 2011
Brys et al. 2015: Subramanian et al. 2016 Hester et al.) 2018 Sun et al. 2018 |Rajeswaran et al,
2018 Nair al. 2018 Le et al. 20188. The key contribution of this paper is an algorithm that can learn how to learn from both demonstrations and rewards. This is quite different from RL algorithms that incorporate demonstrations: learning from scratch with demonstrations and RL involves slow, iterative learning process, while fast adaptation with a meta-trained policy involves extracting in- herently distinet pieces of information from the demonstration and the trials. The demonstration provides information about what to do, while the small number of RL trials can disambiguate the task and how it s refinement. As a result, we get a procedure that significantly exceeds the efficiency of prior approaches. requiring only one demonstration and one trial to adapt to new test task, even from pixel observations, by leveraging previous data. In comparison, single-task methods for learn- ing from demonstrations and rewards typically require hundreds or thousands of trials to learn tasks of comparable difficulty (Rajeswaran et al. 2018 Nair et al. 2018).
3 META-LEARNING FROM DEMONSTRATIONS AND REWARDS
We first introduce some meta-learning preliminaries, then formalize our particular problem state- ment before finally describing our approach and its implementation.
3.1 PRELIMINARIES
Meta-learning, or learning to learn, aims to learn new tasks from very little data. To achieve this meta-learning algorithm can first meta-train a set of tasks (Tr) called the meta-train tasks. We then evaluate how quickly the meta-learner can learn an unseen meta-test task T,. We typi- cally assume that the meta-train and meta-test tasks are drawn from some unknown task distribution P(T) FFin 2018). Through meta-training, a meta-learner can learn some common structure be- tween the tasks in (TT) which it can use more quickly learn new meta-test task.
ri, We define a task Ti as a finite-horizon Markov decision process (MDP), AS,A,ri, H). with continuous state space S, continuous action space A, reward function r; S x A R. unknown dynamics ( St, at), and horizon H. In our manipulation experiments we will restrict the tasks to sparse binary rewards r; : + {0, 13. The state space S, reward r;. and dynamics P. may vary across tasks.
3.2 PROBLEM STATEMENT
Our goal is to meta-train an agent such that can quickly learn new test task T, in two phases:
Phase I: The agent observes and learns from k = 1,..., K task demonstrations Dj- = (d It can then attempt the task in = -,.... L trial episodes {T for which receives reward labels.
Phase II: The agent learns from those trial episodes and the original demos to succeed Ty-
A demonstration is a trajectory d = {(s,, }t} of T states-action tuples that succeeds at the task, while trial episode is a trajectory = {(st, az ra( at))} that also contains reward information.
3

---

Published as a conference paper at ICLR 2020
3.3 LEARNING TO IMITATE AND TRY AGAIN
With the aforementioned problem statement mind, we aim to develop method that can learn to learn from both demonstration and trial-and-error experience. We wish to (1) meta-learn a Phase I policy that is suitable for gathering information about a task given demonstrations, and (2) meta- learn Phase II policy which learns from both demonstrations and trials produced by the Phase I policy. Like prior few shot meta-learning works (Finn et al.) 2017b; Duan et al. 2017). we hope to achieve few shot success
task by explicitly meta- feata-traning Phase and Phase II policies to learn from very little demonstration and trial data.
We write the Phase policy as aa(a, (dr,)), where represents all the learnable parame- ters. x² conditions the task demonstrations {d,,*) } which helps it infer what the unknown task is before attempting the trials. We condition the Phase II policy on both demonstration and trial data and write it as T (als, (d {Te,)). with parameters Φ. One simple yet naive ap- proach would be
single model across both phases: e.g., using a single MAML (Finn) jet al.] 2017a) policy that sees the demonstration, executes trial(s) in the environment and then adapts its own weights from them. However, key challenge in meta-training such single model is that updates based on Phase II behavior (after the trials) will also change Phase be- havior (during the trials). Thus each meta-training update changes the distribution of trial tra- jectories Ti, ~ (a]s, sss, {di, that the Phase II policy learns from. Prior meta-reinforcement learning work (Duan et al. 2016; Finn al. 2017a) have addressed the issue of changing trial distribution by re-collecting on-policy trial trajectories from the environment after every gradient step during meta-training, but this can be difficult in real-world problem settings with broad task distributions, where it is impractical to collect large amounts of on-policy experience. Instead, we represent and train my. n sep- arately, decoupling their optimization. In particular, we train Th first, freeze its weights, and collect trial data [T. from the environment for each meta-training task T;. We then train wy using our col- lected trial data. Crucially, and é are separate so training Th will not change Th behavior, and the distribution of trial data (T.,} for each task remains station- ary. How do we train each of these policies with ooo-cc demonstration and trial data? ne must be trained in way that will provide useful exploration for in- ferring the task. One simple and effec- tive strategy for exploration is posterior Thompson sampling (Russo et al.] [0188] Rakelly al. 2019). I.c. greedily act cording to the policy's current belief of the task. To this end, train n' using a ineta-imitation learning setup, where for each task Ti we assume access to a of demonstrations D. x¹ conditions on K demonstrations (d+,} C Dj and aims maximize the likelihood of the actions under another demonstration of the same task d E DJ (the test demo is distinct from the conditioning demos). This gives the following Phase loss for Ts:
Demos for each task T,
min 0
Σ
C'
[T.
TEIT
1. Train Phase policy mp.
Demo orial
{Tee) ~ sh(aae, (d
each task
2. Collect trials using trained rd.
min 1
Σ c D, v.)
HTH
T.E(T)) Train Phase policy x
First, meta-train m) according Eq. 2 Next, we collect trial trajectories per meta-training task T; in the environment using our trained m We denote the trial (Tr (als, (dr,))), (d.,e) ), and store the resulting dema-trial pairs [(1d,,). {TrJ))) }} in new dataset (blue). Finally, we meta-train n according to Eq, 4
Figure 2: Meta-training Overview:
L (0, D.) E (d.,.) Ear- ((..)) [ [-
log ml (ar|st, (d.,a))
Then we can meta-train mb by minimizing Eq. across the set of meta-train tasks (Ti):
1
min [T.)]
Σ
C'(0,D))
(TE(T.)
We train n in a similar fashion, but additionally condition on L trial trajectories (Tre), which the result of executing It in the environment. Suppose for any task Tr, we have set of demo-trial
4

---

Published as a conference paper at ICLR 2020
Algorithm Watch- Beach-TTryLeaarc Meta-training
Algorithm Watch- testing
Meta-
Input: Training tasks {T:}
2: Input: Demo data D = } per task T;
1: Input: Test tasks {T₃)
3: Input: Number training steps N
Input: Demo data Dj for task T
4: Randomly initialize 0, db
for Tj E (Tz} do
5: for step = ,,...... .N do
Sample K demonstrations
6: Sample meta-training task TV
{d;,}} ~ Dj
7: Update ! with (0 D; (see Eq. !
Collect L trials {Tui} with policy
8: end for
m/(ass, {dz,))
9: for E {T:) do
6: Perform task with re-trial policy T [alas {djah (Taa)
Initialize empty D; for demo-trial pairs.
11: while not done do
end for
Sample K demonstrations {dr,.} D Collect L trials {TrJ) To (als, {d.,)) 14: Update D. - D; U ((((da)))) (T,)))}
end while
16: end for
for =tep=1..... do
18: Sample meta-training task Ti
Update á with V (cb. D, D;) (see Eq.
20:
end for
21:
return 0
pairs D: = x(fdr,)s (.A.))(TT))) (T:SH)}. Then the Phase objective for Ti is:
C (o, D, (id "Dr" (d...) (sr..))))) [- log T aa (dr,), {r,tt)
Eq. encourages n to use and improve upon the trial experience {T:.e), which includes reward information. If a trial has high reward, then n! likely inferred the task correctly from demonstration alone, and x²I should reinforce that high reward trial behavior. If a trial has low reward, then x! probably inferred the task incorrectly and n should avoid that low reward trial behavior. Hence even failed trials can help nh correctly infer the task by showing it what not to do. Note that since Eq. 3 evaluates log likelihood at states and actions from the held out demonstration d! nh cannot simply copy behavior from high reward trials in {T.,e but instead must generalize from the trials to new instance of the same task. We meta- meta-train T by minimizing Eq. across the meta-train tasks:
min 1
Σ T.E(T))
( (...D.,))
KT.H
We refer to our approach as Watch- Tra-Learn (WTL), and describe our meta-training and meta-test procedures in detail in Alg. [ and Alg. 2 respectively. We also illustrate the meta-training flow in Fig, 2 In practice we meta-train tratttrainn T and म by solving Eqs. 2 and Lusing stochastic gradient descent or some variant. We iteratively sample (minibatches of) tasks T; to compute gradient updates on respectively. At meta-test time, for any test task T, receive demonstrations (d,,.) and obtain the lemo-conditioned Phase I policy Tyf (als, {d,,)). We collect trial(s) (T₃₁} using Ti in the environment. Then we obtain the Phase II policy (als,, (d (T,,)). Finally, we execute in the environment to solve the task.
the π. g
3.4
WATCH-TRY-LEARN IMPLEMENTATION
WTL and Alg. I allow for general representations of the Phase policy (a(s,, {d.,+.) so long as conditions on the task demonstrations {d.,). Similarly, the Phase II policy π <a(ss,, s, (dr,)) J. (T.,e) must condition both (d,,e) and the trials (7.e). Hence variety of adaptation mechanisms could
be used.
We choose implement this conditioning in each policy by embedding the demonstration data and (for Phase II) the trial data into context vectors using neural networks. Figure 33 the r architecture assuming single demonstration and trial. The embedding network first applies

---

Published as a conference paper at ICLR 2020
(35 108.3
Clsssian Stidue
Fail
Carconsation (ppack MRLT
Cumeet Coese
Finpa
Coeo Eoaade Gropie Here (i
polby parmsron
Trur Secec the
Figure 3: Our vision-based Phase II architecture: Upper left: pass the RGB ohservation for each timestep through a 4-layer CNN with ReLU activations and layer normalization, followed by spatial softmax layer that extracts 2D keypoints (Levine et al. 2016). We flatten the output keypoints and concatenate them with the current gripper pose, gripper velucity, and context embedding. Upper Right: We pass the resulting vector through the actor network, which predicts the parameters of a Gaussian mixture over the commanded end- effector position, axis-anglc orientation, and finger angle. Lower left: To produce the context embedding, the embedding network applics a vision network to 40 ordered observations sampled randomly from the demo and trial trajectories. We concatenate the demo and trial outputs with the trial episode rewards along the embedding feature dimension, then apply a 10x1 convolution across the time dimension, flatten, and apply a MLP to produce the final context embedding. The Phase I policy architecture (Appendix Fig. is the same as shown here, but omits the concatenation of trial embeddings and trial rewards.
vision network to each demonstration and trial trajectory image 8t, producing demo and trial feature matrices (respectively) where each row corresponds one timestep's features. We concatenate the demo and trial feature matrices together on the feature (horizontal) dimension, along with the trial episode rewards produce single matrix combining demo and trial observation information and trial reward information. We then apply 1-D convolution along the time (vertical) dimension of this matrix and flatten to obtain a single vector that integrates and aggregates information across time. Finally we apply small MLP to produce the context embedding vector which contains information about both the demos (d.,*} } and the trials [T. The Phase policy m/.- illustrated Appendix Figure 8 produces similar context embedding but only uses demonstration data. This architectural design resembles prior contextual meta-learning works (Duan et al.) 2016 Mishra et al.) [2018) Duan et al.) 2017: Rakelly et al.] 2019). which have previously considered how meta-learn efficiently from one modality of data (trials or demonstrations), but not how to integrate multiple sources, including off-policy trial data.
Since each policy is additionally conditioned on the current state 8, we concatenate the context embedding with the current state features before feeding both input an actor network, which produces a Gaussian mixture distribution (Bishop) 1994) over actions. The current state features in- clude visual features produced by another vision network, distinct from the one used to produce the context embedding. Both vision networks use a fairly standard convolutional neural network archi- tecture with final spatial softmax layer that extracts keypoints (Levine et al. 2016). The entire T architecture, illustrated in Figure 3 is trainable end-to-end using backpropagation on Eq. 4 where the parameters é represent the collective weights of all layers. Similarly, the entire Th architecture depicted in Appendix Figure is trained by backpropagation on Eq. 2 where 0 represents the set of weights across all layers. Note that since each neural network architecture is fixed and shared across tasks, we expect the input state dimensions to be the same across tasks, though the content (for example, the objects in the scene) may vary.
4 EXPERIMENTS
In our experiments, aim evaluate our method on challenging few-shot learning domains that span multiple task families, where the agent must use both demonstrations and trial-and -nr-oor-error to effectively infer a policy for the task. Prior meta-imitation benchmarks (Duan al., 2017 Finn

---

Published as a conference paper at ICLR 2020
Button Pressing
Grasping
Pushing
Pick and Place
Figure 5: Illustration of example episodes in four distinct task families: button pressing, grasping, sliding, and pick-and-place Each column shows the first and last frames of an episode lop-to-bottom. We meta-train each model on hundreds of tasks from each of these task families. For each task within task family, unique pair kitchenware objects sampled from set of nearly one hundred different objects.
itt al. 201 generally contain only a few tasks, and these tasks can be easily disambiguated given single demonstration. Meanwhile, prior meta-reinforeemmet learning benchmarks (Duan et al., 2016 Finn et al.] (017a) tend to contain fairly similar tasks that meta-learner can solve with little all. Motivated by these shortcomings, we design two new problems where a meta-learner can leverage a combination of demonstration and trial experience: a toy reaching problem and a challenging multitask gripper control problem, described below. We evaluate how the following methods perform in those environments:
exploration and no demonstration
BC: A behavior cloning method that does not condition on either demonstration or trial-and rial-and-erroo -rroor ex- perience, trained across all meta- veeaatraining data. We train BC policies using maximum log- liglliieldood with expert demonstration actions.
MIL (Finn al. 2017b; James et al. 2018): A meta-imitation learning method that conditions on demonstration data, but does not leverage trie--and-rrror experience. We train MIL policies to [ag--ssing similar to the WTL Phase policy, but MII methods lack Phase II step. To perform a controlled comparison, we use the same architecture for both MII. and WTL.
minimize Eq.
WTL: Our Watch-Try-Leam method, which conditions demonstration and trial experience. In all experiments, the agent receives K- = - demonstration and can take L = trial.
BC SAC: In the gripper environment study how much trial-and-error experience soft actor critic (SAC) (Haarnoja et al.] 20088), state of the art reinforeement learning algo- rithm, would require solve single task.
ReacVing Enwiooment Ratura
Netivl (pent)
Chrort
While WTL meta-learns single model that needs just one trial episode per meta-test task, in "BC SAC" we fine-tune a separate RL agent for each meta-test task and analyze how much trial experience it needs match WTL's single trial performance. We pre-traie train policy similar BC, then fine-tune for each meta-test task using SAC.
Step
Figure Average return of each method held out meta-test tasks in the reaching environment, after one demonstration and one trial. Our Watch-Try- Try-Lemm (WTL) method is quickly able to learn imitate the demonstrator. Each line shows the average over 5 sepa- rate training runs with identical hyperparameters, eval- uated on 50 randomly sampled meta-test tasks. Shaded regions indicate 95% confidence intervals.
4.1 REACHING
ENVIRONMENT EXPERIMENTS
To first verify that our method can actually leverage demonstration and trial experience in simplified problem domain, planar reaching tasks inspired by Finn et al. (2017b) and illustrated in Appendix Fig 7 demon- strator shows which of two objects to reach towards, but the agent's dynamics randomized per task and may not match the demonstrator's s. This simulates a domain adaptive setting such as a robot imitating video of human. Since the demonstrations (d,,a) do not help identify the unknown
begin with toy

---

Published as a conference paper at ICLR 2020
Success Rate (Vision) Success Rate (State Space)
0.7
0.6
WTL (ours)
MIL
0.5
0.3
0.2
0.1
0.6
0.5
0.3
0.1
0.0
All
Button Pressing
Grasping Task Family
Pushing
Pick and Place
Figure 6: The average success of different methods in the gripper control environment, for both state space (non-vision) and vision based policies. The leftmost column displays aggregate results families. Our Batch-Try-Lear (WTL.) method significantly outperforms which turn outperforms the behavior cloning (BC) baseline. We conducted 5 training runs of each method with identical hyperparameters and evaluated each run on 40 held out meta-test tasks. Error bars indicate 93% confidence intervals.
all task meta-imitation (MIL) baseline,
task dynamics, the agent must trial episodes successfully reach the target object. During meta-trainie teata-training WTL meta-learns from demonstrations with the correct dynamics (dl in Eq.3) how adapt unknown dynamics in tasks. To obtain expert demonstration data, first train inforcement learning agent using normalized advantage functions (Gu et al. 2016), where the agent receives oracle observations that show only the true target. With our trained expert demonstration agent, we collect 2 demonstrations per task for 10000 meta-training tasks and 1000 meta-test tasks. For these toy experiments we use simplified versions of the architecture in Fig. 3 as described in Appendix
The results in Fig show WTL is able to quickly learn imitate the expert, while methods that do not leverage trial information struggle due to the uncertainty in the task dynamics.
4.2 GRIPPER ENVIRONMENT EXPERIMENTS
The gripper environment realistic 3-D simulation as shown Figure 5 Gripper tasks fall into four broad task families: button pressing, grasping, pushing, and pick and place. Within task family, each task involves a different pair of kitchenware objects sampled from a set nearly one hundred. Unlike prior meta-imitation learning benchmarks, tasks of the same family that qual- itatively similar still have subtle but consequential differences. Some pushing tasks might require the agent to always push the left object towards the right object, while others might require the agent to always push, for example, the cup towards the teapot. The agent controls a free floating gripper with 7-D action space. The vision based policies receive image observations and gripper state, while state-space policies receive vector of the gripper state and poses for all non-fixed ob- jects in the scene. Gripper environment episodes have a maximum length of seconds and contain sparse binary rewards. In an HTC Vive virtual reality setup, human demonstrator recorded 1536 demonstrations for 768 distinct tasks involving 96 distinct sets of kitchenware objects. We held out 40 tasks corresponding to 5 sets of kitchenware objects for meta-validatin dataset, which used for hyperparameter selection. Similarly, we selected and held out object sets of 40 tasks for meta-test dataset, which used for final evaluations. Refer to Appendix A for more detailed description of the scene setup, task families, and reward functions.
We trained and evaluated MIL, BC, and WTL policies with both state-space observations and vision observations. Appendix C describes hyperparameter selection using the velaavalidation tasks and Appendix C.3 analyzes the sample and time complexity of WTTL. The MII. policy uses identical
8

---

Published as a conference paper at ICLR 2020
architecture and objective to the WTL trial policy, while the BC policy architecture is the same as the WTL trial policy without the any embedding components. For vision based models, we crop and resize image observations from 300 x 220 to 100 x 100 before providing them imput.
We show the meta-test task success rates in Fig. 6 Overall, in both state space and vision domains, find that WTL outperforms MIL and BC by substantial margin, indicating that it can effectively leverage information from the trial and integrate it with that of the demonstration in order to achieve greater performance.
Finally, for the BC + SAC comparison we pre-trained an actor with behavior cloning and fine-tuned 4 RL agents per task with identical hyperparameters using the TFAgents (Guadarrama et al. 2018) SAC implementation. Table [ shows that BC + SAC fine-tuning typically requires thousands of trial episodes per task to reach the same performance our meta-trained WTL method achieves after one demonstration and a single trial episode. Appendix C.4 shows the BC + SAC training curves averaged across the different meta-test tasks.
5 DISCUSSION AND FUTURE WORK
We proposed a meta-learning algorithm that al- lows agent to quickly learn new behavior from a single demonstration followed by trial experience and associated (possibly sparse) re- wards. The demonstration allows the agent to infer the type of task to be performed, and the
METHOD
SUCCESS RATE
BC
.09
MIL
.30 + .02
WTL, 1 TRIAL (OURS)
.42 = .02
trials enable it to improve its performance by resolving ambiguities in new test time situa- tions. We presented experimental results where the agent meta-traine meta-trained on a broad distribution of tasks, after which it is able to quickly learn tasks with new held-out objects from just one demonstration and a trial. We showed that our approach outperforms prior meta-imitation ap- proaches in challenging experimental domains.
RL FINE-TUNING WITH SAC
BC + SAC, 1500 TRIALS
.11 .07
BC + SAC, 2000 TRIALS
.29 + 110
BC + SAC, 2500 TRIALS
.39 + .11
Table I:
Average success rates across meta-test tasks using state space ohservations. For BC SAC we pre- train with behavior cloning and use RI. to fine-tune a separate agent each meta-test task. The table shows BC SAC performance after 1500, 2000, and 2500 tri- als per task.
As illustrated in our qualitative failure analysis (Appendix D) one area of future improvement involves improving the informativeness of even failed trial trajectories generated by n!. In future work, hope to explore alternatives pos- terior sampling that produce more informative trials while maintaining sample and computational efficiency. We also plan to explore ways of extending our approach to be meta-trained on much broader range of tasks, testing the performance of the agent on completely new held-out tasks rather than held-out objects.
The Watch- Try-Learn (WTL.) approach enables a natural way for non-expert users to train agents perform new tasks: by demonstrating the task and then observing and critiquing the performance of the agent on the task if it initially fails. WTL achieves this through a unique combination of demonstrations and trials in the inner loop of meta-learning system, where the demonstration guides the exploration process for subsequent trials, and the use of trials allows the agent to learn new task objectives which may not have been seen during meta-training. We hope that this work paves the way towards more practical and general algorithms for meta-learning behavior.
ACKNOWLEDGEMENTS
We would like to thank Luke Metz and Archit Sharma for reviewing earlier draft of this paper, and Alex Irpan for valuable discussions. We would also like to thank Murtaza Dalal for finding and correcting an error in the "BC+SAC" experimental results from earlier version of this paper.
REFERENCES
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Optimality in Artificial and Biological Neural Networks, pp. 6-8, 1992.
Christopher M Bishop. Mixture density networks. 1994.

---

Published as a conference paper at ICLR 2020
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In IJCAI, 2015.
E Coumans and Y Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. GitHub repository, 2016.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:161 1.02779, 2016.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Neural Information Process- ing Systems (NIPS), 2017.
Chelsea Finn. Learning to Learn with Gradients. PhD thesis, UC Berkeley, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), 2017a.
2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abheel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning. Conference on Robot Learning (CoRL), 2017b.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning, pp. 2829- 2838, 2016.
Sergio Guadarrama, Anoop Korattikara, Osear Ramirez, Pablo Castro, Ethan Holly, Sam Fish- man, Ke Wang, Ekaterina Gonina, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. TF- Agents: A library for reinforcement learning in tensorflow, 2018. URL lhttps:// Comm/eensorfoow/aaee
[Online; accessed 30-November-20188.
Abhishek Gupta, Russell Mendonea, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-
reinforcement learning of structured exploration strategies. arXiv preprint arXiv:1802.0722 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J Lim. Multi- modal imitation learning from unstructured demonstrations using generative adversarial nets. In Neural Information Processing Systems (NIPS), 2017.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Amold, et al. Deep g-learning from demonstra- tions. AAAI, 2018.
Sepp Hochreiter, A Younger, and Peter Conwell. Learning to learn using gradient descent. Interna- fional Conference on Artificial Neural Networks (ICANN), 2001.
Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv:1802.044 2018.
Stephen James, Michael Bloesch, and Andrew J Davison. Task-embedded control networks for few-shot imitation learning. arXiv preprint arXiv:1810.022 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Jens Kober and Jan R Peters. Policy search for motor primitives in robotics. In Neural Information Processing Systems (NIPS), 2009.
10

---

Published as a conference paper at ICLR 2020
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 2013.
Petar Kormushev, Sylvain Calinon, and Darwin G Caldwell. Robot motor skill coordination with em-based reinforcement learning. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pp. 3232-3237. IEEE, 2010.
Hoang M Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III. Hierar- chical imitation and reinforcement learning. arXiv preprint 1803:00800000900 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research (JMLR), 2016.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. International Conference on Learning Representations (ICLR), 2018.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt in dynamic, real-world enviromments through meta- reinforcement learning. International Conference on Learning Representations (ICLR), 2019.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. International Conference on Robotics and Automation (ICRA), 2018.
Tom Le Paine, Sergio Gómez Colmenarejo, Ziyu Wang, Scott Reed, Yusuf Aytar, Tobias Pfaff, Matt W Hoffman, Gabriel Barth-Maron, Serkan Cabi, David Budden, et al. One-shot high-fidelity imitation: Training large-scale deep nets with rl. arXiv preprint 1rX1vv1810.05001,, 2018.
Jan Peters and Stefan Schaal. Policy gradient methods for robotics. In International Conference on Intelligent Robots and Systems (IROS), 2006.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems, 2018.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off off-policy meta-reinforcemment learning via probabilistic context variables. arXiv preprint 1903. 2019.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Oshand, Zheng Wen, al. A tutorial on thompson sampling. Foundations and Trends in Machine Learning, 11(1):1-96, 2018.
Steindór Sæmundsson, Katja Hofmann, and Mare Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint arXiv: 1803:0100 2018.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning. PhD thesis, Institut für Informatik, Technische Universität München, 1987.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
Bradly C Stadie, Ge Yang. Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reiiforeement learning. arXiv preprint 180iv:1803.0 01 / 18, 2018
Kaushik Subramanian, Charles L Isbell Jr, and Andrea L Thomaz. Exploration from demonstration for interactive reinforcement learning. In International Conference on Autonomous Agents & Multiagent Systems. 2016.
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining rein- forcement learning & imitation learning. International Conference on Learning Representations (ICLR), 2018.
11

---

Published as a conference paper at ICLR 2020
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning learn: Meta-critic networks for sample efficient learning. arXiv:1006.09529, 2017.
Matthew E Taylor, Halit Bener Suay, and Sonia Chernova. Integrating reinforcement learning with human demonstrations of varying ability. In International Conference on Autonomous Agents and Multiagemt Systems, 2011.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998.
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv:1611.0576. 2016.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning, Robotics: Science and Systems (RSS), 2018.
12

---

Published as a conference paper at ICLR 2020
A GRIPPER ENVIRONMENT DETAILS
A.1 SCENE SETUP
We created the gripper environment using the Bullet physics engine (Coumans & Bai, 2016). The agent controls a floating gripper that can move and rotate freely in space (6 DOFs), and 2 symmetric fingers that open and close together (1 DOF). For vision-based policies, the environment provides 300 x 220 RGB image observations and the 7-D gripper position vector at each timestep. For state-space policies it provides the 7-D gripper state and the 6-D poses (translation rotation) of all non-fixed objects in the scene. The virtual scene is set up with the same viewpoint and table as in Figure with random initial positions of the gripper, as well as two random kitchenware objects and a button panel that are placed the table. The button panel contains two press-able buttons of different colors. One of the two kitchenware objects is placed onto a random location on the left half of the table, and the other kitchenware object is placed onto random location the right half of the table. After the demonstration episode of each task, the kitchenware objects and button panel are repositioned: kitchenware object on the left half of the table moves random location on the right half of the table, and vice versa. Similarly, the two colored buttons in the button panel swap positions. Swapping the lateral positions of objects and buttons after the demonstration is crucial because otherwise, for example, the difference between the two types of pushing tasks would be meaningless.
A.2 TASK FAMILIES
Our gripper environment has four broad task families: button pressing, grasping, sliding, and pick and place. But tasks in each family may come in one of two types:
Button pressing I: the demonstrator presses the left (respectively, right) button, and the agent must press the left (respectively, right) button.
Button pressing 2: the demonstrator presses one of the two buttons, and the agent must press the button of the same color.
Grasping 1: the demonstrator grasps and lifts the left (respectively, right) object. The agent must grasp and lift the left (respectively, right) object.
Grasping 2: the demonstrator grasps and lifts one of the two objects. The agent must grasp and lift that same object.
Sliding 1: the demonstrator pushes the left object into the right object (respectively, the right object into the left object). The agent must push the left object into the right object (respectively, the right object into the left object).
Sliding 2: the demonstrator pushes one object (A) into the other (B). The agent must push object A into object B.
Pick and Place 1: the demonstrator picks up one of the objects and places it on the near (respectively, far) edge of the table. The agent must pick up the same object and place it on the near (respectively, far) edge of the table.
Pick and Place 2: the demonstrator picks up one of the objects and places it on the left (respectively, right) edge of the table. The agent must pick up the same object and place on the left (respectively, right) edge of the table.
A.3 EPISODES AND REWARDS
A gripper environment episode has a maximum length of 50 timesteps, seconds real time. At each timestep the environment returns a reward of +1 if the agent successfully completes the task, or otherwise. The episode ends immediately upon success, so only the terminal timestep can have nonzero reward. Computing success (and hence the reward) depends on the current task's family. For each task family, the environment determines success at the current timestep if:
Button pressing: the gripper is in contact with the correct button.
Grasping: the correct object's vertical position is certain threshold above the table.
Sliding: the two objects are in contact, and object A has moved more than object B.
Pick and Place: the correct object touches the correct edge of the table, and has cleared some vertical threshold at some prior point in the episode.
13

---

Published as a conference paper at ICLR 2020
Figure 7: Our reaching toy environment with target objects and 2 degrees of freedom (DOFs),
with per-task randomized dynamics.
B REACHER ENVIRONMENT DETAILS
Figure 7 depicts our reaching toy environment. The agent controls a 2 degree of freedom (DOF) arm and should reach one of the two randomly positioned objects. Crucially, the agent's dy- namics are unknown and randomized at the start of each task: each of the two joints may have reversed orientation with 50% probability, and the agent's joint orientations may not match that of the task demonstration d; This simulates a domain adaptive setting, for example the discrepancy in dynamics when robot imitates from human video demonstration. Since simply observing the demonstration is not sufficient to identify the task dynamics, the agent must use its own trials to identify the task dynamics and successfully reach the target object. Following (Brockman et al.) 2016), the environment returns reward at each timestep which penalizes the reacher's s distance to the target object and the magnitude of its control inputs at that timestep. Precisely, let é + g map the agent's state to the reacher tip position, and 2; I be the true target object's position. Then:
ri(s, a) = (1)(()) - zall - |||a|
(5)
C EXPERIMENTAL DETAILS
We trained all policies using the ADAM optimizer (Kingma & Ba 2015). on varying numbers of Nvidia Tesla P100 GPUs Whenever there more than GPU. we use synchronized gradient updates and the batch size refers to the batch size of each individual GPU worker.
C.I REACHER ENVIRONMENT MODELS
For the reacher toy problem, every neural network in both the WTL policies and the basclines policies uses the same architecture: two hidden layers of 100 neurons each, with ReLU activations on each layer except the output layer, which has no activation.
Rather than mixture density networks, our BC and MIL policies are deterministic policies trained by standard mean squared етror (MSE). The WTL Phase II policy nh is also deterministic and trained by MSE, while the Phase I policy n! policy is stochastic and samples actions from a Gaussian dis- tribution with learned mean and diagonal covariance (equivalently, it is simplification of the MDN to single component). We found that a deterministic policy works best for maximizing the MII. baseline's average return, while stochastic WTL trial policy achieves lower returns itself but fa- cilitates easier learning in Phase II. Embedding architectures are also simplified: the demonstration "embedding" is simply the state of the last demonstration trajectory timestep. while the trial em- bedding architecture replaces the temporal convolution and flatten operations with a simple average over per- timestep trial embeddings.
We trained all policies for 50000 steps using a batch size of 100 tasks and a .001 learning rate, using single GPU operating 25 gradient steps per second.
14

---

Published as a conference paper at ICLR 2020
130 108
Creas Stubke
Faili
Faiton
Figa
Comee Poes
CoroEriece Oripee Hord Fow
Figure 8: Our rision-based Phase architecture. This identical the architecture in Figure there are no trial embeddings or trial rewards concatenate in conditioning network.
except that
Table 2: Best Model Hyperparameters for Gripper Environment Experiments
METHOD LEARNING RATE
MDN COMPONENTS
VISION MODELS
WTL
9.693 10
20
MIL
9.397 x 10
20
BC
1.093 x 10-3
20
STATE-SPACE MODELS
WTL
2.138 x 10 3
20
MIL
2.659 10
20
BC
2.138 x 10
20
C.2 GRIPPER ENVIRONMENT MODELS
Algorithm 3 Demonstration and Trial Subsampling
1: Input: Demonstration = {(st,, az)) (or WLOG, trial r), for T > 2
2: Input: Output length = 40
3: Initialize d {(1, a1), (ST, aT)}
4: for step = 1, 1,. U - 2 do
5: Sample uniformly at random from (2,... , T 1}, with replacement.
6: d + d J as),
7: end for
8: return sort(d)
The gripper environment policies illustrated in Figure 3 and Figure S produce context embeddings from demonstrations and (for Phase II) trial trajectories. The demonstrations and trials variable length sequential data, since each gripper environment episode will terminate early if successful. To avoid dealing with variable sized inputs, we subsample 40 timesteps with replacement from any demonstration or trial trajectory to provide as input to our policies. We describe the subsampling process in Alg. 3 Since our gripper environment rewards are sparse and only nonzero on the final timestep, the subsampling process always selects the first and last timesteps. It selects the remaining 38 timesteps uniformly at random from the noninitial and nonterminal timesteps of the original demonstration or trial. It returns the 40 selected timesteps in sorted order for temporal consistency.
For each state-span space and vision method we ran a simple hyperparameter sweep. We tried K = 10 and K 20 MDN mixture components, and for each K sampled 3 learning rates log-unifomlyy from the range [.001,.0] .01] (state-space) or [.0001, .01] (vision), leading to hyperparameter exper- iments per method. We selected the best hyperparameters by average success rate the meta-
15

---

Published as a conference paper at ICLR 2020
We trained all gripper models with 11 GPU workers and batch size of tasks for 500000 (state-space) and 60000 (vision) steps. Training the WTL trial and re-trial policies on the vision-base gripper environment takes 14 hours each.
validation tasks, see Table 2
C.3 ALGORITHMIC COMPLEXITY OF WTL
WTL's trial and re-trial policies trained off- policyy via imitation learning, only two phases of data collection are required: the set of demonstrations for training the trial policy, and the set of trial policy rollouts for training the re-trial policy. The time and sample complexity of data collection is linear in the number of tasks, for which only one demo and one trial is required per task. Furthermore, because the optimization of trial and re-trial policies de-coupled into separate learning stages, the sample complexity is fixed with respect to hyperparameter sweeps. A fixed dataset of demos is used to obtain good trial policy, and fixed dataset of trials (from the trial policy) is used to obtain a good re-trial policy.
The forward-backward pass of the vision network the dominant factor in the computational cost of training the dision-based Phase II architecture. Thus, the time complexity for single SGD update to WTL is (T),, where T the number of sub-sampled frames used to compute the demo and trial visual features for forming the contextual embedding. However, in practice the model size and number of sub-sampled frames = 40 are small enough that computing embeddings for all frames in demos and trials can be vectorized efficiently GPUs.
C.4 BC + SAC BASELINE TRAINING CURVE
Average meta-test success rate vs trial experience.
0.5
0.4
Rate
0.3
Method WTL (Ours)
BCC
0.2
0.1
0.0
10a
10
102
10a
10c
10a
Environment Step (Log Scale)
Figure 9: For "BC+SAC", we pre-train agents with behavior cloning and use RI. to fine-tune each meta- test task. By comparison, WTL uses demo and single trial episode per task (<. 50 environment steps). X-axis (log-scale) shows environment experience steps collected per task. Error bars indicate 95% confidence intervals.
D FAILURE ANALYSIS
We performed a qualitative failure analysis of our vision-based WTL method on meta-test tasks by manually inspecting videos of Phase I and Phase II policy behavior. Table shows matrix of different outcomes, where each episode is classified into one of three categories: 1) successfully completing the task, 2) performing the task but on the wrong object location, 3) moves toward the correct object but misses (often comes close but gets stuck). One common failure mode is when the agent reaches towards but misses the correct object in Phase I, which does not provide the Phase policy with enough information to disambiguate the correct task.
16

---

Published as a conference paper at ICLR 2020
Table 3: WTL outcome matrix in the vision-based gripper environment. This table shows frequen-
cies outcomes after Phase (rows) and after Phase (columns) over 100 meta-test tasks.
Phase II
Missed Object
Wrong Object
Success
Phase
Missed Object
23
12
15
Wrong Object
5
11
8
Success
20
17


Reviews:
[{"review_id": "rJe17q5Z9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a meta-learning approach that can learn from demonstrations and subsequent reinforcement learning trials. The approach works in two phases. First, multiple demonstrations are collected for each task and a meta-learning policy is obtained by embedding the demonstrations of a single task in a context vector that is given to the actor policy.  Subsequently, the meta-policy is used to collect trajectories which are also evaluated with a (sparse) reward. These trajectories, along with the demonstrations, are again used by a second meta-learning policy (with a concatenated context from demonstrations and trajectories) to obtain the final policy. The algorithm is tested on a simple point reaching task as well as on a more complex 3d physics simulation that contains pick-and-place tasks, pushing tasks and button pressing.\n\nThe approach seems really interesting and I think combining demonstrations with reinforcement learning for meta learning is a very promising approach. This is also underlined by the experimental results presented in the paper. The only concern I have is the presentation of the paper. The algorithmic description is very short and many things are left unclear. I would also like to see a bit more ablation studies in the paper. More comments below:\n\n- Just from the text it is not clear how the meta-learning actually works and you have to study the figures in detail to understand that a context is extracted from the demonstrations as well as the RL trajectories. The context should be better introduced (even though this has been shown already in different papers, I think it is a good strategy to write the  \nin a self-contained way).\n- It is unclear to me how the reward signal is used to learn the meta-policy is phase 2. I understand that it is concatenated to the embedding before forming the context (again this is not really described in text but only in the figure), but it is quite unclear to me how the reward is used as optimality criteria. How do you reinforce good trajectories and decrease the influence of poor trajectories? This is maybe one of the most important parts of the paper and needs to be described in much more detail.\n- The reward functions are not properly explained in the paper, even not for the toy task (reaching). This should at least be done in the appendix.\n- I would like to see more ablation studies. For example, it is mentioned that 40 observations are used from the demonstrations and the trajectories. How is this number picked and how does it influence the performance? Also, typically, how long is a single trajectory? It is also unclear to me, if we take 40 random observations and the reward signal is sparse, wouldn't it be quite likely that the observations do not contain any reward values?\n- How does K influence the performance of the algorithm?\n\n \n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syxmdc1CKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on meta learning from both demonstrations and rewards. The proposed method has certain advantages over previous methods:\n(1) In comparison to meta-imitation, it enables the agent to effectively and efficiently improve itself autonomously beyond the demonstration data. \n(2) In comparison to meta-reinforcement learning, it can scale to substantially broader distributions of tasks, as the demonstration reduces the burden of exploration.\n\nI am a bit confused by the writings and don't clearly understand the algorithm.\n1. In eq. 4, should \\theta be \\phi?\n\n2. What does \\pi_\\theta(a_t|s_t, {d_i,k}) mean? I'd like to see an example formulation of the policy. When the parameter \\theta is well trained, does the policy still take demonstrations {d_i,k} as inputs? If yes, why are demonstrations needed? \n\n3. In meta-testing (Algo. 2), will \\theta and \\phi be updated? Currently it looks like that the two parameters are fixed in meta testing. If so, this is a bit strange. Why not update the policies after sampling demonstrations (step 4) and collecting trials (step 5)?\n\n4. \"The state space S, reward ri, and dynamics Pi may vary across tasks.\" I think all the tasks should share the same input space; otherwise, the meta-trained two policies cannot be applied to a test task with different input space.  For example, if the states are 100x100 images in meta training, how to apply the policies to a meta-testing task with 100x200 images as states? \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxKT8BYtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach for combining meta-imitation learning and learning from trial-and-error with sparse reward feedback. The paper is well-written and the experiments are convincing. I found the idea of having separate networks for the two phases of the algorithm (instead of recollecting on-policy trial trajectories) interesting and possibly applicable in other similar settings. \n\nThe paper has a comprehensive analysis of the advantages of the method over other reasonable baselines which do not have the trial-and-error element. However, in terms of the limitation of the method, the paper only has a small failure analysis section in the appendix; adding a more detailed discussion on the limitations can improve the impact of the paper further. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a meta-learning approach that learns from demonstrations and subsequent RL tasks.
The reviewers found this work interesting and promising. There have been some concerns regarding the clarity of presentation, which seems to be addressed in the revised version. Therefore, I recommend acceptance for this paper.
Decision: Accept (Poster)