PERIL: Probabilistic Embeddings for hybrid Meta-Reinforcement and Imitation Learning
Authors: Alvaro Prat
Edward Johns
Venue:  (2021)
Keywords: Meta-learning
Imitation Learning
Reinforcement Learning

Abstract: Imitation learning is a natural way for a human to describe a task to an agent, and it can be combined with reinforcement learning to enable the agent to solve that task through exploration. However, traditional methods which combine imitation learning and reinforcement learning require a very large amount of interaction data to learn each new task, even when bootstrapping from a demonstration. One solution to this is to use meta reinforcement learning (meta-RL) to enable an agent to quickly adapt to new tasks at test time. In this work, we introduce a new method to combine imitation learning with meta reinforcement learning, Probabilistic Embeddings for hybrid meta-Reinforcement and Imitation Learning (PERIL). Dual inference strategies allow PERIL to precondition exploration policies on demonstrations, which greatly improves adaptation rates in unseen tasks. In contrast to pure imitation learning, our approach is capable of exploring beyond the demonstration, making it robust to task alterations and uncertainties. By exploiting the flexibility of meta-RL, we show how PERIL is capable of interpolating from within previously learnt dynamics to adapt to unseen tasks, as well as unseen task families, within a set of meta-RL benchmarks under sparse rewards.

Under review as a conference paper at ICLR 2021
PERIL: PROBABILISTIC EMBEDDINGS FOR HYBRID
META- REINFORCEMENT AND IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Imitation learning is a natural way for a human to describe a task to an agent, and it can be combined with reinforcement learning to enable the agent to solve that task through exploration. However, traditional methods which combine imi- tation learning and reinforcement learning require a very large amount of interac- tion data to learn each new task, even when bootstrapping from a demonstration. One solution to this is to use meta reinforcement learning (meta-RL) to enable agent quickly adapt to new tasks at test time. In this work, we introduce a new method to combine imitation learning with meta reinforcement learning, Probabilistic Embeddings for hybrid Reinfrreinfr and Imitation Learning (PERIL). Dual inference strategies allow PERIL to precondition exploration poli- cies demonstrations, which greatly improves adaptation rates in unseen tasks. In contrast to pure imitation learning, our approach is capable of exploring be- yond the demonstration, making robust to task alterations and uncertainties. By exploiting the flexibility of meta-RL, we show how PERIL is capable of interpo- lating from within previously learnt dynamics to adapt to unseen tasks, as well as unseen task families, within set of meta-RL benchmarks under sparse rewards.
INTRODUCTION
Reinforcement Learning (RL) and Imitation Learning (IL) are two popular approaches for teaching an agent, such as a robot, a new task. However, in their standard form. both require very large amount of data to learn: exploration in the case of RI., and demonstrations in the case of II. In recent years, meta-RI. and meta-ll. have emerged as promising solutions to this, by leveraging a meta-training dataset of tasks to learn representations which can quickly adapt to this new data. However, both these methods have their own limitations. Meta-RL typically requires hand-crafted, shaped reward functions to describe each new task, which is tedious and not practical for non- experts. A more natural way to describe a task is to provide demonstrations, as with meta-IL. But after adaptation, these methods cannot continue improve the policy in the way that RL methods can, and are restricted by the similarity between the new task and the meta-training dataset. A third limitation, which both methods suffer from, is that defining of the environment for efficient learning (as opposed to learning directly from high-ding -igeediinnsiinall images), requires hand-crafting this representation. As with rewards, this not practical for non- experts, but more importantly, it does not allow generalisation across different task families, since each task family would require its own unique representation and dimensionality.
low-dimensional representation
In this work, we propose a new method, PERIL, which addresses all three of these limitations, in a hybrid framework that combines the merits of both RL and IL. Our method allows for tasks to be defined using demonstrations only as with II., but upon adaptation to the demonstrations, it also allows for continual policy improvement through further exploration of the task. Furthermore, we define the state representation using only the agent's 's internal sensors, such as position encoders of robot arm, and through interaction we implicitly recover the state of external environment, such as the poses of objects which the robot is interacting with. Overall, this framework allows for learning of new tasks without requiring any expert knowledge in the human teacher.
PERIL operates by implicitly representing task by latent vector, which is predicted during learn ing of a new task, through two means. First, during meta-training we encourage high mutual infor- mation between the demonstration data and the latent space, which during testing forms a prior for

---

Under review as a conference paper at ICLR 2021
Learn new task
Set of demonstrations of unseen
Learn
Figure 1: Overview of our proposed method. We obtain set of demonstrations of an unseen task and we adapt to through efficient demoontrationn-ondiiione exploration.
the latent space and represents the agent's task belief from demonstrations alone. Second, we allow further exploration to continually update the latent space, by conditioning the robot's s states and actions during exploration of this new task. We model the latent space via an encoder, from which posterior sampling can be done to encode the agent's 's current task belief. In essence, the encoder aims to learn an embedding which can simultaneously (i) infer the task intent, and (ii) output a policy which can solve the inferred task. During meta-training PERIL is optimised end-to-end, by simul- taneously learning both a policy and the embedding function upon which the policy is conditioned.
In our experiments find PERIL achieves exceptional adaptation rates and is capable of exploiting demonstrations to efficiently explore unseen tasks. Through structured exploration, PERIL outper- forms other Meta-RL and Meta-IL baselines and is capable of zero-shot learning. We show how our method is capable of multi-family meta-learning as well as out-of-family meta-learning by clus- tering distinct meta-trained latent space representations. As extension, we also show how to use privileged information during training to create an auxiliary loss for training the embedding function, which helps to form a stronger relationship between the latent space and the true under- lying state which defines the task. Supplementary videos are available at our anonymous webpage https:// //sttes///tts.ee google. com cww//ieewwprrll-lll1
RELATED WORK
Meta-RL was conceptualised as an RNN-based task. Developed by Wang (2016) and Duan et al., the authors use RNNs to feed history of transitions into the model such that the policy can internalise the dynamies. On another line, Finn et al. (2017) developed learning-to-leam strategy, model agnostic meta learning (MAML), which meta-learns an initialisation that adapts the parameters of the policy network and fine tunes it during meta-testing. Although promising results in simple goal- finding tasks, MAML-based methods fail to produce efficient stochastic exploration policies and adapt to complex tasks (Gupta et al., 2018). Meta-learning robust exploration strategies is key in order improve sample efficiency and allow fast adaptation test time. In light of this, context- based RL was developed with the aim of reducing the uncertainty of newly explored tasks. These map transitions 7 collected from an unseen task into latent space z via an encoder (=(=))), such that the conditioned policy (T(T( can efficiently solve said task (Rakelly et al.; Wang & Zhou, 2020). underlining benefit of decoupling task encoding from the policy that disentangles task inference from reward maximisation, whilst gradient-based based and RNN- do this internally. An important remark regarding training conditions of the discussed meta-RI. methods is that they are typically meta-trained using dense reward functions (Rakelly et al.; Wang, 2016; Wang & Zhou, 2020). These dense reward functions provide information-rin the unseen task. Considering that the ultimate goal is to allow agents to solve new tasks in the real world, adaptation during test time must be robust to sparse reward feedback (Schoettler et al., 2020).
meta-RL RL policies
contexts of
In the context of RL, incorporating demonstrations has proven successful in aiding exploration strategies, stabilising learning and increasing sample efficiency (Vecerik et al., 2017). Learning expressive policies from set of demonstrations requires vast amount of expert trajectories (Ross et al.), particularly in high dimensional state-action spaces (Rajeswaran et al.). In contrast, Meta-ll. can be implemented in meta-RL by conditioning the agent with expert trajectories. On the an- other hand, Zhou et al. propose MAML-basee meta-IL approach which averages objective across demonstrations. The latter learns adapt test-time by receiving binary rewards. A similar strat- egy was also developed by and Mendonca et al.. The caveats of these approaches remain that of traditional IL: (i) Imitating expert trajectories hinders the policy from doing better than the demon- strations; (ii) Cloning behaviours reduces flexibility and generalisation capacity.
2

---

Under review as a conference paper at ICLR 2021
3 HYBRID META-REINFORCEMENT AND IMITATION LEARNING
3.1 PROBLEM STATEMENT
We assume access set of tasks T p(T), with each task represented as partially observable Markov decision process (POMDP). Observations are incomplete and only include measurements of the agent's ss internal state. For example, peg-in-hole task would include the robot's (and therefore the peg's () pose in the observation, but not the hole's pose. The full state & contains both the robot's internal state, and the state of the external environment. Our method involves an agent inferring 2, a descriptor of task which provides the information required to solve that task, such as a represen- tation of s. Each task then defined as T = {p(og) [=), p(ol, 2/ lo, a, 2). r(o, al2)), with an unknown initial observation distribution p(Oulz), 2), transition distribution p(or, 2//0, 2), and reward function r(o, a|=), where a is the action taken by the agent. By leveraging task beliefs, PERIL exploits en- riched observational spaces (s ~ {o n z)) with the objective of closing said POMDP into stable MDP form. With the aim of supporting continuous task inference through interaction, we construct probabilistic embedding conditioned on set of recently collected transitions, referred as the context c. We define a contexton cf = (ot, a, Tt, 0t+1) as a transition collected on task T at time-step t such that the context ofr (what we call c) denotes the set of accumulated contextons. During meta-testing, the agent then attempts to find the true posterior p(z[c) over the task belief 2 (z)cc), by conditioning on c.
Demonstrations significantly reduce the search space in exploration whilst providing a natural means of communicating the task. Thus, access expert trajectories provides information-rich context which can be exploited to pre-condition the policy via a prior over z. Subsequently, online adaptation through RL can further disambiguate 2. In our approach, we leverage dual meta-learning objectives to perform hybrid adaptation to new tasks, with conditioning on both demonstration contexts and exploratory contexts, where we define a demonstration as a trajectory of T observations and actions d = {(o,, (/) and an adaptation episode Tg as trajectory of N steps = {((o, az, Tt, 2,) ) J i:
Primal Inference: The agent observes a set of 1,..,, K demonstrations from an expert set Deemo == {dx) to form a context prior Cdemo {dæ 1k
Exploratory Adaptation: The agent explores with initialised context e <- Cdemo and adapts with sampled trajectories I such that at time-step t, hybrid context is formed c < Cdemo O Toat-
Primal task inference aims to provide long-term inference on a task belief: meta-trained inference from previously solved tasks. Using posterior sampling, exploratory adaptation provides a proxy for short-term memory: as the context updated with recent transitions, structured exploration can adjust primal inference by reasoning about the uncertainty of 2, where {Cdemo D( ..... Te T)
3.2 PROBABILISTIC EMBEDDINGS FOR META-RL.
As we do not have access to the posterior p(z[c), we use variational inference methods to produce an approximation (zlc). Through generative processes, we sample z ~ (2(c))) and optimise & by maximising a meta-objective conditioned on z. Variational Encoders (VEs) are capable of producing latent distributions which optimise an arbitrary task-dependent objective G(T).). We average the expectancy over set of tasks from p(T) to formulate a meta-objective which rewards fast adaptation to unseen tasks (Eq. 1), where 8 controls the constraint for mutual information between the inferred variable and context This information bottleneck filters down redundant information and compresses the z into generalisable form. We set the prior p(z) to a unit Gaussian.
o² = arg max ETEN(T) E (9(T))) + 3DKL (go [=)(((])
Traditional meta-RL methods leverage RNN-based inference systems to produce latent features from a history of recently collected transitions. The problem with this approach is that learning from en- tire trajectories leads to massive variances which hinder the learning process (Duan et al.: Humplik et al., 2019). In theory recurrence is not strictly required, since any MDP is defined by a distribu- tion of sequentially invariant transitions. In light of this, we exploit task encoders as suggested by Rakelly et al., where q(=(c) is modelled as a product of Gaussian Factors. Each factor th (2(c,) parameterised by d, is independently computed: dh (z(c)).)) NN ( (cx), I (c.)))
3

---

Under review as a conference paper at ICLR 2021
We build on top of probabilistic meta-RL methods derived by Rakelly et al.. Decoupling task in- ference from task completion allows efficient off-policy objectives to be optimised. Moreover, con- ditioning a policy with diverse task descriptors aims to generalise behaviours amongst different dynamics and rewards. This is generally not the case when employing MAMLL-assd or RNN-based meta-training kernels to "learn how to learn" new tasks, since policy gradients modify the policy pa- rameters online which, in turn, can result in highly unstable and inefficient adaptation to new tasks, requiring up hundreds of sample trajectories in unseen environments.
To that end, we employ meta-adapted maximum entropy RL, where a proxy for the policy r(als) is defined by including the task belief as part of the observational space Ta(alo, 2). resulting in task- conditioned critic Liritic and actor Lu losses for the the Q-function Qe and the policy Te()) respectively (see A. more details). As VEs are a form generative processes, ((T)-) can be defined with the aim of recovering transition dynamics or reconstructing state spaces (Humplik et al., 2019). Instead, we optimise go (2(c)
model-free manner as proposed in Rakelly al.. Here, parameters & can be optimised to maximise expected discounted rewards under a policy (-(2)2) through reconstruction of Qo(:,2). z). We fold Crritic as a proxy for the task-dependent goal (T]
((T + LT (z)
(2)
3.3 CONDITIONING ON DEMONSTRATIONS
By leveraging meta-learning, we suggest that the agent can learn to exploit the overlapping statistical information from within heterogeneous demonstrations belonging to different tasks, with the aim of inferring task embeddings = which can help improve exploration during test-time adaptation. In order to link demonstrations into the probabilistic meta-RL. framework, we formulate an objective based on mutual information [(2; between the demonstration trajectories and the latent space distribution p(z) (Eq. 3). This objective is similar to that used in Yu et al. with the exception that we adapt this to meta-imitation learning and off-policy RL instead of recovering reward functions.
(2; (===p()), ((2)) log p(2)
As direct access to the posterior p(z/7) is not available, [(2; r) is intractable. However, in order to adjust for this, we leverage a variational approximation to p(²(T) using our task belief encoder (2(T). We assume access to distribution of expert demonstrations Pap (1l2). Additionally, include further desiderata over the mutual information objective based on distributional matching:
Learning from demonstrations. Matching generated trajectories those sampled from expert [Dx trajectories: ming [ (P (T|>) [T]=))] This secondary objective can be considered as trying match the distribution of trajectories generated by the agent's policy conditioned on 2, to those from the expert policy (similar to BC). Since they share the same marginal distribution p(z), matching these distributions also encourages matching of the conditionals pwg (=/T) and p(z]7)
Encourage (=(=))) to approx- imate the true posterior p(²[T) such that, given a new demonstration, the encoder properly infers the task. This objective acts regulator as it reduces over-fittie wver-fitting by penalising variance: optimal distributional mismatch is reached when the encoder produces constant information-less represen- tations of The distributional matching objective can therefore be defined as in (4).
Linking variational posterior. ming (r) (p(==)(())))))]>]]>>
min - (2; + Epta) ( (T|2) [(T[2))] + Ep(r) (xcr:
(4)
After several manipulation steps (see section A.1 for more details), we deconstruct this objective into a differentiable form and define two separate loss components C7. (Eq. 5) and LT, (Eq. 6). where n, is the behavioural policy which collected the transitions (truncated importance sampling).
CTe EE (11r)) (elllooe(TTl [T(T)=
(5)
log mo( log no (To (T, 1)
CT -E (a)))
min
log (o)
(6)
As CT conditions the encoder, we aggregate it to the task dependent goal G(T]2) (7).
io
[TT]=) forstte (l))o(=)
4

---

Under review as a conference paper at ICLR 2021
3.4 AUXILIARY SYSTEMS
An inherent problem is that the agent generates predictions Qo(o, a, 2) based on unsupervised esti- mates (z)e)). Whilst this is theoretically generalisable to any MDP, it brings large variances and instabilities during training because the agent must simultaneously (i) distinguish one task from another, and (ii) use the task belief's to solve that task. A plausible solution to mitigate these instabil- ities is to exploit privileged information (a brief task descriptor such as the position and orientation of a door) during training, allowing the encoder (=)c²) to produce task beliefs 2 which provide succinct descriptors of T. Although this information is not available during testing, it could be made available, for example, if training is performed in a full-state simulation, or in the real-world but with extra environment instrumentation. In theory, a perfect task descriptor is that which provides suffi- cient information to close the POMDP. We model ground truth task descriptor for T as vector 6 E Rº where v is the dimension of the task descriptor. During training we wish to condition the encoder to produce latent spaces ~ (o ce)) which, when mapped into the dimensions of b, can produce approximations of b. We use a VAE dx (b]2) parameterised by Ả as auxiliary module, which produces vectors buz by E RR" and optimise it via a log-likelihood maximisation objective (8).
CT
((..))-T,anaatee log (x = 8/=)]
auz
Inference b ~ (b()) from the trajectories in c" is decoupled from RL as gradients do not pass through the actor or the critic. This stabilises the encoder by having a fixed supervised target, and allows off- policy training. Leveraging auxiliary objectives for meta-RL was coined by Humplik et al. (2019), where the authors condition task inference solely on Liuz to train RNN-based poli- cies. However, in PERIL we use simple descriptors such that these solely condition the encoder to converge towards suggested direction. Thus, we extend (T[]2) with the auxiliary loss to produce the ultimate meta-objective (9). Details of the final computational graph is presented in Figure 8. By keeping the critic loss C'ritic in (T]))), we give freedom for the encoder to perform unsupervised learning so that it structure representations of which represent different dynamics and rewards: we solely base task inference on CT a task such as screwing a bolt would output the same task belief as task of inserting peg, even though the tasks are fundamentally different.
(TT + CT (2) + CT CZ] + +TTu (2)
3.5 IMPLEMENTATION
Meta-training requires sampling from a of transitions (ot, at, Ft; O2+ Yr= corresponding dis- tribution of tasks T E p(T) and optimising the meta-objective. We use task-specific replay buffers BT to store collected transitions from which we can randomly sample during training. Following primal inference objective outlined in section 3.1, task inference is initially conditioned via con- textons retrieved from expert demonstrations. Consequently, we store a set of k demonstrations generated by an expert policy on fask-conditioned demonstration buffers DT. This expert policy may consist of human user or, in our case, pre-trained SAC agent with access to the full state of the environment. Meta-training algorithms are presented in Algorithms I and its process is illus- trated in Figure 2. Further details of the meta-training and meta-testing processes in section A.2. We augment DT with imperfect trajectories which lead to demo-like (and increased) sum of rewards. This allows PERIL to: (i) be robust to imperfect demonstrations and surpass performance from the expert; (ii) combat the overfitting nature of BC from a narrow set of k demonstrations. The agent must also exploit posterior sampling to update its task belief in order to accomplish continuous adap- tation objectives. To this end,
exploratory data to update the task belief during adaptation and set additional task-dependent encoder buffers X'T which only stores recently collected contextons. This mitigates distributional shift between off-policy training and on-policy adaptation.
4 EXPERIMENTS
We leverage probabilistic meta-RL methods to find values for = which reconstruct unobserved rep- resentations of a new task. To that end, we focus on validating PERIL in environments where the agent must explore in order understand the dynamics, whilst also utilising sparse rewards to identify the task intent. Furthermore, neural networks, are particularly susceptible to abrupt discon- tinuities. Thus, it is of particular interest to leverage meta-training in situations as such. We devise
5

---

Under review as a conference paper at ICLR 2021
Algorithm PERIL: Meta-training
Require: Batch of TT training tasks (Ti,.=,,... Ts p(T)
1: Initialise TT, Qe. 9p. by and replay BT encoder X² and demo buffers DJ for T e (Ti)e
2: while not done do
3: for each T; (I,..,; do
Collect data
4: Initialise context from demonstration (TE) TE ~ Dj and clear encoder buffer
5: for = I...,K do
6:
Sample from context 2 qo(2|c")
7:
Roll out policy T0(T(2). Add to B and X, Add D, if ER[T] ER(TE] Update context C¹ t (c' U Uz') }. ,, 24
8:
9:
for - I,.,,,,, do
Training for Nr steps
10:
for each E (l,,,,T) do do
Sample buffers x² X, TE D Bt to form hybrid context e {TE Uw' Sample ~ (0(2)e') and compute losses Ciis Sbe Lactor Ceritic: D Laux
12:
13:
Update parameters pn. pq, Φ, via gradient descent
Lgu Linto
B
(.
(x(cg)
(p(olo.2)
Loc Lactor
D
TER
Qa(0,
Leritic
d, (blz)
Laux
X7
Figure 2: Meta-training framework. Environment contains set of tasks T ~ p(T), each with its own ground truth task descriptor b. Samples generated by the policy overwrite the encoder buffer X for the current task T. All samples are added to the replay buffer BT. ① concatenates context from the demonstration and X' to produce a hybrid context cg. Red dotted line copies collected transitions into the demonstration buffer if these result in higher rewards than the demonstrations.
tasks which involve contact-rich interactions and boundaries, factors which are considered to require higher level of perception of the environment (Zhu et al., 2020). More details A.3.
Figure 3: Snapshots of the different task families used. We present two (top and bottom) distinet tasks for each task family: Reach2D, Stick2D, Peg2D, Key2D, Reach3D (left to right). A robotic agent must navigate in order to find the the effector. In Key2D the agent must also rotate the interactive (blue) handle to an arbitrary angle.
goal. In Stick2I and Peg2D the agent must insert
The different task families tested in this study are illustrated in Figure Within a task family, one single task is defined by the configuration of the environment, such as the pose of the object or goal. When leveraging privileged information, this pose is used as the task descriptor. For each task, demonstrations are provided from different robot initial poses, and the agent is then required to perform that task, from any new robot initial pose. meta-RL and IL baselines. To ensure faimesss in all tests, we used the same number of parame- ters during training. Specifically, MetalL (Yu et al.), extends PEARL by conditioning exploration on demonstrations. We also use Noisy-BC with additional noise (20% of p(ool-)). We also consider the case where privileged information is unavailable, thus compare these baselines to both PERIL and PERIL-A In PERIL we omit Laur-
compare our proposed methods to other
baseline where the agent clones demonstration

---

Under review as a conference paper at ICLR 2021
4.1 PERFORMANCE
We evaluate the performance of each method over 5 task families (Figure 4) using k 3 demon- strations per task. Since meta-learning requires trajectory-based adaptation, we record the averaged trajectory return after policy roll-outs. 'The results indicate that, in contrast other baselines, meta-learning without demonstrations through sparse reward feedback is particularly ineffective (PEARL). On the other hand, PERIL-based methods significantly outperform MetalL and Noisy- BC, especially in tasks from Peg2D or Reach2D/3D where the agent must explore efficiently. It is in these cases where the auxiliary module contributes greatest. On the other hand, in task families such as Stick2D and Key2D, few contact-rich interactions with the environment can quickly help decipher the task. this case, goal-oriented auxiliary targets are not as helpful.
PegoO
S1000
-16
15 125 120 120
Figure 4: Test-task performance vs. number of collected transitions during meta-training. By ex- ploiting hybrid inference, PERIL consistently achieves better performance with respect to other baselines. Auxiliary systems are particularly useful in complex tasks which require efficient explo- ration. All results are 3-point averaged.
4.2 INCREASING GENERALISATION
Most meta-RL research involves adaptation to tasks within a single task family (Rakelly et al.; Duan et al.; Yu et al.). But in practice, seek agents capable of interpolating from diverse set of dynamics. To test the representational power behind PERIL we evaluate training and adaptation per- formance in multi-task family settings (Figure 5). Here, train single agent along each task from from within the 2D task families. TSNE plots of the latent space distribution created during meta- testing reveals PERIL-based methods produce structured task beliefs during hybrid inference of new tasks. Moreover, this verifies PERIL'S long-term memory contribution. Last, it showcases PERIL performs structured exploration at the multi-task level, allowing the agent to efficiently switch from one macro policy to another and robustly inherit a diverse set of behaviours. Despite the effec- tiveness of privileged information complex exploration tasks in single task family adaptation, auxiliary systems do not generally contribute much during meta- tatt-mmlti-tan meita-mutt--takking. We believe that task inference substantially more convoluted in the multi-task family case as discerning dynam- ics and intent requires
complex embeddings in the latent space than the position of the goal. We also test generalisation by adapting PERIL out-of-task distributions (Figure 6), where PERIL interpolates from within meta-trained latent space representations to adapt to unseen dynamics.
Metrics during adaptation of unseen tasks reveal PERIL substantially more efficient in exploration with respect to other baselines, and is capable of zero-shol shot learning (Table 1). where the context from the demonstration is sufficient to condition the agent to adapt to an unseen task. Note we record the K-shot the average kth adaptation trajectory until successful completion of the new task.

---

Under review as a conference paper at ICLR 2021
Pugit
-32
240
-5
Deland
D005
Figure 5: Test-task performance number of collected transitions during meta-training (left and middle). PERIL-based methods are capable of generalising within task families. Latent task dis- tributions p(z) during adaptation (right) demonstrate MetalL (top) is incapable of clustering tasks within different task families. In contrast, PERIL (bottom) can discriminate them.
Table 1: Mean adaptation rate (K-shot) along all task families. PERIL. baselines demon- strate superior adaptation to unseen tasks both in adaptation speed and performance and can endure zero-shot learning.
Peach2D
Stek20
Peggo
Model
K-Min
K-Mean
K-Max
PEARL
7.2
0C
MetalL.
1.6
5.1
10.2
PERIL
1.0
2.7
Figure 6: TSNE plot of collected on adaptation.
PERIL-A
0
0.9
3.1
Stick2D is held out of meta-training and PERIL interpolates dynamics from Key2D and Stick2D.
Noisy-BC
1.8
8.4
12.3
4.3 ALTERATIONS ON K
We study the inherent meta-Il. dependency over the number of per-task expert demonstrations We find that increasing the number of demonstrations per task slightly improves sample efficiency but does not alter the asymptotic capacity of PERIL (Figure 7). Improvement is small since, by augmenting the demonstrations online, PERIL capable of self-inducing ad- ditional demos which improves sample efficiency.
Figure 7: Effect on kt
CONCLUSION
We have introduced PERIL, new method for meta imitation and meta reinforcement learning which builds a representation of new task by conditioning both demonstrations, and further explo- ration. This provides framework where new tasks can be defined naturally by a non-expert, without requiring reward shaping or state-space engineering. Experiments across range of tasks show our method is able to adapt not only to novel instances within a task family, but also to entirely novel task families, whilst doing with superior data efficiency to range of baselines. This provides the foundation for further theoretical extensions of this framework, such as learning from imperfect demonstrations, as well as further applications, such real-world contact-rich robot manipulation.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Under review as conference paper ICLR 2017 RL FAST REINFORCEMENT LEARNING VIA SLOW REINFORCEMENT LEARNING. Technical report.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adapta- tion of Deep Networks. Technical report, 2017.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- Reinforcement Learning of Structured Exploration Strategies. Advances in Neural Information Processing Systems, 20188-December::522255111 1802.07244
2018. URL
Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. Technical report.
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. 5 2019. URL http:://axxxi org/abs/ 1905. 06424.
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided Meta-Policy Search. Technical report.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. Technical report. URL http::/sites google.ooooooooooooooooooooooo.. com/
wwe.ddeeeeeeeee weetteroos-maiiulleeee manipulationn
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient Off- Policy Meeaa-Reinforremment Learning via Probabilistic Context Variables. Technical report. URL
https: /ttptb.. wwt.////eeeeeelll///oeeaaaaee/ee
Stéphane Ross, Geoffrey J Gordon, J Andrew Bagnell, and Machine Learning, A Reduction of Imitation Learning and Structured Prediction No-Regret Online Learning. Technical report.
Gerrit Schoettler, Ashvin Nair, Juan Aparicio Ojea, Sergey Levine, and Eugen Solowjow. Meta- Reinforcement Learning for Robotic Industrial Insertion Tasks. 4 2020. URL httttt//p/axiii org/abs/ 2200 4. 14404.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards. 2017. URL
/atrg//bb///// http: http:://axxiv/ org/ abs / 1777. 08817.
Hanlei Wang. Adaptive Control of Robot Manipulators With Uncertain Kinematics and Dynamics. Technical report, 2016.
Haozhe Wang and Jiale Zhou. Learning Context-aware Task Reasoning for Efficient Meta- reinforcement Learning KEYWORDS Multitask Learning, Deep Reinforcement Learning. Tech- nical report, 2020. URL www. faamas org.
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-Inverse Reinforcement Learning with Probabilistic Context Variables. Technical report.
Allan Zhou, Eric Jang Google Brain, Daniel Kappler, Alex X Herzog, Mohi Khansari, Paul Wohlart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn Google Brain. WATCH, TRY, LEARN: META-LEARNING FROM DEMONSTRATIONS AND REWARDS. Technical report. URL httpss://sitsss.goolleee
wttmm/viewwwwwwaeee taahh--rr--- cry-learn project
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Ku- mar, and Sergey Levine. The Ingredients of Real- World Robotic Reinforcement Learning. 2020. URL http:/// / /aaxiiivvvv...../ org/ abss//////////// wttpaa////20202. 2204

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 MATHEMATICAL EXTENSIONS
Probabilistic Kernel
We adopt a MaxEnt RL formulation to define meta-objective for task T E p(7) (10), where H(- (o) represents the observational entropy, and the temperature a controls the weighting term for the entropy-basee based reward.
π = org max
Σ TEA(T)
BEST
Besppelle [r(os, at 2) + a =)] TET
(10)
Optimising the objective defined in (10) possible by using the SAC soft policy iteration approach. The difference to standard SAC is that in our case, the observation is augmented by 2 resulting in ussk-conditioned critic (11) and actor (12) losses for the the Q-function Qo(-) and the policy Te(-) respectively. V corresponds the frozen target value function (no gradients stored in the forward pass).
Notice the term B7 to represent the distribution of transitions in task T, which is modelled by replay buffer. Moreover, the over-line operator in denotes that gradients are detached. You may seek further details on the derivation of actor and critic losses in Appendix of (Haarnoja et al.).
C BenTT [Qolo, (r V(or, 2))]
(11)
xx,(Qa(o,a,2) (o(o)
Cutor EETT Toog (sle) aammsallo.)
DEL To(alo,
(12)
Where (o) is the normalising partition fuucction, which is intractable yet has no effect on the com- putation of the gradients. In the computation of Castor- the gradients for are detached, allowing the policy loss to exploit the representation of the MDP passed on by the critic without conditioning the inference network (=(=)c).
Observe in 11 we allow gradients from the inference process 2, Z pass onto the computation of Qo(o, a, such that meta-optimisation of the latent variable is conditioned on reconstructing the Q-function.
Conditioning on Demonstrations
We minimise the conditioned mutual information term by enduring the following deconstruction process.
min + (ere))+HEeegDDrrroorrrrtttltee +
p(z)
log P(2|T)
-min ( +p(err)
log p(217) (o(2)T)
(13)
-min (P + B log (z(T))
In order to optimise the two terms in (13). we define Lae and Linfo the leftmost and rightmost distribution expectancies respectively. In this definition, the expectancies are not differentiable and requires parametric manipulation. The first step adapted this objective into meta-R RI. form, by approximating the conditional p(T[z) with the policy pe (T(2).
define the first term Lie (0) as conditional behavioural cloning loss.
Lhc Epte) D K (P (r)))
(14)
10

---

Under review as a conference paper at ICLR 2021
By manipulation of the Kullback-Liebler divergence obtain the following expansion.
log PTo (T(2)
min pp)))
= min E po
log pe (T|2)
log Prg. P.r. (T|z)
min E (p(llr))
log po (T|z) = min E2 (a)p(())))) log Pog Prs (T|2)] min E Esmpo(elr) logpe(T
We then substitute policy my the conditional distribution po(r|z) and our variatonal inference approximation for the posterior (=(r))):
min ee-p(()))) log Pre (7|2) min E mo [og))(712))
min H(P=A (T|2)) (e log To (T|2)
Because the entropy term is not dependent on the parameters 0, we derive a lower bound objective which is identical to the maximum likelihood estimator for trajectory distributional matching in BC:
TT = -B,upT (1))) [T]]]
(15)
Note that in the final representation (15), expert trajectory distributions pTg (T) are conditioned by task T as would be the case during meta-training. To obtain a tractable form of Linfo we leverage generative processes switch sampling distributions p(z) to sample z. To that end we sample from both expert distributions T ~ PT (T) and the posterior approximation (encoder) z (0(2(T) to arrive (16).
do not have direct access to the prior
C - Eyppp soag (alr)) log (o((T)))
(16)
Notice how in LT generate of trajectories To from task belief sampled from expert demonstration trajectories. From this we aim to maximise the likelihood of matching the posteriors on z given Tg. In essence, this can viewed as two-player game: we want to match the generated by demonstrations to the generated by policy which is also conditioned on the demonstrations.
In this form, CT is incompatible with off-policy RL as ти is sampled from the current policy mæ- We propose truncated importance sampling by sampling from the distribution which collected ræ (17). We use behavioural policy TA TT(T(2) which is updated every episode as frozen copy of ma-
CT -EropI -og.t min log (T(2)))) log TO (To 1 log (
(17)
A.2 ALGORITHM DETAILS
Computational Graph
Figure illustrates the computational graph and training procedure of PERIL.
Meta Training
Algorithm 2 is detailed extension of the meta-training algorithm presented in the main text.
11

---

Under review as a conference paper at ICLR 2021
Z
Egg
Lactor
Lic
Levitic
(xlc)
Linfo
Te (alo,
Qay (0,
d₂ (b)zz
Leux
Figure 8: Computational graph PERIL. Loss gradients (defined by blue circles) propagate to their corresponding models. These include the task encoder 9á, actor TT,,, critic Q and auxiliary module dx. Black circles denote placeholders where latent task beliefs 2 can be updated (arrow head pointing inwards) evaluated (arrow head pointing outwards). The operator ① denotes summation of the loss gradients. The colour of the arrows pointing outwards from the losses indicates the model dependency for that loss. Notice critic and auxiliary losses are shared within models. Last, the black outlines denote the loss gradients which contribute to the meta-objective G(T]=).
Algorithm 2 PERIL: Meta-training
Require: Batch of T training tasks (Ti),=,,..,r, Ts P(T)
1: Initialise To. Qo. qe- by and Ot QQQ Otg, 0b--
2: Initialise replay BTT encoder X:T and demo buffers D.T for T E (Ti):=,,...
3: while not done do
4: for each Tú, i ( (1,...,T) do
5: Initialise context by sampling demonstrations e² (TE), TE DD
6: X; +
b> Clear task encoder to reduce on-policy mismatch Demonstration-condiiooe trajectories
7:
for I......ppriior do
8:
Sample go (2(c))
9:
Collect trajectories ~ (T(2=) and add to B, and X Exploration--ooniiionee trajectories via posterior sampling
10:
for k= I,...K.Kppss post do
11:
Sample / (2)ct)
12:
T T~ To (T[=), B + B, U
13:
if ER[T ERITE] then
14:
D. (D.UT]
Demo augmentation
15:
d' (d(r"') X
for 1,,,, do
17:
Sample R random training tasks from p(T) Clone behavioural policy Th To
To apply TRIS
19:
for each T. i R do
20:
Sample recent trajectories x² Xi, TE Di, ~ B Form hybrid context é¹ = {TE Ux*)
21:
22:
Sample latent task descriptor 2 ~ (0(2)))
Compute losses Limis Pi fictor Corstiio Lix 2 Li fiave
24:
are V E Ex + DD Lime fa
AQ t 0x --- aa Vn E DiER Liritic
26:
0, + 0q -aq Vo Dier (Luotor + Cix)
27:
- as V, EiER Drnzz
12

---

Under review as a conference paper at ICLR 2021
Meta Testing
Algorithm presentes the process behind meta-testing.
Algorithm Meta-Teess adapation
Require: Networks Ld. test task T ~ p(T), demonstration buffer DJ
1: Initialise context by sampling ic demonstrations c {TE)f ~ DJ
2: for l. do
3: Sample (z(z)))
4: Roll out policy wo o,,)) for N steps and collect trajectory т {(0,, dt, Tt, 1+)))==))
5: Update context e <- {eNT}
A.3 ENVIRONMENTS
2D task families are set up with Pymunk and the 3D reach task families are set up from MuJoCo's ReachEnv. Each task T E P(T) also contains an initial observational distribution po(o) This distribution defines how the robot is initialised in the environment hefore collecting data. Since in this project we want focus on meta-learning the dynamics of contact-rich tasks, we provide an initial distribution po(o) which allows the agent to start relatively close to the interaction. We summarise Po(o) in Table 2, where Po and 9g represent the initial effector position in Pymunk units (PyU) (or meters if Reach3D) and angle (radians). Note that via inverse kinematics, the other links of the robot are automatically set.
Table Initial observational distribution for all task families, where denotes the point of inter- action and is specific to each task family. In Peg2I and Key2D), E is defined as the clearence hole entry point, where in Stick2D, defined by the extrusion's s tip.
Task Family
Distribution
Peg2D, Stick2D, Key2D
100 < PP - E 250, 00 < 4/4
Reach2D
250 < P. - E| < 500, 16 < π
Reach3D
0.05 PPo E 0.1
A.3.1 SPACES
Each task has accessible ground truth task descriptor b E R which describes the exact position in space of its goal G. all 2D tasks, the base (origin) of the robotic manipulator kept constant. The first component of the observational space O is defined by the normalised relative position from the tip of the end effector (P E IR2) to the origin S2 of the robot (18), where w is the vector which defines the width and height in PyU of the visualisation domain (1280, 720). The second component of the observational space corresponds to the sine and cosines of the angle of the end effector 0. The action space A gives complete freedom the effector and includes its cartesian velocity (vz, vy) (PyU/s) and angular velocity (rad/s). Through inverse kinematics, the latter are translated back the other links. Observation and action spaces are summarised in Table 3.
P-Ω
Orel O(P)
(18)
Table 3: State-action spaces for all task families.
Feature
Description
for of cos 0, sin 0 R4
A
(Uas Uyr wo ) E RR'
By definition of the observational space, the agent has no information about the position or condi- tion of the goal. Hence, during adaptation the robot must seek to find the goal and identify the task
13

---

Under review as a conference paper at ICLR 2021
dynamics using its interactions as well the information given by the demonstration. Note that in real world conditions, observation and action spaces could be augemented with kinematic compo- nents such as forces and torques at the end effector. However, in this simulated environment, will show that deciphering the dynamics can be performed by using the aforementioned observational and action spaces.
A.3.2 REWARD FUNCTIONS
Naive dense reward functions provided during meta-training of the critic. These reward functions are unshaped and give no information about the dynamics. essence the reward function r(-) is defined the absolute distance from the effector tip P to the goal G. The only exception is the Key2D task family, where reward functions are augmented with additional incentives to twist the handle to a desired angle a. Table 4 summarises the reward functions used each task family.
Table Dense reward function used for critic meta-training for all task families in MetaSim.
Task Family
Function
Peg2D, Reach2D, ReachåD, Stick2D Key2D
P
GG 0 al
On the other hand, we only provide binary sparse rewards r(-) [0, 1] the task encoder. These
are O in all cases except when the effector has reached its goal, where we provide reward of 1.
A.4 PAEEER-PRMMEERSS
Unless otherwise noted, the default hyperparameters for PERIL are as described below.
Training Loop
Action space |a|: 3
Observational space lo]: 4
Latent space | 8
Maximum path length H: 60
Prior collection steps Krrior: 240
Posterior collection steps K posterior: 240
RL batch size: 256
Encoder batch size: 64
Number of expert demonstrations K:
RL. sparse rewards: False
Task belief sparse rewards: True
Maximum context size: 256
Number of pre-training steps: 1000
Number of training steps: 1000
Number of tasks to average gradients in meta-training R: 16
Number of tasks to collect data from each epoch: 32
Number of Epochs: 100
Discount factor 7: 0.98
Entropy temperature: 0.1
Soft target value 0.005
Kullback-Liebler divergence weighting WKL: :: 0.1
Behavioural cloning weighting whe: 0.5
14

---

Under review as a conference paper at ICLR 2021
Mutual information weighting Wmii 0.2
Auxilliary loss weighting Wauz: 4.0
Actor
Hidden layer size h: 256
Architecture: [|,, a, z|, h, h,
Learning rate Oty x 10
Output: Bounded action [-1,1] via tanh
Critic
Hidden layer size h: 256
Architecture: [a, =/. h,
Learning rate: aq: 3 x 10
Encoder
Latent size: d
Input affinity architecture: [[o, a, r, o|, 256, 256, d]
Output affinity architecture: [d, 64, 64, |=]]
Learning rate: ay: 3 x 10
Belief Module
Task descriptor dimension [6/: Architecture: [|2|, 32, 16, [b]]
Learning rate: ou: 3 x 10-4
15


Reviews:
[{"review_id": "NWmRMUyvkut", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper introduces PERIL, a meta RL method that combines demonstration trajectories and trajectories collected by the policy, in order to adapt to a new task. To this end, the authors combine ideas from metaRL (specifically from PEARL (Rakelly et al. 2019) and Humplik et al (2019)) where a set encoder is used to encode trajectories to a latent vector describing the task, with imitation learning techniques by (a) training this encoder also with demonstrations (b) initialising the latent vector at test time by feeding demonstrations through the encoder, and (c) having additional losses inspired by metaIL techniques. The motivation is that using demonstrations allows us to learn tasks that are difficult otherwise, for example because the rewards (at test time) are sparse. \n\nOverall impression:\n\nI like the idea of using demonstrations for metaRL when tasks are sparse. Many metaRL methods do not work well in sparse reward tasks, and using expert demonstrations is a nice way of guiding the agent towards behaviour that can solve the task. Empirically, the proposed method PERIL outperforms the baselines PEARL and MetaIL, so that is promising. The authors provide analysis of the latent space which nicely illustrates what the method has learned. However, PERIL is quite complex since it consists of many different parts and loss terms (six if I counted correctly), and it needs demonstrations + interactions + (sparse) reward signals at test time. I found it hard to keep track of everything and make sense of how these parts fit together. From both the text and the empirical results, it is not clear to me why all the parts are necessary / what they do, and I am left wondering if a simpler approach would work as well. The notation and mathematical formulation in the paper is not polished enough (there are inconsistencies, variable name clashes, some parts of the objective function not properly introduced and explained) which added to my confusion. Therefore, even though the idea seems promising, I think the paper is not quite ready for publication.\n\nQuestions:\n- In the introduction you say that MetaIL methods have the drawback that \"after adaptation, they cannot continue to improve the policy in the way that RL methods can\". You say that you method PERIL \"allows for continual policy improvement through further exploration of the task\". I have a few questions about this.\n - Since only the latent embedding is updated, doesn't PERIL also suffer from the fact that the policy cannot be improved in the way that RL methods can (but instead, all adaptation is that within the limits of task inference)? \n - Why is additional exploration at test time even necessary, if we have expert rollouts and the policy itself isn't actually updated (the only thing that's adapted at test time is the latent embedding)? If all the demos + trajectories are used for is task inference, then shouldn't the expert demos always be sufficient?\n - You motivate your approach by saying that at test time, it is useful if the expert does not have to provide a shaped reward. However, you do make use of a shaped reward during training - this is a limitation that should be discussed in the paper. In addition, you still need (sparse) rewards at test time. Are those really necessary, given that you have a demonstration of the task? Did you test PERIL without those sparse reward inputs to the encoder?\n- Table 1, how was the agent trained? Was it with number of adaptation trajectories k>0? If so, what if you would train the agent with k=0? On the other hand, can you get good zero-shot adaptation performance by just increasing the number of demonstrations? \n- You say $d_\\lambda$ is a VAE, but if I understand your setup correctly then $d_\\lambda$ is only the decoder of a VAE right? And Eq 8 is the reconstruction loss? Which also means it's not technically a VAE, because in encodes and decodes different things (encodes trajectories, decodes task descriptors - Humplik et al. 2019 describe this as an information bottleneck). Shouldn't there also be a KL term somewhere here?\n- Where is $L_{bc}$ used? It's not part of Eq (7), but I also can't find it anywhere else except in Algo 1 and Fig 2. And what about $L_{mi}$, it's only in Algo 1 but nowhere else? Fig 2 has $L_{KL}$, where is that from? It would really help my understanding of piecing everything together if there was one single equation somewhere, that includes all loss terms. For each loss term, I as the reader want to clearly understand where it comes from, and why it is necessary (see suggestions for additional baselines/ablations below).\n\nSuggestions / Feedback:\n- The problem formulation and the proposed solution don't match. In your problem setting you say you're in a general POMDP where the true state may be partially observed, but in your algorithm you rely on the fact that it's a POMDP only w.r.t. the task (i.e., reward and transition function) and *not* w.r.t. the environment state. That's an important difference! To explain that in more detail: in the introduction and problem setting you say $z$ models the true underlying state $s$ which can change at any moment: your transition function is $p(o',z'|o,a,z)$ where $o$ are observations. However, the entire formulation in your algorithm relies on the fact that $z$ does _not_ change over time, but instead describes a fixed task. That's also what PEARL does, which is what your formulation is based on. I think there's two ways to resolve this: (A) Either change the problem setting such that $z$ is fixed throughout time and define the transition function as $p(s'|s,a,z)$ where the environment state $s$ is now fully observable, or (B) change the algorithm to actually model a belief over a latent $z$ that can change over time. Option (A) is probably an easier fix, but then you might also have to change some of the environments (if I understand correctly, in Key2D the state of the handle is unobserved and can change the unobserved environment state).\n- Section 3.1, I would add explicitly what the objective of the policy is (both in writing and in a mathematical expression). You aim to maximise the return of a policy that conditions on $K_d$ demonstrations, and which has interacted with the environment for $K_r$ rollouts (changing the notation here to make the distinction clear). From there it is easy for the reader to see what happens if you set $K_d=0$ (you get something more similar to PEARL), and what happens when you set $K_r=0$ (which is the zero-shot case). It's good to contrast this for the reader, and explain / show empirically why and when $K_d>0$ and $K_r>0$ is necessary. (See my comment on baselines below.) \n- To understand PERIL better, I would suggest to add a few baselines. \n - PEARL with a pre-initialised buffer that contains the demonstrations. The encoder and policy will be trained as normal, but there's additional data coming from the buffer that contains expert trajectories. Since PEARL uses an off-policy algorithm it is possible to train the policy with this data. I think this is an important comparison, because it's a very simple way to incorporate demonstrations into PEARL and it would be good to understand if/when/why this works/doesn't work.\n - In addition to the above, use the demonstrations at test time to initialise the context in PEARL. This is very close to the setting in PERIL, except that some parts of the objective function are missing ($L_{info}$, $L_{aux}$, $L_{bc}$, $L_{mi}$ - I think).  This would give insight into whether those additional losses are truly necessary (currently you only have ablations on $L_{aux}$).\n - Zero-Shot PERIL. There is some analysis of this in Table 1, but I think it would still be helpful to add this baseline. Does it work well for within-task-distribution adaptation (Fig 4) and not so well for settings that require more generalisation (Fig 5)? What if we just throw in more demonstrations, is that sufficient to do zero-shot adaptation or do we really need the policy rollouts? I think this is a central question that should be very clearly answered in the paper. Table 1 is a good start but this analysis can be expanded.\n - Humplik et al. (2019), but with additional demonstration data to train the encoder/decoder. Again, this is the simplest way to incorporate the demonstration data into this method without explicitly making use of it at test time. This comparison would tell us something about why the demonstrations are necessary - are they necessary during training but not at test time, or the other way around, or are they necessary both during training and testing?\n - Not sure I got everything, there's still $L_{bc}$ and $L_{mi}$ which I'm not entirely sure where they come from and if they are necessary. But basically, I think it's really important to analyse which parts are necessary - and make the method as simple as possible if you find some parts are not necessary.\n\nSmaller comments (didn't influence my score):\n- There's a clash between using the variable k/K for the demonstrations (e.g., Sec 3.1 \"primal inference\", Sec 4.1 first sentence, Fig 7), and for the number of policy rollouts (e.g., algorithm 1 line 5, Table 1). This is confusing, so I strongly suggest using two distinct variable names (something like $k_d$ and $k_r$ also works).\n- Similarly it would help if you use two separate notations for the trajectories $\\tau$ that come from the demonstrations, and the ones that come from the policy. Throughout Section 3 I don't always understand which one of the two you are talking about.\n- Your references need fixing. Some of them are without a year, and some say technical report even though they were published at a conference (e.g. Finn et al., Rakelly et al.). Your \"Wang\" reference for RL2 also seems wrong (first sentence in related work)? It should be Jane Wang et al. \"Learning to reinforcement learn\". The way I always get my bibtex entries is via scholar.google.com: search for the paper there; click on the \"cite\" button under it and then \"BibTeX\" (Double check whether the paper was published somewhere though, google scholar often only puts the arXiv link then actually the paper was published somewhere. Sometimes the authors also put the correct bibtex comment on their homepage/github with the code).\n- Fig 1 left, there's a typo: \"learn to lean\" -> \"learn to learn\"\n- For your experiments, I would call PERIL-A only \"PERIL\" (since this is your full method, including all losses), and then call the *ablations* different, so for example PERIL-noAux when you remove the auxiliary loss.\n- All figures should have some form of indication of the error/std/confidence interval (using shaded regions around the mean for example).\n- Sec 4.2, explain what the multi-task family setting is and why it is challenging. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fr8u3lcaNA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary of the Work\nThe work propose a method which allows us to synthesize meta-RL and meta-IL, by pre-training and conditioning a context-based off-policy meta-RL algorithm on imitation data. Strongly inspired by PEARL (Rakelly, et al 2019) and meta-IL (Yu, et al 2020), this method outperforms previous methods by varying margins on a range of newly-introduced 2D and 3D robotics tasks. The work introduces several new design elements and losses to this family of methods, and the experiments do not make clear which ones are responsible for the increased performance. Additionally, it's not clear the included experiments can fully substantiate the long list of claims provided by the authors, not the least of which is that their method performs zero-shot adaptation to new tasks.\n\n## Pros and Cons\n\n### Pros\n* Addresses several important problems in adaptive RL\n - generalizing to complex tasks and outside-of-distribution tasks\n - using demonstration data to avoid costly random exploration\n - fine-tuning policies acquired with meta-RL/meta-IL\n* Hyperparameters and reward functions are well-documented\n* The visualizations are clear and helpful\n* Overall the work is well-organized\n\n### Cons\n* Makes many claims about the method which are difficult to fully substantiate in such a short paper\n* Treatment of prior work is limited to only a few methods from the past few years, and does not acknowledge many prior works in context-based meta-RL/MTRL. Relationship to the very-similar prior works PEARL and meta-IL is unclear.\n* Experiments section makes it very difficult to compare presented results to prior work\n* Lack of ablations make it unclear which (of many) new design decisions are responsible for the method's performance\n* Experiments don't appear to make a credible simulation of demonstration data which might be encountered in the real world\n* Claims of applicability to robotics necessitates comparisons with robotics benchmarks (See MetaWorld and/or RLBench).\n* Presented method is very complex\n* Experiments lack multiple seeds and any statistical significance tests\n* Terminology and notation often veers far from previous works and conventions, making it difficult to parse. Sometimes writing is ambiguous or unclear.\n\n## Evaluation\n### Quality\n3/5\nThe presented work and experiments are high-quality in their motivation, implementation, and usually their presentation. I think the quality of this work suffers when we consider how well it positions itself with respect to prior work (not well), and the level of detail with which it explores and substantiates each of its claims (of which there are many, making it impossible to address any of them fully). The method section is extensive and recapitulates in detail many concepts from RL, variational inference, IL, etc. It can probably be more sparse to make room for a better treatment of prior work and more experiments/analysis of the work's claims.\n\n### Clarity\n2/5\nThis work suffers from significant clarity issues, mostly around use of language (e.g. zero-shot learning vs few-shot learning), non-standard notation and terminology (e.g. \"primal inference,\" \"privileged information,\" etc.) and the use of equations which are not really necessary to demonstrate a point, \n\n### Originality\n3/5\nWhile original in nature (for instance, introducing privileged information, considering fine-tuning and off-distribution tasks, using demonstrations for pre-training, etc.), the work does not make it easy for the reader to divine how its contributions build on significantly-similar previous works which are highly-cited. This does it and the reader a disservice. \n\n### Significance\n2/5\nIf all claims in the work are well-substantiated, it can be a very significant work, but I don't believe they are substantiated. For instance, the introduction mentions low-dimensional observation spaces as a limitation of current meta-RL/IL work, but the experiments don't seem to contain any use of high-dimensional (e.g. image) observation spaces. It's not clear how the claim of zero-shot learning is supported.\n\n### Misc Editorial Comments and Reviewer's Notes\n\n#### Claims\n* Addresses three limitations of current meta-RL/IL\n - shaped reward functions\n - constrained low-dimensional action/observation spaces\n - requires hand-defining low-dimensional observation/action spaces\n* A hybrid framework which combines the merits of RL and IL\n  - tasks defined only using demonstrations\n  - unlike other (meta)-IL algorithms, allows for improvement of the policy after adaptation\n* Uses only proprioceptive actions from the agent, and implicitly recovers the external environment state\n* Allows for learning new tasks without requiring any expert knowledge in the human teacher\n* Achieves \"exceptional\" adaptation rates and is capable of exploiting demonstrations for efficient exploration\n* Outperforms other meta-RL and meta-IL baselines\n* Is capable of zero-shot learning\n* Is capable of multi-family meta-learning and out-of-family meta-learning through clustering in the latent space\n* Shows how we can use privileged information (during training) to create an auxiliary loss for training the embedding function, allowing us to recover the \"true underlying state\"\n\n#### Mechanisms\n* Represents a task by a latent vector, which is the belief state of the task given a demonstration\n* Meta-training encourages high mutual information between the demonstration data and latent space\n* After adaptation, the agent can explore and update the latent space\n\n#### 1. Introduction\n* \"hand-crafted, shaped reward functions...\" nothing in the meta-RL formulation requires shaped reward functions, as opposed to a sparse ones which are easier to craft. Granted, on-policy meta-RL algorithms are challenges by sparse reward settings in a similar fashion that on-policy RL algorithms are challenged by sparse reward settings, but this is a property of RL in general and not just meta-RL. [2] is a meta-RL method which can cope with sparse rewards, and is extensively-cited in this work.\n\n* \"defining a low-dimensional..\" this is not a limitation per se of meta-RL or meta-IL -- there's nothing in their formulation which necessitates meta-RL operating on low-dimensional state as opposed to images, though it is certainly a design challenge. See [1]\n\n* \"different task families\" Perhaps I have misunderstood the authors, but this seem orthogonal to the purpose of the work and of meta-RL. It is not immediately clear what the authors mean by \"task families.\" While there is certainly work on cross-domain transfer in IL and RL, adapting policies to different action and/or observation spaces is not a typical goal of meta-IL and meta-RL algorithms, so it seems strange to level this critique. \n\n[see \"claims\" above]\n\n#### 2. Related Work\n* Meta-learning and meta-RL far predates Wang and Duan. Please see [3,4, etc.]\n* Modern work on context-based meta-RL and adaptive RL predates PEARL. Please see [5, 6, 7] which all perform variational inference on trajectories to generate a latent context, which can then be used for adaptation\n\n#### 3. Method\n* The proposed approach seems hardly different than PEARL[2] with the following changes. The reviewer may have missed something, but given close relationship between these methods, please make crystal clear for readers the differences between this method and the substantially-similar PEARL method.\n  - Pre-training uses demonstration data rather than RL episodes\n  - This work studies what happens if you continue training after adaptation\n  - Introduces an auxiliary loss which allows conditioning on privileged information\n* 3.2: \"Traditional meta-RL methods leverage RNN-based\" - This is hardly true in any universal sense. Previous meta-RL method have used RNNs [8], variational inference [2], autoregressive models (attention)[9], hierarchy [10], exploration policies [11], etc. to implicitly model the latent task space.\n* 3.3: The included equations don't seem to add much to the paper's story and seem to recapitulate well-known results from RL, variational inference literature, or cited work.\n * 3.5: The use of a SAC expert trained on full-state versions of the environment is not a faithful simulation of expert data, which will be significantly noisier and lower-entropy than a SAC expert, and also is unlikely to be optimal according to any RL loss. I think that this reduces the IL aspects work to a form of offline RL where the offline data source is a SAC policy, and the results demonstrate that the method can reconstruct the privileged information which was available to the SAC agent. The authors note that they augment the demonstration data with \"imperfect demonstrations\", but are silent about how this is achieved (and it must be done with great care).\n\n#### 4. Experiments\n* 4.1: Though it is ambiguous from the text, these experiments seem to either present 1 experiment per method, or the average of 3 experiments per method. This is unfortunately not enough data to make a statistically meaningful comparison of the performance, especially considering the small performance differences involved. Please see [12] for a handy guide on how to compare performance. In short, you will likely need 10 seeds for each experiment and should conduct a statistical test to ensure your differences are real. Please include a 95% confidence interval in your plots for the benefit of readers.\n* 4.1: These experiments are meaningful and helpful, but it's also important to readers that they can verify you have reasonable implementations of the comparison methods. This necessitates providing some results for some of the environments used in Yu, et al and/or Rakelly, et al. How is the reader to know your implementation or hyperparameters are fairly representing the comparison methods?\n* 4.2: I think the reader would benefit from seeing t-SNE plots from the comparison methods as well. The presented plots look very similar to t-SNE plots generated by plotting samples from a PEARL posterior.\n* 4.2: It is very unclear what the authors mean by \"zero-shot\" learning. By my estimation, this method always requires some samples of the target environment to attain the presented performance, making it squarely a few-shot domain.\n* This work introduces many new design elements on top of PEARL and Yu et al, and it's unclear which of them are responsible for the observed performance. Please include ablations which compare your method's performance without each new design element, to demonstrate the impact of each.\n\n[1] https://arxiv.org/pdf/2006.07262.pdf\n[2] http://proceedings.mlr.press/v97/rakelly19a/rakelly19a.pdf\n[3] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.1796\n[4] https://www.sciencedirect.com/science/article/abs/pii/S0893608002002289\n[5] https://openreview.net/pdf/29c35690ff52463c84d9456ab511e4c944ddbea4.pdf\n[6] https://arxiv.org/pdf/1809.10253\n[7] https://arxiv.org/abs/1806.02813\n[8] https://arxiv.org/abs/1611.02779\n[9] https://arxiv.org/abs/1707.03141\n[10] https://arxiv.org/abs/1710.09767\n[11] https://arxiv.org/abs/1802.07245\n[12] https://arxiv.org/abs/1904.06979", "extra_scores": {}, "extra_text": {}}, {"review_id": "72J1jwumBL_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work seeks to efficiently learn new tasks by combining meta-RL and imitation learning (IL). Such a combination is a natural thing to try, as both lines of work improve sample complexity of learning a new task: meta-RL by leveraging experience on prior related tasks, and IL by leveraging demonstrations. Demonstrations also form a natural way of specifying a new task to the agent.\n\nThis paper extends an existing meta-RL approach (PEARL) to additionally leverage demonstrations. This leads to strong results on a set of 2D problems, as well as a 3D reaching task.\n\nSpecifically, PEARL is a Thompson-Sampling approach, consisting of two main components: (i) a learned posterior distribution $q(z \\mid c)$, which encodes a distribution over latent $z$’s reflecting the task conditioned on previously observed states, rewards, and actions $c = s_0, a_0, r_0, …$; and (ii) a context-conditioned policy $\\pi(a \\mid s, z)$, which is used for both exploration to infer the task, by producing more trajectories for the context $c$, and for solving the task, once uncertainty over $z$ is low. Specifically, this paper modifies PEARL in the following ways:\n- The context-conditioned policy is trained with an objective similar to behavior cloning to produce trajectories that match the demonstrations, in addition to learning from normal reward signal.\n- The posterior $q(z \\mid c)$ is trained to produce $z$’s that encode information about the task, given demonstrations for $c$.\n\nStrengths:\n- This paper studies an important problem: how can we quickly learn new tasks? For many real-world RL tasks, we want policies that can quickly adapt to new tasks without retraining from scratch. This paper observes that prior approaches have drawbacks: IL on its own can be data-hungry, requiring additional roll-outs or many demonstrations; and meta-RL can be challenging with sparse rewards. Therefore, combining the two is a natural and promising direction to investigate.\n- The experimental results are generally quite encouraging. PERIL substantially outperforms the baselines, and can even generalize to a fairly wide distribution of 2D tasks (i.e., the same policy can learn to simultaneously do reaching, peg-placing, and key-rotating tasks, while existing works typically learn a narrower task distribution).\n\nWeaknesses:\n- Fairly strong assumption. This paper assumes that an expert distribution $p_{\\pi_E}(\\tau \\mid z)$ over trajectories conditioned on the learned latent $z$ is available. This seems to be a fairly restrictive assumption, since $z$ is learned by PERIL, and therefore, it seems unreasonable for an expert policy to also be able to condition on $z$. Instead, it would be nice if we could relax this to only condition on e.g., the observation $o$.\n- Clarity. While the high-level approach is clear, many of the details are confusing and unclear, which makes it challenging to evaluate this approach. I list the main points of confusion below.\n    - The problem statement defines the task in terms of $z$, which is confusing, because $z$ should be part of the approach, rather than part of the setting. In particular, it’s unclear what it means for the dynamics model to condition on $z$. It seems like this may be mixing the learned latent $z$ with the true state $s$? More generally, the problem statement (Section 3.1) mixes the approach with the problem setting, which makes it confusing to understand what is a constraint due to the setting, and what’s a design choice for the approach.\n    - The principled way to optimize Eq (3) is to maximize variational lower bound (Barber & Agakov, 2003), by substituting the posterior $p(z \\mid \\tau)$ with an arbitrary function $q(z \\mid \\tau)$. This appears to be what the paper is doing, but the current phrasing is pretty unclear. In particular, it’s unclear to me how $\\mathcal{L}_\\text{info}(z)$ is optimized / defined. How is $p(z \\mid \\tau)$ defined? It’s clear how you can do this in the case where the task descriptor is available, e.g., in Section 3.4, but in general, it’s unclear what the learned $z$ should be. Is this from leveraging the latent space of the expert SAC agent? What is $\\pi_b$ in Equation 6?\n    - The notation for the task-dependent objective $G(\\tau)$ seems unnecessary and serves to distract — in particular, it’s not initially clear why we need this and not just maximizing the expected discounted rewards. I would suggest removing this notation, and just saying at the end of the approach: “overall, we minimize the following loss: $\\mathcal{L}_\\text{critic}(z) + \\mathcal{L}_\\text{info}(z) + \\mathcal{L}_\\text{aux}(z) + \\ldots$.\n    - There are quite a few undefined loss functions in line 12 of Algorithm 1, in particular $\\mathcal{L}_\\text{mi}$ and $\\mathcal{L}_\\text{D_KL}$.\n- Related work. This paper generally seems to lack appropriate citations in several key places.\n    - In the introduction, several key areas seem to require citations (e.g., citations for meta-IL, posterior sampling with meta-RL should cite PEARL, claims that meta-RL requires shaped rewards / claims that meta-IL cannot adapt afterwards).\n    - The following references seem highly relevant to related works section on exploration in meta-RL: [1], [2], [3].\n- Experiments.\n    - Why is the behavior cloning baseline with noisy demonstrations? It seems like the fair comparison should be BC w/o noise.\n    - This paper claims that PERIL is capable of exploring beyond demonstrations, but the tasks that this paper evaluates on don’t seem to require much sophisticated exploration. Substantiating these claims seems to require evaluation on tasks requiring more exploration.\n\nI am initially recommending rejection, due to the aforementioned weaknesses. I believe that the related work and clarity could be improved during the rebuttal period, which would help me raise my score, although I find the strong assumption to be a fairly serious weakness.\n\nAdditional comments:\n- Ill-formatted citations. Many of the citations are missing the year, e.g., Zhou et al., Ross et al., Mendonca et al., Duan et al., Yu et al.\n- Contextons —> contexts?\n- Variational Autoencoders are inconsistently abbreviated as VE and VAE. Seems like it should follow the standard of using VAE.\n\nReferences:\n[1] VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, Shimon Whiteson. Oct. 2019. ICLR 2020. https://arxiv.org/abs/1910.08348.\n\n[2] Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning. Evan Zheran Liu, Aditi Raghunathan, Percy Liang, Chelsea Finn. June 2020. ICML LifelongML Workshop 2020. https://openreview.net/forum?id=La1QuucFt8-.\n\n[3] Environment Probing Interaction Policies. Wenxuan Zhou, Lerrel Pinto, Abhinav Gupta. July 2019. ICLR 2019. https://arxiv.org/abs/1907.11740.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6PGapvU3L7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary of work\nThis work proposes PERIL, a method for combined Meta Imitation Learning and Meta Reinforcement Learning using context-based meta-learning. Given a set of demonstrations, a latent variable representing the desired task is inferred, and trajectories are generated conditioned on the inferred latent variable.  The data from the expert demonstrations and trajectories are used for meta-learning updates.\n\n## Review\nKey comments below are included in bold text.\n\n* Related Work\n  * I don't agree that meta-learning was conceptualized as an RNN-based task. There are many variants of meta-learning, RNN-based, MAML (optimization based), Encoder-based (Neural Processes), etc.\n  * It seems that \"Scalable Meta-IRL Through Context-Conditional Policies\" (Ghasemipour et al. 2019) and maybe \"Robust Imitation of Diverse Behaviors\" (Wang et al., 2017), are closely related context conditional meta learning methods (similar to Yu et al. which you cited). They may merit citation.\n  * Bottom of page 2: Why do they have the traditional caveats? They are using rewards, so they should be able to do better.\n* Section 3\n  * Section 3.1: A number of works which also combine Meta-IL & Meta-RL use the setup you use as well (Primal Inference + Exploratory Adaptation). I think you should acknowledge this and provide citations (such as Mendonca et al., Zhou et al., Gupta et al. which you cited in other sections).\n  * Section 3.2: What is a Variational Encoder? This is not a standard term.\n  * Section 3.2: Please clarify how the KL relates to the mutual information. While this may be a simple connection, I think it should be explained better.\n  * __Section 3.2: Where is equation 1 coming from? It looks like the Evidence Lower Bound but $\\mathcal{G}(\\mathcal{T}|z)$ is not specified and could be a non-likelihood objective function. Please clarify.__\n  * __Section 3.2: You mention that you are using an encoder in the same manner as Rakelly et al., but you do not actually describe the method in your manuscript and how you perform context aggregation. Please include a complete description in the main manuscript.__\n  * __Section 3.2: First paragraph of page 4 requires significantly better clarification. Specifically the sentences \"This is generally not ... in unseen environments.\" It is not clear why you are saying your context-based approach is better.__\n* __Section 3.3:__\n  * __While you do acknowledge Yu et al. in the beginning of this section, you understate the extent to which this is similar to their mutual information objective. Equations 3&4, \"Learning from demonstrations\", and \"Linking variational posterior\" are effectively identical to Yu et al.'s with different variable names. You should clarify and attribute credit to their work in a more clear manner.__\n  * __I do acknowledge that you have tried to address the intractability of $\\mathcal{L}\\_{info}$ in a different manner than Yu et al., but the main manuscript would benefit from a brief explanation for why you took this route (even though you have derivations in the appendix)__\n  * __You need to explain why $\\mathcal{L}\\_{BC}$ is not included in your objective. From looking around in your appendix, I think this is because you are estimating the $p(z)$ marginal with the expectation distribution as in equation 5, so you don't take the gradient with respect to $q\\_\\phi$. This still needs explanation in my opinion.__\n  * __Section 3.4: First sentence, what do you mean by unsupervised? You are using the critic to train the model the generates $z$. Please elaborate.__\n  * __Section 3.4: Sentence before equation 9, doesn't this depend on what information is included in the vector $b$?__\n  * __General Question: What is the reinforcement learning algorithm you are using?__\n  * __Section 3.5: I don't understand what you mean by $\\mathcal{D}^{\\mathcal{T}}$?__\n* Section 4\n  * Please explain what the auxiliary informations are for each task.\n  * __Baselines__\n    * __As far as I can tell, you have not compared to any prior method that combines Meta-IL and Meta-RL (such as Mendonca et al. or Zhou et al. which you cite in your work). I think this is a big flaw of your experimental section. Including this would demonstrate whether your latent-based meta-learning approach is better or not from prior works with used other types of meta-learning, which should be your main value proposition. I think latent-based might work better than the MAML-based as in some prior work, but you have not provided any experiments to support this.__\n    * __Why are you comparing to noisy BC and not normal BC (also from your description of noisy BC is not clear what the method is)? Also please clarify what BC loss you are using (e.g. mean-squared error loss, maximum likelihood with learned variances, or whatever you used).__\n    * __I think it would be valuable to also include one setting where only the auxiliary loss + critic loss (no mutual info) is used to see the the value of the additional mutual information losses.__\n  * Section 4.1: How many train/test tasks in each task family. \n  * __Section 4.1: You mention sparse rewards here, but in the appendix there is information about \"dense rewards during meta-training\" for the critic. I don't quite understand what is happening here. Are you using sparse or dense rewards? And is every baseline method that uses rewards use the same type of reward? Please clarify.__\n  * __Section 4.2: Do you mean that you are training a single agent on all tasks simultaneously? How are you doing this when in the observation/action spaces are different? Do all these have the same observation/action space?__\n  * __Section 4.2: Last sentence of paragraph 1 needs elaboration. I don't think you have provided any explanation for these experiments, for example how you are setting up experiments for adapting to unseen dynamics.__\n  * __Section 4.2: Table 1, how is 0 possible? Also, you haven't provided explanation for how these values are computed.__\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method that combines imitation learning and meta-learning, which aims to be able to explore beyond the provided demonstrations. 

While the paper addresses an important topic, and the authors are commended on a productive conversations, there is a consensus among the reviews that the work is not yet ready for publication. The future manuscript should address: reexamine the assumptions and improve presentation.
Decision: Reject