Enhanced Meta Reinforcement Learning via Demonstrations in Sparse Reward Environments
Authors: Desik Rengarajan
Sapana Chaudhary
Jaewon Kim
Dileep Kalathil
Srinivas Shakkottai
Venue: NeurIPS 2022 Accept (2022)
Keywords: Meta-Reinforcement Learning
Learning with Sparse Reward

Abstract: Meta reinforcement learning (Meta-RL) is an approach wherein the experience gained from solving a variety of tasks is distilled into a meta-policy. The meta-policy, when adapted over only a small (or just a single) number of steps, is able to perform near-optimally on a new, related task.  However, a major challenge to adopting this approach to solve real-world problems is that they are often associated with sparse reward functions that only indicate whether a task is completed partially or fully. We consider the situation where some data, possibly generated by a sub-optimal agent, is available for each task. We then develop a class of algorithms entitled Enhanced Meta-RL via Demonstrations (EMRLD) that exploit this information---even if sub-optimal---to obtain guidance during training. We show how EMRLD jointly utilizes RL and supervised learning over the offline data to generate a meta-policy that demonstrates monotone performance improvements. We also develop a warm started variant called EMRLD-WS that is particularly efficient for sub-optimal demonstration data. Finally, we show that our EMRLD algorithms significantly outperform existing approaches in a variety of sparse reward environments, including that of a mobile robot.

Enhanced Meta Reinforcement Learning using Demonstrations in Sparse Reward Environments
Desik Rengarajan" Sapana Chaudhary" Jaewon Kim Dileep Kalathil
Srinivas Shakkottai
Department of Electrical and Computer Engineering, Texas A&M University {desik, sapanac, j jwkim8804, dileeppkalathil kalathil, sshakkot) @tamu. edu
Abstract
Meta reinforcement learning (Meta-RL) is approach wherein the experience gained from solving a variety of tasks is distilled into a meta-policy. The meta- policy, when adapted over only small (or just single) number of steps, is able to perform near-optimally on new, related task. However, a major challenge to adopting this approach solve real-world problems is that they are often associated with sparse reward functions that only indicate whether task is completed partially fully. We consider the situation where some data, possibly generated by sub- optimal agent, is available for each task. We then develop a class of algorithms entitled Enhanced Meta-RL using Demonstrations (EMRLD) that exploit this information-eve even if sub-optimal- obtain guidance during training. We show how EMRLD jointly utilizes RL and supervised learning over the offline data generate meta- meta-y that demonstrates monotone performance improvements. We also develop warm started variant called EMRLD-WS that particularly efficient for sub-optimal demonstration data. Finally, we show that our EMRLD algorithms significantly outperform existing approaches in variety of sparse reward environments, including that of mobile robot.
Introduction
Meta- ReittReeiforremettt Learning (meta-RL) is approach towards quickly solving similar tasks while only gathering a few samples for each task. The fundamental idea behind approaches such as the popular Model Agnostic Meta-Learning (MAML) 9 to determine a universal meta-policy by combining the information gained over working on several tasks. This meta-policy is optimized in manner such that, when adapted using small number of samples gathered for given task, is able to perform near optimally on that specific task. While the dependence of MAMI. on only a small number of samples for task-specific adaptation is very attractive, its also means that these samples must be highly representative of that task for meaningful adaptation and meta-learning. However, in many real-world problems, rewards provided are sparse in that they might only provide limited feedback. For instance, there might be reward only if robot gets close to designated way point, with no reward otherwise. Hence, MAML likely fail on both counts of task rask-specific adaptation and optimization of the meta meta-policy over tasks in sparse reward environments, Making progress towards learning viable meta-policy in such settings is challenging without additional information.
Many real-world tasks are associated with empirically determined policies used in practice. Such policies could be inexpert, but even limited demonstration data gathered from applying these policies could contain valuable information in the sparse reward context. While the fact that the policy generating the data could be inexpert suggests that direct imitation might not be optimal, supervised
Equul contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

learning over demonstration data could be used for enhancing adaptation and learning. How best should we use demonstration data to enhance the process of meta-RL in the sparse reward setting?
Our goal is a principled design of a class of meta-RI. algorithms that can exploit demonstrations from inexpert policies in the sparse reward setting. Our general approach follows two-step algorithms like MAML that employ: (i) Task-specific Adaptation: execute the current meta-policy on task and adapt based on the samples gathered obtain task-specific policy, and (ii) Meta-policy Optimization: execute task-specific policies on an ensemble of tasks to which they are adapted, and use the samples gathered optimize the meta-poliey from whence they were adapted. Our key insight is that we can enhance RL-based policy adaptation with behavior cloning of the inexpert policy to guide task sask-specific adaptation in the right direction when rewards are sparse. Furthermore, execution of such an enhanced adapted policy should yield an informative sample set that indicates how best to obtain the sparse rewards for meta-policy optimization. We aim at analytically and empirically capturing this progression of policy improvement starting from task-specific policy adaption to the ultimate meta-policy optimization. Thus, as long the inexpert policy has an advantage, we must be able to exploit it for meta-policy optimization.
Our main contributions are as follows. (i) We derive a policy improvement guarantee result for MAML-like two-step meta-RL algorithms. We show that the inclusion of demonstration data can further increase the policy improvement bound as long as the inexpert policy that generated the data has an advantage. (ii) We propose an approach entitled Enhanced Meta-RL using Demonstrations (EMRLD) that combines RL-based policy improvement and behavior cloning from demonstrations for task-specific adaptation. We further observe that directly applying the meta-policy to new sparse-reward task sometimes does not yield informative samples, and a warm-start to the meta- policy using the demonstrations significantly improves the quality of the samples, resulting in a variant that we call EMRLD-WS, (iii) We show on standard MuJoCo and two-wheeled robot environments that our algorithms work exceptionally well, even when only provided with just one trajectory of sub-optimal demonstration data per task. Additionally, the algorithms work well even when exposed only to small number of tasks for meta-policy optimization. (iv) Our approach is amenable to a variety of meta-RL problems wherein tasks can be distinguished across rewards (e.g.. whether forward backward motion yields a reward for a task) or across environment dynamics (c.g.. the amount of environmental drift that a wheeled robot experiences changes across tasks). To illustrate the versatility of EMRLD, we not only show simulations control multi-task environments, but also demonstrate its excellent performance via real-world experiments on TurtleBot robot (2). We provide videos of the robot experiments and code at https://gi thub. Soom/Desikeee wte///DDDiikkeee weweggaaaannnneaaa.... /EMLLDD.
different continuous
Related Work: Here, we provide brief overview of the related works. We leave more thorough discussion on related works to the Appendix.
34 g Model- Meta learning: Basic ideas on the meta-learning framework are discussed in IS
agnostic meta-learning (MAML) g introduced the two-step approach described above, and can be used in the supervised learning and RI. contexts. However, in its native form, the RI. variant of MAML can suffer from issues of inefficient gradient estimation, exploration, and dependence on a rich reward function. Among others, algorithms like ProMP
address the issue of 12 deal with the issue of exploration in meta-RL. PEARL [26] takes different approach to RL, wherein task specific contexts are learnt during training, and interpreted from trajectories during testing to solve the task. HTR X relabels the experience replay data of any off-policy algorithm such as PEARL 26 to overcome exploration difficulties in sparse reward goal reaching environments. Different from this approach, we use demonstration data to aid learning and are not restricted to goal reaching tasks.
(8) and DiCE [1]
inefficient gradient estimation. Similarly, E-MAMLL I1
K and MAESN
RL with demonstration: Leveraging demonstrations is an attractive approach to aid learning 21. Earlier work has incorporated data from both expert and inexpert policies to assist with policy learning in sparse reward environments 14 3d 17 27 In particular, 14 utilizes demonstration data by adding it to the replay buffer for Q-learning, (25] proposes an online fine-tuning algorithm by
combining policy gradient and behavior cloning, while (27) proposes a two-step guidance approach
where demonstration data is used to guide the policy in the initial phase of learning.
Meta-RL with demonstration: Meta Imitation Learning [C extends MAML g to imitation
learning from expert video demonstrations. WTL 400 uses demonstrations to generate an exploration algorithm, and uses the exploration data along with demonstration data solve the task. GMPS 20
2

---

extends MAML to leverage expert demonstration data by performing meta-police optimization via supervised learning. Closest to our approach are GMPS [20] and Meta Imitation Learning [TQ, and we will focus on comparisons with versions of these algorithms, along with the original MAMI. 9
Our work differs from prior work on meta-RL. with demonstration in several ways. First, existing works assume the availability of data generated by an expert policy, which severely limits their ability improve beyond the quality of the policy that generated the data. This degrades their performance significantly when they are presented with sub-optimal data generated by an inexpert policy that might be used in practice. Second, these works use demonstration data in purely supervised learning manner, without exploiting the RL structure. We use a combination of loss functions associated with RL and supervised learning to aid the RL policy gradient, which enables our approach to utilize any reward information available. This makes it superior to existing work in the sparse reward environment, which we illustrate in several simulation settings and wal-world robot experiments.
2 Preliminaries
A Markov Decision Processes (MDP) is typically represented as tuple S,A, P. where is the state space, A is the action space, R: A R is the reward function, P: P=:SAAA ((S) is the transition probability function, 7 is the discount factor, and p E (SS) is the initial state distribution. The value function V and the state-action value function Q" of a policy π are defined as V (s)
E E (=g 7 (((ss, (4)s, = 8, do a]. V (s). A policy π generates a P(-[st, at). Since the randomness of 7 is specified by P and r,, we denote it as T ~ (P. T). The goal of reinforcement learning algorithm is to learn policy that maximizes the expected infinite horizon discounted reward defined as (xx = (T.(( 1Er=0'' 7 (R(s, az)]. It is easy to see that (TT = =Bo~~p (VaovoVV(sss (so)].
E [Σ t= 7 R(ss, a4) So s] and ("(ssa)
The advantage function A is defined as as) = (s) trajectory 7 where t (80, d0, 81, "i,,..),, 80 p, dr n ((s..) 8t+1
The discounted state-action visitation frequency of policy π is defined as d (s. a) (1 7) > P'((s, = s. S. = a). 80 ~ p. The discounted state visitation frequency is defined as the marginal d² (s) = EnEA ds a). It is straightforward to see that (s, = x I ad))
The total variation (TV) distance between two distributions p and q is defined as (p, = (1/2) Σg ((x)- - q(x)), the average TV distance between two policies πı and T2, averaged w.r.t. the d is defined as (Tv( m2) Eg-d= (Drv(m1,
Gradient-based meta-learning: The goal of a meta-learning algorithm is to learn to perform optimally in a new (testing) task using only limited data, by leveraging the experience (data) from similar (training) tasks seen during training. Gradient-based meta-learning algorithms achieve this goal by learning a meta-parameter which will yield a good task specific parameter after performing only a few gradient steps w.r.t. the task specific loss function using the limited task specific data.
Meta-learning algorithms consider a set of tasks T with a distribution p over T. Each task i ~ p(T) also associated with a data set D;, which is typically divided into training data D.j used for task specific adaptation and validation data Diaal used for peraa-parameter update. The objective of the gradient-based meta-lcarning is typically formulated as
min EX-D(T) [A, (0 (o, D!), D.))))
where L. is the loss function corresponding task i and is the learning rate. Here, 0,(0) 0 - (0, D.)) is the task specific parameter obtained by one step gradient update starting from the meta- pertamerreeterr&& 6, and the goal is to find the best meta- peta-parrmeter which will minimize the meta loss function ((0) = [C.
Gradient-based meta-reinforcement learning: The -radient-based meta-learning framework is applicable both in supervised learning and reinforcement learning. In RL, each task i corresponds to an MDP with task specific model P. and reward function R,- We assume that the state-action spaces are uniform across the tasks, thus ensuring the first level of task similarity. Task specific data D. is the trajectories Ti.m = (x8" , sI', af" ...................... ,0 ~ P; 0t ~ T )), 8t+1 ~ P.(, at) for 1 S m < M generated according to some policy π. Since the randomness of the trajectory Ti.. specified by P; and n, we denote it as T. ~ TT). We consider the function approximation setting where each policy is represented by function parameterized by 0 c θ and is denoted as
3

---

To. The task specific loss in meta-RL is defined as (f) (To). The gradient VoLki, then be computed using policy gradient theorem.
(f) can
The standard meta-RI. training is done follows. A task ~ p(T) (usually a batch of tasks) is sampled at each iterate & of the algorithm. Now, starting with meta-parameter 0g, the training data D.r for task adaptation is generated the trajectories ( where T..m (Pi, To, ) and the updated parameter f, for task i is computed by policy gradient evaluated on D.. The validation data Di is then collected trajectories (Ts.m) )m where Ta,m ~ Te,,)) and the meta-parte perammerretter On is updated by policy gradient evaluated Di In the next section, will introduce modified approach which will leverage the demonstration data for task adaptation and meta-parameter update.
3 Meta-RL using Demonstration Data
Most gradient-baseds meta-RL RL algorithms learn the optimal meta- mneta-parameter and the task-specific parameter from scratch using on on-policy approaches. These algorithms exclusively rely on the reward feedback obtained from the training and validation data trajectories collected through the on-policy roll-outs of the meta-policy and task-specific policy. However, in RI. problems with sparse rewards, non-zero reward is typically achieved only when the task is completed such sparse rewards settings, trajectories generated according to a policy that is still learning may not achieve any useful reward feedback, especially in the early phase of learning. In other words, since the reward feedback is zero or near-zero, the policy gradient will also be similar, resulting in non-meaningful improvement in the policy. Hence, standard meta-RL algorithms such MAML, which rely crucially on reward feedback, will not be able to make much progress towards learning a valuable task- specific or meta-policy in sparse reward settings.
near-completed. In
Learning the optimal control policy in sparse reward environments has been recognized as a chal- lenging problem even in the standard RL setting, since most state-of-the-art RL algorithms fail to learn any meaningful policies even after a large number of training episodes 25
17 27]. One widely accepted approach to overcome this challenge is known as learning from demonstration, wherein demonstration data obtained from an expert
is used to aid online learning. The intuitive idea is that, even though the demonstration data does not contain any reward feedback, it can be used to guide the learning agent to reach non-zero reward regions of state-action spaces. This guidance, usually in the direction of the goal/target, is achieved by inferring some pseudo reward signal through supervised learning approaches using demonstration data.
2 or inexpert policy 17
2
Can enhance the performance of meta-RL algorithms in sparse reward environments by using demonstration data from sub-optimal experts? Meta-RL. in sparse reward environments is significantly more challenging than that of the standard RL setting. This is because the reward feedback serves the dual objectives of adapting the meta-parameter to specific tasks and for updating the meta- parameter itself. We note that demonstration data helps with both of these objectives. Firstly, use of demonstration data to guide task-specific adaptation becomes important because adaptation is achieved in one or few gradient steps, and policy resulting from each adaptation step might not achieve meaningful reward in sparse reward setting. Secondly, making use of demonstration data for meta-pase meta-parameter update is equally important because of the role of meta-policy as a reward- yielding exploratory policy. Intuitively, the meta-polics should yield trajectories that reach in the vicinity of the reward- achieving region of the state-action spaces. This does not happen in sparse reward environments. However, using the guidance from demonstration data, the task aask-speciiic policy obtained after the task adaptation may be able to generate trajectories that will reach within the reward-achieving region resulting in performance acceleration of the mota-policy.
For meta-learning with demonstration, we assume that each task i is associated with demonstration data Delm, which contains a trajectory generated according to a demonstration policy dedm in an environment with model P,. We do not assume that x dem is the optimal policy for task because in many real-world applications Ddam could be generated using an inexpert policy. Our key idea is to enhance task adaptation using the demonstration data by introducing an additional gradient term corresponding the supervised learning guidance loss. We define the supervised learning loss function for task as LBC (0, Dilem) = - - (x.)) log To (a]s).. We note that, though this loss function is the same as in behavior cloning (BC), we use it directly in the gradient update instead of performing simple warm start. This approach is known to achieve superior performance than naive BC warm starting in standard RL problem under the sparse reward setting [25 17 27 The

---

task adaptation step at iteration k, starting with meta- parameter 8g is now obtained as
0R. - 02 VV (w: (0; + Wbe (00,,,000000 Dele)) 10=02>
(2)
where Wrl and Whe are hyperparameters that control the extant to which RL and demonstration data influence the gradient.
The next question is: how do use demonstration data in the meta-parameter update? One approach is to use only the demonstration data with supervised learning loss function for updating the meta-parameter done in 20 We conjecture that such a reduction supervised learning will severely limit the learning capability of the algorithm. Firstly, if demonstration data is obtained from an inexpert policy, this approach will never be able to achieve the optimal performance. This is because the role of the meta-policy as a reward-yielding exploratory policy will be limited by true performance of the inexpert policy. Secondly, the task-specific policies obtained according to ( may be able to reach within the reward-y swaardyyelding region of state-action space we mentioned before. Hence, the validation data Diaal collected through the roll-out of the policies obtained after task adaptation might contain extremely valuable reward feedback. Utilizing this data could potentially have a significant impact on improving the learning the veta-parameter. Thus, our approach, we update the peta-parr using the RL loss with policy gradient as follows.
Oa 02
NV
( - aVo (w D.) Whe D(em)) ; :Dial)
We note that the demonstration data is indeed used in the meta-parameter update implicitly, as its impact can be observed in G We found empirically that the double use of the demonstration data, either by adding an additional gradient through BC loss function, or by replacing D.aal with Diee results in similar or worse performance than the approach described above.
We now formally present our algorithm called Enhanced Meta-RL. using Demonstrations (EMRLD).
Algorithm 1 Enhanced Meta-RL using Demonstrations (EMRLD)
1: Input: Set T of N tasks, demonstration data Ddemm for task I,..., N
2: Input: Adaptation learning rate a, meta-learning rate B
3: Initialize meta parameter 0g
4: for k = 0, 1...... do
5: for T do
6: Execute the meta policy Te, for task to collect D.
7: Task adaptation: Compute task adapted parameter Oz.; using DD and Dien according to
8: Collect data a by executing adapted policy To.,, for meta-policy update
9: end for
10:
Meta-parameter update: Update meta-policy according to using Dial
11: end for
We now present a theoretical justification of why EMRLD should have a superior performance the sparse reward setting compared to other gradient based algorithms that do not use demonstration data. First, we introduce some notation. Let π = Te, be the meta-policy used at iteration k of our algorithm. Also, let Tk. be the policy obtained after task-specific adaptation for task i. Recall that (T(..))) is the value of the policy for the MDP corresponding to task i. Similarly, we can define the state-action value function and advantage function of policy Tİi for task as Q and A respectively. Also, let d.*..a be the visitation frequency of policy тH, for task i. Now, we can define the value of the meta-policy मı over the ensemble of all tasks as Jmeta (TT))
(J.
If the demonstration data has to be useful, it should provide a reasonable amount of guidance. In particular, we would like the task-specific policy adapted using this data to collect feedback that would ensure good meta-policy updates, particularly in the initial stages of meta- veeta-taiining. Since the capability of the demonstration data to guide adaptation will depend on the demonstration policy n.ee according to which it is generated, we make the following assumption about rrdem.
Assumption 3.1. During the initial stages of meta-training, E ( (A, (s, 2 A, for all E S and i T.
5

---

implies that during the initial stages of meta- ceeta-trainin,, the demonstration policy can provide higher advantage on average than the current policy adapted that task. This is reasonable assumption, since any reasonable demonstration policy is likely to perform much better than an untrained policy in the initial phase of learning. We also note that similar assumption was used in learning from demonstration literature [17 27
Assumption 3.1
We now present the performance improvement result for EMRLD.
Theorem 3.2. Let ThK = =0,, be the meta-policy used at iteration i of our algorithm and let Thi be the policy obtained after task adaptation in task i. Let Assumption 3.7 holds for Tg. Then,
J m (Tk+))) Smera (TT&) 2
E8 (mp(TT)
TK+1,
A, A, (8,
(8,
2C1 1- Bimp(T) (DTV' (T +
1-
A
1-7 E Bldl(T)
201
DDyt (TK+I,0, af("")
1
Theorem 3.2
presents lower bound for the meta policy improvement as sum of two groups of terms. Maximizing the first term in group one with constraint on its second term will ensure a higher lower bound and hence an improvement the meta- meta-parameter training. We notice that this is indeed achieved by the TRPO step used in the mera-parameter update. Hence, this first group is the same for any MAML-Iype of algorithm. The advantage of the demonstration data revealed in the second group of terms. The term A/(1 - -) adds positive quantity to the lower bound, and this contribution from this second group of term can be maximized by minimizing EE~p(T) (DVV R ( ==00)) ). However, this minimization is hard perform in practice because estimating DTV!" requires sampling the data according to Tg+1,,1 and this is not feasible at iteration k. Hence, in practice,
we replace that term by E (TK+1,,, ne =. This can be easily achieved by including the standard maximum likelihood objective in the adaptation step. Thus, EMRLD both exploits the advantage offered by an RI. step, as well as that of behavior cloning for meta-policy optimization.
We can further improve the performance of EMRLD by including behavior cloning warm starting step before performing the update Q We simplify this warm start to a one step gradient as, 0g.s = 02 a VoLCC (0, De
10=0, and then do the task adaptation as in starting with Oz, We call this version of our algorithm EMRLD-WS. Such a warm start is likely to provide more meaningful samples than directly rolling out the meta-policy to obtain samples for tase-spec adaptation. In the next section,
will empirically how our design choices for EMRLD and EMRLD-WS enable them to learn policies that provide higher rewards using only a small amount of (even sub-optimal) demonstration data.
Figure 1: Point2D Navigation (first), Two Wheeled Locomotion (second) and HalfCheetah (last).
4 Experimental Evaluation
We evaluate the performance of EMRLD based on whether the meta-policy generates a good initial condition for task-specitf adaptation in sparse-reward environments over (i) Tasks already seen in training and (ii) New unseen tasks. We seek validate the conjecture that in the sparse-reward setting, EMRLD should be able to leverage even demonstrations of inexpert policies to attain high test performance over previously unseen tasks. We do so with regard to two classes of tasks, namely,
Tasks that differ in their reward functions: Simulation experiments on Point2D Navigation Twoeleeeed Locomotion [12]. and HalfCheetah (38 35
Tasks that differ the environment dynamies: Real-world experiments using a TurtleBot, which tho-wlelled differential drive robot [2]

---

4.1 Experiments on simulated environments
Sparse multi-task environments We present simulation results for three standard environments shown in Figure D and described below. We train over small number of tasks that differ in their reward functions. We generate unknown tasks for test by randomly modifying the reward function.
PointD Navigation is a 2 dimensional goal-reaching environment. The states are the (z. y) location of the agent on 2D plane. The actions are appropriate 2D displacements (dx, dy). Training tasks are defined by a fixed set of 12 goal locations on a semi-cirele of radius 2. The agent is given a zero reward everywhere except when it certain distance near the goal location, making the reward function highly sparse. Within a single task, the objective of the agent is to reach the goal location in the least number of time steps starting from origin. Test tasks are generated by sampling any point on the semicircle the goal.
Two Wheeled Locomotion environment is goal-reaching with sparse rewards, similar Point21) Navigation. However, the robot is constrained by the permissible actions (limits on angular and linear velocity) and trajectories feasible based on the turning radius of the robot. Here, our training tasks are a fixed set of 24 goal locations on a semi-circle of radius 2, while test goals are sampled randomly. Further details on state-space and dynamics are provided in the Appendix.
HalfCheetah Forward- Backward consists of two tasks in which the HalfCheetah agent learns to either move in the forward (task 1) backward (task 2) directions with high velocity possible. The agent gets a reward only after it has moved certain number of units along the x-axis in the correct direction, making the rewards sparse. Training and test are under the same two tasks.
Optimal data and sub-optimal data We provide lim- ited amount of demonstration data the form of just one trajectory per task for guidance. Optimal data consists of (s. a) transitions generated by an expert policy trained using TRPO. Sub-optimal data is generated by an inexpert, partially trained TRPO policy with induced action noise and truncated trajectories as shown in Figure
-2
Figure 2: Optimal and Sub-optimal demonstrations for Point2D Navigation.
Baselines We compare the performance of our algorithm against the following gradient based meta-reinforcement
learning algorithms: (i) MAML: [ The standard MAML algorithm for meta-RL (ii) Meta-BC: A variant of [10: this is a supervised learning/behavior cloning version of MAML, where the maximum likelihood loss is used in the adaptation as well as the neta-optimization steps. (iii) GMPS: Guided meta policy search 20 which uses RL for gradient based adaptation, and uses demonstration data for supervised meta- parameter optimization. The implemention of our algorithms and baselines is based on publicly available meta-learning code base 3 licensed under the MIT License.
Performance with optimal demonstration data: of the different algorithms trained and tested with optimal data in Figure B The top row of Figure 3 shows the average adapted return across training tasks of the meta meta-policy during training iterations. The bottom row of Figure 3 shows the average return of the trained meta-policy adapted across testing tasks over adaptation steps. We see that our algorithms out perform the others by obtaining the highest average return, and are able to quickly adapt testing tasks with just one adaptation step and one trajectory of demonstration data. Additionally, improvement in average return demonstrating stable learning. Meta-BC fails and has unstable training performance the amount of demonstration data available per task is very small. Training over only a small number of tasks further hampers the performance of Meta-BC. MAML and GMPS fail to leam due to sparsity of the environment as the purely RL adaptation step incurs almost zero reward, and hence, negligible learning signal. Furthermore, GMPS is hampered in the meta update step due to availability of only small amount of demonstration data per-task.
We illustrate the training and testing performance
algorithms demonstrate nearly-monolee
Performance with sub-optimal demonstration data: EMRLD uses a combination of RL and imitation, which valuable when presented with sub-optimal demonstrations. For the Point2D Navigation environment, we collect sub-optimal data for each task using a partially trained agent with induced action noise, and truncate the trajectories short of the reward region. Hence, pure imitation cannot reach the goal. For the ThoWheeled Locomotion environment, collect data in
7

---

EMRLD
EMRLD-WS
MAML
Meta-BC
GMPS
Point2D Navigation
TwoWheeled Locomotion
HalfCheetah Forward Backward
80
80
200
0096680
Reward
60
150
40
100
20
50
0
100
200
300
400
500
600
700
0
100
200
300
400
500
100
200
300
400
500
600
700
lteration count
Iteration count
Iteration count
80
100
300
Reward
60
80
60
200
250
40
40
150
100
20
20
-50
-20
-20
-100
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
2.0 Adaptation Steps
1.0
1.5
2.5
3.0
0.0
0.5
1.0
1.5
2.0 2.5 3.0
Adaptation Steps
Adaptation Steps
Figure 3: Training (top) and test (bottom) plots on 2D Navigation, Wheeled locomotion and Half Cheetah with optimal demonstration data. For training curves, solid line corresponds to the mean over 3 seeds and the shaded region corresponds to the standard deviation over them. For the test plots, a solid line corresponds to the mean performance over all testing tasks, and the shaded region corresponds to the standard deviation over them.
EMRLD
EMRLD-WS
MAML
Meta-BC
GMPS
Point2D Navigation
TwoWheeled Lacomation
HalfCheetah Forward Backward
5583886
80
70
200
Reward
60
150
40
100
20
30
50
100
200
300
400
500
600
700
0
100
200
300
400
500
100
200
300
400
500
600
700
Iteration count
Iteration count
Iteration count
100
150
80
Reward
60
80
60
100
40
40
20
20
-50
-0
-20
0.0
1.0
0.0
1.5
25
1.5
2.5
Adaptation Steps
Adaptation Steps
Adaptation Steps
Figure 4: Training (top) and adaptation (bottom) for 2D Navigation, Wheeled locomotion and Half Cheetah with sub-optimal demonstration data. Notation is similar to Figure
similar fashion for all tasks, but remove state-action pairs at the beginning of each trajectory. Since the first few state-action pairs contain information on how to orient the two-wheeled agent towards the goal, this truncation eliminates the possibility of direct imitation being successful. Similarly, in HalfCheetah we use a partially trained policy and truncate trajectories before they reach the reward bearing region. Figure g iilustrates that EMRLD outperforms all the baselines and is quickly able to adapt to unseen tasks, emphasizing the benefit of its RL component. Meta-BC and GMPS fail because they are restricted by the optimality of the data, and the absence of crucial information greatly impacts their performance. MAML again fails due to the sparsity of the reward.
We conclude by presenting in Figure 5 sets of trajectories generated during testing in the TwoWheeled- Lococmotion environment when provided with optimal sub-optimal demonstration data. The variants of EMRLD clearly outperform the others, showing their strength in the sparse reward setting.
4.2 Real-world Experiments on TurtleBot
We demonstrate the ability of EMRLD variants to adapt to sparse-reward tasks when they differ in environment dynamics and have sparse reward feedback. We do so via performance evaluation in the real-world using a TurtleBot shown in Figure 6 (left). We first we modify the Two Wheeled
8

---

X
(a) EMRLD
(b) EMRLD-WS WS
(c) MAMI.
(d) Meta-BC.
(c) GMPS
Figure 5: Trajectories after step of adaptation using optimal demonstration data (top), and after
3 steps of adaptation using sub-optimal data (bottom) for 20 random goal points in Two Wheeled Locomotion. The star indicates the goal, and reward is available only in the shaded region.
Locomotion sparse reward environment by fixing the goal, and changing the dynamics by inducing a residual angular velocity which mimics drifts in the environment. This environmental drift is what differentiates each task. In other words, for a given task, the environment would cause the robot to drift in specific unknown direction. We train values (i.e., 9 different driving environments). We use one trajectory of demonstration data per task collected using expert policy trained using TRPO. Note that all the training and data collection is done in simulation. The results are shown in Figure (middle), where we see that the EMRLD variants clearly outperform the others.
set of 9 tasks with different angular velocity
For testing, we consider the environment where the Turtlebot experiences fixed but unknown residual angular velocity representing environmental drift. Thus, we bias the angular velocity control of the TurtleBot by some amount unknown to the algorithm under test. We first execute the meta policy the TurtleBot in the real world collect 5 trajectories. We also provide one trajectory of simulated demonstration data. We use these samples to adapt the meta-policy, and execute the the TortleBot. The results origin is at (0, 0) and the goal is indicated by star. It clear that the variants of EMRLD are the best
adapted task-specific policy quickly adapting to the drift in the environment and are successful with just step of adaptation.
shown in Figure 6 (right), where the
EMRLD
EMRLD-WS
MAML
Meta-BC BC
GMPS
80
1.5
Average Reward
1.0
0.5
0.0
20
0.5
-1.0
200
400
TurtleBot
Iteration count
X
Figure 6: The TurtleBot (left) is a small, tho-wheeled robot. We train the algorithms simulation (middle) and obtain trained meta-policies. The meta-policies adapted under unknown drift in the real world to control the TurtleBot to attain a given goal point (right).
5 Conclusion
We studied the problem of meta-RL algorithm design for sparse-reward problems. in which demon- stration data generated by a possibly inexpert policy also provided. Our key observation was that simple application of an untrained meta-policy in a sparse-reward environment might not provide meaningful samples, and guidance provided by imitating the inexpert policy can greatly assuage this effect. We first showed analytically that this insight is accurate and that meta-policy improvement might be feasible as long as the inexpert demonstration policy has an advantage. We then developed two meta-RI RL algorithms, EMRLD and EMRLD-WE WS that enhanced by using demonstration data.

---

We show through extensive simulations, well as real world robot experiments that EMRLD is able offer considerable advantage over existing approaches in sparse reward scenarios.
6 Limitations and Future Work
EMRLD inherits the limitations of the gradient-based meta-RL approaches like MAML namely on-policy training, and data collection and gradient computation during test adaptation. A limitation specific to our proposed algorithms is the assumption on availability of task specific demonstration data. However, we reiterate that for small number of train tasks, this assumption is quite practical, further, our framework allows for this data to be sub-optimal.
A possible future direction explore is the context based meta-RL. (that does't require gradient computation during testing) with demonstration data. Another future work direction is to explore usage of demonstration data in off- policy meta-RL algorithms.
7 Acknowledgement
This work was supported in part by the National Science Foundation (NSF) grants NSF-CAREER- EPCN-2045783 and NSF ECCS 2038963, and U.S. Army Research Office (ARO) grant W911NF-19 19- 1-0367. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsoring agencies.
References
[1] Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-leamiig in nonstationary and competitive environments. arXiv preprint arXiv: 1710.0361,, 2017.
[2] Robin Amsters and Peter Slacts. Turtlebot 3 as robotics education platform. In Robotics in Education Current Research and Innovations, Proceedings of the 10th RiE, Advances in Intelligent Systems and Computing, pages 170-181. 2019.
[3] Sébastien M R Amold, Praateek Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. learn2learn: A library for Meta-Learning research. August 2020.
[4] Kai Arulkumaran, Dylan R Ashley, Jürgen Schmidhuber, and Rupesh K Srivastava. All you need is supervised learning: From imitation learning to meta-rl with upside down rl. arXiv preprint arXiv:2202.1 1960, 2022.
[5] Marc Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-82, 2020.
[6] Matthew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 23(5):408-47) 2019.
[7] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1 PMLR, 2016.
[8] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. R12: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference Machine Learning, pages 1126 1726-1133 1135. PMLR, 2017.
[10] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on robot learning, pages 357-368, PMLR, 2017.
10

---

[11]
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte carlo estimator, In International Conference Machine Learning, pages 1529-1538. PMLR, 2018.
[12] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. arXiv preprint arXiv: 1802.07245, 2018.
[13] Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John P. Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI Conference Artificial Intelligence, pages 3223-3230, 2018.
[14] Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, An- drew Sendonaris, Gabriel Dulac-Armold, Ian Osband, John Agapiou, et al. Learning from demonstrations for real world reinforcement learning. 2017.
[15] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pages 87-94. Springer, 2001.
[16] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference Machine Learning. Citeseer, 2002.
[17] Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In International Conference on Machine Learning, pages 2469-2478. PMLR, 2018.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6900 2014.
[19] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (509,055090029)) 2015.
[20] Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. Advances in Neural Information Processing Systems, 32, 2019.
[21] Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292-6299. IEEE, 2018.
[22] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 02999,9999. 2018.
[23] Charles Packer, Pieter Abbeel, and Joseph E Gonzalez. Hindsight task relabelling: Experience replay for sparse reward meta-rl. Advances in Neural Information Processing Systems, 34, 2021.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[25] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1000 2017.
[26] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy meta-reenforeemmnt learning via probabilistic context variables. In International conference muchine learning, pages 5331-5340. PMLR, 2019.
[27] Desik Rengarajan, Gargi Vaidya, Akshay Sarvesh, Dileep Kalathil, and Srinivas Shakkottai. Reinforcement learning with sparse rewards using guidance from offline demonstration. In International Conference on Learning Representations, 2022.
[28] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint (0t::1880.0061 2018
[29] Stefan Schaal. Learning from demonstration. Advances neural information processing systems, 9, 1996.

---

[30] Jüergen Schmidhuber, Jieyu Zhao, and MA Wiering, Simple principles of metaleamingg Technical report IDSIA, 69:1-23, 1996.
[31] Nicolas Schweighofer and Kenji Doya. Meta-learning in reinforcement learning. Neural Networks, 16(1):5-9, 2003.
[32] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvamn, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484-488 2016.
[33] Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and llya Sutskever. Some considerations on learning to explore via meta-reinforrcemen learning. arXiv preprint arXiv:/803.00 /8030/0301101 11000/1111100 2018.
[34] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media,
2012.
[35] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RS international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
(36] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems with sparse rewards. arXiv preprint arXiv: 08817 2017.
[37] Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn.
arXiv preprint arXiv: 161 1.1.0566,, 2016.
[38] Pawel Wawrzyíski. A cat-like robot real-time learning to run. In International Conference on Adaptive and Natural Computing Algorithms, pages 380-390. Springer, 2009.
[39] Tony Z Zhao, Jianlan Luo, Oleg Sushkov, Rugile Pevceviciute, Nicolas Heess, Jon Scholz, Stefan Schaal, and Sergey Levine. Offline meta-reiinorreement learning for industrial insertion. arXiv preprint
10.00276, 2021.
[40] Allan Zhou, Eric Jang, Daniel Kappler, Alex Herzog, Mohi Khansari, Paul Wohlhart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and reward, arXiv preprint 1906. 2019.
12

---

Ethics Statement and Societal Impacts
Our work considers the theory and instantiation of meta-RL algorithms that were trained and tested on simulation and robot environments. No human subjects or human generated data were involved. Thus, we not perceive ethical concerns with our research approach.
While reinforcement learning shows much promise for application to societally valuable systems, applying it to environments that include human interaction must proceed with caution. This is because guarantees are probabilistic, and ensuring that the risk is kept within acceptable limits a must to ensure safe deployments.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] Section 1
(b) Did you describe the limitations of your work? [Yes] Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Section 3
(b) Did you include complete proofs of all theoretical results? [Yes] Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] Supplementary Material (https: /eteps:/ggiiee //ggttbt.b../////////////t/..h.. com DesikRengaaajan /EMMRLLDD))
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] Appendix
(c) Did you report error bars (e-g.. with respect the random seed after running experi- ments multiple times)? [Yes] Section 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Section 4
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
Appendix
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "zWRC4ye2eLW", "reviewer": "Reviewer_y36R", "summary": "The paper proposes a new meta-reinforcement learning algorithm for adaptation to different tasks or environments. The algorithm targets the tasks where the reward function is extremely sparse. In these tasks the standard meta-RL algorithms fail since the reward does not give a signal for adaptation. The authors use demonstrations that can also be suboptimal. The paper shows an elegant way of integrating demonstrations using gradients, instead of doing a supervised RL. This way, the algorithm still performs well (better than demonstrations) in case the demonstrations are not optimal. The authors showcase their algorithm in various environments where rewards and environments are changed. They also apply to a 2 wheeled real robot that has differen angular velocities as environments.", "questions": "Please see above. Could the authors explain or give examples to the set of problems the algorithm tackles. Maybe I'm missing a point in the main motivation of the paper.", "limitations": "The authors cite some limitations of the algorithm, but do not discuss the concern I stated above. I think the class of problems that the algorithhm tackles is another limitation as well.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper is clear for the targeted class of problems where the reward is extremely sparse. The algorithm performs well for these tasks compared to other algorithms as expected. On the other hand, the other algorithms has a clear disadvantage of not having demonstrations or any guidance for these tasks with extremely sparse rewards, so the results are ver much expected.\n\nThe authors did a good job on testing the algorithm on various type of problems. I especially like that they show both the problems where the reward is changing and the problems where the environment is changing. Having a real robot experiment is also another plus.\n\nThe main weakness in my opinion is the set of problems that the authors target. They are looking at problems with extremely sparse reward, but with suboptimal demonstrations. These problems very much exist in real world for reinforcement learning. On the other hand, in the context of meta-RL and adaptation, I cannot think of a clear example where this would be needed, especially for the tasks with different rewards. We are looking into problems where the agent is put to an environment to adapt to an unknown task with a reward function that will mostly return zero unless the agent reaches to a goal that will be different for each task without explicit indicator to the agent. If the reward function for adapting is different, one could potentially train with task information as an input to the policy.  Also, if there are demonstrations, there are ways to break the sparsiticty of the problem. \n\nIn my opinion, adapting to different environments is slightly more likely to appear as a potential real world problem to solve. Even then, it might be feasible to train a robust policy (depending on the problem, of course). Or, in these problems, the authors can potentially look at the state transitions. Indeed I would suggest authors to look at the publication titled \"No-Reward meta learning\" NoRML, bu Yang et al. \n\nOverall I think the authors did a good job at finding an algorithm to address the problem, present their hypothesis, test their hypothesis and presenting the results. I am only concerned with the class of problems that the algorithm tackles.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o_37ojNT05", "reviewer": "Reviewer_nwsU", "summary": "This paper presents a meta-learning algorithm Enhanced Meta-RL using Demonstrations (EMRLD) that leverages potentially suboptimal demonstrations to improve meta reinforcement learning in sparse reward environments. The proposed method assumes access to a set of optimal or suboptimal expert demonstrations, and then uses a combined BC loss on the demos with RL policy improvement loss on online rollouts for the inner loop. Then, the outer loop uses a single RL gradient step on the online rollouts. The main motivation of the method is that compared to Guided Meta-Policy Search (GMPS), EMRLD is able to (1) not be limited by the quality of the initial demonstrations since EMRLD uses an RL outer-loop instead of BC on the original demos and (2) can take advantage of online-explored successes by the meta-policy. An additional version of the method is presented where it is warm-started with BC beforehand, which is called EMRLD-WS. They demonstrate the efficacy of both versions of EMRLD on simulated and real control domains, where EMRLD outperforms all baselines especially when demonstrations are suboptimal.", "questions": "- One major benefit of EMRLD is that online rollouts D_i^{val} enable the meta-policy to not be constrained by the quality of the initial demonstrations. What happens if you give GMPS this similar benefit, by using online-explored successes to the D_i^{val} used for BC in GMPS? \n- One advantage of EMRLD over MAML is the usage of demonstrations. Do you allow MAML to warm-start on demonstrations or incorporate them during training naively at all?\n- IL+RL is a well-studied field that has found the naive weighting of BC and RL loss to be very hyperparameter sensitive. How were w_{rl} and w_{bc} chosen? Do you have an ablation, such as when you set w_{rl} to 0.0 or when you set w_{bc} to 0.0?\n", "limitations": "The authors are proactive in mentioning the restriction to on-policy RL settings that involve increased computation cost and less data reusability.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method is able to incorporate suboptimal demonstrations\n- The method is not much more expensive than related methods Meta-BC and GMPS\n\nWeaknesses:\n- Fixed w_{bc} could still limit performance to be constrained around suboptimal demonstrations\n- Uses more data in the outer loop compared to GMPS, so a fair baseline would be to use self-imitation in the outer loop (see question below)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dm6YuSsQ3xu", "reviewer": "Reviewer_9vCH", "summary": "The paper introduces a new method (EMRLD) for meta reinforcement learning with a focus on sparse reward environments and learning from sub-optimal demonstrations. The authors first introduce meta reinforcement learning and the limitations of approaches like MAML in sparse reward settings, as well as relevant work in reinforcement learning and meta reinforcement learning with demonstrations. Subsequently, the authors describe the problem of learning from suboptimal demonstrations in meta-RL in greater detail and introduce their loss formulation and general algorithm (EMRLD) whose structure is inspired by MAML and makes use of combined loss from RL and supervised learning (behavioral cloning) to learn from sub-optimal demonstrations. The authors then present a theorem on how sub-optimal demonstrations can aid EMRLD in achieving superior performance to gradient-based algorithms that do not use demonstration data and then showcase how their method compares to several baselines in different experiments.\n\nThe experiments generally show that EMRLD perform as well or better compared to the baseline methods in settings with optimal demonstrations and significantly better in settings where demonstration data is suboptimal in the simulated environments. The authors then showcase results for a TurtleBot navigation setting where the robot is trying to reach a given goal. ", "questions": "**General Questions:**\n- Could you clarify the general architectural setup that you are using? Based on the formulation it appears that the meta-policy and task policies share parameters and it would be good to have more clarity and how the overall neural network architectures express that.\n- How do you determine how many task specific policies you use? Is this hyperparameter or is there a general method?\n- In the EMRL-WS setting, how do determine when to stop the BC warm start?\n- Can you provide more clarity on how the variance in the bottom of Figure 3 and Figure 4 (adaptation setting) affects your reasoning about how well your method performs against the baselines? The sub-optimal demonstration setting shows a clearer difference between the methods compared to the expert demonstration setting, but it would still be good to get more clarity on this.\n- Have you thought about doing transferring and /or meta-learning across different environment settings? This might be worth thinking about for future work.", "limitations": "I think the paper could be improved by a more thorough discussion of the limitations of EMRLD as well as future work that it can inspire. Given space limitations, it would be OK to include this in the appendix even though I would recommend it to be in the main paper. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:**\n- Strengths: The paper proposes a new method for meta reinforcement learning in sparse reward settings that can leverage sub-optimal demonstrations.\n- Weaknesses: The authors could have provided more detail on the limitations of their method (discussed below as well).\n\n**Quality:**\n- Strengths: The methods and contributions are generally well supported in the experiments and discussed in detail in the paper.\n- Weaknesses: The authors do not address the variance in their results in Figure 3 and Figure 4 bottom plots where adaptation is shown. The variance in some of the experiments makes it difficult to determine which method performs better compared to others.\n\n**Clarity:**\n- Strengths: The paper is generally well written and well organized with relevant equations, diagrams and descriptions. The theorem provided in Section 3 appears well formulated.\n- Weaknesses: To enhance readability Figure 3 and Figure 4 could use bolded \"subtitles\" such as \"Expert Demonstrations\" and \"Sub-Optimal Demonstrations\" to make the difference between them clear.\n\n**Significance:**\n- Strengths: The paper proposes a new method in a relevant subject area of meta reinforcement learning.\n- Weaknesses: The authors could have provided more detail on the limitations of their method and put it into the broader context of meta reinforcement learning methods (discussed below as well).\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept