Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a new meta-reinforcement learning algorithm for adaptation to different tasks or environments. The algorithm targets the tasks where the reward function is extremely sparse. In these tasks the standard meta-RL algorithms fail since the reward does not give a signal for adaptation. The authors use demonstrations that can also be suboptimal. The paper shows an elegant way of integrating demonstrations using gradients, instead of doing a supervised RL. This way, the algorithm still performs well (better than demonstrations) in case the demonstrations are not optimal. The authors showcase their algorithm in various environments where rewards and environments are changed. They also apply to a 2 wheeled real robot that has differen angular velocities as environments.
strengths_and_weaknesses: The strengths of the paper is clear for the targeted class of problems where the reward is extremely sparse. The algorithm performs well for these tasks compared to other algorithms as expected. On the other hand, the other algorithms has a clear disadvantage of not having demonstrations or any guidance for these tasks with extremely sparse rewards, so the results are ver much expected.

The authors did a good job on testing the algorithm on various type of problems. I especially like that they show both the problems where the reward is changing and the problems where the environment is changing. Having a real robot experiment is also another plus.

The main weakness in my opinion is the set of problems that the authors target. They are looking at problems with extremely sparse reward, but with suboptimal demonstrations. These problems very much exist in real world for reinforcement learning. On the other hand, in the context of meta-RL and adaptation, I cannot think of a clear example where this would be needed, especially for the tasks with different rewards. We are looking into problems where the agent is put to an environment to adapt to an unknown task with a reward function that will mostly return zero unless the agent reaches to a goal that will be different for each task without explicit indicator to the agent. If the reward function for adapting is different, one could potentially train with task information as an input to the policy.  Also, if there are demonstrations, there are ways to break the sparsiticty of the problem. 

In my opinion, adapting to different environments is slightly more likely to appear as a potential real world problem to solve. Even then, it might be feasible to train a robust policy (depending on the problem, of course). Or, in these problems, the authors can potentially look at the state transitions. Indeed I would suggest authors to look at the publication titled "No-Reward meta learning" NoRML, bu Yang et al. 

Overall I think the authors did a good job at finding an algorithm to address the problem, present their hypothesis, test their hypothesis and presenting the results. I am only concerned with the class of problems that the algorithm tackles.
questions: Please see above. Could the authors explain or give examples to the set of problems the algorithm tackles. Maybe I'm missing a point in the main motivation of the paper.
limitations: The authors cite some limitations of the algorithm, but do not discuss the concern I stated above. I think the class of problems that the algorithhm tackles is another limitation as well.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a meta-learning algorithm Enhanced Meta-RL using Demonstrations (EMRLD) that leverages potentially suboptimal demonstrations to improve meta reinforcement learning in sparse reward environments. The proposed method assumes access to a set of optimal or suboptimal expert demonstrations, and then uses a combined BC loss on the demos with RL policy improvement loss on online rollouts for the inner loop. Then, the outer loop uses a single RL gradient step on the online rollouts. The main motivation of the method is that compared to Guided Meta-Policy Search (GMPS), EMRLD is able to (1) not be limited by the quality of the initial demonstrations since EMRLD uses an RL outer-loop instead of BC on the original demos and (2) can take advantage of online-explored successes by the meta-policy. An additional version of the method is presented where it is warm-started with BC beforehand, which is called EMRLD-WS. They demonstrate the efficacy of both versions of EMRLD on simulated and real control domains, where EMRLD outperforms all baselines especially when demonstrations are suboptimal.
strengths_and_weaknesses: Strengths:
- The method is able to incorporate suboptimal demonstrations
- The method is not much more expensive than related methods Meta-BC and GMPS

Weaknesses:
- Fixed w_{bc} could still limit performance to be constrained around suboptimal demonstrations
- Uses more data in the outer loop compared to GMPS, so a fair baseline would be to use self-imitation in the outer loop (see question below)

questions: - One major benefit of EMRLD is that online rollouts D_i^{val} enable the meta-policy to not be constrained by the quality of the initial demonstrations. What happens if you give GMPS this similar benefit, by using online-explored successes to the D_i^{val} used for BC in GMPS? 
- One advantage of EMRLD over MAML is the usage of demonstrations. Do you allow MAML to warm-start on demonstrations or incorporate them during training naively at all?
- IL+RL is a well-studied field that has found the naive weighting of BC and RL loss to be very hyperparameter sensitive. How were w_{rl} and w_{bc} chosen? Do you have an ablation, such as when you set w_{rl} to 0.0 or when you set w_{bc} to 0.0?

limitations: The authors are proactive in mentioning the restriction to on-policy RL settings that involve increased computation cost and less data reusability.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a new method (EMRLD) for meta reinforcement learning with a focus on sparse reward environments and learning from sub-optimal demonstrations. The authors first introduce meta reinforcement learning and the limitations of approaches like MAML in sparse reward settings, as well as relevant work in reinforcement learning and meta reinforcement learning with demonstrations. Subsequently, the authors describe the problem of learning from suboptimal demonstrations in meta-RL in greater detail and introduce their loss formulation and general algorithm (EMRLD) whose structure is inspired by MAML and makes use of combined loss from RL and supervised learning (behavioral cloning) to learn from sub-optimal demonstrations. The authors then present a theorem on how sub-optimal demonstrations can aid EMRLD in achieving superior performance to gradient-based algorithms that do not use demonstration data and then showcase how their method compares to several baselines in different experiments.

The experiments generally show that EMRLD perform as well or better compared to the baseline methods in settings with optimal demonstrations and significantly better in settings where demonstration data is suboptimal in the simulated environments. The authors then showcase results for a TurtleBot navigation setting where the robot is trying to reach a given goal. 
strengths_and_weaknesses: **Originality:**
- Strengths: The paper proposes a new method for meta reinforcement learning in sparse reward settings that can leverage sub-optimal demonstrations.
- Weaknesses: The authors could have provided more detail on the limitations of their method (discussed below as well).

**Quality:**
- Strengths: The methods and contributions are generally well supported in the experiments and discussed in detail in the paper.
- Weaknesses: The authors do not address the variance in their results in Figure 3 and Figure 4 bottom plots where adaptation is shown. The variance in some of the experiments makes it difficult to determine which method performs better compared to others.

**Clarity:**
- Strengths: The paper is generally well written and well organized with relevant equations, diagrams and descriptions. The theorem provided in Section 3 appears well formulated.
- Weaknesses: To enhance readability Figure 3 and Figure 4 could use bolded "subtitles" such as "Expert Demonstrations" and "Sub-Optimal Demonstrations" to make the difference between them clear.

**Significance:**
- Strengths: The paper proposes a new method in a relevant subject area of meta reinforcement learning.
- Weaknesses: The authors could have provided more detail on the limitations of their method and put it into the broader context of meta reinforcement learning methods (discussed below as well).



questions: **General Questions:**
- Could you clarify the general architectural setup that you are using? Based on the formulation it appears that the meta-policy and task policies share parameters and it would be good to have more clarity and how the overall neural network architectures express that.
- How do you determine how many task specific policies you use? Is this hyperparameter or is there a general method?
- In the EMRL-WS setting, how do determine when to stop the BC warm start?
- Can you provide more clarity on how the variance in the bottom of Figure 3 and Figure 4 (adaptation setting) affects your reasoning about how well your method performs against the baselines? The sub-optimal demonstration setting shows a clearer difference between the methods compared to the expert demonstration setting, but it would still be good to get more clarity on this.
- Have you thought about doing transferring and /or meta-learning across different environment settings? This might be worth thinking about for future work.
limitations: I think the paper could be improved by a more thorough discussion of the limitations of EMRLD as well as future work that it can inspire. Given space limitations, it would be OK to include this in the appendix even though I would recommend it to be in the main paper.
review_title: null

Meta-review: metareview: The authors propose EMRLD algorithms, which use potentially suboptimal demonstrations to perform meta-RL in environments where rewards are sparse. The algorithm is illustrated well in Point2D navigation toy examples to illustrate how it solves a multi-task goal reaching environment on both suboptimal and optimal data. Empirical results on twowheeled and halfcheetah forward-backward are compelling, and I appreciated the real-world experiments on Turtlebot. All reviewers have voted to accept. 
confidence: Certain
Decision: Accept