Learning to Reinforcement Learn by Imitation

Abstract: Meta-reinforcement learning aims to learn fast reinforcement learning (RL) procedures that can be applied to new tasks or environments. While learning fast RL procedures holds promise for allowing agents to autonomously learn a diverse range of skills, existing methods for learning efficient RL are impractical for real world settings, as they rely on slow reinforcement learning algorithms for meta-training, even when the learned procedures are fast. In this paper, we propose to learn a fast reinforcement learning procedure through supervised imitation of an expert, such that, after meta-learning, an agent can quickly learn new tasks through trial-and-error. Through our proposed method, we show that it is possible to learn fast RL using demonstrations, rather than relying on slow RL, where expert agents can be trained quickly by using privileged information or off-policy RL methods. Our experimental evaluation on a number of complex simulated robotic domains demonstrates that our method can effectively learn to learn from spare rewards and is significantly more efficient than prior meta reinforcement learning algorithms.

Reviews and author responses:

This paper proposes a meta-learning algorithm for reinforcement learning that incorporates expert demonstrations. The goal is to reduce the sample complexity of meta-RL algorithms in the validation phase. The paper provides a good discussion of the background literature. Experimental results are provided on multi-goal planning problems for three prototypical simulated systems, namely, a 2D point-mass robot, a 7-DOF manipulator and a quadruped crawler.

The theoretical and practical contributions of this paper are minor. The authors propose a straight-forward combination of MAML, importance-weighted policy gradients across the inner-outer loop and off-policy supervised learning of the expert demonstrations, all standard techniques in reinforcement learning and meta-learning. The experimental section is unconvincing and lacking in details. I however find the approach well-motivated and pertinent. Demonstrations on a real robotic platform, where the improved sample complexity is essential, would make this paper much more impressive.

Detailed comments:

1. Can you compare the different algorithms using the number of rollouts as the X-axis? The narrative provides this information but it is difficult to judge the performance based on Figs. 2 and 3.
2. It is unfair to compare MRI (the approach of this paper) with MAML which does not have access to expert demonstrations for validation tasks. The improved sample complexity is thus directly coming from demonstrations. It is difficult to compare MRI and MAML-RL/Imitation on an equal footing. Perhaps the validation tasks could be significantly harder, e.g., sub-goals in the planning problems, or one could consider a large number of validation tasks.
3. It seems the improvement over MAML-RL/Imitation in Fig. 3 is minor. Why is this so?

The paper presents a meta-RL method extends previous work on meta-RL by including an imitation learning step. It is mentioned that the behaviour closing part of this extended algorithm can come from a teacher or some other source. Since this extension is the major contribution, it must be discussed in more detail. I also don't understand why 

My second problem with the paper is reproducibility. The purpose of OpenAI is comparability and reproducibility of algorithms. It is not sufficient to simply state that you have used TensorFlow. We need information about the architecture, etc so that the results can be reproduced. Also, since you use OpenAI, the score for each experiment should be compared to the best scores known from the website so that the performance of the new algorithm can be compared to others.

This work addresses the problem of learning a policy-learning-procedure, through meta-learning, that can adapt quickly to new tasks. This work uses MAML for meta-learning, and with this choice, the problem can be broken down into two loops: 

1) inner loop: adapting a policy \pi_phi based on unseen rollouts, where initial parameters phi were provided by the meta-trainer in the outer loop 
2) outer loop: the meta-trainer tries to learn parameters phi on batches of tasks that provide good initial parameters 

In prior work on meta-reinforcement learning via MAML, both the outer as well as inner objective attempt to minimize a RL objective, leading to an algorithm that has very high sample-complexity. This work uses imitation learning for the outer loop procedure, to significantly decrease sample-complexity.

Technical Contribution:
-----------------------
The idea of using imitation learning for reinforcement learning is well explored in the literature, and so using this idea in itself is not real contribution. There are several issues with the presentation of this work, that make it incredibly difficult to identify a technical contribution:

1. overreaching statements without details to backup: you are writing the paper as if you are learning a "RL algorithm" that can be used to quickly learn new tasks. your manuscript does not really provide a description for this "algorithm". After re-reading several other papers I concluded that what you mean is that you learn an initial set of policy parameters that can quickly adapt to new related tasks and an update rule with which you update these parameters. However, standard MAML uses SGD as an update rule so there is really nothing to be learned here. Unfortunately, your paper provides zero detail on these claims of learning a "RL procedure", so for now I have to assume that you are simply learning a good initial set of policy parameters through meta-learning. If that is the case, then using imitation learning in this setting is really not novel, this has been done by a lot of other people before (you're just using MAML to learn "better" initial parameters).
2. you're technical section (section 4) provides some details on the technical challenges of using demonstrations to perform the outer loop optimization step. Unfortunately, you are not putting your work in the context of existing work ([1], [2]), that discuss and address the importance/issue of sampling in meta-rl with MAML. So it's impossible to know whether there is any new insight here

Experimental Evaluation:
-------------------------
The experimental evaluation is very "thin", other than the original MAML-RL and pure imitation learning no other more recent baselines ([1], [2]) have been compared to. And only 2 relatively simple simulation settings are tested. 

Summary:
-----------
Very minor contribution, a manuscript that is lacking important details and does not relate it's technical section to existing work, with very thin evaluation. 


[1] The Importance of Sampling in Meta-Reinforcement Learning, NIPS 2018
[2] CONTINUOUS ADAPTATION VIA META-LEARNING IN NONSTATIONARY AND COMPETITIVE ENVIRONMENTS, ICLR 2018

This paper describes a meta-RL algorithm through imitation on RL policies. While the paper builds nicely up to the core part, I find essential details missing about the imitation setup. By glancing at previous BC papers (some of which are cited), the quantity for supervised imitations, etc., were clearly defined. 

It will be useful for this reviewer if the authors can provide more clarity in explaining the BC task involved in their algorithm.

Meta-review: This paper proposes a meta-learning algorithm for reinforcement learning that incorporates expert demonstrations. The objective is to improve sample efficiency, which is an important problem. 

The referees find the approach well-motivated and pertinent, but the theoretical and practical contributions of the paper too slim. A concern was also raised in regard to reproducibility of the results, missing details about the implementation and comparisons with previous results. 

The authors did not respond to the reviews. 

The four referees are not convinced by this paper, with ratings from strong reject to ok, but not good enough. 

Decision: rejected-papers