Title: RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning
Authors: Yan Duan
John Schulman
Xi Chen
Peter L. Bartlett
Ilya Sutskever
Pieter Abbeel
Venue: ICLR 2017
Keywords: Reinforcement Learning
Deep learning

Abstract: Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a “fast” reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL^2, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose (“slow”) RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the “fast” RL algorithm on the current (previously unseen) MDP. We evaluate RL^2 experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL^2 is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL^2 on a vision-based navigation task and show that it scales up to high-dimensional problems.

TL;DR: We propose to learn a “fast” reinforcement learning algorithm using standard, off-the-shelf (“slow”) reinforcement learning algorithms, where the “fast” version is represented as an RNN, and fast RL happens inside its activations.

Review discussion:
Although the tasks are distinctly flavored (bandit and other classic regret minimization tasks), isn't the algorithm (learning an RNN to solve POMDPs through a standard RL method) similar to e.g. (https://arxiv.org/pdf/1602.01783.pdf [LSTM layer trained with A3C]), https://arxiv.org/pdf/1512.04455.pdf [RNN trained with variant of DPG, with multiple episode trials], [https://arxiv.org/abs/1507.06527 Recurrent DQN].
Although these methods focus on different tasks, is there an underlying difference in the algorithmic approaches (obviously TRPO is being used here for the RL method)?
Dear reviewer,
Thank you for your question! We apologize for the late response due to NIPS.
You are right in pointing out that the underlying algorithms used in these papers are very similar, since we formulate learning a fast RL algorithm as solving a POMDP, and hence any standard (“slow”) RL algorithm can be applied (A3C / TRPO / Recurrent DQN) to train this fast RL algorithm. In this paper, we focus on the general formulation rather than proposing solutions to specific tasks.
Although previous literature on training RNN policies have used tasks that require memory to test if long-term dependency can be learned (such as the Labyrinth experiment in the A3C paper, and the water maze experiment in the RDPG paper), they focus on the memory aspect of those tasks instead of the fast RL aspect. We have included a discussion about these relevant prior works in the updated version of the paper.
In addition, as we noted in the conclusion, algorithms and policy structures, other than just a plain RNN/LSTM/GRU, that exploit the specific episodic structure of the POMDP may be a promising direction for future work.
Dear authors,
If the intuition being the paper seems to be very interesting, the paper has a lot of missing information making it quite difficult to understand:
First, (not a question): the problem you want to solve is never clearly defined, making the goal of this work very hard to understand.
Second, since the paper is mainly an experimental paper here are some questions concerning the experimental part:
In the MAB problem, you compare the average performance of the methods over a set of uniformly sampled MAB problems.
But, typically, UCB algorithms are designed to perform well in the worst case (upper confidence bound), so could you please give details about the quality of the different models on different problems i.e simple or complex problems. Having an idea about the distribution of the performance over the different problems would be interesting.
Moreover, it is not clear how the hyper-parameters of the baselines are chosen. At last, some other approaches have been developed with similar experimental setups (but different learning algorithms (see end of the comment)
Concerning the tabular MDP setup, are you using a discounted a cumulated reward over each episode ? The observation is the state index, right ?
For the maze experiment: you do not provide any comparison with other learning techniques. Why ? Could you please add a legend over figure 5 ? we don't know the meaning of each curve.
Could you please also comment about some missing references that have already explored this type of problems (with different learning techniques) : Learning Exploration/Exploitation Strategies for Single Trajectory Reinforcement Learning. --  M Castronovo, F Maes, R Fonteneau, D Ernst, or
Automatic discovery of ranking formulas for playing with multi-armed bandits
F Maes, L Wehenkel, D Ernst
European Workshop on Reinforcement Learning, 5-17
Dear reviewer,
Thank you for your questions! We apologize for the late response due to NIPS.
- As mentioned in Section 2.2, the goal of this paper is to learn an RL algorithm that “maximize(s) the expected total discounted reward accumulated during a single trial rather than a single episode.” Our formulation is quite general and hence there is not a specific class of RL problems that we focus on.
Rather we select some representative class of problems to demonstrate the flexibility of our approach.
- (Discussion about UCB) This is a very good suggestion. In the updated version of the paper, we have included a discussion about the performance of different models (e.g. UCB / Gittins / RL^2) on simple / difficult bandit problems in the appendix.
- We perform a hyperparameter search for each of the baseline algorithms. The details are now described in the appendix.
- Re similar approaches, and the listed missing references: we apologize for having overlooked this line of literature. We have included these references in the related work section in the updated version of the paper. The two suggested references have investigated meta-learning an algorithm for solving multi-armed bandits using program search, which is similar in spirit in that they also formulate the problem under a given distribution over MDPs.
However, the proposed techniques require manually designing the primitives used to compose the programs, which needs to be tuned for each specific distribution over MDPs.
In comparison, our approach allows entirely end-to-end training without requiring such domain knowledge.
- For the tabular MDP setup (and for all other problems), the evaluation is with a discount of 1, and we only use a discount of 0.99 in RL^2 for variance reduction, which is applied over the entire trial that spans over multiple episodes. The observation is the state index, which is specified in the appendix.
- For the maze experiment, the classic methods do not apply since we are dealing with a very high-dimensional problem. The recently developed deep RL algorithms, on the other hand, would take orders of magnitudes more episodes to solve each problem.
Hence such comparison would not be very meaningful in analyzing how well the algorithm is performing. As described in the main text, an implicit (optimal) baseline algorithm in this experiment is a strategy that explores around the maze in an efficient manner in the first episode, and after finding the location of the target, returns to it quickly in subsequent episodes. As shown in Figure 6, the RL^2 agent does not learn to perform optimally for this task yet, and hence there is certainly room for improvement that may require new algorithms / architectures to be proposed.
- In Figure 5, the different curves denote different random seeds used to initialize the neural network weights. All other experiment settings and hyperparameters are kept the same. We have made this more explicit in the caption of Figure 5.
The paper proposes to use RL methods on sequences of episodes instead of single episodes. The underlying idea is the problem of 'learning to learn', and the experimental protocol proposed here allows one to understand how a neural network-based RL model can keep memory of past episodes in order to improve its ability to solve a particular problem. Experiments are made on bandit problems, but also on maze problems and show the interesting properties of such an approach, particularly on the maze problem where the agent seems to learn to first explore the maze, and then to exploit its knowledge to quickly find the goal.
The paper is based on a very simple and natural idea which is acutally a good point. I really like the idea, and also the experiment on the maze which is very interesting. Experiments on bandits problem are less interesting since meta-learning models have been already proposed in the bandit problem with interesting results and the proposed model does not really bring additionnal information.  My main concerns is  based on the fact that the paper never clearly formally defines the problem that it attempts to solve.
So, between the intuitive idea and the experimental results, the reader does not understand what  exactly the learning problem is, what is its impact and/or to which concrete application it belongs to. From my point of view, the article clearly lacks of maturity and does not bring yet a strong contribution to the field.
Good:
* Interesting experimental setting
* Simple and natural idea
* Nice maze experiments and model behaviour
Bad:
* No real problem defined, only an intuition is given. Is it really useful ? For which problems ? What is the performance criterion one wants to optimize ? ...
* Bandit experiments do not really bring relevant informations
Dear reviewer,
Thank you for your efforts reviewing the paper and your valuable input! We would like to make a few clarifications to fully incorporate your suggestions to improve our manuscript.
On the value of the bandit experiments: can you clarify what prior work is alluded to wrt to interesting results obtained in the past? We hope to attribute prior work as much as possible. In our opinion, the purpose of the bandit experiment, along with the tabular MDP experiment, is to show that RL^2 can perform as well as (almost) optimal algorithms.
Even if the experiment itself has been performed in the past using different approaches, it is important to verify that our proposed approach can lead to good performance on these benchmark tasks, before moving on to more challenging tasks such as the random maze experiment, where a fair comparison with other RL algorithms is much more challenging to set up.
On a formal definition of the problem: thank you for raising this concern! We actually debated over whether to include a mathematical formulation in the paper. We included it in an earlier draft (snippet available here: https://goo.gl/TOqz6Z) right after giving the MDP notations, but decided that the notation was too dense that it may actually obscure the simplicity of our proposed approach.
However there is certainly value in being precise, and we will add it back (after giving some intuition first) in the updated version of the paper.
The problem of maximising total discounted reward across multiple trials of an unknown MDP (but sampled from a known distribution, e.g. multi-arm bandit) can be formulated as a POMDP problem. A single episode of the POMDP consists of multiple episodes of interaction with the underlying MDP during which the agent should efficiently explore and integrate information about the MDP to minimize reward across the whole trial.
Using this observation (which is not original to this work), the authors compare using an existing RL algorithm (TRPO) to solve POMDPs against classic regret minimization methods on two classic tasks: multi-armed bandits and tabular MDPs.
Additionally, they also demonstrate their approach can scale to a visual navigation task.
While the comparison with classic regret minimization problems is useful, this paper has several weaknesses. It seems very confusing to introduce a new name (RL^2) for an existing class of algorithms (essentially any RL method for solving POMDPs). This terminology and the paper structure obscures to relationship between this and prior work.
The comparison with prior work training RNNs is, while improved from the previous version, still lacking. The distinction the authors make, that prior work “focussed on memory aspect instead of fast RL”, seems somewhat arbitrary. The visual navigation task is conceptually identical to the water maze experiment [Heess et al, 2015] or Labyrinth navigation [Mnih et al, 2016]. These prior tasks require more than just memory, the also requires meta-learning such as exploration and demonstrate “fast RL” with the agent able to improve dramatically after a single episode.
The introduction mentions the need for the use of priors on the environment to create agents which learn quickly and suggests that prior work in DeepRL is data inefficient.
Yet, the recently prior work (e.g. previous paragraph) focussed on POMDPs demonstrated one-shot learning once trained (in the “fast RL” task to use the author’s terminology).
Although the discussion highlights the potential for new algorithms and architectures which are structured to improve performance at these “multi-episode” tasks, no new algorithms or architectures are introduced.
Unfortunately, because of the limited contribution, poor comparisons with prior work and confusing terminology this paper is not suitable for ICLR without substantial revision.
Dear reviewer,
Thank you for your efforts reviewing the paper and your valuable inputs! We would like to make a few clarifications to fully incorporate your suggestions to improve our manuscript.
On the naming “RL^2”: we would like to clarify that this name, or the proposed method, does not refer to any particular RL algorithm or a class of RL algorithms.
Rather, it stands for the general reduction of learning an RL algorithm against a distribution of environments as an RL problem itself. We will clarify more on the naming in the updated manuscript.
On the relation to prior work: we apologize for the ambiguous claim in our last response. You are right that the environment setup in Labyrinth is essentially the same as the random maze environment (we will clarify this). By saying that the prior work focus more on the memory aspect, we really meant considering the entire work (A3C / RDPG) rather than the specific tasks used. What we propose is a general framework for reducing learning an RL algorithm as an RL task itself.
Although prior work have used similar tasks -- and when they do, they certainly exhibit one-shot (or more appropriately, few-shot) learning -- they do not recognize this general structure, and solve the specific task instances in isolation.
In comparison, we consider this general reduction the key contribution of this work, rather than proposing specific RL algorithms or architectures.
The authors try to address the issue of data efficiency in deep reinforcement learning by meta-learning a reinforcement learning algorithm using a hand-designed reinforcement learning algorithm (TRPO in this case). The experiments suggest comparable performance to models with prior knowledge of the distribution over environments for bandit tasks, and experiments on random maze navigation from vision is shown as well, though the random maze experiments would benefit from a clearer explanation. It was not obvious from the text how their experiments supported the thesis of the paper that the learned RL algorithm was effectively performing one-shot learning. The subject of the paper is also strikingly similar to the recently-posted paper Learning to Reinforcement Learn (https://arxiv.org/pdf/1611.05763.pdf), and while this paper was posted after the ICLR deadline, the authors should probably update the text to reflect the state of this rapidly-advancing field.
Dear reviewer,
Thank you for your efforts reviewing the paper and your valuable inputs! We would like to make a few clarifications to fully incorporate your suggestions to improve our manuscript.
On a clearer explanation for the random maze experiments, can you offer more detail on how we can improve the description of the setup?
On how the learned RL algorithm is performing one-shot learning: we never claimed this, and the only mentioning of one-shot learning is in the Related Work section. We do argue that the policy, once trained, effectively acts as an RL algorithm, and this is supported by the empirical comparisons of the performance of the policy to reference algorithms on the bandit and tabular MDP tasks. On the maze task, the algorithm learns to solve the maze in only a few trials, and some may consider such behavior as few-shot learning rather than one-shot learning.
On the recent work Learning to Reinforcement Learn: we will include this work, along with a few other relevant papers (e.g. https://openreview.net/forum?id=SJMGPrcle, https://openreview.net/forum?id=SJ6yPD5xg) in the updated manuscript.

Meta-review:
role: PC
data: comment: This paper fits into the mold of a lot of recent work on "learning to learn". The idea here is to use an RNN to solve tasks in an MDP, and then use a higher level RL algorithm (TRPO) to learn the parameters of this RNN. It's an interesting idea, and one that I think fits in well to a lot of current work on learning methods for optimizing policies.
 
 However, there are a few drawbacks to this paper. First, I believe that the paper is substantially lacking from the point of view of clarity, an assertion that I think is backed up by the three reviewers. Most of the methods are never formally defined, and only a high level description of the actual algorithm is provided. This was particularly confusing to me when reading through the paper, because the entire concept of the paper is suggesting that the authors are learning an RL-based policy. But it's really unclear to me why the "fast" RL policy should really be considered RL at all: it is just a black box policy modelled by an RNN, that the authors then argue exhibits similar behavior as RL algorithms when presented a new domain. This is perhaps a semantic point (maybe we should judge "RL" as virtually any algorithm that can improve over time when run in a domain), but this seems to be an odd distinction, because it certainly seems like the authors are just using TRPO to learn a policy (that happens to be in the form of an RNN). And given that plenty of existing work uses RNNs as policies (and then use an RL algorithm to tune these policies), it's unclear to me how this work differentiates itself, and why it necessitates some new conceptual approach, that specifies the policy itself as an RL algorithm (even if it acts as such in the simple bandit domain presented, for example, in the maze task it seems quite odd to cast this as anything but a policy learned for this domain).
 
 Pros:
 + Potentially nice contribution to the "learning to learn" field
 + Very nice experiments on the maze domain, highlighting a complex task
 
 Cons:
 - Algorithmic presentation is unclear
 - Difficult to see why this method necessitates some new conceptual framework like "RL^2": the authors seem to just be learning an RNN-based policy

Decision: Reject