Venue: ICLR 2017 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The problem of maximising total discounted reward across multiple trials of an unknown MDP (but sampled from a known distribution, e.g. multi-arm bandit) can be formulated as a POMDP problem. A single episode of the POMDP consists of multiple episodes of interaction with the underlying MDP during which the agent should efficiently explore and integrate information about the MDP to minimize reward across the whole trial.

Using this observation (which is not original to this work), the authors compare using an existing RL algorithm (TRPO) to solve POMDPs against classic regret minimization methods on two classic tasks: multi-armed bandits and tabular MDPs. Additionally, they also demonstrate their approach can scale to a visual navigation task.

While the comparison with classic regret minimization problems is useful, this paper has several weaknesses. It seems very confusing to introduce a new name (RL^2) for an existing class of algorithms (essentially any RL method for solving POMDPs). This terminology and the paper structure obscures to relationship between this and prior work.

The comparison with prior work training RNNs is, while improved from the previous version, still lacking. The distinction the authors make, that prior work “focussed on memory aspect instead of fast RL”, seems somewhat arbitrary. The visual navigation task is conceptually identical to the water maze experiment [Heess et al, 2015] or Labyrinth navigation [Mnih et al, 2016]. These prior tasks require more than just memory, the also requires meta-learning such as exploration and demonstrate “fast RL” with the agent able to improve dramatically after a single episode.

The introduction mentions the need for the use of priors on the environment to create agents which learn quickly and suggests that prior work in DeepRL is data inefficient. Yet, the recently prior work (e.g. previous paragraph) focussed on POMDPs demonstrated one-shot learning once trained (in the “fast RL” task to use the author’s terminology).

Although the discussion highlights the potential for new algorithms and architectures which are structured to improve performance at these “multi-episode” tasks, no new algorithms or architectures are introduced.

Unfortunately, because of the limited contribution, poor comparisons with prior work and confusing terminology this paper is not suitable for ICLR without substantial revision.


review_title: Interesting comparisons, but poor comparisons with prior work

final_score: rating: null
review_content: Although the tasks are distinctly flavored (bandit and other classic regret minimization tasks), isn't the algorithm (learning an RNN to solve POMDPs through a standard RL method) similar to e.g. (https://arxiv.org/pdf/1602.01783.pdf [LSTM layer trained with A3C]), https://arxiv.org/pdf/1512.04455.pdf [RNN trained with variant of DPG, with multiple episode trials], [https://arxiv.org/abs/1507.06527 Recurrent DQN]. Although these methods focus on different tasks, is there an underlying difference in the algorithmic approaches (obviously TRPO is being used here for the RL method)?
review_title: Relation to other RL methods that train RNNs

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors try to address the issue of data efficiency in deep reinforcement learning by meta-learning a reinforcement learning algorithm using a hand-designed reinforcement learning algorithm (TRPO in this case). The experiments suggest comparable performance to models with prior knowledge of the distribution over environments for bandit tasks, and experiments on random maze navigation from vision is shown as well, though the random maze experiments would benefit from a clearer explanation. It was not obvious from the text how their experiments supported the thesis of the paper that the learned RL algorithm was effectively performing one-shot learning. The subject of the paper is also strikingly similar to the recently-posted paper Learning to Reinforcement Learn (https://arxiv.org/pdf/1611.05763.pdf), and while this paper was posted after the ICLR deadline, the authors should probably update the text to reflect the state of this rapidly-advancing field.
review_title: Another contribution to the learning-to-learn field, with interesting but not spectacular experimental results

final_score: rating: 3: Clear rejection
review_content: The paper proposes to use RL methods on sequences of episodes instead of single episodes. The underlying idea is the problem of 'learning to learn', and the experimental protocol proposed here allows one to understand how a neural network-based RL model can keep memory of past episodes in order to improve its ability to solve a particular problem. Experiments are made on bandit problems, but also on maze problems and show the interesting properties of such an approach, particularly on the maze problem where the agent seems to learn to first explore the maze, and then to exploit its knowledge to quickly find the goal. 

The paper is based on a very simple and natural idea which is acutally a good point. I really like the idea, and also the experiment on the maze which is very interesting. Experiments on bandits problem are less interesting since meta-learning models have been already proposed in the bandit problem with interesting results and the proposed model does not really bring additionnal information.  My main concerns is  based on the fact that the paper never clearly formally defines the problem that it attempts to solve. So, between the intuitive idea and the experimental results, the reader does not understand what  exactly the learning problem is, what is its impact and/or to which concrete application it belongs to. From my point of view, the article clearly lacks of maturity and does not bring yet a strong contribution to the field. 

Good:
* Interesting experimental setting
* Simple and natural idea
* Nice maze experiments and model behaviour

Bad:
* No real problem defined, only an intuition is given. Is it really useful ? For which problems ? What is the performance criterion one wants to optimize ? ...
* Bandit experiments do not really bring relevant informations

review_title: Review

final_score: rating: null
review_content: 
Dear authors,

 If the intuition being the paper seems to be very interesting, the paper has a lot of missing information making it quite difficult to understand:

First, (not a question): the problem you want to solve is never clearly defined, making the goal of this work very hard to understand.

Second, since the paper is mainly an experimental paper here are some questions concerning the experimental part:

In the MAB problem, you compare the average performance of the methods over a set of uniformly sampled MAB problems. But, typically, UCB algorithms are designed to perform well in the worst case (upper confidence bound), so could you please give details about the quality of the different models on different problems i.e simple or complex problems. Having an idea about the distribution of the performance over the different problems would be interesting. Moreover, it is not clear how the hyper-parameters of the baselines are chosen. At last, some other approaches have been developed with similar experimental setups (but different learning algorithms (see end of the comment)  

Concerning the tabular MDP setup, are you using a discounted a cumulated reward over each episode ? The observation is the state index, right ? 

For the maze experiment: you do not provide any comparison with other learning techniques. Why ? Could you please add a legend over figure 5 ? we don't know the meaning of each curve. 

Could you please also comment about some missing references that have already explored this type of problems (with different learning techniques) : Learning Exploration/Exploitation Strategies for Single Trajectory Reinforcement Learning. --  M Castronovo, F Maes, R Fonteneau, D Ernst, or 
Automatic discovery of ranking formulas for playing with multi-armed bandits
F Maes, L Wehenkel, D Ernst
European Workshop on Reinforcement Learning, 5-17

review_title: pre-review questions

Meta-review: This paper fits into the mold of a lot of recent work on "learning to learn". The idea here is to use an RNN to solve tasks in an MDP, and then use a higher level RL algorithm (TRPO) to learn the parameters of this RNN. It's an interesting idea, and one that I think fits in well to a lot of current work on learning methods for optimizing policies.
 
 However, there are a few drawbacks to this paper. First, I believe that the paper is substantially lacking from the point of view of clarity, an assertion that I think is backed up by the three reviewers. Most of the methods are never formally defined, and only a high level description of the actual algorithm is provided. This was particularly confusing to me when reading through the paper, because the entire concept of the paper is suggesting that the authors are learning an RL-based policy. But it's really unclear to me why the "fast" RL policy should really be considered RL at all: it is just a black box policy modelled by an RNN, that the authors then argue exhibits similar behavior as RL algorithms when presented a new domain. This is perhaps a semantic point (maybe we should judge "RL" as virtually any algorithm that can improve over time when run in a domain), but this seems to be an odd distinction, because it certainly seems like the authors are just using TRPO to learn a policy (that happens to be in the form of an RNN). And given that plenty of existing work uses RNNs as policies (and then use an RL algorithm to tune these policies), it's unclear to me how this work differentiates itself, and why it necessitates some new conceptual approach, that specifies the policy itself as an RL algorithm (even if it acts as such in the simple bandit domain presented, for example, in the maze task it seems quite odd to cast this as anything but a policy learned for this domain).
 
 Pros:
 + Potentially nice contribution to the "learning to learn" field
 + Very nice experiments on the maze domain, highlighting a complex task
 
 Cons:
 - Algorithmic presentation is unclear
 - Difficult to see why this method necessitates some new conceptual framework like "RL^2": the authors seem to just be learning an RNN-based policy
Decision: Reject