Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper provides an analysis of RNN-based meta learning approaches. In particular, it investigates the strategies learned via meta-learning, contrasting strategies involving task-dependent learning vs heuristic or hard-coded solutions. Empirical evidence in two sets of experiments, on a 2-armed bandit toy task and a grid-world navigation task, show that hard-coded strategies can be a function of training task distribution and task complexity as well as task horizon. 

While the experiments are simplistic, they provide a clear and thorough comparison of different agent behaviours across different training regimes. I enjoyed reading the paper and although the results are intuitive and unsurprising, they nicely emphasize the importance of environment and task design choices in strategies learned via memory-based meta-learning.

Comments/questions:
1. I would encourage the authors to use “memory-based/RNN-based meta-learning” instead of “meta-learning” to avoid confusion as these results might not apply more widely across different meta-learning approaches (e.g. gradient based).
2. While the pattern of behaviour looks qualitatively similar across analytical and empirical results reported in Figure 1, I wonder if it would be possible to quantitatively assess where they differ.
3. It would be nice to see error bars for the 5 independent training runs in Figure 5.
4. In the Appendix A.2.3 it is mentioned that the discount factor is annealed from 0.8 to 1. within the first 800k episodes. Could you please expand if this was crucial to achieve your results or just an experimental choice?
 
Overall, I think this is an interesting contribution of perhaps limited scope but still valuable and could encourage interesting future research directions in memory-based meta-learning.

review_title: Official Blind Review AnonReviewer2

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary.** The authors investigate the question of when the optimal behavior for an agent is to learn from experience versus when the optimal behavior is to apply the same (memorized) policy in every scenario. They begin by introducing a simple bandits environment wherein they derive the optimal policy and identify regimes in which it involves memorization vs. learning. Then they train an RL^2 agent and verify that it behaves as expected in these regimes. Next, they expand their approach to a slightly more complicated gridworld environment which does not have an analytic solution to the question. The agent behaves as expected in the gridworld environment.

**Strong points.** This paper tackles a novel question which is fundamental to the field of metalearning. By carefully analyzing the two regimes of learning and memorization in the context of metalearning, this paper will increase awareness about the fact that the two regimes exist. The paper is clearly written and does an excellent job of putting experiments in the context of past ML research. The experimental setup is simple but goes straight to the heart of the issue. Figures and text do a good job of analyzing results and communicating them to the reader. Overall, this paper was an interesting read.

**Weak points.** The idea that “sometimes memorization is best and other times learning is best” does border on the obvious. Indeed, as soon as the authors derive their analytical solution, it becomes clear that we can expect the RL^2 agent to learn the same behavior. For me, there were no surprises in the experimental sections. To the authors: was there anything that was surprising or not obvious to you? What additional information can the experiments tell us, apart from confirming theoretical predictions?

Having said that, I also believe that very simple, well-executed research ideas sometimes make the best papers. This paper appears to be one of those cases. And even though the ideas are simple, they are significant and they are not a major part of the dialogue in the meta-learning community yet. So even if the ideas seem obvious, I think there is value in communicating them well.

I have one concern about the bandit task setup: the authors adjust $\sigma_l$, the width of the Gaussian from which they are sampling the reward, as a proxy for aleatoric uncertainty and hence task complexity. In doing so, they essentially equate “stochasticity of the environment” with “task complexity.” And yet, there are many other ways in which a task can be complex. Sometimes, all the information needed to perform a task is present and yet the task is difficult to solve because one needs to interpret/integrate the information in a particular way. This is why, for example, puzzles are considered difficult tasks. It is also why simulating the 3-body problem is a complex task. To the authors: can you clarify what you mean by “task complexity”?

In the closing paragraph of the paper, the authors claim that their approach “allows us to study the emergence of inductive biases in biological systems” but this claim is not supported by the rest of the paper, which makes almost no connections to biological systems. There are certainly ways in which these results are relevant to learning in biological systems, but the authors did not explore them in this paper, and so this claim is not well supported. In the same paragraph, they bring in contrasting notions of Darwinian and Lamarkian inheritance. Since they do this in one sentence -- the last sentence -- it is hard to understand what their claim is. And it was not clear that this was one of the main takeaways of the paper, as these concepts do not appear anywhere else in the paper. If the authors want to draw these conclusions, then they should add additional discussion on these topics. Otherwise, they risk misleading readers.

One additional minor suggestion would be to invert the color scale of Figure 6, as “white -> red” signifies values of increasing size in all preceding plots, but in Figure 6 it currently signifies values of decreasing size.

Minor grammatical suggestions
-- “the question which aspects of behavior“ -> “the question of which aspects of behavior“
-- When typing quotes in LaTex, use `` and ‘’ instead of “” so as to make them open & close correctly
-- “interplay of the agent’s lifetime,” -> “interplay between the agent’s lifetime,”
-- “We numerically show” -> “We show numerically”
-- “as well as explicit models of memory” -> ”and explicit models of memory” (same issue occurs later)
-- “the agents does not have” -> “the agent does not have”

**Recommendation.** 6 : Marginally above acceptance threshold

**Reasoning.** This paper is well written and the experimental setup is simple, well-executed, and produces results that are relevant to the main question of the paper. The main question of the paper -- when does it make more sense to learn vs. memorize a behavior -- is significant to ICLR and to the field of machine learning. There are a number of relatively minor weaknesses (as described above) but this is overall a nice paper and would be a good contribution to ICLR 2021.
review_title: Review of "Learning not to Learn" for ICLR 2021

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper explores the effect of time horizon on meta-reinforcement learning agents. Using a recurrent meta-learner, it demonstrates that different strategies are learned based on the time horizon during meta-training.

Pros:
Creative proposal to study when incorporating new data to adapt to a task is warranted versus executing an existing behavior.
Results are very nicely presented and writing is good
Cons:
The analogy to “nature versus nurture” seems tenuous
Results seem somewhat obvious (see comments below)

Detailed Comments:

While I appreciate the inspiration of the biological connections argued in this work, I am not convinced that the analogy of learning versus adapting in meta-learning maps neatly onto the concepts of nature versus nurture in biology. If biology terminology is going to be used, it needs to be carefully defined for a machine learning audience, and the limitations of the analogy discussed. Presently, I find that the use of this terminology subtracts from the clarity of the work.

My main criticism of the work is that the results seem fairly obvious. In the finite horizon case, the meta-learner will necessarily learn a strategy optimal for the given horizon because that’s exactly what it’s optimized for. It seems akin to me to training an RNN policy in an MDP and then noticing that the RNN is capable of *not* persisting information across timesteps since it isn’t needed. Since feedforward models are a subset of recurrent models, this seems obvious.

It also seems strange to me that the agent is only tested in 1 episode. It seems like if the agent has learned a new skill, it should be able to repeat that skill if given the chance (e.g., with a reset), and that that is a hallmark of meta-learning. What is happening in this paper seems more like “horizon-dependent RL.” I don’t really see how the behavior in the long-time horizon setting can be called “learning” while the short time-horizon behavior is not. Both seem like the optimal policy for the given MDP. In Section 4, for example, wouldn’t “learning” be defined as figuring out where the colored squares are? Yet in the paper it seems to be defined as executing a policy that moves farther away from the start. It seems to me like exploration and learning are being conflated here. 

Several times throughout the paper it is claimed that the work “investigates the interplay of  three considerations when designing meta-task distributions: The diversity of the task distribution, task complexity and training lifetime.” I don’t see how the first two are analyzed in this paper. Perhaps in the bandit example, but there it seems to be always entangled with the training lifetime.

Besides the comments about the biological terminology mentioned earlier, the paper is clear and easy to read, with the other exception of the description of the 2-arm Gaussian bandit in Section 3. This reviewer had to read that first paragraph about 5 times to understand the setup. The figures are well-done and clearly present the results.

Recommendation:
5. While experiments are thorough and nicely presented, I presently don’t see what insights are gained from the analysis. The claim of analyzing “diversity of the task distribution, task complexity and training lifetime” seems like an over-claim to me. It’s not clear to me that the agent “learns” a new skill, since there is only exploration, and no exploitation.

----Update----
After reading all the other reviews and ensuing discussions, I maintain my original score. If the research question is "How does the optimal policy depend on task parameters such as uncertainty and horizon?" I believe Bayes-adaptive work answers that question. If the question is "How do policies learned by meta-RL algorithms compare to Bayes-optimal policies?" then I think more empiricism is needed (since RL2 in principle can represent the Bayes-optimal policy), or a comparison of multiple methods.

review_title: Creative proposal; analysis seems flawed

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper observes that in meta-RL (and evolutionary biology), sometimes it is advantageous to learn behaviors that adapt to the particular task, while other times not adapting to the task, and instead relying on a task-agnostic “hard-coded” behavior is sufficient. While much meta-RL research typically focuses on the former setting, this paper studies when it is not necessary to learn adaptive behaviors. Specifically, this paper presents three main findings: (i) whether or not it is optimal to learn adaptive behavior strongly depends on the horizon of the task and complexity of learning such adaptive behaviors — if the horizon is too short, or if the adaptive behavior requires complex exploration, then exploring the new task to learn adaptive behaviors may not be worth it; (ii) existing meta-RL agents are capable of choosing not to learn adaptive behaviors, when it is optimal to do so; (iii) existing meta-RL agents generalize poorly to tasks with varying horizon-lengths.

Strengths:
- Novelty. Most meta-RL papers study the case where adapting to a new task requires learning new behaviors. Instead, this paper studies when such adaptation is not necessary, which is an area that has not been well-studied. Therefore, it is interesting and novel to bring attention to the fact that both regimes exist.
- Clarity and Execution. This paper is generally well-executed and clear. The bandit example in Section 3 clearly illustrates the two regimes that this paper studies: when it is necessary to learn adaptive behaviors vs. not, and the results are presented in figures that impressively, clearly illustrate the points made in the text. This section also convincingly shows that existing meta-RL agents roughly learn the Bayes-optimal policy in this case. Similarly, the grid world tasks in Section 4 also clearly illustrate how the behavior of meta-RL agents changes with the horizon of the task. The paper does a good job at thoroughly studying what happens when each parameter (e.g., horizon-length, aleatoric uncertainty, and epistemic uncertainty) varies.

Weaknesses:
- Significance. My primary concern with this work is significance. I believe that the main insights of this paper can be interpreted through the lens of Bayes-adaptive policies [1]. The optimal Bayes-adaptive policy explains when it is optimal to explore a new task and learn adaptive behaviors, depending on the horizon length and amount of exploration needed, and therefore, the main claim of the paper can be framed as observing that existing meta-RL agents can learn the Bayes-adaptive optimal policy in simple tasks. This is an interesting observation, but prior work [2] already shows that meta-RL is equivalent to learning the Bayes-adaptive optimal policy. Furthermore, for harder tasks, this is likely to be confounded with the complexity of learning efficient exploration behaviors that allow learning adaptive behaviors, as shown by prior work [2, 3, 4]. Overall, it is therefore unclear to me what significant takeaways the meta-RL community can gain from this work. One potential takeaway is the observation that meta-RL agents generalize poorly to tasks with varying horizon-lengths, but it’s unclear whether this setting occurs in real tasks, and even if this _is_ necessary, then simply adding the horizon to the observation (state) and varying the horizon during meta-training seems sufficient.

Overall, I found this paper to be a well-written, well-executed, illustration of when it is optimal to learn task-specific adaptive behaviors. However, due to its limited significance, I am unfortunately unable to recommend acceptance.

Additional Questions / Comments:
- In Figure 2, what does it mean with “100 Suboptimal pulls?” Is this over the course of all training? Also, why is the initial policy entropy at 0? I would expect that upon initialization, the policy entropy is not 0.
- I believe that the Bayes-adaptive optimal policy can also be analytically computed for the grid world tasks. It would be nice to report the optimal returns in Figure 5.
- The interplay between the term “lifetime” and episodes was not explicitly defined. It was possible to infer the paper’s intent from reading the experiments, but more carefully defining this could help.
- Framing the exploration trade-off in terms of evolutionary biology is interesting.

References:

[1] Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. Michael O’Gordon Duff. February 2002. https://www.gatsby.ucl.ac.uk/~yael/Okinawa/DuffThesis.pdf

[2] VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, Shimon Whiteson. October 2019. https://arxiv.org/abs/1910.08348

[3] Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables. Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, Sergey Levine. March 2019. https://arxiv.org/abs/1903.08254

[4] Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning. Evan Zheran Liu, Aditi Raghunathan, Percy Liang, Chelsea Finn. June 2020. https://openreview.net/forum?id=La1QuucFt8-

------------

**Edit: Score raised from 4 --> 5 following discussion below.**
review_title: Review

Meta-review: There was fairly detailed discussion among three of the four reviewers. The fundamental concern of the reviewers is regarding the contribution of the paper. During the rebuttal, the authors clarified the following:

> while the effects of varying uncertainty / horizon lengths is well-understood for Bayes-optimal policies, it is not understood for existing meta-RL approaches, which is the topic of this paper

That is, the contribution of the paper is to understand the effects of varying uncertainty/horizon lengths for meta-RL approaches. However, it is known in prior work that meta-RL algorithms such as RL^2 can implement Bayes-optimal policies in principle. As a result, it's not clear whether this contribution is significant relative to prior knowledge, and this paper does not seem to bring any new insights.

An alternative framing of the paper would be to consider the question of how meta-RL solutions compare to Bayes-adaptive optimal policies. While this framing would be interesting and novel, the current version of the paper does not sufficiently answer this question, since the only experiments include RL^2 (and such a study would require experimenting with more sophisticated meta-RL algorithms beyond RL^2).

As such, this paper isn't suitable for publication at ICLR in its current form.
Decision: Reject