Learning not to learn: Nature versus nurture in silico
Authors: Robert Tjarko Lange
Henning Sprekeler
Venue:  (2021)
Keywords: Meta-Learning
Reinforcement Learning

Abstract: Animals are equipped with a rich innate repertoire of sensory, behavioral and motor skills, which allows them to interact with the world immediately after birth. At the same time, many behaviors are highly adaptive and can be tailored to specific environments by means of learning. In this work, we use mathematical analysis and the framework of meta-learning (or 'learning to learn') to answer when it is beneficial to learn such an adaptive strategy and when to hard-code a heuristic behavior. We find that the interplay of ecological uncertainty, task complexity and the agents' lifetime has crucial effects on the meta-learned amortized Bayesian inference performed by an agent. There exist two regimes: One in which meta-learning yields a learning algorithm that implements task-dependent information-integration and a second regime in which meta-learning imprints a heuristic or 'hard-coded' behavior. Further analysis reveals that non-adaptive behaviors are not only optimal for aspects of the environment that are stable across individuals, but also in situations where an adaptation to the environment would in fact be highly beneficial, but could not be done quickly enough to be exploited within the remaining lifetime. Hard-coded behaviors should hence not only be those that always work, but also those that are too complex to be learned within a reasonable time frame.

Under review as a conference paper at ICLR 2021
LEARNING NOT TO LEARN: NATURE VERSUS NURTURE IN SILICO
Anonymous authors Paper under double-blind review
ABSTRACT
Animals are equipped with a rich innate repertoire of sensory, behavioral and motor skills, which allows them to interact with the world immediately after birth. At the same time, many behaviors are highly adaptive and can be tailored to specific environments by means of learning. and the framework of memory-based meta-learning (or "learning to learn') to answer when it is beneficial to learn such an adaptive strategy and when to hard- code a heuristic behavior. We find that the interplay of ecological uncertainty, task complexity and the agents' lifetime has crucial effects on the meta-learned amortized Bayesian inference performed by an agent. There exist two regimes: One in which meta- leat-lennning yields a learning algorithm that implements task-dependent information-integaatio and second regime in which meta-learning imprints a heuristic or 'hard-coded" behavior. Further analysis reveals that non-adaptive behaviors are not only optimal for aspects of the environment that are stable across individuals, but also in situations where an adaptation to the environment would in fact highly beneficial, but could not be done quickly enough to be exploited within the remaining lifetime. Hard-coded behaviors should hence not only be those that always work, but also those that are too complex to be learned within a reasonable time frame.
this work, we use mathematical analysis
INTRODUCTION
The 'nature versus nurture" debate (e.g., Mutti et al., 1996 Tabery 2014) aspects of behavior are 'hard-coded' by evolution, and which are learned from experience is one of the oldest and most controversial debates in biology. Evolutionary principles prescribe that hard-coded behavioral routines should be those for which there is no benefit in adaptation. This is believed to be the case for behaviors whose evolutionary advantage varies little among individuals of a species. Mating instincts or flight reflexes are general solutions that rarely present an evolutionary disadvantage. On the other hand, features of the environment that vary substantially for individuals of a species potentially ask for adaptive behavior (Buss) 2015). Naturally, the same principles should not only apply to biological but also artificial agents. But how can reinforcement learning agent differentiate between these two behavioral regimes?
the question of which
promising approach to automatically learn rules of adaptation that facilitate environment-specife specialization is meta-learning (Schmidhuber) 1987: Thrun & Pratt] 1998). At its core lies the idea of using generic optimization methods learn inductive biases for given ensemble of tasks. In this approach, the inductive bias usually has its own set of parameters (e.g., weights in a recurrent network; Hochreiter et al.) 200T) that are optimized on the whole task ensemble, that is, on long, "vvolutionary" time scale. These parameters in turn control how different set of parameters (e.g.. activities in the network) are updated on a much faster time scale. These rapidly adapting parameters then allow the system adapt to specific task at hand. Notably, the parameters of the system that are subject to "nature" - i.c., those that shape the inductive bias and are common across tasks and those that subject 'nurture' are usually predefined from the start.
In this work, we use the memory-based meta- leta-learning approach for a different goal, namely to acquire qualitative understanding of which aspects of behavior should be hard-coded and which should be adaptive. Our hypothesis is that meta-learning can not only learn efficient learning algorithms, but can also decide not to be adaptive at all, and to instead apply generic heuristic to the whole

---

Under review as a conference paper at ICLR 2021
ensemble of tasks. Phrased in the language of biology, meta-learning can decide whether to hard-code a behavior or to render it adaptive, based on the range of environments the individuals species could encounter.
We study the dependence of the meta-learned algorithm on three central features of the meta- reinforcement learning problem:
Ecological uncertainty: How diverse is the range of tasks the agent could encounter?
Task complexity:
How long does it take to learn the optimal strategy for the task at hand? Note that this could be different from the time it takes execute the optimal strategy.
Expected lifetime: How much time can the agent spend on exploration and exploitation?
Using analytical and numerical analyses, we show that non-adaptive behaviors are optimal in two cases when the optimal policy varies little across the tasks within the task ensemble and when the time it takes to learn the optimal policy is too long to allow a sufficient exploitation of the learned policy.
Our results suggest that not only the design of the meta-task distribution, but also the lifetime of the agent can have strong effects on the meta-learned algorithm of RNN-based agents. In particular, we find highly nonlinear and potentially discontinuous effects of ecological uncertainty, task complexity and lifetime on the optimal algorithm. As a consequence, a meta-learned adaptation strategy that was optimized, e.g., for a given lifetime may not generalize well to other lifetimes. This is essential for research questions that are interested in the conducted adaptation behavior, including curriculum design, safe exploration as well human in toop applications. Our work may provide principled way of examining the constraint- deeendencedee of meta- meta-learned inductive biases.
The remainder of this paper is structured as follows: based meta-reinfooe renttrrriifoormment learning and contrast the related literature. Afterwards, we analyze a Gaussian multi-arm bandit setting. which allows to analytically disentangle the behavioral impact of ecological uncertainty, task complexity and lifetime. Our derivation of the liferimm-ddeendent Bayes optimal exploration reveals highly non-linear interplay of these three factors. We show numerically that memory-basee meta-learning reproduces our theoretical results and can learn not to learn. Furthermore, we extend our analysis more complicated exploration problems. Throughout, we analyze the resulting recurrent dynamics of the network and the representations associated with learning and non-adaptive strategies.
First, we review the background in memory-
2 RELATED WORK & BACKGROUND
Meta-learning 'learning to learn' (e.g.. Schmidhuber 1987 Thrun & Pratt 1998 Hochreiter et al., 2001 Duan al. 2016, Wang et al. 2016 Finn et al.) (017) has been proposed as computational framework for acquiring task distribution sppeciif learning rules. During costly outer loop optimiza- tion, an agent crafts a ssecifie adaptation strategy, which applicable to an engineered task distribution. At inference time, the acquired inner loop learning algorithm is executed for fixed amount of timesteps (lifetime) on a test task. This framework has successfully been applied to a range of applications such as the meta-learning of optimization updates (Andrychowicz al. 2016 Flennerhag et al., 2018 2019), agent (Rabinowitz et al. (2018) and world models (Nagabandi et al. 2018) and explicit models of memory (Santoro et al.) 2016; Bartunov al. 2019). Already, early work by Schmidhuber (198)) suggested an evolutionary perspective on recursively learning the rules of learning. This perspective holds the promise of explaining the emergence of mechanisms underly- ing both natural and artificial behaviors. Furthermore, a similarity between the hidden activations of LSTTM- meta-learners and the recurrent activity of neurons in the prefrontal cortex (Wang et al. 2018 has recently been suggested.
Previous work has shown that LSTM-based meta-learning is capable of distilling a sequential integration algorithm akin to amortized Bayesian inference (Ortega et al. 2019 Rabinowitz 2019 Mikulik et al., 2020). Here we investigate when the integration of information might not be the optimal strategy to meta-learn. We analytically characterize task regime in which not adapting to sensory information is optimal. Furthermore, we study whether SSTM-based meta-learning is capable of inferring when to learn and when execute non-adaptive program. RRabinowiz (2019)
2

---

Under review as a conference paper at ICLR 2021
previously studied the outer loop learning dynamics and found differences across several tasks, the origin of which is however not fully understood. Our work may provide an explanation for these different meta-learning dynamics and the dependence on the task distribution as well as the time horizon of adaptation.
Our work is most closely related to Pardo al. (017)) and Zintgraf et al. 2019 Pardo et al. (2017)
study the impact of fixed time limits and time- awareness on deep reinforcement learning agents. They propose using a timestamp as part of the state representation in order to avoid state-aliasing and the Mon-Marrkvianityy resulting from a finite horizon treatment of an infinite horizon problem. Our setting differs in several aspects. First, we study the case of meta- rentroreifore learning where the agent has to learn within a single lifetime. Second, we focus on a finite horizon perspective with limited adaptation. ZZinggrff et al. (2019), on the other hand, investigate meta reinforcement-leariing for Bayes-adaptive Markov Decision Processes and introduce novel architecture that disentangles task-specific belief representations from policy representations. Similarly to our work, Zintgraf et al. (2119) are interested in using the meta-learning framework to distill Bayes optimal exploration behavior. While their adaptation setup extends over multiple episodes, we focus on single lifetime adaption and analytically analyze when it is beneficial to learn in the first place.
Finally, our work extends upon the efforts of computational ethology (Stephens) [99I] and experimen- tal evolution (Dunlap & Stephens 2009 2016: Marcus et al. 2018). which aims to characterize the conditions under which behavioral plasticity may evolve. Their work shows that both environmental change and the predictability of the environment shape the selection pressure, which evolves adaptive traits. Our work is based on memory- based meta-learning with function approximation and aims to extend these original findings to task distributions for which no analytical solution may be available.
LEARNING NOT TO LEARN
To disentangle the influence of ecological uncertainty, task complexity, and lifetime on the nature of the meta-learned strategy, we first focus a minimal two-arm Gaussian bandit task, which allows for an analytical solution. The agent experiences episodes consisting of T arm pulls, representing the lifetime of the agent. The statistics of the bandit are constant during each episode, but vary between episodes. To keep simple, one of the two arms is deterministic and always returns a reward 0f The task distribution is represented by the variable expected reward of the other arm, which is sampled at the beginning of an episode, from a Gaussian distribution with mean -1 and standard deviation Tp, i.e. μ (,), The standard deviation op controls the uncertainty of the ecological niche. For op < 1, the deterministic arm is almost always the better option. For op >> 1, the chances of either arm being the best in the given episode are largely even. While the mean Jr remains constant for the lifetime T of the agent, the reward obtained in given trial is stochastic and is sampled from second Gaussian, r N(,, oi). This rial-to-trial variability controls how many pulls the agent needs to estimate the mean reward of the stochastic arm. The standard deviation a hence controls how quickly the agent can learn the optimal policy. We therefore use as a proxy for task complexity.
In this simple setting, the optimal meta-learned strategy can be calculated analytically. The optimal exploration strategy is to initially explore the stochastic arm for a given trial number n. Afterwards, chooses the best arm based on its maximum a posteriori-estimate of the remaining episode return. The optimal amount of exploration trials n² can then be derived analytically: []
T Tr m T.o,, = arg maxx[-n + Ep,r [[T - n) x xxxpppp -01]) 0)]
n² arg max E
where j is the estimate of the mean reward of the stochatic arm after the n exploration trials. We find two distinct types of behavior (left-hand side of figure A regime in which learning via exploration is effective and a second regime in which not learning is the optimal behavior. It may be optimal not to learn for two reasons: First, the ecological uncertainty may be so small that it is very unlikely that the stochastic first arm is better. Second, if the trial-to-trial variability is too large relative to the range
Pleasse refer to the supplementary material for detailed derivation of this analytical result as well as the hyperparameters of the numerical experiments.
3

---

Under review as a conference paper at ICLR 2021
Analytical: Optimal Exploration n
Meta-Learned: Conducted Exploration
Lifetime: T= 25
Lifetime: 7= 50
Lifetime: T= 25
Lifetime: 7= 50
Ecological Uncert. Op Ecological Uncert. Op
10.0
25
6.9
3.7
# Exploratioo Trials
0.6
15
Lifetime:
Lifetime:
Lifetime:
Lifetime:
10.0
-10
6.9
0.8
19.0
19.0
19.0
12.7
19.0
Task Complexity
Task Complexity d
Task Complexity d;
Task Complexity
Figure 1: Theory and meta-learned exploration a two-arm Gaussian bandit. Left: Bayes optimal exploration behavior for different lifetimes and across uncertainty conditions 0t, Op- Right: Meta- learned exploration behavior using the RL (Wang al. 2016) framework. There exist two behavioral regimes (learning by exploration and a heuristic non non-explorane noo-exxllraatiee strategy) for both the theoretical result and the numerical meta- leeanee behaviors. The amount of meta- exploration is averaged both over 5 independent training runs and 100 episodes for each of the 400 trained networks.
of potential ecological niches, that it may simply not be possible integrate sufficient information given a limited lifespan. We make two observations:
1. There exists hard nonlinear threshold between learning and not learning behaviors de- scribed by the ratio o; and op. If σ; is too large, the value of exploration (or the reduction in uncertainty) is too small to be profitable within the remaining lifetime of the agent. Instead, it advantageous to hard-code a heuristic choice.
2. The two regimes consistently exist across different lifetimes. As the lifetime grows, the learning regime becomes more and more prevalent. Given sufficient amount of time, learning by exploring the uncertain arm is the best strategy.
Is the common meta-learning framework capable of reproducing these different qualitative behaviors and performing Bayes optimal amortized inference across the entire spectrum of meta-task distri- butions? Or differently put: Can memory based meta- meta-learning yield agents that do not only learn to Iearn but that also learn nor to learn? To answer this question, we train LSTM-based RL² (Wang al. 2016) agents with the standard synchronous actor-critic (Mnih al., 2016) setup the same grid of ecological uncertainties op and "task complexities" dj. The input de to the network at time t consists of the action of the previous timestep, monotonically increasing timestamp within the current episode and crucially the reward of the previous timestep, recurrent weight dynamics of the inner loop can then implement an internal learning algorithm that integrates previous experiences. After collecting set of trajectories, we optimize the weights and initial condition of the hidden state with an outer loop gradient descent update minimize the common actor-critic objective.
{ar-1s 1, é(t), }. The
We obtain the amount of meta-learned exploration by testing the RI.2 agents on hold-out bandits for which we set op 0 and only vary di Thereby, it is ensured that the deterministic arm is the better arm. can then define the number of exploration trials as the pulls from the suboptimal stochastic arm. We observe that meta-learning is capable of yielding agents that behave according to our derived theory of a Bayes optimal agent, which explicitly knows the given lifetime as well as uncertainties op.op (figure l). Importantly, the meta-learned behavior also falls into two regimes: A regime in which the meta-learned strategy resembles a learning algorithm and a regime which the recurrent dynamics encode hard-coded choice of the deterministic Furthermore, the edge between the
4

---

Under review as a conference paper at ICLR 2021
Analytical & Meta-Solution Return & Empirical Behaviour across n
No Learn:
On the Edge:
Learn:
0) 19.5, 0p
0) 9.53, 0p 3.75
0 0,
200
Exp. Return
50
100
50
100
50
100
1.0
1000 Seeds
0.10
Rel. Freq.
0.2
0.5
0.05
0.0
0.0
0.00
50
100
50
100
50
100
n # Stoch. Arm Pulls
n # Stoch. Arm Pulls
n # Stoch. Arm Pulls
Figure 2: Bimodality of the reward landscape. Upper row. The Bayesian model predicts a bimodal dependence of the expected return on the policy. Parameters for lifetime T = 100 in the non-learning (left) and learning regime (right) and close the transition edge (middle). Bottom row. Distribution of the mean number of explorative pulls in 1000 separatively trained network with different random seeds. Close to the edge, the networks fall into two classes: networks either abandon all exploration (peak at 0) or explore and learn. Away from the transition, all 1000 networks adopt similar strategy.
two meta-learned regimes shifts with the agent's lifetime as predicted by the Bayesian theory. As the lifetime increases, wider ecological niches at higher levels of task complexity become solvable and the strategy of learning profitable.
In the Bayesian model, the edge between the two regimes is located at parameter values where the learning strategy and the non-learning strategy perform equally well. Because these two strategies are very distinct, we wondered whether the reward landscape for the memory-based meta-learner has two local maxima corresponding to the two strategies (figure 2 To test this, we trained N = 1000 networks with different initial conditions, for task parameters close to the edge, but in the regime where the theoretically optimal strategy would be to learn. We then evaluated for each network the number of explorative pulls of the stochastic arm, averaged across 100 episodes. The distribution of the number of explorative pulls across the 1000 networks shows i) peak at zero exploration and ti) a broad tail of mean explorative pulls (figure 2 suggesting that there are indeed two classes of networks. One class never pulls the stochastic arm, i.e., those networks adopt a non-learning strategy. The other class learns. For task parameters further away from the edge, this bimodality disappears.
The two behavioral regimes are characterized by distinct recurrent dynamics of the trained LSTM agents. The two left-most columns of figure display the policy entropy and hidden state statisties for a network trained on a 0;, cy-comiination associated with the regime in which learning is the optimal behavior. We differentiate between the case in which the deterministic arm is the better one (μ < 0) and the case in which the second arm should be preferred (μ > 0). In both cases the agent first explores in order to identify the better arm. Moreover, the hidden dynamics appear to display two different attractors, which correspond to either of the arms being the better choice. The better arm can clearly identified from the CCA-diimnsionnalty reduced hidden state dynamics (bottom row of figure 3p. The two right-most columns of figure 3 on the other hand, depict the same statistics for network that was meta- trained on the regime in which the optimal strategy is not to learn. Indeed, the agent always chooses the deterministic arm, regardless of whether it is the better choice. Accordingly, the network dynamics seem to fall into a single attractor.
We examined how these strategies evolve over the course of meta-training and find that there two phases: After an initial period of universal random behavior across all conditions, the distinct behavioral regimes emerge (supplementary figure 9). We note that this observation may be partially
5

---

Under review as a conference paper at ICLR 2021
1. Learning Regime
2. No Learning Regime
Deterministic Arm Better
Stochastic Arm Better
Deterministic Arm Better
Stochastic Arm Better
Policy Entropy over Trials
Policy Entropy over Trials
Policy Entropy over Trials
Policy Entropy over Trials
Subopt.
9 Subopt. Pulls
Subopt Pulls
100 Subopt. Pulls
0.4
Pulls
0.0
100
50 Hidden State Changes
100
100
50 Trial
100
Hidden State Changes
Hidden State Chenges
Hidden State Changes
[mmml
fy Rel. Occ
fy
6 Rel. Doc
By Rel. Occ.
0.2
0.75
PC2
0.50
-0.50
0.50
-0.1
0.255
0.25
0.25
0.0
0.00
Figure 3: Recurrent dynamics of two meta-trained bandits for task conditions that favour learning (blue) and not learning (red), for lifetime T 100. Each bandit is tested both for the case where the deterministic arm is the better option and for the case where the stochastic arm is the better option. First two columns. Bandit for which adaptive strategy is predicted by the theory. The inner loop dynamics integrate information which is reflected by the convergence of the hidden state to two different fixed points depending on which arm is optimal. Last two columns. Bandit for which the heuristic choice of the deterministic arm is the Bayes optimal behavior. Not learning manifests itself non-exi con-explorattve, rigid behavior and activations. The final row visualizes the diiensmnssiin reduced hidden state dynamics (h;) averaged over 100 episodes. A darker color indicates a more frequent occupancy in the discretized PCA space of the transformed hidden states.
caused by the linear annealing of the entropy regularization coefficient in the actor-critic objective which we found to be crucial in training the networks.
In summary, we observe that the meta-learndd strategy shows highly nonlinear, partially discontinu- ous dependence on task parameters. In transition regions between strategies, we find local maxima in the reward landscape that correspond different learning strategies. In the simple bandit setting, these local maxima correspond to a learning and a non-learning strategy, respectively, hence providing minimal model for a sharp nature-nurture trade-off. Next, we investigate whether these insights generalize to more complex domains by studying spatial reasoning.
4 TIME HORIZONS, META-LEARNED STRATEGIES & ENTROPY REDUCTION
While the simple bandit task provides an analytical perspective on the trade-off of learning versus hard-coded behavior, it is not obvious that the obtained insights generalize to more complex situations, i... to distributions of finite-horizon MDPs. To investigate this, we studied exploration behavior in an ensemble of grid worlds task. We hypothesize that meta- meta-learning yields qualitatively different spatial exploration strategies depending on the lifetime of the agent. For short a lifetime, the agent should opt for small rewards that are easy to find. For longer lifetimes, the agent can spend time to explore the environment and identify higher rewards that are harder to find.
To test this hypothesis, of goal locations (top row of figure 4): 9r. (green object), 9m (yellow object) and gs (pink object) with transition rewards R(gh)) > R(gm.) R(gx).
train -ased meta learner to explore maze with three different types

---

Under review as a conference paper at ICLR 2021
Lifetime:
R(-)
9h
6
9m
3
9s
1
Lifetime:
Return test 100) Median, Perc.
80
Episoda Return
60
Lifetime: 100
40
pem -10
7 30
20
100
0.06
2.00
4,00
6.00
8.00
Train Episodes
1e5
Figure 4: Grid navigation task with 3 different rewards, which differ in the amount of reward and the uncertainty in location. Top-Left: Task formulation. Bottom-Left: Learning curves (T = 100) for different training lifetime. We plot the median, 25th and 75th percentile of the return distribution aggregated over 10 independent training runs and 10 evaluation episodes. Agents trained on a shorter lifetimes gradually generalize worse the larger test lifetime. Right: Relative state occupancy of meta-learned exploration strategies for different training lifetimes during meta-learning (averaged over 100 episodes of length 100). With increasing lifetime the agent explores larger parts of the state space and actively avoids suboptimal object location transition.
During episode/lifetime the goal locations fixed. At the beginning of the episode Im and Ih are randomly sampled. The location of gs. on the other hand, remains fixed across all training episodes. We sample the possible locations for gh from the outermost column and row (11 locations) while Im varies along the third row and column (excluding the borders, locations). Thereby, the three goals encode destinations with varying degrees of spatial uncertainty and payoff. The agent can move up. down, left and right. After it (red circle) transitions into goal location, it receives the associated reward and is teleported back the initial location the bottom left corner. Within one episode, the agent can hence first perform one several exploration runs, in which it identifies the object location, and then do series of exploitation runs, in which it takes the shortest path to that location. Importantly, the agent does not observe the goal locations but instead has to infer the locations based on the observed transition rewards.
Depending on the lifetime the agent during the inner loop adaptation, we find that meta-learning imprint 3 qualitatively different strategies (figure right column; figure 5) For small lifetimes the agent executes a hard-coded policy that repeatedly walks to the safe, low-reward pink object. As the lifetime and consequently inner loop adaptation is increased, we find that the agent's meta-les meta-learned policy starts to explore a broader range of locations int the maze, first exploring possible locations of the medium-reward object and for long lifetimes the distant and uncertain high-reward object (figure(Frigh) column). Consistently, the agent exploits increasingly uncertain rewards with increasing lifetime (figure 3)
rrigtt
Furthermore, we investigated how meta-learned strategies generalize across different timescales of adaptation. More specifically, we trained an agent learn (or not) with a given lifetime and tested how the learned behavior performed in setting where there is more or less time available. As predicted, we find that the test time-nome normalized return of the agents decreased with the discrepancy between training and test lifetime (figure This can be problematic in settings where the agent does not have access to its exact lifetime and highlights the lack of time-robustness of meta-adaptation.
7

---

Under review as a conference paper at ICLR 2021
Return Test 7 100
g Transitions
g Transitions
g" Transitions
5.0
10
50
2.5
5
0
0.0
5 15 30 50 70 90 Train Lifetime T
5 15 30 50 70 90 Train Lifetime T
5 15 30 50 70 90
16 30 50 70 90 Train Lifetime T
Train Lifetime
Figure 5: Lifetime dependence of the performance and the visitation counts of the goal locations for an RL agent trained on random 6 x 6 grid worlds and evaluated on TRest 100. For small lifetimes the meta-learned strategy only exploits the small safe object. For larger lifetimes the agent first explores the more uncertain medium (and later high) reward object locations. The displayed statistics (median, 25th/75th percentile) aggregated over 5 independent training runs and 500 evaluation episodes.
The agents displayed clear hallmarks of model-based be- havior and behavioral changes over their lifetime (figure When the agent has encountered the high reward once, it resorts to a deterministic exploitation strategy that fol- lows a shorter trajectory through the environment than the initially used during cxploration. Furthermore, the adaptive policies identify when there not enough time left in the episode to reach the previously exploited goal location. In that case the policies switch towards the easier to reach small goal location. The oscillating policy en- tropy (columns three and four of figure 7 is state-specific and indicates that the meta-learned strategies have cor- rectly learned transition model of the relevant parts of the environment. If action does not affect the overall length of the trajectory to a goal, this is reflected in the entropy of the policy. Finally, we analyzed the distinct recurrent dynamics for the three different strategies (final column of figure 7 We find that the dimensionality of the dynamics increases with the adaptivity of the behavior. As the training lifetime increases, the participation ratio (Gao et al.) 2017) of the hidden state dynamics increases and the explained variance of the first three principal
Time Normalized Return
100
1.0
7
80
Ttrain
60
0.8
40
0.6
20
0.4
20
40
60 7 Ttest
80
100
Figure 6: Episode return for agents trained on Terain and tested on Teest The returns are normalized by the test lifetime. Agents which were meta-trained on a fixed lifetime do not generalize well to smaller larger test lifetimes. The statistics are averaged over independent training runs and 500 test episodes.
components drops.
DISCUSSION & FUTURE WORK
This work has investigated the interplay of three considerations when designing meta-task distribu- tions: The diversity of the task distribution, task complexity and training lifetime. Depending on these, traditional meta-lcarning algorithms are capable of flexibly interpolating between distilling learning algorithm and hard-coding a heuristic behavior. The different regimes emerge in the outer loop of meta- meea-llerringg and are characterized by distinct recurrent dynamics shaping the hidden activity. Meta-learned strategies showed limited generalization timescales they were not trained on, emphasizing the importance of the training lifetime in meta-learning.
key take-home from our results is the highly nonlinear and potentially discontinuous dependence of the meta-learned strategy on the parameters of the task ensemble, For certain parameter ranges, the reward landscape the meta-learning problem features several local maxima that correspond to different learning strategies. The relative propensity of these strategies to emerge over the course of meta- learningnnnng depends on the task parameters and on the initialization of the agent. Generally, this supports the notion that there is not single inductive bias for given task distribution. Rather, there could be a whole spectrum of inductive biases that are appropriate for different amounts of training data. Even for the same task setting, different training runs result in qualitatively different
8

---

Under review as a conference paper at ICLR 2021
TT m0.m(.)
Total Episode Return
Him) >- 0.2
1.0
Policy Entropy
PC Variance Explained:
75
Sh Be 9..g-133
Goal Transe
25
0.0
0
25
50
75
100
25
50
75
100
Eptsode Step
Episode Step
08 -1
Treaee m0.)
Total Episada Return
Mi()) 0.2
Participaton rate: 16.30 Variance
75
Geal Transtions: 0::0, du- 14,
0.5
25
6.0
25
50
75
100
25
50
75
100
Epieodio Step
Epicode Stop
Thali 50,
Total Episode Return
N(m) >> 0.2
1.0
Policy Entropy
PC Variance Explained: 0.57
Porticipation rate: 50.5
75
Goal Transitians 9s: 13. 7
25
0.0
25
50
75
100
50 75 100
Eptsode Step
Episode Step
Figure Characteristic trajectories for three different types of meta-learned strategies. Top to bottom: Episode rollouts (T = 100) for inner loop training lifetimes T {10, 20, 50}. First four columns:
The agents' trajectories, episode return, states with high average policy entropy (red squares) and the policy entropy for the same sampled environment. Final column. PCA- dimensionality reduced hidden state dynamics for different 100 rollout episodes and the agents' respective training lifetimes. A lighter color indicates later episode trials. With increasing lifetime meta-learned strategies become more adaptive and the recurrent dynamics higher dimensional.
solutions, providing note caution for interpretations drawn by pooling over ensembles trained networks.
The observed nonlinear dependence of the obtained solution may be relevant, e.g., for robotic applications, in which rapid adaption of controllers trained in simulation to real-world robotic devices is desirable (e.g., [Nagabandi et al. 2018 Belkhale et al.) 2020 Julian et al. 2020). is beneficial ensure rapid adaptation the real robot, e.g., avoid physical damage. To achieve this, the meta-learner should be optimized for a short horizon. This, however, introduces bias towards learning, or in more complex settings, for heuristic solutions that explore less than required to discover the optimal policy. For such problems, the curation of the meta-lcarning task ensemble may have additionally take into account potential nonlinear and long-lasting trade-offs between final performance and speed of adaptation.
The observed sharp transition between exploratory learning behavior(s) and hard-coded, noa-learning strategies can be seen proof-of-concept example for "nature-nurture" trade-off that adds new aspects to earlier work in theoretical ecology (Stephens) [199)). From this perspective of animal behavior, meta-leane meta-learning with a finite time horizon could provide inroad into understanding the benefits and interactions of instinctive and adaptive behaviors. Potential applications could be the meta-learning of motor skills in biologically inspired agents (Merel et al. 2019) instinctive avoidance reactions to colours or movements. The degree of biological realism that can be reached will be limited by computational resources, but qualitative insights could be gained, e.g., for simple instinctive behaviors. different extension of our analysis is that to non-stationary environments, although suspect qualitative analogy of lifetime in approach and auto-correlation times in non-stationary environments.
REFERENCES
Marcin Andrychowiez, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in newral information processing systems, pp. 3981-3989, 2016.

---

Under review as a conference paper at ICLR 2021
Sergey Bartunov, Jack W Rae, Simon Osindero, and Timothy P Lillicrap. Meta-learnnng deep energy-based memory models. arXiv preprint arXiv:1110.02220,, 2019.
Suneel Belkhale, Rachel Li, Gregory Kahn, Rowan McAllister, Roberto Calandra, and Sergey Levine. Model-based rentroreinoo learning for flight with suspended payloads. arXiv preprint arXiv:2004. 1134, 2020.
David Buss. Evolutionary psychology: The new science of the mind. Psychology Press, 2015.
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Pieter Abbeel. RI 2; Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.0277, 2016.
Aimee S Dunlap and David W Stephens. Components of change in the evolution of learning and unlearned preference. Proceedings of the Royal Society B: Biological Sciences, 276(1670): 3201-3208 2009.
Aimee S Dunlap and David W Stephens. Reliabilitty, uncertainty, and costs in the evolution of animal learning. Current opinion in behavioral sciences, 12:73-79, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1135-1135. JMLR. org, 2017.
Sebastian Flennerhag, Pablo G Moreno, Neil D Lawrence, and Andreas Damianou. Transferring knowledge across learning processes. arXiv preprint arXiv: 2018.
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. arXiv preprint 190.000900.0025, 2019.
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, pp. 214262, 2017.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Efficient adaptation for end-to-end vision-based robotic manipulation. arXiv preprint 20Xi::0004.10100 2020.
Mellissa Marcus, Terence C Burnham, David W Stephens, and Aimee S Dunlap. Experimental evolution of color preference for oviposition in drosophila melanogaster. Journal of Bioeconomics, 20(1):125-140, 2018.
Josh Merel, Diego Aldarondo, Jesse Marshall, Yuval Tassa, Greg Wayne, and Bence Ölveczky. Deep neuroethology of a virtual rodent. arXiv preprint arXiv:1911.0945. 2019.
Vladimir Mikulik, Grégoire Delétang, Tom McGrath, Tim Genewein, Miljan Martic, Shane Legg, and Pedro Ortega. Meta-trained agents implement bayes-optimal agents. Advances in Neural Information Processing Systems, 33, 2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Donald Mutti, Karla Zadnik, and Authony J Adams. Myopia. the nature versus nurture debate goes Investigative ophthalmology & visual science, 37(6):952-99) 957, 1996.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforceemett learning. arXiv preprint arXiv: 1803. /347, 2018.
Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv: /905 030000 2019.
10

---

Under review as a conference paper at ICLR 2021
Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. arXiv preprint arXiv:1712.00378,, 2017.
Neil C Rabinowitz. Meta-learners' learning dynamics are unlike learners', arXiv preprint
frX.v:1905.01320, 2019.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. arXiv preprint arXiv: 1802.07700 2018.
Adam Santoro, Sergey Bartumov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentee neural networks. In International conference on machine learning, pp. 1842-18555 2016.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
DW Stephens. Change, regularity, and value in the evolution of animal learning. Behavioral Ecology, 2(1):77-89, 1991.
James Tabery, Beyond versus: The struggle to understand the interaction of nature and nurture. MIT Press, 2014.
Sebastian Thrun and Lorien Pratt. Learning learn: Introduction and overview. In Learning learn, pp. 3-17. Springer, 1998.
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint /6 1.05763, 2016.
Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinffrcemment learning system. Nature neuroscience, 21(6):860-8666 -68, 2018.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- meta-learming. arXiv preprint arXiv:1910008448 2019.

---

Under review as a conference paper at ICLR 2021
A
SUPPLEMENTARY MATERIALS
A.1
MATHEMATICAL DERIVATION OF OPTIMAL EXPLORATION
In the following section describe the Gaussian two-arm bandit setting analyzed in section 3 The first arm generates stochastic rewards according to hierarchical Gaussian emis- sions. Between individual episodes the mean reward je is sampled from Gaussian distribution with standard deviation Fp. The second arm, on the other hand, arm generates a deter- ministic reward of More specifically, the generative process for rewards resulting from pull of the first arm is described as follows:
Gaussian Multi-Arm Bandit Problem Formulation
H ~
o(-1,0g)
1 n~ 1~11- r NN,, of)
(u+ 1)
$
p(u)oz)
exp
v 2TT
20
Tt
(1)
rt
(2)
=0
p(rlu,o))
1
exp
(r 11)
22T07
20
N(,,o7)
Our Bayesian agent is assumed to spend fixed amount of trials n of the overall lifetime exploring the second stochastic This assumption is justified since corresponding meta-learning agent may easily encode the deterministic nature of the fixed arm 0 and therefore only explore the second non-deterministic arm
Figure 8: Two-arm Gaussian bandit.
P The expected
cumulative reward of such two-phase exploration-exxloitation policy can then be factorized as follows:
T
T
E
rt
n
= E,,
Σ Fr 7
Ep,r
Σ
re
n,
t=1
1=++1
= (-1) x n + Ey
[(T
n))(pa > 0)4l
where ji denotes the maximum posteriori (MAP) estimate of after trials:
á 1 Poo
1 + P
with P, = oz, Pi o7. Peot = + and F 1 Σm ri. For a fixed μ (within a lifetime) this
random variable follows univariate Gaussian distribution with sufficient statistics given by:
B
Varl(i P, +
Poot
The probability of exploiting arm 2 after n exploration trials is then given by the following integral:
p 0) [ p(jidp.
ET.TIIT re|m] may then be evaluated by numerical integration and the resulting optimal n² obtained by searching over range of = 1,...,, T.
Thh second column of figure validates this assumption since the policy entropy quickly vanishes after an initial exploration phase.
12

---

Under review as a conference paper at ICLR 2021
A.2 EXPERIMENTAL DETAILS
A.2.1
MEMORY-BASED BETTARRENFORCEMENT LEARNING
We follow the standard RL paradigm Wang et al. 2016) and train an LSTM-based actor-critic architecture using the A2C objective (Mnih et al. 2016):
LAC = L + B - 3.6
_ E [lo (a) -(a)]]
_ Er [1 (xx))
C E, (H((((a)(z=a))
T-t-1
R₂ = Σ yreti
where R₃ denotes the cumulative discounted reward resulting from the rollout of the episode. The agent interacts with sampled environment for single episode. Between episodes new environment sampled. Unless otherwise stated we ensure a proper scaling of the timestamp input and follow Pardo et al. (2017) by normalizing the time input to lie between -1 and
A.2.2 GAUSSIAN MULTI-ARM BANDITS: HYPERPARAMETERS
All results of section 3 for the two-arm Gaussian bandit setting (and all or, 0g cp-cobbinassoss) may be reproduced using the following set of hyperparameters:
Parameter
Value
Parameter
Value
Parameter
Value
Training episodes
30k
Learning rate Optimizer
0.001
L2 Weight decay
06
Clipped gradient norm
10
Adam
Workers
2
YY
0.999
8e,r
0,005
3u
0.05
YYu
0.4
3e.0
LSTM hidden units
48
Y Anneal time
Ep.
3. Anneal time
30k Ep.
Learned hidden init.
y Schedule
Exponential
3e Schedule
Linear
Forget gate bias init. Orthogonal weight init.
Table 1: Hyperparameters (architecture & training procedure) of the bandit A2C agent.
We use the same set of hyperparameters for all 0;, op and T combinations. Our theoretical results are derived for the case of the temporal credit assignment problem is implicitly defined by the effective length of the time window. We observed that starting with large y often times hindered the network in learning. Hence, we discount factor as a form of implicit curriculum and annealed it accordingly.
1. This poses challenge when training recurrent policies. The difficulty
A.2.3 GRIDWORLD NAVIGATION TASK: HYPERPARAMETERS
Parameter
Value
Parameter
Value
Parameter
Value
Training episodes
IM
Learning
0.001
Lg Weight decay À
Clipped gradient norm
Optimizer
Adam
Workers
0.99
Se.r
0.5
B.
0.1
3. Schedule
Linear
8..0
0.01
LSTM hidden units
256
3. Anneal time
700k
Learned hidden init.
Table 2: Hyperparameters (architecture & training procedure) of the gridworld A2C agent.
In order to train the agents that generated the state occupancies in figure and the rollout trajectories in figure 7 we additionally annealed the discount factor starting at 0.8 to within the first 800k episodes.
13

---

Under review as a conference paper at ICLR 2021
A.3
SUPPLEMENTARY FIGURES
Learning lear:: Exploration training checkpoints (T=100)
Episodes
10k Episodes
15k Episodes
20k Episodes
25k Episodes
10.0
Exploration Thiis
-0
6.9
3.7
-10
0.6
64
12.7
19.0
6.4
12.7
19.0
0.1
6.4
12.7
19.0
0.1
12.7
19.0
0.1
12.7
10.0
Figure 9: Learning to learn and not to learn. Left to right. The meta-learned exploration strategy
visualized for different increasing checkpoints throughout meta-learning (I 100). The initial behavior is random, but as training progresses two distinct regimes (adaptive versus heuristic) emerge. The amount of meta-learned exploration averaged both over independent training runs and 100 episodes for each of the 400 trained networks.
14


Reviews:
[{"review_id": "XM54_WQeCYA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an analysis of RNN-based meta learning approaches. In particular, it investigates the strategies learned via meta-learning, contrasting strategies involving task-dependent learning vs heuristic or hard-coded solutions. Empirical evidence in two sets of experiments, on a 2-armed bandit toy task and a grid-world navigation task, show that hard-coded strategies can be a function of training task distribution and task complexity as well as task horizon. \n\nWhile the experiments are simplistic, they provide a clear and thorough comparison of different agent behaviours across different training regimes. I enjoyed reading the paper and although the results are intuitive and unsurprising, they nicely emphasize the importance of environment and task design choices in strategies learned via memory-based meta-learning.\n\nComments/questions:\n1. I would encourage the authors to use “memory-based/RNN-based meta-learning” instead of “meta-learning” to avoid confusion as these results might not apply more widely across different meta-learning approaches (e.g. gradient based).\n2. While the pattern of behaviour looks qualitatively similar across analytical and empirical results reported in Figure 1, I wonder if it would be possible to quantitatively assess where they differ.\n3. It would be nice to see error bars for the 5 independent training runs in Figure 5.\n4. In the Appendix A.2.3 it is mentioned that the discount factor is annealed from 0.8 to 1. within the first 800k episodes. Could you please expand if this was crucial to achieve your results or just an experimental choice?\n \nOverall, I think this is an interesting contribution of perhaps limited scope but still valuable and could encourage interesting future research directions in memory-based meta-learning.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NMItdzk6Yuy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary.** The authors investigate the question of when the optimal behavior for an agent is to learn from experience versus when the optimal behavior is to apply the same (memorized) policy in every scenario. They begin by introducing a simple bandits environment wherein they derive the optimal policy and identify regimes in which it involves memorization vs. learning. Then they train an RL^2 agent and verify that it behaves as expected in these regimes. Next, they expand their approach to a slightly more complicated gridworld environment which does not have an analytic solution to the question. The agent behaves as expected in the gridworld environment.\n\n**Strong points.** This paper tackles a novel question which is fundamental to the field of metalearning. By carefully analyzing the two regimes of learning and memorization in the context of metalearning, this paper will increase awareness about the fact that the two regimes exist. The paper is clearly written and does an excellent job of putting experiments in the context of past ML research. The experimental setup is simple but goes straight to the heart of the issue. Figures and text do a good job of analyzing results and communicating them to the reader. Overall, this paper was an interesting read.\n\n**Weak points.** The idea that “sometimes memorization is best and other times learning is best” does border on the obvious. Indeed, as soon as the authors derive their analytical solution, it becomes clear that we can expect the RL^2 agent to learn the same behavior. For me, there were no surprises in the experimental sections. To the authors: was there anything that was surprising or not obvious to you? What additional information can the experiments tell us, apart from confirming theoretical predictions?\n\nHaving said that, I also believe that very simple, well-executed research ideas sometimes make the best papers. This paper appears to be one of those cases. And even though the ideas are simple, they are significant and they are not a major part of the dialogue in the meta-learning community yet. So even if the ideas seem obvious, I think there is value in communicating them well.\n\nI have one concern about the bandit task setup: the authors adjust $\\sigma_l$, the width of the Gaussian from which they are sampling the reward, as a proxy for aleatoric uncertainty and hence task complexity. In doing so, they essentially equate “stochasticity of the environment” with “task complexity.” And yet, there are many other ways in which a task can be complex. Sometimes, all the information needed to perform a task is present and yet the task is difficult to solve because one needs to interpret/integrate the information in a particular way. This is why, for example, puzzles are considered difficult tasks. It is also why simulating the 3-body problem is a complex task. To the authors: can you clarify what you mean by “task complexity”?\n\nIn the closing paragraph of the paper, the authors claim that their approach “allows us to study the emergence of inductive biases in biological systems” but this claim is not supported by the rest of the paper, which makes almost no connections to biological systems. There are certainly ways in which these results are relevant to learning in biological systems, but the authors did not explore them in this paper, and so this claim is not well supported. In the same paragraph, they bring in contrasting notions of Darwinian and Lamarkian inheritance. Since they do this in one sentence -- the last sentence -- it is hard to understand what their claim is. And it was not clear that this was one of the main takeaways of the paper, as these concepts do not appear anywhere else in the paper. If the authors want to draw these conclusions, then they should add additional discussion on these topics. Otherwise, they risk misleading readers.\n\nOne additional minor suggestion would be to invert the color scale of Figure 6, as “white -> red” signifies values of increasing size in all preceding plots, but in Figure 6 it currently signifies values of decreasing size.\n\nMinor grammatical suggestions\n-- “the question which aspects of behavior“ -> “the question of which aspects of behavior“\n-- When typing quotes in LaTex, use `` and ‘’ instead of “” so as to make them open & close correctly\n-- “interplay of the agent’s lifetime,” -> “interplay between the agent’s lifetime,”\n-- “We numerically show” -> “We show numerically”\n-- “as well as explicit models of memory” -> ”and explicit models of memory” (same issue occurs later)\n-- “the agents does not have” -> “the agent does not have”\n\n**Recommendation.** 6 : Marginally above acceptance threshold\n\n**Reasoning.** This paper is well written and the experimental setup is simple, well-executed, and produces results that are relevant to the main question of the paper. The main question of the paper -- when does it make more sense to learn vs. memorize a behavior -- is significant to ICLR and to the field of machine learning. There are a number of relatively minor weaknesses (as described above) but this is overall a nice paper and would be a good contribution to ICLR 2021.", "extra_scores": {}, "extra_text": {}}, {"review_id": "F91bjPVYAqN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper observes that in meta-RL (and evolutionary biology), sometimes it is advantageous to learn behaviors that adapt to the particular task, while other times not adapting to the task, and instead relying on a task-agnostic “hard-coded” behavior is sufficient. While much meta-RL research typically focuses on the former setting, this paper studies when it is not necessary to learn adaptive behaviors. Specifically, this paper presents three main findings: (i) whether or not it is optimal to learn adaptive behavior strongly depends on the horizon of the task and complexity of learning such adaptive behaviors — if the horizon is too short, or if the adaptive behavior requires complex exploration, then exploring the new task to learn adaptive behaviors may not be worth it; (ii) existing meta-RL agents are capable of choosing not to learn adaptive behaviors, when it is optimal to do so; (iii) existing meta-RL agents generalize poorly to tasks with varying horizon-lengths.\n\nStrengths:\n- Novelty. Most meta-RL papers study the case where adapting to a new task requires learning new behaviors. Instead, this paper studies when such adaptation is not necessary, which is an area that has not been well-studied. Therefore, it is interesting and novel to bring attention to the fact that both regimes exist.\n- Clarity and Execution. This paper is generally well-executed and clear. The bandit example in Section 3 clearly illustrates the two regimes that this paper studies: when it is necessary to learn adaptive behaviors vs. not, and the results are presented in figures that impressively, clearly illustrate the points made in the text. This section also convincingly shows that existing meta-RL agents roughly learn the Bayes-optimal policy in this case. Similarly, the grid world tasks in Section 4 also clearly illustrate how the behavior of meta-RL agents changes with the horizon of the task. The paper does a good job at thoroughly studying what happens when each parameter (e.g., horizon-length, aleatoric uncertainty, and epistemic uncertainty) varies.\n\nWeaknesses:\n- Significance. My primary concern with this work is significance. I believe that the main insights of this paper can be interpreted through the lens of Bayes-adaptive policies [1]. The optimal Bayes-adaptive policy explains when it is optimal to explore a new task and learn adaptive behaviors, depending on the horizon length and amount of exploration needed, and therefore, the main claim of the paper can be framed as observing that existing meta-RL agents can learn the Bayes-adaptive optimal policy in simple tasks. This is an interesting observation, but prior work [2] already shows that meta-RL is equivalent to learning the Bayes-adaptive optimal policy. Furthermore, for harder tasks, this is likely to be confounded with the complexity of learning efficient exploration behaviors that allow learning adaptive behaviors, as shown by prior work [2, 3, 4]. Overall, it is therefore unclear to me what significant takeaways the meta-RL community can gain from this work. One potential takeaway is the observation that meta-RL agents generalize poorly to tasks with varying horizon-lengths, but it’s unclear whether this setting occurs in real tasks, and even if this _is_ necessary, then simply adding the horizon to the observation (state) and varying the horizon during meta-training seems sufficient.\n\nOverall, I found this paper to be a well-written, well-executed, illustration of when it is optimal to learn task-specific adaptive behaviors. However, due to its limited significance, I am unfortunately unable to recommend acceptance.\n\nAdditional Questions / Comments:\n- In Figure 2, what does it mean with “100 Suboptimal pulls?” Is this over the course of all training? Also, why is the initial policy entropy at 0? I would expect that upon initialization, the policy entropy is not 0.\n- I believe that the Bayes-adaptive optimal policy can also be analytically computed for the grid world tasks. It would be nice to report the optimal returns in Figure 5.\n- The interplay between the term “lifetime” and episodes was not explicitly defined. It was possible to infer the paper’s intent from reading the experiments, but more carefully defining this could help.\n- Framing the exploration trade-off in terms of evolutionary biology is interesting.\n\nReferences:\n\n[1] Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. Michael O’Gordon Duff. February 2002. https://www.gatsby.ucl.ac.uk/~yael/Okinawa/DuffThesis.pdf\n\n[2] VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, Shimon Whiteson. October 2019. https://arxiv.org/abs/1910.08348\n\n[3] Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables. Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, Sergey Levine. March 2019. https://arxiv.org/abs/1903.08254\n\n[4] Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning. Evan Zheran Liu, Aditi Raghunathan, Percy Liang, Chelsea Finn. June 2020. https://openreview.net/forum?id=La1QuucFt8-\n\n------------\n\n**Edit: Score raised from 4 --> 5 following discussion below.**", "extra_scores": {}, "extra_text": {}}, {"review_id": "9lldrgajfe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper explores the effect of time horizon on meta-reinforcement learning agents. Using a recurrent meta-learner, it demonstrates that different strategies are learned based on the time horizon during meta-training.\n\nPros:\nCreative proposal to study when incorporating new data to adapt to a task is warranted versus executing an existing behavior.\nResults are very nicely presented and writing is good\nCons:\nThe analogy to “nature versus nurture” seems tenuous\nResults seem somewhat obvious (see comments below)\n\nDetailed Comments:\n\nWhile I appreciate the inspiration of the biological connections argued in this work, I am not convinced that the analogy of learning versus adapting in meta-learning maps neatly onto the concepts of nature versus nurture in biology. If biology terminology is going to be used, it needs to be carefully defined for a machine learning audience, and the limitations of the analogy discussed. Presently, I find that the use of this terminology subtracts from the clarity of the work.\n\nMy main criticism of the work is that the results seem fairly obvious. In the finite horizon case, the meta-learner will necessarily learn a strategy optimal for the given horizon because that’s exactly what it’s optimized for. It seems akin to me to training an RNN policy in an MDP and then noticing that the RNN is capable of *not* persisting information across timesteps since it isn’t needed. Since feedforward models are a subset of recurrent models, this seems obvious.\n\nIt also seems strange to me that the agent is only tested in 1 episode. It seems like if the agent has learned a new skill, it should be able to repeat that skill if given the chance (e.g., with a reset), and that that is a hallmark of meta-learning. What is happening in this paper seems more like “horizon-dependent RL.” I don’t really see how the behavior in the long-time horizon setting can be called “learning” while the short time-horizon behavior is not. Both seem like the optimal policy for the given MDP. In Section 4, for example, wouldn’t “learning” be defined as figuring out where the colored squares are? Yet in the paper it seems to be defined as executing a policy that moves farther away from the start. It seems to me like exploration and learning are being conflated here. \n\nSeveral times throughout the paper it is claimed that the work “investigates the interplay of  three considerations when designing meta-task distributions: The diversity of the task distribution, task complexity and training lifetime.” I don’t see how the first two are analyzed in this paper. Perhaps in the bandit example, but there it seems to be always entangled with the training lifetime.\n\nBesides the comments about the biological terminology mentioned earlier, the paper is clear and easy to read, with the other exception of the description of the 2-arm Gaussian bandit in Section 3. This reviewer had to read that first paragraph about 5 times to understand the setup. The figures are well-done and clearly present the results.\n\nRecommendation:\n5. While experiments are thorough and nicely presented, I presently don’t see what insights are gained from the analysis. The claim of analyzing “diversity of the task distribution, task complexity and training lifetime” seems like an over-claim to me. It’s not clear to me that the agent “learns” a new skill, since there is only exploration, and no exploitation.\n\n----Update----\nAfter reading all the other reviews and ensuing discussions, I maintain my original score. If the research question is \"How does the optimal policy depend on task parameters such as uncertainty and horizon?\" I believe Bayes-adaptive work answers that question. If the question is \"How do policies learned by meta-RL algorithms compare to Bayes-optimal policies?\" then I think more empiricism is needed (since RL2 in principle can represent the Bayes-optimal policy), or a comparison of multiple methods.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was fairly detailed discussion among three of the four reviewers. The fundamental concern of the reviewers is regarding the contribution of the paper. During the rebuttal, the authors clarified the following:

> while the effects of varying uncertainty / horizon lengths is well-understood for Bayes-optimal policies, it is not understood for existing meta-RL approaches, which is the topic of this paper

That is, the contribution of the paper is to understand the effects of varying uncertainty/horizon lengths for meta-RL approaches. However, it is known in prior work that meta-RL algorithms such as RL^2 can implement Bayes-optimal policies in principle. As a result, it's not clear whether this contribution is significant relative to prior knowledge, and this paper does not seem to bring any new insights.

An alternative framing of the paper would be to consider the question of how meta-RL solutions compare to Bayes-adaptive optimal policies. While this framing would be interesting and novel, the current version of the paper does not sufficiently answer this question, since the only experiments include RL^2 (and such a study would require experimenting with more sophisticated meta-RL algorithms beyond RL^2).

As such, this paper isn't suitable for publication at ICLR in its current form.
Decision: Reject