Learning not to learn: Nature versus nurture in silico

1 INTRODUCTION . The ’ nature versus nurture ’ debate ( e.g. , Mutti et al. , 1996 ; Tabery , 2014 ) – the question of which aspects of behavior are ’ hard-coded ’ by evolution , and which are learned from experience – is one of the oldest and most controversial debates in biology . Evolutionary principles prescribe that hard-coded behavioral routines should be those for which there is no benefit in adaptation . This is believed to be the case for behaviors whose evolutionary advantage varies little among individuals of a species . Mating instincts or flight reflexes are general solutions that rarely present an evolutionary disadvantage . On the other hand , features of the environment that vary substantially for individuals of a species potentially ask for adaptive behavior ( Buss , 2015 ) . Naturally , the same principles should not only apply to biological but also to artificial agents . But how can a reinforcement learning agent differentiate between these two behavioral regimes ? A promising approach to automatically learn rules of adaptation that facilitate environment-specific specialization is meta-learning ( Schmidhuber , 1987 ; Thrun & Pratt , 1998 ) . At its core lies the idea of using generic optimization methods to learn inductive biases for a given ensemble of tasks . In this approach , the inductive bias usually has its own set of parameters ( e.g. , weights in a recurrent network ; Hochreiter et al. , 2001 ) that are optimized on the whole task ensemble , that is , on a long , ’ evolutionary ’ time scale . These parameters in turn control how a different set of parameters ( e.g. , activities in the network ) are updated on a much faster time scale . These rapidly adapting parameters then allow the system to adapt to a specific task at hand . Notably , the parameters of the system that are subject to ’ nature ’ – i.e. , those that shape the inductive bias and are common across tasks – and those that are subject to ’ nurture ’ are usually predefined from the start . In this work , we use the memory-based meta-learning approach for a different goal , namely to acquire a qualitative understanding of which aspects of behavior should be hard-coded and which should be adaptive . Our hypothesis is that meta-learning can not only learn efficient learning algorithms , but can also decide not to be adaptive at all , and to instead apply a generic heuristic to the whole ensemble of tasks . Phrased in the language of biology , meta-learning can decide whether to hard-code a behavior or to render it adaptive , based on the range of environments the individuals of a species could encounter . We study the dependence of the meta-learned algorithm on three central features of the metareinforcement learning problem : • Ecological uncertainty : How diverse is the range of tasks the agent could encounter ? • Task complexity : How long does it take to learn the optimal strategy for the task at hand ? Note that this could be different from the time it takes to execute the optimal strategy . • Expected lifetime : How much time can the agent spend on exploration and exploitation ? Using analytical and numerical analyses , we show that non-adaptive behaviors are optimal in two cases – when the optimal policy varies little across the tasks within the task ensemble and when the time it takes to learn the optimal policy is too long to allow a sufficient exploitation of the learned policy . Our results suggest that not only the design of the meta-task distribution , but also the lifetime of the agent can have strong effects on the meta-learned algorithm of RNN-based agents . In particular , we find highly nonlinear and potentially discontinuous effects of ecological uncertainty , task complexity and lifetime on the optimal algorithm . As a consequence , a meta-learned adaptation strategy that was optimized , e.g. , for a given lifetime may not generalize well to other lifetimes . This is essential for research questions that are interested in the conducted adaptation behavior , including curriculum design , safe exploration as well as human-in-the-loop applications . Our work may provide a principled way of examining the constraint-dependence of meta-learned inductive biases . The remainder of this paper is structured as follows : First , we review the background in memorybased meta-reinforcement learning and contrast the related literature . Afterwards , we analyze a Gaussian multi-arm bandit setting , which allows us to analytically disentangle the behavioral impact of ecological uncertainty , task complexity and lifetime . Our derivation of the lifetime-dependent Bayes optimal exploration reveals a highly non-linear interplay of these three factors . We show numerically that memory-based meta-learning reproduces our theoretical results and can learn not to learn . Furthermore , we extend our analysis to more complicated exploration problems . Throughout , we analyze the resulting recurrent dynamics of the network and the representations associated with learning and non-adaptive strategies . 2 RELATED WORK & BACKGROUND . Meta-learning or ’ learning to learn ’ ( e.g. , Schmidhuber , 1987 ; Thrun & Pratt , 1998 ; Hochreiter et al. , 2001 ; Duan et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017 ) has been proposed as a computational framework for acquiring task distribution-specific learning rules . During a costly outer loop optimization , an agent crafts a niche-specific adaptation strategy , which is applicable to an engineered task distribution . At inference time , the acquired inner loop learning algorithm is executed for a fixed amount of timesteps ( lifetime ) on a test task . This framework has successfully been applied to a range of applications such as the meta-learning of optimization updates ( Andrychowicz et al. , 2016 ; Flennerhag et al. , 2018 ; 2019 ) , agent ( Rabinowitz et al. , 2018 ) and world models ( Nagabandi et al. , 2018 ) and explicit models of memory ( Santoro et al. , 2016 ; Bartunov et al. , 2019 ) . Already , early work by Schmidhuber ( 1987 ) suggested an evolutionary perspective on recursively learning the rules of learning . This perspective holds the promise of explaining the emergence of mechanisms underlying both natural and artificial behaviors . Furthermore , a similarity between the hidden activations of LSTM-based meta-learners and the recurrent activity of neurons in the prefrontal cortex ( Wang et al. , 2018 ) has recently been suggested . Previous work has shown that LSTM-based meta-learning is capable of distilling a sequential integration algorithm akin to amortized Bayesian inference ( Ortega et al. , 2019 ; Rabinowitz , 2019 ; Mikulik et al. , 2020 ) . Here we investigate when the integration of information might not be the optimal strategy to meta-learn . We analytically characterize a task regime in which not adapting to sensory information is optimal . Furthermore , we study whether LSTM-based meta-learning is capable of inferring when to learn and when to execute a non-adaptive program . Rabinowitz ( 2019 ) previously studied the outer loop learning dynamics and found differences across several tasks , the origin of which is however not fully understood . Our work may provide an explanation for these different meta-learning dynamics and the dependence on the task distribution as well as the time horizon of adaptation . Our work is most closely related to Pardo et al . ( 2017 ) and Zintgraf et al . ( 2019 ) . Pardo et al . ( 2017 ) study the impact of fixed time limits and time-awareness on deep reinforcement learning agents . They propose using a timestamp as part of the state representation in order to avoid state-aliasing and the non-Markovianity resulting from a finite horizon treatment of an infinite horizon problem . Our setting differs in several aspects . First , we study the case of meta-reinforcement learning where the agent has to learn within a single lifetime . Second , we focus on a finite horizon perspective with limited adaptation . Zintgraf et al . ( 2019 ) , on the other hand , investigate meta reinforcement-learning for Bayes-adaptive Markov Decision Processes and introduce a novel architecture that disentangles task-specific belief representations from policy representations . Similarly to our work , Zintgraf et al . ( 2019 ) are interested in using the meta-learning framework to distill Bayes optimal exploration behavior . While their adaptation setup extends over multiple episodes , we focus on single lifetime adaption and analytically analyze when it is beneficial to learn in the first place . Finally , our work extends upon the efforts of computational ethology ( Stephens , 1991 ) and experimental evolution ( Dunlap & Stephens , 2009 ; 2016 ; Marcus et al. , 2018 ) , which aims to characterize the conditions under which behavioral plasticity may evolve . Their work shows that both environmental change and the predictability of the environment shape the selection pressure , which evolves adaptive traits . Our work is based on memory-based meta-learning with function approximation and aims to extend these original findings to task distributions for which no analytical solution may be available . 3 LEARNING NOT TO LEARN . To disentangle the influence of ecological uncertainty , task complexity , and lifetime on the nature of the meta-learned strategy , we first focus on a minimal two-arm Gaussian bandit task , which allows for an analytical solution . The agent experiences episodes consisting of T arm pulls , representing the lifetime of the agent . The statistics of the bandit are constant during each episode , but vary between episodes . To keep it simple , one of the two arms is deterministic and always returns a reward of 0 . The task distribution is represented by the variable expected reward of the other arm , which is sampled at the beginning of an episode , from a Gaussian distribution with mean -1 and standard deviation σp , i.e . µ ∼ N ( −1 , σ2p ) . The standard deviation σp controls the uncertainty of the ecological niche . For σp 1 , the deterministic arm is almost always the better option . For σp 1 , the chances of either arm being the best in the given episode are largely even . While the mean µ remains constant for the lifetime T of the agent , the reward obtained in a given trial is stochastic and is sampled from a second Gaussian , r ∼ N ( µ , σl ) . This trial-to-trial variability controls how many pulls the agent needs to estimate the mean reward of the stochastic arm . The standard deviation σl hence controls how quickly the agent can learn the optimal policy . We therefore use it as a proxy for task complexity . In this simple setting , the optimal meta-learned strategy can be calculated analytically . The optimal exploration strategy is to initially explore the stochastic arm for a given trial number n. Afterwards , it chooses the best arm based on its maximum a posteriori-estimate of the remaining episode return . The optimal amount of exploration trials n ? can then be derived analytically : 1 n ? = arg max n E [ T∑ t=1 rt|n , T , σl , σp ] = arg max n [ −n+ Eµ , r [ ( T − n ) × µ× p ( µ̂ > 0 ) ] ] , where µ̂ is the estimate of the mean reward of the stochatic arm after the n exploration trials . We find two distinct types of behavior ( left-hand side of figure 1 ) : A regime in which learning via exploration is effective and a second regime in which not learning is the optimal behavior . It may be optimal not to learn for two reasons : First , the ecological uncertainty may be so small that it is very unlikely that the stochastic first arm is better . Second , if the trial-to-trial variability is too large relative to the range 1Please refer to the supplementary material for a detailed derivation of this analytical result as well as the hyperparameters of the numerical experiments . of potential ecological niches , so that it may simply not be possible to integrate sufficient information given a limited lifespan . We make two observations : 1 . There exists a hard nonlinear threshold between learning and not learning behaviors described by the ratio of σl and σp . If σl is too large , the value of exploration ( or the reduction in uncertainty ) is too small to be profitable within the remaining lifetime of the agent . Instead , it is advantageous to hard-code a heuristic choice . 2 . The two regimes consistently exist across different lifetimes . As the lifetime grows , the learning regime becomes more and more prevalent . Given a sufficient amount of time , learning by exploring the uncertain arm is the best strategy . Is the common meta-learning framework capable of reproducing these different qualitative behaviors and performing Bayes optimal amortized inference across the entire spectrum of meta-task distributions ? Or differently put : Can memory-based meta-learning yield agents that do not only learn to learn but that also learn not to learn ? To answer this question , we train LSTM-based RL2 ( Wang et al. , 2016 ) agents with the standard synchronous actor-critic ( Mnih et al. , 2016 ) setup on the same grid of ecological uncertainties σp and ” task complexities ” σl . The input xt to the network at time t consists of the action of the previous timestep , a monotonically increasing timestamp within the current episode and crucially the reward of the previous timestep , xt = { at−1 , φ ( t ) , rt−1 } . The recurrent weight dynamics of the inner loop can then implement an internal learning algorithm that integrates previous experiences . After collecting a set of trajectories , we optimize the weights and initial condition of the hidden state with an outer loop gradient descent update to minimize the common actor-critic objective . We obtain the amount of meta-learned exploration by testing the RL2 agents on hold-out bandits for which we set σp = 0 and only vary σl . Thereby , it is ensured that the deterministic arm is the better arm . We can then define the number of exploration trials as the pulls from the suboptimal stochastic arm . We observe that meta-learning is capable of yielding agents that behave according to our derived theory of a Bayes optimal agent , which explicitly knows the given lifetime as well as uncertainties σl , σp ( figure 1 ) . Importantly , the meta-learned behavior also falls into two regimes : A regime in which the meta-learned strategy resembles a learning algorithm and a regime in which the recurrent dynamics encode a hard-coded choice of the deterministic arm . Furthermore , the edge between the two meta-learned regimes shifts with the agent ’ s lifetime as predicted by the Bayesian theory . As the lifetime increases , wider ecological niches at higher levels of task complexity become solvable and the strategy of learning profitable . In the Bayesian model , the edge between the two regimes is located at parameter values where the learning strategy and the non-learning strategy perform equally well . Because these two strategies are very distinct , we wondered whether the reward landscape for the memory-based meta-learner has two local maxima corresponding to the two strategies ( figure 2 ) . To test this , we trained N = 1000 networks with different initial conditions , for task parameters close to the edge , but in the regime where the theoretically optimal strategy would be to learn . We then evaluated for each network the number of explorative pulls of the stochastic arm , averaged across 100 episodes . The distribution of the number of explorative pulls across the 1000 networks shows i ) a peak at zero exploration and ii ) a broad tail of mean explorative pulls ( figure 2 ) , suggesting that there are indeed two classes of networks . One class never pulls the stochastic arm , i.e. , those networks adopt a non-learning strategy . The other class learns . For task parameters further away from the edge , this bimodality disappears . The two behavioral regimes are characterized by distinct recurrent dynamics of the trained LSTM agents . The two left-most columns of figure 3 display the policy entropy and hidden state statistics for a network trained on a σl , σp-combination associated with the regime in which learning is the optimal behavior . We differentiate between the case in which the deterministic arm is the better one ( µ < 0 ) and the case in which the second arm should be preferred ( µ > 0 ) . In both cases the agent first explores in order to identify the better arm . Moreover , the hidden dynamics appear to display two different attractors , which correspond to either of the arms being the better choice . The better arm can clearly be identified from the PCA-dimensionality reduced hidden state dynamics ( bottom row of figure 3 ) . The two right-most columns of figure 3 , on the other hand , depict the same statistics for a network that was meta-trained on the regime in which the optimal strategy is not to learn . Indeed , the agent always chooses the deterministic arm , regardless of whether it is the better choice . Accordingly , the network dynamics seem to fall into a single attractor . We examined how these strategies evolve over the course of meta-training and find that there are two phases : After an initial period of universal random behavior across all conditions , the distinct behavioral regimes emerge ( supplementary figure 9 ) . We note that this observation may be partially caused by the linear annealing of the entropy regularization coefficient in the actor-critic objective which we found to be crucial in training the networks . In summary , we observe that the meta-learned strategy shows a highly nonlinear , partially discontinuous dependence on task parameters . In transition regions between strategies , we find local maxima in the reward landscape that correspond to different learning strategies . In the simple bandit setting , these local maxima correspond to a learning and a non-learning strategy , respectively , hence providing a minimal model for a sharp nature-nurture trade-off . Next , we investigate whether these insights generalize to more complex domains by studying spatial reasoning .

Summary: This paper observes that in meta-RL (and evolutionary biology), sometimes it is advantageous to learn behaviors that adapt to the particular task, while other times not adapting to the task, and instead relying on a task-agnostic “hard-coded” behavior is sufficient. While much meta-RL research typically focuses on the former setting, this paper studies when it is not necessary to learn adaptive behaviors. Specifically, this paper presents three main findings: (i) whether or not it is optimal to learn adaptive behavior strongly depends on the horizon of the task and complexity of learning such adaptive behaviors — if the horizon is too short, or if the adaptive behavior requires complex exploration, then exploring the new task to learn adaptive behaviors may not be worth it; (ii) existing meta-RL agents are capable of choosing not to learn adaptive behaviors, when it is optimal to do so; (iii) existing meta-RL agents generalize poorly to tasks with varying horizon-lengths.