Ecological Reinforcement Learning

1 INTRODUCTION . A central goal in current AI research , especially in reinforcement learning ( RL ) , is to develop algorithms that are general , in the sense that the same method can be used to train an effective model for a wide variety of tasks , problems , and domains . In RL , this means designing algorithms that can solve any Markov decision process ( MDP ) . However , natural intelligence – e.g. , humans and animals – exists in the context of a natural environment . People and animals can not be understood separately from the environments that they inhabit any more than brains can be understood separately from the bodies they control . In the same way , perhaps a complete understanding of artificial intelligence can also only be obtained in the context of an environment , or at least a set of assumptions on that environment . There has been comparatively little study in the field of reinforcement learning to understand how properties of the environment impact the learning process for complex RL agents . Many of the environments used in modern reinforcement learning research differ in fundamental ways from the real world . First , standard RL benchmarks , such as the arcade learning environment ( ALE ) ( Bellemare et al. , 2013 ) and Gym ( Brockman et al. , 2016 ) are episodic , while natural environments are continual and lack a “ reset ” mechanism , requiring an agent to learn through continual interaction . Second , most of these environments include detailed reward functions that not only correspond to overall task success , but also provide intermediate learning signal , thus shaping the learning process . These signals can aid in learning , but they can also bias the learning process . Third , the environments are typically static , in the sense that only the agent ’ s own actions substantively impact the world . In contrast , natural environments are stochastic and dynamic : an agent that does nothing will still experience many different states , due to the behavior of natural processes and other creatures . In this paper , we aim to study how these properties affect the learning process . At the core of our work is the concept of ecological reinforcement learning : the idea that the behavior and learning dynamics of an agent , like that of an animal , must be understood in the context of the environment in which it is situated . We therefore study how particular properties of the environment can facilitate or harm the emergence of complex behaviors . We focus our attention on the three properties outlined above : ( 1 ) continual , non-episodic environments where the agent must learn over the course of one “ lifetime , ” ( 2 ) environments that lack detailed reward shaping , but instead provides a reward signal based on a simple “ fundamental drive , ” ( 3 ) environments that are inherently dynamic , evolving on their own around the agent even if the agent does not take meaningful or useful actions . We study how each of these properties affects the learning process . Although on the surface these properties would seem to make the learning process harder , we observe that in some cases , they can actually make reinforcement learning easier . The degree to which these properties make learning easier is highly dependent on the degree of scaffolding that is provided by an environment . For example , an agent tasked with collecting and making food pellets might struggle to learn if it must first complete a complex sequence of actions . However , if food pellets are initially plentiful , the agent can first learn that food pellets are rewarding , and then gradually learn to make them out of raw ingredients as the initial supply becomes scarce . This provides a natural scaffolding and curriculum without requiring manual reward engineering . More generally , “ environment shaping ” can be used as a way to craft the agent ’ s curriculum without modifying its reward function . This benefit is counter-balanced by the fact that non-episodic learning is inherently harder – the resets in episodic tasks provide a more stationary learning problem , preventing the agent from getting “ stuck ” due to a bad initial policy . However , natural environments can also counteract this difficulty : a dynamic environment that gradually changes on its own can provide a sort of “ soft ” reset that can mitigate the difficulties of reset-free learning , and we observe this empirically in our experiments . We illustrate some of these ideas in Figure 1 . The contribution of this work is an empirical study of how the properties of environments – particularly properties that we believe reflect realistic environments – impact reinforcement learning . We study the effect of ( 1 ) continual , non-episodic learning , ( 2 ) learning with and without reward shaping , and ( 3 ) learning in dynamic environments that evolve on their own . We find that , though each of these properties can make learning harder , they can also be combined in realistic ways to actually make learning easier . We also provide an open-source environment for future experiments studying “ ecological ” reinforcement learning , and we hope that our experimental conclusions will encourage future research that studies how the nature of the environment in which the RL agent is situated can facilitate learning and the emergence of complex skills . This exercise helps us determine which types of algorithmic challenges we should focus our development efforts towards in order to solve natural environments that agents might encounter . 2 RELATED WORK . Solving general RL problems can be extremely hard in general ( Kakade & Langford , 2002 ) . Reward shaping is a common technique to guide learning ( Ng et al. , 1999 ; Devlin & Kudenko , 2012 ; Brys et al. , 2015 ) but is usually hand crafted and must be carefully designed by human experts ( Griffith et al. , 2013 ) . Shaping the reward may also lead to suboptimal solutions , as it alters the objective of the learning problem . Curriculum learning can be used to first provide the agent with easier tasks , followed by more challenging tasks ( Bengio et al. , 2009 ; Graves et al. , 2017 ; Randløv & Alstrøm , 1998 ; Wang et al. , 2019a ; Yu et al. , 2018 ; Heess et al. , 2017 ) . Curriculum learning can also be viewed in the context of multiple learning agents in an adversarial or cooperative setting ( Silver et al. , 2016 ; Al-Shedivat et al. , 2017 ; Sukhbaatar et al. , 2017 ; Omidshafiei et al. , 2018 ) or where the curriculum is automatically generated ( Florensa et al. , 2017b ; a ; Riedmiller et al. , 2018 ; Wang et al. , 2019b ) . The “ environment shaping ” that we study in our experiments can be viewed as a kind of curriculum learning , and we argue – and show empirically – that this environment shaping approach can in some cases be more effective than more commonly used reward shaping . Improved exploration methods are a possible solution to solving sparse reward tasks . Prior work has used approximate state-visitation counts ( Tang et al. , 2016 ; Bellemare et al. , 2013 ) , information gain , or prediction error ( Houthooft et al. , 2016 ; Pathak et al. , 2017 ) , or model ensemble uncertainty ( Osband et al. , 2016 ) . A recent work ( Ecoffet et al. , 2019 ) maintains a set of novel states and first returns to the novel states before exploring from this frontier . Our work could be combined with an exploration method , however , this work indicates that sparse reward tasks can be solved with an appropriately shaped environment . Prior work on RL without resets has focused on safe exploration ( Moldovan & Abbeel , 2012 ; Chatzilygeroudis et al. , 2018 ) or learning a policy to reset the environment ( Eysenbach et al. , 2017 ; Han et al. , 2015 ) . Even-Dar et al . ( 2005 ) studies reset free RL in POMDPs and implements a homing strategy which approximately resets the agent . Rather than trying to convert the reset-free problem to one that looks more like a scenario with resets , our experiments study under which conditions reset-free learning can actually be easier , and show that dynamic environments – which we argue better reflect the real world – actually make learning without resets easier . Learning in non-episodic settings has been studied from the perspective of continual learning Ring ( 1997 ) , where a number of tasks are learned in sequence . These algorithms typically consider the problem of “ catastrophic forgetting ” ( Mccloskey , 1989 ; French , 1999 ) , where previously learned tasks are forgotten while learning new tasks . To solve this problem , algorithms use methods such as explicit memorization ( Rusu et al. , 2016 ; Schwarz et al. , 2018 ) , generative replay ( Shin et al. , 2017 ) and explicit weight regularization ( Kirkpatrick et al. , 2016 ; Kaplanis et al. , 2018 ) . These works assume that resets and task boundaries are available whereas we assume that the agent is unable to reset . There has also been work on building more complex tasks in large diverse worlds with Mujoco ( Todorov et al. , 2012 ; Singh et al. , 2019 ; Yu et al. , 2019 ) , Malmo ( Johnson et al. , 2016 ; Guss et al. , 2019 ) , DeepMind Lab ( Beattie et al. , 2016 ) , and many others , however , again , these environments are studied in the context of episodic-learning . 3 PROPERTIES OF NATURAL ENVIRONMENTS . In contrast to most simulated environments that are used for reinforcement learning experiments ( Brockman et al. , 2016 ) , agents learning in natural environments experience a continual stream of experience , without episode boundaries . The typical reward function engineering that is often employed in reinforcement learning experiments is also generally unavailable in the real world , where agents must rely on their own low-level perception to understand the world . Finally , natural environments change on their own , even when the agent does not follow a coordinated or intelligent course of action . This dynamism can create additional challenges , but can also facilitate learning , mitigating some of the issues due to non-episodic and non-resettable learning settings . In this paper , our aim is to study how these aspects of the environment impact the performance of reinforcement learning agents . We term this approach ecological reinforcement learning , in that it deals specifically with the relationship between properties of the environment and the reinforcement learning agent , rather than studying reinforcement learning algorithms in the general case , regardless of the particular properties of the learning environment . We believe that the properties outlined above are broadly reflected in real-world settings , and are often absent in simulated reinforcement learning benchmarks . In this section , we discuss each of these properties , and formulate our hypotheses about how these properties might influence learning . Continual non-episodic learning . In the real world , all learning must at some level be nonepisodic : though we may instrument environments to make them appear episodic , there is always a single underlying temporal process . In general , this makes the learning problem harder : when the agent is not reset to randomly chosen initial states , mistakes early on in training can put it into undesirable situations , from which it might be harder to recover and – more importantly – harder to learn . A non-episodic learning process is non-stationary , and the agent can become trapped in difficult regions of the state space . Hypothesis 1 : Non-episodic learning is more difficult than episodic learning because the agent must handle a non-stationary learning problem , and can become trapped in difficult states . We will study this hypothesis in our experiments , and show how some of the other properties of natural environments can help alleviate this difficulty . Sparse rewards and environment shaping . While in principle RL algorithms can handle relatively uninformative rewards , in practice reward shaping is often an essential tool for getting RL methods to acquire effective policies . For example , an agent that must learn a policy to collect resources to make an axe ( see Figure 2 ) might make use of a reward function that specifies the distance to the nearest resource , or at least provides a small reward for each resource obtained , as opposed to a reward given only for obtaining the final goal . However , well-shaped rewards are generally not available and difficult to provide in the real world , since they require knowledge of privileged state variables ( e.g. , positions of objects ) or the process by which the task must be completed ( e.g. , required resources ) , both of which should in principle be learned automatically by the agent . Furthermore , reward shaping might introduce bias , since the optimal policy for a shaped reward may not in fact be optimal for the original task reward . On the other hand , agents in the real world do not learn in a vacuum : even for humans and animals , it is reasonable to assume a reasonably cooperative environment that has been set up so as to facilitate learning . For humans , this kind of “ scaffolding ” is often provided by other agents ( e.g. , parents and teachers ) . But even without other agents , natural environments might provide automatic scaffolding – e.g. , an animal might find apples that fell from a tree , and thereby learn that apples are a source of food . Once the fallen apples are exhausted , the animal might use its knowledge of the value of apples to learn to climb the tree to obtain the apples on its own . This kind of “ environment shaping ” could serve as a tool for guiding the learning process , without the bias or manual engineering inherent in reward shaping . Hypothesis 2 : Environment shaping can enable agents to learn even with simple sparse rewards , and can in fact result in more proficient policies if applied correctly , as opposed to reward shaping . Dynamic environments . Standard reinforcement learning benchmark tasks are typically situated in static environments ( Brockman et al. , 2016 ; Bellemare et al. , 2013 ) , in the sense that the environment does not change substantially unless the agent takes a coordinated course of action . On the other hand , real-world settings are typically dynamic , in the sense that the environment changes even if the agent does not follow any coordinated course of action : animals will move around , times of day will change , seasons will change , etc . Dynamic environments present their own challenges , but they can also facilitate learning , by automatically exposing the agent to a wide variety of situations . Hypothesis 3 : While dynamic environments could make learning more difficult , in fact they can alleviate some of the challenges associated with non-episodic learning , by providing the agent with a variety of learning conditions even in the absence of coordinated and intelligent behavior ( as is the case , e.g. , early on in training ) .

Summary: The authors study what they refer to as ecological reinforcement learning, defined as the interaction between properties of the environment and the reinforcement learning agent. They introduce environments with characteristics that reflect natural environments: non-episodic learning, uninformative reward signals, and natural dynamics that cause the environment to change. These factors are shown to significantly affect the learning progress of RL agents and, unexpectedly, the agents can sometimes learn more efficiently in these more challenging conditions.