Ecological Reinforcement Learning
Authors: John D. Co-Reyes
Suvansh Sanjeev
Glen Berseth
Abhishek Gupta
Sergey Levine
Venue:  (2020)
Keywords: non-episodic
environment analysis
reward shaping
curriculum learning

Abstract: Reinforcement learning algorithms have been shown to effectively learn tasks in a variety of static, deterministic, and  simplistic environments, but their application to environments which are characteristic of dynamic lifelong settings encountered in the real world has been limited. Understanding the impact of specific environmental properties on the learning dynamics of reinforcement learning algorithms is important as we want to align the environments in which we develop our algorithms with the real world, and this is strongly coupled with the type of intelligence which can be learned. In this work, we study what we refer to as ecological reinforcement learning: the interaction between properties of the environment and the reinforcement learning agent. To this end, we introduce environments with characteristics that we argue better reflect natural environments: non-episodic learning, uninformative ``fundamental drive'' reward signals, and natural dynamics that cause the environment to change even when the agent fails to take intelligent actions. We show these factors can have a profound effect on the learning progress of reinforcement learning algorithms. Surprisingly, we find that these seemingly more challenging learning conditions can often make reinforcement learning agents learn more effectively. Through this study, we hope to shift the focus of the community towards learning in realistic, natural environments with dynamic elements.

least set of assumptions on that Under review as a conference paper at ICLR 2020
ECOLOGICAL REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Reinforcement learning algorithms have been shown effectively learn tasks in a variety of static, deterministic, and simplistic environments, but their application environments which are characteristic of dynamic lifelong settings encountered in the real world has been limited. Understanding the impact of specific envi- ronmental properties on the learning dynamics of reinforcement learning algo- rithms is important as we want to align the environments in which we develop our algorithms with the real world, and this strongly coupled with the type of intelligence which can be learned. In this work, we study what we refer as ecological reinforcement learning: the interaction between properties of the environment and the reinforcement learning agent. To this end, we introduce en- vironments with characteristics that we argue better reflect natural environments: non-episodic learning, uninformative "fundamental drive" reward signals, and nat- ural dynamics that cause the environment to change even when the agent fails to take intelligent actions. We show these factors have profound effect the learning progress of reinforcement learning algorithms. Surprisingly, we find that these seemingly more challenging learning conditions can often make reinforce- ment learning agents learn more effectively. Through this study, we hope to shift the focus of the community towards learning realistic, natural environments with dynamic elements.
INTRODUCTION
central goal in current AI research, especially in reinforcement learning (RL), to develop algo- rithms that are general, in the sense that the same method can be used to train an effective model for wide variety of tasks, problems, and domains. In RI., this means designing algorithms that can solve any Markov decision process (MDP). However, natural intelligence e.g., humans and animals natural environment. People and animals cannot be understood separately
exists in the context from the environments that they inhabit any more than brains can be understood separately from the bodies they control. In the same way, perhaps a complete understanding of artificial intelligence also only be obtained in the context of an environment, environment.
There has been comparatively little study in the field of reinforcement learning understand how properties of the environment impact the learning process for complex RL agents. Many of the envi- ronments used in modern reinforcement learning research differ in fundamental ways from the real world. First, standard RL benchmarks, such as the arcade learning environment (ALE) (Bellemare) al. 2013) and Gym (Brockman et al. 2016) are episodic, while natural environments contin- ual and lack a "reset" mechanism, requiring an agent to learn through continual interaction. Second, most of these environments include detailed reward functions that not only correspond to overall task success, but also provide intermediate learning signal, thus shaping the learning process. These signals can aid learning, but they can also bias learning process. Third, the environments typically static, in the sense that only the agent's In contrast, natural environments are stochastic and dynamic: an agent that does nothing will still experience many different states, due the behavior of natural processes and other creatures.
actions substantively impact the world.
In this paper, we aim to study how these properties affect the learning process. At the core of our work is the concept of ecological reinforcement learning: the idea that the behavior and learning dynamics of an agent, like that of an animal, must be understood in the context of the environment in which it is situated. We therefore study how particular properties of the environment can facilitate

---

Under review as a conference paper at ICLR 2020
Episodic
Non- Episodic
Reward Shaping
Sparse Reward
Env. Shaping
Static
Dynamic
reset
Figure 1: We study three properties of realistic environments that can have large effect on the difficulty of reinforcement learning: (left) non-episodic learning, where the agent is reset automatically the same initial state distribution, but must handle whatever situation it puts itself in; (middle) environment shaping as an alternative to reward shaping, where the agent has single, sparse reward, but the environment is varied so as to provide curriculum (e.g. due to its natural dynamics, or by cooperative teacher agent): (right) dynamic environments, where the environment changes due the actions of other agents and natural phenomena, even the agent does not take coordinated course action such dynamic phenomena can, as we will show, alleviate some of the difficulties in non- non-episodic learning.
or harm the emergence of complex behaviors. We focus our attention on the three properties outlined above: (1) continual, non-episodic environments where the agent must learn over the course of one "lifetime," " (2) environments that lack detailed reward shaping, but instead provides a reward signal based on simple "fundamental drive," (3) environments that are inherently dynamic, evolving on their own around the agent even if the agent does not take meaningful or useful actions. We study how each of these properties affects the learning process. Although the surface these properties would seem to make the learning process harder, we observe that in some cases, they can actually make reinforcement learning easier.
The degree to which these properties make learning easier is highly dependent on the degree of scaf- folding that is provided by an environment. For example, an agent tasked with collecting and making food pellets might struggle to learn if it must first complete a complex sequence of actions. How- ever, if food pellets are initially plentiful, the agent can first learn that food pellets are rewarding, and then gradually learn to make them out of raw ingredients as the initial supply becomes scarce. This provides a natural scaffolding and curriculum without requiring manual reward engineering. More generally, "environment shaping" can be used as way to craft the agent's 's curriculum without modifying its reward function. This benefit is colnter- by the fact that non-episodic learn- ing is inherently harder the resets in episodic tasks provide more stationary learning problem, preventing the agent from getting "stuck" due to bad initial policy. However, natural environments also counteract this difficulty: dynamic environment that gradually changes its can provide a sort of "soft" reset that can mitigate the difficulties of reset-free learning, and we observe this empirically in our experiments. We illustrate some of these ideas in Figure]
The contribution of this work an empirical study of how the properties of environments partic- ularly properties that we believe reflect realistic environments impact reinforcement learning. We study the effect of (1) continual, non-episodic learning, (2) learning with and without reward shap- ing, and (3) learning in dynamic environments that evolve on their own. We find that, though each of these properties can make learning harder, they can also be combined in realistic ways to actually make learning easier. We also provide an open-source environment for future experiments studying "ecological" reinforcement learning, and we hope that our experimental conclusions will encourage future research that studies how the nature of the environment which the RL. agent is situated can facilitate learning and the emergence of complex skills. This exercise helps us determine which types algorithmic challenges we should focus our development efforts towards in order to solve natural environments that agents might encounter.
2 RELATED WORK
Solving general RL. problems can extremely hard in general Kakade & Langford 2002 Reward shaping is a common technique to guide learning (Ng et al. 1999] Devlin & Kudenko, 2012 Brys (ct al.) 2015) but is usually hand crafted and must be carefully designed by human experts (Griffith et al. 2.13). Shaping the reward may also lead to suboptimal solutions, as it alters the objective of the learning problem. Curriculum learning can be used to first provide the agent with easier tasks, followed by more challenging tasks (Bengio et al. 2009; Graves al. 2017: Randløv & Alstrom 1998 Wang et al. 2019a, Yu et al.] 2018: Heess et al. 2017). Curriculum learning can also be
viewed in the context of multiple learning agents in adversarial cooperative setting (Silver
2

---

Under review as a conference paper at ICLR 2020
et al. 2016 Al-Shedivat et al., 2017; Sukhbaatar et al. 2017: Omidshafiei al. 2018 where the curriculum is automatically generated (Florensa et al. 2017b a Riedmiller et al. 2019b). The "environment shaping" that we study in experiments can be viewed as kind of curriculum learning, and we argue- - and show empirically that this environment shaping approach can in cases be more effective than more commonly used reward shaping.
2018 Wang et al.
Improved exploration methods are a possible solution to solving sparse reward tasks. Prior work has used approximate state-visitation counts (Tang al. [0166] Bellemare al. 2013), information gain, or prediction error (Houthooft et al.] 2016 Pathak el al.
2017), or model ensemble uncer- maintains set of novel states and
tainty (Osband al., 2016). A recent work (Ecoffet al. 2019
first returns to the novel states before exploring from this frontier. Our work could be combined with
an exploration method, however, this work indicates that sparse reward tasks can be solved with an appropriately shaped environment.
Prior work on RL without resets has focused on safe exploration (Moldovan Abbeel 2012 | Chatzi- lygeroudis et al., (0118) or learning policy to reset the environment (Eysenbach et al. 2017 Han et al. 2015). Even-Dar et al. (2005) studies reset free RL in POMDPs and implements a homing strategy which approximately resets the agent. Rather than trying to convert the reset-free problem to one that looks more like a scenario with resets, our experiments study under which conditions reset-free learning can actually be easier, and show that dynamic environments which argue better reflect the real world - actually make learning without resets easier.
Learning in non-episodic settings has been studied from the perspective of continual learning Ring [1997), where a number of tasks are learned in sequence. These algorithms typically consider the problem of "catastrophic forgetting" (Mceloskey 1989 French 1999). where previously learned tasks are forgotten while learning new tasks. To solve this problem, algorithms use methods such as explicit memorization (Rusu et al.) 2016 Schwarz et al. 2018)) generative replay (Shin al. 2017) and explicit weight regularization (Kinkpatrick et al. 2016) Kaplanis et al., 2018). These works assume that resets and task boundaries are available whereas we assume that the agent unable to reset. There has also been work on building more complex tasks in large diverse worlds with Mujoco (Todorov et al. 2012 |inghh et al. [2019] Yu et al.) (019), Malmo (Johnson et al.] 2016] Guss al.] 2019). DeepMind Lab (Beattie et 201,,) and many others, however, again, these environments are studied the context of episodic-
3 PROPERTIES OF NATURAL ENVIRONMENTS
In contrast to most simulated environments that used for reinforcement learning experi- ments (Brockman et al., 2016)). agents learning in natural environments experience a continual stream of experience, without episode boundaries. The typical reward function engineering that is often employed in reinforcement learning experiments also generally unavailable in the real world, where agents must rely their low-level perception to understand the world. Finally, natural environments change on their own, even when the agent does not follow a coordinated or intelligent course of action. This dynamism can create additional challenges, but can also facilitate learning, mitigating some of the issues due to non-episodic and non-resettable learning settings. In this paper, our aim is to study how these aspects of the environment impact the performance of rein- forcement learning agents. We term this approach ecological reinforcement learning, in that it deals specifically with the relationship between properties of the environment and the reinforcement learn- ing agent, rather than studying reinforcement learning algorithms in the general case, regardless of the particular properties the learning environment. We believe that the properties outlined above are broadly reflected in real-world settings, and are often absent in simulated reinforcement learn- ing benchmarks. In this section, we discuss each of these properties, and formulate our hypotheses about how these properties might influence learning.
Continual non-episodic learning. In the real world, all learning must at some level be non- episodic: though we may instrument environments to make them appear episodic, there is always single underlying temporal process. In general, this makes the learning problem harder: when the agent is not reset to randomly chosen initial states, mistakes early on in training can put into undesirable situations, from which it might be harder recover and more importantly harder
3

---

Under review as a conference paper at ICLR 2020
to learn. A non-episodic learning process is non-stationary, and the agent can become trapped in difficult regions of the state space.
Hypothesis 1: Non-episodic learning is more difficult than episodic learning because the agent must handle a non-stationary learning problem, and can become trapped in difficult states. We will study this hypothesis in our experiments, and show how some of the other properties of natural environ- ments can help alleviate this difficulty.
Sparse rewards and environment shaping. While in principle RL algorithms can handle rela- tively uninformative rewards, in practice reward shaping is often an essential tool for getting RL methods acquire effective policies. For example, an agent that must learn a policy to collect re- sources to make an axe (see Figure() might make use of reward function that specifies the distance to the nearest resource, or least provides a small reward for each resource obtained, as opposed to a reward given only for obtaining the final goal. However, well-shaped rewards are generally not available and difficult to provide in the real world, since they require knowledge of privileged state variables (e.g., positions of objects) the process by which the task must be completed (e.g., required resources), both of which should in principle be learned automatically by the agent. Fur- thermore, reward shaping might introduce bias, since the optimal policy for shaped reward may not in fact be optimal for the original task reward. On the other hand, agents in the real world do not learn in vacuum: even for humans and animals, it is reasonable assume reasonably cooperative environment that has been set up so as to facilitate learning. For humans, this kind of "scaffolding" often provided by other agents (e.g.. parents and teachers). But even without other agents, natural environments might provide automatic scaffolding - e.g., an animal might find apples that fell from a tree, and thereby learn that apples are a source of food. Once the fallen apples are exhausted, the animal might use its knowledge of the value of apples to learn to climb the tree to obtain the apples on its own. This kind of "environment shaping" could serve as a tool for guiding the learning process, without the bias or manual engineering inherent in reward shaping.
Hypothesis 2: can in fact result in more proficient policies if applied correctly, opposed to reward shaping.
Environment shaping can enable agents to learn even with simple sparse rewards, and
Dynamic environments. Standard reinforcement learning benchmark tasks are typically situated in static environments (Brockman et al.] 2016 Bellemare et al.) 2013), in the sense that the envi- ronment does not change substantially unless the agent takes a coordinated course of action. On the other hand, real-world settings are typically dynamic, in the sense that the environment changes even if the agent does not follow any coordinated course of action: animals will move around, times of day will change, seasons will change, etc. Dynamic environments present their own challenges, but they can also facilitate learning, by automatically exposing the agent to wide variety of situations.
Hypothesis 3: While dynamic environments could make learning more difficult, in fact they can alleviate some of the challenges associated with non-episodic learning, by providing the agent with a variety of learning conditions even in the absence of coordinated and intelligent behavior (as the case, e.g., early on in training).
4 EXPERIMENTAL SETUP
To carry out our ecological RL study, we construct three simulated tasks, The simulator for two of them is built on top of the grid-like environment proposed by Chevalier-Boissert al. (2018).
We chose grid- study the aforementioned properties in isolation, without other confounding factors involving high- dimensional observations and representation learning. The goal is not to simulate completely visu- ally realistic and life-like system, but to study those properties of the MDP that will be particularly important in natural environments, and have not been addressed in detail in prior work.
siccrete-action environment over a more complex, high- nigh-dimensional one to
The environment is an N x N grid of tiles, where each tile contains at most one object, but the agent is free to move over any tile and can pick up and carry one object at a time. The agent can use objects in its environment to construct new ones by dropping a carried object onto an existing object. Objects include wood, metal, deer, axe, and food. The agent combine wood with metal to construct an axe and apply the axe to a deer to produce food. The agent consumes resources such food by picking them up. There are movement actions associated with each of the cardinal
4

---

Under review as a conference paper at ICLR 2020
directions, well to pick up drop object. The environment is partially observed, and the agent receives a local egocentric view around it, represented by the shaded region in Figure 2 which is a 5 x x C grid, where C' is the number of object types, and each grid position contains one-hot vector representation the object type. We use the following two tasks in our evaluation, which are illustrated in Figure
Tool-Making Collect wood Combine
Hunting Pickup axe with ase Eat food
wood with
metal
Figure 2: Tasks in our partially observed stochastic environment crafting an axe (left) and hunting a deer (middle). The agent (purple triangle) receives a local observation (shaded gray square) and must interact with objects in the correct sequences to complete the task. In the food collection task in Unity (right), the agent's goal is to collect green food and avoid red poison.
Tool-making:
The agent must combine wood and metal to craft axe. and then pick up the axe. Wood and metal appear at random locations initially, and continue to spawn with some probability time progresses in the dynamic setting.
Hunting: The agent uses an axe to hunt a deer, which produces food that can pick up to eat. Axes and deer appear at random locations. The deer move around in the environment in the dynamic setting, and can appear at different distances from the agent.
In order to study hypotheses in varied settings, additionally investigate these environmental properties, enumerated at the end of this section, in the context of the Food Collector environment available as part of the Unity ML-Agents toolkit provided by Juliani et al. 2018). The environment features a continuous state space with raycast partial observations representing the directional view of the agent. The action space 27-dimensional, with separate action streams corresponding to forward, lateral, and rotational movement, The task is taken as-is, wherein the agent must maximize the number of healthy food items eaten (represented by green spheres) while avoiding consuming poisonous food (red sphere). We modify the environment dynamics and initial conditions within this task setting to investigate our hypotheses.
To study the hypotheses discussed in the previous section, we vary number of properties of these environments, and examine their effect on the learning process:
Non-episodic learning. The non-episodic version of each task does not allow the agent to reset to the initial state, and instead requires it to learn the task effectively over one very long episode, as illustrated in the figure on the right. For example, in the Unity Food Collec- tor task, the agent must learn to continually collect as many healthy food items possible while avoiding the poisonous food across its lifetime. We will compare this against the episodic case. where the agent is reset an initial state distribution after completing the task or after a fixed time horizon. This will allow us to study hypothesis HI.
Episodic
Non- Mon-Episodii
Reward shaping. Reward shaping is often used to as- sist the agent in finding good policy by providing more signal to the agent. The environment allows us experiment with different reward functions, such as shaped distance-based reward function that rewards the agent for how close it is to the nearest resource it needs, and larger bonus for interacting with the right object. A sparse reward function will be based on a simple "fundamental drive:" whether or not the agent has just acquired the axe (in the tool-making task), deer (hunting), food (food collector).
Environment shaping, Instead of shaping rewards, which can be unrealistic and can bias the learn- ing process, instead change the types and distribution of states the agent observes while
5

---

Under review as a conference paper at ICLR 2020
learning. In tasks, from the agent at which the resources (wood, metal, deer, and food) can appear. We can gradually increase this distance the learning progresses, illustrated the figure the right, which shows the deer appearing progressively further away during training, thus inducing a curriculum. This type of environment shaping remove the need for reward shaping, and potentially alleviates its shortcomings. We will use this setting to study hypothesis H2.
scaffold the learning process by controlling the maximum distance
Dynamic vs. static environments. We can con- struct dynamic versions of both of our tasks by vary- ing the probability that the environment changes any given time, regardless of the agent's actions. This captures the fact that natural environments will change their own, regardless of what the agent does: other animals will move around, weather will change, etc. In both grid domains, we define continuous spectrum of dynamic effects, in terms of a dynamic event probability. In the tool-making task, the dynamic effect probability p controls the resource generation, determining the probability that resource will spawn in empty squares no less than two squares away from the agent at each time step. To keep the expected quantity of resources constant throughout the trajectory allow resources to decay (disappear) after lifespan of ! timesteps, where / is the number of instances of this resource that appear the initial environ- p ment. In the hunting task, the dynamic effect probability p is the probability that a deer will move to random adjacent square on each time step, as illustrated the figure above. The static version of these environments will have O, such that resources are in fixed positions and only respawn as needed when the agent completes the task, and deer do move. In the Unity Food Collector environment, the dynamic property is given by the speed at which both healthy and poisonous food move, which was tested across the range of = for the food objects in the Unity engine. The static setting here corresponds to stationary food. By studying dynamic and static environments episodic and non-episodic settings, we can analyze hypothesis H3.
Sterii
Dynamic
= 16, corresponding to the velocities set
4.1 EVALUATION
In order to compare agents trained under the different environment conditions, must construct single consistent evaluation protocol. We use the same agent network architecture and RL algorithm for all experiments, with details provided in Appendix A We evaluate all agents set of validation tasks that are chosen to be as close as possible to the "standard"' RI. setting, which is episodic and static. We generate 100 validation tasks by randomly generating environments with varying initial resource and agent locations. Performance is measured by the proportion of validation tasks solved. We first study non-episodic learning and dynamic vs. static environments by varying these training settings with sparse reward and no environment shaping. We will then study reward shaping and environment shaping the dynamic non-episodic setting. The training settings are:
Static episodic: In this setting, is set to O so positions of the resources will be static unless the agent moves them. When the agent finishes the task, the environment reset to random initial configuration. The environment is also reset when the episode length reaches 200.
Here, p 0 and there are no resets. When the agent has used up the available resources, more resources generated randomly.
Static non-episodic:
Dynamic non-episodic: This setting explores the lifelong case with a changing environment, where the dynamic property is varied between 0 and 1. For tool-making, p is the probability of resources spawning at each timestep in a random location. There is initially two of each type of resource. This environment lifelong and does reset when the agent completes the task.
Dynamic episodic: This is the same as the dynamic non non-episodic version except that the environ- ment is reset to random initial configuration when the agent completes the task.

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTAL RESULTS
we perform experiments where we train RL. agents on the tasks described above and vary different properties of the environment during training.
To study the hypotheses in Section S
5.1 NON-EPISODIC LEARNING IN DYNAMIC AND STATIC ENVIRONMENTS
In the real world, environments are generally dynamic and non-episodic, meaning that the agent is never reset to initial state distribution, and many parts of the environment are changing without the agent's intervention. In this section we study the effects of both a dynamic, changing environment and the ability reset on the learning agent, corresponding to hypotheses H/ and H3. These settings will use sparse reward and no environment shaping.
Vorying wighe spenttr Morting
Dysamic Npr Cp--Caiidi siral spici
Turinng Rec
Tukking Londu
Weriing Peo.ggeePppeeeeeee Delertee
Syanchonnd honcchonppsee
Figure 3: Proportion of validation tasks solved in cach setting. Agents Icarning in static non-episodic en- vironments struggle to learn useful behaviors, while agents learning dynamic non-episodic environments are substantially more successful. Episodic learning is easier than non-episodic learning on the first task, but non-episodic learning dynamic environments is almost as effective as episodic learning on the hunting task.
Effect of resets on learning, In regard to HI, we find that learning tasks in static, reset-free, non- episodic environments difficult. Figure we compare the performance of the agent trained in each of the four conditions. Recall that all evaluations are conducted in the same setting, with resets and static environments, regardless of how the agent is actually trained. For the tool-making task, observe that removing resets makes learning more difficult in the typical static case. The agent trained in static environment without resets obtains the lowest performance (0% evaluation tasks solved). Adding in resets the static case helps with performance (12%). In the static environ- ment, we observe that the agent frequently becomes stuck in corners of the map or in areas with no resources.
The results indicate that disabling resets makes the standard static learning condition substantially harder. Indeed, the static reset agent is unable to learn effectively for either task, even though the static episodic agent does learn the task to moderate proficiency. However, making the environ- ment dynamic substantially improves performance, in both the episodic and non-episodic setting, as shown in Figure 3 These results suggest that dynamic environments to large extent alleviate the challenges associated with non-episodic learning, confirming hypothesis H3. The lesson that we might draw from this is that, although individual properties of natural environments (such as non-episodic learning) can make the learning process harder, combining these properties (i.e., as in the non-episodic dynamic setting) can actually alleviate these challenges, since the dynamics of the environment naturally cause the agent to experience a variety of different situations, even before has learned to take meaningful and coordinated actions.
Dynamic environments and non-episodic learning. We conclude that, when resets are not avail- able, dynamic environments can also help with non-episodic RL. In this section, study how the frequency of dynamic effects impacts learning. In Figure we can see that making the environment
More details available at httpttt///// httpps:///its.... googl www.cccwviee/eee....e... eco oo cal-ra
7

---

Under review as a conference paper at ICLR 2020
Cynuriic Ablition TockMakknn Task
Dyna Ablittinn Hurting
Saate
Act Pros Hot Pizs =02 Prane
05 "Rossnu -0dt
ressuuee oe mood....
Training Epochs (5000 mestapo
100
150
203
Training Epocho (500
100
120
251 20
Figure 4: Proportion of validation tasks solved as the dynamic effect probability (i.e., resource probability and deer movement probability) is varied in the non-episodic setting. For all tasks, the standard static environment does allow for effective learning, but numher of dynamic environment variants allow the agent leam the task successfully. Evaluation is still carried out in static environment.
more dynamic increases performance to 70%, compared to the static non-episodic case (0%). Hav- ing both dynamic environment and resets achieves the highest performance (95%), indicating that both are helpful on their own. However, an environment that is too dynamic hinders performance, as we observe in Figure where dynamic effect probability of 0.5 performs worse than 0.1. This implies that environment dynamics represent a tradcoff: the environment should be stable enough for the agent to learn meaningful behavior, but dynamic enough to present interesting situations. This in some sense resembles the tradeoff typically encountered with exploration constants, e.g. in e-greedy exploration. From this experiment, particular choice of environment settings can greatly impact learning performance. To understand this better, we analyze and compare the effect of "environment shaping," as defined in Section in the following subsection.
can conclude that, although H3 is generally true, the
5.2 REWARD SHAPING AND ENVIRONMENT SHAPING
Next, we study how reward shaping and environment shaping compare in terms of their capacity to assist learning in reset-free environments, to study hypothesis H2. We perform experiments where we train RL agents on the same tool-making task, in the non-episodic case. We compare shaping the environment during training to shaping the reward function during training. The training envi- ronments have one of each resource, spawning at locations sampled uniformly over the world every 20 timesteps which is much more difficult setting than the ones used in the previous section. We evaluate a range of reward and enviromment shaping conditions. For all methods the agent is given reward of 100 each time it completes the task. Resource interaction means picked up a required resource resource. The methods are:
No shaping with sparse reward: Resources spawn uniformly and the agent receives task comple- tion reward.
Distance reward shaping: The agent is provided with dense distance- based reward which grants (-0.01 * distance to nearest required resource) and (1) for resource interaction.
One-time reward shaping: This is less dense than the distance based reward. The agent is given reward (1) for resource interaction and - -100 for dropping the resource. This resource reward is only granted the first time and resets every time the task is completed.
Environment shaping with subgoal reward: We design simple shaping method that gradually increases the distance away from the agent which resources spawn. This distances increases linearly until the resources are placed uniformly over the grid world, as in the shaped reward version. We tried various schedules and found schedule that starts at a distance of 2 and increases by I every 1e5 environment steps to work well. The subgoal reward is simpler than the one-time reward. The agent is given reward (1) for resource interaction. This reward does not keep track of previous object interactions and grants the bonus multiple times.
Environment shaping with sparse reward: We shape the environment as in the previous method but only use the task reward.
Environment shaping can replace reward shaping. We compare performance of these methods in Figure 5 for both the episodic and non-episodic case. We find that, in the episodic case, environment shaping works well and outperforms reward shaping in the long run. Improper reward shaping can alter the optimal policy, thereby biasing learning and resulting in a solution that is worse
8

---

Under review as a conference paper at ICLR 2020
Sneping Thol-kieing
Humming Spincdes
Distanoo reware shaplng
Dusaane reward
shoping
shaptg
Erry propping sparse reward
shaplng sparee shaplig singge
Ein groppin subgoot reward
Shaplng Methocs for Tcol Makingg Non Epicodic
Shop Nethods for Hunding Eptoocic
Dersonce cheping
Omocline roward shaplig
chaping
No shaping
aainng shuplin spurscrevend Ery shapliee sutapee rocard
Envishu shoppingg ssorse reward
subguat
The Ezochs (50C timaaopp)
Coocht 150CE mesttpp
Shaping Methers
Diszance shaping
shaping Erw rorment shipping
Tou ving Foodba (106 tewerpp
Figure 5: Proportion of validation tasks solved for enviromment shaping with sparse reward, subgoal reward, and different forms of reward shaping. We see that environment shaping can obtain better final performance than reward shaping.
with respect to the desired performance measure, which typically corresponds to the sparse reward. These behaviors are shown in Appendix [
Interestingly, we find that environment shaping works better for the more difficult task of Hunting- As task complexity grows, so does the difficulty of constructing
unbiased shaped reward for the tasks. this case, environment shaping benefits from its ease of use and general applicability to various tasks. Further experiments on harder environment are detailed in Appendix
Human guided environment shaping. In the real world, environment shaping can be done by doing what already do for other human learners (c.g.. children and pupils in school): arranging the environment to conducive to learning. We also conducted a study of this setting, by having an actual human user interactively specify how the environment should be altered to facilitate the agent's 's learning process. In this case, the human user was able to provide an environment shaping schedule that outperformed the we specified manually. Full results for this experiment are provided in Appendix B These results suggest that environment shaping is not only effective, but is also readily intuitive for human user to specify interactively, suggesting that can be a viable way to provide guidance to reinforcement learning agents and may be intuitive to specify, in comparison with reward shaping, which can times be difficult and counter-intuitive.
5.3 EXPERIMENTAL CONCLUSIONS
Our experiments confirm hypothesis H/ by showing that non-episodic is indeed substantially harder than episodic learning in standard static environments. However, our experiments also show that, for all considered tasks, introducing dynamic effects can allow non-episodic learning to succeed, in some cases to a degree that is comparable to the episodic setting, confirming hypothesis H3. How- ever, this result is sensitive to the degree of stochasticity, suggesting that the specific dynamics and design of the environment has a large impact on learning. Based on this conclusion, we study how shaping the environment influences the learning process, and conclude that appropriate environment

---

Under review as a conference paper at ICLR 2020
shaping can, in tasks, supplant the need for more traditional reward shaping, confirming hypoth- esis H2. We further show the human users can effectively select environment shaping schedules manually, suggesting that this is intuitive way guide the learning of reinforcement learning agents. Our conclusions support the notion that ecological reinforcement learning the study of the interaction between an RI agent and its environment is important topic for further study.
DISCUSSION
We study how certain properties of natural environments namely, non-episodic learning without resets, simple "fundamental drive" reward functions, and dynamic environments that evolve on their own even when the agent does not actively intervene affect the reinforcement learning process. We use the term ecological reinforcement learning to refer to this sort of study, which aims to analyze interactions between RL agents and the environment which learning occurs. Although these properties by themselves tend make learning harder, find that environments that exhibit several these traits can actually be easier to learn in, and agents trained in such settings can actually outperform agents trained in more conventional episodic settings on the same evaluation tasks. We conclude that in dynamic environments, the variability of situations created by the environment's dynamics and simple rewards that are difficult for the agent exploit can create kind of natural curriculum that guides an agent through the emergence of increasingly complex behaviors.
Aside from these potentially surprising observations, the framework ecological reinforcement learning also points to a new way to approach the design of RL agents. While reward function design is typically considered the primary modality for specifying tasks RI. agents, ecological reinforcement learning suggests that the form and structure of the environment can help to guide the emergence and specification of skills, Combined with the guidance and curricula afforded by natural environments, this suggests that studying and systematizing the interaction between RL agents and various environment properties is an important and interesting direction for future research.
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning nonstationary and competitive environments. ArXiv, abs/1710.03344, 2017.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, Julian Schrittwieser, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/1612.03801, 2016. URL http:://arxxt www.o////aooo.. abs/ 1612 03801
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael H. Bowling. The arcade learning environment: An evaluation platform for general agents. In J. Artif. Intell. Res., 2013.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML. 2009.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. ArXiv, abs/1 (0//16006014400 2016.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E. Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In IJCAI, 2015.
Konstantinos Chatzilygeroudis, Vassilis Vassiliades, and Jean-Baptiss Mouret. Reset-free trial- and-error learning for robot damage recovery. Robotics and Autonomous Systems, 100:236-250, 2018.
Maxime Chevalier-Boisvert Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: pettts:/giitee h/tp://gtttt/////// com maximecbiieee.eem ggmggygmmmmm sym-minigrid 2018.
Sam Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In AAMAS, 2012.
10

---

Under review as a conference paper at ICLR 2020
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. ArXiv, abs/1901.109955, 2019.
Eyal Even-Dar, Sham M. Kakade, and Yishay Mansour. Reinforcement learning in pomdps without resets. In IJCAI, 2005.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. ArXiv, abs/1711.06782, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. ArXiv, 0ass/7055003666 2017a.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In CoRL, 2017b.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Alex Graves, Mare G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. ICML, 2017.
Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles Lee Isbell, and Andrea Lockerd Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In NIPS, 2013.
William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanic Milani, Sharada Mohanty, Diego Perez Licbana, Ruslan Salakhutdinov, Nicholay Topin, et al. The MineRL competition sample efficient reinforcement learning using human priors. NeurlPS Competition Track, 2019.
Weigiao Han, Sergey Levine, and Pieter Abbeel. Learning compound multi-step controllers under unknown dynamics. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6435-6442, 2015.
Nicolas Manfred Otto Heess, TB Dhruva, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. ArXiv, fbs//107.002288 2017.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In NIPS, 2016.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artifi- cial intelligence experimentation. In Proceedings of the Twenty-Fifth International Joint Confer- on Artificial Intelligence, IJCAI, 2016.
Arthur Juliani, Vincent-Piers Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents, 2018.
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICML, 2002.
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. ICML, fbss/182.00239, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CORR, abs/1412.69800, 2014.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabssa--aarwinkke. Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. CoRR, abs/1612.00796, 2016.
M. Mccloskey. Catastrophic interference in connectionist networks: The sequential learning prob- lem" the psychology. 1989.
11

---

Under review as a conference paper at ICLR 2020
Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. ArXiv, abs/1205.440 4810,,88022000 2012.
Andrew Y. Ng, Daishi Harada, and J. J. Russell. Policy invariance under reward transformations: Theory and application reward shaping. In ICML, 1999.
Shayegan Omidshafiei, Dong-Ki Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher
Amato, Murray Campbell, and Jonathan P. How. Learning to teach in cooperative multiagent reinforcement learning. AAAI, 2018.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In NIPS, 2016.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. coriosity-driven exploration by self-supervised prediction. 2017 IEEE Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), pp. 488-489, 2017.
Jette Randløv and Preben Alström. Learning drive a bicycle using reinforcement learning and shaping. In ICML, 1998.
Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Manfred Otto Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In ICML, 2018.
Mark B. Ring. Child: A first step towards continual learning. In Machine Learning, pp. 77-104, 1997.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, abs/1606.04671, 2016.
Jonathan Schwarz, Jelena Luketina. Wojciech Marian Czarnecki, Agnieszka Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning. ArXiv, abs/1805.06370, 2018
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NeurNIPS, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484 489, 2016.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. ArXiv, abs/ 904.07854 2019.
Sainbayar Sukhbaatar, Ilya Kostrikov, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. ArXiv, abs/ 11177003.550 703.05407, 2017.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In NIPS, 2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI, 2015.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. CoRR, 0tt//90010175. 2019a.
12

---

Under review as a conference paper at ICLR 2020
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open onen-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. ArXiv, abs/1901.01753. 2019b.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. CoRL, 2019.
Wenhao Yu, Greg Turk, and Chuanjian Liu. Learning symmetric and low-energy locomotion, ACM Trans. Graph., 37:144:1- 1-144:12, 2018.
13

---

Under review as a conference paper at ICLR 2020
A AGENT ARCHITECTURE AND TRAINING
We use the same agent network architecture and RL algorithm for all our experiments with minor modification account for the properties we vary such as reset free RL. Agents are parametrized by an MLP. The environment grid size is 8 x 8. The partial grid observation is flattened and processed by 2 layer MLP of size (64, 64, 32). The inventory observation is processed by layer MLP of size (16, 16. 16). These outputs are concatenated and then processed by a final MLP of size (16, action dim). All layers are followed by ReLU nonlinearities except the final layer which uses a softmax to output the action distribution.
We train the agents using double DQN (van Hasselt al. 2015) and the Adam optimizer (Kingma & Ba (014) with a learning rate of 0.0001, selected by sweeping across range of learning rates, with results shown Figure 6 Training done batch mode such that we alternate between collecting 500 environment steps and taking 500 gradient steps (with batch size 256) over the replay buffer of size 5e5. For environments with resets, the horizon length set to 200. We swept over various horizon lengths and found 200 work the best. We also tried setting the horizon length very short (20 and 50) to help with the episodic methods but found no effect. We use epsilon greedy exploration for the policy where epsilon starts and decays linearly by 0.0001 each timestep to 0.1. For each training method we run 10 random seeds.
Vorying Learning Fate for Tool Malling Non Ep codic
Vostinglariing Rote for Tool Making Eoloodi
cemning
saiiing soonig
corting
Lomning
betee
ThE THEE Ezodha (50CE EePonrspe
Trann Coodht SXOCE EmarEee
Figure 6: Proportion of validation tasks solved in each setting. Agents Icarning static non oon-epiiodic en- vironments struggle to learn useful behaviors, while agents learning in dynamic non-episodic environments are substantially more successful. Episodic learning easier than non-episodic learning on first task, but non-episodic learning in dynamic environments is almost effective episodic learning the hunting task.
B HUMAN GUIDED ENVIRONMENT SHAPING
Fuman Gui ded Enwiroment St aping
Envtronment Shoping Schedule Companision
-ntaiia thiping with arae-
crvishecing
Teaniig spocits urmeolags
Training Epocts (5000 Urmeslaps)
Figure 7: Performance of human guided environment shaping. We ask human interactively shape the environment and observe human can effectively guide the shaping compared predefined environment shaping schedule.
In the real world, enviromment shaping can be done by humans. In this section, we study if a human user can effectively guide the environment shaping during training, instead of using our predefined curriculum. We use the tool- tool-making task in the non-episodic setting with sparse reward. The form of environment shaping is setting the distance from the agent within which resources can spawn, which can be increased over time "teach" the agent to reach further-away resources. The human is tasked with providing this distance schedule interactively based on the performance of the agent. At each interaction, the human is given video demonstrating the agent's current behavior the training environment and a graph with the agent's 's validation performance to date. The human user produces two numbers: the resource spawn distance and for how many training epochs to
14

---

Under review as a conference paper at ICLR 2020
continue training before requesting another input. This allows the human to adaptively adjust the environment shaping depending on the agent's 's performance and minimize the amount of human supervision. Interestingly, the human controlled environment shaping does better than our linearly annealed environment shaping, as shown in Figure 7 The human user specifies a slower resource schedule than our programmed environment shaping.
C ROBUSTNESS OF ENVIRONMENT SHAPING VS. REWARD SHAPING
Shaping Methods on Walled Mooi-Making Episodic
Shaping Methods on Walled Tool-Making Non-Episadic
- Devience rewardd
starnca reward shapang
reward shaping
One time reward shaping
Ais shoping
shapang
-Eniironment
Endrronmen shaping
Ern thaping with sulgoaal reward
Enw vubggal
Training Epochs tir eneesees))
Training Enochs timestent)
Figure 8: Performance of environment shaping and reward shaping on axe- na--maiigg task an environment with wall obstacles. The distance-based reward suffers while environment shaping, despite operating off of sparse reward, obtains pcak validation performance. Wc find that this advantage in robustness of environment shaping is present in both the episodic and non-episodic settings, but is enhaneed in the former.
We examine the robustness of the different methods of shaping the learning of the agent by studying the performance of the agent in more challenging environment which contain walls and are more maze-like. This makes the environment less easily navigable and provides more chances for the agent to become trapped in particular region of the state space. We find in Figure X that environ- ment shaping is the best best-performing method under this structural challenge under both episodic and non-episodic settings. However, the episodic setting demonstrates larger gap between the perfor- mance of environment shaping and that of reward shaping. We visualize the state visitation counts of the agent under the different shaping methods in Figures (non-episodic) and 10 (episodic) to understand the differences in performance. In the non-episodie setting, the distance- based reward shaping results in the agent getting trapped in corners and therefore spending a high proportion of time there. This demonstrates that reward shaping jective. In contrast environment shaping methods results in greater coverage of the grid. This is consistent with their superior performance in Figure g
be easier exploit as it alters the true ob-
D LEARNED BEHAVIOR
In Figure 11
demonstrate the learned behavior under environment shaping with sparse reward and reward shaping with the one-time reward. With environment shaping, the agent accomplishes the desired task in timesteps. On the other hand, despite the fact that the one-time reward provides a reward only for the first interaction with the metal, the reward-shaped agent obtains the metal and repeatedly drops and picks it up afterwards, eventually failing to solve within the allotted 100 timesteps, demonstrating the biasing effect of reward shaping.
In Figure 12 we visualize trajectories from the humting environment and analyze the learned be- havior of two environment-shaped agents, a distance-based reward shaped agent, and a one-time reward shaped agent. The first tetrromment-shapee agent able to use resources that start out on opposite sides of the world, such that they are never both in view of the agent at the same time. This notable because the form of environment shaping used is one wherein the agent provided with resources near it and gradually weaned off over time. The second environment-shaped agent, while presented with a task in which the resources start out on adjacent squares, faces the challenge of the deer moving right before the agent approaches it. We observe that the trained policy is able to make a second attempt at catching the deer, and is successful. The agent trained with distance-based reward shaping displays suboptimal behavior of approaching the and then the deer while failing
15

---

Under review as a conference paper at ICLR 2020
State Visitation Count (Distance reward shaping)
Epoch
Epoch 700
Epoch 1300
Epoch 1900
State Visitation Count (One-Time reward shaping)
Epech
Epoch 200
Epoch 1300
Epoch 1900
State Visitation Count (No shaping)
Epoch
Epoch 700
Epoch 13300
Epoch 1900
State Visitation Count (Ervironment shaping)
Epoch
Epoch 1300
Epoch 1900
State Visitation Count (Env shaping with subgoal reward)
Epcch
Epoch 700
Epoch 1300
Epoch 1900
Figure 9: State visitation counts for the non-episodic setting visualized at stages during training under the different methods of shaping. Yellow corresponds to high visitation, dark purple corresponds to low visitation. The darkest purple around the borders and the map correspond walls, which cannot be traversed by the agent. We find that distance- based reward shaping results while the one-time reward and both environment shaping methods result in the most uniform state visitation distribution over the grid during training, indicating that they were able to traverse the grid and explaining their superior performance shown in Figure 8
agent getting stuck in corners of grid,
to interact with either. which can be seen as a bias resulting from a reward that incentivizes prox- imity resources. Finally, the agent trained with one-time reward shaping also shows suboptimal behavior that is explained by the biases of the reward. The agent picks up the axe and then remains stationary throughout the remainder of the trajectory, failing hunt the deer due to a reward that provides it small reward bonus for accomplishing the first portion of the task.
16

---

Under review as a conference paper at ICLR 2020
State Visitation Count (Distance reward shaping)
Epech
Epoch 200
Epoch 1300
Epech 1900
State Visitation Count (One-Time reward shaping)
Epech 0
Epoch 700
Epoch 1300
Epoch 1900
State Visitation Count (No shaping) Epoch 1300
Epoch
Epoch 1900
State Visitation Count (Environment shaping) Epoch 700
Epech
Epoch 1300
Eooch 1900
State Visitation Count (Env shaping with subgoal reward)
Epoch 1300
Figure 10: State visitation for the episodic setting visualized stages during training under the different methods of shaping. Yellow corresponds to high visitation, dark purple corresponds to low visitation. The darkest purple around the horders and the map correspond to walls, which cannot he traversed by the agent. All shaping methods result in more uniform state visitation distribution than in the non-episodic setting Figure 9 which aligns with intuition since the resets in the episodic setting help the agent get "unstuck."
Figure 11: Sample trajectories validation environments demonstrating learned behavior trained under vironment shaping sparse reward (left) as as under shaping with the one-time reward (right), both the non-episodic setting. The shaded region represents the agent's 's ego-centric partial view of the environment.
17

---

Under review as a conference paper at ICLR 2020
f1
h r
Figure 12: Sample trajectories validation environments demonstrating learned behavior trained under vironment shaping with sparse reward (top two), distance-baat baased reward shaping (third), and one-time reward shaping (bottom), all the non-episodic setting. While environment shaped agents accomplish desired task within 15 timesteps, biased task specification the last two result in interpretable but suboptimal behav- ior. The distance- based reward shaped agent goes to the correct resources in order, but without interacting with either. The one- time reward shaped agent picks up the axe. but fails to do anything afterwards. Both reward shaped agents here fail solve the task within the allotted 100 timesteps.
18


Reviews:
[{"review_id": "r1eZLxow9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study what they refer to as ecological reinforcement learning, defined as the interaction between properties of the environment and the reinforcement learning agent. They introduce environments with characteristics that reflect natural environments: non-episodic learning, uninformative reward signals, and natural dynamics that cause the environment to change. These factors are shown to significantly affect the learning progress of RL agents and, unexpectedly, the agents can sometimes learn more efficiently in these more challenging conditions.\n\nClarity:\n\nThe paper seems to be clearly written. The code will be made publicly available.\n\nNovelty:\n\nThe main contribution from the paper seems to be two novel benchmark problems with characteristics that reflect natural environments and an exhaustive evaluation of the performance of standard algorithms on them. While the experimental results show some light about the performance of existing methods in the proposed environment, the paper does not contain any methodological contributions. Because of this, it is hard to assess the novelty of the work.\n\nQuality and significance:\n\nWhile the paper makes some interesting points, I feel that the proposed environments are too few, and too simple and unrealistic when compared to real-world problems. Because of this, one cannot know how general and significant the conclusions obtained are.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "S1gUDMlCtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper discussed several properties of environments used in reinforcement learning research experiments. The main conclusion is the environment should be dynamic and non-episodic, and the environment shaping is introduced to be effective. In general, the idea is well-presented and easy to follow. However, I have some concerns about the proposed method and experiments:\n\n----\n1. In general, I think the arguments in the paper are still a bit vague to be demonstrated using only experimental results. One improvement could be using some mathematical formulation to describe the argument and conducting some analysis. For instance, the paper can formulate the environment shaping and reward shaping concretely and prove that environment shaping could replace reward shaping. \n\n\n2. For the experiment comparing the reward shaping and environment shaping: the environment shaping method is designed and more complicated than the reward shaping, I think it could be more convincing if the authors investigate and develop more approaches for reward shaping. Otherwise, it is a bit hard to argue the environment shaping could outperform reward shaping a lot. \n\n3. To argue that the non-episodic environment is better than episodic ones, I think the paper should consider more tasks besides the two mentioned in the experiment section. From figure 3, the non-episodic dynamic environment is not very clearly better than episodic one from all scenarios. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1gXFeAOFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper discusses the value of creating more challenging environments for training reinforcement learning agents. Specifically, the paper focuses on three characteristics of the environment that the paper claims are necessary for developing intelligent agents. The first of these properties is stochasticity in the environment transitions, specifically stochasticity that is independent of the action taken by the agent. The next is sparsity of rewards; discontinuing the use of reward shaping to define desired behavior. Finally the paper argues that environments should not be episodic, that natural environments are continuing tasks so research focus should be around solving continuous tasks.\n\nReview\n\nMy primary concern about this paper is the largely insufficient literature review. Many of the claims made in the motivation of this paper are not novel to this paper and are, in fact, incredibly vibrant sub-communities of study within the field of Reinforcement Learning. A more careful literature review should have easily found these communities and more nuanced claims could have been made. I will give concrete examples of the most important cases of missing literature in the following paragraphs, but this list is not exhaustive.\n\nThe paper claims that most standard RL environments include detailed reward functions that unnecessarily shape learning and inject bias into the learning process. While I agree that this is problematic, I disagree that this paper provides any novel insights towards this problem. The problem of learning from sparse rewards is well-known in the RL community and is a hot-topic of study. Even in the standard environments cited by the paper we have Montezuma's Revenge and Pitfall, two environment notorious for their difficulty due to sparse reward; each of which with its own host of literature surrounding only the single environment (for instance, I encourage the authors to investigate the highly controversial Go-Explore paper by Uber and its references). Other environments not considered by this paper include the Malmo (Minecraft) learning environment, around which NeurIPS 2019 hosted an extremely sparse reward competition. Another (overlapping) community of RL research interested in the sparse reward setting is the intrinsic reward community, one such paper being Riedmiller and Hafner et al. 2018.\n\nThis paper claims that all standard environments are episodic. Of the environments listed as \"standard\" by this paper, this claim does not even hold. However, there is a large chunk of the RL community that is not represented here. The continual learning and life-long learning communities are focused exclusively on the problem of non-episodic learning. Some example environments used by the community include Malmo, MuJoCo, DeepMind's Lab environment, and many smaller toy domains designed to showcase individual problems including Cart-Pole, RiverSwim, Pendulum, and Acrobot; with the smaller environments from OpenAi Gym cited by this paper. Another smaller community to investigate would be the average reward formulation of the RL problem, which fairly exclusively focuses on the continual learning problem.\n\nFinally, this paper seems vaguely reminiscent of a few particular environments that I have seen in the literature previously. For example, Berkeley's robot task discovery playpen (see for example Singh, Yang, Hartikainen, Finn, Levine 2019). Or an even more similar simulated environment being the Playroom environment by Singh, Barto, Chentanez 2005. Finally, Malmo has been used in a similar way as the tool-building examples mentioned in this paper.\n\nThere were a few key issues with the experiments discussed in this paper. The first of which being \"Hypothesis 1\" which states: \"Non-episodic learning is more difficult than episodic learning because the agent must handle a non-stationary learning problem.\" This hypothesis alone does not appear to be uniformly true. In fact, imagine a simple 5-state random walk markov chain environment without termination. Each state is visited infinitely many times, so the chain is ergodic and there are no non-stationary points. The empirical section uses meta-parameters that were ill-motivated with no discussion about meta-parameter selection. It is critical to point out that the stepsize used for an episodic problem will likely not be the optimal stepsize for the corresponding non-episodic problem, as the magnitude and variance of the considered returns are necessarily different (in response to Figure 3). Further, evaluating over 3 random seeds simply is not sufficient to make any statistically significant claims when comparing any of these curves (for instance in Figure 5).\n\nAdditional Comments (do not influence rating)\n\nFor a paper exclusively introducing a new control environment, it is critical to include a discussion about the exploration problem. At the very least, I would appreciate seeing sensitivity curves for values of epsilon.\n\nThis paper makes many strong claims about the nature of intelligence that are neither supported in the work or are accepted in the community. While it is intuitive that the environment plays a critical role in developing intelligence, the lack of universal definition of intelligence makes this a non-falsifiable claim. Although I appreciate the point the authors are trying to make, which is that RL research frequently is done in the realm of toy simulated domains, I do not think that this paper includes the appropriate supporting evidence to validate such lofty claims.\n\nIt would be interesting to change the exploration method from epsilon-greedy to sampling according to the softmax action distribution. This can have dramatically improved performance on non-adversarial exploration problems, and reduces the need for scheduled epsilon decay. It additionally reduces the need for two extra meta-parameters, allowing the empirical claims to be made more strongly without performing some sweep over parameters.\n\n--------------------\nEdit after reading discussion, rebuttal, and edits to manuscript.\n\nThe paper's intended contribution was different than I had realized during the initial review phase. I appreciate the effort the author put into the response and the changes made to the literature review section of the paper. I think these help to demonstrate the scope and placement of the work considerably.\n\nI still believe the paper over-states the novelty of the results and I still find it difficult to understand their utility. I believe that the entirety of the paper falls under the domain of exploration in RL, but this is not made clear through the introduction or related works section (though the updates to the related works section help considerably). \n\nStochasticity in the environment helping the agent to learn is not a surprising finding at all if the exploration method is insufficient. To give a small example, imagine an agent wandering in a tabular gridworld. If the agent has no method of exploration, and the environment has sparse reward, then it is not surprising that the agent would get stuck in a corner. If the environment was modified so that each transition had a 5% chance of randomly going another direction (e.g. the \"right\" action has a 5% chance of becoming an \"up\" action), then we've effectively encoded epsilon-greedy exploration through the environment dynamics. All of this comes down to say, it is difficult to separate the dynamics of the environment from the agent's exploration method. I think it would require a careful study that considers these aspects more explicitly.\n\nTo summarize, I think the paper could easily be accepted to a future conference, but I think it is important to:\n- Make connection between the insights and exploration clear, specifically designing the introduction, lit review, and experiments around this connection.\n- Make sure the contributions are extremely clear in the writing. Demonstrate those contributions directly in the empirical section.\n- Tone down the claims in the writing. Many of the claims about the state of the field in RL are demonstrably incorrect. I agree with the sentiment trying to be expressed, but the absolutism makes it difficult to separate that sentiment from a deep understanding of the current RL literature with only the most important papers cited, or a fundamentally insufficient lit review. To make claims about the state of a field the lit review should be rather extensive.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how the properties of an environment affect the success of reinforcement learning, and in particular finds that random dynamics and non-episodic learning makes learning easier, even though these factors make learning more difficult when applied individually. The paper was reviewed by three experts who gave Reject, Weak Reject, and Weak Reject recommendations. The main concerns are about missing connections to related work, overstating some contributions, and experimental details. While the author response addressed many of these issues, reviewers felt another round of peer review is really needed before this paper can be accepted; R2's post-rebuttal comments give some specific, constructive, concrete suggestions for preparing a revision.
Decision: Reject