Title: JELLY BEAN WORLD: A TESTBED FOR NEVER-ENDING LEARNING

Abstract: Machine learning has shown growing success in recent years. However, current machine learning systems are highly specialized, trained for particular problems or domains, and typically on a single narrow dataset. Human learning, on the other hand, is highly general and adaptable. Never-ending learning is a machine learning paradigm that aims to bridge this gap, with the goal of encouraging researchers to design machine learning systems that can learn to perform a wider variety of inter-related tasks in more complex environments. To date, there is no environment or testbed to facilitate the development and evaluation of never-ending learning systems. To this end, we propose the Jelly Bean World testbed. The Jelly Bean World allows experimentation over two-dimensional grid worlds which are filled with items and in which agents can navigate. This testbed provides environments that are sufficiently complex and where more generally intelligent algorithms ought to perform better than current state-of-the-art reinforcement learning approaches. It does so by producing non-stationary environments and facilitating experimentation with multi-task, multi-agent, multi-modal, and curriculum learning settings. We hope that the Jelly Bean World will prompt new interest in the development of never-ending learning, and more broadly general intelligence.

1. INTRODUCTION:

Machine learning has witnessed growing success across a multitude of applications over the past years. However, despite these successes, current machine learning systems are each highly specialized to solve one or a small handful of problems. They have much narrower learning capabilities as compared to humans, often learning just a single function or model based on statistical analysis of a single dataset. One reason for this is that current machine learning paradigms are restricted and specialized to a particular problem and/or dataset. An alternative learning paradigm that more closely resembles the generality, diversity, competence, and cumulative nature of human learning is never-ending learning (Mitchell et al., 2018) . The thesis of never-ending learning is that we will never truly understand machine learning until we can build computer programs that, like people: (i) learn many different types of knowledge or functions, (ii) from years of diverse, mostly selfsupervised experience, (iii) in a staged curricular fashion, where previously learned knowledge enables learning further types of knowledge, and (iv) where self-reflection and the ability to formulate new representations and new learning tasks enable the learner to avoid stagnation and performance plateaus. Building computer programs with these properties necessitates well-defined and robust ways to evaluate whether a system is indeed capable of never-ending learning. However, there are currently no ways to achieve that. There only exists one large-scale case study on never-ending learning with the Never-Ending Language Learning (NELL) system by Mitchell et al. (2018) , which uses the internet as the environment with which the system interacts. While the internet does have significant complexity, it is unwieldy to use as a testbed. It is very difficult to focus on a particular aspect of the system or the environment, or to tweak the algorithm and restart experiments to observe the effects of changes. Furthermore, oftentimes tasks require manual annotation which can be very expensive. Thus, a good testbed for never-ending learning (and machine learning more generally) needs to provide the experimenter with a high degree of control. To this end, we propose a novel evaluation framework-the Jelly Bean World (JBW)-that can enable and facilitate research towards the goal never-ending learning. We have designed the JBW to be highly versatile, enabling evaluation of systems that have any number of the aforementioned abilities.

We consider never-ending learning in the context of reinforcement learning. Let s t ∈ S denote the state of the environment at time t, a t ∈ A denote the action performed by the learning agent at time t, ω t ∈ Ω denote the observation of the world that the learning agent receives at time t, and r t ∈ R denote the reward provided to the learning agent at time t. The distribution of the next state of the world s t ∼ T (s t-1 , a t-1 ) has the Markov property (i.e., it depends only on the previous state and action) and the initial state of the world is given by a distribution s 0 ∼ W . The observation ω t ∼ O(s t ) depends only on the current state of the world (perhaps deterministically). The reward r t is given by a function R(s t-1 , a t-1 , s t ) of the current state, the previous state, and the previous action taken. The environment is a tuple containing all these elements E (W, T, O, R). Then, the goal of reinforcement learning is to find a learning algorithm π that, given the history of previous observations, actions, and rewards, outputs the next action so that the obtained reward is maximized. We deliberately blur the distinction between the policy and the algorithm that learns the policy, which is why we call π a "learning algorithm."

This formalism does not distinguish between learning algorithms that are highly specialized to a single task and learning algorithms that are capable of learning a wide variety of tasks and adapting to richer and more complex environments, which are hallmarks of general intelligence. In order to more formally describe general intelligence, we posit that there is an underlying measure of complexity of the environment E such that: (i) highly specialized and non-general learning algorithms can perform well in environments with low complexity, but (ii) environments with high complexity require successful learning agents to possess more general learning capabilities. It is in these more complex environments where we can characterize never-ending learning. We can formalize this notion of complexity by letting π * be the (computable) learning algorithm that maximizes expected reward in an environment E. Then we define the complexity of E to be the length of the shortest program (Turing machine) that implements π * :

complexity(E) = min{|T | : T is a Turing machine that implements π * } complexity(E) 0 π * exhibits specialized intelligence π * exhibits more general intelligence

We can equivalently define complexity(E) = K(π * ), where K(•) is the Kolmogorov complexity and is related to the minimum description length and minimum message length (Nannen, 2010; Kolmogorov, 1963) . The Kolmogorov complexity of the environment K(E) is bounded below by K(π * ) minus a constant. This bound is shown in Section A.1 of the appendix.

Never-ending learning is in many respects similar to lifelong learning, also called continual learning (Chen & Liu, 2018) . Like never-ending learning, lifelong learning is distinguished from multi-task learning by the never-ending nature of the learning problem. However, in never-ending learning, and unlike multi-task learning and lifelong learning (to the best of our understanding), a well-defined set of tasks is not assumed a priori. Rather, never-ending learning is more similar to real-world settings in this respect, where the notion of a task or subtask naturally emerges from the complexity of the environment, and the distinction between tasks is not always so sharp. Regardless, due to their similarities, a good testbed for never-ending learning will also be a good testbed for lifelong learning.

In contrast to most popular reinforcement learning settings, never-ending learning focuses on environments with high complexity. In never-ending learning, agents can only exist in a single reset-free environment (i.e., we explicitly disallow the agent π from learning across multiple episodes or in multiple environments, which is closer to human learning). We require π to only have access to a single episode. During its lifetime, π can only use the information provided by its past observations {ω t } and actions {a t } in a single world. Thus, never-ending learning explicitly removes the distinction between training and testing that is common to many other classical machine learning paradigms. Additionally, note that in the general reinforcement learning formalism, s t can contain information about t, and the reward function R can be time-varying, thus rendering the environment non-stationary.

In never-ending learning, we are interested in the full generality of non-stationary environments, as the assumption of stationarity is not realistic in even simple adversarial and multi-agent settings. We thus argue that a testbed for never-ending learning should have the following properties:

1. Non-Episodic: It should disallow agents from resetting the environment and "retrying". The testbed should also force them to only learn within a single environment (i.e., not transfer information across environments). This is in contrast with most popular reinforcement learning environments and, as we show in Section 4, poses significant challenges to existing algorithms. 2. Non-Stationary: The testbed should allow for easy experimentation with non-stationary environments, where the reward R can depend on time. 

3.:

Multi-Task: It should support settings in which reward is maximized not by learning how to perform a single task repetitively, but by learning how to perform a general variety of tasks, and learning how to switch between them and/or combine them to better perform other tasks (e.g., by composing them). We posit that, at a sufficiently high level of task complexity, optimal learning agents will be required-either explicitly or implicitly-to perform abstract reasoning over concepts and make informed decisions about actions in the environment. 4. Multi-Modal: It should support multiple data modalities that agents receive as input. These modalities should not contain the same information, but rather be complementary to each other so that the agents are forced to learn from diverse types of experiences. Multi-modality provides yet another way to increase the complexity of the world. 5. Controllable: It should be easy for experimenters to modify the complexity and richness of the learning problems in the testbed, make changes to it, and restart it (e.g., as opposed to NELL). 6. Efficient: It should run on readily available hardware and allow for quick experimentation.

Ideally, we should not have to wait for days, weeks, or months (e.g., NELL) to obtain results. 7. Reproducible: It should make it easy to reproduce results and experiments, which would facilitate scientific research. This also requires that it allows for seamlessly saving and loading state and for reproducing results outside the environment in which they were first obtained. The testbed should also not require access to specialized hardware, which can be expensive.

Many of the above properties are means to increase the complexity of the world. These properties are in fact very closely related to the characteristics of "AGI Environments, Tasks, and Agents" outlined by Laird & Wray III (2010) and later refined by Adams et al. (2012) . The proposed JBW has all of these properties. It aims to provide an easy way to create sufficiently complex environments allowing researchers to experiment with never-ending learning, while remaining simple enough to control the problem and enable rapid prototyping. The JBW is a two-dimensional grid world with simple physics, but is extensible enough to admit a wide variety of complex and inter-related tasks. We present a

MAP:

Manages the infinite world map.

AGENTS:

Manages agents and handles their interaction with the simulator.

SCENT:

Simulates the diffusion of scent in the world.

Represented as a vector.

VISION:

Simulates the visual field of all managed agents.

Field of View:

Occlusion Represented as a 3D tensor.

Advances time after all managed agents have acted, invoking modules as needed.

Simulator:

Interface for reinforcement learning.

Environment REWARD:

Specifies the reward given to the agent for each possible state transition.

Collect[JellyBean] ∧ Avoid[Onion]

REWARD SCHEDULE:

Specifies the reward for each time step. 

Visualizer:

Distributed simulations are also supported using MPI. comparison with related work in Table 1 , showcasing the ways in which the JBW is a novel and highly versatile evaluation framework. The JBW is written in C++ and we provide C, Python, and Swift APIs, and is available at https://github.com/eaplatanios/jelly-bean-world. It is also worth noting that the JBW has already been used for the instruction of the "Deep Reinforcement Learning" and "Never-Ending Learning" graduate courses at Carnegie Mellon University.

2. DESIGN:

The Jelly Bean World (JBW) consists of the following main modules (illustrated in Figure 1 ): (i) the simulator, which comprises the central component (the other modules only interact with the simulator), (ii) the environment, which provides a simple interface for performing reinforcement learning experiments in the never-ending learning setting as well as utilities for evaluating neverending learning systems, and (iii) the visualizer, which provides the ability to visualize and debug the behavior of learning agents. Note that the visualizer is completely asynchronous and can be attached, reattached, and detached to and from existing simulator instances, without affecting the simulations.

2.1. SIMULATOR:

The simulator manages a map and a set of agents. At a high-level, the map is an infinite twodimensional grid where each grid cell can contain items (e.g., jelly beans and onions) and/or agents. Each item has a color and a scent that agents can perceive. Each agent has a direction and a position, and can navigate the world map and collect or drop items. The action space of each agent is: to turn, move, collect items, drop items, or do nothing. The action space is configurable and can be constrained by the user. These constraints are described later in this section. Time in the simulator is discrete, and all agent-map interactions are turn-based, meaning that the simulator will first wait for all managed agents to request an action and will then simultaneously execute all actions and advance the current time. Thus, the simulator also controls the passage of time.

Map. In order to truly support never-ending learning, we have designed the JBW map to be infinite, meaning that it has no boundaries and agents can keep exploring it forever. To achieve this, the map is a procedurally-generated two-dimensional grid. We simulate it by dividing it into a collection of disjoint (P × P )-sized patches and only generating patches when an agent moves sufficiently close to them. The map also contains items of various types which are distributed according to a pairwise-interaction point process over the two-dimensional grid (Baddeley & Turner, 2000) . More specifically, for a collection of items I {I 0 , . . . , I m }, where I i = (x i , t i ), x i ∈ Z 2 is the position of the i th item, t i ∈ T is its type, and T is the set of all item types:

EQUATION

where f (I i ) is the intensity of item I i and g(I i , I j ) is the interaction between I i and I j , which are provided as part of the item's type. The intensity function characterizes the (log) probability of the existence of an item I i independent of other items in the world. The interaction function can be understood as a description of the (log) probability of the existence of an item I i given the existence of all other items I j . For example, the interaction function can be used to increase the log probability of an item when it appears near other items, producing a clustering effect. Since the world is subdivided into (P × P )-sized patches, the maximum distance of interaction between items is P . The red line corresponds to an example path followed by an agent. Once the agent enters a patch that is not fixed, then that patch is sampled, along with its non-fixed neighboring patches in order to avoid boundary effects. Item Types. Each item type t ∈ T defines the following:

New:

-Color: Fixed-size vector specifying the item color.

-Scent: Fixed-size vector specifying the item scent.

-Occlusion: Occlusion of an item (relevant to the vision modality, described later in this section). -Intensity Function: Maps from item locations to real values.

-Interaction Functions: Collection of functions that map from pairs of item locations to real values. The collection contains one function for each item type.

The number of item types and their properties are configurable.

The specific parametric forms for the intensity and interaction functions that are supported are described in Section A.2 of the appendix. Note that each item type also specifies additional properties that are described later in this section.

Procedural Generation. When the simulator is instantiated the map is empty (i.e., no patches have been generated). Whenever a new agent is added to the simulator, a patch centered at its location is generated. In addition, whenever an existing agent moves sufficiently close to a region where no patch exists, a new patch is generated. The patch generation process consists of two main steps: (i) add a new empty (P × P )-sized patch to the collection of map patches (note that the new patch will be neighboring at least one existing patch and that all patches are disjoint), and (ii) fill the new patch with items. The second step is performed by using Metropolis-Hastings (MH) (Robert & Casella, 2010) to sample the items that the new patch contains, from the distribution defined in Equation 1.

The proposal density we use is defined as follows: (i) add a new item I m+1 = (x m+1 , t m+1 ) with probability 1/(2P 2 • |T |) (i.e., uniform in position and type), and (ii) remove an existing item I i with probability 1/2m where m is the current number of items in the patch. Before sampling, the patch is initialized by first randomly selecting an existing patch and copying its items into the new patch. This is intended to facilitate rapid mixing of the Markov chain, and reduce the number of MH iterations. Note that if we use small patches and only sample new patches as the agents visit them, boundary effects may be observed due to the missing neighboring patches further away from the agent. For this reason, we actually also sample all missing neighboring patches while sampling each new patch, but do not finalize them (i.e., they are still considered missing and may be resampled later on). This prevents boundary effects during the procedural generation process. An example is shown in Figure 2 .

Each item has a color and a scent that is specified by its type and can be perceived by agents. The JBW thus supports two perception modalities, vision and scent. These modalities are complementary and agents can benefit by learning to combine them, as we explain at the end of this section.

Vision. Each agent has a visual range property that specifies how far they can see. Vision is represented as a three-dimensional tensor, where the first two dimensions correspond to the width and the height of the agent's visual field, and the third dimension corresponds to the color dimensionality. The visual field is always centered at the agent's current position and the color observed at each cell within the visual field is the sum of the color vectors of all items and agents located at that map location. Agents also have a field of view property that specifies their field of view angle (i.e., 180 • denotes that the agent can only see the forward-facing half of the visual field, whereas 360 • denotes that the agent can see the whole visual field). The part of the visual field that is outside an agent's field of view is masked out and appears black to the agent. Another important aspect of vision is that items also have an occlusion property as part of their type. This is used to simulate partial or complete visual occlusion (details in Section A.3). If an item with occlusion 1 is in an agent's visual field, then the colors behind that item are not visible to the agent. An example is shown in Figure 3 .

Scent. Scent is represented as a fixed-dimensional vector, where each dimension can be used to model orthogonal/unrelated scents. Each agent and each item has a pre-specified scent vector that is provided as part of the world configuration (similar to their colors). At each time step, agents can perceive the scent at their current grid position. The physics of scent are described by a simple diffusion difference equation on the world grid. We define the scent at location (x, y) at time t as:

S t x,y = C t

x,y current items/agents scent

+ λS t-1

x,y previous scent

+ α S t-1 x-1,y + S t-1 x+1,y + S t-1 x,y-1 + S t-1

x,y+1

EQUATION

)

where λ is the rate of decay of the scent at each location, α is the rate of diffusion of the scent from neighboring grid cells, and C t x,y = I∈I t

x,y scent(I)

+ A∈A t

x,y scent(A), where I t x,y is the set of all items at time t and location (x, y), and A t x,y is the set of all agents at time t and location (x, y). Our simulator ensures that the scent (or lack thereof) diffuses correctly, even as items are created, collected, dropped, and destroyed. It does so by keeping track of the creation, collection, drop, and destruction times of each item in the world. Note also that, while simulating this diffusion, we also take into account the non-fixed patches that have been sampled in order to avoid boundary effects.

Vision-Scent Complementarity. Vision and scent are complementary. Vision has high precision, in the sense that the agent can see the actual color of each grid cell in its visual field and can thus relatively accurately determine what items may exist in that cell. However, it has low recall-the agent can only see as far as its visual range allows and it has no visual information about the rest of the map. On the other hand, scent has low precision-the scent at the current cell is a linear combination of the scents of all items in the world and it may be very difficult to learn to interpret and use it effectively. However, scent has high recall-the scent at the current cell contains information about items in a much larger range. Thus, learning to use both modalities will be beneficial to agents. In Section 4, we also provide some experimental results supporting this argument.

Constraints. The simulator enforces multiple constraints on the actions that agents are allowed to take. We have designed the following small set of constraints with the goal of providing a computationally efficient way to support arbitrarily complex tasks and learning problems:

-Agent Collision: This occurs when multiple agents attempt to move to the same location at the same time. This conflict can be resolved in one of three ways: (i) allow multiple agents to occupy the same location, (ii) first-come-first-serve (only allow the first agent who made a move request for that location to actually move-this is the current default), or (iii) randomly choose one of the agents and satisfy their request (ignoring the requested action of the others). -Item Blocking Movement: Item types may specify that they block agent movement (e.g., a Wall item type). This means that agents are not allowed to move to locations with items of that type. -Item Collection Requirements: Item types may specify that in order to collect items of that type, an agent has to have first collected a specified number of other items (e.g., collecting Wood may only be allowed if the agent has first collected an Axe). -Item Collection Costs: Similar to the collection requirements, item types may specify that in order to collect items of that type, an agent has to drop or destroy a specified number of other items (e.g., collecting an Axe may require destroying a piece of Metal and a piece of Wood that the agent has previously collected).

Interface. Users interact with the simulator programmatically. JBW provides functions to add or remove agents from the world, query the current vision and scent perception of each agent, and to direct agents to perform actions. Users can choose to add multiple agents to the world, thus enabling experimentation with multi-agent settings. Multi-agent interactions provide another controllable source of complexity in the JBW. Users can then request actions for each agent in the simulation (i.e., turn, move, do nothing, etc.). Once all agents have requested actions, the simulator executes these actions and advances time, appropriately updating the state of the world.

Server/Client Support. The JBW also provides a TCP server-client interface where the simulator can be setup to run as a server. Users (i.e., clients) can then connect to the server, and interact with the simulator by sending messages to the server. This allows use cases such as a class setting where students can each control an agent in a common simulated world, or perhaps a hackathon where participants can compete in a common world. It also allows debugging and visualization tools to be attached and detached to and from running simulator instances, without affecting the simulations. In fact, this is how our visualizer, which is described in Section 2.3, communicates with the simulator.

Persistence. Simulations in the JBW can be saved to and loaded from files, which can then be distributed across platforms. This facilitates reproducibility. The simulator guarantees uniform random number generation behavior across all platforms and machines (e.g., in distributed settings).

The state of the pseudorandom number generator is also saved and loaded along with the simulation.

2.2. ENVIRONMENT:

Environments manage simulator instances and provide an interface for performing reinforcement learning experiments using the JBW. We provide implementations of the JBW environments for OpenAI Gym (Brockman et al., 2016) in Python and for Swift RL (Platanios, 2019) in Swift. JBW environments support batching by design, with support for parallel execution of the multiple simulator instances being managed (i.e., one simulator for each batch entry). Perhaps the most important aspect of JBW environments is that they require the user to specify a reward schedule to use for each experiment. This schedule effectively defines the tasks that the agents are learning to perform. A reward schedule provides a function that, given a simulation time, returns a reward function to use at that time. A reward function returns a scalar reward value, given the current and previous states of the agent and the world (e.g., the world map). 1 We provide a simple domain-specific language (DSL) for composing and combining multiple reward functions in arbitrary ways, to allow for the design of composable learning tasks. This enables endless possibilities in the realms of multi-task learning, curriculum learning, and more generally never-ending learning. Currently environments are limited to single agent reinforcement learning settings, but we plan to support multi-agent settings in the future (this is easy because the JBW simulator already supports multiple agents for each simulation).

2.3. VISUALIZER:

Visualization can be instrumental when developing, debugging, and evaluating never-ending learning systems. To this end, we have implemented a real-time visualizer using Vulkan2 in which the user can see any part of the simulated JBW, at any scale and simulation rate. The visualizer utilizes the simulator server-client interface to visualize simulations running in different processes or on remote servers, in a fully asynchronous manner. Rendering is multithreaded to provide a smooth and responsive user interface. Finally, the visualizer can be attached, detached, and re-attached to existing simulation server instances, without affecting the running simulations.

3. LEARNING TASKS:

Learning tasks can be defined in terms of reward functions and reward schedules, which were defined in Section 2.2. The JBW allows researchers to easily define their own reward functions and schedules, but it also provides a few primitives and ways to compose them in order to effortlessly allow for quick experimentation and prototyping. In fact, all learning tasks used in Section 4 were defined using these primitives. The currently supported primitives are:

Reward Functions Reward Schedules:



Action[v]:

Give v to agents when they take an action (i.e., not a no-op).

Fixed[r]:

The reward function is always fixed to r, and is thus stationary.

Collect[i,v]:

Give v to agents for each item of type i that they collect.

Curriculum[{ri, ti} R i=1 ]

Use reward function r1 for the first t1 steps, then r2 for t2 steps, ..., and keep using r R after the list of reward functions is exhausted.

Explore[v]:

Give v to agents each time they move further away from their starting position in the world map.

Cyclical[{ri, ti} R i=1 ]

Use reward function r1 for the first t1 steps, then r2 for t2 steps, ..., and then repeat after the list of reward functions is exhausted.

Reward Function Compositions:

Combined[r1,r2] r1 ∧ r2

Applies both r1 and r2 and returns the sum of their rewards.

We note that Collect is a sparse reward function, whereas Action and Explore are not. For conciseness, we omit the v argument in reward functions when it is set to 1 and we also define

Avoid[i,v]=Collect[i,-v].

nels. The resulting tensor is flattened and passed through a dense layer with size 512. If scent is used, then the scent vector is passed through two dense layers: one with size 32, and one with size 512. If both modalities are being used, the two hidden representations are concatenated. Finally, the result is processed by a Long Short-Term Memory (LSTM) network (Hochreiter & Schmidhuber, 1997) which outputs a value for the agent's current state, along with a distribution over actions. Learning is performed using Proximal Policy Optimization (PPO); a popular on-policy reinforcement learning algorithm proposed by Schulman et al. (2017) .

The experiments are implemented using Swift for TensorFlow.4 

4.1. CASE STUDIES:

For all experiments we evaluate performance using the reward rate metric, which is defined as the amount of reward obtained per step, computed over a moving window. The size of that window varies per experiment and is reported together with the results. This is an appropriate metric for this task as we want to measure the improvement in the ability of an agent to learn (i.e., the gradient of the reward rate), while also making sure the agent does not get stuck (i.e., the reward rate goes to zero). Whenever possible, we also report the results obtained by the greedy vision-based agent described in Section A.7 of our appendix.

The greedy agent makes additional assumptions about the world, and thus, doesn't generalize to more complex environments, it provides a lower bound on the optimal reward rate. Note that a perfect upper bound cannot be obtained as that would require solving an NP-hard discrete optimization problem. 

Time:

Step

Collect[JellyBean]∧ Avoid[Onion]

Reward Rate (pts/s)

Figure 4 : Non-episodic experiment result. The reward rate is computed using a 100,000-step window and the shaded bands correspond to standard error over 20 runs.

Case Study #1: Non-Episodic. The goal of this case study is to show that the JBW allows for experimenting with never-ending learning agents and to also show how current machine learning methods (e.g., PPO used to train an LSTM-based agent) are failing to effectively perform never-ending learning.

For this experiment we use the fixed reward function Collect[JellyBean]∧ Avoid [Onion] and let agents interact with the JBW for 10 million steps.

Our results are shown in Figure 4 . The agents seem to be learning effectively for the first 1 million steps, but start to underperform later on, eventually getting stuck and being unable to collect any reward. This is the case for multiple different learning agents that we experimented with; both using different models and using different learning algorithms, such as Deep Q-Networks (DQNs) proposed by Mnih et al. (2013) . After connecting the visualizer to observe what happens we see that all agents either: (i) get stuck in an area of the map that they have already explored and exhausted of jelly beans, or (ii) get stuck constantly rotating and not moving to new grid cells at all. This indicates that the JBW is indeed challenging for current machine learning methods when it comes to never-ending learning. Perhaps some sort of reward shaping or curriculum learning could help the agents. However, our goal with this paper is not to solve these hard problems but rather point them out and show how the JBW provides a testbed with which to tackle them.

Case Study #2: Non-Stationary. The goal of this case study is to demonstrate that the JBW allows for experimenting with non-stationary and multi-task learning problems. To this end, we perform two experiments: (i) one using a cyclical/periodic reward function schedule where every 100,000 steps we alternate between the Collect[JellyBean]∧Avoid [Onion] and Avoid[JellyBean] ∧Collect[Onion] reward functions, and (ii) one testing a couple of curriculum reward schedules for eventually learning to Collect[JellyBean]∧Avoid [Onion] . The results are shown in Figure 5 and we observe that current standard machine learning approaches are not able to efficiently alternate between different learning problems and are effectively learning each problem from scratch whenever they switch, eventually ending up unable to learn either one. We also observe that agents who first learn to collect jelly beans and then switch to the full reward function are able to learn to collect jelly beans and avoid onions faster than agents that first learn to avoid onions or face the final learning problem directly from the beginning. Eventually all agents perform similarly, but this showcases how the JBW enables research in curriculum learning.

Case Study #3: Multi-Modal. The goal of this case study is to: (i) show how computationally efficient features, such as the field of view mask and visual occlusion, allow for increasing the learning problem complexity in a controllable manner and, perhaps most importantly, (ii) show how the perception modalities of the JBW are complementary. We thus perform three experiments. For the first two we use the fixed reward function Collect [JellyBean] and for the last one we use Collect [Onion] . We change the reward function in order to show how easy it is to experiment with different tasks in the JBW. In the first experiment, we vary the field of view of the agents. The results are shown in the left plot of Figure 6 . We see that decreasing the field of view allows us to make the learning task harder for agents, while maintaining the same computational cost for the environment. Similarly, in the second experiment we measure the effect that visual occlusion has on performance. The results are shown in the middle plot of Figure 6 and we observe that enabling visual occlusion makes the learning task harder. Finally, with the third experiment our goal is to show that vision and scent are complementary. The results are shown in the right plot of Figure 6 . We see that "vision" agents do better than "scent" agents, indicating that vision is perhaps an easier perception modality to use in the context of this learning task. Surprisingly though, the "vision" agents also do better than the "vision+scent" agents. This indicates a limitation of the model because, even though scent contains useful information that vision does not, the agents seem to get confused by it and do not seem able to use it properly. It also shows the need for better multi-modal algorithms and the utility of the JBW in testing such algorithms. The relative utility of scent and vision depends on the environment and the agent model. We demonstrate this in Section A.4 by providing a different configuration where "scent" agents are able to outperform both "vision" and "vision+scent" agents. 

No curriculum:

The reward function is always set to the final one.

Curriculum Length:

The reward rates are not comparable before this point

Collect[Onion] ∧ Avoid[JellyBean]:

Curriculum #1 For the first 100,000 steps the reward function is set to

Avoid[Onion]

Curriculum #2 For the first 100,000 steps the reward function is set to

Collect[JellyBean] Final Reward Function Collect[JellyBean] ∧ Avoid[Onion]

Figure 5 : Non-stationary experiment results. The reward rate is computed using a 100,000-step window and the shaded bands correspond to standard error over 20 runs. Figure 6 : Multi-modal experiment results. The braces on top of the plots specify the reward function used in each case. The reward rate is computed using a 100,000-step window and the shaded bands correspond to standard error over 20 runs. "Greedy Visual" refers to the reward rate obtained by the greedy visual agent baseline described in Section A.7 (which, as explained in that section, is not able to handle occlusion or avoidance).

5. CONCLUSION AND FUTURE EXTENSIONS:

We presented a new testbed designed to facilitate experimentation with never-ending learning agents, where the complexity of the learning problems is higher than that of existing testbeds and evaluations, while maintaining controllability, performance, and reproducibility. In order to produce more complex environments, the JBW supports non-stationary environments, with multiple distinct but inter-related tasks and complementary perception modalities. The JBW also explicitly restricts learning to a single never-ending episode. It is highly configurable and performant, and provides users with tools to easily save, load, distribute environments, and reproduce and visualize results. We also showed how easily we can define learning tasks in the JBW, for which current machine learning methods struggle.

The space of potential extensions to the JBW is large. Although the current intensity and interaction functions are stationary with respect to space (i.e., they are independent of position x, y), it is not difficult to define new non-stationary functions, in order to generate worlds with non-stationary item distributions. The JBW supports multiple agents running asynchronously in the world, and so it would also be interesting to experiment with multi-agent settings. However, agents currently don't have an easy way to communicate with each other, and so adding a mechanism for communication, perhaps via new agent-item interactions (e.g., reading/writing note items), would be interesting. Another way to add complexity is via items that can contain "strings" (e.g., notes) in an agent-specific language, or even natural language. These notes could, for example, contain task specifications. Scent currently does not interact with items in the world, meaning that it can pass through Wall items without any hinderance. Thus, another possible extension would be to support more complex item-scent interactions. It would be interesting to explore interactions between items and the properties of agents (e.g., a Telescope could extend the visual range of an agent while narrowing its field of view). Finally, another interesting way to add complexity is to generate richer relationships between item types, perhaps even an ontology. We look forward to continue improving the JBW, and hope that a standardized testbed for never-ending learning will motivate research into more generally-intelligent learning algorithms. 

. RadialHash[∆,s,c,k]: ft(x, y) = c -k • M ( x 2 + y 2 /s + ∆),

where M : R → [0, 1] is the linear interpolation of M ( t )/(2 32 -1) and M ( t + 1 )/(2 32 -1), and M : Z → Z is the last mixing step of the 32-bit MurmurHash function (Appleby, 2008) . This provides pseudorandomness to the intensity function.

For interaction functions, let (x , y 1 ) be the input position of the first item, t 1 be the type of the first item, (x 2 , y 2 ) be the position of the second item, and t 2 be the type of the second item. Interaction functions are indexed by pairs of item types, so each pair of item types can be given its own interaction function: g(((x 1 , y 1 ), t 1 ), ((x 2 , y 2 ), t 2 )) g t1,t2 ((x 1 , y 1 ), (x 2 , y 2 )). The JBW currently supports four interaction functions: 1. Zero: gt 1 ,t 2 ((x1, y1), (x2, y2)) = 0.

2. PiecewiseBox[U ,V ,u,v]: ,u,v,α,β] : 4. CrossHash[s, c, k, δ, u, v, α, β] is the same as Cross[U ,V ,u,v,α,β] except for the fact that U and V are given by:

gt 1 ,t 2 ((x1, y1), (x2, y2)) =    u, if d < U, v, if U ≤ d < V, 0, otherwise, where d = (x1 -x2) 2 + (y1 -y2) 2 . 3. Cross[U ,V

gt 1 ,t 2 ((x1, y1), (x2, y2)) =            u, if d = 0, D ≤ U, α, if d = 0, D ≤ U, v, if d = 0, U < D ≤ V, β, if d = 0, U < D ≤ V, 0,

U = c + k • M (x1/s), V = U + δ,

where M (•) is defined as above in the RadialHash intensity function.

Even though this is a small set of intensity and interaction functions it can allow for creating worlds with many interesting features (e.g., we use the Cross interaction function to create contiguous wall segments that are axis-aligned, and the PiecewiseBox interaction function to create irregularly shaped clusters of trees forming forests). Note that the unspecified intensity and interaction functions in Table 3 are set to Zero by default.

A.3 FIELD OF VIEW AND VISUAL OCCLUSION:

To compute the color of a cell with respect to the agent's field of view, let the cell position be (x, y) and consider a circle of radius 1 2 centered at (x, y). Project this circle onto the circle of radius 1 centered at the agent position. Let θ denote this projection (an arc). Let θ fov be the arc on the agent's circle centered on a point in the current agent direction. The length of θ fov is specified by the field-of-view parameter in the configuration. The color of cell c x,y is then computed as:

EQUATION

where ĉx,y is the original color of the cell. In order to compute how much a cell at position (x, y) is occluded, we consider a circle of radius 1 2 centered at (x, y), and project this circle onto the circle of radius 1 centered at the agent position. Let θ denote this projection (an arc). Each item in the agent's visual field is similarly projected onto the agent's circle, each producing an arc θ i . The color of the cell c x,y is then computed as:

EQUATION

where ĉx,y is the original color of the cell, and o i is the occlusion parameter of the i th item, as specified by the item's type. If a cell is affected by both the field of view and visual occlusion, the above effects are composed (both multiplicative factors are applied to the original color).

Similarly, in environments containing items that provide negative reward, the above greedy algorithm does not allow the agent to move over tiles with such items. Thus, given sufficient time, the agent will find itself moving between two clusters of avoided items, where one cluster is shaped like a "∪" and the other like a "∩". And so when the agent moves downward into the "∪" of one cluster, it will make a 180-degree turn, leaving the cluster, and moving upward toward the other cluster. The other cluster is shaped symmetrically, and the agent will repeat the behavior deterministically: enter the "∩", make a 180-degree turn, leave the cluster, and move down toward the first cluster again. This will repeat indefinitely.

:

A simple reward function could be one that gives the agent 1 reward point for each JellyBean it collects.

Information on Vulkan can be found at https://www.khronos.org/vulkan/.

An example non-stationary world configuration is also presented in Section A.5 of our appendix.

https://www.tensorflow.org/swift.

Constant[v]: ft(x, y) = v.

Note that since the configurations differ, the reward rates between the two figures are not directly comparable.

Figures and Tables:



Figure 1: Overview of the modules comprising the Jelly Bean World.

Figure 3: Rendering of an agent's perspective from the JBW visualizer.

3

otherwise, where d = min{|x1 -x2|, |y1 -y2|} and D = max{|x1 -x2|, |y1 -y2|}.

Existing reinforcement learning environments positioned based on our desired properties.

Table content: <table><tr><td>Environment</td></tr></table>



Table content: <table><tr><td/><td/><td/><td/><td/><td/><td>∼</td></tr><tr><td>Atari and Retro Games</td><td>Games end</td><td>The game</td><td>Each game has</td><td>Agents only</td><td>Modifying the</td><td>Can run on</td></tr><tr><td>(Bellemare et al., 2013),</td><td>when the</td><td>mechanics are</td><td>a single fixed</td><td>observe the</td><td>task complex-</td><td>small machines</td></tr><tr><td>(Pfau et al., 2018)</td><td>player wins or</td><td>stationary</td><td>reward</td><td>game video</td><td>ity/richness is</td><td>but models are</td></tr><tr><td/><td>loses</td><td/><td>function</td><td>frames</td><td>not possible</td><td>slow to train</td></tr><tr><td/><td/><td/><td/><td/><td/><td>∼</td></tr><tr><td>Continuous Control (Duan et al., 2016), (Todorov et al., 2012)</td><td>Some tasks are non-episodic (e.g., swimmer)</td><td>Stationary rewards and (i.e., physics) environments</td><td>Some of the tasks have hierarchical interesting</td><td>Agents only observe information positional</td><td>The tasks and environments configurable are non-</td><td>Efficiency varies widely across tasks</td></tr><tr><td/><td/><td/><td>structures</td><td>and joint angles</td><td/><td/></tr><tr><td>Evolutionary Robotics</td><td>Episodic in a</td><td>Stationary</td><td>Only navigation</td><td>Multiple</td><td>Configurable</td><td>Fast 2D</td></tr><tr><td>(Mouret &amp; Doncieux, 2012)</td><td>finite world</td><td>environments</td><td>goals are</td><td>different kinds</td><td>using XML</td><td>simulation</td></tr><tr><td/><td/><td>(i.e., physics)</td><td>supported</td><td>of sensors</td><td/><td>written in C++</td></tr><tr><td>BabyAI</td><td>Episodic in a</td><td>Fixed set of</td><td>Handful of</td><td>Agents given</td><td>Existing levels</td><td>Built on fast</td></tr><tr><td>(Chevalier-Boisvert et al., 2018)</td><td>finite world</td><td>levels and</td><td>tasks to</td><td>visual input and</td><td>are not</td><td>MiniGrid</td></tr><tr><td/><td/><td>rewards</td><td>perform</td><td>instructions</td><td>configurable</td><td>simulator</td></tr><tr><td/><td/><td/><td/><td/><td>∼</td><td/></tr><tr><td>Adversarial Games like</td><td>Games end</td><td>Non-stationary</td><td>Each game has</td><td>Agents observe</td><td>There is limited</td><td>Experiments</td></tr><tr><td>Go (Silver et al., 2017),</td><td>when the</td><td>(without</td><td>a single fixed</td><td>the game video</td><td>control over</td><td>are typically</td></tr><tr><td>StarCraft (Vinyals et al., 2019),</td><td>player wins or</td><td>assumptions</td><td>reward</td><td>frames and the</td><td>things like the</td><td>extremely com-</td></tr><tr><td>and Dota (OpenAI, 2019)</td><td>loses</td><td>about the</td><td>function</td><td>game state</td><td>adversary's</td><td>putationally</td></tr><tr><td/><td/><td>adversaries)</td><td/><td/><td>competence</td><td>expensive</td></tr><tr><td/><td/><td>∼</td><td>∼</td><td/><td/><td>∼</td></tr><tr><td/><td>Levels have a</td><td>Levels have</td><td>Levels have</td><td>Agents only</td><td>The complexity</td><td>Requires</td></tr><tr><td>DeepMind Lab</td><td>time limit</td><td>different</td><td>predefined</td><td>observe the</td><td>of each level is</td><td>rendering of a</td></tr><tr><td>(Beattie et al., 2016)</td><td/><td>rewards but</td><td>rewards</td><td>game video</td><td>fixed</td><td>3D world</td></tr><tr><td/><td/><td>same physics</td><td/><td>frames</td><td/><td/></tr><tr><td/><td>∼</td><td/><td/><td/><td/><td/></tr><tr><td>Malmö and MineRL (Johnson et al., 2016), (Guss et al., 2019)</td><td>Tasks have a pre-specified time limit but that is typically</td><td>Stationary rewards and map generation is based on</td><td>Supports 6 complex tasks but also allows for new ones</td><td>Agents observe the game video frames and the game state</td><td>Modifying the task complex-ity/richness is difficult and</td><td>Requires rendering of a 3D world and slow training of</td></tr><tr><td/><td>very long</td><td>Perlin noise</td><td/><td/><td>expensive</td><td>large models</td></tr><tr><td>Jelly Bean World</td><td>Agents live "forever" in an</td><td>The rewards and the world</td><td>Composable and dynamic</td><td>Vision and scent are</td><td>Modifying the task complex-</td><td>Experiments can run</td></tr><tr><td>(Proposed Environment)</td><td>infinite open</td><td>can both be</td><td>tasks are</td><td>designed to be</td><td>ity/richness is</td><td>efficiently on</td></tr><tr><td/><td>world</td><td>non-stationary</td><td>supported</td><td>complementary</td><td>very easy</td><td>small machines</td></tr></table>

Peer review (iclr2020):
Summary This paper introduces a new environment for testing lifelong or never-ending learning. The goal of the environment is to act as a new benchmark testbed for challenging existing agents and models across areas of research, encouraging and pushing new research towards solving challenges in curriculum learning, exploration, representation learning, and continual learning. The contributions in this paper extend upon previous work by building an easily controllable environment generator with key necessary features for lifelong learning including: non-stationarity, multiple task specification, and multiple sets of observable features. Review The paper highlights many key characteristics of an environment that are challenging to current RL models. This focus on building a benchmark upon which further research can measure performance is important. I find the proposed environment to be incredibly intriguing and would find it valuable to the field of lifelong learning (or continual learning or never-ending learning, etc.). I think the size and scope of the environment generator is impressive, showing a considerable amount of engineering effort has gone into its design. The largest overarching issue that I would like to point out is the limited study of modelling choices. I am not an expert on applied Reinforcement Learning, so I can make very few claims about the validity of the chosen network architecture or use of the PPO policy-gradient algorithm for this environment. However, it is critical, in my view, that a paper introducing a new environment studies these effects itself; demonstrating how various degrees of learning capacity or wider ranges of learning algorithms behave in the given environment. If a slightly larger network architecture trivially solves each task in this environment, can this still be considered a benchmark task? A key result in the paper that I would like to see further investigated (even with only a different network architecture) would be Figure 6, the comparison between scent, vision, and vision+scent. It is unclear to me why the scent features would be so challenging to learn from and specifically why they would harm the representation so permanently. A deeper study using only the scent features would be valuable to me. In its current state, it appears that these feature provide no additional information and are thus not necessary to include in the environment; breaking one of the primary motivating features of JBW: the multi-modality. I recognize that the paper comments on the orthogonality of the scent playing a role, and notes that further results are included on a not-yet-available website (presumably to maintain anonymity). However, I would like to see these results included in the appendix of the paper so I could better assess the utility of the scent features. Perhaps an additional result showing the average reward versus the cosine distance (or other measure of orthogonality) between *jellybeans* and *onions* would additionally motivate the utility of the scent features. The paper empirically investigates the use of curriculum learning to accelerate learning for a particular task. The paper then claims that curriculum learning improves learning speed, but ultimately does improve final performance. This demonstration is intended to showcase the use of the proposed environment (JBW) for curriculum learning. However, there are few key issues with this empirical study. First, the paper shows the reward rate of 3 different curricula but does mention the metric used to compare the agents during the time the curricula is active. It is implied that the metric is the reward rate of each individual agent; however, each agent has a unique reward function making comparisons between agents impossible. Curriculum #2 can only receive positive rewards while Curriculum #1 can only receive negative rewards. Naturally this means that Curriculum #2 must have strictly greater or equal reward rate over Curriculum #1. Even in the case that the final objective specifies the metric used, these are still highly non-comparable entities. A suggestion to improve this result would be to run each curricula for 100k as a *pretraining* phase, then to restart the agents to the same state in the environment and measure their performance from there. The case study measuring the effects of non-stationarity of the rewards does not provide sufficient evidence that the proposed environment contributes a novel ability to investigate non-stationarity. First, the given study of non-stationarity focuses solely on an alternating reward function, clearly demonstrating the problem of catastrophic forgetting. While this is a motivating demonstration, it is not novel and the issue of catastrophic forgetting in our models has been known since at least the 90s (e.g. French 1999 and related). Carefully and scientifically investigating such an issue is best done in a far less complex environment where more precise results can be drawn. Further, the ability to oscillate a reward function in this way is not unique to this environment and can be trivially done in most environments. Secondly, it is unclear if JBW allows for non-stationarity in the transition probabilities in the MDP. This is a critical component to non-stationarity and would be a necessary feature for me to claim non-stationarity is widely supported in the environment. The paper starts with a motivating conversation about environment complexity, with interesting insights into measuring the complexity of an environment based on the complexity of the policy used to solve that environment. However this conversation is ignored until the conclusion of the paper, where the paper claims to have built an environment of greater complexity than already existing environments. Without any supporting evidence in the body of the paper, it is impossible to verify the validity of this statement, and it is still an open question to me whether this claim is even falsifiable in the first place. As a concrete counter-claim, I would claim that the Minecraft environment (Malmo) has similar or higher complexity to the proposed environment in most aspects. Minecraft has a far greater diversity of objects, a third dimension of movement, adversarial components, hunger and health, etc. each of which adding a large level of complexity not achievable in the proposed environment. This is not to say that I expect the proposed environment to contain these features, but rather to point out that claims of greater complexity may be ill-founded. Additional Comments (not affecting score) I do slightly question if ICLR is the appropriate venue for such work. While I recognize that the scope of this conference has shifted considerably over the past few years, this paper (as written) does not further understanding or study of learning representations. I believe a more careful demonstration of the representation induced by characteristics of the environment is within easy reach of the paper, but is not currently presented. ----------- After the author response, reading other reviews/responses, and looking at the edited draft: I am convinced of the utility of the domain, the scope of the engineering effort put into building, and the ease with which it can be configured by the user to test many applicable settings (partial observability, stochasticity in transitions and rewards, etc.). I remain slightly skeptical of the amount of benefit the proposed provides over the Malmo environment for any of the settings discussed in the case-studies. I specifically feel my concerns about the stochasticity in the transitions and environment complexity have been well addressed. My concerns about the curriculum learning demonstration are partially addressed to a point where I am satisfied. My concerns about the modeling choice are also partially satisfied, with one lingering concern. I am unclear if the environment is trivially solvable by using more computation resources (e.g. bigger networks). However, after reconsideration I decided this concern bares less weight than I previously considered. All this considered, I am changing my rating from 3 -> 6.