Jelly Bean World: A Testbed for Never-Ending Learning
Authors: Emmanouil Antonios Platanios
Abulhair Saparov
Tom Mitchell
Venue:  (2020)

Abstract: Machine learning has shown growing success in recent years. However, current machine learning systems are highly specialized, trained for particular problems or domains, and typically on a single narrow dataset. Human learning, on the other hand, is highly general and adaptable. Never-ending learning is a machine learning paradigm that aims to bridge this gap, with the goal of encouraging researchers to design machine learning systems that can learn to perform a wider variety of inter-related tasks in more complex environments. To date, there is no environment or testbed to facilitate the development and evaluation of never-ending learning systems. To this end, we propose the Jelly Bean World testbed. The Jelly Bean World allows experimentation over two-dimensional grid worlds which are filled with items and in which agents can navigate. This testbed provides environments that are sufficiently complex and where more generally intelligent algorithms ought to perform better than current state-of-the-art reinforcement learning approaches. It does so by producing non-stationary environments and facilitating experimentation with multi-task, multi-agent, multi-modal, and curriculum learning settings. We hope that this new freely-available software will prompt new research and interest in the development and evaluation of never-ending learning systems and more broadly, general intelligence systems.

Published as a conference paper at ICLR 2020
JELLY BEAN WORLD: A TESTBED FOR DEVER-ENDING LEARNING
Emmanouil Antonios Platanios", Abulbair Saparov' & Tom Mitchell Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA 15213, USA le.a. w.a.ppaaaanosss. asaparov, com.mitche 11 @css cma. edu
ABSTRACT
Machine learning has shown growing success in recent years. However, current machine learning systems are highly specialized, trained for particular problems domains, and typically on a single narrow dataset. Human learning, on the other hand, is highly general and adaptable. Never-ending learning is machine learning paradigm that aims to bridge this gap, with the goal of encouraging researchers to design machine learning systems that can learn to perform a wider variety of inter-related tasks in more complex environments. To date, there is no environment or testbed to facilitate the development and evaluation of never-ending learning systems. To this end, we propose the Jelly Bean World testbed. The Jelly Bean World allows experimentation over two-dimensional grid worlds which are filled with items and in which agents can navigate. This testbed provides environments that are sufficiently complex and where more generally intelligent algorithms ought to perform better than current state-of-the-ar art reinforcement learning approaches. It does so by producing non-stationary environments and facilitating experimentation with multi-task taski multi-agent, multi-modal, and curriculum learning settings. We hope that the Jelly Bean World will prompt new interest in the development of never-ending learning, and more broadly general intelligence.
INTRODUCTION
Machine learning has witnessed growing success across a multitude of applications over the past years. However, despite these successes, current machine learning systems are each highly specialized to solve one or a small handful of problems. They have much narrower learning capabilities as compared to humans, often learning just a single function or model based on statistical analysis of a single dataset. One reason for this is that current machine learning paradigms are restricted and specialized particular problem and/or dataset. An alternative learning paradigm that more closely resembles the generality, diversity, competence, and cumulative nature of human learning is never-ending learning (Mitchell et al. (018). The thesis of never-ending learning is that we will never truly understand machine learning until we can build computer programs that, like people: (i) learn many different types of knowledge or functions, (ii) from years of diverse, mostly self- supervised experience, (ii) in a staged curricular fashion, where previously learned knowledge enables learning further types of knowledge, and (iv) where self-reflectton reflection and the ability to formulate new representations and new learning lasks enable the learner avoid stagnation and performance plateaus. Building computer programs with these properties necessitates well-defined and robust ways to evaluate whether a system is indeed capable of never-ending learning. However, there currently no ways to achieve that. There only exists one targe-scale case study on never-ending learning with the Never- EndiinEnding Language Learning (NELL) system by Mitchell et al. (2018), which uses the internet as the environment with which the system interacts. While the internet does have significant complexity, it is unwieldy to use testhed. It is very difficult to focus on particular aspect of the system or the environment, or to tweak the algorithm and restart experiments to observe the effects of changes. Furthermore, oftentimes tasks require manual annotation which can be very expensive. Thus, a good testbed for never-ending learning (and machine learning more generally) needs to provide the experimenter with a high degree of control. To this end, we propose a novel evaluation framework-the Jelly Bean World (JhW)-that can enable and facilitate research towards the goal never-ending learning. We have designed the JBW W to be highly versatile, enabling evaluation of systems that have any number of the aforementioned abilities.
We consider never-ending learning in the context of reinforcement learning. Let 8, E denote the state of the environment at time I, E A denote the action performed by the learning agent at
"Equal contribution (listed in alphabetical order).

---

Published as a conference paper at ICLR 2020
time t, Wa S denote the observation of the world that the learning agent receives at time t, and I' E IR denote the reward provided to the learning agent at time I. The distribution of the next state of the world 8; 1
has the Markov property (i.e., it depends only on the previous state and action) and the initial state of the world is given by a distribution 80 ~ W. The observation wi ~ O(s;) depends only on the current state of the world (perhaps deterministically) The reward ry is given by function R(s--1, a;-1, St) of the current state, the previous state, and the previous action taken. The environment is tuple containing all these elements & - (W, O, R). Then, the goal of reinforcement learning is to find learning algorithm æ that, given the history of previous observations, actions, and rewards, outputs the next action so that the obtained reward is maximized. We deliberately blur the distinction between the policy and the algorithm that learns the policy, which is why we call "learning algorithm."
This formalism does not distinguish between learning algorithms that are highly specialized to a single task and learning algorithms that are capable of learning a wide variety of tasks and adapting to richer and more complex environments, which are hallmarks of general intelligence. In order to more formally describe general intelligence, we posit that there is an underlying measure of complexity of the environment & such that: (i) highly specialized and non- con-general learning algorithms can perform well in environments with low complexity, but (ii) environments with high complexity require successful learning agents possess more general learning capabilities. It is in these more complex environments where we can characterize never-ending learning. We can formalize this notion of complexity by letting n' be the (computable) learning algorithm that maximizes expected reward in an environment E. Then we define the complexity of & to be the length of the shortest program (Turing machine) that implements '=:
complexity(E) = min : T is a Turing machine that implements x
complexity())
exhibits specialized intelligence
exhibits more general intelligence
We can equivalently define complexity(E) = -K(m K(TT), where K(-) is the Kolmogorov complexity and is related to the minimum description length and minimum message length (Nannen 2010
Kolmogorov 1963). The Kolmogorov complexity of the environment K (E) is bounded below by
K(T") minus constant. This bound is shown in Section A. of the appendix.
Never-ending learning is in many respects similar to lifelong learning, also called continual learning (Chen & Liu 2018). Like never-ending learning, lifelong learning is distinguished from multi-task learning by the never-ending nature of the learning problem. However, in never-ending learning. and unlike multi-task learning and lifelong learning (to the best of our understanding), a well-defined set of tasks is not assumed a priori. Rather, never-ending learning is more similar to real-world settings in this respect, where the notion of a task or subtask naturally emerges from the complexity of the environment, and the distinction between tasks is not always so sharp. Regardless, due to their similarities, a good testbed for eever- learning will also be a good testbed for lifelong learning.
In contrast to most popular reinforcement learning settings, never-ending learning focuses on envi- ronments with high complexity. In never-ending learning, agents can only exist in single reset-free environment (i.e., we explicitly disallow the agent æ from learning across multiple episodes or in multiple environments, which is closer to human learning). We require single episode. During its lifetime, π can only use the information provided by its past observations {w:) } and actions {ar} in single world. Thus, never-ending learning explicitly removes the distinc- tion between training and testing that is common to many other classical machine learning paradigms. Additionally, note that in the general reinforcement learning formalism, S; can contain information about f, and the reward function R can be vvaring thus rendering the environment non-stationn statiootton In never-ending learning, we are interested in the full generality of -ta-ioation environments, as the assumption of stationarity is not realistic in even simple adversarial and multi-agent settings. We thus argue that a testbed for never-ending learning should have the following properties:
only have access to
1. Non-Episodic: It should disallow agents from resetting the environment and "retrying". The testbed should also force them to only learn within a single environment (i.e., not transfer information across environments). This is in contrast with most popular reinforcement learning environments and, as we show in Section 4 poses significant challenges existing algorithms.
2. Non-Stationary: The testbed should allow for easy experimentation with non-stationary envi- ronments, where the reward R can depend time.
2

---

Published as a conference paper at ICLR 2020
Table I: Existing reinforcement learning environments positioned based desired properties.
Environment
Non-Episodic
Non-Stationary
Multi-Task
Multi-Modal
Controllable
Efficient
X Games end when the player wins loses
X
X
x Agents only observe the game video frames
X Modifying the task complex-
Atari and Retro Games
The game mechanics are stationary
Each game has single fixed reward function
Can run on small machines but models slow to train
Bellemare al. 2013
Plau et a 2015
ity/richness is not possible
x Stationary rewards and environments (i.e., physics)
/ Some of the tasks have interesting hierarchical structures
X Agents only observe positional information and joint angles
X The tasks and environments are non- configurable
~v Efficiency varies widely across tasks
Some tasks are non-episodic (c.g., swimmer)
Continuous Control Duan all 2016 Todorov et 2012
X Episodic in a finite world
x Stationary environments (i... physics)
x Only navigation goals are supported
/ Multiple different kinds of sensors
V Fast 2D simulation written in C++
Evolutionary Robotics Mouret & Doncieux 2012
Configurable using XML
X
X Existing levels are not configurable
BabyAl
Episadic in finite world
Fixed set of levels and rewards
Handful of tasks to
Agents given visual Input and instructions
Built fast MiniGrid simulator
Chavaller Boisvert al. 2018
perform
X Each game has single fixed reward function
Adversarial Games like Go Silver et al. 2017|
Games end when the player wins loses
Non-stationary (without assumptions about the adversaries)
Agents observe the game video frames the game state
There limited control over things like the adversary's competence
Experiments are typically extremely com- putationally expensive
StarCraft Vanyal 2019
and Dota OpenA/ 2019
X Agents only observe the game video frames
X
Levels have time limit
Levels have different rewards but same physics
Levels have predefined rewards
The complexity of each level fixed
Requires rendering of world
DeepMind Lab Beattie al. 2
Malmö MineRL Johnson al 2016)
Tasks have pre-specifed
Stationary rewards and map generation based on Perlin noise
Supports 6 complex tasks but also allows for new ones
Agents observe the game video frames and the game state
Modifying the task complex-
Requires
rendering of 3D world and slow training of
time limit but that is typically
ity/richness is difficult and expensive
Glas 2019
very long
largo models
/ Modifying the task complex- ity/richness is
Agents live "forever" in an infinite open world
The rewards and the world can both be non-stationary
Composable and dynamic tasks supported
Vision and scent are designed to be complementary
Experiments can run efficienlly on small machines
Jelly Bean World (Proposed Enviranment)
very easy
3. Multi-Task: It should support settings in which reward is maximized not by learning how to perform a single task repetitively, but by learning how to perform a general variety of tasks, and learning how to switch between them and/or combine them to better perform other tasks (e.g., by composing them). We posit that, at sufficiently high level of task complexity, optimal learning agents will required either explicitly or implicitly over concepts and make informed decisions about actions in the environment.
perform abstract reasoning
Multi-Modal: It should support multiple data modalities that agents receive as input. These modalities should not contain the same information, but rather be complementary to each other forced learn from diverse types of experiences. Multi-modality provides
so that the agents yet another way to increase the complexity of the world.
5. Controllable: It should be easy for experimenters to modify the complexity and richness of the learning problems in the testbed, make changes to it, and restart it (c.g., as opposed to NELL).
6. Efficient: It should run on readily available hardware and allow for quick experimentation. Ideally, we should not have to wait for days, weeks, or months (e.g.. NELL) to obtain results.
7. Reproducible: It should make it easy to reproduce results and experiments, which would facilitate scientific research. This also requires that allows for seamlessly saving and loading state and for reproducing results outside the environment in which they were first obtained. The testbed should also not require access to specialized hardware, which can be expensive.
Many of the above properties are means to increase the complexity of the world. These properties are in fact very closely related to the characteristics of "AGI Environments, Tasks, and Agents" outlined by Laird& & Wray III (2010) and later refined by AAams et al. (2012). The proposed JBW has all of these properties. It aims to provide an easy way to create sufficiently complex environments allowing researchers to experiment with never-ending learning. while remaining simple enough control the problem and enable rapid prototyping. The JBW is a two-dimensional grid world with simple physics, but is extensible enough admit wide variety of complex and inter-related tasks. We present
3

---

Published as a conference paper at ICLR 2020
Simulator
Environment
Advances time after managed agents have acted, invoking modules as needed.
Interface reinforcement leamiig.
MAP
VISION
REWARD
Manages the infinite world
Simulstes the visuel field of all managed agents.
Specifies the reward given to the agent for each possible state transition.
Collect (JellyBaa))) [Onion)
Ocolusion
Represevieed as tansoc
REWARD SCHEDULE
Specifies the reward for each time step Fixed Perfodic Randon
Hibld of view
AGENTS
SCENT
Visualizer
Manages agents and handles their interaction with the simulator.
Simulates the diffusion of scent in the world.
Asynchronous simulation visualizor.
Represonted
Distributed simulations are also sugported using MPt.
Figure 1: Overview of the modules comprising the Jelly Bean World.
comparison with related work in Table 1 showcasing the ways in which the JBW is novel and highly versatile evaluation framework. The JBW is written in C++ and we provide C. Python, and Swift APIs, and is available at https: / / com/ heappapppaaaiosoooo je jellyy bean wor 1d It is also worth noting that the JBW has already been used for the instruction of the "Deep Reinforcement Learning" and "Never-Ending Learning" graduate courses at Carnegie Mellon University.
DESIGN
The Jelly Bean World (JBW) consists of the following main modules (illustrated in Figure Th: (i the simulator, which comprises the central component (the other modules only interact with the simulator), (ii) the environment, which provides simple interface for performing reinforcement learning experiments in the never-ending learning setting as well as utilities for evaluating never- ending learning systems, and (iii) the vismalizer, which provides the ability to visualize and debug the behavior of learning agents. Note that the visualizer is completely asynchronous and can be attached, reattached, and detached to and from existing simulator instances, without affecting the simulations.
2.1 SIMULATOR
The simulator manages map and a set of agents. At a high-level, the map is an infinite two- dimensional grid where each grid cell can contain items gc.g..jelly beans and onions) and/or agents. Each item has color and a scent that agents can perceive. Each agent has direction and position, and can navigate the world map and collect or drop items. The action space of each agent is: to turn, move, collect items, drop items, or do nothing. The action space is configurable and can be constrained by the user. These constraints are described later in this section. Time in the simulator is turn-based, meaning that the simulator will first wait for
discrete, and all agent- map interactions all managed agents to request an action and will then simultancously execute all actions and advance the current time. Thus, the simulator also controls the passage of time.
Map. In order to truly support never-ending learning. we have designed the JBW map to be infinite, meaning that it has no boundaries and agents can keep exploring it forever. To achieve this, the map proceduralll-generaae
two-dimensional grid. We simulate it by dividing it into collection of disjoint (P x P)-sized patches and only generating patches when an agent moves sufficiently close them. The map also contains items of various types which caarwise-intractiine point process over the two-dimensional grid (Baddeley & Turner 2000). More
distributed according
specifically, for a collection of items I A [To...., Im), where I = =(x,, fx), 24 E 2² the position of the 7th item, & E T is its type, and T is the set of all item types:
p(I) x exp Σ f(.)+ + ((T,,11)) E
(1)
where f(T.) is the intensity of item I, and g (1, 13) is the interaction between 7, and 7,. which are provided as part of the item's type. The intensity function characterizes the (log) probability of the
4

---

Published as a conference paper at ICLR 2020
existence of an item I, independent of other items in the world. The interaction function can be understood as a description of the (log) probability of the existence of item I; given the existence of all other items I5. For example, the interaction function can be used to increase the log probability of an item when it appears near other items, producing clustering effect. Since the world is subdivided into (P x Pi-izzdd patches, the maximum distance of interaction between items is P.
Item Types. Each item type T defines the following:
Non-final patches that need be sampled to
New patches that need to be sampled
Color: Fixed-size vector specifying the item color.
Scent: Fixed-size vector specifying the item scent.
avoid boundary
Occlusion: Occlusion of item (relevant to the vision modality, described later in this section).
effects
Intensity Function: Maps from item locations real values.
Interaction Functions: Collection of functions that map from pairs of item locations to real values. The collection contains one function for each item type.
The number of item types and their properties are configurable The specific parametric forms for the intensity and interaction functions that are supported are described Section AA22 of the appendix. Note that each item type also specifies additional properties that are described later in this section.
Figure 2: Illustration of the procedu- ral generation algorithm for the infinite world map. The 32 x patches shown white have already been sampled and those in gray have been sampled but not fixed in order to avoid boundary effects. The red line corresponds to an exam- ple path followed by an agent. Once the agent enters patch that is not fixed, then that patch sampled, along with its non- non-fixed neighboring patches in order avoid boundary effects.
Procedural Generation. When the simulator is instantiated the map is empty (i.e., patches have been generated). When- ever a new agent is added to the simulator, a patch centered at its location is generated. In addition, whenever existing agent moves sufficiently close to a region where no patch ex- ists, new patch is generated. The patch generation process consists of two main steps: (i) add a new empty (P P)-sized patch to the collection of map patches (note that the new patch will be neighboring at least one existing patch and that all patches are disjoint), and (ii) fill the new patch with items. The second step is performed by using Metropolis-Hastings (MH) (Robert & Casella 2010) to sample the items that the new patch contains, from the distribution defined in Equation D The proposal density we use is defined as follows: (i) add a new item 4m+1 (Im+1, fm+1) with probability 1/(2p22 [T]) (i.c., uniform in position and type), and (ii) remove an existing item I; with probability 1 1/2m where m is the current number of items in the patch. Before sampling, the patch initialized by first randomly selecting an existing patch and copying its items into the new patch. This intended to facilitate rapid mixing of the Markov chain, and reduce the number of MH iterations. Note that if we use small patches and only sample new patches as the agents visit them, boundary effects may be observed due to the missing neighboring patches further away from the agent. For this reason, we actually also sample all missing neighboring patches while sampling each new patch, but do not finalize them (i.e., they are still considered missing and may be resampled later on). This prevents boundary effects during the procedural generation process. An example is shown in Figure 2
Scent Diffusion
Visual Field
Occlusion
Agent
Field-of- viwee Mask
Rendering of an agent's per- spective from the JBW visualizer.
Figure 3:
Each item has color and a scent that is specified by its type and can be perceived by agents. The JBW thus supports two perception modalities, vision and scent. These modalities are complementary and agents can benefit by learning combine them, we explain at the end of this section.
Vision. Each agent has a visual range property that specifies how far they can see. Vision is represented as a three-dimensional tensor, where the first two dimensions correspond to the width and the height of the agent's visual field, and the third dimension corresponds to the color dimensionality. The visual field is always centered at the agent's current position and the color observed at each cell within the visual field the sum of the color vectors of all items and agents located at that map location. Agents also have field of view property that specifies their field of view angle (i.e., 180 denotes that the agent can only see the forward- -acing half of the visual field, whereas 360 denotes
5

---

Published as a conference paper at ICLR 2020
that the agent can the whole visual field). The part of the visual field that is outside an agent's field of view is masked out and appears black to the agent. Another important aspect of vision is that items also have an occlusion property as part of their type. This is used to simulate partial or complete visual occlusion (details in Section A.3) If an item with occlusion is in an agent's visual field, then the colors behind that item are not visible to the agent. An example is shown in Figure S
Scent. Scent is represented as lixcd-dimensional vector, where each dimension can be used to model orthogonal/unrelatee scents. Each agent and each item has a pre-specified scent vector that is provided part of the world configuration (similar to their colors). At each time step, agents can perceive the scent at their current grid position. The physics of scent are described by a simple diffusion difference equation on the world grid. We define the scent at location (z, v) at time t as:
X.St
+ +a + +
+ S - S 11
(2)
current Sternsdagents scent
previous scent
neighboring cells diffused scent
where \ is the rate of decay of the scent at each location, is the rate of diffusion of the scent from neighboring grid cells, and Czy = ΣxEIS,, scent(I) + ΣeEAt, scent (A), where Th,y is the set of all items at time and location (x, y), and Ai., is the set of all agents at time L and location (æ, y). Our simulator ensures that the scent (or lack thereof) diffuses correctly, even as items are created, collected, dropped, and destroyed. It does so by keeping track of the creation, collection, drop, and destruction times of each item in the world. Note also that, while simulating this diffusion, we also take into account the non-fixed patches that have been sampled in order avoid boundary effects.
Vision-Scent Complementarity. Vision and scent are complementary. Vision has high precision, in the sense that the agent see the actual color of each grid cell in its visual field and can thus relatively accurately determine what items may exist in that cell. However, it has low recall- the agent can only see as far as its visual range allows and it has no visual information about the rest of the map. On the other hand, scent has low precision- the scent at the current cell is a linear combination of the scents of all items in the world and it may be very difficult to learn to interpret and use effectively. However, scent has high recall- -the scent the current cell contains information about items in a much larger range. Thus, learning to use both modalities will be beneficial to agents. we also provide some experimental results supporting this argument.
In Section 4
Constraints. The simulator enforces multiple constraints on the actions that agents are allowed to take. We have designed the following small set of constraints with the goal of providing computationally efficient way to support arbitrarily complex tasks and learning problems:
Agent Collision: This occurs when multiple agents attempt to move to the same location at the same time. This conflict can be resolved in one of three ways: (i) allow multiple agents to occupy the same location, (ii) first-come-first-serr (only allow the first agent who made a move request for that location to actually move-this is the current default), or (iii) randomly choose one of the agents and satisfy their request (ignoring the requested action of the others).
Item Blocking Movement: Item types may specify that they block agent movement (e.g., a Wall item type). This means that agents are not allowed to move to locations with items of that type. Item Collection Requirements: Item types may specify that in order to collect items of that type, an agent has to have first collected a specified number of other items (e.g., collecting wood may only be allowed if the agent has first collected an Axe).
Item Collection Costs: Similar to the collection requirements, item types may specify that in order to collect items of that type, an agent has to drop or destroy specified number of other items (c.g., collecting an Axe may require destroying a piece of Metal and a piece of Nood that the agent has previously collected).
Interface. Users interact with the simulator programmatically. JBW provides functions to add or remove agents from the world, query the current vision and scent perception of each agent, and to direct agents to perform actions. Users can choose to add multiple agents to the world, thus enabling experimentation with multi-agent settings. Multi-agent interactions provide another controllable source of complexity in the JBW. Users can then request actions for each agent in the simulation (i.e., turn, move, do nothing, etc.). Once all agents have requested actions, the simulator executes these actions and advances time, appropriately updating the state of the world.
Server/Client Support. The JBW also provides a TCP server-client interface where the simulator can be setup to run as a server. Users (i.e., clients) can then connect to the server, and interact with the simulator by sending messages to the server. This allows use cases such class setting where

---

Published as a conference paper at ICLR 2020
students each control agent in common simulated world, or perhaps hackathon where participants can compete a common world. It also allows debugging and visualization tools to be attached and detached to and from running simulator instances, without affecting the simulations. In fact, this how our visualizer, which described Section 2.3 communicates with the simulator.
Persistence. Simulations in the JBW can be saved to and loaded from files, which can then be distributed across platforms. This facilitates reproducibility. The simulator guarantees uniform random number generation behavior across all platforms and machines (e.g.. s. in distributed settings). The state of the pseudorandom number generator is also saved and loaded along with the simulation.
2.2 ENVIRONMENT
Environments manage simulator instances and provide an interface for performing reinforcement learning experiments using the JBW. We provide implementations of the JBW environments for OpenAl Gym (Brockman et al. 2016) in Python and for Swift RI. (Platanios) 2019) in Swift. JBW environments support batching by design, with support for parallel execution of the multiple simulator instances being managed (i.c., one simulator for each batch entry). Perhaps the most important aspect of JBW environments is that they require the user to specify reward schedule to use for each experiment. This schedule effectively defines the tasks that the agents are learning to perform. A reward schedule provides function that, given a simulation time, returns reward function to use at that time. A reward function returns a scalar reward value, given the current and previous states of the agent and the world (e.g., the world map) []] We provide simple domain-specifer language (DSL) for composing and combining multiple reward functions in arbitrary ways, allow for the design of composable learning tasks. This enables endless possibilities in the realms of multi-task learning, curriculum learning, and more generally never-ending learning. Currently environments are limited to single agent reinforcement learning settings, but we plan to support multi-agent settings in the future (this is easy because the JBW simulator already supports multiple agents for each simulation).
2.3 VISUALIZER
Visualization can be instrumental when developing, debugging, and evaluating never-ending learning systems. To this end, we have implemented a real-time visualizer using Vulkar in which the user can see any part of the simulated JBW, at any scale and simulation rate. The visualizer utilizes the simulator server-client interface to visualize simulations running in different processes on remote servers, in fully asynchronous manner. Rendering is multithreaded to provide a smooth and responsive user interface. Finally, the visualizer can be attached, detached, and re-attached to existing simulation server instances, without affecting the running simulations.
3 LEARNING TASKS
Learning tasks can be defined in terms of reward functions and reward schedules, which were defined in Section 2.2 The JBW allows researchers to easily define their own reward func- tions and schedules, but it also provides a few primitives and ways to compose them in or- der to effortlessly allow for quick experimentation and prototyping. In fact, all learning tasks used in Section were defined using these primitives. The currently supported primitives are:
Reward Functions
Reward Schedules
Give agents when they take action (i.e., not no-op).
The reward function always foxe to r. and is
[ction[o]
F Fixed [r]
thus stationary.
Give to agents for each item of type i that they collect.
Use reward function r. for the first (y steps, then ra for steps, and keep using rx after the list of reward functions is exhausted. Use reward function for the first steps, then ra for bo steps, and then repeat
Collect [4,
Curriculun (rr, 41
Give agents each time they move further away from their
Explore [+]
Cyclicall (r).
starting position in the world map.
after the reward functions exhausted.
We note that Co eect is a sparse reward function, whereas Action Lon and Exple Explore are not. For concise- ness, we omit the v argument in reward functions when it is set to 1 and we also define Avoid (i, v] =collcet -v].
Reward Function Compositions
Combined
Applies both and my and returns the sum their rewards.
Pa Ar
A simple reward function could he one that gives the agent 1 reward point for each Je1 1 yyean it collects. e Information Vulkan can be found at httttp /www. wwrokos wwwp:uuoooooooooooooo/oooooo kan/
7

---

Published as a conference paper at ICLR 2020
4 EXPERIMENTS
Table 2: Simulator configuration.
The goal of this section is to show how the non-episodic, non-stationary, multi-modal, and multi-task aspeets of the JBW make it a challenging environment for existing ma- chine learning algorithms, through few example case studies. For all experiments we use the simulator config- uration and item types shown in Tables 2 and 3 Due space, the case studies focus on the single-agent setting, We use different agent models depending on which modal- ities are used in each experiment. If vision is used, then the visual field is passed through convolution layer with stride 2, 3 x 3 filters, and 16 channels, and another one
Scent Dimensionality
Color Dimensionality
Map
Patch Size
64
MH Sampling Iterations
10, 000
Scent Decay (A)
0.4
Scent Diffusion (cx)
0.14
Color
[0.00, 0.00, 0.00]
Scent
[0.00, 0.00, 0.00)
Agent
Povaaor
Action Space
TurnbeIt,
and TurnRight
Visual Range
Field-of-View
experiment-sper
with stride 1, 2 filters, and 16 chan- nels. The resulting tensor is flattened and passed through dense layer with size 512. If scent is used, then the scent vector is passed through two dense layers: one with size 32, and one with size 512. If both modalities are being used, the two hidden representations Finally, the result is processed by Long Short-Term Memory (LSTM) network (Hochreiter & Schmidhuber 1997) which outputs& value for the agent's current state, along with distribution over ac- tions. Learning is performed using Prox- imal Policy Optimization (PPO); popu- lar on- on-policy reinforcement learning algo- rithm proposed by (Schulman al. (017)) The experiments are implemented using
Table 3: Item types. See Section A.2 for details on the func- tional forms of the intensity and interaction functions.
Je JellyBean: Jeliy beans appear close to bananas.
Scent
[1.64, 0.54, 0.40)
Color
0.82, 0.27.0.20
Occlusion
0.0
Blocks Agents
concatenated.
Intensity
Constant [1.5]
JellyBean: Pwwcewiseexx (10, 100,
Interactions
Janana
:PiecewiseBox (10, 2,,,1,,,,,,,2000
PPecceisseoo [50, 1100,,01110--0011
Hanana: Bananas appear close to jelly beans and away from walls.
Scent
1.92, 1.76, 0.40
Color
0.996, 0.28)
Occlusion
Blocks Agents
Folse
Intensity
Constant (1.5)
yelly Piecewt seRox (10, 0, 2,, 2, CD)
Interactions
Sanana
<.iecewiseBox [10, 2,0,,00000 130, 06-
Swift for TensorFlow
Mall
PPeeoewiseee [50, 130, -100,
Onion: Onions appear scattered all over the world.
0.68, 0.01
4.1 CASE STUDIES
00668, 0.99
Occlusion
For all experiments we evaluate per- formance using the reward rale metric, which is defined as the amount of re- ward obtained per step, computed over moving window. The size of that win- dow varies per experiment and is reported together with the results. This is an ap- propriate metric for this task as we want to measure the improvement in the abil- ity an agent to learn (i.e., the gradi- ent of the reward rate), while also mak- ing sure the agent does not get stuck (i.e.. the reward rate goes to zero). Whenever possible, we also report the results ob- tained by the greedy dision-based agent described in Section A.7 of our appendix. The greedy agent makes additional as- sumptions about the world, and thus, doesn't 't generalize to more complex envi- ronments, it provides lower bound the optimal reward rate. Note that per- feet upper bound cannot be obtained
Blocks Agents
Palse
Intensity
Constant [1.5]
Interactions
None
Wa Walls
contiguous axis-aligned.
Scent
0.000 0.00, 0.00]
Color
[0.20, 0.47, 0.67]
Occlusion
1. experiments with occlusion. 0.0 otherwise
Blocks Agents
True
Intensity
Constant (-12)
Interactions
Mal1
: Cross (23, 40, 8,-1000, 1111000011100 -1500,
Tree: Trees cluster together Irregular shapes.
Scent
[0.00, 0.47, 0.06)
Color
[0.00, 0.47, 0.06]
Occlusion
0.1 in experiments with occlusion, 0.0 otherwise
Blocks Agents
True
Intensity
Constant [2]
Interactions
Tree
:PiecewiseBox [100, 500, 0, 0.1]
Truffle Truffles appear in forests and are very rare.
Scent
88.00, 4.80, 2.60)
Color
(0.42, 0.13
Occlusion
0.0
Blocks Agents
zalse
Intensity
Constant(0) [0]
Truffla
:Piecowisesox (30, 1000, -0.3, Pieccexi 200, 2,00
Intcractions
Tree
An example non-stationary world configuration is also presented in Section A.5 of our appendix.
https: http://wwwwwtoofffooooooo
8

---

Published as a conference paper at ICLR 2020
that would require solving an NP-hard discrete optimization problem.
Collect Collect(DetyBaan)/Avcdddoooi
Case Study #1: Non-Episodic. The goal of this case study is to show that the JBW allows for ex- perimenting with never-ending learning agents and to also show how current machine learning methods (e.g., PPO used to train an LSTM-based agent) are failing to effectively perform never-ending learning. For this experiment we use the fixed reward func- and
Reward Rate (pts/s)
0.03-
0.02-
0.01
mm
tion Ccllect [JellyBean] A Avoid[Cnion. let agents interact with the JBW for 10 million steps. Our results are shown in Figure E The agents seem to be learning effectively for the first 1 million steps, but start to underperform later on, eventually getting stuck and being unable to collect any reward. This is the case for multiple different learning agents that we experimented with; both using different models and using different learning algorithms,, such Deep Q-Networks (DQNs) proposed by MMihh et al. (2013). After connecting the visualizer to observe what happens
0.00
2M
4M
6M
8M
10M
Time Step
Figure 4: Non-episodic experiment result. The reward rate is computed using 100,000-step window and the shaded bands correspond to stan- dard error over 20 runs.
that all agents either: (i) get stuck in an area of the map that they have already explored and exhausted of jelly beans, or (ii get stuck constantly rotating and not moving to new grid cells at all. This indicates that the JBW is indeed challenging for current machine learning methods when it comes to never-ending learning. Perhaps some sort of reward shaping or curriculum learning could help the agents. However, our goal with this paper is not to solve these hard problems but rather point them out and show how the JBW provides a testbed with which to tackle them.
Case Study #2: Non-Stationary. The goal of this case study is to demonstrate that the JBW allows for experimenting with non-stationary and multi-task learning problems. To this end, we perform two experiments: (i) one using a cyclical/periodic reward function schedule where every 100,000 steps alternate between the Collect [JellyBean] AAvoid [nnoon] and Avoid [ JellyBear: ACollect [Onion] reward functions, and (ii) one testing a couple of curriculum reward schedules for eventually learning to Collect [JellyBean] AAvoid Cnion]. ). The results are shown in Figure 5 and we observe that current standard machine learning approaches are not able to efficiently alternate between different learning problems and are effectively learning each problem from scratch whenever they switch, eventually ending up unable to learn either one. We also observe that agents who first learn to collect jelly beans and then switch to the full reward function are able to learn to collect jelly beans and avoid onions faster than agents that first learn avoid onions or face the final learning problem directly from the beginning. Eventually all agents perform similarly, but this showcases how the JBW enables research in currieulum learning.
Case Study #3: Multi-Modal, The goal of this case study is to: (i) show how computationally efficient features, such as the field of view mask and visual occlusion, allow for increasing the learning problem complexity in controllable manner and, perhaps most importantly, (ii) show how the perception modalities of the JBW are complementary. We thus perform three experiments. For the first two we use the fixed reward function Collect (JellyBean and for the last one we use Co Collect [nnio]] We change the reward function in order to show how easy it is to experiment with different tasks in the JBW. In the first experiment, we vary the field of view of the agents. The results are shown in the left plot of Figure We see that decreasing the field of view allows us to make the learning task harder for agents, while maintaining the same computational cost for the environment. Similarly, in the second experiment we measure the effect that visual occlusion has on performance. The results are shown in the middle plot of Figure and we observe that enabling visual occlusion makes the learning task harder. Finally, with the third experiment our goal is to show that vision and scent are complementary. The results are shown in the right plot of Figure 6 We see that "vision" agents do better than "scent" agents, indicating that vision is perhaps an easier perception modality to use in the context of this learning task. Surprisingly though, the "vision" agents also do better than the "vision+scent" agents. This indicates a limitation of the model because, even though scent contains useful information that vision does not, the agents seem to get confused by it and do not seem able use it properly. also shows the need for better multi-modal algorithms and the utility of the JBW in testing such algorithms. The relative utility of scent and vision depends on the environment and the agent model. We demonstrate this in Section A.4 by providing a different configuration where "scent" agents are able to outperform both "vision" and "vision+scent" agents.

---

Published as a conference paper at ICLR 2020
Cyclical Schedule
Curriculum Schedule
Fital Reward Function L [3olty
Curiculumm Length
Reward Rate (pts/s)
0.08
0.025
The renardiisee
collact[@vion)
Reward Rate (ots
areiil sperporabee
curriclle
0.04
0.020
bofore This point
The onccton aay set the fraal one.
0.00
CollectrDellybcan)
0.010
0.015
Forthe fest reetee lundionis Avo (oo [Onion]
Dumicukum #1
steps
Avaid
Dumicalam #2 the reward funesonus set Coll Twott (See (ythee))
0.08
0000
800k
1.2M
1dM
600k
800k
me Step
Time Step
Figure 5: Non-stationary experiment results. The reward rate is computed using 100,000-step window and the
shaded bands correspond to standard error over 20 runs.
(ellect(DellyBean|
Co Ceutect| [nion]]
Field-of-View
Visual Occlusion
Vision/Scent Complementarity
Reward Rate ipts/s)
Greedy Visual: 0.164
0.04
0.03
0.02
Vision+Seent
0.051
360
Vision
0.01
Greedy
Visual
0.050
270"
0.01
Without Occlusion
0.03
Scent
0.055
90'
With Occlusion
0.00
0.00
0.00
100k
200k
300k
400k
500k
0
100k
2008
300k
400k
500k
200k
400k
600k
800k
1M
Time
Step
Step
Time
Step
Figure 6: Multi-modal experiment results. The braces on top of the plots specify the reward function used each case. The reward rate is computed using a 100,000-stey window and the shaded bands correspond to standard error over 20 runs. "Greedy Visual" refers to the reward rate obtained by the greedy visual agent baseline described in Section A.7 (which, as explained in that section, is not able to handle occlusion or avoidance).
CONCLUSION AND FUTURE EXTENSIONS
We presented a new testbed designed to facilitate experimentation with never-ending learning agents, where the complexity of the learning problems is higher than that of existing testbeds and evaluations, while maintaining controllability, performance, and reproducibility. In order to produce more complex environments, the JBW supports non-stationary environments, with multiple distinct but inter-related tasks and complementary perception modalities. The JBW also explicitly restricts learning to single never-ending episode. It is highly configurable and performant, and provides users with tools to easily save, load, distribute environments, and reproduce and visualize results. We also showed how easily we can define learning tasks in the JBW, for which current machine learning methods struggle.
The space of potential extensions to the JBW is large. Although the current intensity and interaction functions stationary with respect space (i.e., they are independent of position x, y), it is not difficult to define new non-stationary functions, in order to generate worlds with non-stationary item distributions. The JBW supports multiple agents running asynchronously in the world, and so it would also be interesting to experiment with multi-agent settings. However, agents currently don't have an easy way to communicate with each other, and so adding mechanism for communication, perhaps via new agent-item interactions (e.g., reading/writing note items), would be interesting. Another way to add complexity is via items that can contain "strings" (c.g.. notes) in an agent-specific language, or even natural language. These notes could, for example, contain task specifications. Scent eurrently does not interact with items in the world, meaning that it can pass through Wal 1 items without any hinderance. Thus, another possible extension would be to support more complex item-scent interactions. It would be interesting to explore interactions between items and the properties of agents (e.g.. a Telescope could extend the visual range of an agent while narrowing its field of view). Finally, another interesting way to add complexity is to generate richer relationships between item types, perhaps even an ontology. We look forward to continue improving the JBW, and hope that a standardized testbed for eever-ending learning will motivate research into more generally-intelligenn learning algorithms.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their helpful comments and the Spring 2018 class in "Never- Ending Learning" at Carnegie Mellon University for the impulse and discussion that led to the devel- opment of this framework. This research was supported in part by AFOSR grant FA9550171021
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Sam Adams, Itmar Arel, Joscha Bach, Robert Coop, Rod Furlan, Ben Goertzel, J Storrs Hall, Alexei Samsonovich, Matthias Scheutz, Matthew Schlesinger, et Mapping the Landscape of Human-Level Artificial General Intelligence. Al magazine, 33(1):25 42, 2012.
Austin Appleby. MurmurHash. httpss///stees..gggll..c./////es
2008.
Adrian Baddeley and Rolf Turner. Practical maximum pscudolikelihoodd for spatial point patterns. Australian & New Zealand Journal of Statistics, 42(3):283-322, 2000. doi: 10.1111/1467-842) 00128.
Charles Beattie, Joel 7. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefraneq. Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/1612.03801 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. 2016.
Zhiyuan Chen and Bing Liu. Lifelong Machine Learning, Second Edition. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2018.
Maxime Snevalier-Boisvere Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: First steps towards grounded language learning with a human in the loop. CoRR, abs/18 8 100.0222, 2018. URL http: /ttt:://bpp aap.uni-tee trrer
be//bb/jornas wot.aaalcccooooooooooeoee
810 html #abs 181 C 08272
E. W. Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1: 269-271, 1959. ISSN 0029-599X. 1..00//B001866900
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Picter Abbeel. Benchmarking Deep Rein- forcement Learning for Continuous Control. In International Conference Machine Learning, pp. 1329-1338, 2016.
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. MineRL: A Large-Scale Dataset of Minecraft Demonstrations. In International Joint Conference on Artificial Intelligence, pp. 2442-2448, 2019. doi: 10.24963/ 20ca932019/339
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9()): 1735-1788 1780, 1997.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The Malmo Platform for Artificial Intelligence Experimentation. In International Joint Conferences on Artificial Intelligence, pp. 4246-4247. 2016.
A. N. Kolmogorov. On tables random numbers. Sankhyā: The Indian Journal of Statistics, Series A (1961-2002) 25(4):369-77, 1963. ISSN 0581572X.
John E Laird and Robert E Wray III. Cognitive Architecture Requirements for Achieving AGI. In Conference on Artificial General Intelligence (AGI-2010). Atlantis Press, 2010.
Tom M Mitchell, William W Cohen, Estevam R Hruschka Jr, Partha Pratim Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhanava Dalvi, Matt Gardner, Bryan Kisiel, Jayant Krish- namurthy, Ni Lao, Kathryn Mazaitis, Thahir P Mohamed, Ndapakula Nakashole, Emmanouil A Platanios, Alan Ritter, Mehdi Samadi, Burr Settles, Richard C Wang, Derry Wijaya, Abhinav Gupta, Xinlei Chen, Abulhair Saparov, Malcolm Greaves, and Joel Welling. Never-Ending Learning. Communications of the ACM, 6135:10331115, April 2018. ISSN 0001-0782. doi: 10.1 145119533
11

---

Published as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. arXiv preprint arXiv:1312.5602, 2013.
Jean-Baptiste Mouret and Stéphane Doncieux. Encouraging behavioral diversity in evolutionary robotics: An empirical study. Evolutionary Computation, 20(1):91-133, 2012.
Volker Nannen. A short introduction to kolmogorov complexxty. CoRR, abs/1005. 24000 2010.
OpenAI. OpenAl Five. hutpss///pperaa..oomffiie 2019.
Vicki Plau, Alex Nichol, Christopher Hesse, Larissa Schiavo, John Schulman, and Oleg Klimov. OpenAl Gym Retro. htttps://opeaai: httt:://opeeaa..co//ooooo/////// wwt.blcogooommmm/ee/e.ooooo..ee
2018. 2018.
Emmanouil Antonios Platanios.
Swift RL.
Swi tt rl. 2019.
Christian Robert and George Casella. Introducing Monte Carlo Methods with R. Springer New York, New York, NY, 2010. ISBN 978- 178-1-4419-1582-5
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv: (707,0677770634 2017.
David Silver, Julian Schrittieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the Game of Go without Human Knowledge. Nature, 550(7676)3354, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEE, 2012.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. AlphaStar: Mastering the real-time strategy game StarCraft II. DeepMind Blog, 2019.
A APPENDIX
A.1 COMPLEXITY BOUND
We show that K(E) must be at least K(r') up constant. We write algorithm श that enumerates all possible sequences of environment states, observations, actions, and rewards from time t up to time T: (st, Wt, de, Tz),.... (ST, (ST,WT, WT, aT, (T)) Then â computes the action that maximizes Since the images im(#t) S A and im(r") are discrete, and
the expected reward (1R((rr)TT lom arg max., [[ (( [] arg max., ( (Trx) } there is sufficiently large finite T' such that the action computed by â is the same that computed by म². Note that ğ relies on subroutine that simulates the environment in order to first enumerate the environment states, and the subroutine perform the optimization is independent of E, and so K(#t) K(E) for
constant c. Since K(T*) ) < (f)), we have that K(E)= 2 K(x²) -c
A.2 INTENSITY AND INTERACTION FUNCTIONS
The JBW currently only supports small number of implemented intensity and interaction functions to control the distribution of items in the world. However, is very straightforward to implement new customized intensity and interaction functions. Let (x..) 2² be position and E T be an item type. Intensity functions are indexed by item type, and each item type is assigned its intensity function: f((2, v).t) A fe(x,)) The JBW currently supports three intensity functions:
1. Zero: fi(2, v)
2. Constant [w]: f( v)
12

---

Published as a conference paper at ICLR 2020
3. RadialRash c,kk
=cc.))--kk-oo M +v//s
where M : [0, 1] is the linear interpolation of M (L and M( 11)((22)) 1). and
+ is the last mixing step of the 32-hit MurmurHash function (Apleeh 2008 This provides pseudorandomness to the intensity function.
For interaction functions, let (æ1, /1) ) be the input position of the first item, l be the type of the first item, (x2, (2) the position of the second item, and t2 be the type of the second item. Interaction functions are indexed by pairs of item types, so each pair of item types can be given its own interaction function: ((((11 y1). f1), ((z2, V2), t2)) _ It ((21, v1). (I22, v2)). The JBW currently supports
four interaction functions:
1. Zero: //1. ((21, (iee,,)),, v1). (2))
PiecewiseBox (U, V, u, w.
u, if < U, v, if S d V,
It (1),
0, otherwise,
where d = (x1 x22)² (yn y2)²
3. Cross (U, V, U, a, B]:
u, -f
a, if
I1. (1112(( (32,
v, =
8, 0,
0, otherwise,
where d min((a - 121, val) and D max(I=i 22/, Iyn yal)-
4. CrossHash (s, c, 8,u, a, B1 is the same as Cross (U, V. u, a, 81 except for the fact that U and are given by:
U == == c k k (21/s))
V U + 8,
where M-) is defined as above in the Radia a Hash intensity function.
Even though this is small set of intensity and interaction functions it can allow for creating worlds with many interesting features (e.g., we use the Cross interaction function to create contiguous wall segments that are axis-aligned, and the Piecewissee interaction function create irregularly shaped clusters of trees forming forests). Note that the unspecified intensity and interaction functions in Table 3 are set to Zero by default.
A.3 FIELD OF VIEW AND VISUAL OCCLUSION
To compute the color of cell with respect the agent's field of view, let the cell position be (1r, y) and consider circle of radius centered at (I, y). Project this circle onto the circle of radius 1 centered at the agent position. Let denote this projection (an arc). Let Ojw the are on the agent's 's circle centered on point in the current agent direction. The length of frr is specified by the held-of-viev parameter in the configuration. The color of cell Cr.y is then computed as:
1pov no
Cr.y
Cr.yy
18]
where &g.y is the original color of the cell. In order to compute how much a cell at position (a, is occluded, we consider circle of radius 2 centered (æ, y). and project this circle onto the circle of radius 1 centered at the agent position. Let @ denote this projection (an are). Each item in the agent's 's visual field similarly projected onto the agent's circle, each producing an 0,. The color of the cell Cr.y is then computed as:
Ca.y Ex.y max 1
<1-Σ
Σ
10
where õz.y is the original color the cell, and o is the occlusion parameter of the 7th item, as specified by the item's 's type. If a cell is affected by both the field of view and visual occlusion, the above effects are composed (both multiplicative factors applied to the original color).
13

---

Published as a conference paper at ICLR 2020
A.4 RELATIVE UTILITY OF SCENT AND VISION
The relative difficulty of utillzzig scent vision to effectively navigate in the JBW environments depends on the environment configuration as well as the method used by the learning agent. For example, if the learning agent does not possess memory, it cannot remember the scent of any pre- viously visited tile, and thus it will not be able determine the direction from which the scent is dif- fusing. Therefore. such methods will not benefit from the information provided by the scent modal- ity. In addition, scent is not necessarily blocked by items that block movement (e-g., walls). Thus, in environments with such items, scent is more difficult utilize, since the simple strategy of following paths of monotonically increasing scent could lead to a wall. The agent could be fooled to believe item is nearby when, in fact, is behind a wall. This is illustrated in Figure
Scent pesses through walls and the agen: needs to leam that even though can smell an item being close, that docs not mean the shortest path feasible and needs to combine the scent information with the visua information.
Figure 7: Example showing one of the challenges the scent modality poses for agents.
Table 4: Simulator configuration.
Scent Dimensionality
Color Dimensionality
7
Map
Patch Size
Sampling Iterations
and is possibly agent underperforms the "vision" agent in Figure
of the main reasons the "vision+scent"
Scent Decay
Scent Diffusion
In order to showcase how scent can provide useful infor- mation, we also designed simpler environment configu- ration that does not contain any walls. The configuration for this world is shown Tables [nd Given the task of collecting Jel 1 1 Beans, we expect the scent modality to be very useful memory. The results of using an LSTM-
[0.00, 0.00, 0.00
Scent
[0.00, 0.00, 0.00]
Agent
MoveForwand
Action Space
Turnte ft,
and TurnR
Visual Range
Field-of-View
6U"
long the agent has some sort of
based agent are shown in Figure 8 We ob- serve that the "scent" agent outperforms both the "vision" and the "vision+scent" agents. This is the opposite pattern of what we observe in Figure 6 This can partly be explained by the fact that, this case, the visual field of the agent is more restricted, thereby limiting the agent's reliance vi- sion its learning. Ideally agents should be able to each perception modality opti- mally and not be "confused" by the fact that one of them may be harder to utilize than the other. Thus, these experiments show- case how the JBW can be used to evaluate multi-modal machine learning algorithms.
Table Item types for the simple environment where the scent modality dominates the vision modality. See Sec- tion A.2 for details on the functional forms of the intensity and interaction functions.
JellyBean: Jelly beans appear close to bananas.
Scent
[0.0, 0.0, 1.0
Color
[0.0, 0.0
Occlusion
0.0
Blocks Agents
False
Intensity
Constent [-5.3]
Jellysean: PateeaannPiccoooo Piccowicceoxx 200,,00,,40
Interactions
Bananz : Pieceewi GeRox 200,200,22 -100]
On1 : Pieeew seRox (230, Co, -100)
Banana: Bananas appcar close to jelly bcans.
Scent
[0.0, 1.0, 0.0)
Color
[0.0, 1.0, 0.0
Occlusion
0.0
0.02
Vision+Scent
Blocks Agents
False
Vision
Intensity
Constant 5.3]
Scent
JellyDean: PiecewiseBox(11,
100]
Interactions
Banane [PiecewiseSox(10,222 06
0.01
Onion
PPiecewisesxx(200 -6,
On on: Onions appear scattered, away from jellybeans and bananas.
Scent
0.0,,000,00
Color
1.0,0.0.0.0
0.00
Occlusion
0.0
200k
400k
600k
800k
1M
Blocks Agents
Falae
Time Step
Intensity
Constent: (-5)
Figure Results of experiments on the configu- ration shown Tables and
Interactions
Jell yRean: Piecoew seRox (200, 0, -100
:Piecewi seRox [200, -6,
Note that since the configurations differ, the reward between the figures directly comparable.
14

---

Published as a conference paper at ICLR 2020
Figure 9: Visualization of an example environment with spatial non statiionntn Each tile is colored according to its scent. Jel1yl Bs are shown in blue, Bananas in green, and On ons in red. Wal 1s depicted as grey squares. This environment generated using the configuration shown in Tables and
A.5 EXAMPLE OF SPATIAL NON-STATIONARITY
To demonstrate the ability generate spatially non-stationary worlds in JBW, provide configu- ration that makes use of non- stationary intensity and interaction functions. The configuration shown in Tables 6 and 7 and visu- alization is provided in Figure JellyBeans and Onions appear to- gether in clusters and, since this con- figuration uses the non-stationary in- tensity function Radia laas,, these clusters are arranged concentric cir- cles around the origin that are irreg- ularly spaced. Radía al Hash uses hash function to induce pseudoran- dom relationship between the distance to the origin and the likelihood of find- ing such clusters. Walls in this en- vironment also have a non-stationary distribution. In some regions, they smaller and more frequent, whereas in other regions they are longer and appear more sporadically.
Table 6: Item types for the non-stationary environment. See Sec- tion A.2 for details on the functional forms of the intensity and interaction functions.
yHean: Jelly appear
Scent
0.0,, 1.0]
Color
[0.0, 0.0, 1.0)
Occlusion
0.0
Blocks Agents
False
Intensity
Radi a a (530, -3.03. 144
Jallyhean: riecewiseRor (10, 200,0, 0, -5]
Interactions
Banana
: FFeecewsseoo (10, 200, 2, -130]
Onion
FFeecewiseoox (200, 100,-100]
Banana: Bananas appear close to jelly beans.
Scent
0.0, 0.0
Color
0.0
Occlusion
0.0
Blocks Agents
False
Intensity
Radiallash 60, -,
bellysean: (10, 100, -1000]
Intoractions
Banana
:Piecevisesox 200, 0, -6]
Onion
: pPiccowicasox (200, 0, -6,
Onion: Onions appear scattered, away from jollybeans and bananas.
Table Simulator configuration.
Scent
0.0, 0.0)
Color
0.0, 0.0
Scent Dimensionality
3
Occlusion
0.0
Color Dimensionality
Blocks Agents
False
Map
Patch Size
Intensity
Constant
Sampling Iterations
JellyBean: Piecewiseedd (200, -000, -100]
Decay (X)
Interactions
Banana
:Piecewise seBox (200, 6]
Diffusion (o)
Wa Wal.:: Walls tend to be contiguous and axis-aligned.
Color
[0.00, 0.00, 0.00]
Scent
0.00, 0.0,0.0.0 0.0]
Scent
[0.00, 0.00, 0.00]
Color
0.5. 0.5]
Agent
Action Space
Turnbeft, ft,
MovaFormatt
Occlusion
1.0 in experiments with occlusion. 0.0 otherwise
and TurnR'
Blocks Agents
True
Visual Range
5
Intensity
Constant[0]
Field-of-
experiment-speeda
Interactions
wall
:CrossHash [60, 4, 20, -200, -2,
15

---

Published as a conference paper at ICLR 2020
A.6 PERFORMANCE
The JBW is implemented in optimized C++, with performance being highly prioritized in both its design and its implementation. This would allow less time and hardware resources to be spent simulating the world and more time and resources be allocated for the machine learning algorithms. Additionally, the JBW is perceptually quite simple, being a two-dimensional grid world with limited vision and scent inputs. This allows the machine learning algorithm to focus less on perceptual information processing and more on abstract information processing, which we think is a hallmark of never-ending learning. As rough indication of performance, 5820K (released in 2014) at 3,5GHz, the JBW can generate 8.56 patches per second, each of size 64 x 64 (i.c., 35, 062 grid cells), using the configuration described in Section
single core of an Intel Core i7
A.7 GREEDY ALGORITHM
As benchmark and for the sake of comparison, we also implemented simple greedy agent that searches its visual field for cells of a particular color, and then computes the shortest path to those cells. This algorithm makes the assumption that reward maximized simply by collecting items of single color, ad infinitum. It also assumes that this color is known a priori. Additionally, it assumes the color of obstacles (items that block agent movement that should be avoided as part of the reward function) is known a priori, and is distinet from the color of items that provide reward. The shortest path computes is such that it never goes through any such obstacles.
Algorithm 1: Pseudocode for the greedy aision-based algorithm.
Input: Color of rewarding items o, and color of obstacles Cu. Initialize best_ best_path = mull
2 Function Greedyvisionnoll by (visual field we)
shortest_path = ShortestPach(wu, Gr, Go)
best best_ppath path = null or shortest patá < lbest patal
assign best_patstt__b_eepeeeeeeaeaaaeee = shortest path
besb_pathh null
if the cell immediately in front of the agent has color "Che for any 7 >> 0
8
return MoveForward
else
10
return TurnLeft or TurnRight uniformly at random
11
else
12
next_action dequeue the next action from best_patte path if best jast_ppahh has no further actions assign best patt = null
13
14
15
return next_action
The algorithm is shown in pseudocode in Algorithm [ The function ShortestPath is simply Dijkstra's 's algorithm on directed graph where each vertex corresponds to unique agent position and direction within its visual field wi, and each edge corresponds to possible action that transitions between agent states (Dijkstra) 1959). Let er be the color of items that provide reward, and cn be the color of items that block agent movement. The algorithm returns a shortest path from the agent's s current position and direction to a cell that has color 7c,, for any 7 >> 0, while avoiding cells that have color yyc. for any y > (we match any color the direction of the vectors c, and ow order to detect partially occluded items). If no such path exists, Shortest Path returns nul 1. In the case where the agent's 's field of view is limited, ShortestPath only returns paths that pass through cells within the agent's field of view. Also, in the experiment where the agent must additionally avoid Onion items, ShortestPath avoids them in the same way that it avoids obstacles: it avoids cells that have color yc, for any 7 > where c, is the color of the Onlon item type.
In environments with visual occlusion, if items with high occlusion are arranged in line (such as wall), and the agent is adjacent to the wall and facing it, the portions of the wall further from the agent will be occluded by the portion of the wall closer to the agent. Since we currently do not distinguish between empty cells and completely occluded cells. ShortestPath: will return paths that may pass through the wall. If no other paths are returned, the agent will continuously try to move through the wall and make progress.
16

---

Published as a conference paper at ICLR 2020
Similarly, in environments containing items that provide negative reward, the above greedy algorithm does not allow the agent to move over tiles with such items. Thus, given sufficient time, the agent will find itself moving between two clusters of avoided items, where one cluster is shaped like "U" and the other like a "n". And so when the agent moves downward into "U" of one cluster, it will make a 180-degree turn, leaving the cluster, and moving upward toward the other cluster. The other cluster is shaped symmetrically, and the agent will repeat the behavior deterministically: enter the "n", make a 180-degree turn, leave the cluster, and move down toward the first cluster again. This will repeat indefinitely.
17


Reviews:
[{"review_id": "Hkeqnqz0FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\n\nThe authors introduce a simulator (JBW) with the goal of supporting continual learning. They demonstrate that RL agents struggle with a lot of the tasks in JBW. The majority of the paper describes the technical details of JBW, and show that RL agents can struggle to solve continually changing tasks in JBW.\n\n2. Decision (accept or reject) with one or two key reasons for this choice.\n\nI'm borderline. It is valuable to have environments that support continual learning, although the experimental investigation into different forms of non-stationarity would be more informative. \n\nRe new implementations: the continual learning setting is certainly important and interesting, but existing environments (see BabyAI, https://arxiv.org/abs/1810.08272 (focus on NLP)) do feature multiple tasks and it is not hard to augment these to run `forever'.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJeyARJKtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper introduces a new environment for testing lifelong or never-ending learning. The goal of the environment is to act as a new benchmark testbed for challenging existing agents and models across areas of research, encouraging and pushing new research towards solving challenges in curriculum learning, exploration, representation learning, and continual learning. The contributions in this paper extend upon previous work by building an easily controllable environment generator with key necessary features for lifelong learning including: non-stationarity, multiple task specification, and multiple sets of observable features.\n\nReview\n\nThe paper highlights many key characteristics of an environment that are challenging to current RL models. This focus on building a benchmark upon which further research can measure performance is important. I find the proposed environment to be incredibly intriguing and would find it valuable to the field of lifelong learning (or continual learning or never-ending learning, etc.). I think the size and scope of the environment generator is impressive, showing a considerable amount of engineering effort has gone into its design.\n\nThe largest overarching issue that I would like to point out is the limited study of modelling choices. I am not an expert on applied Reinforcement Learning, so I can make very few claims about the validity of the chosen network architecture or use of the PPO policy-gradient algorithm for this environment. However, it is critical, in my view, that a paper introducing a new environment studies these effects itself; demonstrating how various degrees of learning capacity or wider ranges of learning algorithms behave in the given environment. If a slightly larger network architecture trivially solves each task in this environment, can this still be considered a benchmark task? A key result in the paper that I would like to see further investigated (even with only a different network architecture) would be Figure 6, the comparison between scent, vision, and vision+scent. It is unclear to me why the scent features would be so challenging to learn from and specifically why they would harm the representation so permanently. A deeper study using only the scent features would be valuable to me. In its current state, it appears that these feature provide no additional information and are thus not necessary to include in the environment; breaking one of the primary motivating features of JBW: the multi-modality. I recognize that the paper comments on the orthogonality of the scent playing a role, and notes that further results are included on a not-yet-available website (presumably to maintain anonymity). However, I would like to see these results included in the appendix of the paper so I could better assess the utility of the scent features. Perhaps an additional result showing the average reward versus the cosine distance (or other measure of orthogonality) between \"jellybeans\" and \"onions\" would additionally motivate the utility of the scent features.\n\nThe paper empirically investigates the use of curriculum learning to accelerate learning for a particular task. The paper then claims that curriculum learning improves learning speed, but ultimately does improve final performance. This demonstration is intended to showcase the use of the proposed environment (JBW) for curriculum learning. However, there are few key issues with this empirical study. First, the paper shows the reward rate of 3 different curricula but does mention the metric used to compare the agents during the time the curricula is active. It is implied that the metric is the reward rate of each individual agent; however, each agent has a unique reward function making comparisons between agents impossible. Curriculum #2 can only receive positive rewards while Curriculum #1 can only receive negative rewards. Naturally this means that Curriculum #2 must have strictly greater or equal reward rate over Curriculum #1. Even in the case that the final objective specifies the metric used, these are still highly non-comparable entities. A suggestion to improve this result would be to run each curricula for 100k as a \"pretraining\" phase, then to restart the agents to the same state in the environment and measure their performance from there.\n\nThe case study measuring the effects of non-stationarity of the rewards does not provide sufficient evidence that the proposed environment contributes a novel ability to investigate non-stationarity. First, the given study of non-stationarity focuses solely on an alternating reward function, clearly demonstrating the problem of catastrophic forgetting. While this is a motivating demonstration, it is not novel and the issue of catastrophic forgetting in our models has been known since at least the 90s (e.g. French 1999 and related). Carefully and scientifically investigating such an issue is best done in a far less complex environment where more precise results can be drawn. Further, the ability to oscillate a reward function in this way is not unique to this environment and can be trivially done in most environments. Secondly, it is unclear if JBW allows for non-stationarity in the transition probabilities in the MDP. This is a critical component to non-stationarity and would be a necessary feature for me to claim non-stationarity is widely supported in the environment.\n\nThe paper starts with a motivating conversation about environment complexity, with interesting insights into measuring the complexity of an environment based on the complexity of the policy used to solve that environment. However this conversation is ignored until the conclusion of the paper, where the paper claims to have built an environment of greater complexity than already existing environments. Without any supporting evidence in the body of the paper, it is impossible to verify the validity of this statement, and it is still an open question to me whether this claim is even falsifiable in the first place. As a concrete counter-claim, I would claim that the Minecraft environment (Malmo) has similar or higher complexity to the proposed environment in most aspects. Minecraft has a far greater diversity of objects, a third dimension of movement, adversarial components, hunger and health, etc. each of which adding a large level of complexity not achievable in the proposed environment. This is not to say that I expect the proposed environment to contain these features, but rather to point out that claims of greater complexity may be ill-founded.\n\nAdditional Comments (not affecting score)\n\nI do slightly question if ICLR is the appropriate venue for such work. While I recognize that the scope of this conference has shifted considerably over the past few years, this paper (as written) does not further understanding or study of learning representations. I believe a more careful demonstration of the representation induced by characteristics of the environment is within easy reach of the paper, but is not currently presented.\n\n-----------\n\nAfter the author response, reading other reviews/responses, and looking at the edited draft:\n\nI am convinced of the utility of the domain, the scope of the engineering effort put into building, and the ease with which it can be configured by the user to test many applicable settings (partial observability, stochasticity in transitions and rewards, etc.). I remain slightly skeptical of the amount of benefit the proposed provides over the Malmo environment for any of the settings discussed in the case-studies.\n\nI specifically feel my concerns about the stochasticity in the transitions and environment complexity have been well addressed. My concerns about the curriculum learning demonstration are partially addressed to a point where I am satisfied. My concerns about the modeling choice are also partially satisfied, with one lingering concern. I am unclear if the environment is trivially solvable by using more computation resources (e.g. bigger networks). However, after reconsideration I decided this concern bares less weight than I previously considered.\n\nAll this considered, I am changing my rating from 3 -> 6.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1gpQlCnKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Jelly Bean World: A Testbed for Never-Ending Learning\nThis work introduces a domain for evaluating and experimenting with algorithms for never-ending learning. These are variants of grid worlds which have multi-task, multi-modal, dynamic settings and can lead to interesting learning challenges. \n\nI’m referring to never-ending learning as NEL throughout. \n\nIntroduction\nComment: It’s good to tell the reader as early as possible why never ending learning is different than multi-task or continual or lifelong learning? Those are more commonly used terms in the community so it should be situated properly. All of this stuff about NEL seems very similar to continual learning/lifelong learning and we should really reference it and describe the difference?\n\nI didn’t find “In order to more formally describe general intelligence, we posit that there is an underlying measure of complexity of the environment E such that: (i) highly specialized and non-general learning algorithms can perform well in environments with low complexity, but (ii) environments with high complexity require successful learning agents to possess more general learning capabilities” to provide much clarity. Can we either remove it or stick to the later formalism?\n\nIn never-ending learning, we explicitly disallow the learning agentπfrom learning across multiple episodes or in multiple environments, which is closer to humanlearning. -> Is this just the same as saying you’re reset free and in a single environment? This sentence is a bit confusing. \n\nOne potential criticism of using simplified simulated worlds like JBW is why should we believe that insights that we get from JBW would carry over to the real world natural environments that NEL ideally cares about. Why is this actually representative of the real world? Because that is really what we care about with NEL. There is merit to simulation in this setting but only if we believe that either insights, algorithms or policies also hold in the real world and we can representatively model the worlds complexity. Can we verify this somehow? \n\nDesign\nDoes the user have control over all the agents? Or how are they programmed\n\nI would move the details of procedural generation to the appendix, they’re a bit distracting from the point. \n\nI would also tell the readers why things like scent, intensity, interaction etc are important early on, otherwise it’s confusing what their purpose is. \n\nIn general I quite like the setup, it seems like it has the sufficient amount of complexity in modality, interaction and multi-agent systems to be useful. I wonder if it’s also useful to introduce autonomous self-powered agents which move on their own in the environment and introduce dynamic non stationarities. \n\nA little more description of the multi-agent, multi-task, curriculum stuff would be useful in the design section. \n\nThe reward functions are all sparse? Or do they need guidance to get to objects as well?\n\nI’m still a bit confused about the interactions functions. Could those be described a bit further?\n\nPerhaps a practical question is how does this relate to the work described in the BabyAI/Minigrid stuff from MILA and other simulated gridworld style environments with multiple agents and such. \n\nExperiments: \nIn the case studies, are things multi-agent?\nI wonder if in the reset-free experiment, if we just use dynamic agents in a multi-agent setup, would this just work?\n\nIs it a little odd that the without occlusion performance comes back down to around the same as with occlusions?\n\nIs the scent just perhaps misconfigured/too hard to learn from coz it never seems like it’s doing well with scent?\n\nOverall, I like the paper and the introduced environment. I think it’s important to study scenarios such as the ones described here and this provides a tractable way to start. I am however concerned that the environments are too simplistics and perhaps too far from the real world for the insights to carry over to more realistic scenarios. Some suggestions would be to try and make the environment a bit more realistic and less toy so that insights might also more easily transfer to real world scenarios. But I think with some of the clarifications above and a bit more description, this would be a valuable contribution on topics which are not thought about enough in RL. I also think that actual visuals and videos on an actually accessible website would make it easier for the reviewers/readers to understand the importance of this. I'm currently listing it as a weak accept but I would like the authors to better clarify some of the points mentioned above, discuss how realistic the setting is and also provide us with videos of the environment to better gauge things. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a flexible environment for studying never ending learning. During the discussion period, all reviewers found the paper to be borderline.

Pros:
- we don't have good lifelong or never-ending RL environments, and this paper seems to provide one
- includes a number of interesting features such as multiple input modalities, non-episodic interactions, flexible task definitions

Cons:
- procedurally generated, toy environment
- unclear if the environment reflects the characteristics of real world NEL problems

In the balance, I think the environments add value to the RL community, and being presented at ICLR would increase its visibility.
Decision: Accept (Poster)