Jelly Bean World: A Testbed for Never-Ending Learning

1 INTRODUCTION . Machine learning has witnessed growing success across a multitude of applications over the past years . However , despite these successes , current machine learning systems are each highly specialized to solve one or a small handful of problems . They have much narrower learning capabilities as compared to humans , often learning just a single function or model based on statistical analysis of a single dataset . One reason for this is that current machine learning paradigms are restricted and specialized to a particular problem and/or dataset . An alternative learning paradigm that more closely resembles the generality , diversity , competence , and cumulative nature of human learning is never-ending learning ( Mitchell et al. , 2018 ) . The thesis of never-ending learning is that we will never truly understand machine learning until we can build computer programs that , like people : ( i ) learn many different types of knowledge or functions , ( ii ) from years of diverse , mostly selfsupervised experience , ( iii ) in a staged curricular fashion , where previously learned knowledge enables learning further types of knowledge , and ( iv ) where self-reflection and the ability to formulate new representations and new learning tasks enable the learner to avoid stagnation and performance plateaus . Building computer programs with these properties necessitates well-defined and robust ways to evaluate whether a system is indeed capable of never-ending learning . However , there are currently no ways to achieve that . There only exists one large-scale case study on never-ending learning with the Never-Ending Language Learning ( NELL ) system by Mitchell et al . ( 2018 ) , which uses the internet as the environment with which the system interacts . While the internet does have significant complexity , it is unwieldy to use as a testbed . It is very difficult to focus on a particular aspect of the system or the environment , or to tweak the algorithm and restart experiments to observe the effects of changes . Furthermore , oftentimes tasks require manual annotation which can be very expensive . Thus , a good testbed for never-ending learning ( and machine learning more generally ) needs to provide the experimenter with a high degree of control . To this end , we propose a novel evaluation framework—the Jelly Bean World ( JBW ) —that can enable and facilitate research towards the goal never-ending learning . We have designed the JBW to be highly versatile , enabling evaluation of systems that have any number of the aforementioned abilities . We consider never-ending learning in the context of reinforcement learning . Let st ∈ S denote the state of the environment at time t , at ∈ A denote the action performed by the learning agent at ∗Equal contribution ( listed in alphabetical order ) . time t , ωt ∈ Ω denote the observation of the world that the learning agent receives at time t , and rt ∈ R denote the reward provided to the learning agent at time t. The distribution of the next state of the world st ∼ T ( st−1 , at−1 ) has the Markov property ( i.e. , it depends only on the previous state and action ) and the initial state of the world is given by a distribution s0 ∼ W . The observation ωt ∼ O ( st ) depends only on the current state of the world ( perhaps deterministically ) . The reward rt is given by a function R ( st−1 , at−1 , st ) of the current state , the previous state , and the previous action taken . The environment is a tuple containing all these elements E , ( W , T , O , R ) . Then , the goal of reinforcement learning is to find a learning algorithm π that , given the history of previous observations , actions , and rewards , outputs the next action so that the obtained reward is maximized . We deliberately blur the distinction between the policy and the algorithm that learns the policy , which is why we call π a “ learning algorithm. ” This formalism does not distinguish between learning algorithms that are highly specialized to a single task and learning algorithms that are capable of learning a wide variety of tasks and adapting to richer and more complex environments , which are hallmarks of general intelligence . In order to more formally describe general intelligence , we posit that there is an underlying measure of complexity of the environment E such that : ( i ) highly specialized and non-general learning algorithms can perform well in environments with low complexity , but ( ii ) environments with high complexity require successful learning agents to possess more general learning capabilities . It is in these more complex environments where we can characterize never-ending learning . We can formalize this notion of complexity by letting π∗ be the ( computable ) learning algorithm that maximizes expected reward in an environment E . Then we define the complexity of E to be the length of the shortest program ( Turing machine ) that implements π∗ : complexity ( E ) = min { |T | : T is a Turing machine that implements π∗ } complexity ( E ) 0 π∗ exhibits specialized intelligence π∗ exhibits more general intelligence We can equivalently define complexity ( E ) = K ( π∗ ) , where K ( · ) is the Kolmogorov complexity and is related to the minimum description length and minimum message length ( Nannen , 2010 ; Kolmogorov , 1963 ) . The Kolmogorov complexity of the environment K ( E ) is bounded below by K ( π∗ ) minus a constant . This bound is shown in Section A.1 of the appendix . Never-ending learning is in many respects similar to lifelong learning , also called continual learning ( Chen & Liu , 2018 ) . Like never-ending learning , lifelong learning is distinguished from multi-task learning by the never-ending nature of the learning problem . However , in never-ending learning , and unlike multi-task learning and lifelong learning ( to the best of our understanding ) , a well-defined set of tasks is not assumed a priori . Rather , never-ending learning is more similar to real-world settings in this respect , where the notion of a task or subtask naturally emerges from the complexity of the environment , and the distinction between tasks is not always so sharp . Regardless , due to their similarities , a good testbed for never-ending learning will also be a good testbed for lifelong learning . In contrast to most popular reinforcement learning settings , never-ending learning focuses on environments with high complexity . In never-ending learning , agents can only exist in a single reset-free environment ( i.e. , we explicitly disallow the agent π from learning across multiple episodes or in multiple environments , which is closer to human learning ) . We require π to only have access to a single episode . During its lifetime , π can only use the information provided by its past observations { ωt } and actions { at } in a single world . Thus , never-ending learning explicitly removes the distinction between training and testing that is common to many other classical machine learning paradigms . Additionally , note that in the general reinforcement learning formalism , st can contain information about t , and the reward functionR can be time-varying , thus rendering the environment non-stationary . In never-ending learning , we are interested in the full generality of non-stationary environments , as the assumption of stationarity is not realistic in even simple adversarial and multi-agent settings . We thus argue that a testbed for never-ending learning should have the following properties : 1 . Non-Episodic : It should disallow agents from resetting the environment and “ retrying ” . The testbed should also force them to only learn within a single environment ( i.e. , not transfer information across environments ) . This is in contrast with most popular reinforcement learning environments and , as we show in Section 4 , poses significant challenges to existing algorithms . 2 . Non-Stationary : The testbed should allow for easy experimentation with non-stationary environments , where the reward R can depend on time . 3 . Multi-Task : It should support settings in which reward is maximized not by learning how to perform a single task repetitively , but by learning how to perform a general variety of tasks , and learning how to switch between them and/or combine them to better perform other tasks ( e.g. , by composing them ) . We posit that , at a sufficiently high level of task complexity , optimal learning agents will be required—either explicitly or implicitly—to perform abstract reasoning over concepts and make informed decisions about actions in the environment . 4 . Multi-Modal : It should support multiple data modalities that agents receive as input . These modalities should not contain the same information , but rather be complementary to each other so that the agents are forced to learn from diverse types of experiences . Multi-modality provides yet another way to increase the complexity of the world . 5 . Controllable : It should be easy for experimenters to modify the complexity and richness of the learning problems in the testbed , make changes to it , and restart it ( e.g. , as opposed to NELL ) . 6 . Efficient : It should run on readily available hardware and allow for quick experimentation . Ideally , we should not have to wait for days , weeks , or months ( e.g. , NELL ) to obtain results . 7 . Reproducible : It should make it easy to reproduce results and experiments , which would facilitate scientific research . This also requires that it allows for seamlessly saving and loading state and for reproducing results outside the environment in which they were first obtained . The testbed should also not require access to specialized hardware , which can be expensive . Many of the above properties are means to increase the complexity of the world . These properties are in fact very closely related to the characteristics of “ AGI Environments , Tasks , and Agents ” outlined by Laird & Wray III ( 2010 ) and later refined by Adams et al . ( 2012 ) . The proposed JBW has all of these properties . It aims to provide an easy way to create sufficiently complex environments allowing researchers to experiment with never-ending learning , while remaining simple enough to control the problem and enable rapid prototyping . The JBW is a two-dimensional grid world with simple physics , but is extensible enough to admit a wide variety of complex and inter-related tasks . We present a comparison with related work in Table 1 , showcasing the ways in which the JBW is a novel and highly versatile evaluation framework . The JBW is written in C++ and we provide C , Python , and Swift APIs , and is available at https : //github.com/eaplatanios/jelly-bean-world . It is also worth noting that the JBW has already been used for the instruction of the “ Deep Reinforcement Learning ” and “ Never-Ending Learning ” graduate courses at Carnegie Mellon University . 2 DESIGN . The Jelly Bean World ( JBW ) consists of the following main modules ( illustrated in Figure 1 ) : ( i ) the simulator , which comprises the central component ( the other modules only interact with the simulator ) , ( ii ) the environment , which provides a simple interface for performing reinforcement learning experiments in the never-ending learning setting as well as utilities for evaluating neverending learning systems , and ( iii ) the visualizer , which provides the ability to visualize and debug the behavior of learning agents . Note that the visualizer is completely asynchronous and can be attached , reattached , and detached to and from existing simulator instances , without affecting the simulations .

Summary: This paper introduces a new environment for testing lifelong or never-ending learning. The goal of the environment is to act as a new benchmark testbed for challenging existing agents and models across areas of research, encouraging and pushing new research towards solving challenges in curriculum learning, exploration, representation learning, and continual learning. The contributions in this paper extend upon previous work by building an easily controllable environment generator with key necessary features for lifelong learning including: non-stationarity, multiple task specification, and multiple sets of observable features.