Rewardless Open-Ended Learning (ROEL)

1 INTRODUCTION . Development of machine learning over the past several decades has principally relied upon researchers solving a series of challenges proposed by experts within the community ( e.g . MNIST Lecun et al . ( 1998 ) , ATARI Bellemare et al . ( 2013 ) and robotics Yu et al . ( 2021 ) ) . Learning algorithms typically solve these challenges through a huge number of observations , and whilst they have exceeded human-level performance on a variety of these tasks ( Silver et al. , 2017 ) , ( Berner et al. , 2019 ) , they often require significantly more examples before coming close to matching human performance ( Toromanoff et al. , 2019 ) . Humans in contrast leverage many diverse experiences to solve complex tasks ( Wang et al. , 2018 ) often without a well defined objective or reward . Should we attempt to teach reinforcement learning ( RL ) agents in a similar fashion ? We use this question to motivate the design a procedure that aims to learn a diverse set of increasingly complex novel behaviors without supervision ( labels ) or incentives ( external rewards ) . Open-ended algorithms aim to automatically generate challenges to be solved ( Forestier et al. , 2020 ) , ( Schmidhuber , 2012 ) , ( Standish , 2002 ) , ( Secretan et al. , 2011 ) that can serve as stepping stones to increasingly complex problems . These algorithms are useful in designing a general unsupervised learning procedure because they do not rely upon manually selected challenges ; instead they search for challenges that form stepping stones to solve larger ones , collecting a diverse set of experiences along the way . Designing algorithms to achieve these open-endedness can be challenging . Often requiring a careful balance between diversity , optimization ( Mouret & Clune , 2015 ) , ( Pugh et al. , 2016 ) , and a domain complex enough for sufficient experiences/challenges to be generated in order to learn a suitable curricula . The Paired Open-Ended Trailblazer ( POET ) algorithm ( Wang et al. , 2019 ) , ( Wang et al. , 2020 ) is a recent example of an attempt to solve these challenges in the context of supervised RL . POET uses the Minimal Criterion Coevolution ( MCC ) algorithm ( Brant & Stanley , 2017 ) to co-evolve increasingly complex environments and agents and then directly optimizes an RL objective either with evolution strategies ( ES ) ( Salimans et al. , 2017 ) or proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) . Within POET both the MCC for suitable learning outcomes and agent optimization rely upon the extrinsic reward within the environment . This approach works well for the reward-dense bipedal walker control problem ( Brockman et al. , 2016 ) but can often become challenging , even in conceivably simple RL problems , to generate desired outcomes ( Vecerik et al. , 2018 ) . This problem becomes more challenging when the reward is sparse , especially with a long-horizon , where the agent effectively has no reward until termination . Reward-shaping can be used ( Ng et al. , 1999 ) to help improve the density of reward and aide exploration . However , reward-shaping does not encourage open-ended learning , because it rewards the agent for following a hand-designed objective driven path , which limits open-endedness and often leads to poor performance ( Lehman & Stanley , 2011b ) . An alternate approach is found in the field of unsupervised RL where we don ’ t use an extrinsic reward function at all . Instead the objective is to learn a group of behaviors and hope to observe the agent solving task through one of our explorations , an approach that shares some characteristics with the problem of novelty search ( NS ) in open-ended learning ( Lehman & Stanley , 2011a ) . The acquisition of a diverse set of useful skills without extrinsic reward is in itself no-easy task . We consider a skill as a latent-variable conditioned policy that alters the state of the environment consistently ( Gregor et al. , 2016 ) , ( Eysenbach et al. , 2018 ) . The utility of a skill is typically based on its diversity to another , for example whilst one skill might be standing still , walking , dancing and running are all vastly different and a good algorithm will aim to learn the entire set of skills . Skills are then often selected for a task specific problem either by imitation learning or by directly selecting an encoding on the latent-variable conditioned policy . The learned behaviors can also be used as a pre-conditioned policy for regular RL with rewards similar to pre-training on ImageNet ( Deng et al. , 2009 ) In this paper we propose Rewardless Open-Ended Learning ( ROEL ) an unsupervised deep RL algorithm based upon the POET framework . The key-idea of this work is to use constant environment mutation to generate a diverse set of skills capable of solving increasingly complex and novel problems . We hypothesize this combination of open-ended learning with unsupervised RL will generate sets of behaviors that are more diverse and general than unsupervised RL agents simply trained on a static environment . Behaviors can then be selected for a specific task by directly selecting from a latent encoding , or by comparing the state-space transition to a demonstration . Then , if a skill can only be demoed on a single environment , our environment generalist approach ensures this type of behavior will generalize across many environments . Our paper makes the following contributions : • We present a new algorithm , ROEL , that combines unsupervised RL with open-ended learning to generate diverse and general behaviors ( Section 3 ) . An approach that has not been attempted before with agent-environment co-evolution ( Section 2 ) . • We train an unsupervised learning algorithm on the bipedal-walker environment . An environment with a difficult task objective and a non-stationary environment transition . • We prove empirically , that our approach is capable of learning more general and diverse groups of policies than single environments , especially when deployed to environments with dynamics that differ from the training set ( Section 5 ) . 2 RELATED WORK . Central to recent open-ended learning solutions is quality diversity ( QD ) , based on NS , ( Lehman & Stanley , 2011a ) , ( Lehman & Stanley , 2011b ) QD algorithms aim to return a diverse set of highquality solutions by carefully balancing diversity and optimization ( Pugh et al. , 2016 ) , ( Mouret & Clune , 2015 ) . In traditionally hard RL exploration problems Ecoffet et al . ( 2021 ) produced significant performance improvements by maintaining an archive of stepping stones from state to state . QD itself however does not necessarily provide open-ended learning , most approaches for openended learning rely upon constantly generating new challenges for the agent to overcome . This can be through co-evolution ( de Jong & Pollack , 2004 ) , with an adversarial agent ( Florensa et al. , 2018a ) or simply the agent itself ( Team et al. , 2021 ) . In a similar vein research into self-generated curricula aims to automatically generate a sequence of objectives that efficiently allow the agent to learn complex behaviors . These approaches include : reverse goal-oriented curriculum generation ( Florensa et al. , 2018b ) , teacher-student curriculum learning ( Matiisen et al. , 2017 ) , and procedural content generation ( Justesen et al. , 2018 ) , ( Hafner , 2021 ) . Another approach , world-agent based co-evolution , aims to constantly evolve the environment with the agent to generate pressure for agents to gather new skills . To facilitate open-ended learning we make use of a similar framework to POET ( Wang et al. , 2019 ) ( Wang et al. , 2020 ) where environment-agent pairs are evolved based on their QD using insights from the minimal criterion coevolution algorithm ( MCC ) ( Brant & Stanley , 2017 ) . The MCC performance ranking of agents in POET is based upon the agents total average episodic return from the extrinsic reward . Our approach extends this problem to the unsupervised domain , allowing for the discovery of diverse and useful sets of behaviors . PAIRED ( Dennis et al. , 2021 ) co-evolves 2 agents , a protagonist and antagonist where the protagonist aims to solve tasks generated by the antagonist . Regret is then defined as the average score of the protagonist and the best score of the antagonist over multiple trials . A separate adversary agent is then also trained , with the objective to minimax regret between the 2 agents by automatic environment generation . In contrast POET effectively maintains a population of minimax agents . We found the POET framework to be a suitable procedure to design a rewardless open ended learning algorithm . The results in PAIRED are inconclusive as to whether POETs population QD approach outperforms PAIREDs adversarial approach , we leave this investigation to future work . Our methodology aims to extend open-ended learning to the unsupervised RL setting , in an effort to generate increasingly sophisticated predictable and diverse behaviors . Significant previous work in fixed environment unsupervised RL has focused on intrinsic motivation ( Oudeyer , 2007 ) , ( Oudeyer et al. , 2007 ) , ( Chentanez et al. , 2005 ) , with examples including : empowerment ( Klyubin et al. , 2005 ) , ( Mohamed & Rezende , 2015a ) , state count based exploration ( Bellemare et al. , 2016 ) , curiosity driven exploration ( Pathak et al. , 2017 ) , surprise maximization ( Achiam & Sastry , 2017 ) and minimization ( Berseth et al. , 2021 ) . Compared to intrinsic motivation skill discovery aims to explore a group of skills that can then be exploited by imitation learning ( Schaal , 1997 ) , inverse reinforcement learning ( Russell , 1998 ) , or fine-tuning . Skill discovery has generally made use of relations between RL and information theory ( Ziebart et al. , 2008 ) , ( Florensa et al. , 2017 ) , ( Daniel et al. , 2012 ) , ( Eysenbach et al. , 2018 ) , ( Wang et al. , 2018 ) , ( Sharma et al. , 2020 ) . DIAYN ( Eysenbach et al. , 2018 ) explicitly aimed to exploit HRL by designing a reward function to maximize stateentropy , so that skills are differentiated by visiting different state-space regions . DADS ( Sharma et al. , 2020 ) subsequently aimed to discover skills that are both diverse and predictable . Our work then develops the methodology in DADS to train agents over a range of increasingly complex environments , creating unending environmental pressure for agents to evolve ever more complex and diverse skills . Gupta et al . ( 2020 ) provides another interesting approach to the problem of generalized unsupervised reinforcement learning by combining meta-learning ( Finn et al. , 2017 ) with DIAYN ( Eysenbach et al. , 2018 ) to automatically generate meta-training tasks . This is a useful approach for the fixed environment setting , but does not address the challenge of open-ended learning , only meta-training on a static state-transition distribution . 3 REWARDLESS OPEN-ENDED LEARNING ( ROEL ) . Building on previous work in unsupervised RL and mutual-information based exploration ( Eysenbach et al. , 2018 ) , ( Sharma et al. , 2020 ) , ROEL uses the POET framework ( Wang et al. , 2019 ) to endlessly generate agents capable of exhibiting increasingly complex , diverse and predictable behaviors . The policies learnt by ROEL can then be exploited to maximize a task specific objective using some form of supervision . Importantly our algorithms use of non-stationary environmental representations allows for the discovery of new emergent behaviors based on previous simpler behaviors .

Summary: Rewardless Open-Ended Learning (ROEL) is fundamentally a combination between two Paired Open-Ended Trailblazer (POET) and Dynamics-Aware Unsupervised Discovery of Skills (DADS). POET presents a framework for "open-ended learning" which automatically generating progressively more difficult environments to elicit agents with novel behaviors. DADS introduces a method of "unsupervised RL" which learns a variety of skills without rewards solely based on mutual information between state transitions and skill-determining latent variable. The result is an RL framework which generates environments to elicit controllable behaviors without a reward function from the environment.  The following empirical evaluations are performed: - Qualitative assessment of skills learned by ROEL - Qualitative comparison or ROEL and DADS - Measurement of rewards of different skills learned by ROEL and DADS - Measurement of state predictability of skills in different environments by ROEL and DADS  The primary contributions of the paper are: - Combining open-ended RL and unsupervised RL - ROEL yields a more robust set of policies in a bipedal-walker environment than the DADS baseline.