Rewardless Open-Ended Learning (ROEL)
Authors: Alexander Quessy
Thomas Stuart Richardson
Venue: ICLR 2022 Submitted (2022)
Keywords: unsupervised reinforcement learning
open-ended learning
skill discovery

Abstract: Open-ended learning algorithms aim to automatically generate challenges and solutions to an unending sequence of learning opportunities. In Reinforcement Learning (RL) recent approaches to open-ended learning, such as Paired Open-Ended Trailblazer (POET), focus on collecting a diverse set of solutions based on the novelty of an agents pre-defined reward function. In many practical RL tasks defining an effective reward function a priori is often hard and can hinder an agents ability to explore many behaviors that could ultimately be more performant. In this work we combine open-ended learning with unsupervised reinforcement learning to train agents to learn a diverse set of complex skills. We propose a procedure to combine skill-discovery via mutual information, using the POET algorithm as an open-ended framework to teach agents increasingly complex groups of diverse skills. Experimentally we demonstrate this approach yields agents capable of demonstrating identifiable skills over a range of environments, that can be extracted and utilized to solve a variety of tasks.

Under review as a conference paper at ICLR 2022
REWARDLESS OPEN-ENDED LEARNING (ROEL)
Anonymous authors
Paper under double-blee bbinn review
ABSTRACT
Open-ended learning algorithms aim to automatically generate challenges and solutions to an unending sequence of learning opportunities. In Reinforcement Learning (RL) recent approaches to open-ended learning, such as Paired Open- Ended Trailblazer (POET), focus on collecting a diverse set of solutions based on the novelty of an agents pre-defined reward function. In many practical RL. tasks defining an effective reward function a priori is often hard and can hinder agents ability to explore many behaviors that could ultimately be more perfor- mant. In this work we combine open-ended learning with unsupervised reinforce- ment learning to train agents learn diverse set of complex skills. We propose procedure to combine skill- discovery via mutual information, using the POET al- gorithm open-ended framework teach agents increasingly complex groups of diverse skills. Experimentally we demonstrate this approach yields agents ca- pable of demonstrating identifiable skills over a range of environments, that can be extracted and utilized to solve a variety of tasks.
INTRODUCTION
Development of machine learning over the past several decades has principally relied upon searchers solving a series of challenges proposed by experts within the community (e.g. MNIST Leeun et al. (1998). ATARI [Blllemare al. (2013) and robotics Yu al. (2021)). Learning al- gorithms typically solve these challenges through a huge number of observations, and whilst they have exceeded human-leve performance on a variety of these tasks (Silver et al.] (017)) (Berner et al. 2019). they often require significantly more examples before coming close to matching hu- man performance (Toromanoff et 2019). Humans in contrast leverage many diverse experiences to solve complex tasks (Wang et al. 2018) often without well defined objective reward. Should we attempt to teach reinforcement learning (RL) agents in a similar fashion? We use this question to motivate the design procedure that aims to learn a diverse set of increasingly complex novel behaviors without supervision (labels) or incentives (external rewards).
Open-ender algorithms aim to automatically generate challenges to be solved (Forestier et al. 2020), (Schmidhuber (2012), (Standish) 2002). (Secretan et al. [01 that can serve as stepping stones to increasingly complex problems. These algorithms learning procedure because they do not rely upon manually selected challenges; instead they search for challenges that form stepping stones to solve larger ones, collecting a diverse set of experiences along the way. Designing algorithms to achieve these open-endedness can be challenging. Often requiring a careful balance between diversity, optimization (Mouret & Clune 2013), (Pugh al.) 2016), and domain complex enough for sufficient experiences/challengee to be generated in order learn a suitable curricula.
useful in designing general unsupervised
The Paired EndedE Trailblazer (POET) algorithm (Wang et al. 2019). (Wang et al. 2020 is a recent example of an attempt to solve these challenges the context of supervised RI.. POET uses the Minimal Criterion Coevolution (MCC) algorithm (Brant & Stanley 2017) to co-evolve in- creasingly complex environments and agents and then directly optimizes an RL objective either with evolution strategies (ES) (Salimans et al. 2017) proximal policy optimization (PPO) (Schulman et al. 2017). Within POET both the MCC for suitable learning outcomes and agent optimization rely upon the extrinsic reward within the environment. This approach works well for the reward-dens bipedal walker control problem (Brockman et al. 2016) but can often become challenging, even in conceivably simple RL problems, to generate desired outcomes (Vecerik et al. 2018).

---

Under review as a conference paper at ICLR 2022
This problem becomes more challenging when the reward is sparse, especially with long- long-horizon, where the agent effectively has no reward until termination. Reward-shaping can be used (Ng et al, 1999) to help improve the density of reward and aide exploration. However, reward-shaping does not encourage open-ended learning, because rewards the agent for following a hand-designed objective driven path, which limits open-endedness and often leads to poor performance (Lehman & (annlyy220110 An alternate approach is found in the field of unsupervised RL where we don't It use an extrinsic reward function at all. Instead the objective to learn group of behaviors and hope to observe the agent solving task through one of our explorations, an approach that shares some characteristics with the problem of novelty search (NS) in open- ended learning (Lehman & Stanley 2011a).
The acquisition of diverse set of useful skills without extrinsic reward is in itself no-easy task. We consider skill as a latent-variable conditioned policy that alters the state of the environment consistently (Gregor et al. 2016). (Eysenbach et af., 2018). The utility of a skill is typically based on its diversity to another, for example whilst one skill might be standing still, walking, dancing and running are all vastly different and good algorithm will aim to learn the entire set skills. Skills are then often selected for task specific problem either by imitation learning or by directly selecting an encoding on the latent-variable conditioned policy. The learned behaviors can also be used pre-conditioned policy for regular RL with rewards similar to pre-training on ImageNet (Dng et al. 2009)
In this paper we propose Rewardless Open-Ended Learning (ROEL) an unsupervised deep RL algo- rithm based upon the POET framework. The key-idea of this work is to use constant environment mutation to generate diverse set of skills capable of solving increasingly complex and novel prob- lems. We hypothesize this combination of open-ended learning with unsupervised RL will generate sets of behaviors that are more diverse and general than unsupervised RL agents simply trained on a static environment. Behaviors can then be selected for specific task by directly selecting from a latent encoding, or by comparing the state-space transition to demonstration. Then, if a skill can only be demoed on a single environment, our environment generalist approach ensures this type of behavior will generalize across many environments.
Our paper makes the following contributions:
We present a new algorithm, ROEL, that combines unsupervised RL with open-ended learning to generate diverse and general behaviors (Section 3). An approach that has not been attempted before with agent-environment co-evolution (Section 2
We train an unsupervised learning algorithm on the sopedal-walkee environment. An envi- ronment with difficult task objective and a non-stationary environment transition.
We prove empirically, that our approach is capable of learning more general and diverse groups of policies than single environments, especially when deployed to environments with dynamies that differ from the training set (Section
2 RELATED WORK
Central to recent open-ended learning solutions is quality diversity (QD), based on NS, (Lehman Stanley a0 (Lehman & Stanley 2011b) QD algorithms aim return a diverse set of high- quality solutions by carefully balancing diversity and optimization (Pugh et al. 2016), (Mouret & Clune| 2013). In traditionally hard RL exploration problems Ecoofft et al. (222 produced signifi- cant performance improvements by maintaining archive of stepping stones from state to state.
QD itself however does not necessarily provide open-ended learning, most approaches for open- ended learning rely upon constantly generating new challenges for the agent to overcome. This can be through co-evolution (de Jong & Pollack 2004), with an adversarial agent (Florensa et al.) 201 8a) or simply the agent itself (Team et al. 2021). In similar vein research into self-generated curricula aims to automatically generate a sequence of objectives that efficiently allow the agent to leam complex behaviors. These approaches include: reverse goal-oriented curriculum generation (Florensa et al. (2018)). teacher-student curriculum learning (Matiisen et al. 2017). and procedural content generation (Justesen et al. 2018), (Hafner 2021
2

---

Under review as a conference paper at ICLR 2022
Another approach, world-agent based co-evolution, aims to constantly evolve the environment with the agent to generate pressure for agents to gather new skills. To facilitate open-ended learning
make use of similar framework to POET (Wang et al., 2019) (Wang et al. 2020) where environment-agent pairs are evolved based on their QD using insights from the minimal criterion coevolution algorithm (MCC) (Brant & Stanley 2017). The MCC performance ranking of agents in POET is based upon the agents total average episodic return from the extrinsic reward. Our approach extends this problem to the unsupervised domain, allowing for the discovery of diverse and useful sets of behaviors. PAIRED (Dennis et al., 2021) co-evolves 2 agents, protagonist and antagonist where the protagonist aims solve tasks generated by the antagonist. Regret is then defined as the average score of the protagonist and the best score of the antagonist over multiple trials. A separate adversary agent is then also trained, with the objective to minimax regret between the 2 agents by automatic environment generation. In contrast POET effectively maintains population of minimax agents. We found the POET framework to be a suitable procedure to design a rewardless open ended learning algorithm. The results in PAIRED are inconclusive to whether POETs population QD approach outperforms PAIREDs adversarial approach, we leave this investigation to future work.
Our methodology aims to extend open- learning to the unsupervised RL setting, in an effort to generate increasingly sophisticated predictable and diverse behaviors. Significant previous work in fixed environment unsupervised RL has focused on intrinsic motivation (Oudeyer 2007), (Oudeyer
iet al. 2007). (Chentanez et al.] (005)), with examples including: empowerment (Klyubin et al.)
2005 (Mohamed & Rezende (2015a), state count based exploration (Bellemare et al., 2016)) cu-
riosity driven exploration (Pathak et al. 2017). surprise maximization (Achiam & Sastry 2017) and
minimization (Berseth et al.) 2021). Compared to intrinsic motivation skill discovery aims to explore group of skills that can then be exploited by imitation learning (Schaal 1997). inverse reinforce- ment learning (Russell 1998), or tinettuning Skill discovery has generally made use of relations between RL and information theory (Ziebart et al. 2008). (Florensa et al. (017), (Daniel et al.) 2012), (Eysenbach et al. 2018), (Wang et al. 2018), (Sharma et al. 22200) DIAYN (Eysenbach) ct al. 2018) explicitly aimed to exploit HRL by designing a reward function to maximize state- entropy, so that skills are differentiated by visiting different state-space regions. DADS (Sharmal et al. 2020) subsequently aimed to discover skills that are both diverse and predictable. Our work then develops the methodology in DADS to train agents over a range of increasingly complex en- vironments, creating unending environmental pressure for agents to evolve ever more complex and diverse skills.
Gupta et al. 2020) provides another interesting approach to the problem of generalized unsupervised reinforcement learning by combining meta-learning (Finn et al.] 2017) with DIAYN (Eysenbach) al. 2018) automatically generate meta-training tasks. This is a useful approach for the fixed environment setting, but does not address the challenge of open-ended learning, only meta-training on a static state-transition distribution.
3 REWARDLESS OPEN-ENDED LEARNING (ROEL)
Building on previous work in unsupervised RL and mutual-information based exploration (Eysen- bach et al. 2018), (Sharma et al. 2020), ROEL uses the POET framework (Wang et al., 2019) to endlessly generate agents capable of exhibiting increasingly complex, diverse and predictable be- haviors. The policies learnt by ROEL can then be exploited to maximize a task specific objective using some form of supervision. Importantly our algorithms use of non-stationary environmen- tal representations allows for the discovery of new emergent behaviors based on previous simpler behaviors.
3.1 POET FRAMEWORK
We use POET as framework to endlessly discover agents with increasingly complex novel behav- iors. POET accomplishes this through growing and optimizing a paired population of agents and unique environments (ea-pairs), contained within an environment-action list (ea-list). To control the evolution of the ea-list each environment is represented by an environmental encoding (EE), which maps environment attributes to environment representation, and can easily be encoded and mutated. The Envirommental Characterization (EC) is then used to describe the qualities of that en- vironment, in ROEL use the domain-general Performance of All Transferred Agents (PATA-EC)
3

---

Under review as a conference paper at ICLR 2022
from (Wang et al. 2020). The procedure to generate and search for behaviors within POETs main loop can then be summarized in the following steps:
1. Mutate: Every M steps, provided ea-pair exhibits suitable performance, it is mutated to a generate new child EEs. The children are then filtered via a minimal criterion (MC) (Brant & Stanley 2017) to ensure the environment is not too easy or hard. EEs that satisfy the MC are then ranked based on the performance of the best agents and the most novel EEs added to the active population. When the active population reaches a predefined size the least novel pairs are dropped from the active environment and added to set of archived environments. Novelty of newly-generater environments is based on how different the ranking of agents trained on the new environment changes compared to the ranking of agents on old environments, based on the idea that useful challenges should make novel distinctions amongst agents in system (de Jong & Pollack 2004).
2. Optimize: Every step POET optimizes every agent to maximize its reward on the given environment. In the original POET architecture this is the extrinsic reward from Markov decision process, optimized with the Evolution Strategies (ES) algorithm (Salimans et al.,
2017 In ROEL we optimize the mutual-ing intuuuaaiinoommiiin based reward as an RL optimization problem using Soft-Actor Critic (Haarnoja et al. (0188).
3. Transfer: Every N steps of POET's outer loop, the performance
active agents compared and ranked. If an agent differentt environment outperforms the current agent then the agent is replaced with the more performant agent. This constant transfer proved critical in POETs original implementation (Wang et al.) 2019) for discovering stepping stones to more performant behaviors and to drive useful open-ended learning.
3.2 REWARDING ROEL
ROEL aims to create agents that exhibit diverse set of predictable and useful behaviors across range of problems. POET uses the extrinsic return generated from the agents interaction with the environment to generate an objective minimize, as a supervised learning problem. The return of each ea-pair in POET is key to calculate each EEs PATA-EC and decide how to evolve the ca-pair. Subsequently in ROEL we use mutual information I to construct reward function that aims to drive open-ended learning and motivate exploration, similar to Houthooft et al. (2017) and Mohamed & Rezende (2015b).
I(x;)) -
Following the definition of mutual information, shown in equation I Maximizing I with respect to Y is equivalent maximizing the entropy H(X) whilst minimizing conditional entropy H(XJY), where and Y are functions of state and actions respectively. This objective therefore aims to find actions to maximize state entropy, and is therefore exploratory as an RL policy. This property was recently used by AAchiam et al. (2018), Eysenbach et al. (2018), Gregor et al. (2016) and Sharma al. 2020) to learn a diverse and distinet set of skills in a policy.
In ROEL we make use of the reward function specified by DADS (Sharma et al. 2020) to learn policy conditioned on a latent variable 2, sampled from the transition dynamics prior p(2) Z. DADS differs from previous conditional entropy based approaches through the insight to leverage work model based reinforcement learning (MBRL), where an objective to learn a model to estimate the dynamics of the system p(s' (s,))s.)..
a). Therefore DADS learns 2 functions:
A skill-conditioned policy ग [a|s;; 2) where skills are drawn from the larger skill set such that Z.
Skill-dynamics: A skill-conditioned transition function (s(((1s))) that predicts the state transition given skill
I(s/;=1s) H(2]s) - H(2)s',
DADS then uses the nutual-informatioon between the next state with respect to a skill 2, given the current state, to quantify how informative skill is of the transition s' 8, this is shown in
4

---

Under review as a conference paper at ICLR 2022
equation Using suitable approximation and the definition of conditional mutual information a reward function can be defined, as shown equation 3 (full derivation found in Sharma et al. 2020))
4o ( (s E (s(5'|
(s s') log
+ +ogg(L
(3)
The reward in equation 3 encourages agents to exhibit skill diversity by sampling L skills from the prior 2; ~ p(z), be used with the skill dynamics estimate qs- We then use the return generated over several episodes to generate statistic for the average return, that can be used to mutate and transfer agents within ROEL under the POET framework. This approach introduces -sattioonnnaity into the skill transition and offers the following distinct advantages over simply training on a static environment.
ROEL learns an open-ended diverse set of skills
The combination of mutation and transfer learning was the key to POET learning increasingly com- plex skills in supervised setting. For ROEL this procedure of gradually improving novelty via increased environmental complexity allows for simple, foundational, skills to be uncovered early in training. These are then used by ROEL as stepping stones to more complex skills when the chal- lenge of more complex environments more complex behaviors). Generally this has the effect of improving the diversity of the set skills uncovered as there is simply larger distribution of possible state trajectories in more complex envi- ronments. But, designing building blocks to transfer the set of skills to the agent is key to efficiently discovering large of skills.
presented (which in turn provide the opportunity to learn
ROELs outer loop searches for environments that allow access to greater intrinsic reward, balancing novelty over performance via MCC algorithm. This is implemented with PATA-EC (Perfor- mance of All Transferred Agents Environmental Characterization) ranking, described in Wang et al. 2200 where each agent is ranked on an environment based on its performance against all other environments. EEs that change the ordering of other agents PATA-EC are favoured as this implies that the EE provides new set of challenges the set of active agents. overcome this new chal- lenge the agent will need to optimize different behaviors improving the diversity of skills learnt in the long run.
ROEL is able to generalize over a large group of environments
The search for novel environments in ROELs outer loop can also serve to improve the generalization of ROEL at test-time. By evolving EEs to generate novel behaviors ROEL able to sample large distribution of possible environments, but unlike domain randomization this search is directed to find environments that explicitly cause a change behavior. This allows for ROEL to replicate the skills it learns across much broader range of environments than if we were to simply train single-static environment. We explore this characteristic further section 5.2 by comparing the performance of ROEL DADS when tested hard EE unobserved at train-time.
EXPERIMENT SETUP
Using an implementation similar DADS (Sharma al., 2020), we use Soft Actor-Critt (SAC) (Haarnoja et al.) 2019). an off-policy actor-critic deep RL algorithm, specifically EC-SAC, to train our policy ( 8, 2). SAC (Haarnoja et al. 2018h) based on the maximum entropy framework and uses entropy regularization to tradcoff robustness for exploration, policies encouraged to discover action-sequences with similar trajectories under a given skill.
To test ROEL we use the bipedal control environment (Brockman al. 2016) with the custom EE extension described in Wang et al. (2019). allowing us to mutate the environment by varying a set of hand- picked obstacle hyper-parane parameerr,,,sss, such stump shape, crrfaccerouugness.. and gap width. This encoding offers sufficient opportunities for ROEL to discover diverse behaviors and is described in more detail in Appendix B In future work we would like investigate how much improving the
5

---

Under review as a conference paper at ICLR 2022
EEs expressiveness effects ROELs performance by using CPPNs (Compositional Pattern Producing Networks) (Stanley 2007). similar to Wang et al. (2020).
For the experiments in section 5 we trained ROEL over 24 iterations (5400 episodic iterations) using a total of 224 processing cores distributed over nodes (each with dual socket CPUs) for s 12 hours. ROEL is highly parallel allowing for each ea-pair to be asynchronously during the training step. Our baseline DADS environment is trained using the same parameters as the ROEL inner-loop, using the original bipedal normal environment. We find that ROEL takes slightly longer to train than DADS (+ hours) as there is no need to calculate the PATA-EC and minimal criterion the mutation and transfer steps. We also train a domain randomized (DR) DADS baseline on the parameters contained within the ROEL EC to test the capabilities of ROELs auto-curricull to generalize to unseen environments. We provide further details of our specific training-evaluution architecture and hyperparameters in Appendix
RESULTS
this section, we present our results from using ROEL with the bipedal walker environment. Ini- tially we qualitatively evaluate the types of skills learnt by ROEL and compare ROELs in and out- of-sample performance when used on a fixed skill z. We then quantitatively evaluate ROEL using DADS baseline, aiming assess initial hypothesis that ROELs of open-endedness al- lows it to learn skills that are more diverse and general than unsupervised RL, trained on a single environment.
5.1 QUALITATIVE ANALYSIS
Shuffling Forward
7
Leap
T
Spics
7
Shuffling Dackwards
Parching
Figure 1:
Examples of novel skills learnt by ROEL. Images are generated through policy rollout under a fixed skill z specified at the beginning of each episode.
Figure shows of the 20 skills leartt by ROEL hand selected from a single run. Some of the skills are clearly motivated by the environments ROEL is exposed during training. For example the shuffling forward skill allows the agent to move over pits generated the training environment and keeps the LIDAR sensor close to the ground, helping improve sensor accuracy. The leap

---

Under review as a conference paper at ICLR 2022
skill likely improves the intrinsic return when hills present in the environment. The one-leg and perching skills may not immediately appear useful, but could prove key stepping stones to learn more complex modes of locomotion. For example the one-leg prior has learnt how to use the ungrounded leg to maintain balance for short period.
Selecting Stance
ROEL
Crossing the
Climbing
Selectin. Stance
DADS
Crossingt Cap
Figure 2: ROEL and DADS solving the same out-of-sample EE. ROEL is able to leverage its knowl- edge of previous tasks to rapidly cross a gap and climb the hill, whilst DADS meanders around the gap.
Figure demonstrates how ROELs behavior differs from DADS out-of-sample environments, using an EE containing gaps and roughness different from that seen by any agent at during training. When the ROEL agent figure crosses the pit rapidly places a leg on the other side and pushes itself over the hole, it is then able to move its leg up to pull itself over the hill. The DADS agent on other hand stops upon reaching the pit and explores the hole with its legs ground sensor, agent then slowly shuffles over the hole, it never gets over the hill in the time horizon. We find this a good qualitative example of how ROEL is able to leverage experience gained through mutation and goal-switching respond out-of-sample challenges.
5.2 QUANTITATIVE ANALYSIS
To assess the ability of ROEL to perform out-of-sample original environmental encoding to generate samples under each skill prior.
the EE seen in figure and the
It difficult to directly measure the diversity of each agents prior based on samples from the vironment. Observing diversity from the agents interaction with the environment alone requires us characterize each with suitable statistic. If the environment itself features stationary agents- environment dynamics, the variance of the trajectories can be used to measure behavioral diversity the behavior is time-invariant. This is the case in Mujoco (Todorov et 2012) where the environ- ment is flat-ple fla-plane: DADS therefore compares novelty via the agents snate-transition. With bipedal, we cannot make this assumption. because the terrain is non-uniform and the agent-envirrnment dynamics change with time. For ROEL, this shift in dynamics is also made intentionally via our constant evolution of the EE.
An ideal statistic for ROEL would characterize each skills policy dynamics, independent of the environment based on observations alone. We find that the state-space bayesian forecasting literature West & Harrison (1997) does offer some interesting possible solutions to this problem but this is still challenging. We would need to design transfer function capable of capturing the skills fundamental dynamics but we find this to be outside the scope of this paper and leave this future work. Alternatively, we use the return (cumulative reward) from each episode to characterize each prior. This is based the intuition that if the reward function itself describes example of novel

---

Under review as a conference paper at ICLR 2022
behavior, the difference in return of priors can be used heuristic to measure diversity. In bipedal, the reward function is based on the ability the agent to reach the end of the episode.
In-Sample
Out-of-Sample
0.10
ROEL
ROEL
0.05
0.00
0.10
DADS
DADS
0.05
0.00
250
-200
150
100
-50
250
-200
150
100
-50
Return
Return
Dispersion of skill returns using the original bipedal extrinsic reward function. Each skill sample has been fitted to normal distribution and assigned unique color.
Figure 3:
Figure 3 shows the fitted distribution of returns from each prior, both for ROEL and DADS, on the in- and out-of-sample environments. To fit the distribution we sample 100 episodes from each prior and calculate the empirical mean and variance of each to fit normal distribution. In-sample, the prior of each sampled mean is more dispersed than in DADS. This suggests greater skill diversity, helping explain the behavior each prior in ROEL is generally unchanged, whilst DADS features a large change in dispersion. This indicates that the skills learnt by ROEL generalize well this new out-of-sample challenge and highlights the importance of providing a diverse set of environments during training if we want learn a diverse set of skills.
qualitatively observe in figure [ Out-of-sample, the return of
Another useful indicator of the capability of our agent to generalize to new environments is the error between the expected observation trajectory from the skill-dynamies network and the actual observation trajectory. If the skill-dynamics network is capable of accurately predicting the future observations, then the value of intrinsic motivation used train agents will promote skills that are more diverse and predictable. The implementation of probabilistic ensembles with trajectory sam- pling (PETS) in Chua et al. (2018) suggests, that accurate state estimation is key for successful model based reinforcement learning, and adds degree of confidence in our approach. The accu- racy of our skill-dynamic network is therefore a useful heuristic for the performance of ROEL on downstream tasks.
Figure plots the error in the prediction from the skill-dynamics network for both in- and out-of- sample EES the core hull dynamics. For our baseline, instead used DADS trained with domain randomization (DR) across the same EE as ROEL. This provides a more accurate comparison of how our auto-curriclla based approach in ROEL compares to a naive DR approach (when training without DR the error of DADS out-of-sample diverges to 100 times that of ROEL due to over-fitting). In-sample, we find DADS outperforms ROEL consistently with smaller mean and variance all 3 parameters.
-ample however, DADS often fails to predict the skill dynamics, rapidly diverging an error 3 times that of ROEL in some cases whilst ROEL is generally unchanged. An exception to this is in the y-velocity, where the mean of ROEL and DADS-DR is generally the same, but DADS-DR has less variance. The x-velocity state offers some of the most interesting results, showing DADS-DR rapidly diverging in both mean and variance, whilst the error ROEL is minimal. This difference can be explained by the differences in the DADS-DR and ROEL training procedures. In ROEL, the auto-curricula encourages longer episodes so that the behavior dynamics are longer. However, in DADS-DR, DR, the environment selection is random and this means that the hardest obstacles often appear within the training environment so that the agent finds it difficult to traverse and this causes the episode terminate earlier. This helps to explain why the x-velocity prediction is better, be-
8

---

Under review as a conference paper at ICLR 2022
Hull Angular Velocity
y- Velocity
X- Velocity
0.00
Absolgte Eiror (dogle)
In-Sample
6100
0:00
0.00
Absolutc
40.35
10.03
810
-0.35
810
0,30
Out-of-Same -Sampl
0.15
Abgolue Erpor (dbgic)
1012
0.09
0.30
0..5
Abvelute Error
+2.22
000
com
000
-0.10
steps
steps
steps
Figure 4: Comparison of the in and out-of-sample performance of the skill-dynamics network, sam- pled from random discrete priors. The average error between the predicted and actual observation is plotted and enclosed by a shaded region out to I-SD. Note the different y-axis scale in each graph.
cause ROEL is able observe
practical set of trajectories to learn generally diverse skills. Whilst the y-velocity prediction is relatively accurate for both ROEL and DADS-DR the reduced variance in ROEL is likely due to the environment observing larger range of failures caused by the environment, which are more likely in the off-sample harder bipedal environment.
The fact that ROELs prediction error is unchanged in the out-of-sample environment is evidence that the skill-dynamics accurately generalize to unknown environments. Consequently, this suggests that the intrinsic reward received by ROEL during training captures the general environment dynamics. ROEL is therefore able to learn skills that generalize over much larger distribution of possible environments. These skills are then pushed to more diverse, as the outer loop of ROEL encourages the development of environments that lead to long term maximization of intrinsic reward leading to a set novel, predictable and useful skills.
CONCLUSION
In this work described how combining previous work in conditional entropy based unsupervised RL with POETs open-ended framework lead to the development of ROEL. An algorithm that aims automatically generate increasingly complex and useful skills by generating its curricula without the need for supervision. We demonstrated that this approach produces agents with skills that are more robust, when used in out-of-sample environments, than simply training on single static environment.
The compelling potential of this work lies in the open-ended nature of our methodology. Our skills are automatically generated by solving challenges presented by co-evolution, but we do not need specify an extrinsic reward function, provided suitably expressive EC, ROEL should learn to generate skills indefinitely that constantly increase in complexity as the agent is presented with increasingly difficult challenges.
In future work, we aim to further investigate the open-endedness of ROEL,, by training the agent for many more iterations and aiming to discover how diverse the learnt skills may become. Further, plan to investigate how ROEL can be combined with CPPNs to generate a more expressive EC Stanley 2007). and how this could be directed to guide novel skill generation beyond the MCC criteria. Following this, plan to investigate how ROELs learnt skills be used in concert to exhibit more complex behaviors at test time, perhaps by simultancously learning a meta-controller (Gupta al. 2020 that searches for novel skill combinations.

---

Under review as a conference paper at ICLR 2022
STATEMENT OF REPRODUCIBILITY
To ensure reproducibility of the algorithm and results described within our paper we have included parameters of the setup used to train and evaluate ROEL in Appendix A along with a general description of our implementation in section After opening the discussion forums for all submitted papers shall make directed comment the reviewers and area chairs with an anonymous repository containing:
Source Code of our model, including methods to train, test and evaluate ROEL.
Videos of demonstrations of the behavior of our trained agents.
Models and Datasets used evaluate the agents and generate figures in the report.
REFERENCES
Joshua Achiam and Shankar Sastry, Surprise-based intrinsic motivation for deep reinforcement 01732
learning. CoRR, abs/1703.01133 2017. URL.
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational Option Discovery Algorithms. arXiv: 1807. [cs], July 2018.
M. G. Bellemare, Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253-279, June 2013. ISSN 1076-9757. doi: 10. 16.1/3//airr.99)
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 1479-1487, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019. URL http: / arxiv. org/ abs / 1 912.99220 06680
Glen Berseth, Daniel Geng, Coline Manon Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Ja- yaraman, and Sergey Levine. Smirl: Surprise minimizing reinforcement learning in unstable environments. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httptttt////peeeeiiiiiiiee Eorum? -id c00oDoooo
Jonathan C. Brant and Kenneth O. Stanley. Minimal criterion coevolution: A new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 67-74, Berlin Germany, July 2017. ACM. ISBN 978 978-1-4503-44900-8 doi: 10.1145/3071178. 3071186.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. arXiv:1606.0155 005400 [es], June 2016.
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. In L. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Process- ing Systems, volume 17. MIT Press, 2005. URL http:
neurips. cc/
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement
learning in a handful of trials using probabilistic dynamics models. CoRR, abs/ 1tts//805.12111 14, 2018. URL http::tt///eee //axx/rrxii wwwprwwwvv...r/ee/ommooo htgg/abs//sat 12114
10

---

Under review as a conference paper at ICLR 2022
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Neil D. Lawrence and Mark Girolami (eds.), Proceedings of the Fifteenth International Con- ference Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 273-281, La Palma, Canary Islands, 21-23 Apr 2012. PMLR. URL https: //proceedings ml wtw.mm.....r......m v22 hwt./aanieell.. el... html
Edwin D. de Jong and Jordan B. Pollack. Ideal Evaluation from Coevolution. Evolution- ary Computation, 1292)119-112, June 2004. ISSN 1063-6560, 1530-9304. doi: 10.1162/
1063656047739551399
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPRO9, 2009.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design. arXiv:2012.02096 [cs], February 2021.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580- February 2021. ISSN 0028-0836, 1476-4687. doi: 10. 10.10088/411586--00-01155-----
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is All You Need: Learning Skills without Reward Function. arXiv:/802.0660 [cs], October 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. aggostig meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp- 1126-111 1135. PMLR, 06-11 Aug 2017. URL https::/ httpps:/pproceelingg ml w....prres/vv. v00/
Finnl7a. html
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic Neural Networks for Hierarchical Rein- forcement Learning. arXiv: N0KV::/704.30122 [cs], April 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic Goal Generation for Reinforcement Learning Agents. arXiv:1705.066 [cs), July 2018a.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse Cur- riculum Generation for Reinforcement Learning. arXiv: 1707.0000 [cs], July 2018b.
Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning. arXiv:700..02190 [cs], July 2020.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. arXiv:1611.07507 [cs], November 2016.
Variational
Intrinsic Control.
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Efi Kokiopoulou, Luciano Sbaiz, Jamie Smith, Gábor Bartók, Jesse Berent, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. TF-Agents: A library for reinforcement learning in tensorflow. https:://itttttit com tensorf Low agents, 2018. URL htt.a//pss/gggtt...ccc//eeeeerep 2019].
[Online; accessed 25-June-
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised Meta- Learning for Reinforcement Learning. arXiv:1806.0 04640 [cs, stat), April 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870 PMLR, 10-15 Jul 2018a. M ttattps://proceeiii ngs. ml wtt...pree/////eee rno ja 8b. html hb.html
Machine Learning,

---

Under review as a conference paper at ICLR 2022
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv: [cs, stat], August 2018b.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algo- rithms and Applications. arXiv: 1812.0599 [cs, stat], January 2019.
Danijar Hafner. Benchmarking the Spectrum of Agent Capabilities. arXiv:2109.06670 [cs], Septem- ber 2021.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational Information Maximizing Exploration. 1605 [cs, January 2017.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation. arXiv::806. 10729 [cs, stat], November 2018.
Diederik P. Kingma and Jimmy Ba. abs/1412 (bss/411..990,, 2015.
Adam: A method for stochastic optimization. CoRR,
A.S. Klyubin, Polani, and C.I., Nehaniv. Empowerment: Universal Agent- Agent-Centric Measure of Control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135, Edin- burgh, Scotland, UK, 2005. IEEE. ISBN 978-0-7803-9363-9, doi: 10.1109/CEC2220555544777
Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):278-2334, 1998. ISSN 00189219. doi: 10.1109/5.726791
Joel Lehman and Kenneth O. Stanley, Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation GECCO '11 pp. 211, Dublin, Ireland, 2011a. ACM Press. ISBN 978-1-4503-0557-0 doi: 10.1 00.114522011566220016666
Joel Lehman and Kenneth O. Stanley. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation, 19(2):189-22) 223, June 201 ISSN 1063-6560, 1530- 9304. doi: 10. 10.1162/EVC....00 a00022..
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-Student Curriculum Learning, arXiv:1707.0001 [cs], November 2017.
S. Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In NIPS, 2015a.
Shakir Mohamed and Danilo J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS 15, pp. 2015b. MIT Press.
Cambridge, MA, USA,
Bean-Baptiste Mouret and Jeff Clune. arXiv: 1504. 04909 [es, q-bio], April 2015.
Illmminating search
spaces by mapping elites.
Andrew Y. Ng, Daishi Harada, and Stuart.J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Confer- ence on Machine Learning. ICML '99, pp. 278-287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122 122.
Pierre-Yve Oudeyer. What is intrinsic motivation? typology of computational approaches. Fron- tiers in Neurorobotics, 1, 2007. ISSN 1662-5218. doi: 00339///uro.12.006.....
Pierre- Yves Oudeyer, Frdric Kaplan, and Verena V. Hafner. Intrinsic Motivation Systems for Au- tonomous Mental Development. IEEE Transactions on Evolutionary Computation, 11(2):265- 286, April 2007. ISSN 1089-778X. doi: 10. 109/TEVC.220069900000
12

---

Under review as a conference paper at ICLR 2022
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity- exploration by self-suue prediction. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 2778-2783 JMLR.org, 2017.
Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality Diversity: A New Frontier for Evolutionary Computation. Frontiers in Robotics and Al, 3, July 2016. ISSN 2296-9144. doi: 10.3389/frobb.2016.00000
Stuart Russell. Learning agents for uncertain environments (extended abstract). In Proceedings of the Eleventh Annual Conference Computational Learning Theory, COLT 98, pp. 101-103, New York, NY, USA. 1998. Association for Computing Machinery. ISBN 1581130570. doi: 279964
10.1145/27993..2796.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution Strategies as Scalable Alternative Reinforcement Learning. arXiv: 10t:::103008600 [cs, stat), September 2017.
Stefan Schaal. Learning from demonstration. In M. C. Mozer, M. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems, volume 9. MIT Press, 1997. URL http::::///prroeenggs.s.ttsm neurips. cc pttpppp///9///1llll/ Pappe00000aaaaae..pppppp
Jürgen Schmidhuber. POWERPLAY: Training an Increasingly General Problem Solver by Con- tinually Searching for the Simplest Still Unsolvable Problem. 11Xiv:111..5309 [es), November 2012.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. vrXiv:/707.066 [cs], August 2017.
Jimmy Secretan, Nicholas Beato, David B. D' D'mbrosiio, Adelein Rodriguez, Adam Campbell, Jeremiah Folsom-Kovait and Kenneth O. Stanley. Picbreeder: A Case Study in Collabo- rative Evolutionary Exploration of Design Space. Evolutionary Computation, 19(3):373-403 September 2011. ISSN 1063-6560, 1530-9304. doi: 10.1162/EVCO a a.00030.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-Aware Unsupervised Discovery of Skills. 1xXxx:1977.01577 [cs, stat), February 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354 359, October 2017. ISSN 0028-0836, 1476-4687. doi: 1..0088/aurre222700
Russell K. Standish. Open-Ended Artificial Evolution. arXiv:nlim/0210027, October 2002.
Kenneth O. Stanley. Compositional pattern producing networks: A novel abstraction of devel- opment. Genetic Programming and Evolvable Machines, 8(2):131-162, June 2007. ISSN 1389-2576. doi: 10.1007/110710-007-903----
URL htttp:://cc..rr///// 10077
s10710 007 9028-982888
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg. Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmies Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes- Fitt, Valentin Dalibard, and Wojciech Marian Czarneck.. Open-Ended Learning Leads to Gener- ally Capable Agents, arXiv:2107. 12808 [cs], July 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.
Marin Toromanoff, Emilic Wirbel, and Fabien Moutarde. Is Deep Reinforcement Learning Really Superhuman Atari? Leveling the playing field. arXiv:1908.04633 [cs], November 2019.
13

---

Under review as a conference paper at ICLR 2022
Mel Vecerik, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester, and Jon Scholz. A Prac- tical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning. arXiv:1810.01531 [cs], October 2018.
Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforreement learn- ing system. Nature Neuroscience, 2166))660-868 June 2018. ISSN 1097-6256, 1546-1726. doi: 10. 000088/611533018-0147777
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions. arXiv:
[cs], February 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeff Clune, and Kenneth O. Stanley, En- hanced POET: Open- Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions. arXiv:2003.0m 2805086000 [cs], April 2020.
Mike West and Jeff Harrison. Bayesian Forecasting and Dynamic Models (2nd Ed.). Springer-
Verlag, Berlin, Heidelberg, 1997. ISBN 0387947256.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. arXiv: 10897 [cs, stat), June 2021.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, AAAI008, pp. 1433-1438. AAAI Press, 2008. ISBN 9781577353683.
A IMPLEMENTATION DETAILS
In this appendix we provide details of our training setup for our DADS baseline and ROEL imple- mentation.
A.1 BASELINE
We use similar hyperparameters to the DADS implementation for our baseline, All skills are pa- rameterized via a one-hot vector sampled from the uniform prior p(z) = 1 where D 20 the number of skills, we only resample at the beginning of episode. We use the EC-SAC imple- mentation (Haarnoja et al., 2018a) to optimize the policy (a]s;; z). based on the implementation found in TF-Agents Guadarrama al. 2018). The SAC actor and critic networks both use 2 hidden layers with dimensions (512, 512). The critic network updated using Adam (Kingma & Ba 2015) with fixed learning rate 3 x 10-4 trained over a fixed initial entropy coefficient
0.1, updated via a soft update target function with co-efficient 0.005. Similar to DADS the policy stochastic and parameterized matrix that transforms the output to be in the range [- +1] via tanh function.
normal distribution (N[[s (s, =), (s, 2)), where Σ is a diagonal covariance
The skill-dynamics network the dimensions (512, 512) as the actor and critic networks. Our implementation uses replay buffer to collect samples of reward and trajectory, with batch size of 256. The policy and critic networks are then updated via gradient descent every 500 steps for 64 steps of Adam.
A.2 ROEL
Our ROEL implementation built on-top of the DADS baseline and use the same architecture and hyper-parameters to calculate ROELs intrinsic reward. To train ROEL we maintain list of 6 active ea-pairs ranked via their PATA-EC (Wang et al., 2020). Each active ea- ea-pair is eligible to reproduce if it's average intrinsic reward is > -0.1, the child will then be considered for addition if its average intrinsic reward over episodes on any active agent satisfies an MCC (Brant & Stanley) (017)) in the range [0.05, 100). We found that min MCC of 0.05 provided frequent updates but prevented
14

---

Under review as a conference paper at ICLR 2022
Table 1: Bipedal walker observations
Num
Observation
Min
Max
Units
Hull Angle
0
2x
Hull Angular Velocity
+00
9/8
Vel x
+1
m
Vel y
-1
+1
m/
Hip Joint Angle
-00
+00
Hip Joint Speed
+00
Knee Joint Angle
00
+00
Knee Joint Speed
00
+00
8
Leg Ground Contact Flag
FALSE
TRUE
Hip Joint Angle
30
+00
10
Hip Joint Speed
-00
+00
0/8
11
Knee Joint Angle
-00
+00
12
Knee Joint Speed
+00
0/8
Leg Ground Contact Flag
FALSE
TRUE
14-23
10x LIDAR Readings
Table 2: Bipedal walker actions
Num
Name
Min
Max
Hip
Knee
Hip 2
Knee
the worst environments, we did not investigate the effect of reducing the max criterion. The same parameters are used re-rank based on PATA-EC and conduct transfer learning across the ea-list every 8 steps of ROEL.
We train each environment in the active ea-pairs list for steps every ROEL iteration, each parent can only produce 3 offspring each reproduction step, which itself occurs every 3 steps of ROELs outer loop. Therefore, during the first 3 loops of ROEL we only have 3 children and for the rest of training the entire capacity of the ea-pairs list is full. This produces total number of 5400 training steps on all ea-pairs using 24 ROEL iterations: ((3 x 3 x 40) + (3 x 21 40)). The baseline is trained for 5600 steps, the closest value to 5400 when check-pointing every 400 iterations.
B BIPEDAL ENVIRONMENT
B. WALKER
The bipedal walker environment used is an openAl gym environment (Brockman al. 2016) with continuous observation type Box(24) and continuos action space, type Box(4). Table shows the 24 observations received by the agent at each state that we represent with our definition of ROEL. Similarly, table 2 shows the 4 actions available to the walker. Figure shows the states and actions labelled on the bipedal walker.
B.2 EXTRINSIC REWARD
Algorithm Ctrorithmmooutiie outlines the procedure used to obtain the extrinsic reward from each step of the bipedal walker environment. This is based principally the distance from the start point in the horizontal direction and motor action states, encoded as tuple, a = (Ahips Ohipz Qkneet 4kneez) reward shaping term (A) is also included to encourage the agent move forward and ensure the head is
15

---

Under review as a conference paper at ICLR 2022
y
Hull
LIDAR (10 Samples)
Hip
Leg
Knee
Leg 1
Ground Contact Flag
Figure Components of the bipedal walker
kept straight. The episode terminates reaches the finish position (miss))) or falls over and collides with the ground. All terms normalized by scale variable ensure reward behavior is generated correctly regardless of the rendering viewport size.
Algorithm 1: Bipedal Walker Reward reward function takes an input of hull æ position from the state-transition actions and reward shaping term A preesooss
Function reward(æ. a, Aprevious) is
curvent
130r.
5/2]
Calculate reward shaping
seale
A previious currenti
for a4 in a do
r (0.00035 Tmator clip((a), 1)):
Slow motor rate
end
fallen True then
+ 100
Punissh model if fallen over
end
previous +- : return T, A previous
b Update reward shaping term
end
B.3 ENVIRONMENTAL ENCODING (EE)
Our EE is inherited from the original POET bipedal implementation WWang et al. 2019) and includes the following attributes:
Ground roughness, controls how much the terrain moves up and down. ROEL sees max of 1.0 at train time and our out-of-sample evaluation uses 5.0
Pit, controls how large the pit gaps are in the environment, we use a large range at train time but the out-of-sample evaluation [0.5, 0.5] small gap is not seen during train time.
Stump, controls how large stumps made in the environment and are used in several train environments and are not used during the out-of-sample environment.
Stair, adds stairs for the agent traverse and are not used during training or out-of-sample.
The default EC used to start ROEL and train the DADS baseline sets all the EE variables zero.
16


Reviews:
[{"review_id": "fiAW3AyrPZy", "reviewer": "Reviewer_2GeS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n\n1. Relevance of problem setting - \nThe paper considers the very important setting of reward-free learning, since it is difficult to specify reward functions that align with our intended objectives and further cumbersome to provide rewards for real systems. The paper considers leveraging open-ended learning to try to learn a more diverse set of skills, since automatically generating challenging environments can enable new skills to be discovered. \n\nWeaknesses \n\n1. Experimental Evaluation  - missing evaluation on DADs environments\n\nThe proposed method can be viewed as DADs, augmented with a scheme to train on autonomously generated diverse environments (from POET). In the experiment section, the authors compare the performance of ROEL to that of DADs on held-out environment variations, and argue that ROEL is superior. However, they only use the bipedal walker which has very low action space (only 4 dimensions), and don’t test on any of the environments used in the DADs papers [2,3] which include control of much more complex agents such as the gym-ant, humanoid or the d-kitty robot. Since the skill-discovery from DADs has been shown to enable effective control in these agents, we need to evaluate if open-ended learning can scale to these agents as well. This is quite critical for general adoption of open-ended learning in unsupervised learning.  \n\n2. Experimental Evaluation - issues with current evaluation\n\nFor the only included environment (bipedal-walker), performance comparison on the out of distribution environment would be much more compelling if the authors included quantitative measures of performance such as a success metric. Instead the authors compare performance by showing visualization of 1 run for each method (Fig.2), but this could be susceptible to noise, it’s unclear if this behavior can be reliably replicated, and further from the visualizations, it’s unclear if the ROEL agent actually cross the gap successfully. \n\nThe authors do include some quantitative analysis of the prediction error of the observation trajectory from the skill network (Fig. 4), and show that DADs skill network overfits. But this is an argument for ROEL discovering more diverse skills, not whether these skills actually enable better performance on a given OOD environment, for which success metrics should be used.\n\nIn Fig.3 the authors plot returns from behavior resulting from different skill priors, and find that the return plot for ROEL is more diverse. They argue that diverse returns can be used as a heuristic to imply the corresponding skills are diverse, but I’m not sure this is necessarily true. For example, different skill priors could essentially learn to perform the same behavior but to varying degrees of proficiency, thus resulting in a diverse return plot without having skills that are really diverse. \n\n3. Novelty \n\nThe novelty of the approach is very limited, as it essentially takes the POET algorithm for open-ended learning, and replaces the external environment reward with the exact reward used in DADs. It would have been interesting to consider how the DADs skill conditioned dynamics model that is learned could be used for ranking or selecting environments/agents in the open-ended learning part.  \n", "summary_of_the_paper": "This paper proposes combining unsupervised skill-discovery with open ended learning, where the environment and agents are continually evolved together. Specifically the proposed method (ROEL) is a combination of POET[1] with the intrinsic reward for skill-discovery in DADS[2], and is evaluated on a simulated bipedal walker environment. \n", "summary_of_the_review": "While the premise of generating new environments to enable more diverse skills to be learned is intriguing, the experiments do not convincingly demonstrate this. The paper lacks evaluation on more complex environments where DADs is known to work well, and also doesn’t include quantitate performance metrics for the simpler bipedal walker considered. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dmqBd_ZMMAZ", "reviewer": "Reviewer_kQnm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- The combination of open-ended and rewardless learning is very natural and synergistic -- it moves strongly in the direction of learning without hand-crafted structures.\n- The algorithm is clearly presented and represents a paradigm with wide applicability.\n\n## Weaknesses\n- One of the main selling points of ROEL is that it is rewardless (unlike POET), which means that is should easily transfer to other environments. This is not evaluated, though I understand that the original version of POET is restricted to one environment.\n- It is difficult to see a significant qualitative difference between ROEL and DADS as illustrated in Figure 2.\n    A video would likely better serve this end.\n- The quantitative analysis is primarily indirect; it supports the hypothesis but not in a definitive way.\n- A comparison against POET would be particularly salient as it is the open-ended learning baseline.\n\n\n## Recommendation\nI recommend \"accept\", although I could be convinced either way upon further discussion.\n\n\n## Justification\nAlthough I point out a number of weaknesses with the evaluation, in particular, I believe the evaluation is an intrinsically difficult step for this type of work\nThe evaluation that is performed supports the claims of the paper, and the overall quality is still sufficient for acceptance.\nFurthermore the novelty of the algorithm and its clear presentation lay a good foundation for future work.\n\nSpecific criteria:\n- Correctness: 3\n- Technical Novelty and Significance: 3\n- Empirical Novelty and Significance: 2\n\n## Questions\n- What were the particular difficulties in combining these two algorithms/paradigms?\n\n\n## Additional Comments\nA caveat of my review is that while I am familiar with reinforcement learning, I am not up-to-date on the literature surrouding POET and DADS.\nI can judge the work on its internal consistency and how much further it develops these two approaches, but if there are other similar approaches, I might not be aware of them.\n\n- s1 p6: \"We hypothesis\" -> \"We hypothesize\"\n- s4.1 Figure 2: This sequence of frames is not altogether convincing of the point being made. A video might be necessary.\n- s4.1 p2: \"this is a good qualitative example of how ROES is able to leverage experience gained through mutation & goal-switching\" -- this claim does not seem adequately defended as I think the example only shows that ROEL might be more robust than DADS rather than specifically leveraging experience.\n- s4.1 p2: \"mutation & goal-switching\" -> \"mutation and goal-switching\"\n- s4.2 p2: \"the reward function is based on the ability of the agent to reach the end of the episode\" --\n    Does reach the end of the episode mean that the agent moves rightward or simply that it does not fall before some number of timesteps?\n- s4.2: \"in sample\" -> \"in-sample\", \"out of sample\" -> \"out-of-sample\"\n- Figure 4: Highlight the fact that the in-sample plots have a highly zoomed-in y-axis (which is hard to read).\n    I did not realize this at first and thought ROEL was somehow highly noisy compared to DADS in-sample, but in reality, the scale is too small to make a difference.\n    Maybe it would just be better to keep the in-sample and out-of-sample plots at the same scale to demonstrate that ROEL is very similar to DADS in-sample, relatively speaking.\n- s4.2: Probably beyond the scope of this paper (no expectation of implementing it), but a behavior/skill embedding space would be a great way to measure the diversity of behaviors, although this might be very difficult to implement.\n- s4.2 last paragraph: \"predictable & useful skills\" -> \"predictable and useful skills\"\n", "summary_of_the_paper": "Rewardless Open-Ended Learning (ROEL) is fundamentally a combination between two Paired Open-Ended Trailblazer (POET) and Dynamics-Aware Unsupervised Discovery of Skills (DADS).\nPOET presents a framework for \"open-ended learning\" which automatically generating progressively more difficult environments to elicit agents with novel behaviors.\nDADS introduces a method of \"unsupervised RL\" which learns a variety of skills without rewards solely based on mutual information between state transitions and skill-determining latent variable.\nThe result is an RL framework which generates environments to elicit controllable behaviors without a reward function from the environment.\n\nThe following empirical evaluations are performed:\n- Qualitative assessment of skills learned by ROEL\n- Qualitative comparison or ROEL and DADS\n- Measurement of rewards of different skills learned by ROEL and DADS\n- Measurement of state predictability of skills in different environments by ROEL and DADS\n\nThe primary contributions of the paper are:\n- Combining open-ended RL and unsupervised RL\n- ROEL yields a more robust set of policies in a bipedal-walker environment than the DADS baseline.\n", "summary_of_the_review": "ROEL is a logical and synergistic combination primarily of rewardless learning (DADS) and open-ended learning POET.\nThe algorithm has a clear applicability and a good presentation.\nThe experiments are reasonable and support the primarily claims of the paper, although the experiment measure indirect aspects of the claims as direct experimentation is intrinsically (to the claims) difficult.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KuDHmr-odBn", "reviewer": "Reviewer_Bne6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- Open ended learning and the unsupervised discovery of diverse skills is a very interesting problem and the paper suggests an extension of prior work to this intersection of approaches.  \n- The paper is well written, and presents a complicated approach clearly.\n\n\n### Concerns\n\nThe evaluations of the presented algorithm are unconvincing.  \n\n- Primarily, the baseline is an algorithm trained on a single environment.  It is not surprising that a method trained on multiple environments (co-evolved or not) will generalize better to unseen environments.  If the contribution is meant to highlight the improved performance of co-evolution over a static distribution of tasks (or some other naive multi-task method), it would be nice to see that comparison.  The differences observed in Figures 3 and 4 could reasonably be explained by single vs multiple environment training, rather than the POET-like curriculum learned through ROEL.\n\n- Second, the quantitative evaluation of unsupervised learning methods is a difficult problem in itself, but it would be nice to see some evaluation of fine-tuning to a particular “downstream” task with extrinsic reward (again, comparing to sensible baselines).\n\nWithout proper experimental comparisons, the significance and impact of the ROEL algorithm is difficult to assess.\n\n## Post rebuttal update\n\nThank you for the response.  The evaluation against DR is a step in the right direction.  However, the evaluation still needs to be more mature (see my second point above) before I would be comfortable accepting this paper.  I encourage you to keep working on it and submit to another conference.", "summary_of_the_paper": "This paper presents a method for learning skills in an open-ended, reward-free setting using coevolution of agents and environments trained to maximize the mutual information between a skill latent code and state transition.", "summary_of_the_review": "Although the method is interesting and tackles a relevant problem, the experimental evaluations are not mature enough to provide a clear indication of the method's merit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DNBjClM0b9w", "reviewer": "Reviewer_HJjY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros:\n\n1. The paper addresses an important problem: unsupervised skill discovery in (kind of) open-ended RL environments. This is exactly the kind of problem where specifying a concrete reward limit what can be achieved in terms of skill discovery. To me, the problem is real and practical.\n2. The proposed framework of skill discovery in open-ended domains is novel, although combines two existing methods with each other.\n3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed method.\n\n## Cons:\n\n1. I don't quite understand how exactly was the DADS baseline trained. The paper says it was trained \"using the original bipedal default environment\". There are two versions of the walker environment, according to its OpenAI gym implementation, namely normal (with slightly uneven terrain) and hardcore (ladders, stumps, pitfalls). Which one was it trained on? If DADS is trained on a single environment, it is not really fair to compare it with one trained on multiple environments for generalisation purposes.\n2. Have authors tried training DADS using Domain Randomization (DR), where the training environment is sampled from a parameterized distribution, i.e., randomly sampling hand-picked obstacle hyperparameters, such as stump shape, surface roughness and gap width? The authors should show that ROEL works better than DADS trained on DR, otherwise, it is hard to argue about the effectiveness of the method.\n3. I'm not convinced with the statement in Section 4.1 claiming that ROEL \"outperforms\" DADS in off-sample environments. What does \"outperform\" even mean in this particular experiment and how is it measured? All I see is a few cherry-picked environments where some skills of ROEL were able to pass through an obstacle where some skills of DADS weren't able to do this. First, I don't understand how the skills are selected in these experiments. How is z chosen? Have you tried all z's? Have you tried more example environments? Please elaborate on exactly what is going on here.\n4. The structure of the paper can also be improved. Particularly, it is a bit unusual to see Implementation details (Section 3.3) in what seems like the Method (Section 3). The authors could consider adding an Experimental Setting section (after section 3) where such details will be described. Also, various future directions for work are described throughout the paper. Moving all those ideas to the Conclusion would improve the readability of the paper.\n\n## Minor issues\n\n- In PAIRED, a separate adversary generates the environments for the protagonist and antagonist. The antagonist doesn't do this.\n\n## Additional issues\n\nThe issues below did not influence my score, but please address them in the updated version of the manuscript.\n\n### Citations format:\n- There are two basic citations commands in the `natbib` package\n    - `\\citet` for **textual** citations. This prints *Hinton et al. (2006)*\n    - `\\citep` for **parenthetical** citations. This prints *(Hinton et al.,* 2006*).*\n- In the entire paper, authors make use of the `\\citet` command (or generic `\\cite` which can be problematic, see [natbib documentation](https://ctan.org/pkg/natbib), page 6). However, the majority of them should actually use the `\\citep` command. According to the [ICLR 2022 formatting instructions](https://github.com/ICLR/Master-Template/raw/master/archive/iclr2022.zip):\n    - \"Citations within the text should be based on the natbib package and include the authors’ last names and year (with the “et al.” construct for more than two authors). When the authors or the publication are included in the sentence, the citation should not be in parenthesis using `\\citet{}` (as in “See Hinton et al. (2006) for more information.”). Otherwise, the citation should be in parenthesis using `\\citep{}` (as in “Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007).”). \"\n\n### Academic language and abbreviations:\n- Please DO NOT use \"&\" instead of \"and\", \"w/\" instead of \"with\", etc, throughout the paper\n\n### Punctuation errors:\n- There are large number punctuation mistakes in sentences all over the paper. Especially, commas are very often missing. Please fix these.\n\n# Post-rebuttal update\n\nI thank the authors for their response and improving the manuscript based on my suggestions. Specially, I appreciate the authors running DADS on domain randomised version of the bipedal walker environment and making cosmetic changes to the paper.\n\nHowever, after taking a look at the current version of the paper, I can't yet recommend that it is accepted in its current form. It seems like the qualitative and some quantitative analysis is still compared against DADS on the normal version of the bipedal environment. The performance gains against this baseline are expected and the comparison is unfair, as far as I'm concerned.\n\nI therefore recommend that the authors continue improving the paper and submit it for another review process at one of the upcoming conferences.", "summary_of_the_paper": "In this work, the authors a method that lies in the intersection of two subfields of RL, namely open-endedness and unsupervised skill discovery. Specifically, they introduce a method called Rewardless Open-Ended Learning (ROEL) which extends a recent method called POET to perform skill discovery in a reward-free setting rather than traditional supervised RL. To this end, they use mutual-information-based skill discovery techniques which have recently become a popular approach for unsupervised RL. The authors empirically demonstrate that ROELD is able to learn identifiable skills in bipedal walker environment.", "summary_of_the_review": "While I think the paper addresses an important problem, I have major concerns about the claims the authors make given the experiments they have. Specifically, I've raised questions on the choice of baseline agents and comparison with them. Furthermore, the writing can be improved significantly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and active participation in the discussions. This papers is borderline with the majority of three reviewers leaning towards rejection and one leaning towards acceptance. All four reviewers unanimously agree on the empirical validation being a major weakness of this paper with reviewer kQnm putting less emphasis on this shortcoming. Overall, I side with reviewers Bne6, 2GeS and HJjY, and believe the paper needs to a more thorough set of experiments. I therefore recommend rejection.
Decision: Reject