Augmentative Topology Agents For Open-Ended Learning
Authors: Muhammad Umair Nasir
Michael Beukman
Steven James
Christopher Wesley Cleghorn
Venue: Submitted to ICLR 2023 (2023)
Keywords: Open-Ended Learning
NeuroEvolution

Abstract: In this work, we tackle the problem of Open-Ended Learning by a method that simultaneously evolves agents and increasingly challenging environments. Unlike previous open-ended approaches that optimize agents using a fixed neural network topology, we hypothesize that generalization can be improved by allowing agents' controllers to become more complex as they encounter more difficult environments.  Our method, Augmentative Topology EPOET (ATEP), extends the Enhanced Paired Open-Ended Trailblazer (EPOET) algorithm by allowing agents to evolve their own neural network structures over time, adding complexity and capacity as necessary. Empirical results demonstrate that ATEP results in general agents capable of solving more environments than a fixed-topology baseline. We also investigate mechanisms for transferring agents between environments and find that a species-based approach further improves the performance and generalization of agents.

Under review as a conference paper at ICLR 2023
AUGMENTATIVE TOPOLOGY AGENTS FOR OPEN-ENDED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we tackle the problem of open-ended learning by introducing a method that simultaneously evolves agents and increasingly challenging environ- ments. Unlike previous open-ended approaches that optimize agents using a fixed neural network topology, we hypothesize that generalization can be improved by allowing agents' controllers to become more complex as they encounter more dif- ficult environments. Our method, Augmentative Topology EPOET (ATEP), ex- tends the Enhanced Paired Open-Ended Trailblazer (EPOET) algorithm by allow- ing agents to evolve their own neural network structures over time, adding com- plexity and capacity necessary. Empirical results demonstrate that ATEP results in general agents capable of solving more environments than a fixed-topology baseline. We also investigate mechanisms for transferring agents between envi- ronments and find that a species- based approach further improves the performance and generalization of agents.
INTRODUCTION
Machine learning has successfully been used to solve numerous problems, such as classifying im- ages (Krizhevsky et al. (0122) writing news articles (Radford al. 2019 Schick & Schütze 2021) or solving games like Atari (Mnih et al., (215)) chess (Silver et al., 2018). While impressive, these approaches still largely follow a traditional paradigm where a human specifies task that is subsequently solved by the agent. In most cases, this is the end of the agent's '' learning-one it can solve the required task, no further progression takes place.
Through the motivation by the fact that humans have always learnt and innovated in an open-ended manner, Open-ended learning research field emerged (Stanley et al. [017). For instance, humans did not invent microwaves to heat food, but to study radars. Vacuum tubes and electricity was invented for very different reason but we stumbled upon computers through them (Stanley) 2019).
In perspective of agent, Open-ended learning is a research field that rather than converge specific goal, the aim is to obtain an increasingly growing set of diverse and interesting behaviors (Stanley) et al. (017). One approach is allow both the agents, well as the environments, change, evolve and improve over time (Brant & Stanley 2017 Waag et al. 2019). This has the potential to discover large collection of useful and reusable skills (Qessy Richardson 2021). well interesting and novel environments (Gisslén et al. 2021). Open-ended learning is also much more promising way to obtain truly general agents than the traditional single task-oriented paradigm (Team et al., 2021).
The concept of open-ended evolution has been a part of artificial life (ALife) research for decades now, spawning numerous artificial worlds (Rayl 1991 Ofria & Wilke 2004 Spector et al.) 2007 Yaeger Sporns 2006 (Soos Stanley 2014). These worlds consist of agents with various goals, such as survival. predation, reproduction. Recently, open-ended algorithms have received newed interest (Stanley) 2019), with Stanley et al. (2017) proposing the paradigm as a path towards the goal of leve artificial intelligence.
major breakthrough in open-ended evolution was that of NeuroEvolution of Augmenting Topolo- capable of efficiently solving complex
gies (NEAT) (Stanley & Miikkulainen 2002), which inforcement learning tasks. Its key idea was to allow the structure of the network to evolve alongside the weights, starting with simple network and adding complexity the need arises. This inspired

---

Under review as a conference paper at ICLR 2023
future research about open-endedly evolving networks indefinitely (Soros & Stanley 2014). Specif-
ically, novelty search (Lchman et al.] 2008). used the idea of novelty to drive evolution, instead of
traditional objective-based techniques. This in turn led to the emergence of quality diversity (QD)
algorithms (Lehman & Stanley 2011 I Mouret & Clune 2013] Hcoffet et al. 2019 Nilsson & Cully)
2021], which are based combining novelty with an objective sense of progress, where the goal is to obtain a collection of diverse and high-performing individuals.
While QD has successfully been used in numerous domains, such as robotic locomotion (Cully et al. 2013 Mouret & Clune| 2013 Tarapore et al., 2016)) video game playing (Ecoffet et al,, 2019 and procedural content generation (Khalifa et al. 2018 Earle al., 2022). still is not completely open-ended, where completely open-ended means to run indefinitely and create novel artifacts. One reason for this that the search space for phenotypical behavior characteristics (or behavioral descriptors) remains fixed (Mouret & Clune 2015). A second reason is that in many cases, the environment remains fixed, which limits the open-endedness of the algorithm (Wang) (et al. 2019). A way to circumvent this is to co-evolve problems and solutions, as is done by Minimal Criterion Coevolution (MCC) (Brant & Stanley 2017). This co-evolutionary pressure allowed more complex mazes to develop, and better agents to solve them emerged, giving rise to an open-ended process.
However, MCC had some limits; for instance, it only allows new problems if they solvable by individuals in the current population. This leads to only slight increases in difficulty, and complexity which only arises randomly. Taking this into account, Paired Open-ended Trailblazer (POET) (Wang et al. 2019) builds upon MCC, but instead allows the existence of unsolvable en- vironments, if was likely that some individuals could quickly learn to solve these environments. POET further innovates by transferring agents between different environments, to increase the like- lihood of solving hard problems. While POET obtained state of the art results, its diversity slows down as it evolves for longer. Enhanced POET (Wang et al. 2020) adds improved algorithmic com- ponents to the base POET method, resulting in superior performance and less stagnation. Enhanced POET, however, uses agents with fixed topology neural network controllers. While this approach works well for simple environments, it has an eventual limit on the complexity of tasks it can solve: at some point of complexity, the fixed topology agents may not have sufficient capacity to solve the environments.
To address this issue, we propose Augmentative Topology Enhanced POET (ATEP), which uses NEAT to evolve agents with variable, and potentially unbounded, network topologies. We argue that fixed-topology agents will cease solve environments after certain level of complexity and empirically show that ATEP outperforms Enhanced POET (EPOET) in a standard benchmark do- main. Finally, we find that using NEAT results in improved exploration and better generalization compared to Enhanced POET.
RELATED WORK
POET (Wang et al. 2019) and EPOET (Wang et al. 2020) are the founding algorithms of the field of open-ended learning, building upon prior approaches such MCC (Brant & Stanley 2017). This has led to an explosion of new use cases such as PINSKY (Dharna et al. 2020 2022). which uses POET on 2D Atari games. This approach extends POET to generate 2D Atari video game levels alongside agents that solve these levels. Quessy & Richardson 2021) uses unsupervised skill discovery (Campos et al. 2020; Eysenbach et al. | to discover a large repertoire of useful skills. Meier& & Mujika (2022) also investigate unsupervised skill discovery through reward functions learned by neural networks. Other uses of POET include the work by Zhou & Vanschoren| (2022). who obtain diverse skills in a 3D locomotion task. POET has also been shown to aid evolving robot morphologies (Stensby et al., 2021) and avoiding premature convergence which is often the result when using handcrafted curricula. Norstein et al. (2022) use MAP-Elites (Mouret & Clune (0155) to open-endedll create a structu red repertoire of various terrains and virtual creatures. Hejna III et al. (2021) introduces TAME that evolves morphologies without tasks, potentially creating a system of open-ended morphology evolution.
/hhamaa et al. 2020) in the context of POET
Adversarial approaches are commonly adopted when developing open-ended algorithms. [Denni jet al.] (2020) propose PAIRED, a learning algorithm where an adversary would produce an envi- ronment based on the difference between the performance of an antagonist and protagonist agent.
2

---

Under review as a conference paper at ICLR 2023
Domain randomization (Sadeghi & Levine 2016), prioritized level replay (Jiang et al., [022] 1) and Adversarially Compounding Complexity by Editing Levels (ACCEL) (Parker-Holder et al., 2022) adopt a similar adversarial approach, where teacher agents produce environments and student agents solve them.
Several domains and benchmarks have been proposed with the aim of encouraging research into open-ended, general agents. [Team et al.] (202T) introduce the XLand environment, where a sin- gle agent is trained on 700k 3D games, including single and multi-agent games, resulting zero- shot generalization on holdout test environments, [Battht et al. (2022) introduced an autoencoder (Kingma & Welling, 2013) and CPPN-NEAT based open-ended evolutionary algorithm evolve Minecraft (Duncan 2011; Cipollone et al., (214)) buildings. They showed how differences in the training of the autoencoders can affect the evolution and generated structures. Fan et al. (2022) cre- ate a Minecraft- environment, MineDojo, which has numerous open-ended tasks. They also introduced MineCLIP as an effective language-conditioned reward function that plays the role of an automatic metric for generation tasks. Gan al. (2021) introduce the Open-ended Physics En- vironment (OPEn) to test learning representations, and tested many RL-based agents. Their results indicate that agents that make use of unsupervised contrastive representation learning, and impact- driven learning for exploration, achieve the best result.
NEUROEVOLUTION OF AUGMENTING TOPOLOGIES
We leverage NeuroEvolution of Augmenting Topologies (NEAT) to evolve the structure of an agent's controller. NEAT starts with population of simple neural networks (NNs), where the input neu- rons are directly connected to the output neurons without any hidden layers. Crossover is performed between two parents and the resulting children are mutated by adding connections and nodes, or perturbing weights. In this way, the NN will gradually be complexified. Crossover and mutations are illustrated [Section E.2 of [Appendix E One of the major problem overcome is the Per- mutations or Competing Convention Problem Radcliffe (1993); Montana et al. (199)). Competing conventions describes the case in which the crossover networks that represent the same solution but are encoded differently (e.g. different ordering of neurons) can lead to a loss of information and significantly worse child. NEAT addresses this by introducing a method keep track of the historic origin of gene by using the innovation number. Using this innovation number, identical genes from two parents can be aligned, while genes that only occur in one (denoted excess or disjoint genes depending numbers) can be inherited from the fitter parent. Finally, NEAT introduces speciation (Mahfoud) 1995], where individuals with similar topologies are grouped together, and share fitness. This protects innovation and ensures diversity. This speciation calculation is shown by [Equation ijin (Ap- pendix D. In this equation, C1 Cg and c3 coefficients that indicate the importance of each factor while N is the number of genes in the larger genome. E and D denote the number of excess and disjoint genes respectively. W is the average weight difference of similar genes. then, indicates how close two genomes are; if is less than some threshold, then the two genomes belong to the same species.
whether it occurs within outside the range of the other parent's innovation
NEAT has demonstrated superior performance when compared to fixed topology approaches, and has been used numerous subsequent research works to great success (Stanley 2007 Lehman
et al. 2008 (Stanlyy et al. 2009. |Schrum et al. 20200 Clei & Bellec 2022)
4 ENHANCED POET
Since our method is heavily based on EPOET, we briefly describe this method, as well as the orig- inal POET algorithm. POET focuses on evolving pairs of agents and environments in an attempt to create specialist agents that solve particular environments. POET uses modified version of 2D Bipedal Walker Hardcore environment from OpenAl Gym (Brockman et al., 2016) as a benchmark. The first environment is flat surface, and as evolution progresses, the environments become harder with the addition of more obstacles. POET also transfers agents across environments, which prevent stagnation and leverage experience gained solving another. An Environment- Aerronmettt-genttt (EA) pair is eligible to reproduce when the agent crosses a preset reward threshold on this environment. The next generation of environments is formed by mu-
environment as step towards
3

---

Under review as a conference paper at ICLR 2023
tating the current population and selecting only those environments that neither too easy too hard. Finally, environments are ranked by novelty, and only the most novel children pass through to the next generation. More information about the hyperparameters of POET is listed in Appendix C
EPOET improves upon POET by adding in two algorithmic improvements: (1) a general method of evaluating the novelty of challenges and (2) improved approach to deciding when agents should transfer to new environments. In the original POET, the way to evaluate novelty was to compare the environment characterization (EC) of different environments. This is obtained by using some fixed, domain-specific static features, such as the roughness of the terrain. This inherently limits the exploration of the algorithm, as it is restricted to explore within these preset confines. Enhanced POET introduces an improved EC, Performance of All Transferred Agents EC (PATA-EC), which based on the performance of different agents in the environment. Secondly, the original transfer mechanism in POET was generally inefficient, as it increased the required computation (as each agent needed to be fine-tuned), and resulted in subpar transfers as was too easy to qualify for transfer. Enhanced POET makes this process more strict, only transferring very promising agents.
Enhanced POET also improves upon the environmental encoding used in the original algorithm, which was fixed and thus had limited number of unique and diverse environments it could repre- sent. The solution to this problem is to use a more expressive encoding in the form of compositional pattern producing networks (CPPNs) (Stanley 2007). A CPPN is specific neural network, which can take in x; coordinates and produce a specific pattern when evaluated across an entire region. These CPPNs are evolved using the NEAT (Stanley & Miikkulainen 2002) algorithm, which in- creases the complexity of the environments as evolution progresses.
Lastly, the authors introduce Accumulated Number of Novel Environments Created and Solved (AN- NECS), a metric for open-ended learning that, intuitively, describes the amount of new content that generated by the algorithm. ANNECS counts the number of environments that satisfy two con- straints: (1) it must neither be too easy nor too hard and (2) it must be eventually solved by some agents in the future. Thus, if the ANNECS metric increases as time goes on, it indicates that the algorithm is continually producing novel and meaningful environments.
5 OPEN-ENDEDLY EVOLVING THE TOPOLOGY OF AGENTS
Many of the approaches introduced in prior work have been implemented using fixed topology ap- proach in conjunction with optimizers such as evolutionary strategies (ES) (Salimans et al. 2017). V-MPO (Song et al.) 2019) and Proximal Policy Optimization (Schulman et al. (017), which moti- vates us to explore NEAT and the benefits it brings to the open-ended learning framework. We will describe the overall approach in Section 5.1
5.1 AUGMENTATIVE TOPOLOGY ENHANCED POET (ATEP)
In this section, discuss the basic building blocks of our algorithm and the different variants experimented with. ATEP combines EPOET with NEAT to allow the agents' network topologies to evolve. This means that the algorithmic steps very similar to EPOET, and the main differences are (1) the optimizer used: we use NEAT to optimize the variable-topology agents whereas EPOET used Evolution Strategies to optimize fixed-topology agents; and (2) the transfer mechanism, which will be discussed later in this section. The detailed flow of ATEP is described in Figure 1
We first use NEAT evolve population for each environment. The valid environments (those that pass the minimal criterion) then reproduce to create a new generation of (slightly harder) environ- ments. We then take the environment that is the most novel (as measured by the Euclidean distance between the PATA-EC scores), and create a new environment-gge these environments is then evaluated, and if there are valid transfers available, we can move agents between environments. In EPOET, transfer is performed as follows: we compare the fitness of the candidate agent to the fitness of the target agent, over the previous 5 generations. If the candidate's fitness is greater than all previous 5 fitness scores, we fine-tune it on the target environment and both of these checks
pair. The transfer eligibility of
again compare against the best fitness from the previous 5 generations. passed we transfer the candidate and replace the target. For ATEP, we experiment with two different transfer mechanisms, the first being inspired by the approach used by EPOET, denoted as Fitness-Based Transfer ATEP (FBT-ATEP). In this case, we compare the best genome in the candi-
4

---

Under review as a conference paper at ICLR 2023
HEAT
Feert oseet
Chams
Coap CARBO
Twents
A flowchart demonstrating the flow of the ATEP framework, with blocks in green being where ATEP differs from PORT. For both EPOET and ATEP, each environment is associated with an agent, represented by an ES population for EPOET and NEAT population for ATEP. Mechanisms
Figure 1:
used in NEAT are descrtibed in Section 3 while the hyperparameters are in Table 3 of Appendix C PATA-EC and ANNECS are described in Section 4 with other components used in EPOET. [Sec-] tion 5.1 describes the transfer mechanisms and/Appendix D iillustrates pseudocodes for the transfer mechanisms used in ATEP. The environment images used in the chart were created by ATEP.
date population to the best genome from the target population. We then perform the same checks as EPOET, and if both are passed, replace the entire target population with the candidate.
For the second transfer mechanism, we use the speciation inherent in NEAT to influence transfer. Specifically, we check if the best genome in the candidate population is within ô threshold (using the speciation calculation in EEquation 1) of any target environment's 's best genome. If this is the case, we transfer the candidate species and replace the target species with it. This approach. called Species-Basee Transfer ATEP (SBT-ATEP), skips the step of comparing fitness scores and has its own advantages which we discuss in the next section. Finally, we also consider random transfer (RT-ATEP) and no transfer (NT-ATEP) to investigate whether the transfer mechanisms have large impact on the results.
5.2 EXPERIMENTAL SETUP
Now we describe the experimental setup for ATEP, its variants, and our baselines. In ATEP, we use NEAT as the algorithm evolve the topology and weights To reduce the computational load, we change one aspect of the original EPOET paper, reducing the number of active environment- agent pairs from 40 to 20. We make this change for both EPOET and ATEP, so the results are still comparable.
We set up two baselines: the first, denoted EPOET40x40, EPOET with the original controller consisting of two hidden layers with 40 nodes each. The second baseline, EPOET20x20, is a con- troller with two layers of 20 nodes each. Having a baseline that has lesser nodes than the original EPOET's controller allows to gather insights having lesser number of nodes, which we then compare with larger number of nodes, which is the original EPOET's controller. This also allows us to evaluate the effect of having small fixed topology, comparatively larger fixed topology, and variable topology. Furthermore, this allows us to confirm our hypothesis that fixed topology agents will stagnate after certain level of complexity. For further details the controllers, please refer to Appendix C
RESULTS AND DISCUSSION
In this section, we discuss and analyze our results. We break the results into 3 different categories: Open-Ene -een-Endddness, nodes complexity exploration and generalization ability. All results are gathered based on 4 seeds due the expensive computational load, with each algorithm requiring approxi-
"ypperarraeeter settings for the various methods listed in Appendix C
5

---

Under review as a conference paper at ICLR 2023
Algortimm
100
80
POET4004
RT-ATEP
ANNDS
NT.ATEP
18k
lteraions
Figure 2: Accumulated Number of Novel Environments Created and Solved (ANNECS). The results gathered on seed. Solid line represents mean and the error bars represent standard daviation across the seeds. To save compute, we stop the NT- and RT-ATEP experiments carly, as it is clear that they perform poorly.
mately 50, 000 to 200. 000 CPU hours for single run. EPOET20x20 required the least amount of computation, while SBT-ATEP required the most. Each algorithm was run in parallel on a cluster consisting of 264 Intel Xeon cores, with the runtime ranging between 10 and 30 days. Each algo- rithm had maximum of 10 function evaluations per run. One function evaluation means one run of an episode that could be 2000 timesteps long.
The change from 20 40 active anviromment-agent (EA) pairs limits the agent have minimum number of optimization steps to 3000 from 6000, which was the case in the original EPOET research. This number is calculated as we try make an EA pair every 150 iterations and we have active EA pairs, thus it becomes possible that if an EA pair is created every 150 iterations and active EA pair threshold is reached, we archive the oldest. Thus, giving less time for the agent to optimise.
6.1 OPEN-ENDEDNESS
As mentioned, Wang al. 2020) introduce the ANNECS metric to capture the open-endedness of an algorithm; take it most important score judge which algorithm performs better complex environments. Refer to Apppendxx F for more explanation on ANNECS.
Figure shows the ANNECS score function of training time. We see that there is significant difference between EPOET20x20 and FBT- and SBT-ATEP, indicating that the small network results in solving fewer environments. EPOET40x40 performs substantially better than EPOET20x20, and is competitive with ATEP early on during training. The rate of increase in ANNECS, however, does decrease after about 13k iterations, whereas ATEP increases ates our hypothesis that fixed topology agents will start stagnating at some level of environment complexity, due capacity issues. While we can improve the results by increasing the size of the network, that will merely delay the onset of performance plateau.
consistent rate. This substanti-
FBT-ATEP outperforms EPOET40X40, although it also slows down slightly as time progresses. This is due to replacing the entire target population with the transferred population, which may eliminate all useful skills learned by the target population. SBT-ATEP, on the other hand, only replaces single species that is close to the candidate species, leaving the rest of the population intact. We also find that SBT-ATEP has negligible performance plateaus during the run of our experiments though it performed similarly to FBT-ATEP and EPOET40x40
in solving environments and, carly on during training, starts to outperform these the second half the experiment. This, will show later, is partly due to SBT-ATEP exploring more actions. We further note that the variations using no transfer (NT-ATEP) or random transfer (RT-ATEP) perform poorly, indicating that intelligent transfer mechanisms necessary.
Although ATEP outperforms EPOET, more computationally expensive, as measured by the number of function evaluations. One function evaluation means individual being evaluated

---

Under review as a conference paper at ICLR 2023
environment. SBT-ATEP has the most function evaluations since species transfers from one population to another, becomes highly probable that it can transfer in the opposite direction because they may now be within the Othreshold range. This increases the population size, resulting in more function evaluations. The tradeoff here is of function evaluations to performance, which is justified as the performance confirms our hypothesis. |Figure djdsplays the total number of function evaluations.
302
252
Alyortmm PST-ATEP
EXCETENO
Nisden
Figure 3: Cumulative of the number of function evaluations, with the Y-axis converted to log- scale. While ATEP requires signif- icantly more function evaluations, we find that its total wall-time is only 3 times more than EPOET, as the neural networks are generally smaller and each evaluation does not take as long.
Figure 4: Mapping of fitness to nodes. We plot values every 1000 iterations, starting iteration num- ber 150. Each dot represents a specific iteration, as well the mean fitness over all environments and the mean number of nodes in the population. Distribution on top represents distribution of nodes while distribution the left repre- sents fitness distribution.
6.2 NODES COMPLEXITY EXPLORATION
We have now shown that SBT-ATEP outperforms all of the other tested methods based on the AN- NECS score. We also find that it generally a smaller neural network with fewer nodes than the other algorithms. Figure 4] shows the number of nodes and corresponding fitness value for each algorithm. We can see that SBT-ATEP generally has high fitness, but fewer nodes than the other approaches. This is echoed in Figure 5c, where SBT-ATEP has the least number of nodes for most of the experiment, although it gradually adds nodes and complexity. FBT-ATEP, on the other hand, adds nodes very rapidly. This again indicates that the transfer mechanism in EPOET is critical. Refer to Appppdiii for evolved networks from SBT-ATEP to illustrate emerging network complexity.
Inspired by this, we further look into simple Fitness to Nodes ratio (FNR) metric, shown in Fig- ure 5a and find that SBT-ATEP outperforms all other algorithms on this metric for the majority of the run. This indicates that SBT-ATEP outperforms all other algorithms per-environment basis, while using fewer nodes. This leads us to believe that a better-curated transfer mechanism, based on SBT-ATEP, will sustain the FNR for longer runs.
Furthermore, in Figure 5b we calculate an ANNECS to Nodes ratio (ANR) metric with the intent to observe the role of nodes in the Open-Endedness of the agents, i.e. to have the ability to complexify over time. We observe that SBT-ATEP performs significantly better than the other models. FBT- ATEP has the lowest ANR, it adds nodes much faster than the rate of increase in ANNECS.

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(C)
Figure 5: Analysis with respect to the number of nodes. Figure (a) shows FNR along iterations, (b)
shows ANR along iterations, (c) shows the addition of nodes along iterations
6.3 GENERALIZATION ABILITY
We next evaluate the generalization ability of our open-ended agents, as prior work (Team et al. [02T] has shown that these agents have the potential generalize to unseen environments. To concretely test this, we first take the 20 latest environments from each method. For each environ- ment, we take the latest agent that could solve this environment from the method under considera- tion. Each of these agents is now evaluated on the selected environments from the other methods (60 in total). We perform 30 runs per environment-agent pair and calculate the mean and maximum of the rewards. We split the results into three categories: environments with fitness scores above 300, between 200 and 300, and below 200. Scores below 200 indicate that the environment has not been solved by the agent. |Figure 6d] shows the performance of each method when evaluated on the 60 other environments. We observe that SBT-ATEP outperforms all other models, with only 10% of the environments remaining unsolved.
Secondly, we test the generalization capabilities agents on all the environments created by their own algorithm. We exclude EPOET20x20 as it fails to solve 80 environments in the whole run. We take into account 80 environments that were solved by the model itself and observe how each agent performs all of them. Figures 6a 6b bc show the results. Here, early-stage agents perform worse and late-stage agents are shown to have generalization abilities on previously unseen landscapes. The transfer mechanism plays key role in this generalization, it exposes agents to more environments. Despite not having seen all environments, late-stage agents generalize much better. SBT-ATEP generalizes the best, with the lowest proportion of unsolved environments, in contrast to the lower-performing EPOET40x40 and FBT-ATEP.
Finally, we briefly investigate potential reasons why SBT-ATEP outperforms FBT-ATEP. We find that SBT-ATEP explores more actions, as it only transfers single species instead of replacing the target population
is done by FBT-ATEP. This allows the new species complement the actions that were already explored by the existing population. (Appnddxx B] shows the action distribution of each action for SBT-ATEP, FBT-ATEP and EPOET40X40.
7 CONCLUSION AND FUTURE WORK
This work investigated the effect of having an Augmentative topology agent an open-ended learn- ing algorithm's performance. We hypothesized that using a fixed topology would result in agents that exhibit delays in solving an environment after a certain point in environment complexity. We showed that this is indeed the case, and addressed this limitation by introducing ATEP, which allows the network topology of the agents change and add complexity as necessary. We demonstrated that this approach outperforms existing methods in terms of the ANNECS score and generalization ability, while using fewer nodes in the neural networks. Our approach, however, does require more function evaluations than competing approaches. Thus, promising future direction would be to use NEAT with Novelty Search (Lehman et al. 2008) or Surprise Search which tends to converge faster than simple NEAT (Gravina et al. 2016). QD algorithms may also be worthwhile explore in the context of open-ended learning as they have the ability to produce a population of high- performing and diverse individuals (Bhatt et al. 2022). Exploring Neurogenesis (Maile et al.) 2022 Draclos jet al. 2017), where neurons are added to a single neural network based on various external triggers, could also be promising direction. To reduce computational load, it would also be promising to
8

---

Under review as a conference paper at ICLR 2023
Finess
Finess
Above 500
Above 360
20010-300
230 10 300
EnWorrmment
60
Boron
Envirrmments
Biloo
49
Agents
Agents
(a) EPOET40x40
(b) FBT-ATEP
Finess
Fiteess
Above 500
Allowo 600
200
200 200-300
EnWrrrments
Boion 200
Envrormentts
Below 203
40
281
FST
EPOE 140440 EPOET20022
Agents
Algorthm
(c) SBT-ATEP
(d)
Figure 6: Figures showing generalization capabilities. Figures (a), (b) and (c) show agents of 80 solved environments being tested on all 80 environments, for EPOET40x40, FBT-ATEP and SBT- ATEP respectively. Note that EPOET20x20 does not take part in this test failed to produce 80 environments in the run. Figure (d) shows each algorithm being tested on the 20 latest environments created by all other algorithms, i.e., each algorithm is evaluated on environments. The Y-axis shows the percentage of environments in each category. Each test is conducted for 30 runs and the mean scores are taken.
look into developing single-population open-ended learning methods without losing the exploration abilitics of EPOET.
Furthermore, have opened up possible future research into transfer mechanisms. We compared simple approaches such as FBT and SBT, but more advanced approaches could yield further perfor- mance improvements. For instance, we could combine both FBT and SBT in weighted manner, or transfer only a certain percentage of a species or population. Finally, this work provides a starting point, like EPOET itself, into open- open-ended learning with augmentative topology agents. We there- fore used the modified version of 2D BipedalWalker as our benchmark. Future work should compare ATEP with standard EPOET different and more complex environments. Ultimately, we hope that this new approach furthers research into open-ended algorithms that do not slow down over time, and can keep up with an ever-changing environment.
REPRODUCIBILITY AND ETHICAL STATEMENT
For reproducibility, we have provided a GitHub repo where users can follow instructions to repro- duce the experiments. We have also provided detailed hyperparameter tuning Appendix C and pseudocodes in Appendix D for user reference. ATEP is an Open-Ended Learning algorithm that has stochastic elements, similar to many other machine learning algorithms. critical for users to perform standard evaluations as the user would do for other machine learning algorithms. A full run of ATEP may be computationally expensive and will take approximately 50, 000 to 200, 000 CPU hours.
TThe GitHub repo will be provided after acceptance, but zip file containing the source code will be sub- mitted alongside this paper.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Matthew Barthet, Antonios Liapis, and Georgios N Yannakakis. Open-ended evolution for Minecraft building generation. IEEE Transactions on Games, 2022.
Varun Bhatt, Bryon Tjanaka, Matthew C. Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. CoRR, 0ttp/2206.041999 2022. doi: 10.48550aaXiw.22060040999 URL /ttp::::///o...//11... 11tp://...444505. 2206. 221.. 04199
Jonathan C Brant and Kenneth O Stanley. Minimal criterion coevolution: approach to open- ended search. Proceedings of the Genetic and Evolutionary Computation Conference, pp- 67-74, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540 2016.
Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Proceed- ings of the 37th International Conference on Machine Learning, volume 119, pp. 1317-1327, 2020.
Maria Cipollone, Catherine Schifter, and Rick A Moffat. Minecraft as a creative tool: A case study. International Journal of Stame-Based Learning (IJGBL), 4(2):1-14, 2014.
Maximilien Le Clei and Pieree Bellec. Neuroevolution of recurrent architectures control tasks. In Jonathan E. Fieldsend and Markus Wagner (eds.), Genetic and Evolutionary Computation Con- ference, Companion Volume, pp. 651-654, 2022.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Ban Sran-Baptiste Mouret. Robots that can adapt like animals. Nature, 521((553):503-577, 2015.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in Neural Information Processing Systems, 33:13049-13061, 2020.
Aaron Dharna, Julian Togelius, and Lisa B Soros. Co-generation of game levels and game-playing agents. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pp. 203-209, 2020.
Aaron Dharna, Amy K Hoover, Julian Togelius, and Lisa Soros. Transfer dynamics in emergent cvolutionary curricula. IEEE Transactions on Games, 2022.
Timothy J Draelos, Nadine E Miner, Christopher C Lamb, Jonathan A Cox, Craig M Vineyard, Kristofor D Carlson, William M Severa, Conrad D James, and James B Aimone. Neurogenesis deep learning: Extending deep networks to accommodate new classes. In 20/7 International Joint Conference on Neural Networks (IJCNN), pp. 526-533. IEEE, 2017.
Sean C Duncan. Minecraft, beyond construction and survival. 2011.
Sam Earle, Justin Snider, Matthew Fontaine, Stefanos Nikolaidis, and Julian Togelius. Illumi nating diverse neural cellular automata for level generation. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 68-76, 2022.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploraie exploratiooonnn problems. arXiv preprint arXiv:1901.10995, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian lbarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representa- tions, 2019.
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint 2rXiv:2206.08853 2022.
10

---

Under review as a conference paper at ICLR 2023
Chuang Gan, Abhishek Bhandwaldar, Antonio Torralba, Joshua B Tenenbaum, and Phillip Isola. Open: An open- open-ended physics environment for learning without a task. In 2021 JEEE/RSJ Inter- national Conference Intelligent Robots and Systems (IROS), pp.
IEEE, 2021.
Linus Gisslén, Andy Eakins, Camilo Gordillo, Joakim Bergdahl, and Konrad Tollmar. Adversarial reinforcement learning for procedural content generation. In 2021 IEEE Conference on Games, pp. 1-8. IEEE, 2021.
Daniele Gravina, Antonios Liapis, and Georgios Yannakakis. Surprise search: Beyond objectives and novelty. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pp- 677-684, 2016.
Donald Hejna III, Pieter Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. arXiv preprint arXiv:2 102. 13100, 2021.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940-4950, 2021.
Ahmed Khalifa, Scott Lee, Andy Nealen, and Julian Togelius, Talakat: Bullet hell generation through constrained map-elites. In Proceedings of The Genetic and Evolutionary Computation
Conference, pp. 1047-1054, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1312.61 14, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1 1106-111 114, 2012.
Joel Lehman and Kenneth Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 211-218, 2011.
Joel Lehman, Kenneth O Stanley, et al. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp. 329-336, 2008
Samir W Mahfoud. Niching methods for genetic algorithms. PhD thesis, University of Illinois BirbanaChhappajy 1995.
Kaitlin Maile, Emmanuel Rachelson, Hervé Luga, and Dennis Wilson. When, where, and how to add new neurons ANNs. arXiv preprint arXiv:2202.08533 2022.
Robert Meier and Asier Mujika. Open-ended reinforcement learning with neural reward functions. arXiv preprint arXiv:2202.08266 082666 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5177540):559-533 533, 2015.
David J Montana, Lawrence Davis, et al. Training feedforward neural networks using genetic al- gorithms. In International Joint Conference Artificial Intelligence, volume 89, pp. 762-767, 1989.
Jean- CaantBapistte Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv: 1504, 04909, 2015.
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 866 875, 2021.
Emma Stensby Norstein, Kai Olav Ellefsen, and Kyrre Glette. Open-ended search for environments and adapted agents using map-elites. In International Conference the Applications of Evolu- tionary Computation (Part of EvoStar), pp. 651-666. Springer, 2022.
Charles Ofria and Claus Wilke. Avida: A software platform for research in computational evolu- tionary biology. Artificial life, 10(2):191-229, 2004.

---

Under review as a conference paper at ICLR 2023
Jack Porder-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint wrXiv:2233.01300 2022.
Alexander Quessy and Thomas Stuart Richardson. Rewardless open-ended learning (ROEL). 2021.
Nicholas J Radcliffe. Genetic set recombination and its application to neural network topology optimisation. Neural Computing & Applications, 1(1):67-90, 1993.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Thomas Ray. An approach to the synthesis life. Artificial life II. pp. 371-408, 1991.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprint arXiv: /6/ I.04201, 2016.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative reinforcement learning. arXiv preprint vxXiv1170.00886 2017.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339-2352, 2021.
Jacob Schrum, Vanessa Volz, and Sebastian Risi. CPPN2GAN: Combining compositional pattern producing networks and gans for large-scale pattern generation. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 139-147, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. International Conference on Learning Representations, 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre. Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- play. Science, 362(6419): 1140- 1144, 2018
H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-MPO: On-policy maximum posteriori policy optimization for discrete and continuous control. arXiv preprint arXiv: 1909. 12288, 2019.
I. Soros and Kenneth Stanley. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. In ALIFE 14: The Fourteenth International Conference on the Synthesis and Simulation of Living Systems, pp. 793-800, 2014.
Lee Spector, Jon Klein, and Mark Feinstein. Division blocks and the open-ended evolution of development, form, and behavior. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 316-323, 2007.
Kenneth O Stanley. Compositional pattern producing networks: novel abstraction of development. Genetic programming and evolvable machines, 8(2):131-162, 2007.
Kenneth O Stanley. Why open-endedness matters. Artificial life, 25(3):232 225,, 2019.
Kenneth O Stanley and Risto Mükkulainenn. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2):99-127, 2002.
Kenneth O Stanley, David D' Ambrosio, and Jason Gauci. A hypercube- based encoding for evoly- ing large-scale neural networks. Artificial life, 1552)228--11, 2009.
12

---

Under review as a conference paper at ICLR 2023
Kenneth O Stanley, Joel Lehman, and Lisa Soros. Open-endedness: The last grand challenge you' ve never heard of. O'Reilly Online, 2017. httpw///ttsss.w....a.....cc/////
Emma Hjellbrekke Stensby, Kai Olav Ellefsen, and Kyrre Glette. Co-optimising robot morphol- ogy and controller in a simulated open-ended environment. In International Conference on the
Applications of Evolutionary Computation, pp. 34-49, 2021.
Danesh Tarapore. Jeff Clune, Antoine Cully, and Jean Jean-Baptian Baptiste Mouret. How do different encod- ings influence the performance of the map-elites algorithm? In Proceedings of the Genetic and Evvoutionary Computation Conference, pp. 173-180, 2016.
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107. 128088 2021.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (POET): Endlessly generating increasingly complex and diverse learning environments and their solutions. 2019.
arXiv preprint arXiv:1901.07
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. International Conference on Machine Learning, pp. 9940-9951, 2020.
Larry Yaeger and Olaf Sporn.. Evolution of neural structure and complexity in a computational ecology. In Artificial Life X: Proceedings the Tenth International Conference the Simulation and Synthesis of Living Systems, pp. 330-336, 2006.
Fangqin Zhou and Joaquin Vanschoren. Open-ended learning strategies for learning complex loco- motion skills. arXiv preprint arXiv:2206.06779, 2022.
A
ABBREVIATIONS
Table 1: Abbreviations for commonly used terminologies.
Abbreviation
Termonology
MCC
Minimal Criterion Coevolution
POET
Paired Open-Endes Ended Trailblazer
EPOET
Enhanced Paired Open-Ended Trailblazer
NEAT
NeuroEvolution of Augmenting Topologies
ATEP
Augmentative Topology Enhanced POET
NN
Neural Network
FBT-ATEP
Fitness-Based Transfer ATEP
SBT-ATEP
Species-Based Transfer ATEP
RT-ATEP
Random Transfer ATEP No Transfer ATEP
NT-ATEP
EPOET40x40
EPOET with NN of two hidden layers, 40 nodes each
EPOET20X20
EPOET with NN of two hidden layers, 20 nodes each
ANNECS
All New and Novel Environments Created and Solved
PATA-EC
Performance of All Transferred Agents Environment Charactization
EA
Environment-Agae Ageett
FNR
Fitness to Nodes Ratio
ANR
ANNECS Nodes Ratio
B ACTION DISTRIBUTION FIGURES
In Figure 7 we have added figures for the action distributions of each algorithm.
13

---

Under review as a conference paper at ICLR 2023
019
300
020
013
044
100
002
AddL
630
Adiont
031
6.14
414
0150
500
000
000
Figure 7: Action Distributions for (top row) SBT-ATEP, (middle row) FBT-ATEP and (bottom row) EPOET40x40. Each column represents one specific dimension of the action array.
C HYPERPARAMETER SETTINGS
This appendix is for hyperparameter settings which were used for ATEP and the baselines. Table 2 shows hyperparameters for EPOET. Table shows settings for NEAT in ATEP and Table shows parameter configurations for CPPNs. General hyperparameters for reproduction are given in Table 5
Table 2: ES hyperparameter settings
Hyperparameter
Setting
ES Population
512
Weight update method
Adam
Initial learning
0.01
Decay factor of learning rate
0.9999
Initial noise standard deviation
0.1
Lower bound of noise standard deviation
0.01
Decay factor of noise standard deviation
0.999
14

---

Under review as a conference paper at ICLR 2023
Table 3: NEAT hyperparameter settings
Hyperparameter
Setting
Population size
1000
Crossover probability
Weight mutation (small) probability
0.85
Weight mutation (large) probability
0.15
Weight mutation (small) range
-0.1-
Weight mutation (large) range
-1++1
Connection mutation probability
0.85
Node mutation probablity
0.15
Maximum stagnation
60
c_
1.0
c-(2]
1.0
c_(3]
3.7
Delta threshold
3.0
Initial condition
full
Activation function
tanh
Number of inputs
24
Number of outputs
Table 4: CPPN hyperparameter settings
Hyperparamer
Setting
Initial condition
full
Activation default
identity
Activation options
identity sin sigmoid square tanh
Aggregation default bias init stdev bias init type bias max value bias min value
sum
0.1
gaussian
10.0
-10.0
bias mutate power
0.1
bias mutate rate num inputs num outputs
0.75
1
1
response init mean response init type response max value response min value single structural mutation structural mutation surer
1.0
gaussian
10.0
-10.0
True
default
weight init stdev weight init type
0.25
gaussian
weight max value
10.0
weight min value
-10.0
weight mutate power
0.1
weight mutate rate
0.75
15

---

Under review as a conference paper at ICLR 2023
Table EPOET general hyperparameter settings for ATEP
Hyperparameter
Setting
Reward threshold
200
Environment difficulty MC
25 340
Transfer check
25
Reproducibility check
150
Active environments
20
D TRANSFER MECHANISMS
This appendix shows pseudocodes of Species-based and Fitness-based transfer mechanisms. Algo- rithms shown in Algorithm [ and 2 respectively. 2 in partieular, is very similar the transfer
algorithm described by Wang et al. 20200) find della(.) (-) in Algorithm iis calculated by the equa- tion Equation
4E N
8
W
(1)
N
Algorithm 1: Species- Based Transfer
Input -Candidate population's 's best individual I A function delta() that calculates
delta score and Ōthreshold-
Let M All environments (Candidate environment]
foreach m E . do
Im = best individual of environment
det = find delta(Le, Im) using Equation
if 6 S Sthreshhld then
delete target species
Transfer candidate species to target population
else
Transfer is not possible
end end
Algorithm 2: Fitness- Bassd Transfer
Input : Candidate population's best individual 1, a function Score(.) that calculates the maximum of the target agent's 5 most recent fitness scores.
Let M = All environments [Candidate environment]
foreach in do
Compute direct transfer Tp;
if DD > Score(m) then Compute fine-tuning transfer Tp;
if 1p > Score(m) then
Add m to _canddddteess
else
Transfer not possible
end
else
Transfer not possible
end
end
Delete whole population of canndddates
Transfer whole candidate population T conddaaass
16

---

Under review as a conference paper at ICLR 2023
E EVOLVED NETWORKS
E.1 ToY EXAMPLE
The toy example is presented to illustrate mutation and crossover operators in NEAT.
(a)
(b)
(c)
(e)
(I)
Figure 8: A toy example of the mutation of an individual in NEAT. (a) shows simple network, (b) mutates by adding a connection from node 3 to node 2 and (e) mutates by adding node 5 between node and node 2. (d), (e) and (f) shows toy example of the crossover between two individuals of NEAT. (d) and (e) shows two individuals.
is the result of the crossover between (d) and (e),
which results in adding new connection from node to node 4 in individual (e). These illustrations are motivated by NEATs' original paper (Stanley Miikkulainen 2002).
E.2 AUGMENTED NEURAL NETWORKS
For illustration purposes took later stage agents' neural network (NN) which eventually became the best performing individual for the solved environment. Yellow nodes represent inputs, aqua represents hidden nodes while white maps outputs. Figure 9a shows the start of the NN without any evolution applied. Figure 95] is the state after 2000 iterations. It shows to have a few nodes in hidden layer one and couple of nodes in hidden layer two. Complexity has not yet been emerged as it can be observed, Figuure 10alis the NN after 4000 iterations. We can observe emerging complexity. Some residual connections can be seen, for example node 5, an input node connects hidden node 30, but also connects to an output node 26. Figure 10b] is the evolved network that became the best individual in the population, after 6000 iterations of when it was created. Here complexity is observed in many levels. Firstly, an input node connects to input node O, where node 46 appears to be having forward pass to the input and 0. Secondly, we observe an interesting small block of 6 nodes conventionally fully connected, input 22 and 23 connected to 40 and 42 which further connects to 41 and 43, that connects to the outputs. Overall depth the network seems to be variable.
17

---

Under review as a conference paper at ICLR 2023
(b)
Figure 9; From left to right: NNs at iteration 0 and 2000 when it was created.
18

---

Under review as a conference paper at ICLR 2023
(a)
(b)
Figure 10: From left to right: NNs at iteration 4000 and 6000 of when was created.
19

---

Under review as a conference paper at ICLR 2023
F KEY CONCEPTS FROM EPOET
F.1 PATA-EC
Performance of All Transferred Aggett-Envvronmnn Characterization (PATA-EC) introduced by EPOET as domain-genes genierl environment characterization score. PATA-EC is score that describes the environment behaviour through all active and archived agents behaviour on the environment. Four steps to calculate PATA-EC are: (1) Evaluate: Each created environment evaluates all agents and stores their raw scores in a vector. (2) Clip: The cores are clipped between lower and upper bound. By doing we clip the possibilities of having agent that is failure (too low score) the environment is to easy for the agent (too high score). (3)
We the sort the raw scores by their rankings, and normalize the scores in between the range of [-0.5,0.5]. This allows us to use direct euclidean distance on the PATA-EC scores. Eventually PATA-EC score helps in creating most novel environments and is not domain-specifie thus could be used in any environment.
F.2 ANNECS
All New and Novel Environments Created and Solved is measure of progress in open-ended system. The intuition behid it eventually gets solved by the agents, thus measuring useful environments. The environments that are useful gets solved by an agent thus progressing itself in an open-ended system. To be counted in ANNECS score, environment created at particular iteration must: (1) pass the minimal criteria (i.e. not too easy or too hard) measured against all active and archived agents generated over the entire run far, (2) eventually solved by the system, which means the algorithm will not receive credit for producing unsolvable environments. As ANNECS metrics constantly goes up indicates the algorithm making meaniungful environments.
if the system is generating environments that are novel but also
20


Reviews:
[{"review_id": "w9oMgCqbwY", "reviewer": "Reviewer_SyVu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an ML pipeline to evolve agent topologies for open ended learning by combining previous works EPOET and NEAT. Experiments are done on the 2D bipedal environment.", "summary_of_the_review": "Incremental idea with poor execution.", "strength_and_weaknesses": "1. The paper lacks sufficient insights/results/illustrations to understand how the agent topologies are changing, most of the methodology and evaluation is borrowed from EPOET whose focus was more on novel environment generation rather than agent topology, so the results and discussion seem inadequate. Further EPOET already introduced the idea of using NEAT for environment generation which the authors have adapted for topology evolution.\n\n2. Statistical significance of the results is poor, only 2 seeds have been used for the comparisons. Performance of RL based agents is typically high variance, so it is not justified to compare the agent policies/performance with so less data.\n\n3. Currently the work comes short on novelty and lacks reusable insights that can actually be informed for progressing open ended learning. Are there any interesting patterns in the way the topologies evolve? What insights can be drawn about choosing the hyperparameters for fitness and mutation for open ended learning? \n\n4. To me, the empirical evaluation seems lacking as no adequate baseline for topology evolution is used.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "sp0pY160pB", "reviewer": "Reviewer_TYB9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use NEAT to evolve NNs with varying structures to solve an increasingly complex collection of reinforcement learning problems under the framework of open-ended learning. It also studied two alternative transfer mechanisms to transfer evolved NNs across different environments. Experiments show that the newly proposed algorithm may achieve good performance for one benchmark of open-ended learning.", "summary_of_the_review": "It is interesting and important to study the problem of open-ended learning. However, the technical novelty of the paper does not seem to be sufficiently strong. The newly proposed algorithm is also computationally costly to run. Consequently, the practical usefulness of the new algorithm requires more investigations.", "strength_and_weaknesses": "Strength:It is interesting and important to study the problem of open-ended learning. It is also interesting to consider structure/topology evolution in order to improve the effectiveness of evolved NNs towards solving complex problem instances.\n\nWeakness:This paper is motivated by the assumption that generalization can be improved by allowing evolved NNs to become increasingly complex. However, the validity of this assumption may be questionable. In fact, as the experiment results showed, the NNs evolved by NEAT tend to have small sizes, which help to maintain good generalization performance. Hence the experiment results reported do not seem to be fully compatible with the key assumption of the algorithm design.\n\nSeveral key concepts were used in the paper without thorough definition. For example, it remains unclear what \"completely open-ended\" on page 1 means. How to measure the degree of open-endedness? In line with this question, it remains unclear how open-ended learning is defined. What does it mean by \"complexity arises randomly\"? I cannot find any thorough mathematical definition of this learning problem. It is also unclear how the open-ended learning problem is related to other learning paradigms such as transfer learning and multitask learning.\n\nFurthermore, the practical significance of open-ended learning is not investigated with sufficient depth in the paper. In Section 2, the authors introduced some practical applications of open-ended learning. However, it seems that none of such applications were investigated in the paper based on the newly proposed algorithm. Hence, the practical usefulness of the new algorithm remains questionable to a certain extent.\n\nOne major contribution of this paper appears to be using NEAT instead of ES (or other reinforcement learning algorithms) to drive the evolution of NNs. The introduction of NEAT in Subsection 4.1 appears to be following exactly the original design of NEAT. It is not clear what are the technical difficulties of using NEAT for open-ended learning and how the technical difficulties were actually tackled in this paper. A direct adoption of NEAT does not seem to be sufficiently novel. Furthermore, since Subsection 4.1 introduces only an existing algorithm, it may not be appropriate to put it in the section on new methods (or new algorithms).\n\nAnother major concern is regarding the proposed transfer mechanisms. According to the paper, \"transfers agents across environments can prevent stagnation and leverage experience gained on one environment as a step towards solving another\". However, what kind of \"stagnation\" is considered in the paper? Is this a common problem for existing algorithms, such as POET and EPOET? Why can the newly introduced transfer mechanism prevent stagnation? Without clear answers to all these questions, the technical novelty and contribution of the newly developed transfer mechanisms remain doubtful. The mechanism design also appears to be heuristic in nature. It is not clear what theoretical principles are being followed while designing the transfer mechanisms.\n\nMore technical details should be provided in the paper. For example, the first criteria for environment counting on page 3 is that the environment must neither be too easy nor too hard. However, what are the precise definitions of easy and hard environments and why?\n\nThe new algorithm appears to have several new hyper-parameters to be fine-tuned, for example the threshold delta. It may not be easy to tune these hyper-parameters. Hence it may be practically difficult to use the new algorithm for some real-world open-ended learning applications.\n\nThe new algorithm is computationally costly to run (up to 200k cpu hours according to the paper). As a result, only 2 seeds have been tested in the paper. It may not be easy to draw any solid conclusions from the experiment results on 2 seeds. Furthermore, the network architectures (40x40 or 20x20) experimented appear to be much smaller than the typically used architectures among the majority of deep reinforcement learning algorithms. It is unclear whether the new algorithm can still outperform the baselines upon increasing the complexity of the network architectures. In general, as the benchmark problem studied is simple, the claim that NEAT enables the new algorithm to solve challenging or complex problems is not strongly verified in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SnhT291b-g", "reviewer": "Reviewer_JKxU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an extension of the EPOET algorithm for co-evolving a population of RL (PPO) agents with a population of task configurations, such that the topology of agents can also be co-adapted throughout training. Specifically, this work introduces Augmentative Topology EPOET (ATEP), which uses NEAT instead of PPO to update the population of agents, and thus their topology in addition to network weights throughout training. As NEAT requires a population, ATEP effectively co-evolves populations of populations of agents, and thus requires new transfer mechanisms for transferring agents evolved to solve one environment to become the \"elite\" for another environment, as necessitated by EPOET. To this end, the paper proposes two such transfer mechanisms, Fitness-Based Transfer (FBT) and Species-Based Transfer (SBT). The main results show that ATEP methods result in more robust agent populations (i.e. capable of solving more task configurations) and leads to faster growth in the number of novel, meaningfully different tasks that are generated compared to baseline methods.", "summary_of_the_review": "While I like the ideas in this paper, the overall evaluation is only over 2 training runs and the quality of the presentation and writing (including eliminating the existing incorrect claims previously detailed) can be much improved. Importantly, the authors do not provide reassurance that ATEP is not outperforming EPOET purely due to its use of a larger effective population size—since NEAT is a population-based approach and each agent in the population is now optimized according to a NEAT population. Further, I would like to see more evidence for the fidelity of their implementation of EPOET compared to the original, given the deviation in ANNECS score. For these reasons and those stated previously in my review, I cannot recommend this paper for acceptance in its current state. I am very open to accepting this paper if the authors can provide an improved version of this work that addresses the issues described.", "strength_and_weaknesses": "### Strengths\n- Open-endedness is an important, underexplored topic in ML research.\n- The paper provides an interesting extension of EPOET that allows agent topologies to be co-evolved with the environment.\n- The paper proposes two new transfer mechanisms for EPOET that are required for applying NEAT in updating the agent population, both of which serve as novel extensions to EPOET's standard transfer mechanism.\n\n### Weaknesses\n- It seems the comparison to EPOET is unfair because **NEAT requires ATEP to effectively maintain more agents than EPOET**. It is not clear if this paper controls for the larger population size of ATEP.\n- The main experimental results (e.g. Figure 2, Figure 5) are only based on two training runs.\n- The ANNECS values in Figure 2 for EPOET do not seem to match that from the original paper, where at 20k updates, EPOET reaches an ANNECS score of 100, thus matching the performance of SBT-ATEP in this work. This may be due to the authors reduction of the number of active environments from 40 to 20 with respect to the original study. However, the authors should run their implementation of EPOET on the original setting of 40 environments on 3 seeds to ensure their implementation is correct, or otherwise state that they reuse the official open source implementation of EPOET.\n- The agent network size seems to play a significant role in improved ANNECS and robustness. Therefore, authors should also provide more detailed discussion on how the choice of 20x20 and 40x40 compares to the original EPOET implementation, which achieved much higher ANNECS scores in the same number of updates.\n- The paper is unclear about several key details, described in the Clarity section.\n- The paper claims the original POET work uses BipedalWalkerHardcore, but this is inaccurate. POET and EPOET used modifications of the BipedalWalker environment, not the specific BipedalWalkerHardcore environment, which refers to a specific setting of the BipedalWalker environment parameters.\n- At the top of Page 6, the authors claim SBT-ATEP does not stagnate, but this does not follow from the results, because it is unknown if SBT-ATEP will stagnate beyond the number of training steps investigated.\n- The paper does not provide the correct citation for the VAE. It should be Kingma, Diederik P., and Max Welling. \"Auto-encoding variational bayes.\" 2013.\n- The introduction mentions prior works are limited in open-endedness due to the finiteness of the environment. However, the authors do not state this shared weakness in their own experimental environment.\n- The authors do not provide any measures of environment complexity over the course of training, e.g. as done in prior works. This would be useful for comparing the difference in kinds of environments produced by ATEP vs. EPOET.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1PaKadOHWsP", "reviewer": "Reviewer_TUdn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "this paper propose to simultaneously evolve the environments and the agents by increasing complexity of the later. the proposed method extends EPOET. ", "summary_of_the_review": "the lack of simple experiments demonstrating the key points and the lack of explicit definitions prevents me from recommending this paper for publication in ICLR. ", "strength_and_weaknesses": "1. the paper should be reorganized in the sake of readability. \n\nFor example \"EPOET improves upon POET by adding in two algorithmic improvements\" (1) ... (2). \nWhat section does example (1) and (2)? If those the main contributions and improvements, one would expect to see them explicitly in the text.\n\n2. a few relevant works are not mentioned. E.g., \"Task-Agnostic Morphology Evolution\" and similar.\n\n3. I am not sure that a method can be called reproducible if it takes 200000 CPU hours. A simple examples clarifying the main points one by one will be helpful. \n\n4. \"Accumulated Number of Novel Environments Created and Solved (ANNECS), a metric for open-ended learning that, intuitively, describes the amount of interesting new...\". What is the definition of \"interesting\"?\nis Novel and interesting the same? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject