Augmentative Topology Agents For Open-ended Learning
Supplementary Material
Muhammad Umair Nasir, Michael Beukman, Steven James and Christopher Cleghorn

TABLE II: NEAT hyperparameter settings

A PPENDIX IACTION D ISTRIBUTION F IGURESIn Figure 1 we have added figures for the action
distributions of each algorithm.

A PPENDIX IIH YPERPARAMETER S ETTINGS

This appendix is for hyperparameter settings whichwere used for ATEP and the baselines. Table I shows
hyperparameters for ES in EPOET. Table II shows settings for NEAT in ATEP and Table III shows parameter
configurations for CPPNs. General hyperparameters forreproduction are given in Table IV.

TABLE I: ES hyperparameter settings
Hyperparameter

Setting

ES Population
Weight update method
Initial learning rate
Decay factor of learning rate
Initial noise standard deviation
Lower bound of noise standard deviation
Decay factor of noise standard deviation

512
Adam
0.01
0.9999
0.1
0.01
0.999

*University of the Witwatersrand, South Africa

Hyperparameter

Setting

Population size
Crossover probability
Weight mutation (small) probability
Weight mutation (large) probability
Weight mutation (small) range
Weight mutation (large) range
Connection mutation probability
Node mutation probablity
Maximum stagnationc1c2
c3
Delta threshold
Initial condition
Activation function
Number of inputs
Number of outputs

1000
0.3
0.85
0.15
-0.1 - +0.1
-1 - +1
0.85
0.15
60
1.0
1.0
3.7
3.0
fulltanh
244

TABLE III: CPPN hyperparameter settings
Hyperparamer

Setting

Initial condition
Activation default
Activation options
Aggregation default
bias init stdev
bias init typebias max value
bias min value
bias mutate powerbias mutate ratenum inputs
num outputs
response init mean
response init typeresponse max value
response min value
single structural mutation
structural mutation surer
weight init stdev
weight init typeweight max value
weight min value
weight mutate power
weight mutate rate

fullidentity
identity sin sigmoid square tanhsum
0.1
gaussian
10.0
-10.0
0.1
0.75
11
1.0
gaussian
10.0
-10.0
True
default
0.25
gaussian
10.0
-10.0
0.1
0.75

Fig. 1: Action Distributions for (top row) SBT-ATEP, (middle row) FBT-ATEP and (bottom row) EPOET40x40.
Each column represents one specific dimension of the action array.
TABLE IV: EPOET general hyperparameter settings forATEP
Hyperparameter

Setting

Reward threshold
Environment difficulty MC
Transfer check
Reproducibility check
Active environments

20025 - 340
2515020A PPENDIX III
T RANSFER M ECHANISMSThis appendix shows pseudocodes of Species-based
and Fitness-based transfer mechanisms. Algorithms areshown in Algorithms 1 and 2, respectively. Algorithm 2,in particular, is very similar to the transfer algorithm
described by Wang et al. [1].

Algorithm 1: Species-Based TransferInput : Candidate population’s best individual
Ic . A function f ind delta(.) thatcalculates delta score and δthreshold .
Let M = All environments - {Candidateenvironment}
foreach m ∈ M do
Im = best individual of environment m
δct = find delta(Ic , Im ) using Equation 1
if δct ≤ δthreshold thendelete target species
Transfer candidate species to target
populationelse
Transfer is not possible
endendAlgorithm 2: Fitness-Based Transfer
Input : Candidate population’s best individual
I, a function Score(.) that calculates themaximum of the target agent’s 5 mostrecent fitness scores.
Let M = All environments - {Candidateenvironment}
foreach m in M do
Compute direct transfer ID ;if ID > Score(m) thenCompute fine-tuning transfer IP ;if IP > Score(m) thenAdd m to Tcandidates
elseTransfer not possible
end
else
Transfer not possible
end
end
Delete whole population of Tcandidates
Transfer whole candidate population to Tcandidates
R EFERENCES[1] R. Wang, J. Lehman, A. Rawal, J. Zhi, Y. Li,J. Clune, and K. Stanley, “Enhanced POET: Openended reinforcement learning through unbounded
invention of learning challenges and their solutions,”in International Conference on Machine Learning,
2020, pp. 9940–9951.