Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
Authors: Chris Reinke
Mayalen Etcheverry
Pierre-Yves Oudeyer
Venue:  (2020)
Keywords: deep learning
unsupervised Learning
self-organization
game-of-life

Abstract: In many complex dynamical systems, artificial or natural, one can observe self-organization of patterns emerging from local rules. Cellular automata, like the Game of Life (GOL), have been widely used as abstract models enabling the study of various aspects of self-organization and morphogenesis, such as the emergence of spatially localized patterns. However, findings of self-organized patterns in such models have so far relied on manual tuning of parameters and initial states, and on the human eye to identify interesting patterns. In this paper, we formulate the problem of automated discovery of diverse self-organized patterns in such high-dimensional complex dynamical systems, as well as a framework for experimentation and evaluation. Using a continuous GOL as a testbed, we show that recent intrinsically-motivated machine learning algorithms (POP-IMGEPs), initially developed for learning of inverse models in robotics, can be transposed and used in this novel application area. These algorithms combine intrinsically-motivated goal exploration and unsupervised learning of goal space representations. Goal space representations describe the interesting features of patterns for which diverse variations should be discovered. In particular, we compare various approaches to define and learn goal space representations from the perspective of discovering diverse spatially localized patterns. Moreover, we introduce an extension of a state-of-the-art POP-IMGEP algorithm which incrementally learns a goal representation using a deep auto-encoder, and the use of CPPN primitives for generating initialization parameters. We show that it is more efficient than several baselines and equally efficient as a system pre-trained on a hand-made database of patterns identified by human experts.

Published as a conference paper at ICLR 2020
INTRINSICALLY MOTIVATED DISCOVERY OF DIVERSE PATTERNS IN SELF-ORGANIZING SYSTEMS
Chris Reinke" Mayalen Etcheverry', Pierre-Yves Oudeyer
Flowers Team
Inria, Univ. Bordeaux, Ensta ParisTech (France)
{chris re reinke, het.ya/.....cceeeee...eereeeeee. oetp:re-yyeeeouudderrrreraaaaa
ABSTRACT
In many complex dynamical systems, artificial or natural, one can observe self- organization of patterns emerging from local rules. Cellular automata, like the Game of Life (GOL), have been widely used as abstract models enabling the study of various aspects of self-organization and morphogenesis, such as the emergence of spatially localized patterns. However, findings of self-organized patterns in such models have so far relied on manual tuning parameters and initial states, and the human eye to identify "interesting" patterns. In this paper, we for- mulate the problem of automated discovery of diverse self-organized patterns such -isensiienni complex dynamical systems, well as framework for experimentation and evaluation. Using a continuous GOL as a testbed, we show that recent mntrisically-motivvcee machine learning algorithms (POP-IMGEPS), initially developed for learning of inverse models in robotics, can be transposed and used in this novel application area. These algorithms combine intrinsically- motivated goal exploration and unsupervised learning of goal space representa- tions. Goal space representations describe the "interesting" features of patterns for which diverse variations should be discovered. In particular, we compare vari- ous approaches define and learn goal space representations from the perspective of discovering diverse spatially localized patterns. Moreover, we introduce an ex- tension state-of-the-ee POP-IMGER algorithm which incrementally learns a goal representation using deep auto-encoder, and the of CPPN primitives for generating initialization parameters. We show that it is more efficient than several baselines and equally efficient system pre-trained hand-made database patterns identified by human experts.]
INTRODUCTION
Self-organization of patterns that emerge from local rules is a pervasive phenomena in natural and artificial dynamical systems (Ball) 1999). It ranges from the formation of snow flakes, spots and rays on animal's 's skin. to spiral galaxies. Understanding these processes has boosted progress in many fields, ranging from physics to biology (Camazine et al., 2003). This progress relied importantly on the use of powerful and rich abstract computational models self-organization (Kauffman) 1993). For example, cellular automata like Conway's Game of Life (GOL) have been used to study the emergence of spatially localized patterns (SLPs) (Gardener) 1970), informing theories of the origins life (Gardener) 1970 Beer 2004]. SLPs, such the famous glider in GOL (Gardner al. 1983), are self-organizing patterns that have a local extension and can exist independently of other patterns. However, finding such novel self-organized patterns, and mapping the space of possible emergent patterns, has so far relied heavily on manual tuning of parameters and initial states. Moreover, the dependence of this exploration process on the human eye identify "interesting" patterns is strongly limiting further advances.
We formulate here the problem of automated discovery of diverse set of self-organized patterns in such high-dimensional, complex dynamical systems. This involves several challenges. A first
"Equal contribution.
Source code and supplementary material at wttphtt:/aatttrtteeeeee scovery.github

---

Published as a conference paper at ICLR 2020
challenge consists in determining representation of patters,, possibly through learning, enabling to incentivize the discovery of diverse "interesting" patterns. Such a representation guides exploration by providing measure of (di-)simillan adii)iiiilar between patterns. This problem is particularly challenging in domains where patterns are high-dimensional as in GOL. In such cases, scientists have a limited intuition about what useful features are and how represent them. Moreover, low-dimensional representations of patterns are needed for human browsing and the visualization of the discoveries. Representation learning shall both guide exploration, and fed by self-collected data.
A second challenge consists in how to automate exploration of high-dimensional, continuous initial- ization parameters to discover efficiently "interesting" patterns, such SLPs, with limited budget of experiments. Sample efficiency is important to enable the later use of such discovery algorithms for physical systems (Grizou al. 2020), where experimental time and costs are strongly bounded. For example, in the continuous GOL used in this paper as a testbed, initialization consists in deter- mining the values of a real- high-dimensional 256 256 matrix besides 7 additional dynamics parameters. The possible variations of this matrix are too large for simple random sampling to be efficient. More structured methods are needed.
To address these challenges, we propose to leverage and transpose recent intrinsically motivated learning algorithms, within the family of population- based Intrinsically Motivated Goal Exploration Processes
denoted simply as IMGEPs below, Baranns & Oudeyer (2013): Péré et al. (2118) They were initially designed to enable autonomous robots to explore and learn what effects be produced by their actions, and how to control these effects. IMGEPs self-define goals in goal space that represents important features of the outcomes of actions, such as the position reached by an arm. This allows the discovery of diverse novel effects within their goal representations. was recently shown how deep neuronal auto-encoders enabled unsupervised learning of goal repre- sentations IMGEPs from raw pixel perception of robot's visual scene (Laversanne- Finot al,, 2018). We propose to use similar mechanism for automated discovery of patterns by unsupervised learning of low-dimensional representation of features of self-organized patterns. This removes the need for human expert knowledge to define such representations.
Moreover, key ingredient for sample efficient exploration of IMGEPs for robotics has been the use of structured motion primitives to encode the space of body motions (Pastor et al. 2013). We propose to use a similar mechanism to handle the generation of structured initial states in GOL-like complex systems, based on specialized recurrent neural networks (CPPNs) (Stanley 2006).
In summary, we provide in this paper the following contributions:
We formulate the problem of automated discovery of diverse self-organized patterns in high-dimensional and complex game-of-life types of dynamical systems.
We show how to transpose POP-IMGEPs algorithms to address the associated joint chal- lenge of learning to represent interesting patterns and discovering them in a sample efficient manner.
We compare various approaches to define or learn goal space representations for the sample efficient discovery of diverse SLPs in continuous GOL testbed.
We show that an extension of state-of-the-art POP-IMGEP algorithm, with incremen- tal learning of goal space using deep auto-encoder, is equally efficient than system pretrained on a hand-made database of patterns.
2 RELATED WORK
Automated Discovery in Complex Systems Automated processes have been widely used to ex- plore complex dynamical systems. For example, evolutionary algorithms have been applied to search specific patterns or rules of cellular automata (Mitchell et al.) 1996,, [Sapin et al., (003)) However, their objective is to optimize a specific goal instead of discovering diversity of patterns. Another line of experiments represent active inquiry-based learning strategies which query which set of experiments to perform to improve a system model, i.e. a mapping from parameters to the system outcome. Such strategies have been used in biology (King al. 2004 2009). chemistry (Raccuglia et al., (2066; Reizman et al.) 2016 Duros et al., (017) and astrophysics (Richards et al.) 2011). However, these approaches have relied expert knowledge, and focused on automated opti-
2

---

Published as a conference paper at ICLR 2020
mization of pre-defined target property. Here, we interested automatically discover and map a diversity of unseen patterns without prior knowledge of the system. An exception is the concurrent work of Grizou et al. (2020), which showed how a simple POP-IMGEP algorithm could be used to automate discovery of diverse patterns in oil-droplet systems. However, it used a low-dimensional input space, and hand-defined low-dimensional representation of goal spaces, identified major limit of the system.
Intrinsically motivated learning Intrinsically-motiiatte motivated learning algorithms (Baldassarre & Mirolli 2013; |Baranes & Oudeyer (2013) autonomously organize an agent's 's exploration curriculum in order to discover efficiently a maximally diverse set of outcomes the agent can produce in an un- known environment. They are inspired from the way children self-develop open repertoires of skills and learn world models. Intrinsically Motivated Goal Exploration Processes (IMGEPs) (Baranes & Oudeyer 2013 Forestier al.) 2017) are family of curiosity-driven algorithms developed allow efficient exploration of high-dimensions. complex real world systems, Population-base versions of these algorithms,, which leverage episodic memory, hindsight learning, and structured dynamic motion primitives to parameterize policies, enable sample efficient acquisition of nigh-dimenssioal skills in real world robots (Forestier et al.) 2017: Rolf al. 2010). The discovered repertoires of di- verse behaviors can be reused to solve hard exploration downstream tasks (Colas et al. 2018 Conti al. 2018). Recent work (Laversanne-Finot et al., 2018: Péré et al. 2018) studied how auto- matically learn the goal representations with the use of deep variational autoencoders. However, training was done passively and in an early stage on a precollected set available observations. Recent approaches (Nair al. 2018 Pong al., 2019) introduced the use of online training of variational autoencoders VAES to learn the important features of goal space similar to the methods in this paper. However, these approaches focused on the problem of sequential decisions in MDPs, incurring a cost on sample efficiency. This problem is observed in various intrinsically motivated RL approaches (Bellemare al.] 2016 Burda et al., (0019b). These approaches are orthogonal to the automated discovery framework considered here with independent experiments allowing the use of memory-based sample efficient methods. A related family of algorithms in evolutionary com- putation is novelty search (Lehman & Stanley, 2008) and quality-diversity algorithms (Pugh et al., 2016). which can be formalized as special kinds of population based IMGEPs.
Representation learning We using representation learning methods to learn autonomously goal spaces for IMGEPs. Representation learning aims at finding low-dimensional explanatory fac- tors representing nigh-dimensional input data (Bengio et al. 2013). It key problem in many areas in order to understand the underlying structure of complex observations. Many state-of-the-art methods (Tschannen et al., 2018) have built on top of Deep VAEs (Kingma & Welling 2013), using varying objectives and network architectures. However, studies of the interplay between represen- tation learning and autonomous data collection through exploration of an environment have been limited so far.
ALGORITHMIC METHODS FOR AUTOMATED DISCOVERY
3.1 POPULATION-BASEE INTRINSICALLY MOTIVATED GOAL EXPLORATION PROCESSES
An IMGEP is an algorithmic process generating a sequence of experiments to explore the param- eters of a system by targeting relf-generated goals (Fig. D. It aims to maximize the diversity of observations from that system within a budget of N experiments. In population- based IMGEPs, an explicit memory of the history H of experiments and observations is used to guide the process.
The systems are defined by three components. A parameter space θ corresponding to the control- lable system parameters An observation space where an observation o a vector representing all the signals captured from the system. For this paper, the observations are time series of images which depict the morphogenesis of activity patterns. Finally, an unknown environment dynamic D: θ O which maps parameters to observations.
To explore a system, an IMGEP uses a goal space T that represents relevant features of its obser- vations, computed by encoding function ğ R(o). For exploration of patters, such features may describe their form or extension. The exploration process iterates times through: 1) sample goal from goal sampling distribution g G(H); 2) infer corresponding parameter 0 using
3

---

Published as a conference paper at ICLR 2020
0k
Initial System State
Ok
System Rollout
CPPN
1
A 200
4
Parameter for Dynamics
Deep Neural
R
0.2
Observation Space O
Autoencoder
T=
f 0.4
/ 0.8
Ok
A 1200
o 0.02
4) encode observation
3) perform experiment
update encoder every K steps
Parameter Space
Goal Space T
R(os)
History H
[0 R(o))] 10, ok. R(or)]
&R
1) sample goal:
select parameter: -1 (Bc.
Figure I: Population-based intrinsically motivated goal exploration process with incremental learn- ing of goal space (IMGEP-OCH algorithm) used to explore continuous GOL model.
parameter sampling policy IT Pr(0; g. H): 3) roll-out an experiment with 0, observe outcome o, compute encoding R()); 4) store (0, R(o)) in history H. The parameter sampling policy II and in some cases the goal sampling distribution G take into account previous explorations stored history H. Therefore, the history is first populated through exploring Ninat randomly sampled parameters. After this initial phase the described intrinsically motivated goal exploration process starts.
Different goal and parameter sampling mechanisms can be used within this architecture (Baranes & Oudeyer 2013 Forestier Oudeyer 2016). the experiments below, goals are sampled uniformly over hyperrectangle defined in T. The hyperrectangle is chosen large enough to allow sampling of a large goal diversity. The parameters are sampled by 1) given a goal, selecting the parameter from the history whose corresponding outcome is most similar in the goal space; 2) then mutating it by random process.
3.2 ONLINE LEARNING OF GOAL SPACES WITH DEEP AUTO-ENCODERS
For IMGEPs the definition of the goal space T and its corresponding encoder R critical part, because it biases exploration of the target system. One approach to define a goal space by selecting features manually, for example by using computer vision algorithms detect the positions of pattern and its form. The diversity found by the IMGEPs will then be biased along these pre-defined features. A limit of this approach is its requirement of expert knowledge to select helpful features, particularly problematic in environments where experts do not know in advance what features are important, or how formulate them.
Another approach is to learn goal space features by unsupervised representation learning. The aim to learn a mapping R()) from the raw sensor observations a compact latent vector IR. This latent mapping can be used as a goal space where a latent vector z g is interpreted as goal. Previous IMGEP approaches already learned successfully their goal spaces with VAES (Laversanne- Finot et al.) 2018 Péré et al.) 2018). However, the goal spaces were learned before the start of the exploration from à prerecorded dataset of observations from the target environment. During the exploration the learned representations were kept fixed. A problem with this pretraining approach that it limits the possibilities to discover novel patterns beyond the distribution of pretraining examples, and requires expert knowledge.
4

---

Published as a conference paper at ICLR 2020
Algorithm 1: IMGEP-OGL.
Initialize goal space encoder VAE with random weights
for i 1- 1 to N do
if Niit then
// Initial random iterations to populate H
L Sample 0 ~ (()
else
// Intrinsically motivated iterations
Sample goal ((H) based on space represented by R
Choose gg, H)
Perform an experiment with 0 and observe o
Encode reached goal 0 = R(o)
Append (0, o, g) to the history H
if mod K == 0 then
// Periodically train the network
for E epochs do
Train R on observations in H with importance sampling
for (0, 9) E H do
// Update the database of reached goals
H < R(o)
In this paper we attempt to address this problem by continuously adapting the learned representation to the novel observations encountered during the exploration process. For this purpose, propose an online goal space learning IMGEP (IMGEP-OGL) which learns the goal space incrementally during the exploration process (Algorithm [T. The training procedure of the VAE is integrated in the goal sampling exploration process by first initializing the VAE with random weights (Appendix(B.6/ The VAE network then trained every K explorations for E epochs the observation collected in the history H. Importance sampling is used to give more weight to recently discovered patterns by using a weighted random sampler. samples for 50% of the training batch samples patterns from the last K iterations and for the other 50% patterns from all other previous iterations.
3.3 STRUCTURING THE PARAMETER SPACE IN IMGEPS: FROM DMPs TO CPPNS
A key role in the generation of patterns in dynamical systems is their initial state At IMGEPs sample these initial states and apply random perturbations to them during the exploration. For the experiments this paper this state is a two-dimensional grid with 256 x 256 cells. Performing directly random sampling of the 256 x 256 grid cells results in initial patterns that resemble white noise. Such random states result mainly in the emergence of global patterns that spread over the whole state space, complicating the search for spatially localized patterns. This effect is analogous to similar problem in the exploration of robot controllers. Direct sampling of actions for individual actuators microscopic time scale is usually inefficient. A key ingredient for sample efficient exploration has been the use of structured primitives (dynamic motion primitives DMPs) to encode the space of possible body motions (Pastor al., 2013).
We solved the sampling problem for the initial states by transposing the idea of structured primitives. Indeed, "actions" consist here in deciding the parameters of an experiment, including the initial state. We propose to use compositional pattern producing networks (CPPNs) (Stanley) (2066) to produce structured initial patterns similar do DMPs. CPPNs are recurrent neural networks that allow the generation of structured initial states. (Appendix B.4 Fig. 14p They are defined by their network structure (number of neurons, connections between neurons) and their connection weights. Thus, instead of using directly an initial state part of the parameters 0 to control the dynamical system, a CPPN is used. If system roll-out for the parameters are performed, then the initial state ===1 generated by the CPPN. Moreover, instead of sampling and mutating directly an initial state, the weights and structure of the CPPN are randomly generated and mutated. Please note, the number of parameters in is therefore not fixed and open-ended (yet starts small) because the structure and the number of weights of randomly sampled and mutated CPPNs differ.
5

---

Published as a conference paper at ICLR 2020
(a) Evolution in Lenia from CPPN to animal
(b) Lenia animals discovered by IMGEP-OGL.
(c) Lenia animals
discovered by Chan (2019)
[=1
[=50
1=100
2000
Figure 2: Example patterns produced the continuous GOL system (Lenia). Illustraioon of the dynamical morphing from an initial CPPN image to animal (a). The automated discovery (b) is able to find similar complex animals as a human-expen manual search (c) by Chan (2019).
EXPERIMENTAL METHODS
We describe here the continuous Game of Life (Lenia) we use as a testbed representing a large class of high-dimensional dynamical systems, as well as the experimental procedures, the evaluation methods used to measure diversity and detect SL.s, and the used algorithmic baselines and ablations. Implementation details and parameter settings for all procedures are given in Appendix B
4.1 CONTINOUS GAME OF LIFE AS A TESTBED
Lenia (Chan 2019) continuous cellular automaton (Wolfram 1983) similar to Conway's Game of Life (Gardener 1970). Lenia, in particular, represents nigh-dimensionll complex dynamical system where diverse visual structures can self-organize and yet are hard find by manual explo- ration. It features the richness of Turing-complete game-of-life models. It is therefore well suited to test the performance of pattern exploration algorithms for unknown and complex systems. The fact that GOL models have been used widely to study eelf-organization in various disciplines, ranging from physics to biology and economics (Bak et al. 1989), also supports their generality and potential of reuse of our approach for discovery in other computational or wet high-dimensional systems.
two-dimensional grid of cells E [o, ||256x256 where the state each cell real-valued scalar activity A'(x) [0, 1]. The state of cells evolves over discrete time steps t (Fig. 2 a). The activity change is computed by integrating the activity of neighboring cells. Lenia's behavior controlled by its initial pattern A==1 and several settings that control the dynamics of the activity change (R2 T, μr, a, 31,32,33) (3)).
Lenia consists
self-organd self-organizing morphogenetic system. Its parameters for the initial pattern == and dynamics control determine the development of morphological patterns. Lenia can produce diverse patterns with different dynamics (stable, non-stable chaotic). Most interesting. spatially localized coherent patterns that resemble in their shapes microscopic animals can emerge (Fig. 2 b, c). These pattern types, which we will denote "animals" as short name, are key reason scientists have used GOL models to study theories of the origins of life (Gardener) 1970] Beer 2004]. Therefore, in our evaluation method based on measures of diversity (see below), we will in particular study the performance of IMGEPs, and the impact of using various approaches for goal space representation, on finding a diversity different pattern classifiers to analyze the exploration results. Initially we differentiate between dead and alive patterns. A pattern is dead if the activity of cells are either 0 or 1. Alive patterns are separated into animals and non-animals. Animals are finite, i.c. which do not infinitely cross several borders. All other patterns are non-animals whose activity usually spreads over the whole state space.
Lenia can be understood
animal patterns. We implemented for this purpose
connected areas of positive activity which

---

Published as a conference paper at ICLR 2020
4.2 EVALUATION BASED ON THE DIVERSITY OF PATTERNS
The algorithms are evaluated based on their discovered diversity of patterns. Diversity is measured by the spread of the exploration in an analytic behavior space. This space is externally defined by the experimenter as in previous evaluation approaches in the IMGEP literature. For example, in Péré al. (2018), the diversity of discovered effects of robot that manipulates objects is measured by binning the space of object positions and counting the number of bins discovered. A difference here is that the experimenter does not have access to an easily interpretable hand-defined low-dimensional representation of possible patterns, equivalent to the cartesian coordinate of rigid objects. The space of raw observations O, i.e. the final Lenia patterns A ==200 is also too nigh-dimensional for a mean- ingful measure of spread in it. We constructed therefore an external evaluation space. First, latent representation space was build through the training of a 3-VAE (Higgins et al. (017)) to learn the important features over large dataset of 42500 Lenia patterns identified during the many exper- iments over all evaluated algorithms. This large dataset enabled to cover a diversity of patterns orders of magnitude larger than what could be found in any single algorithm experiment, which ex- perimental budget was order of magnitude smaller. We then augmented that space by concatenating hand-defined features (the same features were used for hand-defined goal space IMGEP).
For each experiment, all explored patterns were projected into the analytic behavior space. The diversity of the patterns is then measured by discretizing the space into bins of equal size by splitting each dimension into 7 sections (results were found to be robust to the number of bins per dimension, see Appendix B.77.. This results in 713 bins. The number of bins in which at least one explored entity falls is used as measure for diversity.
We also measured the diversity in the space of parameters θ by constructing an analytic parameter space. The 15 features of this space consisted of Lenia's parameters (R, T, n,, a,, B1, B2, 8a) and the latent representation of a V-E The 3-VAE was trained on a large dataset of initial Lenia states (A used over the experimental campaign. This diversity measures also used 7 bins per dimension.
4.3 ALGORITHMS
The exploration behaviors of different IMGEP algorithms were evaluated and compared to a random exploration. The IMGEP variants differ in their way how the goal space is defined or learned.
Random exploration: The IMGEP variants were compared to a random exploration that sampled randomly for each of the N exploration iterations the parameters including the initial state A=11
Goal exploration with a hand-defined goal space: The first IMGEP uses hand- defined goal space that is composed of 5 features used in Chan (2019). Each feature measures certain property of the final pattern 2=200 that emerged in Lenia: T) the sum over the activity of all cells, 2) the number of activated cells, 3) the density of the activity center, 4) an asymmetry measure of the pattern and 5) a distribution measure of the pattern.
IMGEP-HGS
IMGEP-PGL.
Goal exploration with pretrained goal space: For this IMGEP variant the goal space was learned with 38- VAE approach on training data before the exploration process started. The training set consisted of 558 Lenia patterns: half were animals that have been manually identi- fied by Chan 2019);; the other half randomly generated with CPPNs (see Section 4.4
IMGEP-OGL- Goal exploration with online learning of the goal space: Algorithm
IMGEP-RGS Goal exploration with a random goal space: An ablated IMGEP using a goal space based on the encoder of VAE with random weights.
4.4 EXPERIMENTAL PROCEDURE AND HYPPERAARAMETERS
For each algorithm 10 independent repetitions of the exploration experiment were conducted. Each experiment consisted of N = 5000 exploration iterations. This number was chosen to be compatible with the application of the algorithms in physical experimental setups similar to Grizou et al.) 2000)) planned in future work. For IMGEP variants the first Nmm 1000 iterations used random parameter sampling to initialize their histories H. For the following 4000 iterations each IMGEP approach sampled goal g via uniform distribution over its goal space. The ranges for sampling in the hand-
7

---

Published as a conference paper at ICLR 2020
defined goal space (HGS) are defined in Table 3 (Appendix B.5 The ranges for the B- B-V based goal spaces (PGL, OGL) were set to [-3. 3] for each of their latent variables. Then, the parameter 0k from previous exploration in H. was selected whose reached goal År had the minimum euclidean distance to the current goal within the goal space. This parameter was then mutated to generate the parameter & that explored.
The parameters I consisted of a CPPN (Section 3.3) that generates the initial state A for Lenia and the settings defining Lenia's dynamics: 0 [CPPN + At=1, R, T, p, G, 82, 3a]. CPPNs were initialized and mutated by a random process that defines their structure and connection weights as done by Stanley (006). The random initialization of the other Lenia settings for the dynamics was done by an uniform distribution and their mutation by Gaussian distribution around the original values. The meta parameters to initialize and mutate the parameters were the same for algorithms. They were manually chosen without optimizing them for specific algorithm.
RESULTS
We address several questions evaluating the ability of IMGEP algorithms identify a diverse set of patterns, and in particular diverse "animal" patterns (i.e. spatially localized patterns).
Does goal exploration outperform random parameter exploration? In robotics/agents contexts where scenes populated with rigid objects, various forms of goal exploration algorithms outper- form random parameter exploration (Laversanne-Finng et al. 2018). We checked whether this still holds in continuous GOL which have very different properties. Measures of the diversity in the analytic behavior space confirmed the advantage of IMGEPs with hand-designed (HGS) or learned goal spaces (PGL/OGL) over random explorations (Fig. 3 b). The organization resulting from goal exploration is also visible through the diversity in the space of parameters. IMGEPs focus their
(a) Diversity in Parameter Space
(b) Diversity in Behavior Space
5000
Random
Random
2500
IMGEP-RGS
IMGEP-RGS -GRS
4000
2000
diversity
IMGEP-HGS
diversity
IMGEP-HGS
3000
IMGEP-PGL
1500
IMGEP-PGI.
2000
IMGEP-OGI
1000
IMGEP-OGL
1000
500
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
explorations
explorations
(c) Behavior Space Diversity for Animals
(d) Behavior Space Diversity for Non- Animals
Random
1500 ... Random
IMGEP-RGS
IMGEP-RGS
diversity
1000
IMGEP-HGS
diversity
1000
IMGEP-HGS
IMGEP-PGL
IMGEP-PGL
500
IMGEP-OGL
500
IMGEP-OGI.
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
explorations
explorations
Figure 3: (a) Although random explorations reach the highest diversity in the analytic parameter space, (b) IMGEPs reach a higher diversity in the analytic behavior space (except when using ran- dom representations). (c) IMGEPs with learned goal space discovered larger diversity of animals compared to hand-defined goal space. (d) Learned goal spaces are as efficient as a hand-defined space for finding diverse non-animals. Overall, IMGEPs with unsupervised learning of goal features are efficient to discover diversity of diverse patterns. Depicted is the average diversity (n 10) with the standard deviation as shaded area (for some not visible because is too small). The final diversity is significantly different ( (Welchs t-test, < 0.01) for algorithms between the braces plotted on the right each figure. See Figs. to 10 for a qualitative visual illustration of these results.
8

---

Published as a conference paper at ICLR 2020
(a) IMGEP-HGS Goal Space
(b) IMGEP-OGL Goal Space
dead
non-animal
animal
Figure 4: (a) Hand-defined and (b) learned goal spaces have major differences shown here by a t- SNE visualization. The different number and size of clusters of animals or non-animals can explain the differences in their resulting diversity between the algorithms (Fig.
exploration on subspaces that are useful for targeting new goals. In contrast, random parameter exploration is unguided, resulting in higher diversity in the parameter space (Fig. 3 b).
What the impact of learning a goal space vs. using random or hand-defined features? We compared also the performance of random VAE goal spaces (RGS) to Icarned goal spaces (PGL/OGL). For reinforcement learning problems, using intrinsic reward functions based on random features of the observations can result in high or boosted performance (Burda al., 2019a b). In our context however, using random features (RGS) collapsed the performance of goal exploration, and did not even outperform random parameter exploration for all kinds of behavioral diversity (Fig. 3. Results also show that hand-defined features (HGS) produced significantly less global di- versity and less "animal" diversity than using learned features (PGL/OGL). However, HGS found an equal diversity of "non- -nimaani These results show that in this domain, the goal-space has a critical influence on the type and diversity of patterns discovered. Furthermore, unsupervised learn- ing is an efficient approach to discover a diversity of diverse patterns, i.e. both efficient at finding diverse animals and diverse non-animals.
Is pretraining on a database of expert patterns necessary for efficient discovery of diverse animals? A possibility to bias exploration towards patterns of interest, such as "animals", is to pretrain a goal space with a pattern dataset hand-made by an expert. Here PGL is pretrained with a dataset containing 50% animals. This leads PGL to discover high diversity of animals. How- ever, the new online approach (OGL) is as efficient as PGL to discover diverse patterns (Fig. [3] b,c,d). Taken together, these results uncover an interesting bias of using learned features with a VAE architecture, which strongly incentivizes efficient discovery of diverse spatially localized patterns.
How do goal space representations differ? We analyzed the goal spaces of the different IMGEP variants to understand their behavior by visualizing their reached goals in wwo-dimensional space. T-SNE (Maaten & Hinton 2008) was used reduce the nigh-dimensional goal spaces.
The hand-defined (HGS) and learned (OGL) goal spaces show strong differences between each other (Fig.A. We believe this explains their different abilities find either high diversity of non-animals or animals (Fig. 3 c, d). The goal space of the HGS shows large areas and several clusters for non- animal patterns (Fig. a). Animals form only few and nearby clusters. Thus, the hand-defined features seem poor to discriminate and describe animal patterns in Lenia. As consequence, when goals are uniformly sampled within this goal space during the exploration process, then more goals generated in regions that describe non-animals. This can explain why HGS explored higher diversity of non-animal patterns but only a low diversity of animal patterns. In contrast, features learned by OGL capture better factors that differentiate animal patterns. This is indicated by the several clusters of animals that span a wide area in its goal space (Fig. 4 b).
We attribute this effect to the difficulty of VAEs capture sharp details (Zhao et al. 2017). They therefore represent mainly the general form of the patterns but not their fine- one-grained structures. Animals differ often in their form whereas non-animals occupy often the whole cell grid and differ

---

Published as a conference paper at ICLR 2020
in their fine- fine-grained details. The goal spaces learned by VAEs seem therefore better suited for exploring diverse sets animal patterns.
6 CONCLUSION
We formulated novel application for machine learning: the problem of automatically discover- ing self-organized patterns in complex dynamical systems with high-dimensions both the param- eter space and the observation space. We showed that this problem calls for advanced methods quiring the dynamic interaction between sample efficient autonomous exploration and unsupervised representation learning. We demonstrated that population-based IMGEPs mic framework to address this challenge, by showing how it can discover diverse self-op organizg patterns continuous GOL. In particular, we introduced a new approach for learning goal space representation online via data collected during the exploration process. It enables sample efficient discovery of diverse sets of animal-like patterns, similar to those identified by human experts and yet without relying on such prior expert knowledge (Fig. 2). We also analyzed the impact of goal space representations on the diversity and types of discovered patterns.
promising algorith-
The continuous game of life shares many properties with other artificial or natural complex systems, explaining why GOL models have been used in many disciplines to study self ooff-organzzatinn, see Bak et al. (1989). We therefore believe this study shows the potential of IMGEPs for automated discovery in other systems encountered in physics, chemistry or even computer animation. In further work, we aim apply this approach in robotized wet experiments such the one presented in Grizou et al. (2220) addressing the fundamental understanding how proto-cells can self-organize. We are also working together with bio-chemists map the space of behaviors of certain complex biochemistry systems for which no appropriate model exists. Once such a map has been discovered, it can be used optimize the system for target properties by leveraging the diversity patterns found through the unsupervised discovery process.
ACKNOWLEDGMENTS
We thank Bert Wang- Chaa Chan for his helpful discussions about the Lenia system. We also thank Jonathan Grizou and Cedric Colas for useful comments on the paper.
REFERENCES
Per Bak, Kan Chen, and Michael Creutz. Self-organized criticality in the game of life. Nature, 342 (6251):780, 1989.
Gianluca Baldassarre and Marco Mirolli. Intrinsically motivated learning in natural and artificial systems. Springer, 2013.
Philip Ball. The self-made tapestry: pattern formation in nature, volume 198. Oxford University Press Oxford, 1999.
Adrien Baranes and Pierre-Yves Oudeyer. Active learning of inverse models with intrinsically mo- tivated goal exploration in robots. Robotics and Autonomous Systems, 61(1) 61(1):49-73, 2013.
Randall Beer. Autopoiesis and cognition in the game life. Artificial Life, 10(3):30--326, 2004.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-188 1828, 2013.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. 2019a.
10

---

Published as a conference paper at ICLR 2020
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019b.
Scott Camazine, Jean-Louis Deneubourg, Nigel R Franks, James Sneyd, Eric Bonabeau, and Guy Theraula. Self-organization biological systems. Princeton university press, 2003.
Bert Wang-Chak Chan. Lenia: Biology of artificial life. Complex Systems, 28(3):251-266, 2019.
Cédric Colas, Olivier Sigaud, and Pierre-Yve Oudeyer. Gep-pg: Decoupling exploration and
ploitation in deep reinforcement learning algorithms. International Conference on Machine Learning (ICML), 2018.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via pop- ulation of novelty-seeking agents. In Advances in neural information processing systems, pp- 5027-5038, 2018.
Vasilios Duros, Jonathan Grizou, Weimin Xuan, Zied Hosni, De-Liang Long, Haralampos N Miras, and Leroy Cronin. Human versus robots in the discovery and crystallization of gigantic polyox- ometalates. Angewandre Chemie, (9966)))0955--0990 2017.
Sébastien Forestier and Pierre-Y Oudeyer. Modular active curiosity -riven discovery of tool use. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3965-3972. IEEE, 2016.
Sébastien Forestier, Yoan Mollard, and Pierre-
Pierre-Yves Yves Oudeyer. Intrinsically motivated goal explo- ration processes with automatic curriculum learning. preprint 17Xiv:1708.022 09000, 2017.
Martin Gardener. Mathematical games: The fantastic combinations of john conway's 's new solitaire game" life.". Scientific American, 223:120-123, 1970.
Martin Gardner, Martin Gardner, Martin Gardner, and Martin Gardner. Wheels, life, and other mathematical amusements, volume 86. WH Freeman New York, 1983.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Jonathan Grizou, Laurie Points, Abhishek Sharma, and Leroy Cronin. curious formulation robot enables the discovery of novel protocell behavior. Science Advances, 6(5))eaay4237, 2020. 2020..
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representationss volume 3, 2017.
I. T. Jolliffe. Principal Component Analysis and Factor Analysis, pp. 115-128. Springer New York, New York, NY, 1986. ISBN 978-1-4757-1960 doi: 10.10077978-144457-19990
Stuart A Kauffman. The origins of order: Self-organization and selection in evolution. OUP, 1993.
Ross King, Kenneth Whelan, Ffion Jones, Philip GK Reiser, Christopher H Bryant, Stephen H Muggleton, Douglas B Kell, and Stephen G Oliver. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427(6971):247,, 2004.
Ross D King, Jem Rowland, Stephen G Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, Larisa N Soldatova, et al. The automation of science. Science, (24(5523) 89,,899.78 2009.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv:1412.6980, 2014.
preprint
Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv: 1312.61 14, 2013.

---

Published as a conference paper at ICLR 2020
Adrien Laversanne-Fing Alexandre Pere, and Pierre Yves Oudeyer. Curiosity driven exploration of learned disentangled goal spaces. In Proceedings of The 2nd Conference on Robot Learning, volume 87 of PLMR, 2018.
Joel Lehman and Kenneth O Stanley. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp. 329-336, 2008.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (NNov):55792260)) 2008.
Melanie Mitchell, James P Crutchfield, Rajarshi Das, et al. Evolving cellular automata with genetic algorithms: A review of recent work. In Proceedings of the First International Conference on Evolutionary Compulation and Its Applications (EvCA96), volume 8. Moscow, 1996.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
Peter Pastor, Mrinal Kalakrishnan, Franziska Meier, Freek Stulp. Jonas Buchli, Evangelos Theodorou, and Stefan Schaal. From dynamic movement primitives associative skill mem- ories. Robotics and Autonomous Systems, 61(4)3551-31,, 2013.
Alexandre Péré, Sébastien Forestier, Olivier Sigaud, and Pierre- Oudeyer. Unsupervised Learn- ing of Goal Spaces for Intrinsically Motivated Goal Exploration. In ICI.2208- 6th International Conference on Learning Representations, Vancouver, Canada, April 2018.
Vitchyr H Pong. Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. preprint arXiv: 1903:0303909 2019.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolu-
tionary computation. Frontiers Robotics and AI, 3:40, 2016.
Paul Raccuglia, Katherine Elbert, Philip DF Adler, Casey Falk, Malia Wenny, Aurelio Mollo, Matthias Zeller, Sorelle A Friedler, Joshua Schrier, and Alexander J Norquist. Machine-learning assisted materials discovery using failed experiments. Nature, 533(161 2016.
Brandon J Reizman, Yi-Ming Wang, Stephen L Buchwald, and Klavs F Jensen. Suzuki Suzuki-miyaun cross-coupling optimization enabled by automated feedback. Reaction chemistry & engineering, 1(6):658-666, 2016.
Joseph W Richards, Dan L Starr, Henrik Brink, Adam A Miller, Joshua Bloom, Nathaniel Butler, J Berian James, James P Long, and John Rice. Active learning to overcome sample selection bias: application photometric variable star classification. The Astrophysical Journal, 122 201
Matthias Rolf, Jochen J Steil, and Michael Gienger. Goal babbling permits direct learning of inverse kinematics. JEEE Transactions on Autonomous Mental Development, 2(3):216-22) 2010.
Emmanuel Sapin, Olivier Bailleux, and Chabrier Jean-Jacques. Research of cellular automaton simulating logic gates by evolutionary algorithms. In European Conference on Genetic Program- ming, pp. 414-423. Springer, 2003.
Kenneth Stanley. Exploiting regularity without development. In Proceedings of the AAAI Fall Symposium on Developmental Systems, pp. 37. AAAI Press Menlo Park, CA, 2006.
Kenneth Stanley and Risto Miikkulainen. Efficient evolution neural network topologies. In Proceedings of the 2002 Congress on Evolutionary Computation,, volume 2. IEEE, 2002.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation Icarning. preprint arXiv:1812.00 05069, 2018.
Stephen Wolfram. Statistical mechanics of cellular automata. Rev. of mod. physics, 55(3):601, 1983.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards deeper understanding of variational autoencoding models. preprint arXiv: 1702.0868.. 2017.
12

---

Published as a conference paper at ICLR 2020
A ADDITIONAL RESULTS AND FIGURES
We provide visualizations of discovered patterns, as well as of learned goal-spaces. A dataset and interactive visualization of all discovered patterns can be found at https:// wwwnattddddiicvyyyyy.eaomoo wwwcovery.gitninnee...e
io/ The visualization serves as a support in qualitatively evaluating the diversity of discovered patterns. Integrated into interactive interface, these graphs are also useful for human to easily explore and visualize the different types of found patterns. In addition, provide statistical results quantifying the proportion of sub types of discoverd patterns.
A.1 DISCOVERED PATTERNS
Fig. to 10 illustrate examples of discovered patterns per class (animal, non-animal, dead) for each algorithm. The patterns have been randomly sampled from the results of single exploration repetition experiment. Please note that the shown examples represent only small fraction of all discovered patterns. A database with all patterns can found on the website for the paper.
A.2
VISUALIZATION GOAL SPACES
The goal spaces of all IMGEP algorithms are visualized (Fig. 11) via two-dimensional reductions: PCA (Jolliffe 1986) and t-Distributed Stochastic Neighbor Embedding (t-SNE) (Maaten Hinton 2008 The visualizations were constructed by using for each algorithm its goal space represen- tations
discovered patterns from a single repetition experiment. All goal representations were normalized so that the overall minimum value became O and the maximum value 1 for each dimension. PCA was performed to detect the two principle components. T-SNE was executed using the default standard Euclidean distance metric and perplexity set 50.
A.3 PROPORTION OF DISCOVERED PATTERNS OF DIFFERENT CLASSES
We used the measure of diversity of the found patterns to compare the performance algorithms. Another measure compare the algorithms the average proportion of dead, animals, non-animals patterns discovered by each algorithm (Fig. 5 For animal patterns the results follow the diversity results (Fig. 3. i.e. OGL and PGL find the highest proportion of animal patterns, followed by HGS, then RGS and Random. A corollary is that RGS, random and HGS find in proportion higher percentage of non-animal patterns than OGL and PGL. As the number of different non- animal patterns is as high for OGL and PGL, this shows the higher sample efficiency of OGL and PGL to find diverse patterns. In contrast, Random, RGS and HGS approaches tend to find non- animal patterns that are more similar to each other average.
0.9
dead
0.8
non-animals
0.7
animals
0.6
proportion
0.5
0.4
0.3
0.2
0.1
0
Random
IMGEP-RGS
IMGEP-HGS
IMGEP-PGL
IMGEP-OGL
Figure 5: Proportion of patterns for each class and algorithm. Each dot besides the boxplot shows the proportion of found patterns for each repetition (n 10). The box ranges from the upper to the lower quartile. The whiskers represent the upper and lower fence. The mean is indicated by the dashed line and the median by the solid line.
13

---

Published as a conference paper at ICLR 2020
Random Exploration
animal: 8.1%
non-animal:
57.3%
dead: 34.6% %
Figure 6: Randomly selected examples of patterns discovered by the random exploration algorithm during single exploration with 5000 iterations.
14

---

Published as a conference paper at ICLR 2020
IMGEP-RGS
animal: 8.1%
non-animal:
82.4%
dead: 9.5% %
Figure 7: Randomly selected examples of patterns discovered by the IMGEP-RGE RGS algorithm during a single exploration with 5000 iterations.
15

---

Published as a conference paper at ICLR 2020
IMGEP-HGS
animal: 14.3%
non-animal: 70.0%
dead: 15.7%
Figure 8: Randomly selected examples of patterns discovered by the EMGEP-HGS algorithm during a single exploration with 5000 iterations.
16

---

Published as a conference paper at ICLR 2020
IMGEP-PGL
animal: 33.4%
dead: 23.6%
non-animal:
43.0%
Figure 9: Randomly selected examples of patterns discovered by the IMGEP-PGLL algorithm during a single exploration with 5000 iterations.
17

---

Published as a conference paper at ICLR 2020
IMGEP-OGL
animal: 32.5%
dead: 19.7%
non-animal:
47.8%
Figure 10: Randomly selected examples of patterns discovered by the IMGEP-OGL algorithm dur- ing a single exploration with 5000 iterations.
18

---

Published as a conference paper at ICLR 2020
PCA
T-SNE
RGS
HGS
PGL
OGL
dead
non-animal
animal
Figure 11: PCA and I-SNE visualization of the goal spaces for the IMGEP variants show that HGS has more area (PCA) and clusters (t-SNE) for non-animals compared to learned goal spaces (PGL and OGL) and vice versa for animals. t-SNE shows that the fand-defined goal space (HGS) and learned goal spaces (PGL and OGL.) structure and cluster more the discovered patterns compared to random goal space (RGS).
19

---

Published as a conference paper at ICLR 2020
B IMPLEMENTATION DETAILS AND HYPERPARAMETER SETTINGS
This section provides implementation details and hyperparameter settings of all experiments. The content is organized in the following manner:
B.1
Settings of Lenia
B.2
Description of the classifiers for dead, animal and non-animal Lenia patterns
B.3
Description of the statistical measures about Lenia patterns used as features for the
IMGE-HHSS goal space and the analytic behavior space.
B.4
Sampling mechanisms for Lenia's initial state via CPPNs and Lenia's other dynamic
parameters.
B,5
Details about the IMGEP-HGS
B.6
Details about IMGEPs utilizing deep variational autoencoders (RGS, PGL, OGL).
Description of the measurement method of diversity in the analytic parameter and behavior space
B.7
B. LENIA SETTINGS
A full description of Lenia can be found in (Chan 2019). For all experiments the following config- urations of Lenia were used:
Grid size: 256 256 (A [0, 1]256x256)
Number of steps: M 200
(expp-((ux)) (-(u-pe
Exponential growth mapping: G(u; ((,; 2exp
20
Exponential kernel function: Ko(r) = exp (a I-((-T)) with =
Kernel shell: ()( = BL (Br mod 1), with = (B1, B2, (3)
The controllable parameters of Lenia are the kernel size R E N, time step T E N, E R and a E R that control the growth mapping, and B1, B2, B3 E that control the kernel shell. Additionally the initial state A E [0, 1 2266x 266 controls the system dynamics.
B.2 CLASSIFICATION OF LENIA PATTERNS
We classified 3 types of Lenia patterns: dead, animals and non-animals. The categories were used to analyze if the exploration algorithms showed differences in their behaviors by identifying different types of patterns. The classifiers only categorize the final pattern into which the Lenia system morphs after M 200 time steps.
Dead Classifier: For dead patterns the activity of all cells are either 0 or 1.
Animal Classitier: A pattern is classified as an animal if it is a finite and connected pattern of activity. Cells v, are connected as a pattern if both are active (A(r) 2 0.1 and A(y) 2 0.1) and they influence each other. Cells influence each other when they are within their radius of the kernel K as defined by the parameter R.
Furthermore, the connected pattern must be finite. In Lenia finite and infinite patterns can be differ- entiated because the opposite borders of Lenia's 's cell grid are connected, so that the space is similar ball surface. Thus, pattern can loop around this surface making infinite. We identify infi- nite patterns by the following approach. First, all connected patterns are identified for the case of assuming an infinite grid cell, i.e. opposite grid cell borders are connected. Second, all connected patterns for the case of finite grid cell, i.e. opposite grid cell borders are not connected, are iden- tified. Third, for each border pair (north-south and east-west) it is tested cells within a distance of R from both borders exists, that are part of connected pattern for the infinite and finite grid cell case. such a pattern exists than it is assumed to be infinite, because it loops around the grid cell surface of Lenia (Fig. 12 a). All other patterns considered be finite (Fig. 12] b). Please
20

---

Published as a conference paper at ICLR 2020
that this method has drawback. It can not identify certain infinite pattems that loop several borders, for example, if a pattern connects the north to east and then the west to south border (see the third animal in Fig. for example).
Moreover, there are two additional constraints that an animal pattern must fulfill. First, the cells of the connected pattern P {01.-.-- Zn,) must have at least 80% of all activation, i.e. Exep A Z 0.8 My Aty). Second, pattern must exists for the last two time steps Both constraint are used to avoid that too small patterns or chaotic entities which change drastically between time steps are classified as animals. See Figs.
M and / = M 1).
and To examples of animal patterns.
Non-Animal Classifier: We also classified non-animal patterns which are all entities that were not dead and not an animal. These patterns spread usually over the whole state space and are connected via borders. See Figs. 6 and for examples of non-animal patterns.
B.3 STATISTICAL MEASURES FOR LENIA PATTERNS
We defined five statistical measurements for the final patterns A f that emerge in Lenia. The measures were used as features for hand-defined goal spaces of IMGEPs and define partly the analytic behavior space in which the results of the exploration experiments were compared.
Activation mass MA Measures the sum over the total activation of the final pattern and normalizes it according to the size of the Lenia grid:
1 A T2
MA
where 1,2 256 256 is the number cells of the Lenia system.
Activation volume VA: Measures the number of active cells and normalizes it according the size of the Lenia grid:
VA L (vz A
with 10
Activation density D Measures how dense activation is distributed average active cells:
MA
DA
VA
Activation asymmetry Ax: Measures how symmetrical the activation is distributed according to an axis that starts in the center of the patterns activation mass and goes along the last movement direction of this center. This measure was introduced to especially characterize animal patterns. The center of the activity mass is usually also the center of the animals and analyzing the activity along their movement axis measures how symmetrical they are.
As first step, the center of the activation mass is computed for every time step of the Lenia simu- lation and the Lenia pattern recentered to this location. This ensures that the center all the time
(a) Infinite Pattern
(a) Finite Pattern
infinite segmentation
finite segmentation
infinite
finite segmentation
pattern
pattern
segmentation
Figure 12: Classification of finite and infinite patterns. Infinite patterns form loops between image borders which are identified segment is connected between two borders the infinite and finite segmentation. Finite patterns form no loops. They have connected segments between borders in the infinite not finite segmentation. Segments are colorized yellow, green and blue.
21

---

Published as a conference paper at ICLR 2020
conrectly computed in the case the animal moves and reaches one border to appear the opposite border in the uncentered pattern. The center (zt 7) for time step t is calculated by:
Moo
Moi
(i, f) =
with Mpg ΣΣ
x" y (x,,))
Moo
Mon
where Mpg measures the image moment (or raw moment) of order (p + q) for p, E N.
Based on the center (.. D) the pattern A recentered Al by shifting the and y indexes
according to the center:
1z( A'((2 Z) mod L, (y mod
(1)
where is width and length of the Lenia grid and the indexing x, y 0, .... L 1, After each time step the center is recomputed and the pattern recentered:
2
renenter
Lenia step
recenter
Lenia step
Please note, the simulations and all figures of patterns in the paper are done with the uncentered pattern. The centered version is only computed for the purpose statistical measurements.
The recenter step by (ā, f) defines also the movement direction of the activity center:
(ma, myye ) (i, 1). - (s.i ( = 2.iid y midd)
where x mid gid = L-1 are the coordinates for the grid's middle point. A line can be defined that starts in the midpoint (I yid of the final centered pattern A5 and goes in and opposite to the movement direction of the activity mass center (mz, my)t= M This line separates the grid two equal areas. The asymmetry is computed by comparing the activity amount in the grid right Mright and left M of the line. The normalized difference between both sides is the final measure:
2
A
1 MA
(M M
Activation centeredness Ca: Measures how strong the activation is distributed around the activity mass center:
Way AC (I, with Wey -01111111
d(x, 3)
ΣΣ CA MA
ma88 d(x,
2
where d(r, y) (x rumidd j2 (y - ymid)a is the distance from the point (x, y) to the center point ma midd) At-M(x y) is the centered activation that is updated every time step as for the asymmetry measure (Eq. M. The weights Way decrease the farer point is from the center. Thus, patterns that are concentrated around the center have a high value for CA close to 1. Whereas, patterns whos activity is distributed throughout the whole grid have a smaller value. For patterns that are equally distributed (Va.a! A(x) = A(x') is CA = 0 defined as centeredness measure.
B.4 SAMPLING OF PARAMETERS FOR LENTA
All algorithms explore Lenia by sampling the parameters 9 that control Lenia The parameters comprised of the initial pattern A and the parameters which control the dynamic behavior (R, T, j. a, 31, 32, 3z). There are two operations to sample parameters: 1) random initialization and 2) mutating existing parameter 0. CPPNs are used for the random initialization and mutation of the initial pattern A==1 The details of this process are described in the next section. Afterwards, the initialization and mutation of Lenia's 's parameter that control its dynamics are described.
B.4.1 SAMPLING OF START PATTERNS FOR LENIA VIA CPPNS
Compositional Pattern Producing Networks (CPPNs) recurrent neural networks (Stanley 2006) which we to generate and mutate the initial state Lenia - The CPPNs generate Lenia activity patterns cell by cell by taking input bias value, the and y coordinate of the cell (mapped to
22

---

Published as a conference paper at ICLR 2020
CPPN:
Pattern:
(x. 3
1- lp
y
-2
-2
2
Figure 13: CPPNs recurrent neural networks. Their input is bias of 1, the æ and y coordinate of grid cell and its center distance d. Their output is the activity value of the grid cell.
2- and 2)) and its distance to the grid center (Fig. 13 Their output defines the activity of the cell (A(x. y) - (pl) between and for the given (I.y) coordinate.
CPPNs consist of several hidden neurons (typically 4 to 6 in our experiments) that can have recurrent and self connections. Each CPPN has one output neuron. Two activation functions were used for the hidden neurons and the output neuron. The first is Gaussian and the second is sigmoidal:
gauss(r) = eexp (-(2.5x)² 1,, sigm(x) = 1 exp( 5z)
(2)
To randomly initialize a Lenia initial pattern A a CPPN is randomly sampled by sampling the number of hidden neurons, the connections between inputs and neurons and neurons to neurons, their connection weights and the activation functions for neurons. Afterwards the initial pattern is generated by it. In the history H. of the IMGEPs is then the CPPN part of the parameter added. If the parameter is mutated, then the weights, connections and activation functions of the CPPN are mutated and the new initial pattern A generated by CPPN is defined over network structure (number of nodes, connections of nodes) and its connection weights.
We used the neat-pythor package for the random generation and mutation of CPPNs. It is based on the NeuroEvolution of Augmenting Topologies (NEAT) algorithm for the evolution of neural networks (Stanley & Miikkulainen 2002). The meta-parameters for initialization and mutation of CPPNs listed in Table [ The random sampling and mutation of CPPNs allows to generate complex patterns as illustrated in Fig. 14 Please note, the neat-python package allows also the setting and mutation of response and bias weights for each neuron. Those settings were used
Initialization
Mutation
2d Mutation
3rd Mutation
Mutation
5th Mutation
O
Figure 14: CPPNs can generate complex patterns via their random initialization and successive
mutations. Each row shows generated patterns by CPPN and its mutations.
23

---

Published as a conference paper at ICLR 2020
Parameter
Value
Initial number of hidden neurons
4
Initial activation functions
gauss, sigm random connections with probability 0.6
Initial connections
Initial synapse weight
Gaussian distribution with μ 0, 0.4
Synapse weight range
[-3,3]
Mutation neuron add probability
0.02
Mutation neuron delete probability
0.02
Mutation connection add probability
0.05
Mutation connection delete probability Mutation rate activation functions Mutation rate of synapse weights Mutation replace rate synapse weights
0.01
0.1
0.05
0.06
Mutation power of synapse weights œM Mutation enable/disable rate of synapse weights
1
0.02
Table 1: Settings for the sampling of CPPNs to generate Lenia's 's initial states.
for the experiments. Moreover, we adjusted the sigmoid and Gaussian function in the neal-python package to the ones defined in Eq. 2 to be able to replicate similar images as in Stanley (2006).
B.4.2 SAMPLING OF LENIA'S DYNAMIC PARAMETERS
The parameters that control the dynamics of Lenia (R, T, p4, a, 82, By) are initialized and mutated via uniform and Gaussian distributions. Table for each parameter the meta parameters for their initialization and mutation. Each parameter is initialized by uniform sampling 0g U(a, b) with a and b as upper and lower border. An existing parameter 0; mutated by the following equation:
0, < [0, + NO0 am)
where am is the mutation power and [r:] min(max(mn) a), the clip function. For natural numbers 0g E N the resulting value is rounded.
Parameter
Type
Value Range
Mutation оM
[2, 20
0.5
T
N
[1, 20
0.5
IR
[0, 1]
0.05
[0.001, 0.3]
0.01
$1.
12, B3
R
[0, 1]
0.05
Table 2: Settings for the initialization and mutation of Lenia system parameters 6.
B.5 IMGEP-HGS DETAILS
The 5 statistical features that are given in Appendix B.3 were used to define the goal space of the IMGEP-HGS approach (Algorihm 2). Goals this space were sampled from a uniform distribution within the ranges defined in Table
Feature
min
max
Feature
min
max
mass M
0
asymmetry A centeredness CA
-1
volume VA
density DA
0
1
Table 3: HGS Goal Space Ranges
24

---

Published as a conference paper at ICLR 2020
B.6 IMGEPS WITH DEEP VARIATIONAL AUTOENCODERS
B.6.1 VAE FRAMEWORK AND IMPLEMENTATION DETAILS
We applied deep variational autoencoders (VAEs) to learn latent representations of Lenia patterns. VAEs have two components: a neural encoder and decoder. The encoder q(z[x, x) represents given data point in a latent representation data point by representative distribution in the latent space of reduced dimension d. standard Gaussian prior p(z) (0, I) and diagonal Gaussian posterior q x) (N, σ)) used. Given data point z, the encoder outputs the mean μ and variance in the latent space. The decoder p(x z, v) tries to reconstruct the original data x from sampled latent representation z for the distribution given by the encoder. Under these assumptions, training done by maximizing the computationally tractable evidence lower bound (with 8 = 1):
In variational approaches the encoder describes
the representative distribution
C(x, +b (x(2)] (x (x)x) P -3 (xllax()(x)]lptl
The first term (a) represents the expected reconstruction accuracy while the second (b) is the KL. divergence the approximate posterior from the prior.
b
(4)
In particular, used the 3- framework (Higgins et al. 2017) which re-weights the b term by factor 3 > 1, aiming to enhance the disentangling properties of the learned latent factors. The weight factor set to 3 5 for all experiments. All 3-VAEs used for this paper the same architecture (Table 4). The encoder network has as input the Lenia pattern and as outputs for each latent variable z, the mean (4; and log-variance log(o?). The decoder takes input during the training for each latent variable a sampled value 2; ~ A NJ4,, of). For validation runs and the generation of all reconstructed patterns shown in figures the decoder takes the mean 2; #4 as input. Its output is the reconstructed pattern.
The training objective (Bq. results in the following loss function for batch:
d
Loss( 2. a +
b;
(5)
where r are the input patterns, å are the reconstructed patterns, μl, a are the outputs of the decoder network and is the number of latent dimensions. The reconstruction accurray part a the loss given by a binary cross entropy with logits:
N
2
a
ΣΣ N
(zj,n log (ā)..) ) + (1 -jj.)) log( - ((tj,))))
where the index j for the single cells (pixel) of the pattern and n for the datapoint in the current batch, N is the batch size and (xx
I- The KL divergence terms 6; are given by:
N
1
b.
Σ
(o + př -oog(o?,) -1)
2 N
Encoder
Decoder
Input pattern 256 x 256 x
Input latent vector x: 8
Cony layer: 32 kemels
stride 2, 1-padding ReLU
FC layers 256 ReLU, 16
ReLU
Cony layer: 32 kemels 4 x 4, stride 2, -ppdddng + ReL.U
TransposeConv layer: 32 kemells 4 x 4, stride 2, 1-padding ReLUU
Conv layer: 32 kemels
striee 2, 1-padding + ReLU
TransposeConv layer: 32 kernels
stride 2, 1-padding ReLU
Cony layer: 32 kemels
striee 2. 1-padding ReLU
TrunsposeConv layer: 32 kernels
stride 2, I-padding ReLU
FC layers: : 256 ReLU, 256 ReLUU FC: 2
TransposeConv layer: 32 kemels 4 x stride 2,
1-padding
Table 4: 3-VAE architecture for the pretrained and online experiments.
25

---

Published as a conference paper at ICLR 2020
All 3- VAES were trained for 2000 epochs and initialized with pytorch default initialization. We used the Adam optimizer (Kingma & Ba 2014) (Ir = 1e-3, 31 = 0.9, B2 = 0.999, = -e--,, 8, weight decay=le-5) with a batch size of 64.
The patterss from the datasets were augmented by random x and y translations (up to half the pattern size and with probability 0.3), rotation (up to 40 degrees and with probability 0.3), horizontal and vertical flipping (with probability 0.2). The translations and rotations were preceded by spherical padding to preserve Lenia spherical continuity.
B.6.2 IMGEP VARIANTS
IMGEP-RGS (random goal space): IMGEP with a goal space defined by an encoder network with random weights (Algorithm 2 The weights were initialized according to the Xavier method Glloro & Bengio 2010 The network architecture the encoder is the same that the one of the VAEs used for IMGEP with learned goal spaces. In the other IMGEP algorithms (HGS/PGL/OGL) the goals are sampled uniformly within fixed-range boundarics that are chosen in advance. However, in the case of random goal spaces, do not know in advance in which region of the space goals will be encoded. Therefore, we set the sampling range for each latent variable to the minimum and maximum value of the latent representations of all so far explored patterns.
IMGEP-PGL (prelcarned goal space): IMGEP with a goal space defined by a 3-VAE that was trained before the exploration starts (Algorithm 2 The 3-VAE was trained on a dataset consisting of 558 precollected Lenia patterns (training: 75%, validation: 10%, testing: 15%). Half of the patterns (279) were manually identified animal patterns by Chan (2019). The other half (279) were randomly initialized CPPN patterns (see Section [B.41). The best 3-VAR model obtained during training (highest accuracy on the validation data) was used as goal space for the exploration. During the exploration, goals were uniformly sampled in a [-3, 3J hypercube. These values were chosen because the encoder of the 3-V is trained to match a prior standard normal distribution. Therefore, we can assume that most area of the covered goal space will fall into that hypercube.
IMGEP-OGL (online learned goal space): IMGEP with a goal space that was online learned during the exploration via an 3-VAE (Algorithm 1). Every K 100 explorations the 3-VAE model was trained for 40 epochs resulting in 2000 epochs in total (less if there is not enough data after the first T runs to start the training). The datasets were constructed by collecting non-dead patterns during the exploration. One pattern in every ten is added to the validation set (10%). All others are used for the training set. At the initial period of training, the training dataset has approximately 50 patterns and at the last period approximately 3425 patterns. The validation dataset only serves for checking purposes and has no influence on the learned goal space. Importance sampling is used to give the patterns a different weight during the training. A weighted random sampler is used that samples newly discovered patterns half of the time. Each pattern that has been added to the training dataset during the last period of 100 explorations has a probability of 0.5 N to be sampled (N is the 0.5 total number of new patterns in the dataset). Older patterns are with probability [Dy]-N- As a result, newer discovered patterns have a higher weight and stronger influence on the training.
Algorithm 2: IGGEP-HGS, IMGEP-RGS, and IMGEP-PGI.
Initialize goal space encoder R with handdefined features (HGS), random weights (RGS), or pretrained weights (PGL)
for i 4- 1 to N do if < Nenit then
// Initial random iterations to populate H
L Sample 0 ~ (())
else
// Intrinsically motivated iterations
Sample goal g G(H) based on space represented by R
Choose [T H)
Perform an experiment with 0 and observe o Append (o, R(o)) to the history H
26

---

Published as a conference paper at ICLR 2020
B.6.3 VAE TRAINING RESULTS
The 3- 3-VAE learning saturates for both the precollected dataset and the online collected dataset (Fig. 15). Their ability reconstruct patterns based the encoded latent representation is also qualitatively similar. For both datasets the 3-VAEs are able to learn the general form of the activity pattern (Fig. [T6]. Nonetheless, the compression of the images to 8 8-dimensional vector results in general blurriness in the reconstructed patterns. As result, the 3-VAEs are not able to encode finer details and textures of patterns.
IMGEP-PGL. V-E
IMGEP-OGL- B- 3-VAE
40k
30k
60k
loss
20k
40k
20k
500
1000 epoch
1500
500
1000
1500
epoch
Figure 15: experiments.
Averaged learning curves (n 10) of the 3-VAEs for the IMGEP-PGL and OGL
a) Reconstruction Examples of the 3-VAE used for IMGEP-PGL
b) Reconstruction Examples of the 3-VAE used for the IMGEP-OGL.
Figure 16: Examples of patterns (left) and their reconstruction (right) by 3-VAE networks used for the IMGEP-PGL. (a) and OGL (b). The patterns are sampled from their validation dataset. For the PGL (a) the dataset composed of animal patterns (row 1) from Chan (2019) and randomly generated CPPN patterns (row 2). For the OGL (b) it is composed of animal patterns (row 1) and non-animal patterns (row 2).
27

---

Published as a conference paper at ICLR 2020
B.7 MEASUREMENT OF DIVERSITY IN THE ANALYTIC PARAMETER AND BEHAVIOR SPACE
B.7.1 DIVERSITY MEASURE
Diversity is measured by the area that explored parameters or observations cover in their respective
spaces. The parameter space consisted of the initial start state of Lenia (A'=1 E [0, 1 |255 [ 256 and the settings for Lenia's s dynamics (Rt T, n4, 31, 32, Bz). The space consist therefore of 256 dimensions, each for a single grid cell the initial pattern, plus dimensions for the dynamic settings. The observation space consists of the final patterns A * [0, 1] 1|256x266 resulting in 256² dimensions for the space. Each single exploration results in new point in those spaces.
The diversity measures how much area the algorithms explored in those spaces (Fig. 17 The mea- surement is done by discretizing the space with a spatial grid and counting the number of discretized areas in which at least point falls. For the discretization minimum and maximum border is defined for each space dimension. Each dimension is then split in equally sized bins between those borders. The areas with values falling below the minimum or above the maximum border counted as two additional bins.
The number of dimensions of the original parameter and observation space are too large measure diversity, because the initial and the final pattern have 256² dimensions. We constructed therefore an analytic parameter and behavioral space where the latent representations of 3- 3-VAE were used to reduce the nigh-dimensional patterns to 8 dimensions. 5 bins (7 with the out of range values) per dimension were used for the discretization of those spaces for all experiments in the paper.
Figure 17: Illustration of the diversity measure in a two-dimensional space. Ranges for the dimen- sions are [-5, 5] and [O, 0.3]. The number bins per dimension resulting in 72 49 discretized areas. The diversity is the number of areas (here 12) in which points exist (grey areas).
B.7.2 ANALYTIC PARAMETER AND BEHAVIOR SPACE
The analytic parameter space constructed by the 7 Lenia parameters that control its dynamics and latent representation dimensions of a 3- (Table The 3- VAE was trained on initial pat- terns A used during the experiments. The analytic behavior space was constructed by combining the 5 statistical measures for final Lenia patterns (Section B.3) and also 8 latent representation di- mensions of 3-VAE (Table 5). This 8-VAE was trained on final patterns -4=2000 observed during the experiments. The datasets for both 3-VAEs were constructed by randomly selecting 42500 pat- terns (37500 as training set, 5000 as validation set) from the experiments of all algorithms and each of their 10 repetitions. The 3- VAES used the same structure, hyper- parameters, loss function and
Analytic Parameter Space Definition
Analytic Behavior Space Definition
Parameter
min
max
Parameter
min
max
20
mass MA
T
2
10
volume VA
0
0
1
density DA
0
0.3
asymmetry AA
-1
1
B1, B2 33 B-VAE latent 1 to 8 (trained
0
1
centeredness CA
3-VAE latent 1 to (trained on final patterns A -*=M
-5
5
-5
5
on intial states 1=11)
Table 5: Features of the analytic parameter and behavior space with their min and max values.
28

---

Published as a conference paper at ICLR 2020
a) Reconstruction Examples of the Analytic Parameter Space 3- 3-VAE
b) Reconstruction Examples of the Analytic Behavior Space 3- VAE
Examples of patterns (left) and their reconstruction (right) by the 3-VAE used for the analytic parameter (a) and behavior space (b). The patterns are sampled from the validation dataset.
Figure 18:
learning algorithm described in Section B.6 They were trained for more than 1400 epochs. The encoder which resulted the minimal validation set error during the training was used. According to their reconstructed patterns they represent the general form of patterns but often not individual details such as their texture (Fig. 18).
B.7.3 DEPENDENCE OF DIVERSITY ON THE NUMBER BINS PER DIMENSION
The diversity measure has as parameter the number bins per dimension. We analyzed the influences of this parameter on the diversity measure (Fig. 19 Although the diversity difference between algorithms depends on the number of bins per dimension for each space, the order of the algorithms generally invariant to it. Only if the number of bins per dimension grows large (> 10) the order of the algorithms changes in some cases. The order starts to follow the same order as seen for the proportion of identified patterns (Fig. (5.. In this case the discretization of the space becomes too fine. Fach pattern falls into its own discretized area. We chose therefore a smaller number of bins per dimension of compare the algorithms in a meaningful way.
(a) Diversity in Parameter Space
(b) Diversity in Behavior Space
5000
5000
4000
4000
diversity
diversity
3000
Random
3000
IMGEP-RGS
2000
2000
IMGEP-HGS
IMGEP-HGS
1000
IMGRP-PGI.
1000
Random
IMGEP-PGL
IMGEP-OGL
IMGEP-RGS
IMGEP-OGI.
20
25
15
bins per dimension
bins per dimension
(c) Behavior Space Diversity for Animals
(d) Behavior Space Diversity for Non- Animals
3000
diversity
diversity
Random
1000
IMGEP-RGS
2000
IMGEP-HGS
IMGEP-HGS
500
IMGRP-PGI
1000
Random
IMGEP-PGL
IMGRP-OGI.
IMGEP-RGS
IMGEP-OGI
5
20
25
10
15
20
bins per dimension
bins per dimension
Figure 19: Influence of the number of bins per dimensions on the diversity measure. Depicted is the average diversity (n 10) with the standard deviation as shaded area.
29


Reviews:
[{"review_id": "ryg4ZdOhtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper uses the continuous Game of Life as a testing ground for algorithms that discover diverse behaviors. The problem is interesting, under-explored, and rich. The  combines a variety of interesting ideas including compositional pattern producing networks (CPPNs) to learn structured primitives. Although the authors do propose formal measures of behavioral diversity and so show performance improvements, at the end of the day this work, like much empirical work on generative adversarial networks, is drifting towards art -- where performance is ultimately judged by human eyes rather than quantiative metrics. \n\nComments:\nThe paper refers to hand-designed goal spaces and talks, on p28, about “the statistical measures used to define the goal space”. At the same time, the analytic behavior space is also defined in terms of statistical measures, but it is *not* referred to as hand-designed. At this point, the profusion of spaces and measures means that I am no longer sure what counts as hand-crafted or not. Please clarify.\nThe hypothesis on p34, sec E.4.2 that the VAE’s 8-dim bottleneck helps focus on animals rather than non-animals (which are differentiated more in terms of textures and details) is important and should be checked. \nSome of the decisions about what to check and vary are unclear. For example, section E.1 considers the effect of different initializations (“pytorch”, “xavier” and “kaiming”). The choice of initialization is important mostly to do with improving gradients to improve the rate of convergence (or convergence at all) in deep nets. It’s not clear why initializations are an parameter to vary when considering diversity of solutions. Or, rather, why initializations are more interesting to consider various other architectural considerations. More broadly, looking at Fig 17, the x-axis doesn’t make much sense. The experiments along the x-axis vary according to initialization, but also according to the nature of the goal space and other features. It seems a bit incoherent. \n\nOverall I think this is a good paper. The results are novel and even better, they are fun. However, the paper is extremely long, and it feels as though the authors have to some extent lost control of the material. I could add more comments but TL;DR it needs a lot of editing and pruning. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Hyez9WiNcr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes an algorithm to find diverse patterns in Lenia (a continuous CA system) by using a CPPN to generate initial states, and a stochastic exploration algorithm to mutate parameters of the CPPN + CA parameters. The fitness is the closeness of a generated set of latents to a set of latents produced through one of several possible processes; hand-design, pretraining, or online training on previously generated CA settings. \n\nThe core results are in Figures 27 to 31 in an appendix. Initial inspection reveals that handdesigned goal states produce the most interesting non-animal patterns. With regard to animal forms, it appears to me that Online goal learning harms the diversity of animal forms considerably compared to PGL and perhaps HGS. High frequency spatial structure seems to be lost there. \n\nI would like to see a further analysis of maybe 10000s of such images generated, and an understanding of exactly why RGS produces the same kind of red linear patterns, and why HGS produces the distribution of pattern types in Figure 29, and why non-animal types differ in PGL vs HGS, and why high frequency spatial structure is lost in OGL. How robust are these over many runs? The results should NOT be shown just for the first repetition of the experiment but for all independent runs of the experiments, e.g averaged over 30 independent CPPN evolutions, for PGL, OGL, Random, and HGS! \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BJxu9RJAYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The focus of the presented paper is on formulating the automated discovery of self-organized patterns in high-dimensional dynamic systems. The introduced framework uses cellular automata (game of life) as a testbed for experimentation and evaluation and existing machine learning algorithms (POP-IMGEPs). The goal of the paper is to show that these algorithms can be used to discover and represent features of patterns. Moreover, an extension of SOTA algorithms is introduced and several approaches to define goal space representations are compared. \n\nOverall, I have the impression this is an interesting paper that could be accepted to ICLR. The idea of applying IMGEPs to explore parameters of a dynamic system is novel and interesting, which could also simulate further research in this field. Furthermore, the paper well-written, technically sound, and the results are interesting. The overall contribution of the paper is in applying IMGEP algorithms to exploring parameters of dynamic systems and in comparing different algorithms along with an extensive set of experiments. As a point of criticism, a lot of (interesting) material was pushed to the Appendix. Resolving the references makes reading the paper harder. Moreover, given that this paper has more than 35 pages appendix material, it seems this work would better be suited for a journal as for a conference. There is a reason for papers to have a page limit and this work circumvents this limit by presenting a lot of additional material. Therefore, I am not willing to strongly support this work. \n\nSpecific Comments: \n\n- Section 3.1: It is not clear how the initial system state is established. In Section 3.1. the text states that 'parameters are randomly sampled and explored' before the process starts, but it is not clear why a random sampling is used and what this means for the subsequent sampling. Later in the text (3.3) it becomes more clear, but here this appears too unclear.\n-  Section 3.1: \"distribution over a hypercube in \\mathcal{T} chosen to be large enough to bias exploration towards the frontiers of known goals to incentivize diversity.\" This sentence is not clear and needs more details. How is the distribution chosen exactly?\n- Section 3.2 appears a bit repetitive and could be more concise. I don't think it is necessary here to contrast manual vs learned features of the goal space.\n- Section 3.2 (P3): the last sentence of this paragraph reads as if there exists no approaches for VEAs in online settings. This should be toned down or backed up by a reference.\n- Section 3.2: (last sentence): it is not clear how the history is used exactly to train the network. Which strategy is used to sample from the history of observations?\n- Section 3.3: What is meant by \"The CPPNs are used of the parameters \\{theta}\"? The details provided after this sentence are not clear and need more details. \n- Section 4.2: Please provide more details what \"very large\" dataset means.\n- Section 4.2: 'HGS algorithm' is not defined.\n- Section 5: It seems unnecessary to explain what t-SNE does as a method. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a framework for automatically detecting diverse, self-organized patterns in a continuous Game of Life environment, using compositional pattern producing networks (CPPNs) and population-based Intrinsically Motivated Goal Exploration Processes (POP-IMGEPs) to find the distribution of system parameters that produce diverse, interesting goal patterns.

This work is really well-presented, both in the paper and on the associated website, which is interactive and features source code and demos. Reviewers agree that it’s well-written and seems technically sound. I also agree with R2 that this is an under-explored area and thus would add to the diversity of the program.

In terms of weaknesses, reviewers noted that it’s quite long, with a lengthy appendix, and could be a bit confusing in areas. Authors were responsive to this in the rebuttal and have trimmed it, although it’s still 29 pages. My assessment is well-aligned with those of R2 and thus I’m recommending accept. In the rebuttal, the authors mentioned several interesting possible applications for this work; it’d be great if these could be included in the discussion. 

Given the impressive presentation and amazing visuals, I think it could make for a fun talk.

Decision: Accept (Talk)